path: root/DESIGN

opentracker design notes
========================

Decisions that cost measurement to reach, and the alternatives that were
tried and rejected.

What the tracker is optimised for
---------------------------------

UDP carries most of the traffic. An HTTP request costs about ten times a UDP
announce, 26 microseconds of tracker CPU against 2.5. Nearly all of that is
the kernel setting up and tearing down a connection rather than tracker
code.

A connection carries one request and closes. Keep-alive works and is rarely
asked for.

Statistics are approximate on purpose. Counters are per thread and summed
when read, so a /stats read can under-report once.


Decisions
---------

One UDP socket per worker, where the kernel spreads datagrams over a group
bound to one address. Cost per datagram then stays flat as workers are added,
0.71 to 0.72 microseconds from one worker to eight. Sharing one socket grows
it eightfold over the same range, 0.71 to 6.07.

listen.udp.workers defaults to 1. One worker beats none by 19 to 37 per cent
everywhere, because a blocking read costs one system call where the event
loop costs two. Beyond one the gain depends on the kernel balancing, so
raising it is the operator's decision.

A thread parked in a blocking read is woken by a signal, not by closing or
shutting down its socket:

    mechanism           Linux     FreeBSD   macOS
    close()             blocked   blocked   0 ms
    shutdown(SHUT_RD)   1 ms      1 ms      blocked
    SO_RCVTIMEO         works     works     works
    pthread_kill        1 ms      1 ms      1 ms

Only the signal works everywhere. A receive timeout also works, but wakes
every thread on every tick for the life of the process to catch something
that happens once: at eight workers it doubled idle context switches, 255 to
495 over thirty seconds.

The cleaner is the exception, and waits on a pipe nobody reads. A poll
timeout is an interval rather than a deadline, so no clock enters into it.

Each binary is built in one compiler invocation. Without object files a build
cannot link objects compiled with different FEATURES, which was a real
failure mode. It costs incremental rebuilds, 0.075 seconds becoming 1.56.

A socket is read or written, never both. http_interest_set replaces an
add-and-drop pair so that the invariant cannot be broken by accident, and the
event layer hears about a change once instead of twice.

Accept waits until the client's request has arrived: TCP_DEFER_ACCEPT on
Linux, the accf_data filter on FreeBSD. Measured against a flood of silent
connections, the filter serves more requests than no filter, not fewer, so it
is not the denial of service vector that FreeBSD-SA-02:26 once described.


The torrent table is split into 2^OT_BUCKET_COUNT_BITS buckets, 1024 by
default, set from the Makefile. Inserting a torrent opens a slot in a sorted
vector, so the bytes moved grow with the bucket's occupancy. Insert cost per
torrent, Linux:

                2^10     2^12     2^14     2^16
  200k        466 ns            439 ns
  1M          834 ns   579 ns   529 ns
  4M         2348 ns  1163 ns   770 ns   674 ns

Re-announcing an existing torrent moves by under ten per cent over the same
range, so the cost is paid while the torrent population is built and not in
steady state. Below a few hundred thousand torrents there is nothing to win.
2^14 captures most of the gain for two per cent more resident memory.

The cleaner's wake rate does not follow the bucket count. A cap of ten wakes
a second sets the sleep, and a wake cleans as many buckets as it must to
finish a sweep in OT_CLEAN_INTERVAL_MINUTES. At the default bucket count that
is one bucket every 117 ms.


A refused announce is answered over UDP with a well formed announce for an
empty swarm rather than with the error action. BEP 15 binds a client to the
interval it is handed and says nothing about what a client should do with an
error, so an error puts the client on its own retry schedule instead. Over
five minutes against one blacklisted torrent, libtorrent 2.0.8 sent seven
connects and three announces to a tracker answering with the error and two of
each to one answering with an empty swarm, and was still going on a widening
backoff. Transmission 3.00 retries twenty seconds after an error, then at
five, fifteen, thirty, sixty and finally every hundred and twenty minutes.
That last figure sets OT_CLIENT_REJECT_INTERVAL: at anything under two hours
Transmission's own backoff would be the cheaper of the two in steady state.
Neither client bounds a tracker's interval from above, and libtorrent clamps
it from below at five minutes.

The reject interval is a constant and not a config entry, because the
ordinary announce interval is a constant too and an operator has nothing to
measure a different value against. Over HTTP a refusal stays a failure
reason, which is the only place a user is told why.

The state file keeps a wall clock minute, and it is the only thing in the
tracker that reads that clock. Everything else runs off a monotonic one,
where a time step cannot evict a swarm, but a file outlives the process that
wrote it and minutes since this boot mean nothing to the next run. So the
dump converts on the way out and load_state converts on the way back in, and
what lands on disk is the format opentracker has written for years. A step
between writing and reading mis-dates a torrent by the size of the step,
which costs it part of its last day and nothing else.

Build hardening the shipped binary
----------------------------------

The production and debug binaries carry -fstack-protector-strong, -fPIE and
_FORTIFY_SOURCE=2 on every platform; the GNU-style linkers also get -pie and
full RELRO (-z relro,-z now). Measured cost on
the announce hot path is 300 to 305 nanoseconds, under two per cent, and the
individual flags sit inside the run-to-run noise: add_peer has few stack
buffers, most buffer sizes are known at compile time so FORTIFY emits no
runtime check, and PIE on arm64 is almost free. On x86-64 PIE costs a little
more. End to end it is invisible, since a UDP announce is mostly kernel time.

The link flags -pie and RELRO are gated on the operating system: Apple's
linker rejects -z, and flags a bare -pie in the combined compile-and-link as
unused, while macOS links executables PIE by default, so leaving both off
there costs no hardening. The branch runs in the shell through a != assignment,
the one form both bmake and GNU make 4 accept; $(shell) is GNU only and GNU
make 3.81, which stock macOS ships, has neither. On that old make the != line
is inert and leaves the flags empty, which is the right answer for macOS
anyway, so every platform resolves correctly without a make conditional.

The flags are not in OT_CFLAGS, so the sanitizer test builds do not inherit
them. AddressSanitizer checks harder than FORTIFY does, and the two clash on
FreeBSD, where a FORTIFY plus ASan build aborted a multi hash scrape that the
hardened production binary serves without complaint.

Startup and teardown order
--------------------------

Nothing that consults the access list runs before the list has been read.
The loader used to do its first read inside its own thread, so whatever ran
next saw an empty list: a state file read with -l was checked against it and
a blacklist build restored torrents it should have refused, a whitelist build
restored none at all. Announces arriving in the same window went the same
way, 3089 of them refused over 373 ms against a two million entry whitelist.
The first read is therefore synchronous in accesslist_init, and the thread
only handles reloads.

Nothing that adds a peer starts before the state file has been read. That is
the TCP and UDP listeners and live sync, so main starts all three after
load_state rather than leaving live sync inside trackerlogic_init. A cluster
packet arriving in that window created the torrent, and the saved state for it
was then dropped on the exactmatch add_torrent_from_saved_state bails on,
losing that torrent's completed count.

Teardown stops the cleaner first. It is the one worker that calls into other
modules, accesslist_cleanup and stats_cleanup, so stopping it after them left
it locking a mutex accesslist_deinit had already destroyed: five shutdowns in
sixty at a bucket count low enough to make the window easy to hit.

Signal handlers are installed before the first thread exists. A thread
created earlier runs with whatever disposition a signal still has, and
SIGUSR1, the one that wakes a blocked worker, terminates a process by
default.


Paths not taken
---------------

A keyed hash for the bucket index. Info hashes are SHA-1 output and already
uniform, so it changes no distribution. It would buy resistance to deliberate
bucket flooding and cost the ordering a full scrape depends on: buckets are
ordered by hash prefix and sorted within, so walking them emits the sorted
dictionary bencode wants, with no sort and no copy.

Cache line padding for the per-bucket locks and vector headers. The
contention it was meant to remove is not there. On glibc, where a 48-byte
mutex straddles a line, padding gains 3 to 8 per cent, but gains as much at
one thread where no false sharing can exist, and scaling from one to eight
threads is identical either way. On FreeBSD, where eight 8-byte mutexes share
a line and the risk should be highest, padding is 15 per cent slower at one
thread and neutral to 6 per cent slower at four and eight.

accf_http, the HTTP-aware accept filter. It duplicates a parser the tracker
has anyway, puts more kernel code in the path, and holds a connection that
sends "GET /" and stops, which accf_data hands straight over.

recvmmsg and sendmmsg. No measurable gain on either FreeBSD machine, between
-4 and +10 per cent, and they do not exist on Darwin.

Link time optimisation. It cuts .text by 13 per cent, 93,497 bytes to 81,630,
and changes throughput not at all. A tracker waits on memory and system
calls, not on instruction fetch: the announce path is 6.6 KB of code against
a 32 KB instruction cache at worst.

A directory for object files. bmake has no pattern rules and the Makefile
builds under FreeBSD's make today. Building each binary in one invocation
clears the objects out of the tree without giving that up.



Platform differences that cost us code
--------------------------------------

SO_REUSEPORT means three different things. Linux spreads datagrams over the
group; FreeBSD permits the bind and needs SO_REUSEPORT_LB to spread. Darwin
hands every datagram to whichever socket bound last, so a group there starves
all but one reader.

Waking a blocked reader differs per kernel. see the table above.

Deferring accept is TCP_DEFER_ACCEPT, or accf_data, or nothing.

make is bmake on FreeBSD and GNU make elsewhere, and the Makefile has to
build under both.