opentracker design notes ======================== Decisions that cost measurement to reach, and the alternatives that were tried and rejected. What the tracker is optimised for --------------------------------- UDP carries most of the traffic. An HTTP request costs about ten times a UDP announce, 26 microseconds of tracker CPU against 2.5. Nearly all of that is the kernel setting up and tearing down a connection rather than tracker code. A connection carries one request and closes. Keep-alive works and is rarely asked for. Statistics are approximate on purpose. Counters are per thread and summed when read, so a /stats read can under-report once. Decisions --------- One UDP socket per worker, where the kernel spreads datagrams over a group bound to one address. Cost per datagram then stays flat as workers are added, 0.71 to 0.72 microseconds from one worker to eight. Sharing one socket grows it eightfold over the same range, 0.71 to 6.07. listen.udp.workers defaults to 1. One worker beats none by 19 to 37 per cent everywhere, because a blocking read costs one system call where the event loop costs two. Beyond one the gain depends on the kernel balancing, so raising it is the operator's decision. A thread parked in a blocking read is woken by a signal, not by closing or shutting down its socket: mechanism Linux FreeBSD macOS close() blocked blocked 0 ms shutdown(SHUT_RD) 1 ms 1 ms blocked SO_RCVTIMEO works works works pthread_kill 1 ms 1 ms 1 ms Only the signal works everywhere. A receive timeout also works, but wakes every thread on every tick for the life of the process to catch something that happens once: at eight workers it doubled idle context switches, 255 to 495 over thirty seconds. Each binary is built in one compiler invocation. Without object files a build cannot link objects compiled with different FEATURES, which was a real failure mode. It costs incremental rebuilds, 0.075 seconds becoming 1.56. A socket is read or written, never both. http_interest_set replaces an add-and-drop pair so that the invariant cannot be broken by accident, and the event layer hears about a change once instead of twice. Accept waits until the client's request has arrived: TCP_DEFER_ACCEPT on Linux, the accf_data filter on FreeBSD. Measured against a flood of silent connections, the filter serves more requests than no filter, not fewer, so it is not the denial of service vector that FreeBSD-SA-02:26 once described. The torrent table is split into 2^OT_BUCKET_COUNT_BITS buckets, 1024 by default, set from the Makefile. Inserting a torrent opens a slot in a sorted vector, so the bytes moved grow with the bucket's occupancy. Insert cost per torrent, Linux: 2^10 2^12 2^14 2^16 200k 466 ns 439 ns 1M 834 ns 579 ns 529 ns 4M 2348 ns 1163 ns 770 ns 674 ns Re-announcing an existing torrent moves by under ten per cent over the same range, so the cost is paid while the torrent population is built and not in steady state. Below a few hundred thousand torrents there is nothing to win. 2^14 captures most of the gain for two per cent more resident memory. The cleaner's wake rate does not follow the bucket count. A cap of ten wakes a second sets the sleep, and a wake cleans as many buckets as it must to finish a sweep in OT_CLEAN_INTERVAL_MINUTES. At the default bucket count that is one bucket every 117 ms. Paths not taken --------------- A keyed hash for the bucket index. Info hashes are SHA-1 output and already uniform, so it changes no distribution. It would buy resistance to deliberate bucket flooding and cost the ordering a full scrape depends on: buckets are ordered by hash prefix and sorted within, so walking them emits the sorted dictionary bencode wants, with no sort and no copy. Cache line padding for the per-bucket locks and vector headers. The contention it was meant to remove is not there. On glibc, where a 48-byte mutex straddles a line, padding gains 3 to 8 per cent, but gains as much at one thread where no false sharing can exist, and scaling from one to eight threads is identical either way. On FreeBSD, where eight 8-byte mutexes share a line and the risk should be highest, padding is 15 per cent slower at one thread and neutral to 6 per cent slower at four and eight. accf_http, the HTTP-aware accept filter. It duplicates a parser the tracker has anyway, puts more kernel code in the path, and holds a connection that sends "GET /" and stops, which accf_data hands straight over. recvmmsg and sendmmsg. No measurable gain on either FreeBSD machine, between -4 and +10 per cent, and they do not exist on Darwin. Link time optimisation. It cuts .text by 13 per cent, 93,497 bytes to 81,630, and changes throughput not at all. A tracker waits on memory and system calls, not on instruction fetch: the announce path is 6.6 KB of code against a 32 KB instruction cache at worst. A directory for object files. bmake has no pattern rules and the Makefile builds under FreeBSD's make today. Building each binary in one invocation clears the objects out of the tree without giving that up. Platform differences that cost us code -------------------------------------- SO_REUSEPORT means three different things. Linux spreads datagrams over the group; FreeBSD permits the bind and needs SO_REUSEPORT_LB to spread. Darwin hands every datagram to whichever socket bound last, so a group there starves all but one reader. Waking a blocked reader differs per kernel. see the table above. Deferring accept is TCP_DEFER_ACCEPT, or accf_data, or nothing. make is bmake on FreeBSD and GNU make elsewhere, and the Makefile has to build under both.