1 files changed, 99 insertions, 0 deletions

diff --git a/DESIGN b/DESIGN
new file mode 100644
index 0000000..ff37706
--- /dev/null
+++ b/DESIGN
@@ -0,0 +1,99 @@
+opentracker design notes
+========================
+
+Decisions that cost measurement to reach, and the alternatives that were
+tried and rejected.
+
+What the tracker is optimised for
+---------------------------------
+
+UDP carries most of the traffic. An HTTP request costs about ten times a UDP
+announce, 26 microseconds of tracker CPU against 2.5. Nearly all of that is
+the kernel setting up and tearing down a connection rather than tracker
+code.
+
+A connection carries one request and closes. Keep-alive works and is rarely
+asked for.
+
+Statistics are approximate on purpose. Counters are per thread and summed
+when read, so a /stats read can under-report once. Nothing drifts.
+
+
+Decisions
+---------
+
+One UDP socket per worker, where the kernel spreads datagrams over a group
+bound to one address. Cost per datagram then stays flat as workers are added,
+0.71 to 0.72 microseconds from one worker to eight. Sharing one socket grows
+it eightfold over the same range, 0.71 to 6.07.
+
+listen.udp.workers defaults to 1. One worker beats none by 19 to 37 per cent
+everywhere, because a blocking read costs one system call where the event
+loop costs two. Beyond one the gain depends on the kernel balancing, so
+raising it is the operator's decision.
+
+A thread parked in a blocking read is woken by a signal, not by closing or
+shutting down its socket:
+
+    mechanism           Linux     FreeBSD   macOS
+    close()             blocked   blocked   0 ms
+    shutdown(SHUT_RD)   1 ms      1 ms      blocked
+    SO_RCVTIMEO         works     works     works
+    pthread_kill        1 ms      1 ms      1 ms
+
+Only the signal works everywhere. A receive timeout also works, but wakes
+every thread on every tick for the life of the process to catch something
+that happens once: at eight workers it doubled idle context switches, 255 to
+495 over thirty seconds.
+
+Each binary is built in one compiler invocation. Without object files a build
+cannot link objects compiled with different FEATURES, which was a real
+failure mode. It costs incremental rebuilds, 0.075 seconds becoming 1.56.
+
+A socket is read or written, never both. http_interest_set replaces an
+add-and-drop pair so that the invariant cannot be broken by accident, and the
+event layer hears about a change once instead of twice.
+
+Accept waits until the client's request has arrived: TCP_DEFER_ACCEPT on
+Linux, the accf_data filter on FreeBSD. Measured against a flood of silent
+connections, the filter serves more requests than no filter, not fewer, so it
+is not the denial of service vector that FreeBSD-SA-02:26 once described.
+
+
+Paths not taken
+---------------
+
+accf_http, the HTTP-aware accept filter. It duplicates a parser the tracker
+has anyway, puts more kernel code in the path, and holds a connection that
+sends "GET /" and stops, which accf_data hands straight over.
+
+recvmmsg and sendmmsg. No measurable gain on either FreeBSD machine, between
+-4 and +10 per cent, and they do not exist on Darwin.
+
+Link time optimisation. It cuts .text by 13 per cent, 93,497 bytes to 81,630,
+and changes throughput not at all. A tracker waits on memory and system
+calls, not on instruction fetch: the announce path is 6.6 KB of code against
+a 32 KB instruction cache at worst.
+
+A directory for object files. bmake has no pattern rules and the Makefile
+builds under FreeBSD's make today. Building each binary in one invocation
+clears the objects out of the tree without giving that up.
+
+A src/ directory. 52 files sharing an ot_ prefix are a module list rather
+than clutter, and build output was most of what made the tree look crowded.
+
+
+Platform differences that cost us code
+--------------------------------------
+
+SO_REUSEPORT means three different things. Linux spreads datagrams over the
+group; FreeBSD permits the bind and needs SO_REUSEPORT_LB to spread. Darwin
+hands every datagram to whichever socket bound last, so a group there starves
+all but one reader.
+
+Waking a blocked reader differs per kernel. see the table above.
+
+Deferring accept is TCP_DEFER_ACCEPT, or accf_data, or nothing.
+
+make is bmake on FreeBSD and GNU make elsewhere, and the Makefile has to
+build under both.