fix: per-worker test-port bands so the mutation gate stops crash-scoring (D-033) - #20
Merged
Merged
Conversation
…ket (D-033) Two concurrent `bun test` processes over one port-heavy file collide: run A passes 68/68, run B dies EADDRINUSE after one test. The mutation gate runs Stryker at --concurrency 4 and scores a crashed run as a KILLED mutant, so the gate has been nondeterministic in both directions. Arbitration now lives in the kernel's port table, not in any worker-id scheme. test/preload.ts imports test/ports.ts first; that module binds a sentinel on 127.0.0.1:(11700 + band) to claim a band, and every port literal in [11800, 19999] goes through port(base) / portStr(base). Band 0 is the identity map, so an ordinary local run binds exactly the numbers written in the source. Bands >= 1 relocate each distinct base into a compact 512-wide window (COMPACT_BASE 20480, MAX_BAND 24, top port 32767 — below the Linux ephemeral floor of 32768, which is the binding constraint on macOS too). - node:net, not Bun.listen: measured on Bun 1.3.14, `unref()` on a Bun.listen listener RELEASES the socket, while node:net's keeps the bind and only stops the handle holding the event loop open. node:net's listen(port, "127.0.0.1") settles the bind synchronously for a literal IP, which the claim requires. - The sentinel binds an explicit "127.0.0.1" and never sets reusePort: with SO_REUSEADDR (default on listeners) BSD/macOS grants two live binds on one port when the bound addresses differ, where Linux refuses. - Probe-and-skip, never a reaper: `offbook up` spawns a DETACHED server, so a SIGKILLed run frees the sentinel while the orphan keeps the ports. A band is kept only when its sentinel binds AND a bind-probe of DETACHED_BASES finds them free; otherwise the sentinel is released and the skip is announced on stderr. An in-run reaper at concurrency 4 would SIGKILL sibling workers' live runs and score their mutants KILLED. - DETACHED_BASES pre-seeds every band's index map so those bases map identically in every process — the precondition the cross-process orphan probe rests on. - OFFBOOK_TEST_BAND=<0..24> forces a band and skips the scan, for a machine where a third party holds a base port (Expo defaults to 19000/19001). Sweep rules: one port(BASE) call per DISTINCT value so multi-role clusters stay equal; ports rendered into asserted text use the same call; out-of-window contract literals (9001/1883/9080, the fixture sentinels) untouched; comment digits rewritten in prose, never mechanically offset.
…ith a plant
The scheme's guarantee is totality, and band 0 is the identity map — so every
defect in it is invisible in a local single-process run and bites only in a
second concurrent process. These gates are what make it a fact rather than a
convention. Every rule below was proved by planting the exact violating shape,
observing red, and removing it.
test/port-hygiene.test.ts:
- The timeout exemption is narrowed to the trailing ARGUMENT position of
test/it/test.each/it.each/setTimeout/setInterval. It previously exempted any
literal anywhere inside a test body, which swallowed the two most natural
ways to add a port (`test("x", () => { const WS = 19010; … })` and a
`{ controlPlanePort: 19470 }` object built in a test). Disabling the narrowed
exemption shows it now covers exactly the four legitimate 15_000 timeouts.
- Numeric separators and radix prefixes are decoded before the window check.
`19_010` was never recorded as a literal at all (the old scan dropped any
run containing `_`), and `0x4a3a` (19002) fell out on the identifier guard.
- Accounting replaces the literal-only count: 187 scanned bases plus five
declared arithmetic families = the real 317, against BAND_WIDTH 512. The
declaration is checked in both directions — an undeclared computed port()
call site fails, and so does a declaration whose call site is gone — plus a
64-slot margin, because the per-family n-ranges are hand-read.
- PORTABILITY pins the sentinel's explicit "127.0.0.1" and the absence of
reusePort at the source level (deleting the host argument keeps every test
green on Linux and only breaks on BSD).
- ORPHANS pins DETACHED_BASES against every --ws-port/--tcp-port/--ctrl-port/
--port flag site, both directions.
- Arithmetic on a port() RESULT (`port(19010) + 1`) is flagged: it lands on an
unrelated base's slot in every band but 0.
- Every path is anchored at import.meta.dir. Several suite tests process.chdir,
and a chdir escaping its finally turned this gate into an ENOENT crash (8
failures, measured) — which the mutation gate scores as a kill.
- The scan boundary (test/**/*.ts + src/**/*.test.ts, nothing else) is stated
in the header so the next helper under scripts/ or demo-app/ is a decision.
test/import-style.test.ts fences the new #test/* alias: only *.test.ts files
may use it. It is a src->test edge, and test/ports.ts binds a socket and writes
to stderr as an import side effect (the R-043 log parsers are stderr-sensitive,
D-030).
.github/workflows/mutation.yml: the canary's probe guard was a grep for a
DIRECT import of ports.ts, so a probe importing a helper that transitively
claims a band would keep it green with the preload dropped — the exact
blindness it exists to prevent. It is now a measured negative control: run the
probe under a preload-free config and require silence (plus a non-vacuous pass
count), then run it under the runner-sanitized config and require the band
line. Verified: a one-hop transitive import passes the old grep and fails this.
DECISIONS.md gains D-033: the reproduced collision, the geometry with the ephemeral-range reasoning (Linux 32768-60999 is the binding constraint; macOS starts at 49152), band 0 = identity, the BSD SO_REUSEADDR requirement, the probe-and-skip rule, and the two rejected options with their measurements — worker-id arithmetic (core spreads ...process.env AFTER setting STRYKER_MUTATOR_WORKER, so a parent env collapses every worker into one band; the id is a monotonic spawn counter, not a slot index) and serializing the gate (conc-1 measured past the 15-minute job timeout where conc-4 finishes in 7.7 min, and it does nothing for the local concurrent run that was reproduced first). The intake file is resolved, its Resolution names D-033 and records why the intake's own recommended option 1 was rejected on measurement, and it moves to docs/archive/intake/ (check-docs errors on a resolved file left in intake). D-032's closing pointer follows it to the archive path. AGENTS.md gains a working note on reading band-mapped ports (band 0 = identity, port()/portStr() mandatory in 11800-19999, the hygiene gate, OFFBOOK_TEST_BAND as the escape hatch), extends the internal-imports note with the fenced #test/* alias, and updates the sandbox-orphan note: a leaked server now makes the next run skip that band loudly instead of failing inside it.
The mutation gate surfaced these when the port sweep pulled runfile.ts into the changed-file scope: nothing proved the retry HAPPENS after a silent first answer, nothing proved it does NOT happen after a good one, nothing measured its doubled budget, and no body exercised the shape check one field at a time (a body wrong in every field passes an OR'd check too). Seven mutant families were hand-applied and each now dies.
… flakes The gate job was cancelled at its 15-minute backstop on the first CI run. Cause is the fix working: a mutant run that used to die instantly on EADDRINUSE was scored KILLED, so collisions made the campaign fast for the wrong reason. Honest runs finish their covering tests (397 mutants: 5m50s locally at conc-4, over 15min on a 4-vCPU runner), so the timeout goes to 30. Also: the retry pin now measures the two attempts as a ratio of each other instead of absolute milliseconds (60ms rather than a second per covering run, and it cannot flake on a loaded runner), and the non-repo-cwd test gets its own state dir — the suite-wide one collects every spawning test's pointer, and a dangling one prints a reap note that broke its empty-stderr assertion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the mutation gate's port-collision nondeterminism (D-033), the open intake item from #19.
The bug, reproduced rather than theorized
Two concurrent
bun testprocesses over one port-heavy file collide: run A passes 68/68, run B dies withEADDRINUSEafter a single test. The CI mutation gate runs Stryker at--concurrency 4, so its workers hit this constantly —Failed to start server. Is port 19001 in use?appears throughout #19's job log. A crashed run is scored as a KILLED mutant, so the gate was nondeterministic in both directions and flipped a required check both ways on that PR: three mutants were "killed" with an emptykilledBy(crash luck), while a genuinely killable one survived on CI's filesystem.The fix: kernel-arbitrated sentinel bands
Every port the suite binds in
[11800, 19999]now goes throughport(base)/portStr(base). Each test process claims a band at preload time by binding a sentinel socket on127.0.0.1:(11700 + band)— arbitration lives in the kernel's port table, so it needs no cooperation, survives crash/respawn, and works for plain concurrentbun testtoo, not just Stryker.Band 0 is the identity map:
port(19010) === 19010, so an ordinary local run binds exactly the literals in the source and every debugging habit still works. Bands ≥ 1 relocate each distinct base into a compact 512-wide window.Why not the alternatives (both rejected with reasons, in D-033)
env: { STRYKER_MUTATOR_WORKER: workerId, ...process.env }— the spread comes after, so a parent env carrying that variable collapses all four workers into one band silently. The id is also a monotonic spawn counter, not a slot index, so any modulus is pigeonhole-doomed.--concurrency 1). Measured at ~4x wall time on a path already past the 15-minute job timeout, and it does nothing for the local collision reproduced above. Kept as the documented fallback.Geometry, and why these numbers
The binding constraint is the Linux ephemeral source-port floor (32768): a band port at or above it can be stolen by a kernel-assigned ephemeral port, reintroducing the exact flake. The top band's last port is
20480 + 23*512 + 511 = 32767. macOS's ephemeral range starts at 49152, so satisfying Linux satisfies macOS with 16k to spare. A dense lazy index is required rather than a modular map — measured over the real port set, mod 512 leaves 29 colliding residues, mod 2048 still leaves 8.Portability (macOS/BSD)
The sentinel must bind an explicit
127.0.0.1and never setreusePort: listeners setSO_REUSEADDRby default, and BSD then permits two live binds on one port when the bound addresses differ, where Linux refuses. Deleting the host argument keeps every test green on Linux, so it is pinned at the source level. No Linux-only interface is read at runtime; the claim-failure hint offers bothssandlsofcommands.Bands arbitrate our own processes only — they cannot stop a third party holding a base port (Expo defaults to 19000/19001, which this suite allocates), so
OFFBOOK_TEST_BAND=<0..24>forces a band, and the failure message names it.The orphan rule: probe-and-skip, never a reaper
The sentinel is a process claim, not a port claim. When a mutant run is SIGKILLed on timeout, the OS frees the sentinel but any detached
offbook upserver it spawned keeps holding that band's ports. So a band is kept only when its sentinel binds and a bind-probe finds that band's spawned-server ports free; otherwise the sentinel is released and the scan moves on. Never a kill: an in-run reaper at concurrency > 1 would SIGKILL sibling workers' live runs, scoring their mutants KILLED — a silent drift toward false green, strictly worse than the EADDRINUSE it replaces.Enforcement
test/port-hygiene.test.tsmakes the scheme a fact rather than a convention: totality (no raw in-window literal anywhere), base accounting, band math, the portability pins, orphan-list completeness, and preload load order. Every gate was proved with a planted violation observed red and then removed — including the shapes that defeated the first draft: a literal declared in a test body, atest.eachtable,19_010,0x4a3a, and arithmetic on aport()result.One thing the gate then surfaced (not caused by this branch)
With the sibling test files changed, the gate's changed-file selection pulls
runfile.tsinto scope on its own for the first time — and 11 mutants survived inprobeServer. They are pre-existing test gaps, not sweep damage: therunfile.test.tsdiff is purely mechanical (every removed line has aport()-wrapped replacement, zero assertion changes), and the mutants decode to the retry semantics — nothing proved the retry happens after a silent first answer, nothing proved it does not happen after a good one, nothing measured its doubled budget — plus an identity shape check no body ever exercised one field at a time. Earlier campaigns always mutated all four modules together, where other coverage masked them; the one test that did kill the shape mutant spawns the CLI as a child process, and child coverage never returns to Stryker, soperTestattribution could never select it.Fixed with three in-process tests. All seven mutant families were hand-applied and each now dies.
Verification
bun test0 — 733 pass / 0 fail.--concurrency 4(the exact failing condition): pass, score 100.00 — 370 killed, 0 survived, 27 ignored.