Skip to content

fix: per-worker test-port bands so the mutation gate stops crash-scoring (D-033) - #20

Merged
nzneit merged 5 commits into
mainfrom
fix/mutation-gate-port-namespacing
Aug 20, 2026
Merged

nzneit merged 5 commits into
mainfrom
fix/mutation-gate-port-namespacing

Conversation

@nzneit

@nzneit nzneit commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Closes the mutation gate's port-collision nondeterminism (D-033), the open intake item from #19.

The bug, reproduced rather than theorized

Two concurrent bun test processes over one port-heavy file collide: run A passes 68/68, run B dies with EADDRINUSE after a single test. The CI mutation gate runs Stryker at --concurrency 4, so its workers hit this constantly — Failed to start server. Is port 19001 in use? appears throughout #19's job log. A crashed run is scored as a KILLED mutant, so the gate was nondeterministic in both directions and flipped a required check both ways on that PR: three mutants were "killed" with an empty killedBy (crash luck), while a genuinely killable one survived on CI's filesystem.

The fix: kernel-arbitrated sentinel bands

Every port the suite binds in [11800, 19999] now goes through port(base) / portStr(base). Each test process claims a band at preload time by binding a sentinel socket on 127.0.0.1:(11700 + band) — arbitration lives in the kernel's port table, so it needs no cooperation, survives crash/respawn, and works for plain concurrent bun test too, not just Stryker.

Band 0 is the identity map: port(19010) === 19010, so an ordinary local run binds exactly the literals in the source and every debugging habit still works. Bands ≥ 1 relocate each distinct base into a compact 512-wide window.

Why not the alternatives (both rejected with reasons, in D-033)

  • Arithmetic worker-id bands. Stryker forks workers with env: { STRYKER_MUTATOR_WORKER: workerId, ...process.env } — the spread comes after, so a parent env carrying that variable collapses all four workers into one band silently. The id is also a monotonic spawn counter, not a slot index, so any modulus is pigeonhole-doomed.
  • Serialize the gate (--concurrency 1). Measured at ~4x wall time on a path already past the 15-minute job timeout, and it does nothing for the local collision reproduced above. Kept as the documented fallback.

Geometry, and why these numbers

The binding constraint is the Linux ephemeral source-port floor (32768): a band port at or above it can be stolen by a kernel-assigned ephemeral port, reintroducing the exact flake. The top band's last port is 20480 + 23*512 + 511 = 32767. macOS's ephemeral range starts at 49152, so satisfying Linux satisfies macOS with 16k to spare. A dense lazy index is required rather than a modular map — measured over the real port set, mod 512 leaves 29 colliding residues, mod 2048 still leaves 8.

Portability (macOS/BSD)

The sentinel must bind an explicit 127.0.0.1 and never set reusePort: listeners set SO_REUSEADDR by default, and BSD then permits two live binds on one port when the bound addresses differ, where Linux refuses. Deleting the host argument keeps every test green on Linux, so it is pinned at the source level. No Linux-only interface is read at runtime; the claim-failure hint offers both ss and lsof commands.

Bands arbitrate our own processes only — they cannot stop a third party holding a base port (Expo defaults to 19000/19001, which this suite allocates), so OFFBOOK_TEST_BAND=<0..24> forces a band, and the failure message names it.

The orphan rule: probe-and-skip, never a reaper

The sentinel is a process claim, not a port claim. When a mutant run is SIGKILLed on timeout, the OS frees the sentinel but any detached offbook up server it spawned keeps holding that band's ports. So a band is kept only when its sentinel binds and a bind-probe finds that band's spawned-server ports free; otherwise the sentinel is released and the scan moves on. Never a kill: an in-run reaper at concurrency > 1 would SIGKILL sibling workers' live runs, scoring their mutants KILLED — a silent drift toward false green, strictly worse than the EADDRINUSE it replaces.

Enforcement

test/port-hygiene.test.ts makes the scheme a fact rather than a convention: totality (no raw in-window literal anywhere), base accounting, band math, the portability pins, orphan-list completeness, and preload load order. Every gate was proved with a planted violation observed red and then removed — including the shapes that defeated the first draft: a literal declared in a test body, a test.each table, 19_010, 0x4a3a, and arithmetic on a port() result.

One thing the gate then surfaced (not caused by this branch)

With the sibling test files changed, the gate's changed-file selection pulls runfile.ts into scope on its own for the first time — and 11 mutants survived in probeServer. They are pre-existing test gaps, not sweep damage: the runfile.test.ts diff is purely mechanical (every removed line has a port()-wrapped replacement, zero assertion changes), and the mutants decode to the retry semantics — nothing proved the retry happens after a silent first answer, nothing proved it does not happen after a good one, nothing measured its doubled budget — plus an identity shape check no body ever exercised one field at a time. Earlier campaigns always mutated all four modules together, where other coverage masked them; the one test that did kill the shape mutant spawns the CLI as a child process, and child coverage never returns to Stryker, so perTest attribution could never select it.

Fixed with three in-process tests. All seven mutant families were hand-applied and each now dies.

Verification

  • Full gate set by exit code: biome 0, typecheck 0, check-docs 0, bun test 0 — 733 pass / 0 fail.
  • Gate replica at --concurrency 4 (the exact failing condition): pass, score 100.00 — 370 killed, 0 survived, 27 ignored.
  • The original repro harness, 4-way: 4/4 runs 68 pass / 0 fail, zero EADDRINUSE (before: one passed, the rest died).
  • Two independent adversarial audits (desync lens, guarantee lens) over the 20-file sweep: no desyncs; 12 simultaneous claimers took 12 distinct bands. Their 14 findings are all fixed in this branch.

nzneit added 5 commits August 20, 2026 05:14
…ket (D-033)

Two concurrent `bun test` processes over one port-heavy file collide: run A
passes 68/68, run B dies EADDRINUSE after one test. The mutation gate runs
Stryker at --concurrency 4 and scores a crashed run as a KILLED mutant, so the
gate has been nondeterministic in both directions.

Arbitration now lives in the kernel's port table, not in any worker-id scheme.
test/preload.ts imports test/ports.ts first; that module binds a sentinel on
127.0.0.1:(11700 + band) to claim a band, and every port literal in
[11800, 19999] goes through port(base) / portStr(base). Band 0 is the identity
map, so an ordinary local run binds exactly the numbers written in the source.
Bands >= 1 relocate each distinct base into a compact 512-wide window
(COMPACT_BASE 20480, MAX_BAND 24, top port 32767 — below the Linux ephemeral
floor of 32768, which is the binding constraint on macOS too).

- node:net, not Bun.listen: measured on Bun 1.3.14, `unref()` on a Bun.listen
  listener RELEASES the socket, while node:net's keeps the bind and only stops
  the handle holding the event loop open. node:net's listen(port, "127.0.0.1")
  settles the bind synchronously for a literal IP, which the claim requires.
- The sentinel binds an explicit "127.0.0.1" and never sets reusePort: with
  SO_REUSEADDR (default on listeners) BSD/macOS grants two live binds on one
  port when the bound addresses differ, where Linux refuses.
- Probe-and-skip, never a reaper: `offbook up` spawns a DETACHED server, so a
  SIGKILLed run frees the sentinel while the orphan keeps the ports. A band is
  kept only when its sentinel binds AND a bind-probe of DETACHED_BASES finds
  them free; otherwise the sentinel is released and the skip is announced on
  stderr. An in-run reaper at concurrency 4 would SIGKILL sibling workers' live
  runs and score their mutants KILLED.
- DETACHED_BASES pre-seeds every band's index map so those bases map identically
  in every process — the precondition the cross-process orphan probe rests on.
- OFFBOOK_TEST_BAND=<0..24> forces a band and skips the scan, for a machine
  where a third party holds a base port (Expo defaults to 19000/19001).

Sweep rules: one port(BASE) call per DISTINCT value so multi-role clusters stay
equal; ports rendered into asserted text use the same call; out-of-window
contract literals (9001/1883/9080, the fixture sentinels) untouched; comment
digits rewritten in prose, never mechanically offset.
…ith a plant

The scheme's guarantee is totality, and band 0 is the identity map — so every
defect in it is invisible in a local single-process run and bites only in a
second concurrent process. These gates are what make it a fact rather than a
convention. Every rule below was proved by planting the exact violating shape,
observing red, and removing it.

test/port-hygiene.test.ts:
- The timeout exemption is narrowed to the trailing ARGUMENT position of
  test/it/test.each/it.each/setTimeout/setInterval. It previously exempted any
  literal anywhere inside a test body, which swallowed the two most natural
  ways to add a port (`test("x", () => { const WS = 19010; … })` and a
  `{ controlPlanePort: 19470 }` object built in a test). Disabling the narrowed
  exemption shows it now covers exactly the four legitimate 15_000 timeouts.
- Numeric separators and radix prefixes are decoded before the window check.
  `19_010` was never recorded as a literal at all (the old scan dropped any
  run containing `_`), and `0x4a3a` (19002) fell out on the identifier guard.
- Accounting replaces the literal-only count: 187 scanned bases plus five
  declared arithmetic families = the real 317, against BAND_WIDTH 512. The
  declaration is checked in both directions — an undeclared computed port()
  call site fails, and so does a declaration whose call site is gone — plus a
  64-slot margin, because the per-family n-ranges are hand-read.
- PORTABILITY pins the sentinel's explicit "127.0.0.1" and the absence of
  reusePort at the source level (deleting the host argument keeps every test
  green on Linux and only breaks on BSD).
- ORPHANS pins DETACHED_BASES against every --ws-port/--tcp-port/--ctrl-port/
  --port flag site, both directions.
- Arithmetic on a port() RESULT (`port(19010) + 1`) is flagged: it lands on an
  unrelated base's slot in every band but 0.
- Every path is anchored at import.meta.dir. Several suite tests process.chdir,
  and a chdir escaping its finally turned this gate into an ENOENT crash (8
  failures, measured) — which the mutation gate scores as a kill.
- The scan boundary (test/**/*.ts + src/**/*.test.ts, nothing else) is stated
  in the header so the next helper under scripts/ or demo-app/ is a decision.

test/import-style.test.ts fences the new #test/* alias: only *.test.ts files
may use it. It is a src->test edge, and test/ports.ts binds a socket and writes
to stderr as an import side effect (the R-043 log parsers are stderr-sensitive,
D-030).

.github/workflows/mutation.yml: the canary's probe guard was a grep for a
DIRECT import of ports.ts, so a probe importing a helper that transitively
claims a band would keep it green with the preload dropped — the exact
blindness it exists to prevent. It is now a measured negative control: run the
probe under a preload-free config and require silence (plus a non-vacuous pass
count), then run it under the runner-sanitized config and require the band
line. Verified: a one-hop transitive import passes the old grep and fails this.
DECISIONS.md gains D-033: the reproduced collision, the geometry with the
ephemeral-range reasoning (Linux 32768-60999 is the binding constraint; macOS
starts at 49152), band 0 = identity, the BSD SO_REUSEADDR requirement, the
probe-and-skip rule, and the two rejected options with their measurements —
worker-id arithmetic (core spreads ...process.env AFTER setting
STRYKER_MUTATOR_WORKER, so a parent env collapses every worker into one band;
the id is a monotonic spawn counter, not a slot index) and serializing the gate
(conc-1 measured past the 15-minute job timeout where conc-4 finishes in 7.7
min, and it does nothing for the local concurrent run that was reproduced
first).

The intake file is resolved, its Resolution names D-033 and records why the
intake's own recommended option 1 was rejected on measurement, and it moves to
docs/archive/intake/ (check-docs errors on a resolved file left in intake).
D-032's closing pointer follows it to the archive path.

AGENTS.md gains a working note on reading band-mapped ports (band 0 = identity,
port()/portStr() mandatory in 11800-19999, the hygiene gate, OFFBOOK_TEST_BAND
as the escape hatch), extends the internal-imports note with the fenced #test/*
alias, and updates the sandbox-orphan note: a leaked server now makes the next
run skip that band loudly instead of failing inside it.
The mutation gate surfaced these when the port sweep pulled runfile.ts into
the changed-file scope: nothing proved the retry HAPPENS after a silent
first answer, nothing proved it does NOT happen after a good one, nothing
measured its doubled budget, and no body exercised the shape check one
field at a time (a body wrong in every field passes an OR'd check too).
Seven mutant families were hand-applied and each now dies.
… flakes

The gate job was cancelled at its 15-minute backstop on the first CI run.
Cause is the fix working: a mutant run that used to die instantly on
EADDRINUSE was scored KILLED, so collisions made the campaign fast for the
wrong reason. Honest runs finish their covering tests (397 mutants: 5m50s
locally at conc-4, over 15min on a 4-vCPU runner), so the timeout goes to 30.

Also: the retry pin now measures the two attempts as a ratio of each other
instead of absolute milliseconds (60ms rather than a second per covering
run, and it cannot flake on a loaded runner), and the non-repo-cwd test
gets its own state dir — the suite-wide one collects every spawning test's
pointer, and a dangling one prints a reap note that broke its empty-stderr
assertion.
@nzneit
nzneit merged commit 716ec46 into main Aug 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant