Skip to content

fix: isolate L3 dispatch per composed server (main CI failure root cause) - #18

Merged
nzneit merged 1 commit into
mainfrom
fix/dispatch-singleton-bleed
Aug 13, 2026
Merged

nzneit merged 1 commit into
mainfrom
fix/dispatch-singleton-bleed

Conversation

@nzneit

@nzneit nzneit commented Aug 13, 2026

Copy link
Copy Markdown
Owner

What this PR does

This PR repairs the cause of the main-branch CI failure of 2026-08-13 (run 31668911331). It changes two source files and two test files. Each composed server now has its own L3 dispatch registry.

The failure

The test POST /v1/publish: a valid fromClient publish re-enters onInbound and fires the reactive L2 scenario failed on main. A re-run passed. The test was not the problem.

The cause

The investigation found this chain:

  • A test in dispatch.test.ts loads two handler files. The handlers have no hooks. Their pattern is command/{deviceId}/set. The registrations go into defaultDispatch.
  • defaultDispatch is one object for the full process. Each engine uses it. No function removes a registration.
  • An L3 handler owns its full topic. This is the design. A handler with no hooks makes the engine drop the inbound message. There is no scenario, no violation, and no state entry.
  • Bun runs test files in the order of the filesystem directory listing. Each fresh checkout can have a different order. When dispatch.test.ts runs before control-plane/index.test.ts, the leaked handler eats the control-plane test's message, and the test fails.

The failure is constant for one checkout. It looks random across CI runs because each CI run makes a fresh checkout. A scratch checkout with the bad order failed 12 of 12 runs. The main local checkout has the good order and always passed.

The fix

Two parts. The G11 contract does not change:

  1. dispatch.ts — While loadHandlers imports a handler file, the free register() sends the registration to the registry that does the import, not to the singleton. A direct register() call, with no import in flight, goes to the singleton as before.
  2. compose/index.ts — Each composed server gets its own registry. Two servers in one process can no longer share handlers. This closes a latent product defect, not only the test defect.

The polluting test now uses an isolated registry.

The tests

All new tests were red (or order-dependent) before the fix:

  • A unit test: loadHandlers puts handler-file registrations into the loading registry, and the singleton stays clean.
  • An integration test with the incident shape: server A loads a hook-less handler; server B's scenario must still fire.
  • A contract pin: a direct register() with no load in flight goes to the singleton, also after a load completed. This pin also kills the mutants of the new lines for the mutation gate.

Verification

  • bun scripts/check-docs.ts: exit 0.
  • bun run lint: exit 0.
  • bun run typecheck: exit 0.
  • bun test: 573 pass, 0 fail, exit 0.
  • The proof: a scratch checkout with the bad directory order reproduced the CI failure in each of 12 runs. With this fix, the same checkout and the same order run green.

Note for review

The mutation gate will run on dispatch.ts. The new tests target the new lines. If a mutant survives, it gets handled in this PR.

…use)

The 2026-08-13 main CI failure ('a valid fromClient publish re-enters
onInbound and fires the reactive L2 scenario') was not a timing flake:
dispatch.test.ts's loadHandlers fixtures registered hook-less handlers
on command/{deviceId}/set into defaultDispatch — the process-wide
singleton every composed engine shares — and nothing ever unregisters.
With bun's readdir-dependent test-file order (fresh checkout = fresh
order), the leaked handler could run ahead of the control-plane file
and silently L3-shadow its channel: inbound swallowed, no scenario, no
violation, empty state. Deterministic per checkout, coin-flip per CI
run.

Fix, inside the G11 contract: dispatch.ts routes handler-file
registrations to the registry whose loadHandlers is importing them (an
active-loader slot behind the free register()), and compose gives each
server its own registry — which also closes the latent product bug of
two in-process servers sharing L3 handlers. Direct register() with no
load in flight still targets the singleton (pinned).

Tests (red first): loadHandlers isolation unit test; an incident-shaped
integration pin (server A's handlersDir must not shadow server B's
scenarios); the singleton-contract pin doubling as mutant-kill for the
new lines. The culprit fixture test migrates to an isolated registry.
Verified: full gates green (573/0), and the hostile-readdir-order
worktree that reproduced the CI failure 12/12 runs green with the fix
under the same order.
@nzneit
nzneit merged commit 7350d65 into main Aug 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant