Skip to content

fix(runtime): settle a parked run on its leader's disconnect, never on silence alone - #159

Merged
nikolasd merged 5 commits into
mainfrom
leader-grace-window
Sep 10, 2026
Merged

nikolasd merged 5 commits into
mainfrom
leader-grace-window

Conversation

@nikolasd

Copy link
Copy Markdown
Owner

A parked run used to be settled to lost by the inactivity sweep once it went quiet, conflating a worker gone quiet while its leader was still connected with a leader that had itself vanished. Only the second is actually abandoned; silence alone is never evidence a leader gave up, only its connection actually being gone is.

Mechanism

ipc::leader_registry::LeaderRegistry: a refcounted registry keyed by leader instance id, in ipc::server::Shared beside active_connections, populated from inside ipc::connection::handle (the negotiated instance id doesn't exist until the initialize handshake completes). Refcounted rather than a set or a flag, because more than one physical connection can share one instance id (the crew-extension fallback), and plain addition resolves both connect/disconnect race orderings correctly.

Each entry is {count, zero_since}, not a bare count. zero_since records the instant of the entry's most recent 1→0 transition and is cleared on any 0→1, closing a bug in the simpler design: a leader that disconnects, reconnects, and disconnects again within one grace window could otherwise leave a stale timer reading count == 0 correctly, but for the wrong disconnect.

The registry's decrement is a drop guard, not a statement paired by hand at each exit from the dispatch loop: a future early return would otherwise leak the registration silently, failing the feature closed with nothing to notice. The guard's on_gone callback fires exactly on the drop that causes the true 1→0 transition, arming the grace-window timer from inside Drop::drop itself.

The grace window is 60 seconds. Teardown reuses run_cancel's own mechanical shape via a shared cancel_and_settle_run helper parameterized on the target state: run_cancel asks for the literal "cancelled", checked against the requesting principal; the new path asks for RunState::unrendered_verdict() and bypasses the ownership check the same way the removed backstop did. A WorkerTimeout { kind: LeaderGone } fact is journaled first, before the state changes because of it.

Daemon restart: at startup nothing has connected to the fresh registry yet, so every leader owning a non-terminal run is, from this process's own perspective, disconnected as of right now. Each such owner is seeded with zero_since set to the daemon's own startup instant and the same grace-window timer armed. The clock starts at the daemon's own startup instant, never at a run's last-seen activity: a leader cannot reconnect to a daemon that is not running, so downtime is the daemon's own absence, not the leader's, and must never be charged against the window. The disconnect grace window is a ServerConfig field now, defaulting to the production constant, so tests can shrink it instead of waiting out the real window.

Removes the inactivity sweep's settle_abandoned_turn path entirely, along with its three dedicated tests. A parked run whose leader is still connected is never settled by the runtime now, however long it stays silent — the inactivity fact is still journaled, but it no longer drives a state change. lost returns to its original, narrower meaning: the evidence itself is missing or ambiguous, never "the leader went quiet."

Review response

A leader that reconnects while settle_leader_gone is still working through its owned runs (a query plus two DB round trips per run — not instantaneous for a leader with several parked runs) previously had every one of those runs settled anyway, after it was demonstrably back. settle_leader_gone now re-checks gone_for_at_least immediately before each run's own settle, not only once via the caller's check before the loop starts. The caller's check stays as the cheap early-out; the new one is what actually protects a leader that comes back mid-loop. Proven with a driver whose first cancel_run call blocks until released, landing the reconnect deterministically between two runs' settles rather than racing a guessed sleep.

Two hardening changes from the same review: gone_for_at_least on an instance id with no registry entry at all now answers false, not true — both real callers always create the entry first, so the branch is a guard against future misuse, never a path either takes today, but the destructive answer must never be the default for an unestablished question. And the LeaderGone arm in the inactivity sweep and ActivityClock::mark_journaled, both invariants that should hold but run inside a long-lived background task, now log an error and continue instead of panicking.

Docs

Adds ADR-0036, amending ADR-0027 (the backstop-settles-on-silence half of its Decision point 3 is superseded; the "never succeeded" half is inherited unamended) and noting the crew-extension shared-instance-id hazard the refcounting has to be safe under without resolving the pre-existing ownership ambiguity that id already carries elsewhere. Updates docs/architecture.md and docs/future-features.md, which described the now-removed backstop as current behavior.

Tests

9 unit tests on LeaderRegistry covering the refcount/zero_since mechanics directly, including the reconnect-within-window stale-zero case. 4 unit tests on OrchestrationService's restart-seeding and settle paths against a real database — unit rather than integration because the mechanism under test is deliberately internal, never a surface a leader talks to. 6 integration tests against the real IPC server covering a live disconnect's teardown, the shared settle helper, an in-window reconnect, cross-leader isolation, and the inactivity fact's continued journaling without settlement.

bun run generate re-run for the new TimeoutKind::LeaderGone variant.

nikolasd and others added 5 commits September 10, 2026 00:13
…n silence alone

A parked run (waitingUser + turnSettled) used to be settled to `lost` by
the inactivity sweep once it went quiet past the inactivity deadline
(ADR-0027 wave 3). That conflated two different situations: a worker
gone quiet while its leader is still connected and free to steer or
finish it, and a leader that has itself vanished, leaving the run
unfinishable. Silence looks identical from the runtime's side in both
cases, but only the second is actually abandoned. Silence alone is
never evidence a leader gave up -- only the leader's connection
actually being gone is.

Adds ipc::leader_registry::LeaderRegistry: a refcounted registry keyed
by leader instance id, living in ipc::server::Shared beside the
existing active_connections counter but populated from inside
ipc::connection::handle (the negotiated instance id doesn't exist
until the initialize handshake completes, which admit's spawn wrapper
never sees). Refcounted rather than a set or a flag, because more than
one physical connection can share one instance id (the crew-extension
fallback used by any leader session with no real session id of its
own), and plain addition resolves both connect/disconnect race
orderings correctly without extra care.

Each entry is {count, zero_since}, not a bare count. zero_since records
the instant of the entry's most recent 1->0 transition and is cleared
on any 0->1 -- closing a real bug in the simpler design: a leader that
disconnects, reconnects, and disconnects again within one grace window
can otherwise leave a stale timer (armed for the first disconnect)
waking after the window and reading count == 0, correctly, but for the
second disconnect, not the one it was spawned to watch. Recording the
transition's instant lets a woken timer ask the actual question --
has THIS absence lasted the full window -- rather than merely whether
the count happens to read zero right now.

The registry's decrement is a drop guard (LeaderConnectionGuard), not a
statement paired by hand at each exit from the dispatch loop: that
loop's decrement needs the instance id, which only exists once
execution is already inside it, so unlike active_connections it cannot
be bookkept from the spawn wrapper, and a future early return would
otherwise leak the registration silently, failing this feature closed
with nothing to notice. The guard's on_gone callback fires exactly on
the drop that causes the true 1->0 transition, arming the grace-window
timer from inside Drop::drop itself, so arming is as structural as the
decrement it rides on.

The grace window is 60 seconds (LEADER_DISCONNECT_GRACE_WINDOW): short
enough that an abandoned run does not tie up a concurrency slot
indefinitely, long enough to absorb an ordinary restart or reconnect
blip. Teardown reuses run_cancel's own mechanical shape rather than a
second implementation of it: the guts of run_cancel are factored into
a shared cancel_and_settle_run helper parameterized on the target
RunState. run_cancel still asks for the literal "cancelled", checked
against the requesting principal; the new settle_leader_gone path asks
for RunState::unrendered_verdict() and passes no principal, bypassing
the ownership check the same way the removed backstop did. Before
settling, a WorkerTimeout { kind: LeaderGone } fact is journaled first
-- the same report-before-decide shape the inactivity sweep already
uses for Inactivity/Total.

Daemon restart: at startup nothing has connected to the fresh registry
yet, so every leader owning a non-terminal run is, from this process's
own perspective, disconnected as of right now.
OrchestrationService::seed_leader_registry_after_restart seeds each
such owner with zero_since set to the daemon's own startup instant and
arms the same grace-window timer a live disconnect would get. The
clock deliberately starts at the daemon's own startup instant, never
at a run's last-seen activity: a leader cannot reconnect to a daemon
that is not running, so any time the daemon spent stopped is the
daemon's own absence, not the leader's, and must never be charged
against the window -- doing otherwise would settle live, reconnecting
leaders' runs on every restart that followed a stop longer than the
window. The disconnect grace window is now a ServerConfig field
(leader_disconnect_grace) rather than a bare constant reference at
each call site, defaulting to the production constant, so tests can
shrink it instead of waiting out the real window.

Removes the inactivity sweep's settle_abandoned_turn path entirely,
along with its three dedicated tests. A parked run whose leader is
still connected is never settled by the runtime now, however long it
stays silent -- the inactivity fact is still journaled, for
run/timeoutAck to react to, but it no longer drives a state change.
lost returns to its original, narrower meaning: the evidence itself is
missing or ambiguous, never "the leader went quiet."

Adds ADR-0036 recording this decision, amending ADR-0027 (the
backstop-settles-on-silence half of its Decision point 3 is
superseded; the "never succeeded" half is inherited unamended) and
noting the crew-extension shared-instance-id hazard the refcounting
has to be safe under without resolving the pre-existing ownership
ambiguity that id already carries elsewhere. Updates
docs/architecture.md and docs/future-features.md, which described the
now-removed backstop as current behavior.

Tests: 9 unit tests on LeaderRegistry covering the refcount/zero_since
mechanics directly (including the reconnect-within-window stale-zero
case above); 2 unit tests on
OrchestrationService::seed_leader_registry_after_restart against a
real database (a parked run whose leader never reconnects is settled
after the window; one that reconnects within it is left untouched and
journals nothing) -- unit tests rather than integration tests because
the mechanism under test is deliberately internal, never a surface a
leader talks to; and 6 integration tests against the real IPC server
covering a live disconnect's teardown, the shared settle helper, an
in-window reconnect, cross-leader isolation, and the inactivity fact's
continued journaling without settlement.

bun run generate re-run for the new TimeoutKind::LeaderGone variant.
…settle, not only once

settle_leader_gone took its caller's `gone_for_at_least` check as final:
the actual decision happened before the DB query for the leader's owned
runs, and was never revisited across the loop that settles each one. For
a leader with several parked runs, that loop is a query plus two DB
round trips per run -- not instantaneous. A leader that reconnects
inside that window had every one of its runs settled anyway, after it
was demonstrably back.

"Abandoned" means the leader is gone now, having been gone the full
window -- not merely that it was gone for the window at some earlier
instant. A leader that has reconnected is not abandoned, and settling
its work serves no purpose this feature exists for.

settle_leader_gone now takes the LeaderRegistry and re-checks
gone_for_at_least immediately before each run's own settle, inside the
loop. The caller's own check stays as the cheap early-out that skips the
query entirely in the common case; the new check is what actually
protects a leader that comes back mid-loop, run by run, rather than only
at the loop's start. Both call sites (a live disconnect's own timer, the
daemon-restart seed) pass their registry through. Doc comment rewritten
to state the rule instead of describing the old, narrower behavior.

Proven with a test using a driver whose first cancel_run call blocks
until released, rather than a guessed sleep: the reconnect is registered
the instant the first run's settle is confirmed committed (its cancel_run
has started), landing deterministically between the two runs' settles.
The run reached first is still settled; the run reached after the
reconnect is left untouched.

Two more hardening changes from the same review:

gone_for_at_least on an instance id with no registry entry at all now
answers false, not true. Both real callers (a live connection's
register, the restart seed's seed_disconnected_since) always create the
entry before ever asking this question, so the branch is unreachable
from either path today -- but the destructive answer must never be the
default for a question nobody has established one for. Logs a warning
when it happens; the corresponding test now asserts false and states
why the branch is a guard, not a path.

The LeaderGone arm in the inactivity sweep's since_ms computation, and
in ActivityClock::mark_journaled, no longer panics on an invariant that
should hold but runs inside a long-lived background task rather than a
test. Both log an error and continue (the sweep skips journaling that
one kind for that tick; the clock skips setting a flag), so a future
change that violates the invariant in error degrades to a missed fact
rather than taking the sweep down.
@nikolasd
nikolasd merged commit d9a280b into main Sep 10, 2026
19 checks passed
@nikolasd
nikolasd deleted the leader-grace-window branch September 10, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant