Skip to content

feat(effect-daemon-cluster): standby singletons report ready only after shard acquisition - #38

Merged
systemfsoftware-bot merged 3 commits into
mainfrom
feat/daemon-lifecycle-dogfood
Nov 19, 2026
Merged

systemfsoftware-bot merged 3 commits into
mainfrom
feat/daemon-lifecycle-dogfood

Conversation

@systemfsoftware-bot

@systemfsoftware-bot systemfsoftware-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A cluster singleton whose shard another runner owns was announced ready at registration: readyWhen(ownsSingleton(...), signalled) resolved readiness immediately for a standby runner. Consequences:

  • a rest_for_one dependent started (and ran) against an uninitialized singleton;
  • startTimeoutMillis bounded registration-to-ready, which for a standby is a lie — it could not bound the real post-acquisition initialization;
  • status and the trace could not distinguish standby from active;
  • the conformance suite had no two-runner standby → acquisition → active → handoff → re-promotion coverage.

Change (head c679145, review round 2 applied)

Spec (kernel + medium contract, additive):

  • Started evidence gains an optional activity (initiallyStandby + a Stream of Acquired | Ready | Released). Media without it (fiber, process, socket) behave exactly as before.
  • New kernel events ChildStandby / ChildAcquired / ChildReleased and a new standby child status. Standby children are not bound by the start deadline; ChildAcquired arms a fresh start_deadline, so the timeout measures post-acquisition initialization. ChildReleased is a handoff back to standby: no termination, no restart, no intensity charge.
  • Acquisition epoch (P0-2): child timer targets carry an epoch bumped on every acquisition, so a deadline or liveness tick armed before a release cannot fire against the re-acquisition, and duplicate ticks after re-promotion are stale.
  • Single ordered readiness stream (P0-3): children with an activity stream report readiness only through it (the readiness watcher is not forked for them), and ChildReady can no longer promote a standby child — a release processed ahead of an in-flight ready cannot mark a non-owner ready. Re-promotion readiness is re-reported through the stream (readiness is not one-shot per incarnation).
  • Phase-preserving activity transitions (P0-1): standby/acquired/released apply to the core in every live phase; a group backoff preserves its pending plans while the singleton's acquisition and timer land.
  • Gated starts: children start in declared order, each gated on the readiness (or removal) of the ones before it — boot, one_for_all/rest_for_one group restarts, and dynamic starts. one_for_one restarts of independent children issue directly (P1-1): an ungated sibling restarts immediately while a standby singleton waits. The gate drains in the same decision when predecessors are ready, so existing trees observe no timing change; the catalogue still conforms.
  • TOCTOU-free registration (P1-2): the medium no longer pre-checks ownership; every singleton declares standby and emits Acquired at the start of every run.
  • awaiting drops entries whose incarnation is gone (P1-3); stopping a singleton shuts its activity queue so the watcher ends with the incarnation (P1-4).
  • statusOf shows standby; traceOf shows the full transition history. The conformance kit projects standby/acquired/released pacing and stale-timer no-ops out of the compared trace (the two-runner lifecycle test reads the raw trace).

Cluster medium: a singleton's ready resolves only after acquisition + run readiness; acquisition, re-promotion readiness, and handoff are emitted on the activity stream. Handoff still reports nothing and holds the registration.

Counterexample tests (one per review finding)

  • Re-acquired singleton crosses the deadline armed before its shard moved away — the stale epoch misses nothing (readyEvents: 2, zero terminations).
  • Singleton interrupted before signalling readiness stays standby; its dependent never starts (zero ready events, zero terminations).
  • one_for_one sibling restarts at once while the standby singleton waits (worker gen 1 ready, gate standby).
  • Owned-at-registration singleton still declares standby (initiallyStandby === true) and becomes ready.
  • Singleton acquiring during a group backoff keeps the acquisition and timer (both children back at gen 1, acquired:leader × 2).
  • Dynamic child accepted but withheld while the declared singleton is standby; runs once it is ready.
  • Kernel property laws: phase-preserving Restarting transitions, epoch-matched deadlines (stale epoch → Stale, current epoch → missed), ready-while-standby rejection, ungated one_for_one restarts, acquisition blocking vs readiness draining the queue, dead-entry pruning.

Verification (final tree, head c679145)

  • vitest: spec 270/270, conformance 86/86, cluster 19/19, process 33/33, socket 29/29 — exit 0.
  • test:types 41/41; lint, dprint check, tsc -b, build + api:check, attw — clean on all touched packages (pre-push hook green).
  • Pre-fix red (first round, unchanged): the standby/dependent/timeout scenarios fail against the unfixed code.

Out of scope (stacked follow-ups, per the dogfooding list)

Unified readiness/timeouts (readiness folded fully onto the activity stream), typed retry-aware termination, post-startup liveness, keyed dynamic children.

…er shard acquisition

A cluster singleton whose shard another runner owns used to be announced
ready at registration, so rest-for-one dependents started against an
uninitialized singleton and startTimeoutMillis could not bound the real
post-acquisition initialization.

The medium now reports standby truthfully: a standby singleton is not
ready, is not bound by the start deadline, and children declared after it
do not start until it owns its shard and its run signals ready. The start
deadline is armed when the shard arrives, and losing the shard is a
handoff back to standby rather than a termination: nothing is reported,
restarted, or charged against the restart intensity.

The supervisor starts children in declared order, each gated on the
readiness of the ones before it, matching the documented OTP behaviour.
Status and the trace expose standby, acquired, ready and released
transitions; media that cannot standby are unchanged.
Review fixes on top of the standby-readiness change:

- P0-1: ChildStandby/ChildAcquired/ChildReleased now apply to the core
  in every live supervisor phase (Restarting keeps its pending backoff
  plans, CoolingDown its window), so an acquisition during a group
  backoff is never dropped.
- P0-2: child timer targets carry an acquisition epoch; a start
  deadline or liveness tick armed before a release cannot fire against
  the re-acquisition, and duplicate ticks after re-promotion are stale.
- P0-3: children with an activity stream report readiness only through
  that ordered stream (the readiness watcher is not forked for them),
  and ChildReady no longer promotes a standby child, so a release
  processed ahead of an in-flight ready cannot mark a non-owner ready.
- P1-1: one_for_one restarts issue directly instead of gating on
  earlier children, so an independent sibling restarts immediately
  while a standby singleton waits.
- P1-2: the medium no longer pre-checks shard ownership at
  registration (a TOCTOU race); every singleton declares standby and
  emits Acquired at the start of every run.
- P1-3: awaiting drops entries whose incarnation is gone, and dynamic
  starts route through the same enqueue/drain gating.
- P1-4: stopping a singleton shuts its activity queue down so the
  watcher fiber ends with the incarnation.

Counterexample tests cover each case: acquisition during a group
backoff, a re-acquisition crossing the pre-release deadline,
interruption before readiness, an ungated one_for_one restart,
owned-at-registration standby declaration, dynamic-start gating, plus
kernel property laws for phase-preserving transitions, epoch-matched
timers, ready-while-standby rejection, ungated restarts, and dead-entry
pruning. The conformance kit projects standby/acquired/released pacing
and stale-timer no-ops out of the compared trace.
… restarts children

cooledRestartOf spread the old child record, keeping a non-zero epoch
from before the cool-down while drainAwaitingOf arms the fresh
start_deadline at epoch 0, so the elapse would read as stale and a
hung incarnation would never miss its deadline. A restart resets the
epoch like restartingOf already does.

New property law ∀c_CoolDown_=ResetsEpoch: a cooling-down child with a
non-zero epoch comes back at epoch 0 with an epoch-0 start_deadline
armed; it fails against the unfixed fold.
@systemfsoftware-bot

Copy link
Copy Markdown
Contributor Author

Round 2 finding addressed in 194b859.

Fix: cooledRestartOf now resets epoch: 0 on the children it restarts (same invariant as restartingOf), so the epoch-0 start_deadline that drainAwaitingOf arms after a cool-down is never stale and a hung incarnation still misses its deadline.

Law: ∀c_CoolDown_=ResetsEpoch — a cooling-down child carrying a non-zero epoch comes back at epoch 0 with an epoch-0 start_deadline armed at the resumed instant. Verified red against the unfixed fold (the law constructs the CoolDown state directly; routing through the exhaustion path masks the hole because restartedChildrenOf already resets), green with the fix.

Checks on the final tree: spec suite 271/271 (incl. the new law), lint/dprint/typecheck/build/api:check clean, pre-push hook green. Cluster/conformance/process/socket suites unchanged by this commit (kernel-internal, non-standby paths untouched).

@systemfsoftware-bot
systemfsoftware-bot merged commit 81509a2 into main Nov 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant