Repository navigation
feat(effect-daemon-cluster): standby singletons report ready only after shard acquisition - #38
Conversation
…er shard acquisition A cluster singleton whose shard another runner owns used to be announced ready at registration, so rest-for-one dependents started against an uninitialized singleton and startTimeoutMillis could not bound the real post-acquisition initialization. The medium now reports standby truthfully: a standby singleton is not ready, is not bound by the start deadline, and children declared after it do not start until it owns its shard and its run signals ready. The start deadline is armed when the shard arrives, and losing the shard is a handoff back to standby rather than a termination: nothing is reported, restarted, or charged against the restart intensity. The supervisor starts children in declared order, each gated on the readiness of the ones before it, matching the documented OTP behaviour. Status and the trace expose standby, acquired, ready and released transitions; media that cannot standby are unchanged.
Review fixes on top of the standby-readiness change: - P0-1: ChildStandby/ChildAcquired/ChildReleased now apply to the core in every live supervisor phase (Restarting keeps its pending backoff plans, CoolingDown its window), so an acquisition during a group backoff is never dropped. - P0-2: child timer targets carry an acquisition epoch; a start deadline or liveness tick armed before a release cannot fire against the re-acquisition, and duplicate ticks after re-promotion are stale. - P0-3: children with an activity stream report readiness only through that ordered stream (the readiness watcher is not forked for them), and ChildReady no longer promotes a standby child, so a release processed ahead of an in-flight ready cannot mark a non-owner ready. - P1-1: one_for_one restarts issue directly instead of gating on earlier children, so an independent sibling restarts immediately while a standby singleton waits. - P1-2: the medium no longer pre-checks shard ownership at registration (a TOCTOU race); every singleton declares standby and emits Acquired at the start of every run. - P1-3: awaiting drops entries whose incarnation is gone, and dynamic starts route through the same enqueue/drain gating. - P1-4: stopping a singleton shuts its activity queue down so the watcher fiber ends with the incarnation. Counterexample tests cover each case: acquisition during a group backoff, a re-acquisition crossing the pre-release deadline, interruption before readiness, an ungated one_for_one restart, owned-at-registration standby declaration, dynamic-start gating, plus kernel property laws for phase-preserving transitions, epoch-matched timers, ready-while-standby rejection, ungated restarts, and dead-entry pruning. The conformance kit projects standby/acquired/released pacing and stale-timer no-ops out of the compared trace.
… restarts children cooledRestartOf spread the old child record, keeping a non-zero epoch from before the cool-down while drainAwaitingOf arms the fresh start_deadline at epoch 0, so the elapse would read as stale and a hung incarnation would never miss its deadline. A restart resets the epoch like restartingOf already does. New property law ∀c_CoolDown_=ResetsEpoch: a cooling-down child with a non-zero epoch comes back at epoch 0 with an epoch-0 start_deadline armed; it fails against the unfixed fold.
|
Round 2 finding addressed in Fix: Law: Checks on the final tree: spec suite 271/271 (incl. the new law), lint/dprint/typecheck/build/api:check clean, pre-push hook green. Cluster/conformance/process/socket suites unchanged by this commit (kernel-internal, non-standby paths untouched). |
Problem
A cluster singleton whose shard another runner owns was announced ready at registration:
readyWhen(ownsSingleton(...), signalled)resolved readiness immediately for a standby runner. Consequences:rest_for_onedependent started (and ran) against an uninitialized singleton;startTimeoutMillisbounded registration-to-ready, which for a standby is a lie — it could not bound the real post-acquisition initialization;Change (head
c679145, review round 2 applied)Spec (kernel + medium contract, additive):
Startedevidence gains an optionalactivity(initiallyStandby+ aStreamofAcquired | Ready | Released). Media without it (fiber, process, socket) behave exactly as before.ChildStandby/ChildAcquired/ChildReleasedand a newstandbychild status. Standby children are not bound by the start deadline;ChildAcquiredarms a freshstart_deadline, so the timeout measures post-acquisition initialization.ChildReleasedis a handoff back to standby: no termination, no restart, no intensity charge.ChildReadycan no longer promote a standby child — a release processed ahead of an in-flight ready cannot mark a non-owner ready. Re-promotion readiness is re-reported through the stream (readiness is not one-shot per incarnation).one_for_all/rest_for_onegroup restarts, and dynamic starts.one_for_onerestarts of independent children issue directly (P1-1): an ungated sibling restarts immediately while a standby singleton waits. The gate drains in the same decision when predecessors are ready, so existing trees observe no timing change; the catalogue still conforms.Acquiredat the start of every run.awaitingdrops entries whose incarnation is gone (P1-3); stopping a singleton shuts its activity queue so the watcher ends with the incarnation (P1-4).statusOfshowsstandby;traceOfshows the full transition history. The conformance kit projects standby/acquired/released pacing and stale-timer no-ops out of the compared trace (the two-runner lifecycle test reads the raw trace).Cluster medium: a singleton's
readyresolves only after acquisition + run readiness; acquisition, re-promotion readiness, and handoff are emitted on the activity stream. Handoff still reports nothing and holds the registration.Counterexample tests (one per review finding)
readyEvents: 2, zero terminations).one_for_onesibling restarts at once while the standby singleton waits (workergen 1ready,gatestandby).initiallyStandby === true) and becomes ready.acquired:leader × 2).Restartingtransitions, epoch-matched deadlines (stale epoch →Stale, current epoch → missed), ready-while-standby rejection, ungatedone_for_onerestarts, acquisition blocking vs readiness draining the queue, dead-entry pruning.Verification (final tree, head
c679145)vitest: spec 270/270, conformance 86/86, cluster 19/19, process 33/33, socket 29/29 — exit 0.test:types41/41; lint,dprint check,tsc -b,build+api:check,attw— clean on all touched packages (pre-push hook green).Out of scope (stacked follow-ups, per the dogfooding list)
Unified readiness/timeouts (readiness folded fully onto the activity stream), typed retry-aware termination, post-startup liveness, keyed dynamic children.