Skip to content

gate repeated_failure on an applied transition, not a guess - #167

Merged
nikolasd merged 2 commits into
mainfrom
crew-125-126-repeated-failure
Sep 10, 2026
Merged

nikolasd merged 2 commits into
mainfrom
crew-125-126-repeated-failure

Conversation

@nikolasd

Copy link
Copy Markdown
Owner

Gates the repeated_failure escalation on an applied transition to failed, populates its question field, and drops the extension's own unconditional "two consecutive failures" sentence in favor of it.

The bug

observe_process_exited computed what a vendor process's exit would mean in isolation (terminal_state_for), walked the run toward that value, and then raised EscalationRaised { reason: "repeated_failure" } off the computed value regardless of what the walk actually did.

That's wrong whenever the walk is a no-op. A run reaches a terminal state independently of process exit whenever the leader's own run/finish settles it first — which also clears turnSettled, the exact fact terminal_state_for branches on. So the ordinary shape "leader finishes the run, the vendor process happens to exit a moment later" made the exit handler compute failed for a run that was actually succeeded, and the escalation fired on that guess: a run that never failed could raise repeated_failure as if it had, so long as its task's prior run genuinely had.

The fix

walk_to now returns whether this call is the one that committed the run into its target — false whenever nothing was actually applied (already at target, already some other terminal state, no legal hop, or the commit itself failed), true only when a fresh commit lands exactly on target. observe_process_exited gates the escalation on that return value together with the computed terminal being failed, instead of the computed value alone.

This gate is narrower than what it replaced, not just more accurate

run/finish { outcome: "failed" } commits failed through its own path and never raises this escalation itself — it never has. Before this fix, a racing ProcessExited could still trigger the escalation by accident, riding the very bug this closes, so a leader-adjudicated failure sometimes got the notice and sometimes didn't, depending on exit timing. Now it never does, on any timing, even when the task's predecessor genuinely also failed.

Decided rather than merely inherited: a run the leader explicitly failed already carries the leader's own verdict — a materially different state of knowledge than the silent, no-verdict-rendered death this escalation exists to catch. Raising it there too would mean threading the same task-history check into run_finish, the run/submit start-error backstop, and the boot recovery sweep — every other place a run can reach failed — which is a real expansion of this escalation's scope, not a bug fix, and out of scope here. Documented at the gate itself.

Also

Populates the escalation's question field, previously always None. The runtime-authored text mirrors first_run_gate_question's own pattern and is only ever reached from behind previous_run_for_task_also_failed, so it's never sent on a task's actual first failure.

On the extension side, the "two consecutive failures" sentence that used to be appended to every plain failed digest — unconditionally, true on a task's first failure as much as its second — is gone from that digest entirely. The guidance now lives only in the escalationRaised digest, which already existed and already only fires on a genuine repeat; with the runtime now populating its question, that digest carries the information the removed sentence used to assert regardless of whether it was true.

Tests

  • A fixture-driven regression for the exact race: a run settled to succeeded ahead of a racing exit that computes failed, with a genuinely failed predecessor on the same task — asserts the run stays succeeded and no escalation is journaled.
  • A second, paired regression pinning the narrowing above: a run already settled to failed (standing in for run/finish) gets no escalation off a racing exit either, even with a genuinely failed predecessor.
  • A third, positive case proving the gate doesn't suppress a genuine second failure reached through this sink's own walk, and that its escalation carries the populated question.
  • Verified each regression actually discriminates by reverting the gate and watching it fail (left: ["repeated_failure"], right: []), and did the same for the digest change by reintroducing the removed sentence and watching the new TS test fail, before restoring every fix.

Gates

cargo fmt --all, cargo clippy --all-targets --all-features -D warnings, bun run check (markers, generate --check, format:check, typecheck, extension build, bun test 434/434, cargo test --workspace all green) — all clean on the committed tip.

nikolasd and others added 2 commits September 10, 2026 23:56
…uess

`observe_process_exited` computed what a vendor process's exit would mean
in isolation (`terminal_state_for`), walked the run toward that value, and
then raised the repeated-failure escalation off the computed value
regardless of what the walk actually did. That is wrong whenever the walk
is a no-op: `walk_to` already refuses a second edge on a run that is
already terminal, and a run reaches that state independently of process
exit whenever the leader's own `run/finish` settles it first -- which also
clears `turnSettled`, the exact fact `terminal_state_for` branches on. So
the ordinary shape "leader finishes the run, the vendor process happens to
exit a moment later" makes the exit handler compute `failed` for a run
that is actually `succeeded`, and the escalation fired on that guess: a
run that never failed could raise `EscalationRaised { reason:
"repeated_failure" }` as if it had, so long as its task's prior run
genuinely had.

`walk_to` now returns whether *this call* is the one that committed the
run into its target -- false whenever nothing was actually applied
(already at target, already some other terminal state, no legal hop, or
the commit itself failed), true only when a fresh commit lands exactly on
target. `observe_process_exited` gates the escalation on that return value
together with the computed terminal being `failed`, instead of the
computed value alone. A prior version of the fix reached for a second,
independent `current()` read before the walk to decide "already
terminal"; using the walk's own return instead needed no second read and
no separate race window to reason about.

Also populates the escalation's `question` field -- the sink previously
carried `None` here, which is why the field being always-empty was
scoped as its own follow-up. The runtime-authored text mirrors the
first-run-gate question's own pattern: names what happened (this task's
previous run also failed) and what to do (read both runs' output, then
retry or escalate; crew will not retry it for you), and is only ever
reached from behind `previous_run_for_task_also_failed`, so it is never
sent on a task's actual first failure.

On the extension side, the "two consecutive failures" sentence that used
to be appended to every plain `failed` digest -- unconditionally, true on
a task's first failure as much as its second -- is gone from that digest
entirely. The guidance now lives only in the `escalationRaised` digest,
which already existed and already only fires on a genuine repeat; with
the runtime now populating its `question`, that digest carries the same
information the removed sentence used to assert regardless of whether it
was true.

This gate is narrower than what it replaced, not just more accurate --
worth stating plainly rather than leaving it as a side effect nobody
wrote down. `run/finish { outcome: "failed" }` commits `failed` through
its own path (`OrchestrationService::run_finish`), never through this
walk, and never raises this escalation itself; it never has. Before this
fix, a `ProcessExited` racing in afterward could still trigger the
escalation by accident, riding the very bug this change closes -- so a
leader-adjudicated failure sometimes got the notice and sometimes did
not, depending on exit timing. Now it never does, on any timing, even
when the task's predecessor genuinely also failed. Decided rather than
merely inherited: a run the leader explicitly failed already carries the
leader's own verdict, a materially different state of knowledge than the
silent, no-verdict-rendered death this escalation exists to catch.
Raising it there too would mean threading the same task-history check
into `run_finish`, the run/submit start-error backstop, and the boot
recovery sweep -- every other place a run can reach `failed` -- which is
a real expansion of this escalation's scope, not a bug fix, and out of
scope here. Documented at the gate itself, not just in this message.

Tests: a fixture-driven regression proving the exact race -- a run settled
to `succeeded` ahead of a racing exit that computes `failed`, with a
genuinely failed predecessor on the same task (the one condition that
would have tripped the old bug) -- asserts the run stays `succeeded` and
no escalation is journaled. A second, paired regression pins the
narrowing above: a run already settled to `failed` (standing in for
`run/finish`) gets no escalation off a racing exit either, even with a
genuinely failed predecessor -- the old code, riding the same bug, would
have raised one here. A third, positive case proves the gate does not
suppress a genuine second failure reached through this sink's own walk,
and that its escalation carries the populated question. Verified each
regression actually discriminates by reverting the gate and watching it
fail (`left: ["repeated_failure"], right: []` both times), and did the
same for the digest change by reintroducing the removed sentence and
watching the new TS test fail, before restoring every fix and re-running
everything green.

cargo fmt --all, cargo clippy --all-targets --all-features -D warnings,
bun run check (markers, generate --check, format:check, typecheck,
extension build, bun test 434/434, cargo test --workspace all green)
all verified clean.
@nikolasd
nikolasd merged commit 8c4ebfd into main Sep 10, 2026
19 checks passed
@nikolasd
nikolasd deleted the crew-125-126-repeated-failure branch September 10, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant