Repository navigation
gate repeated_failure on an applied transition, not a guess - #167
Merged
Merged
Conversation
…uess
`observe_process_exited` computed what a vendor process's exit would mean
in isolation (`terminal_state_for`), walked the run toward that value, and
then raised the repeated-failure escalation off the computed value
regardless of what the walk actually did. That is wrong whenever the walk
is a no-op: `walk_to` already refuses a second edge on a run that is
already terminal, and a run reaches that state independently of process
exit whenever the leader's own `run/finish` settles it first -- which also
clears `turnSettled`, the exact fact `terminal_state_for` branches on. So
the ordinary shape "leader finishes the run, the vendor process happens to
exit a moment later" makes the exit handler compute `failed` for a run
that is actually `succeeded`, and the escalation fired on that guess: a
run that never failed could raise `EscalationRaised { reason:
"repeated_failure" }` as if it had, so long as its task's prior run
genuinely had.
`walk_to` now returns whether *this call* is the one that committed the
run into its target -- false whenever nothing was actually applied
(already at target, already some other terminal state, no legal hop, or
the commit itself failed), true only when a fresh commit lands exactly on
target. `observe_process_exited` gates the escalation on that return value
together with the computed terminal being `failed`, instead of the
computed value alone. A prior version of the fix reached for a second,
independent `current()` read before the walk to decide "already
terminal"; using the walk's own return instead needed no second read and
no separate race window to reason about.
Also populates the escalation's `question` field -- the sink previously
carried `None` here, which is why the field being always-empty was
scoped as its own follow-up. The runtime-authored text mirrors the
first-run-gate question's own pattern: names what happened (this task's
previous run also failed) and what to do (read both runs' output, then
retry or escalate; crew will not retry it for you), and is only ever
reached from behind `previous_run_for_task_also_failed`, so it is never
sent on a task's actual first failure.
On the extension side, the "two consecutive failures" sentence that used
to be appended to every plain `failed` digest -- unconditionally, true on
a task's first failure as much as its second -- is gone from that digest
entirely. The guidance now lives only in the `escalationRaised` digest,
which already existed and already only fires on a genuine repeat; with
the runtime now populating its `question`, that digest carries the same
information the removed sentence used to assert regardless of whether it
was true.
This gate is narrower than what it replaced, not just more accurate --
worth stating plainly rather than leaving it as a side effect nobody
wrote down. `run/finish { outcome: "failed" }` commits `failed` through
its own path (`OrchestrationService::run_finish`), never through this
walk, and never raises this escalation itself; it never has. Before this
fix, a `ProcessExited` racing in afterward could still trigger the
escalation by accident, riding the very bug this change closes -- so a
leader-adjudicated failure sometimes got the notice and sometimes did
not, depending on exit timing. Now it never does, on any timing, even
when the task's predecessor genuinely also failed. Decided rather than
merely inherited: a run the leader explicitly failed already carries the
leader's own verdict, a materially different state of knowledge than the
silent, no-verdict-rendered death this escalation exists to catch.
Raising it there too would mean threading the same task-history check
into `run_finish`, the run/submit start-error backstop, and the boot
recovery sweep -- every other place a run can reach `failed` -- which is
a real expansion of this escalation's scope, not a bug fix, and out of
scope here. Documented at the gate itself, not just in this message.
Tests: a fixture-driven regression proving the exact race -- a run settled
to `succeeded` ahead of a racing exit that computes `failed`, with a
genuinely failed predecessor on the same task (the one condition that
would have tripped the old bug) -- asserts the run stays `succeeded` and
no escalation is journaled. A second, paired regression pins the
narrowing above: a run already settled to `failed` (standing in for
`run/finish`) gets no escalation off a racing exit either, even with a
genuinely failed predecessor -- the old code, riding the same bug, would
have raised one here. A third, positive case proves the gate does not
suppress a genuine second failure reached through this sink's own walk,
and that its escalation carries the populated question. Verified each
regression actually discriminates by reverting the gate and watching it
fail (`left: ["repeated_failure"], right: []` both times), and did the
same for the digest change by reintroducing the removed sentence and
watching the new TS test fail, before restoring every fix and re-running
everything green.
cargo fmt --all, cargo clippy --all-targets --all-features -D warnings,
bun run check (markers, generate --check, format:check, typecheck,
extension build, bun test 434/434, cargo test --workspace all green)
all verified clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gates the
repeated_failureescalation on an applied transition tofailed, populates its question field, and drops the extension's own unconditional "two consecutive failures" sentence in favor of it.The bug
observe_process_exitedcomputed what a vendor process's exit would mean in isolation (terminal_state_for), walked the run toward that value, and then raisedEscalationRaised { reason: "repeated_failure" }off the computed value regardless of what the walk actually did.That's wrong whenever the walk is a no-op. A run reaches a terminal state independently of process exit whenever the leader's own
run/finishsettles it first — which also clearsturnSettled, the exact factterminal_state_forbranches on. So the ordinary shape "leader finishes the run, the vendor process happens to exit a moment later" made the exit handler computefailedfor a run that was actuallysucceeded, and the escalation fired on that guess: a run that never failed could raiserepeated_failureas if it had, so long as its task's prior run genuinely had.The fix
walk_tonow returns whether this call is the one that committed the run into its target —falsewhenever nothing was actually applied (already at target, already some other terminal state, no legal hop, or the commit itself failed),trueonly when a fresh commit lands exactly on target.observe_process_exitedgates the escalation on that return value together with the computed terminal beingfailed, instead of the computed value alone.This gate is narrower than what it replaced, not just more accurate
run/finish { outcome: "failed" }commitsfailedthrough its own path and never raises this escalation itself — it never has. Before this fix, a racingProcessExitedcould still trigger the escalation by accident, riding the very bug this closes, so a leader-adjudicated failure sometimes got the notice and sometimes didn't, depending on exit timing. Now it never does, on any timing, even when the task's predecessor genuinely also failed.Decided rather than merely inherited: a run the leader explicitly failed already carries the leader's own verdict — a materially different state of knowledge than the silent, no-verdict-rendered death this escalation exists to catch. Raising it there too would mean threading the same task-history check into
run_finish, the run/submit start-error backstop, and the boot recovery sweep — every other place a run can reachfailed— which is a real expansion of this escalation's scope, not a bug fix, and out of scope here. Documented at the gate itself.Also
Populates the escalation's
questionfield, previously alwaysNone. The runtime-authored text mirrorsfirst_run_gate_question's own pattern and is only ever reached from behindprevious_run_for_task_also_failed, so it's never sent on a task's actual first failure.On the extension side, the "two consecutive failures" sentence that used to be appended to every plain
faileddigest — unconditionally, true on a task's first failure as much as its second — is gone from that digest entirely. The guidance now lives only in theescalationRaiseddigest, which already existed and already only fires on a genuine repeat; with the runtime now populating itsquestion, that digest carries the information the removed sentence used to assert regardless of whether it was true.Tests
succeededahead of a racing exit that computesfailed, with a genuinely failed predecessor on the same task — asserts the run stayssucceededand no escalation is journaled.failed(standing in forrun/finish) gets no escalation off a racing exit either, even with a genuinely failed predecessor.left: ["repeated_failure"], right: []), and did the same for the digest change by reintroducing the removed sentence and watching the new TS test fail, before restoring every fix.Gates
cargo fmt --all,cargo clippy --all-targets --all-features -D warnings,bun run check(markers,generate --check,format:check,typecheck, extension build,bun test434/434,cargo test --workspaceall green) — all clean on the committed tip.