Skip to content

fix(guardrails): stop yolo sessions waiting on the guardrail checker - #2519

Open
rdimitrov wants to merge 4 commits into
mainfrom
claude/mecatl-performance-investigation-7372a3
Open

rdimitrov wants to merge 4 commits into
mainfrom
claude/mecatl-performance-investigation-7372a3

Conversation

@rdimitrov

@rdimitrov rdimitrov commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Summary

Yolo-mode sessions no longer wait on the guardrails checker. By default they skip it entirely; operators who want the advisory signal can opt in with guardrails.yoloAdvisory: true, which runs reviews in the background.

Why

In yolo mode a configured checker is documented as advisory only: no veto, no ask, no fail-closed. The engine still ran every review inline, though. Each matching Read, Grep, Glob, ListDir, Shell, Edit, Write, web and MCP call waited for a full checker-model run (up to 90s on a stalled checker), and the verdict was then discarded. Shell and web calls paid two reviews each, serially. A checker turns on whenever the guardrail slot resolves, including through its fallback to the cheap slot, so many yolo users paid this without choosing it.

Two yolo paths could also still fail closed, contradicting the documented posture: terminal evidence failures, and principal-revision changes during an action review.

What changes

  • Default: a yolo-mode session calls no checker. Narrower delegated children of a yolo parent keep enforcing, as before.
  • Opt-in (guardrails.yoloAdvisory: true): evidence prep stays on the dispatch goroutine; the checker call runs in the background. The tool runs and its result reaches the model immediately. Verdicts are emitted as advisory EvHook guardrail events (pass_advisory / execute / release_result). Checker failures never fail closed in yolo.
  • Run end: drainChildren and the run goroutine's teardown cancel in-flight background reviews and wait at most a 1s grace for them to unwind; verdicts still in flight are dropped. Advisory events go out through the child registry's sealed, abortable emit (safeEmit), so none follows the terminal event and a stalled consumer cannot wedge run end.
  • Engine API: adds agent.YoloReviewPolicyProvider (Added). Recorded under Changed: yolo skips reviewers that do not opt in.
  • Reporting: the startup posture line, the guardrails_checker_reason, and the yolo note now say whether yolo skips the checker or runs it in the background.
  • Docs: permissions-and-posture page, generated configuration reference, and the governance architecture claim.

API compatibility

  • Assessment: no wire break. No proto or field changes, and disposition values are unchanged.
  • Observable changes: in yolo sessions, guardrail EvHook events are now absent by default. With the opt-in, they arrive after the reviewed tool's result rather than before it, still before the run's terminal event. These events are informational in yolo: they never gated a result or an ask, and the TUI already handles guardrail outcomes arriving after a result is released.
  • Major/feature: none.
  • Go source: additive interface (YoloReviewPolicyProvider). Embedders whose ToolReviewer relied on inline yolo review must implement it to keep receiving yolo reviews. Changelog entry classifies this as Changed, pre-v1 minor.
  • Stored data: unaffected.
  • Upgrade implications: operators who want yolo warnings set guardrails.yoloAdvisory: true.

Test plan

  • New engine tests (engine/agent/yolo_review_test.go): no opt-in means zero checker calls; opt-in releases the tool result before the parked checker answers, with both phases advisory and usage recorded; terminal evidence failure in yolo releases the result with no ask; run end cancels parked reviews at once. Mutation-checked against the previous inline behavior; stable over 20 race runs.
  • Composition tests through the production reviewer (internal/app/guardrails_yolo_test.go) and a settings-parse test for yoloAdvisory.
  • Updated TestGuardrailE2EExecutedYoloAutoDefaultTransitions to assert the checker is skipped in yolo while the same checker still blocks in auto. Moved TestBuildWiresFiniteScriptEvidenceToContextualReviewer from yolo to auto, where the review it checks still runs inline.
  • task lint: root and engine clean. The one remaining finding is pre-existing in environment/microvm/launch_ownership_darwin.go, which this change does not touch.
  • task test:race: engine/agent and internal/adapter/permconfig pass. The remaining failures are environmental on the macOS dev host. The internal/* failures were reproduced on unmodified HEAD (forker/worktree, temp-path symlink, microvm, TLS key size, BSD stat, missing timeout). CI is the authoritative run.
  • go run ./cmd/mecademo shows the tool call, permission ask/approval, and result.
  • task docs, task site:build.

🤖 Generated with Claude Code

Yolo-mode sessions demoted guardrail review to advisory, but the engine
still ran every review inline: each matching Read, Grep, Shell, Edit and
web call waited for a full checker-model run whose verdict was then
discarded. Two paths (terminal evidence failures and principal-revision
changes) could still fail closed in yolo, contradicting the documented
posture.

Yolo sessions now skip the checker by default. Operators can opt in with
guardrails.yoloAdvisory: true, which runs matching action and inbound
reviews in the background: the tool never waits, is never blocked or
held, checker failures never fail closed, and verdicts are emitted as
advisory guardrail events. Run end joins in-flight reviews for up to 10s
(plus 1s grace) so their usage is recorded, then cancels the rest.
Narrower delegated children of a yolo parent keep enforcing.

Adds agent.YoloReviewPolicyProvider (engine API, Added) and records the
yolo behavior change in the engine changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d edges

Publish advisory review events through the child registry's sealed,
abortable emit instead of holding a private lock across the send, so a
stalled consumer can no longer wedge run end. Drain background reviews
from the run goroutine's teardown as well as drainChildren, so no review
outlives the run on any exit path.

Simplify the review set: create its context once per run, drop the
sealed flag, cancel in-flight reviews at run end instead of waiting up
to 10s, and share one launcher between action and inbound reviews.
Revert the yolo guards on the inline terminal-failure path, which yolo
no longer reaches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ncel test

Create the background review context on first use again: the eager
per-run context and run-end drain added 13 allocations to every turn in
the allocs gate even when no review runs. Drain now returns without
allocating when nothing started.

The run-end cancel test could end the run before either review reached
the checker; it now lets the run finish only after both reviews park.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@ChrisJBurns ChrisJBurns left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • medium · correctness — Action review now runs at the same time as the tool it reviews (engine/agent/actionreview.go, engine/agent/backgroundreview.go)

    For Edit, Write and Shell, the checker can read evidence after the tool has already changed it. A version change marks the evidence incomplete, so the verdict becomes unresolved and is reported as pass_advisory. That makes the advisory signal weakest for exactly the riskiest tools.

  • low · operational — Usage from background reviews can go unrecorded at teardown (engine/agent/backgroundreview.go, engine/agent/loop.go)

    Defers run in reverse order, so closeAuxiliaryUsageOwnership runs before backgroundReviews.drain. Reviews that are abandoned or still unwinding enqueue their usage after ownership has closed, so checker tokens that were spent can be under-reported.

  • nit · maintainability — heldFinalTurn test helper is duplicated across packages (engine/agent/yolo_review_test.go, internal/app/guardrails_yolo_test.go)

    The same LLM wrapper is defined in both test files. If the two copies drift, the hold-the-final-turn timing in the engine and app tests could stop matching.

Comment thread engine/agent/backgroundreview.go
Comment thread engine/agent/backgroundreview.go
Comment thread engine/agent/backgroundreview.go
…iews

A background action review runs while the reviewed call executes, so a
lazily read file could already be changed by Edit, Write or Shell and the
verdict degrade to unresolved. Background reviews now set
ReviewEvidencePreparation.Snapshot and the preparer serves the bounded
first read it already performs from memory, reporting larger files as
incomplete. Preparation cost is unchanged.

Cap concurrent background reviews at four per run; calls over the cap
are not reviewed. Stop recording advisory verdicts in the shared review
trajectory, so enforcing reviews of delegated children see facts in
dispatch order rather than in checker completion order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rdimitrov

Copy link
Copy Markdown
Member Author

Thanks for the review. On the summary items, as of d1f3b21:

  • Action review racing the tool (fixed): background reviews now set ReviewEvidencePreparation.Snapshot. The preparer keeps the bounded first read it already does as in-memory evidence, so the checker sees each referenced file as it was before the call ran. Files larger than that first read are reported incomplete rather than read live. Preparation cost is unchanged. Covered by TestSnapshotWorkspaceEvidenceSurvivesLaterChange (mutation-checked) and TestYoloBackgroundReviewsSnapshotEvidence.
  • Usage at teardown (no change): on every normal terminate path, drainChildren cancels and joins background reviews first, then closeAuxiliaryUsageOwnershipAndDrain records their usage. The defer order only matters on the backstop path, where pending usage isn't drained into the session for any auxiliary work. Reordering the defer there wouldn't record anything more, and reviews that outlive the 1s grace drop usage under the existing "late auxiliary usage is dropped" contract.
  • Duplicated heldFinalTurn (no change): sharing it would mean exporting a test-only helper from the engine module (for example via mockllm). For a 15-line wrapper, I'd rather keep a copy per package, since each copy's timing matches the test it serves.

🤖 Addressed by Claude Code

This branch was successfully deployed

1 active deployment
Preview — d1f3b213 Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants