feat(libsy): advisor review gate as a native algorithm and advisor route type by eric-liu-nvidia · Pull Request #371 · NVIDIA-NeMo/Switchyard

eric-liu-nvidia · 2026-08-11T20:37:47Z

Adds the advisor review gate to the Rust server as a new route type: type = "advisor".

Benchmark results (Terminal-Bench 2.1)

Configuration	Accuracy
Nemotron 3 Ultra (baseline)	43.8%
Nemotron 3 Ultra + Opus 4.8 as advisor review gate	54.7% ± 0.7 pp

With $64.76 ± $4.95 additional Opus 4.8 cost, the advisor review gate lifts Nemotron 3 Ultra accuracy by almost 25%.

How it works

There are two models: an executor (the model doing the work) and an advisor (a stronger model that checks the work).

The client talks to the executor as usual. While the executor is still working — its replies contain tool calls — everything passes straight through. The advisor is not involved.
When the executor produces a reply with no tool calls, that means it thinks it is done (or it is proposing a plan). The proxy holds that reply back instead of returning it.
The proxy sends the conversation and the held-back reply to the advisor and asks one question: is this work complete and correct? The advisor answers APPROVE or REDO.
APPROVE → the held-back reply is released to the client unchanged.
REDO → the client never sees that reply. The proxy puts the advisor's feedback into the conversation ("not done yet — here is what is missing") and calls the executor again, so it keeps working.
The review happens once per session (max_reviews, default 1). After that, every call is a plain passthrough with zero overhead.

If the advisor is down or replies with something unparseable, the gate fails open: the executor's reply goes to the client as if approved. Advisor problems never block the executor.

Why this shape: giving advice up front made the executor trust the plan and skip its own test-and-iterate loop. Checking only the final "I'm done" claim leaves the executor's normal behavior untouched and catches exactly one failure mode — stopping too early.

Example

[routes.gated]
id = "switchyard-advisor"
type = "advisor"
executor_target = "executor"
advisor_target = "advisor"   # judge-only; set max_retries = 0 on its llm_client

Note for reviewers: this adds regex as a production dependency of libsy (for the configurable review trigger pattern).

🤖 Generated with Claude Code

github-actions · 2026-08-11T20:39:03Z

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-371/
Built to branch `gh-pages` at 2026-08-11 21:14 UTC. Preview will be ready when the GitHub Pages deployment is complete.

coderabbitai · 2026-08-11T20:44:48Z

Walkthrough

Changes

Advisor gate

Layer / File(s)	Summary
Public API and route contract `Cargo.toml`, `crates/libsy/Cargo.toml`, `crates/libsy/src/algorithms.rs`, `crates/libsy/src/algorithms/advisor_gate.rs`, `crates/libsy/src/lib.rs`	Adds the `AdvisorGate` configuration types, public exports, module wiring, and workspace dependencies.
AdvisorGate execution flow `crates/libsy/src/algorithms/advisor_gate.rs`, `crates/libsy/src/algorithms/util/*`	Buffers executor turns, invokes the advisor on configured triggers, handles `APPROVE` and `REDO`, supports streaming replay, enforces scoped budgets, and records telemetry.
Server route construction `crates/switchyard-server/src/config.rs`	Adds `Advisor` routes, target resolution, trigger validation, capability handling, defaults, and configuration tests.
Advisor gate statistics `crates/switchyard-server/src/stats/algorithms.rs`, `crates/switchyard-server/src/stats/algorithms/advisor_gate.rs`	Projects advisor-gate Prometheus counters into reset-relative serialized snapshots.
End-to-end advisor coverage `crates/switchyard-server/tests/server.rs`	Tests approval, redo, budgets, streaming, routing logs, token counting, failures, and statistics resets.

Estimated code review effort: 4 (Complex) | ~60 minutes

Poem

A rabbit reviews each turn,
“APPROVE” makes bright replies return.
“REDO” sends feedback down the track,
Streams replay, then hop right back.
Budgets, stats, and tests all bloom.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 86.96% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: adding the AdvisorGate algorithm and introducing the advisor route type.

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/switchyard-server/src/stats/algorithms.rs`:
- Line 31: Add concise Rust doc comments for the public advisor_gate field in
crates/switchyard-server/src/stats/algorithms.rs (lines 31-31), describing when
it appears in the serialized response. Also document every public field in
crates/switchyard-server/src/stats/algorithms/advisor_gate.rs (lines 42-53),
specifying each field’s count and grouping semantics.

In `@crates/switchyard-server/src/stats/algorithms/advisor_gate.rs`:
- Line 192: Update the stats binding in the test around StatsAccumulator::new to
be mutable, so the later StatsAccumulator::reset(&mut self) call compiles.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 585f7428-96a2-4510-9e3b-71963e67e64e

📥 Commits

Reviewing files that changed from the base of the PR and between 58f355a and 315d4b0.

⛔ Files ignored due to path filters (1)

Cargo.lock is excluded by !**/*.lock, !Cargo.lock

📒 Files selected for processing (11)

Cargo.toml
crates/libsy/Cargo.toml
crates/libsy/src/algorithms.rs
crates/libsy/src/algorithms/advisor_gate.rs
crates/libsy/src/algorithms/util/llm_judge.rs
crates/libsy/src/algorithms/util/prompts.rs
crates/libsy/src/lib.rs
crates/switchyard-server/src/config.rs
crates/switchyard-server/src/stats/algorithms.rs
crates/switchyard-server/src/stats/algorithms/advisor_gate.rs
crates/switchyard-server/tests/server.rs

coderabbitai · 2026-08-11T20:44:51Z

 #[derive(Clone, Debug, Default, PartialEq, Serialize)]
 pub(crate) struct AlgorithmStatsSnapshot {
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub advisor_gate: Option<AdvisorGateStatsSnapshot>,


📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the new public statistics fields.

crates/switchyard-server/src/stats/algorithms.rs#L31-L31: document when advisor_gate is present in the serialized response.

crates/switchyard-server/src/stats/algorithms/advisor_gate.rs#L42-L53: document the count and grouping semantics of each public field.

As per coding guidelines, add concise /// documentation for public Rust items.

📍 Affects 2 files

crates/switchyard-server/src/stats/algorithms.rs#L31-L31 (this comment)

crates/switchyard-server/src/stats/algorithms/advisor_gate.rs#L42-L53

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/stats/algorithms.rs` at line 31, Add concise Rust doc comments for the public advisor_gate field in crates/switchyard-server/src/stats/algorithms.rs (lines 31-31), describing when it appears in the serialized response. Also document every public field in crates/switchyard-server/src/stats/algorithms/advisor_gate.rs (lines 42-53), specifying each field’s count and grouping semantics.

Source: Coding guidelines

coderabbitai · 2026-08-11T20:44:51Z

+            .unwrap_or_else(|error| panic!("failed to build metrics exporter: {error}"));
+        let provider = SdkMeterProvider::builder().with_reader(exporter).build();
+        let meter = provider.meter("switchyard");
+        let stats = StatsAccumulator::new(registry, ["advisor_gate"]);


🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Make stats mutable.

Line 236 calls StatsAccumulator::reset(&mut self), but this binding is immutable. The test does not compile.

Proposed fix

- let stats = StatsAccumulator::new(registry, ["advisor_gate"]); + let mut stats = StatsAccumulator::new(registry, ["advisor_gate"]);

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

let stats = StatsAccumulator::new(registry, ["advisor_gate"]);

let mut stats = StatsAccumulator::new(registry, ["advisor_gate"]);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/stats/algorithms/advisor_gate.rs` at line 192, Update the stats binding in the test around StatsAccumulator::new to be mutable, so the later StatsAccumulator::reset(&mut self) call compiles.

Signed-off-by: zengyuanl <zengyuanl@nvidia.com>

ayushag-nv · 2026-08-12T01:17:42Z

@eric-liu-nvidia can you address coderabbit first

nachiketb-nvidia

Let's break this up into 2 or 3 MRs

one for core logic
one more for potential telemetry
one more for server side config changes

also, if we have any prompts, let's make sure they're also knobs

grahamking · 2026-08-12T12:53:51Z

+}
+
+#[tokio::test]
+async fn advisor_route_approve_flow_and_stats() -> TestResult {


Can you wrap all of your tests in a #[cfg(test)] mod tests {? That will ensure they don't get compiled unless you are running tests. Grep for those strings in most other files for an example.

eric-liu-nvidia requested a review from a team as a code owner August 11, 2026 20:37

eric-liu-nvidia mentioned this pull request Aug 11, 2026

feat(advisor): review-gate advisor backend and advisor route type #359

Closed

10 tasks

coderabbitai Bot reviewed Aug 11, 2026

View reviewed changes

eric-liu-nvidia force-pushed the claude/advisor-review-gate-rust branch from 315d4b0 to 136bc21 Compare August 11, 2026 21:13

eric-liu-nvidia added 4 commits August 11, 2026 21:25

feat(libsy): advisor_gate review-gate algorithm

e10e5f8

Signed-off-by: zengyuanl <zengyuanl@nvidia.com>

feat(server): advisor route type

038161b

Signed-off-by: zengyuanl <zengyuanl@nvidia.com>

feat(server): advisor_gate stats projection

ee2ae14

Signed-off-by: zengyuanl <zengyuanl@nvidia.com>

test(server): advisor route end-to-end coverage

136bc21

Signed-off-by: zengyuanl <zengyuanl@nvidia.com>

nachiketb-nvidia requested changes Aug 12, 2026

View reviewed changes

grahamking reviewed Aug 12, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(libsy): advisor review gate as a native algorithm and advisor route type - #371

feat(libsy): advisor review gate as a native algorithm and advisor route type#371
eric-liu-nvidia wants to merge 4 commits into
mainfrom
claude/advisor-review-gate-rust

eric-liu-nvidia commented Aug 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 11, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-08-11 21:14 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Aug 11, 2026

Uh oh!

coderabbitai Bot Aug 11, 2026

Uh oh!

ayushag-nv commented Aug 12, 2026

Uh oh!

nachiketb-nvidia left a comment

Uh oh!

grahamking Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	let stats = StatsAccumulator::new(registry, ["advisor_gate"]);
	let mut stats = StatsAccumulator::new(registry, ["advisor_gate"]);

Uh oh!

Conversation

eric-liu-nvidia commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results (Terminal-Bench 2.1)

How it works

Example

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Built to branch gh-pages at 2026-08-11 21:14 UTC. Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

coderabbitai Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

ayushag-nv commented Aug 12, 2026

Uh oh!

nachiketb-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

grahamking Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eric-liu-nvidia commented Aug 11, 2026 •

edited

Loading

github-actions Bot commented Aug 11, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-08-11 21:14 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading