feat(server): opt-in cost savings endpoint and live dashboard - #378
feat(server): opt-in cost savings endpoint and live dashboard#378michaelneale wants to merge 4 commits into
Conversation
Adds per-target pricing (USD per 1M tokens) to the server TOML and, when any target is priced, registers GET /v1/savings (JSON) and GET /dashboard (self-contained live HTML page). Savings compare actual routed spend against a baseline model - what the same traffic would have cost if every request had been served by the most capable target. Classifier calls are counted as routing overhead against the savings. Pricing semantics (base input / cache read / cache write / output buckets) match switchyard.cli.launchers.cost_estimator. Fully opt-in: no pricing in config means no new routes and no behaviour change. Signed-off-by: Michael Neale <michael.neale@gmail.com>
…by model id Signed-off-by: Michael Neale <michael.neale@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
WalkthroughThe change adds optional model pricing and savings accounting. Configured servers expose ChangesSavings reporting
Estimated code review effort: 3 (Moderate) | ~30 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
crates/switchyard-server/src/savings_dashboard.html (2)
130-144: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse
fmtfor the call count, asrenderModelsdoes.Line 141 interpolates
m.callsdirectly.renderModelspasses the same field throughfmt(m.calls)on line 153. Align the two so the distribution row groups thousands the same way as the table.This also removes the only raw interpolation of a server-supplied value in the template. The static analysis
inner-outer-htmlwarnings on lines 133-142 and 159-163 are handled:escalready wraps every model name, and the remaining values are numeric. Themanual-sanitizationhint recommends DOMPurify, which does not apply here. The page is embedded withinclude_str!and is intentionally dependency-free.♻️ Proposed consistency change
- <div class="pct">${m.calls} calls · ${pct.toFixed(1)}%</div> + <div class="pct">${fmt(m.calls)} calls · ${pct.toFixed(1)}%</div>🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings_dashboard.html` around lines 130 - 144, Update renderDist to format m.calls with the existing fmt helper in the distribution row, matching renderModels while leaving the percentage calculation unchanged.Source: Linters/SAST tools
195-199: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReport a failed reset to the user.
The reset handler ignores the response status. If
POST /v1/stats/resetreturns an error, the page callsrefresh()and shows the unchanged counters with the status still set to "live". The operator gets no signal that the reset failed.♻️ Proposed change
$("reset").addEventListener("click", async () => { - await fetch("/v1/stats/reset", {method:"POST"}); - refresh(); + try { + const r = await fetch("/v1/stats/reset", {method:"POST"}); + if(!r.ok) throw new Error(await r.text()); + } catch(e) { + $("status").textContent = "reset failed"; + return; + } + refresh(); });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings_dashboard.html` around lines 195 - 199, Update the reset click handler near the $("reset") listener to inspect the POST /v1/stats/reset response before calling refresh(). On a non-success response, report the reset failure through the page’s existing user-visible status/error mechanism and avoid presenting the unchanged counters as successfully reset; retain the refresh flow only for successful resets.crates/switchyard-server/src/savings.rs (2)
91-102: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider rounding the per-model costs too.
computeappliesround6to the snapshot totals but stores the rawf64forModelSavings::costandModelSavings::baseline_cost. The JSON response therefore mixes rounded totals with full-precision per-model values, and the per-model column can show long floats to any consumer that does not format them. The bundled dashboard formats withtoFixed, so nothing is visibly wrong today.Apply
round6to both per-model fields for a consistent response contract.♻️ Proposed consistency change
- cost, - baseline_cost: would_be, + cost: round6(cost), + baseline_cost: round6(would_be),Also applies to: 199-205
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings.rs` around lines 91 - 102, Apply the existing round6 helper to both cost and baseline_cost when constructing each ModelSavings entry in compute, while leaving the raw calculations and priced flag unchanged.
45-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRestrict
is_emptyunless it is part of the external API. No in-repository code callsSavingsConfig::is_empty; usepub(crate)or remove it if external callers do not need it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings.rs` around lines 45 - 48, Restrict the visibility of SavingsConfig::is_empty from public external API access to pub(crate), or remove the method if it is not required by external callers; preserve its existing pricing.is_empty behavior if retained.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/switchyard-server/src/config.rs`:
- Around line 129-159: Validate every pricing rate in apply_savings before
converting entries with into_model_price. Reject negative or non-finite f64
values, including NaN and infinities, by returning a ServerError during
configuration loading; only construct SavingsConfig after all rates pass
validation, preserving the existing baseline_model checks.
In `@crates/switchyard-server/src/savings_dashboard.html`:
- Around line 167-202: Update refresh and the polling setup to prevent
concurrent requests: track an in-flight state, return immediately when refresh
is already running or document.hidden is true, and clear the state in all
completion paths. Replace the unconditional setInterval polling with
visibility-aware scheduling that resumes on visibility changes and preserves
manual refresh behavior; do not apply the React-specific setstate-same-var hint.
In `@crates/switchyard-server/src/savings.rs`:
- Around line 105-119: Update the classifier-cost loop in the savings
calculation to append each classifier model lacking a price to the existing
unpriced_models collection, matching the routed-model loop’s behavior. Keep its
cost at zero while ensuring the model is surfaced for reporting and dashboard
warnings.
---
Nitpick comments:
In `@crates/switchyard-server/src/savings_dashboard.html`:
- Around line 130-144: Update renderDist to format m.calls with the existing fmt
helper in the distribution row, matching renderModels while leaving the
percentage calculation unchanged.
- Around line 195-199: Update the reset click handler near the $("reset")
listener to inspect the POST /v1/stats/reset response before calling refresh().
On a non-success response, report the reset failure through the page’s existing
user-visible status/error mechanism and avoid presenting the unchanged counters
as successfully reset; retain the refresh flow only for successful resets.
In `@crates/switchyard-server/src/savings.rs`:
- Around line 91-102: Apply the existing round6 helper to both cost and
baseline_cost when constructing each ModelSavings entry in compute, while
leaving the raw calculations and priced flag unchanged.
- Around line 45-48: Restrict the visibility of SavingsConfig::is_empty from
public external API access to pub(crate), or remove the method if it is not
required by external callers; preserve its existing pricing.is_empty behavior if
retained.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 65f3e994-00e7-43b1-9749-e66f9d453e24
📒 Files selected for processing (7)
CHANGELOG.mdcrates/switchyard-server/src/config.rscrates/switchyard-server/src/lib.rscrates/switchyard-server/src/savings.rscrates/switchyard-server/src/savings_dashboard.htmldocs/operations/cost_savings.mdmkdocs.yml
| fn apply_savings(&self, state: ServerState) -> ServerResult<ServerState> { | ||
| if self.pricing.is_empty() { | ||
| if self.savings.is_some() { | ||
| return Err(ServerError::new( | ||
| "[savings] requires a [pricing] table with at least one model", | ||
| )); | ||
| } | ||
| return Ok(state); | ||
| } | ||
| let pricing: BTreeMap<String, ModelPrice> = self | ||
| .pricing | ||
| .iter() | ||
| .map(|(model, config)| (model.clone(), config.into_model_price())) | ||
| .collect(); | ||
| let baseline = match self | ||
| .savings | ||
| .as_ref() | ||
| .and_then(|s| s.baseline_model.as_ref()) | ||
| { | ||
| Some(model) => { | ||
| if !pricing.contains_key(model) { | ||
| return Err(ServerError::new(format!( | ||
| "savings baseline_model {model} has no [pricing.\"{model}\"] entry" | ||
| ))); | ||
| } | ||
| Some(model.clone()) | ||
| } | ||
| None => None, | ||
| }; | ||
| Ok(state.with_savings(SavingsConfig::new(pricing, baseline))) | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Validate the pricing rates before you build ModelPrice.
apply_savings accepts any f64 from the TOML, including negative, inf, and nan. A nan or inf rate propagates through SavingsConfig::compute into actual_cost, baseline_cost, saved, and saved_pct. serde_json cannot serialize non-finite floats, so GET /v1/savings then fails at response time instead of at startup. A negative rate silently produces negative spend.
Other numeric configuration in this file is validated at build time (base_threshold, max_retries, context_window). Apply the same treatment here.
🛡️ Proposed fix to reject non-finite and negative rates
- let pricing: BTreeMap<String, ModelPrice> = self
- .pricing
- .iter()
- .map(|(model, config)| (model.clone(), config.into_model_price()))
- .collect();
+ let mut pricing: BTreeMap<String, ModelPrice> = BTreeMap::new();
+ for (model, config) in &self.pricing {
+ let price = config.into_model_price();
+ for (field, rate) in [
+ ("input", price.input),
+ ("output", price.output),
+ ("cached", price.cached),
+ ("cache_write", price.cache_write),
+ ] {
+ if !rate.is_finite() || rate < 0.0 {
+ return Err(ServerError::new(format!(
+ "pricing {model} {field} must be finite and greater than or equal to 0"
+ )));
+ }
+ }
+ pricing.insert(model.clone(), price);
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| fn apply_savings(&self, state: ServerState) -> ServerResult<ServerState> { | |
| if self.pricing.is_empty() { | |
| if self.savings.is_some() { | |
| return Err(ServerError::new( | |
| "[savings] requires a [pricing] table with at least one model", | |
| )); | |
| } | |
| return Ok(state); | |
| } | |
| let pricing: BTreeMap<String, ModelPrice> = self | |
| .pricing | |
| .iter() | |
| .map(|(model, config)| (model.clone(), config.into_model_price())) | |
| .collect(); | |
| let baseline = match self | |
| .savings | |
| .as_ref() | |
| .and_then(|s| s.baseline_model.as_ref()) | |
| { | |
| Some(model) => { | |
| if !pricing.contains_key(model) { | |
| return Err(ServerError::new(format!( | |
| "savings baseline_model {model} has no [pricing.\"{model}\"] entry" | |
| ))); | |
| } | |
| Some(model.clone()) | |
| } | |
| None => None, | |
| }; | |
| Ok(state.with_savings(SavingsConfig::new(pricing, baseline))) | |
| } | |
| fn apply_savings(&self, state: ServerState) -> ServerResult<ServerState> { | |
| if self.pricing.is_empty() { | |
| if self.savings.is_some() { | |
| return Err(ServerError::new( | |
| "[savings] requires a [pricing] table with at least one model", | |
| )); | |
| } | |
| return Ok(state); | |
| } | |
| let mut pricing: BTreeMap<String, ModelPrice> = BTreeMap::new(); | |
| for (model, config) in &self.pricing { | |
| let price = config.into_model_price(); | |
| for (field, rate) in [ | |
| ("input", price.input), | |
| ("output", price.output), | |
| ("cached", price.cached), | |
| ("cache_write", price.cache_write), | |
| ] { | |
| if !rate.is_finite() || rate < 0.0 { | |
| return Err(ServerError::new(format!( | |
| "pricing {model} {field} must be finite and greater than or equal to 0" | |
| ))); | |
| } | |
| } | |
| pricing.insert(model.clone(), price); | |
| } | |
| let baseline = match self | |
| .savings | |
| .as_ref() | |
| .and_then(|s| s.baseline_model.as_ref()) | |
| { | |
| Some(model) => { | |
| if !pricing.contains_key(model) { | |
| return Err(ServerError::new(format!( | |
| "savings baseline_model {model} has no [pricing.\"{model}\"] entry" | |
| ))); | |
| } | |
| Some(model.clone()) | |
| } | |
| None => None, | |
| }; | |
| Ok(state.with_savings(SavingsConfig::new(pricing, baseline))) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/switchyard-server/src/config.rs` around lines 129 - 159, Validate
every pricing rate in apply_savings before converting entries with
into_model_price. Reject negative or non-finite f64 values, including NaN and
infinities, by returning a ServerError during configuration loading; only
construct SavingsConfig after all rates pass validation, preserving the existing
baseline_model checks.
| async function refresh(){ | ||
| try { | ||
| const r = await fetch("/v1/savings"); | ||
| if(!r.ok) throw new Error(await r.text()); | ||
| const d = await r.json(); | ||
| $("saved_pct").textContent = d.saved_pct.toFixed(1) + "%"; | ||
| $("saved_usd").textContent = money(d.saved); | ||
| $("actual").textContent = money(d.actual_cost); | ||
| $("baseline").textContent = money(d.baseline_cost); | ||
| $("requests").textContent = fmt(d.total_requests); | ||
| $("classifier").textContent = money(d.classifier_cost); | ||
| $("baseline_model").textContent = d.baseline_model || "—"; | ||
| renderDist(d.models || {}); | ||
| renderModels(d.models || {}); | ||
| const unpriced = d.unpriced_models || []; | ||
| if(unpriced.length){ | ||
| $("unpriced").style.display = "block"; | ||
| $("unpriced").textContent = | ||
| "No price configured for: " + unpriced.join(", ") + " — costed at $0."; | ||
| } else { | ||
| $("unpriced").style.display = "none"; | ||
| } | ||
| $("status").textContent = "live"; | ||
| } catch(e) { | ||
| $("status").textContent = "fetch failed"; | ||
| } | ||
| } | ||
|
|
||
| $("refresh").addEventListener("click", refresh); | ||
| $("reset").addEventListener("click", async () => { | ||
| await fetch("/v1/stats/reset", {method:"POST"}); | ||
| refresh(); | ||
| }); | ||
|
|
||
| refresh(); | ||
| setInterval(refresh, 2000); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Guard against overlapping polls.
setInterval(refresh, 2000) starts a new request every two seconds regardless of whether the previous one finished. fetch has no timeout and no AbortController. If the server stalls or the network is slow, requests accumulate and saturate the browser's per-origin connection pool. The interval also keeps polling while the tab is hidden.
Add an in-flight flag, and pause polling when the document is hidden.
The ast-grep setstate-same-var hint on line 201 is a React rule and does not apply to this page.
🛡️ Proposed fix
+let inFlight = false;
async function refresh(){
+ if(inFlight) return;
+ inFlight = true;
try {
const r = await fetch("/v1/savings");
if(!r.ok) throw new Error(await r.text()); $("status").textContent = "live";
} catch(e) {
$("status").textContent = "fetch failed";
+ } finally {
+ inFlight = false;
}
} refresh();
-setInterval(refresh, 2000);
+setInterval(() => { if(!document.hidden) refresh(); }, 2000);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async function refresh(){ | |
| try { | |
| const r = await fetch("/v1/savings"); | |
| if(!r.ok) throw new Error(await r.text()); | |
| const d = await r.json(); | |
| $("saved_pct").textContent = d.saved_pct.toFixed(1) + "%"; | |
| $("saved_usd").textContent = money(d.saved); | |
| $("actual").textContent = money(d.actual_cost); | |
| $("baseline").textContent = money(d.baseline_cost); | |
| $("requests").textContent = fmt(d.total_requests); | |
| $("classifier").textContent = money(d.classifier_cost); | |
| $("baseline_model").textContent = d.baseline_model || "—"; | |
| renderDist(d.models || {}); | |
| renderModels(d.models || {}); | |
| const unpriced = d.unpriced_models || []; | |
| if(unpriced.length){ | |
| $("unpriced").style.display = "block"; | |
| $("unpriced").textContent = | |
| "No price configured for: " + unpriced.join(", ") + " — costed at $0."; | |
| } else { | |
| $("unpriced").style.display = "none"; | |
| } | |
| $("status").textContent = "live"; | |
| } catch(e) { | |
| $("status").textContent = "fetch failed"; | |
| } | |
| } | |
| $("refresh").addEventListener("click", refresh); | |
| $("reset").addEventListener("click", async () => { | |
| await fetch("/v1/stats/reset", {method:"POST"}); | |
| refresh(); | |
| }); | |
| refresh(); | |
| setInterval(refresh, 2000); | |
| let inFlight = false; | |
| async function refresh(){ | |
| if(inFlight) return; | |
| inFlight = true; | |
| try { | |
| const r = await fetch("/v1/savings"); | |
| if(!r.ok) throw new Error(await r.text()); | |
| const d = await r.json(); | |
| $("saved_pct").textContent = d.saved_pct.toFixed(1) + "%"; | |
| $("saved_usd").textContent = money(d.saved); | |
| $("actual").textContent = money(d.actual_cost); | |
| $("baseline").textContent = money(d.baseline_cost); | |
| $("requests").textContent = fmt(d.total_requests); | |
| $("classifier").textContent = money(d.classifier_cost); | |
| $("baseline_model").textContent = d.baseline_model || "—"; | |
| renderDist(d.models || {}); | |
| renderModels(d.models || {}); | |
| const unpriced = d.unpriced_models || []; | |
| if(unpriced.length){ | |
| $("unpriced").style.display = "block"; | |
| $("unpriced").textContent = | |
| "No price configured for: " + unpriced.join(", ") + " — costed at $0."; | |
| } else { | |
| $("unpriced").style.display = "none"; | |
| } | |
| $("status").textContent = "live"; | |
| } catch(e) { | |
| $("status").textContent = "fetch failed"; | |
| } finally { | |
| inFlight = false; | |
| } | |
| } | |
| $("refresh").addEventListener("click", refresh); | |
| $("reset").addEventListener("click", async () => { | |
| await fetch("/v1/stats/reset", {method:"POST"}); | |
| refresh(); | |
| }); | |
| refresh(); | |
| setInterval(() => { if(!document.hidden) refresh(); }, 2000); |
🧰 Tools
🪛 ast-grep (0.45.1)
[warning] 201-201: Avoid using the initial state variable in setState
Context: setInterval(refresh, 2000)
Note: [CWE-710] Improper Adherence to Coding Standards. Security best practice.
(setstate-same-var)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/switchyard-server/src/savings_dashboard.html` around lines 167 - 202,
Update refresh and the polling setup to prevent concurrent requests: track an
in-flight state, return immediately when refresh is already running or
document.hidden is true, and clear the state in all completion paths. Replace
the unconditional setInterval polling with visibility-aware scheduling that
resumes on visibility changes and preserves manual refresh behavior; do not
apply the React-specific setstate-same-var hint.
Source: Linters/SAST tools
| // Classifier / judge calls are pure routing overhead: they add to the | ||
| // actual spend but a baseline deployment would not make them at all. | ||
| let mut classifier_cost = 0.0; | ||
| for (model, m) in &stats.classifier.models { | ||
| let tokens = TokenBuckets { | ||
| prompt: m.prompt_tokens, | ||
| completion: m.completion_tokens, | ||
| cached: m.cached_tokens, | ||
| cache_creation: m.cache_creation_tokens, | ||
| }; | ||
| if let Some(price) = self.price_for(model) { | ||
| classifier_cost += tokens.cost(price); | ||
| } | ||
| } | ||
| actual_cost += classifier_cost; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Report unpriced classifier models in unpriced_models.
The routed-model loop pushes every model with no price into unpriced_models. The classifier loop does not. A classifier target with no [pricing] entry is therefore costed at zero and stays invisible.
docs/operations/cost_savings.md states that models which served traffic without a pricing entry are costed at zero and listed in unpriced_models so the under-count is visible. Classifier traffic currently breaks that statement. The dashboard warning box also stays hidden in this case.
🐛 Proposed fix to surface unpriced classifier models
if let Some(price) = self.price_for(model) {
classifier_cost += tokens.cost(price);
+ } else if !unpriced_models.contains(model) {
+ unpriced_models.push(model.clone());
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Classifier / judge calls are pure routing overhead: they add to the | |
| // actual spend but a baseline deployment would not make them at all. | |
| let mut classifier_cost = 0.0; | |
| for (model, m) in &stats.classifier.models { | |
| let tokens = TokenBuckets { | |
| prompt: m.prompt_tokens, | |
| completion: m.completion_tokens, | |
| cached: m.cached_tokens, | |
| cache_creation: m.cache_creation_tokens, | |
| }; | |
| if let Some(price) = self.price_for(model) { | |
| classifier_cost += tokens.cost(price); | |
| } | |
| } | |
| actual_cost += classifier_cost; | |
| // Classifier / judge calls are pure routing overhead: they add to the | |
| // actual spend but a baseline deployment would not make them at all. | |
| let mut classifier_cost = 0.0; | |
| for (model, m) in &stats.classifier.models { | |
| let tokens = TokenBuckets { | |
| prompt: m.prompt_tokens, | |
| completion: m.completion_tokens, | |
| cached: m.cached_tokens, | |
| cache_creation: m.cache_creation_tokens, | |
| }; | |
| if let Some(price) = self.price_for(model) { | |
| classifier_cost += tokens.cost(price); | |
| } else if !unpriced_models.contains(model) { | |
| unpriced_models.push(model.clone()); | |
| } | |
| } | |
| actual_cost += classifier_cost; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/switchyard-server/src/savings.rs` around lines 105 - 119, Update the
classifier-cost loop in the savings calculation to append each classifier model
lacking a price to the existing unpriced_models collection, matching the
routed-model loop’s behavior. Keep its cost at zero while ensuring the model is
surfaced for reporting and dashboard warnings.
Signed-off-by: Michael Neale <michael.neale@gmail.com>
1 participant
What
Adds opt-in cost savings reporting to
switchyard-server: an optional[pricing]table in the deployment TOML enablesGET /v1/savings(JSON) andGET /dashboard(self-contained live HTML page) that compare actual routed spend against a baseline model — "what would this traffic have cost if every request went to the most capable target".Why
The routing algorithms (llm_classifier, stage_router, escalation) exist largely to spend the capable model only on turns that need it. Today the only way to see what that actually saved is post-hoc analysis of
/v1/statsdumps (as the Pythoncost_estimatorin the launchers does). This makes the payoff visible live while a coding agent runs through the proxy.How
Purely additive. No
[pricing]table → the endpoints are not registered and behavior is unchanged. Pricing never influences routing decisions.Config:
[pricing]is keyed by model id (target.id), matching how stats are keyed, so one entry covers all targets sharing a model.Pricing semantics (base input / cache read / cache write / output buckets) match
switchyard.cli.launchers.cost_estimator.Classifier/judge traffic is priced separately as
classifier_costand deducted from savings, so routing overhead is charged honestly against the result.Models serving traffic without a pricing entry are costed at zero and surfaced in
unpriced_models(and as a dashboard warning) so under-counting is visible.The dashboard is a single embedded HTML file with no external dependencies, polling
/v1/savingsevery 2s. Counters reset with the existingPOST /v1/stats/reset.Endpoint registration is gated the same way as the existing session-stats route.
Sample
/v1/savingsoutputFrom a real session (goose coding agent through an llm_classifier route, cheap judge, Sonnet weak / Opus strong, plus some Opus-pinned A/B traffic):
{ "total_requests": 11, "actual_cost": 0.2864, "baseline_cost": 0.4722, "classifier_cost": 0.0026, "saved": 0.1858, "saved_pct": 39.35, "baseline_model": "claude-opus-5", "models": { "claude-opus-5": { "calls": 4, "cost": 0.1581, "baseline_cost": 0.1581 }, "claude-sonnet-5": { "calls": 7, "cost": 0.1256, "baseline_cost": 0.3141 } }, "unpriced_models": [] }Testing
[savings]without pricing rejected, unpriced baseline rejected)cargo test -p switchyard-server— 62 tests green; fmt and clippy cleanNotes for reviewers
Summary by CodeRabbit
/v1/savingsfor savings metrics and/dashboardfor a live cost and savings dashboard.