Skip to content

fix(routing): share cost-tier forwarding for Auto and Jev - #139

Open
ayush-or wants to merge 4 commits into
mainfrom
codex/jev-cost-tier-forwarding
Open

ayush-or wants to merge 4 commits into
mainfrom
codex/jev-cost-tier-forwarding

Conversation

@ayush-or

@ayush-or ayush-or commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

Forward benchmark costTier through one request builder for Auto Router, Auto Beta Router, and Jev. Previously, the harness only constructed Auto plugins, so Jev silently omitted the requested tier; SWE Atlas and Deep SWE also passed undefined into their sandbox tier-forwarding check. A run labeled high could therefore use the API default or an old experiment instead of explicit live high.

What changed?

  • Replace the separate Auto/Jev helpers and Auto SDK-to-wire conversion with buildRouterPlugin, returning wire-format plugins for ordinary Responses requests, search requests, and sandbox agents. Normalize model variants once. Reuse Switchyard's existing algorithm builder unchanged.
  • Use the existing post-serialization extraBody transport for every router, removing the split between Auto's SDK path and Jev's raw-body path. The pinned SDK does not model Jev; no SDK update, cast, new proxy, or deployment-specific dependency is needed.
  • Preserve router-specific controls: Auto and Auto Beta retain cost_quality_tradeoff and pin_model; Jev receives only its explicit cost_tier. Omitted controls leave API defaults intact. Unrelated caller plugins survive, and an explicit same-ID extraBody.plugins override wins without creating duplicate router plugins.
  • Forward coding benchmark tiers into the existing sandbox capability check. The built-in mini_swe client and pi support forwarding; unsupported native sandbox agents fail before launching rather than silently benchmarking the wrong tier.
  • Parameterize the coding-benchmark regression test across Auto, Auto Beta, and Jev. Cover all three tiers on subsequent Responses turns, search web-plugin/server-tool coexistence, model variants, Auto-only controls, omission, and raw plugin overrides.
  • Generalize the README's tier instructions and explain why live-tier evaluations must omit old experiment IDs.

Before: typesafe/jev-router with costTier: "high" emitted no tier selector. After: it emits plugins: [{ "id": "jev-router", "cost_tier": "high" }]; Auto variants use their corresponding plugin IDs through the same builder.

How to test

  1. Run bun run format:check, bun run check, bun run typecheck, bun test, and bun run build.
  2. Inspect the wire tests for low/medium/high on Auto, Auto Beta, and Jev: the selected plugin must survive SDK serialization and subsequent turns, preserve the web plugin, and omit experiment headers unless requested.
  3. Inspect the SWE Atlas / Deep SWE layer tests: unsupported sandbox agents must reject explicit tiers for all three routers before dataset loading or agent launch. Check the separate pi forwarding tests and default-omission cases.
  4. For an eventual live smoke run, omit experiment IDs, select an explicit tier, and verify both config_version and cost_tier in Datadog before running a full matrix.

Validation at a3f44415c6fe518ea5928ccd5063591d40ed611e: formatting, lint, typecheck, all 1,693 tests (151 files, four snapshots), and declaration build passed locally after the final push. The first refactor push had a switch-case-braces lint failure; the follow-up fixes it, and the complete validation was rerun. Existing unrelated lint warnings remain. CI for this head and both CodeQL analyses passed. Wire tests use the real SDK serialization with an existing recorded response fixture, not a paid provider call.

Rollout and Harbor / Argo

Merge upstream, sync packages/bench-harness into openrouter-web through the existing subtree workflow, and deploy the consuming benchmark worker/images. Kepler on Argo still calls this harness through gcp-bench-trial, so moving orchestration does not remove the need for this fix. Do not edit the vendored copy directly.

Native Harbor has a separate, already-generic trial proxy: its requestOverlay.plugins can inject either router plugin after the agent serializes its request. Reuse that path for native Harbor runs; this PR does not add another Harbor adapter or claim that its deployed behavior was tested. The legacy sandbox rejection is not a blanket restriction on those agents under Harbor.

Remove jevtier-* experiment IDs from new live-tier runs. The separate API max-to-deep fallback fix must also deploy before rechecking the Atlas policy-404 case.

Reviewer focus

  • Keep the router plugin in wire form at the HTTP boundary; moving it back through the current SDK would reject Jev's unsupported plugin type.
  • Check that numeric tradeoff/model pinning never leak into Jev, explicit raw plugin overrides remain intact, and unsupported legacy sandbox agents still fail closed.
  • No production KV writes, benchmark launches, new infrastructure, or Harbor runtime changes are included.

Checklist

  • No secrets, restricted datasets, or private benchmark artifacts added.
  • HIPAA / ePHI impact: None — benchmark request controls and deterministic synthetic tests only; no new content sinks or changes to API HIPAA eligibility/access controls.
  • Local validation complete on the final pushed head.
  • CI and CodeQL complete; subtree sync, deployment, and a production smoke test remain separate rollout steps.

@ayush-or
ayush-or marked this pull request as ready for review October 9, 2026 01:43
@ayush-or
ayush-or requested a review from a team as a code owner October 9, 2026 01:43
@ayush-or ayush-or changed the title fix(routing): forward Jev cost tiers in benchmark requests fix(routing): share cost-tier forwarding for Auto and Jev Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant