Repository navigation
Conversation
ayush-or
marked this pull request as ready for review
October 9, 2026 01:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Forward benchmark
costTierthrough one request builder for Auto Router, Auto Beta Router, and Jev. Previously, the harness only constructed Auto plugins, so Jev silently omitted the requested tier; SWE Atlas and Deep SWE also passedundefinedinto their sandbox tier-forwarding check. A run labeled high could therefore use the API default or an old experiment instead of explicit live high.What changed?
buildRouterPlugin, returning wire-format plugins for ordinary Responses requests, search requests, and sandbox agents. Normalize model variants once. Reuse Switchyard's existing algorithm builder unchanged.extraBodytransport for every router, removing the split between Auto's SDK path and Jev's raw-body path. The pinned SDK does not model Jev; no SDK update, cast, new proxy, or deployment-specific dependency is needed.cost_quality_tradeoffandpin_model; Jev receives only its explicitcost_tier. Omitted controls leave API defaults intact. Unrelated caller plugins survive, and an explicit same-IDextraBody.pluginsoverride wins without creating duplicate router plugins.mini_sweclient andpisupport forwarding; unsupported native sandbox agents fail before launching rather than silently benchmarking the wrong tier.Before:
typesafe/jev-routerwithcostTier: "high"emitted no tier selector. After: it emitsplugins: [{ "id": "jev-router", "cost_tier": "high" }]; Auto variants use their corresponding plugin IDs through the same builder.How to test
bun run format:check,bun run check,bun run typecheck,bun test, andbun run build.piforwarding tests and default-omission cases.config_versionandcost_tierin Datadog before running a full matrix.Validation at
a3f44415c6fe518ea5928ccd5063591d40ed611e: formatting, lint, typecheck, all 1,693 tests (151 files, four snapshots), and declaration build passed locally after the final push. The first refactor push had a switch-case-braces lint failure; the follow-up fixes it, and the complete validation was rerun. Existing unrelated lint warnings remain. CI for this head and both CodeQL analyses passed. Wire tests use the real SDK serialization with an existing recorded response fixture, not a paid provider call.Rollout and Harbor / Argo
Merge upstream, sync
packages/bench-harnessintoopenrouter-webthrough the existing subtree workflow, and deploy the consuming benchmark worker/images. Kepler on Argo still calls this harness throughgcp-bench-trial, so moving orchestration does not remove the need for this fix. Do not edit the vendored copy directly.Native Harbor has a separate, already-generic trial proxy: its
requestOverlay.pluginscan inject either router plugin after the agent serializes its request. Reuse that path for native Harbor runs; this PR does not add another Harbor adapter or claim that its deployed behavior was tested. The legacy sandbox rejection is not a blanket restriction on those agents under Harbor.Remove
jevtier-*experiment IDs from new live-tier runs. The separate API max-to-deep fallback fix must also deploy before rechecking the Atlas policy-404 case.Reviewer focus
Checklist