bench: tb21 harness carry + usage accounting, now synced with main (merge, no rebase) - #5145
Open
tlongwell-block wants to merge 8 commits into
Open
bench: tb21 harness carry + usage accounting, now synced with main (merge, no rebase)#5145tlongwell-block wants to merge 8 commits into
tlongwell-block wants to merge 8 commits into
Conversation
Brings the benchmark harness (accounting, container runtime, manifests, personas, summarize/leaderboard tooling) forward onto main. main does not touch benchmarks/, so this is the branch tree verbatim. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>
Two harness-facing deltas main does not already carry: - acp.rs: `tool_call: shell (other)` records that a shell ran but not what it ran, and the command is not persisted anywhere else in a session bundle. Without it a teammate that read the repo and one that rewrote it leave identical logs, which is the whole content of a read-only role. - config.rs: tests pinning that the benchmark's endpoint spellings survive the effort clamp. An unrecognised model silently clamps down, which turns a graded sweep into a null that reads as a finding. Everything else this branch carried in crates/ (cache_control breakpoints, cached-token accounting, provider_extra/thoughtSignature, turn-start logging) has landed on main independently, so main's version is taken. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>
An OpenRouter model id is a family of upstream deployments, not one deployment. `deepseek-v4-flash-0731` is served by nine upstreams spanning fp4 to fp8 quantization, 262K to 1M context and a 1.6x price spread; `kimi-k3` by ten spanning 1.5x. Unpinned, consecutive requests land on different ones, so a benchmark cell measures a mixture whose composition moves with provider load rather than a fixed condition. It also decides whether prompt caching happens at all. Measured today against a fixed 21.8K-token prefix, three calls each: deepseek, unpinned cached 0, 0, 21760 (GMICloud, Cloudflare x2) deepseek gmicloud/fp8 cached 21760 x3 $0.00059/call deepseek siliconflow/fp8 cached 0 x3 $0.00306/call kimi, unpinned cached 0, 0, 20480 (Fireworks, Moonshot x2) kimi moonshotai/mxfp4 cached 20480 x3 $0.00715 vs $0.06245 That is a 5-9x swing in input cost for identical work, decided by routing luck. The `/endpoints` metadata is no guide: it advertises `supports_implicit_caching: false` for every endpoint measured caching above, and `true` only for one that is not routable on this account. So: `OPENROUTER_PROVIDER_ORDER` (comma-separated, accepting both the bare `moonshotai` slug and the `gmicloud/fp8` slug/quantization form) sets `provider.order`. It is strictly opt-in -- unset leaves the body byte identical, which is what keeps the existing "no body shape adds a provider routing filter" assertions honest. Paired with `allow_fallbacks: false`, because `order` alone is only a preference: OpenRouter still serves from elsewhere when the named upstream is busy, which is exactly the silent mid-run condition change this exists to prevent. A pin that quietly falls back is not a pin. This is not `require_parameters` -- that filter screens on advertised parameters and 404s otherwise-valid model ids, which is why the reasoning block nearby stays away from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Atish Patel <atish@squareup.com>
Three cells, all at reasoning effort `high`, all on pinned OpenRouter
upstreams:
tb-solo-kimi-k3-high TB 2.1 solo, moonshotai/kimi-k3
tb-solo-deepseek-v4-flash-high TB 2.1 solo, deepseek-v4-flash-0731
lhtb-or-kimi-lead-2deepseek-high LHTB-46 team: kimi lead, deepseek
scout, deepseek worker
The first cells in the study on open-weight models and on a route other
than direct-OpenAI or Databricks. The solo pair spans a 22x input price
ratio ($0.133 vs $3.00/Mtok), making it the open-weight analogue of the
luna -> sol ladder; the team cell is the open-weight restaging of
tb-gt-sol-luna-terra-high, which is currently the study's only clean
long-horizon team win.
NO MEDIUM-EFFORT CELLS. The original plan had a high/medium pair per
model. Neither model supports it: both advertise supported_efforts
["max","high","low"], and while OpenRouter accepts `medium` without
error, measured reasoning length on deepseek did not separate low from
medium from high (medians 1097 / 1102 / 1067, within-level spread wider
than between-level). A cell labelled `medium` would name a condition the
provider does not implement, so the effort axis is dropped rather than
faked.
Effort is still pinned on all three cells even though `high` matches
deepseek's provider default -- an unpinned cell records no effort in its
manifest, so a later provider-side default change would silently redefine
the condition without moving the hash. kimi defaults to `max`, so there
`high` is a deliberate step down.
Prices are the PINNED endpoints' own rates, not the model ids' headline
rates -- gmicloud/fp8 bills $0.133/$0.266 where the model id advertises
$0.14/$0.28, and two of kimi's ten upstreams bill 1.5x the row here. The
pin is what makes these rows the correct ones, so dropping it would
require restating cost as well as re-establishing the condition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Atish Patel <atish@squareup.com>
`_settle_usage` waited for the agent to report its tokens before teardown killed it, but only on the success path. The timeout path fell straight through to the kill, on the stated reasoning that "a turn that never completed has no usage to flush". That was true while buzz-agent reported once per turn. It is not true now that it reports after every provider round, and it was always the wrong path to skip: under `continue_until_timeout` every phase but the last ends here, so the skip applied to essentially the whole run. It is the harness half of the 97%-zero-rows / 5.9x-undercount measured on the OpenRouter LHTB-46 cell; the agent half is the per-round reporting. Moving the call into the `finally` covers both paths and keeps the ordering that matters — settle, then kill. It stays cheap: it returns on the first poll once a usage line exists, which after the first round it does, so only a phase that never completed a single round can spend the full budget, and that phase has nothing to report anyway. Depends on the per-round reporting in buzz-agent to be worth much on its own. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>
buzz-agent emitted its `usage_update` notification in exactly one place: after `ctx.run()` returned. Until then a turn's token counters lived only in the prompt task's stack frame. A turn that was killed mid-flight therefore reported nothing at all — the provider had already billed every round it completed, and no consumer ever saw any of it. That is not a corner case for anything that terminates a turn on a clock. Measured on a long-horizon benchmark run that relaunches its agent between phases, 97% of the recorded usage rows came back all zeros, and the run's recorded cost was $79.23 against the $485 the provider's own ledger billed for the same model over the same window — a 5.9x undercount, concentrated on the seat being relaunched. Nothing was truncated after the fact; the tokens were never captured in the first place. The run loop now emits a session-cumulative `usage_update` after every usage-bearing provider response, so an interrupted turn has reported everything but its single in-flight request. Emitting more than once per turn is already part of the contract — buzz-acp's UsageTracker advances its committed baseline only at publish time, and goose behaves the same way, which is why the tracker was written to tolerate it. The turn-start session baseline is snapshotted into RunCtx so the mid-turn figure stays session-cumulative rather than turn-local; a turn-local number would be discarded by a high-water-mark consumer and lose the turn entirely. It is taken by value rather than as a handle because the loop reports once per round, and taking the sessions lock on each of those would serialise concurrent sessions behind one another's provider round-trips. Both call sites now build the payload through one helper so their wire shapes cannot drift — a divergence there would present as tokens silently vanishing, which is the failure this reporting exists to prevent. A SIGTERM handler was the obvious shape and does not work: at signal time the counters are not anywhere a handler could reach. Making usage durable during the turn is what fixes it. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>
Brings the bench branch (forked from main at 3d7712c, Aug 1) up to current main (f53bbd1) as a merge commit — no rebase, no history rewrite — so the branch stops silently drifting behind agent and runtime fixes. Notably picks up: - #4896: recover from unsupported image input instead of poisoning the turn (the 'Poisoned Polaroid' that tainted 15 tb21 trials) - #5136: mention the orchestrator by pubkey when posting the task (the 'Register Grabber' that killed large-scale-text-editing) One conflict, container_runtime.py: the bench branch's solo-mode _wait_for_done changes vs #5136's _send mention plumbing. Resolved by keeping both (same resolution as the validated cherry-pick 2944a88). Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…8 pin (run 2) Commits the tb21 Meli-solo condition that until now lived only as untracked files on the runner box, plus the take-2 variant Tyler asked for (thread 74a65a09, event f7457e0b): - tb-meli-solo.yaml + personas/bench/meli-solo.md: the exact manifest and persona (sha256 de633019...) that produced run tb21-solo-1 on the gmicloud/fp8 pin. Recorded for provenance. - tb-meli-solo-baseten.yaml: identical cell, condition tb-meli-solo-baseten, priced at the baseten/fp8 listed rates (prompt $0.13/M, completion $0.26/M — OpenRouter endpoints API, 2026-08-07). - testbed/endpoints/openrouter-baseten.json: the provider pin, OPENROUTER_PROVIDER_ORDER=baseten/fp8. A separate endpoint file so openrouter-live.json (which run 1's provenance points at) stays byte-identical. Run 2 executes on the post-merge harness (9cafe8f, PR #5145), which carries #4896 (vision-input recovery) and #5136 (explicit-mention task post) that run 1 predated. Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this branch is
benchmark/harness-accounting-and-solois the branch both tb21 A/B benchmark boxes run (soloi-0f04fcd287e19da4e, teami-05e08e50c1c714f47, deepseek-v4-flash via OpenRouter). Until now it lived as an orphan branch — never in a PR — which is how it silently drifted 128 commits behind main and missed agent-level fixes like #4896. This PR gives it a review surface and a drift banner.The 6 bench-only commits (what main doesn't have)
c6d7b356bench: carry the harbor-buzz-orchestra harness onto latest main90b56ba2bench: log what a tool call touched, and pin the effort-wave endpoints10f09396bench: pin the OpenRouter upstream so a cell is one condition131891fcbench: add the OpenRouter wave — kimi-k3 and deepseek-v4-flash cellsb915d567fix: settle usage on the timeout path, not only after DONE1cf841affix: report agent usage per provider round, not once per turn(Author of the bench/accounting commits: @atish — flagging for review of which of these should graduate to main.)
The new merge commit (
9cafe8f1)Merges current
origin/mainin as a merge commit — no rebase, no history rewrite, no force push. This picks up, among 128 commits:@aVim-macro mention-resolver crash that killedlarge-scale-text-editingin both conditions)One conflict:
container_runtime.py— the bench branch's solo-mode_wait_for_donechanges vs #5136's_send(mention=)plumbing. Resolved by keeping both; identical to the resolution already e2e-validated on the scratch box (brancheva/rg-fix-on-bench-branch2944a88, rerun jobrg-fix-1, reward 1.0, zero RuntimeLaunchError).Validation
benchmarks/harbor-buzz-orchestra: full pytest suite at merge head 9cafe8f — 273 passed.tests/test_container_runtime.py(unusedFORWARDERimport) — present at bench head 1cf841a before the merge, not introduced here.git merge-base --is-ancestorconfirms fix(agent): recover from unsupported image input instead of poisoning the turn #4896 (8a7eb8d), fix(bench): mention the orchestrator by pubkey when posting the task #5136 merge commit (f53bbd1), and full origin/main are all ancestors of 9cafe8f.Intent / merge posture
Primarily a visibility PR: review what's bench-only and decide what graduates to main. Safe to leave open as the bench branch's standing PR. The two in-flight A/B boxes are still pinned to 1cf841a and are unaffected; post-run reruns of tainted trials will build from this merge head after a re-smoke on the scratch box.
Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b