diff --git a/benchmarks/harbor-buzz-orchestra/.gitignore b/benchmarks/harbor-buzz-orchestra/.gitignore index 356ac54922..5acf871b18 100644 --- a/benchmarks/harbor-buzz-orchestra/.gitignore +++ b/benchmarks/harbor-buzz-orchestra/.gitignore @@ -4,3 +4,4 @@ __pycache__/ .pytest_cache/ .benchmark/ jobs/ +jobs-archive/ diff --git a/benchmarks/harbor-buzz-orchestra/README.md b/benchmarks/harbor-buzz-orchestra/README.md index 0358c954e7..966855789c 100644 --- a/benchmarks/harbor-buzz-orchestra/README.md +++ b/benchmarks/harbor-buzz-orchestra/README.md @@ -36,6 +36,19 @@ roster: `endpoint_config` maps those endpoint names to providers, URLs, and API-key environment variables. The adapter contains no fixed roster or model. +Exactly one orchestrator is required; **workers are optional**. A roster of one +orchestrator and nothing else is the single-agent baseline — see +`manifests/tb-solo-sonnet.yaml` and `personas/solo-tb.md`. The lone agent gets +byte-identical wiring to a worker (same binaries, same MCP toolset, same env), +because a handicapped baseline would flatter every multi-agent condition +measured against it. + +`generation` has no `temperature`: `buzz-agent` exposes no temperature +environment variable, so the value could never reach the provider. It used to +be accepted, hashed into the condition identity, and silently discarded — with +a default of `0.0` while the measured provider default is `1.0`. It is now +rejected outright rather than quietly lying. + ## Run With the production compose stack and model endpoints already running, execute @@ -62,13 +75,237 @@ rather than deletes that channel, leaving the relay/Postgres event timeline and the per-agent acp/agent logs (downloaded into the trial's `buzz/` artifacts) available for analysis. +## Trial artifacts + +Every trial writes a self-contained bundle to its `buzz/` artifact directory — +the container is gone once the trial ends, so this is what survives: + +``` +manifest.json the frozen condition and its sha256 +.system-prompt.md the composed prompt as that agent saw it +.stdout/.stderr.log +receipts.jsonl one priced usage row per agent +endpoints.redacted.json provider and env-var *names* only, never values +summary.json the index to start from +``` + +The bundle exists so system prompts can be tuned after a sweep: the exact +prompt bytes sit next to what that agent cost. It is written even when the +trial fails or times out — a trial that burned tokens and then stalled still +cost money, and dropping it would bias cost figures toward successes. + +Because every agent runs with a live provider token and a Nostr private key in +its environment, the bundle is scanned for credential-shaped strings and the +result is recorded in `summary.json` under `secret_scan`. Check it before +sharing a bundle. + +### Container trust store + +Many Terminal-Bench images ship no `ca-certificates` package, so +`/etc/ssl/certs/ca-certificates.crt` does not exist and **every https client in +the container fails** — apt, curl, git, pip. `buzz-agent` is the exception: it +gets `SSL_CERT_FILE=/opt/buzz/ca-certificates.crt`, which nothing else reads. + +That asymmetry cost real scores. On the A1 sweep an agent hit a transient apt +error, rewrote `sources.list` from http to https to work around it, and left +the container in a state where the **verifier's** own `apt-get update` could +not validate a certificate. The verifier reported `E: Unable to locate package +curl`, never installed pytest, and the task was recorded as reward 0.0 — +identical in `result.json` to a model that got the answer wrong. + +So `_install_stack` copies the same bundle to `/etc/ssl/certs/ca-certificates.crt` +before the agents launch. It is offline (the bundle is already uploaded), so it +adds no egress dependency, and it **only writes when that path is missing or +empty** — a task whose subject is certificate handling keeps whatever store its +image shipped. + +The outcome is recorded per trial in `result.json` under +`agent_result.metadata.container_trust_store`: + +| value | meaning | +|---|---| +| `present` | the image had its own store; we changed nothing | +| `seeded` | the image had none and we installed ours | +| `failed` | the image had none and we could not install ours | + +`failed` is deliberately not fatal — the agent still reaches its provider +through `SSL_CERT_FILE`, and killing a runnable trial would trade a partial +handicap for a total one. But **read a 0.0 on a `failed` trial as suspect**, +not as a wrong answer. + +### Verifier dependencies + +81 of the 89 Terminal-Bench `test.sh` files begin with `apt-get install -y +curl`, to fetch the uv installer that installs pytest. Only 3 of 12 sampled +task images ship curl, so most trials do that install at scoring time — the one +moment when a transient apt failure is unrecoverable and lands as a 0.0. + +So the runtime installs `VERIFIER_DEPS` itself, and the timing is the point: + +- **After the agents are stopped.** Installing before would hand the agent a + tool its task image chose not to ship, which changes the thing being + measured. Doing it in teardown changes only what the verifier finds. +- **Before Harbor's verifier phase**, which is what needs it. + +The win is not saving the verifier's apt call — it still makes one. It is that +with curl already in dpkg's status file, `apt-get install -y curl` resolves +from the installed version and succeeds *even when `apt-get update` left the +index empty*. That is exactly the failure that scored `compile-compcert` 0.0. + +Recorded as `agent_result.metadata.container_verifier_deps`: `present` (image +shipped it) | `installed` | `unavailable`. Best-effort, like the trust store — +measured at ~7s per trial on the EC2 runner, which at `--n-concurrent 16` +is under a minute across a full 89-task sweep. + +The package list is deliberately just `curl`. The rest of the task set's apt +requests are a long tail no blanket pre-install should chase: git 3, binutils +1, everything else once. + +### Preflight + +`benchmark.py` refuses to start a sweep that cannot be scored. Two checks run +before the stack, the binaries, and the money: + +- `check_tls_not_intercepted()` — the **host's** path to PyPI is not being + MITM'd. A Cloudflare WARP session once turned a live 89-task sweep into 0/89. +- `check_container_can_be_scored()` — a **container** can seed its trust store + and install curl. The host reaching PyPI says nothing about this: the docker + proxy, the image's trust store and apt are a separate path, and it is the one + every verifier depends on. + +The container check runs the trials' own two setup commands — +`seed_trust_store_command()` and `install_verifier_deps_command()`, imported +from `container_runtime`, not reimplemented — against `ubuntu:24.04`, the base +most task images derive from and which likewise ships neither `ca-certificates` +nor curl. A preflight that proved something subtly different from what the +trials do would be worse than none, because it would read as a clean bill of +health. It costs ~7s. + +Both follow the same rule: a check that *cannot see* (no docker, no openssl, a +pull that timed out) prints `skipping check` and lets the run proceed. Only a +positive signal of breakage stops it. + +### What none of this covers + +An agent that switches apt to a mirror the network cannot reach at all. +`compile-compcert` failed that way — the agent moved apt to +`https://azure.archive.ubuntu.com`, whose Azure address is unreachable from the +Square egress path at :443 regardless of trust anchors. Restoring +`sources.list` before the verifier runs would close it; that is not implemented. + +`verifier_health.py` carries `unable to locate package` as the backstop, so a +verifier that still cannot install its dependencies is reported as broken by +`summarize.py` rather than averaged in as a zero. + +### Token and cost accounting + +`buzz-agent` reports cumulative token counts per turn, `buzz-acp` logs them +under the `acp::usage` tracing target, and the runtime parses them out of the +downloaded logs to price each agent against the manifest's `prices` table. The +runtime sets `RUST_LOG` to enable that target; if an endpoint config sets its +own `RUST_LOG`, the directive is appended rather than replaced, because losing +it would present as a free trial rather than as an error. + +**Cache reads are estimated, not measured.** `LlmResponse.input_tokens` is an +inclusive sum of plain, cache-read, and cache-write input and nothing +downstream carries the split. Each endpoint's `Price` therefore declares a +`cache_read_rate` (0.0–1.0); that fraction of input is billed at +`cached_input_per_million_usd` and the rest at the full rate. Every receipt +also carries `cost_usd_no_cache_discount`, so the assumption's leverage is +visible rather than baked into one number. + +The rate lives in the manifest, not in code, so it is frozen into the condition +hash — two cost figures are only comparable under the same assumption. + +Choosing a rate: + +- **Anthropic-route endpoints: `0.0`, and that is exact, not conservative.** + `buzz-agent` never sends `cache_control` and Anthropic prompt caching is + opt-in, so no cache reads occur. (It also means Buzz is leaving a real + discount on the table — worth fixing in the agent, not in the harness.) +- **OpenAI-route endpoints: non-zero.** OpenAI caches automatically above ~1k + prompt tokens and reports `cached_tokens`, and an agent loop resends a + growing prefix every turn, so the true rate is substantial. Calibrate it from + the gateway's own `cached_tokens` on a real trial rather than guessing. + +Leaving every rate at 0.0 gives the old upper-bound behaviour, but it is not +neutral across conditions: it over-charges exactly the endpoints that really do +cache, which biases OpenAI-route workers against Anthropic-route ones. + +**Reasoning tokens are not separable, and this does not affect cost.** Providers +bill thinking at the output rate, so reasoning tokens inside `output_tokens` are +already priced correctly. Only the thinking-vs-answer split is unavailable — +an analysis detail, not a cost error. + +**Auto-compaction is billed but unreported, so it is bounded instead.** When a +session crosses its compaction threshold, `buzz-agent` summarises the history +into a fresh context (a "handoff"). That summarisation is a real provider call, +but `Llm::summarize` returns a bare `String` and throws its usage away, so those +tokens never reach the counters above. On a long trial with the default +`max_handoffs` of 80 the omission can exceed everything else we model. + +The runtime parses the handoff log lines — which carry the pre-handoff context +size — and publishes a worst case beside the metered figure: each handoff is +charged at most its pre-handoff context as input plus tokens of output, at +full rate. So `cost_usd` is the floor, `cost_usd_including_handoff_bound` is the +ceiling, and the truth is between them. Receipts keep `handoffs`, +`handoff_input_tokens_upper_bound`, and `handoff_cost_usd_upper_bound` separate +from the metered fields; nothing is silently merged. Getting the exact number +requires a `buzz-agent` change (return usage from `summarize` and fold it into +the turn counters). + +Handoffs do **not** fail reconciliation — the metered numbers are still genuine — +but the reconciliation note says the total is a floor whenever any occurred. + +A separate signal to watch: if the handoff cap is reached or summarisation +fails, the agent falls back to *truncating* its own history. The runtime counts +those as `handoff_truncations` and raises a warning, because dropped context is +a task-fidelity problem, not just a cost one. + +A trial whose agents ran but reported no tokens is marked +`accounting_reconciled: false` with a reason. Such a trial is an +instrumentation failure, not a free trial, and must not be averaged into a cost +figure as though it were a real zero. + +### Compaction threshold + +`buzz-agent` fires a handoff at whichever comes first: a percentage of the +context window (`BUZZ_AGENT_HANDOFF_PERCENT`, default 90) or an absolute +ceiling (`BUZZ_AGENT_HANDOFF_AT_TOKENS`, default 272,000; `0` disables it). The +ceiling is inert at a 200k window, where 90% (180k) binds first, and takes over +on large ones — at 1M, 90% would mean summarising ~900k tokens of history in a +single call, and every request would sit above OpenAI's 272k long-context +pricing boundary. + +Two manifest fields pass straight through, both optional: + +```yaml +generation: + max_output_tokens: 4096 + context_window_tokens: 1000000 + compact_at_percent: 30 # → BUZZ_AGENT_HANDOFF_PERCENT + compact_at_tokens: 272000 # → BUZZ_AGENT_HANDOFF_AT_TOKENS +``` + +Leave them unset to inherit the agent's defaults; the runtime then omits the +variables entirely, so a variable's presence in a trial bundle means the +condition really did pin it. `context_window_tokens` is always reported to the +agent as the real window — the harness never bends it to move the trigger. + +`compact_at_tokens` is validated against `context_window_tokens - max_output_tokens`, +since buzz-agent reserves room for the response and a target above that could +never fire. A ceiling merely larger than the percentage is fine — that is the +default at 200k. Because compaction cadence moves both cost and task +performance, both fields are part of the condition hash. + ## Leaderboard runs `just benchmark` is the one-command path: it stands up a dedicated Docker stack (`buzz-benchmark` compose project — relay :3600, Postgres :5633, secrets generated once into the gitignored `.benchmark/`), applies the benchmark -schema, and defaults to leaderboard-eligible settings (Terminal-Bench 2.1, -5 attempts per problem, the Sonnet+Haiku team). All selectors pass through: +schema, and defaults to leaderboard-eligible settings +(`terminal-bench/terminal-bench-2-1`, 5 attempts per problem, the Sonnet+Haiku +team). All selectors pass through: ```bash just benchmark # full TB 2.1, k=5 @@ -108,6 +345,14 @@ uv run --project benchmarks/harbor-buzz-orchestra/testbed \ --provisioner-config ``` +Harbor accepts two dataset forms, resolved from different places: `org/name[@ref]` +from the hub package registry, and `name[@version]` from `registry.json`. +Terminal-Bench 2.1 exists only as a package — `terminal-bench@2.0` in +`registry.json` is the older 2.0 cut — which is why the default is the +slash form. Whichever form is given is validated against the right source +before the stack comes up, so a typo or an upstream rename fails in a second +rather than after the first trial. + `--path` replaces `--dataset` for local task directories; `--include-task` / `--exclude-task` filter by glob; `--dry-run` prints the underlying `harbor run` command. After the job finishes the script derives a `metadata.yaml` from the diff --git a/benchmarks/harbor-buzz-orchestra/docs/PERSONAS.md b/benchmarks/harbor-buzz-orchestra/docs/PERSONAS.md new file mode 100644 index 0000000000..dba25663fe --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/docs/PERSONAS.md @@ -0,0 +1,453 @@ +# Persona library and condition matrix + +Working document. Tracks what each persona is for, which conditions use it, and +what is still unresolved. Update it in the same change that edits a persona — +a stale entry here is worse than no entry, because manifests pin persona +content by hash and a drifted description hides which condition actually ran. + +The personas live in `personas/bench/`. The previous generation +(`personas/*-tb.md`, `*-m1.md`) is kept for reference and for the m1 wiring +proof; nothing new should use it. + +## 1. What the agents are actually running inside + +Every persona is written against these mechanics. They are not incidental — a +persona that ignores them produces a stalled trial, not a worse score. Each was +read out of the source rather than assumed. + +**Agents wake only on @mention.** Each agent is a `buzz-acp` process with +`BUZZ_ACP_SUBSCRIBE=mentions`, all subscribed to one shared channel. An agent +runs a turn only when a message @mentions it by its exact display name (the +harness sets each profile name to the agent id). `ignore_self` defaults on, so +an agent's own message never wakes it. + +**A message that mentions nobody wakes nobody.** This is the dominant +multi-agent failure mode: every agent asleep, no pending mention, the trial +burning its 900s timeout to a failure. Every persona therefore ends with a hard +rule about who must be mentioned. + +**Only the orchestrator ends the trial.** The harness polls the channel as the +user identity and stops when the roster's single `kind: orchestrator` agent +publishes a message starting with `DONE:` +(`container_runtime.py:504` `_wait_for_done`). A worker posting `DONE:` +achieves nothing. Personas for non-orchestrator slots forbid the prefix +explicitly, so a helpful worker cannot end the trial early by accident. + +**Round budget resets per wake.** `BUZZ_AGENT_MAX_ROUNDS` caps LLM/tool rounds +per turn, and the counter is local to each `run()` call +(`crates/buzz-agent/src/agent.rs:82`). A solo agent is woken exactly once, so +its cap is its entire budget for the task; a team member gets the same cap per +assignment, across arbitrarily many assignments. At the old default of 32 that +handed every team roughly a 6× compute advantage over the solo baseline — +larger than any prompt-scheme effect the study is trying to measure — and it +failed silently: a turn that hits the cap ends with no message, so the solo +agent never posts `DONE:` and the trial burns its full 900s to a zero. + +`DEFAULT_MAX_AGENT_ROUNDS` is now **300**, chosen so no condition realistically +reaches it, which makes the trial timeout and the cost ceiling the binding +constraints — and those apply to every condition equally. Every persona states +the number and tells the agent to publish what it has by round 280, so a cap +hit degrades to a partial report rather than a silent freeze. + +**A 12KB `[Base]` prompt is prepended to every persona.** +`crates/buzz-acp/src/base_prompt.md`, composed as +`[Base]\n{base}\n\n[System]\n{persona}`. It is written for a production Buzz +workspace, and in a graded container a good third of it is actively wrong: +`buzz feed get` startup recovery, `RESEARCH/`/`PLANS/`/`AGENTS.md`, git +worktrees and PRs, `core` memory curation, "keep exploration inside your +working directory", "run the full test suite for the package you touched", +"stop and ask if git email is empty", "narrate as you go", and — the one that +kills trials — "publishing is optional and silence is usually correct." + +Each persona opens with a byte-identical block that names and overrides these +specifically rather than gesturing at them, because a general "ignore the above" +does not beat a specific MUST. Whether to suppress it outright is settled for +now: keep it, and measure it (A1n, §3). + +**Context isolation is a convention, not a mechanism.** Channel history is +injected automatically only for threaded replies or DMs, capped at 12 messages +(`crates/buzz-acp/src/pool.rs:2493`), so a flat top-level message gives its +recipient nothing but that message — *but* `format_context_hints` +(`crates/buzz-acp/src/queue.rs:1305`) appends `Hint: Use buzz messages get +--channel for recent messages if needed` to every top-level wake. An +agent that follows the hint sees everything. Personas therefore both mandate +flat posting and explicitly forbid `buzz messages get`/`thread`/`search`/`feed +get`, naming the hint so it can be ignored on purpose. Without that, how much +context each agent had would vary run to run. Threaded-versus-flat remains a +real future axis (§6). + +**One filesystem, shared.** All agents exec in the same task container. +Concurrent writers clobber each other, so every persona carries an ownership +rule. + +**The harness appends a `## Your team` block** to the persona at launch +(`container_runtime.py:696`): the agent's own id and pubkey, the channel UUID, +the user to report to, and a table of teammates. Personas must not restate any +of it, and refer to it by name. + +The table's Role column carries the **manifest's** `role` string, not the +roster kind. That distinction is load-bearing: personas address each other by +job ("the teammate whose Role column reads `critic`"), and in the three-agent +critic condition the implementer and the critic are both `kind: worker`, so a +table rendered from the kind would print two identical rows and leave the lead +unable to tell which teammate edits and which only verifies. Keep manifest +`role` values drawn from the vocabulary the personas use: `solo`, `lead`, +`implementer`, `critic`, `driver`, `navigator`, and — for the `gt/` generation +(§2.1) — `scout` and `worker`. + +**A misspelled @mention fails silently.** `extract_at_mentions_with_known` +(`crates/buzz-sdk/src/mentions.rs:107`) resolves names against known members; +an unmatched name yields zero pubkeys, the send still reports success, and +nobody wakes. Every persona says to copy names from the table character for +character and calls this out as the most fragile thing an agent writes. It is +also unfixable from the prompt side in the general case — a transient relay +hiccup during resolution produces the same silent freeze, which argues for a +harness-side quiet-channel watchdog (§5). + +## 2. The personas + +Every file shares two byte-identical sections — "This trial is not a Buzz +workspace" and "Messaging" — carrying the `[Base]` overrides, the round budget, +the mention rules, and the stdin form of `buzz messages send`. They are +duplicated rather than injected at compose time so that manifest `prompt.sha256` +still covers the whole prompt; the cost is ~600 tokens per persona and the +benefit is that a diff between two personas shows only the intended delta. If +you edit one of those blocks, edit all eleven and verify: + +```sh +for f in personas/bench/*.md; do + awk '/^## Messaging$/,/^ first message of a turn is your report\.$/' "$f" | + shasum -a 256 | cut -c1-12 +done | sort -u # must print exactly one line +``` + +| File | Slot | Terminal | Ends trial | Used by | +|------|------|----------|------------|---------| +| `solo.md` | solo | read+write | yes | A1–A3, A1n | +| `lead-delegate.md` | orchestrator | none | yes | B1, B5, C1, C2, C3, C4, C5 | +| `lead-research.md` | orchestrator | read-only | yes | B2 | +| `lead-implement.md` | orchestrator | read+write | yes | B3 | +| `peer-driver.md` | orchestrator | read+write | yes | B4 | +| `peer-navigator.md` | worker | read-only | no | B4 | +| `worker.md` | worker | read+write | no | B1, B2, B5, C1, C2 | +| `worker-chatty.md` | worker | read+write | no | C4 | +| `worker-deep.md` | worker | read+write | no | C5 | +| `worker-divergent.md` | worker | read+write | no | C3 | +| `critic.md` | worker | read-only | no | B3, C2 | + +**`solo.md`** — the control. Everything else is measured against it, so it says +plainly that there is one turn and nobody to delegate to, and tells the agent +not to spend rounds narrating. + +**`lead-delegate.md`** — pure coordination: no terminal, one assignment per +message, verification assigned to a different worker than the one being +verified. This is the "manager" archetype and the base for most team +conditions. + +**`lead-research.md`** — the lead investigates read-only and hands down +pre-scoped edits quoting the lines to change. Tests whether putting the +expensive model's attention on *understanding* beats spending it on +*coordination*. The read-only constraint is what makes it a different condition +rather than a slower solo run. + +**`lead-implement.md`** — inverts the usual shape: the strong model does the +work and the second agent exists only to check it. Cheapest way to ask whether +independent verification is worth an extra agent at all. + +**`peer-driver.md` / `peer-navigator.md`** — two strong models as equals, one +holding the keyboard. The navigator is read-only both because the filesystem is +shared and because it forces the value to come from judgement rather than from +a second pair of hands. Handoffs are pinned to three moments (before committing +to an approach, on contradiction, after the check passes) so the condition is +not just "two agents talking." + +**`worker.md` / `worker-chatty.md` / `worker-deep.md`** — the reporting +granularity axis, and the only axis on which they differ. The three files are +identical apart from the title line and one `## Reporting cadence` section; +`diff worker.md worker-deep.md` should show nothing else, and if it does the +A/B is measuring something other than granularity. The first draft failed this: +deep also had more error-recovery autonomy, a looser reading of scope, and a +self-verification rule the other two lacked, so a deep win would have been +unattributable. All three now share the same failure rule (report verbatim and +stop), the same verify-before-reporting rule, and the same scope language. + +`worker.md` completes an assignment and reports once. `worker-chatty.md` +reports wherever the result could change the plan and at least every three +commands, then waits. `worker-deep.md` reports once at the end and is the only +one that explicitly overrides `[Base]`'s "narrate as you go, never go dark", +which would otherwise pull it toward the chatty arm and compress the effect. + +The chatty arm is the timeout risk: a report-and-wait cycle costs a worker turn +plus a lead turn, and at 25-45s per medium-effort turn that is roughly 60-110s +per command against a 900s budget. Unbounded, it buys about a dozen commands +for the whole trial. The "at least every three commands, and not for a command +whose outcome was never in doubt" bound exists to keep the arm finishable; even +so, report its timeout rate separately rather than folding it into mean score. + +**`worker-divergent.md`** — for the two-angles condition. The whole persona is +about not converging: pursue the assigned approach even if the other looks +better, do not talk to the sibling, report a dead end as a real result. +Includes a stricter file-ownership rule than the other workers, since parallel +divergent work is exactly where two writers collide. + +**`critic.md`** — read-only, adversarial by default, verdict-first (`PASS` / +`FAIL`), and explicitly told not to write the fix. Read-only matters for a +reason beyond collisions: the state the critic assesses has to be the state the +grader sees. + +## 2.1 The `gt/` generation — goosetown-derived + +`personas/bench/gt/` is a second, self-contained persona family, added +2026-07-29 after every team cell in the study came in at or below its matching +solo baseline (B1 0.536 / C1 0.494 against A1's 0.545; C3 0.831 against A2's +0.843 at 64% more money). It is derived from the `goosetown-*` skill set — +`goosetown-orchestrator`, `goosetown-worker`, `goosetown-reviewer`, and the +eight `goosetown-researcher-*` skills, which all share one template. + +| File | Slot | Terminal | Ends trial | Used by | +|------|------|----------|------------|---------| +| `gt/gt-lead.md` | orchestrator | **read-only** | yes | G0, G1, G1s, G2, G2s | +| `gt/gt-scout.md` | worker | read-only | no | G0, G1, G1s, G2, G2s | +| `gt/gt-worker.md` | worker | read+write | no | G0, G1, G1s, G2, G2s | + +Three deltas against the `lead-delegate.md` family, each chosen because it maps +onto something the measured numbers say went wrong. None of them is a style +edit. + +**The lead cannot write.** `lead-delegate.md:40` says "Do the work directly when +that is the shorter path" and frames delegation as something that must earn its +round trip. That describes a solo agent with an expensive habit, and it is what +C3 measured: score at solo-opus level, input tokens at 1.81× solo opus. +`gt-lead.md` replaces the trade-off with a boundary — reads are the lead's, +every byte written is a teammate's — and carries **no trivial-write exception**, +because the exception is the loophole the old persona fell through. Enforcement +is prose only: there is no per-seat tool gating (`container_runtime.py:297` +gives every agent the same MCP toolset), so "read-only" in the table above is a +rule the persona states, not a sandbox. + +**A read-only recon seat exists.** This is new to the study. Every previous +teammate could write, so any two of them were a potential collision and the lead +had to serialise — which is why the team cells effectively ran in series and +paid round trips for it. `gt-scout.md` carries a hard read-only boundary, so +scouts cannot collide with each other or with the worker, and deliberate overlap +between them costs tokens and nothing else. That is what makes a genuinely +parallel recon phase safe, and it is the only reason the `count: 2` scout cells +(G2, G2s) are a different experiment rather than a slower one. + +**Reports carry an envelope.** `gt-worker.md` closes with +`STATUS: complete | partial | blocked` plus `DELIVERABLE` / `EVIDENCE` / +`NOTES`; `gt-scout.md` closes with either `BRIEF:` + +`FINDINGS` / `GOTCHAS` / `GAPS` or `VERDICT: pass | pass_with_notes | fail` plus +per-finding severity. Freeform prose made the lead's next decision expensive and +left the failure taxonomy hand-labelled; a fixed shape costs the delegate nothing +and is parseable. All three files also tell the agent to send **the decisive +output, not the transcript**, because every pasted log line is re-sent on every +subsequent round of the trial — the cost mechanism doc 02 §2 is built on. + +Two ideas kept from the existing family because they are better than the +goosetown originals. `gt-scout.md`'s verify half keeps `critic.md`'s +**re-derive, do not re-run** rule — `goosetown-reviewer` has no equivalent, and +without it two agents run one script, reproduce one mistake, and both certify +it. And all three files keep the byte-identical "This trial is not a Buzz +workspace" block, which is load-bearing against `[Base]`. + +One idea taken from goosetown that has no precedent here and is a **score** +lever rather than a safety one: *"A cancelled writer with 8 of 10 sections on +disk is useful"* (`goosetown-writer/SKILL.md:63`). The grader reads the +container when the clock stops, so `gt-lead.md` and `gt-worker.md` both say to +land the simplest thing that passes the task's own check and refine from there, +rather than assembling the finished answer and writing it once. + +### `gt-lead.md` diverges from the shared Messaging block, on purpose + +The §2 verification loop globs `personas/bench/*.md` and therefore does not +reach `gt/`, which is correct: **`gt-lead.md`'s Messaging section is +deliberately different.** The shared block says "Every turn you take ends with +exactly one published message". `gt-lead.md` says *at least* one, and adds a +bullet stating that dispatching two teammates means two messages in the same +turn. + +That is not a drift to be tidied up. A message wakes exactly one agent, so +"exactly one message per turn" caps a lead at one delegate in flight and +collapses G2/G2s into serial recon — the whole thing they exist to measure. The +invariant that actually matters is the anti-freeze one (never end a turn having +woken nobody), and the reworded bullet preserves it. `gt-scout.md` and +`gt-worker.md` keep "exactly one", since they only ever report to the lead. + +Also inherited rather than re-derived: `gt-lead.md` repeats +`lead-delegate.md:64`'s claim that teammates cannot read channel history. Per §1 +that is a convention, not a mechanism — `format_context_hints` tells every woken +agent how to fetch history, and no persona in either family forbids it. The +statement is kept verbatim so the G1-vs-C3 comparison differs only where +intended, but read it as an instruction to write self-contained assignments, +not as a guarantee about what a delegate can see. + +## 3. Condition matrix + +Naming: `tb---`. Model shorthand: `opus` = Opus 5, +`sol` = GPT-5.6 Sol, `luna` = GPT-5.6 Luna. All at medium thinking effort +(`container_runtime.THINKING_EFFORT`). + +### Tier 1 — solo baselines (persona held constant, model varies) + +| ID | Condition | Model | Persona | +|----|-----------|-------|---------| +| A1 | `tb-solo-luna` | luna | `solo.md` | +| A2 | `tb-solo-sol` | sol | `solo.md` | +| A3 | `tb-solo-opus` | opus | `solo.md` | +| A1n | `tb-solo-luna-nobase` | luna | `solo.md`, `[Base]` off | + +These are the reference points for every cost and score claim. Run them first +and run them at the same `k` as everything else. + +**A1n is the `[Base]` sensitivity check**, not a headline condition. Its +manifest is byte-identical to A1's except `include_platform_prompt: false`, +which sets `BUZZ_ACP_NO_BASE_PROMPT=1` and suppresses buzz-acp's ~12KB +production-workspace section. Any score, cost, or token delta between A1 and +A1n is attributable to that section alone, which is what turns "every persona +overrides `[Base]` in prose" from an assertion into a measurement. Run it once, +at the same `k`, and report the delta as a caveat. A1 stays the baseline: the +study is about Buzz, and `[Base]` is what a real Buzz agent receives. + +### Tier 2 — two agents + +| ID | Condition | Scheme | Orchestrator | Worker | +|----|-----------|--------|--------------|--------| +| B1 | `tb-2-delegate-opus-luna` | delegate | opus / `lead-delegate.md` | luna / `worker.md` | +| B2 | `tb-2-research-opus-luna` | research | opus / `lead-research.md` | luna / `worker.md` | +| B3 | `tb-2-verify-opus-luna` | implement+critic | opus / `lead-implement.md` | luna / `critic.md` | +| B4 | `tb-2-peer-opus-sol` | peer | opus / `peer-driver.md` | sol / `peer-navigator.md` | +| B5 | `tb-2-delegate-opus-sol` | delegate | opus / `lead-delegate.md` | sol / `worker.md` | + +B1 and B2 are the pair the question "should the smart model research or +delegate?" reduces to — same models, same slots, one persona different. That +is the cleanest comparison in the whole matrix; keep it that way. + +B4 versus B1 changes *both* scheme and worker model, so it cannot attribute +anything on its own. B5 is what makes it readable: B1→B5 isolates the worker +model at a fixed scheme, and B5→B4 isolates the scheme at fixed models. If +budget forces a cut, cut B4 before B5 — a peer result with no control is not +publishable. + +Optional if budget allows: `tb-2-peer-opus-luna` closes the 2×2. + +### Tier 3 — three agents (orchestrator opus, workers luna, unless noted) + +| ID | Condition | Scheme | Roster | +|----|-----------|--------|--------| +| C1 | `tb-3-delegate-opus-2luna` | delegate | `lead-delegate.md` + 2× `worker.md` | +| C2 | `tb-3-critic-opus-2luna` | manager+worker+critic | `lead-delegate.md` + `worker.md` + `critic.md` | +| C3 | `tb-3-divergent-opus-2luna` | two angles | `lead-delegate.md` + 2× `worker-divergent.md` | +| C4 | `tb-3-chatty-opus-2luna` | chatty workers | `lead-delegate.md` + 2× `worker-chatty.md` | +| C5 | `tb-3-deep-opus-2luna` | deep workers | `lead-delegate.md` + 2× `worker-deep.md` | + +C1, C4, C5 are one axis with three points — `worker.md` is the midpoint. Read +them together or not at all. + +B1→C1 is the team-size axis at a fixed scheme: one worker versus two, same +personas, same models. It is the only place the matrix answers "does a third +agent help?", so it is worth more than any of C2–C5 individually. + +### Roster ids + +Class ids become agent ids by appending `-`, and +`_classes_by_agent_id` splits on the last hyphen, so a class id must not end in +`-`. Use `lead`, `impl`, `critic`, `worker`. Two workers with the +same persona are one class with `count: 2` (`worker-1`, `worker-2`); two +workers with different personas are two classes with `count: 1`. + +## 4. Run order + +1. **Smoke, 1 task, k=1**: A1, then B1, then C1. Confirms the solo path, the + two-agent handoff, and the three-agent fan-out before any money is spent. + Read the session bundles by hand — the point is to see the actual message + traffic, not the score. +2. **Tier 1 complete** (A1–A3). Baselines before anything is compared to them. +3. **B1, B2, B5** — the scheme and model axes that carry the argument. +4. **C1, C4, C5** — the granularity axis. +5. **B3, B4, C2, C3** — the remaining shapes, budget permitting. + +## 5. Open issues that affect these personas + +**Round budget — settled.** `DEFAULT_MAX_AGENT_ROUNDS` is 300, high enough that +no condition should reach it, so the trial timeout and cost ceiling bind +instead and every condition faces the same limits. The residual asymmetry is +that a team still gets 300 *per assignment*; the mitigation is that nothing +should get near 300 in the first place. G11 stays open as a **disclosure** +rather than a fix: state the per-turn cap and its reset semantics in the post, +and report total LLM rounds per trial alongside cost so a reader can check that +no condition was actually round-bound. If any condition's round counts cluster +near 300, the comparison at that point is invalid and the cap must go higher. + +**Wall-clock, not rounds, is now the binding constraint** — and it is not +symmetric. Every handoff costs a full agent turn (25-45s at medium effort), so +a chatty three-agent condition can exhaust 900s on protocol overhead while +doing a fraction of the solo agent's work. Timeouts also cost the *full* 900s, +so a condition that times out often is penalised twice, in score and in mean +wall-clock. Report timeout rate as a first-class metric per condition, not +folded into the averages, and consider whether 900s is the right budget for the +three-agent tiers before reading anything into their wall-clock numbers. + +**A silent-freeze watchdog is worth considering.** Several failure modes end +the same way — a mention that resolved to nobody, a relay hiccup during +resolution, a turn that hit the round cap — with every agent asleep and the +harness polling to timeout. The personas reduce the prompt-side causes but +cannot touch the transport-side ones. A harness watchdog that re-posts the last +message's mention after N seconds of channel silence would convert a class of +silent 900s losses into recoverable ones. It changes the protocol, so it needs +a methodology footnote if adopted — but an uncorrected transient that reads as +"the team failed the task" is the worse distortion. + +**`[Base]` — settled, and measured.** Keep it for production parity in every +headline condition; A1n (§3) quantifies what it costs. Suppressing it entirely +via `include_platform_prompt: false` is now a manifest field and part of the +condition hash. The remaining option not taken is a trimmed benchmark base via +`BUZZ_ACP_BASE_PROMPT_FILE`: it would cut roughly 2,500 tokens from every round +of every turn and delete most of what the personas currently spend ~600 tokens +each rebutting. Worth revisiting if A1n shows `[Base]` is materially hurting, +since at that point "production parity" is preserving a known handicap. + +**Endpoint names for Opus 5 are not established.** `databricks-live.json` +currently maps only `databricks-gpt-5-6-sol` and `databricks-gpt-5-6-luna`. +Every Tier-2 and Tier-3 condition above assumes an Opus 5 endpoint on the same +gateway; A3, B1–B5 and C1–C5 are blocked until it exists and its list prices +are recorded. Substitute `sol` as the strong model if it does not. + +**Persona hashes must be updated with the text.** Manifests pin +`prompt.sha256` and `_verify_artifact` refuses to launch on a mismatch, which +is the intended behaviour — it makes silent prompt drift impossible. Recompute +with `shasum -a 256 personas/bench/` and update every manifest that +references the file in the same change. Current values: + +``` +critic.md b253b735a31ae6eb3c1b8ee318e5e83cef9b6821d49eabff8fe14370ba6b00c6 +lead-delegate.md 82df9751aeb5235f7122adc06bafdde12084181b0be84b91721e0d1b5e4a14f7 +lead-implement.md 8a1baf78170334ad24510255a0a48af551afcb61646c023e44e95ea8efe37d59 +lead-research.md fc67d8ae2ca96cbd1c0133b5c16bd4ac5700cec9927535168f7ce784dbadbb6a +peer-driver.md 4461b3b7e735b9cdf7c30646a6a54d6b429879c37ea3eb4c8b29b2daceaa7a1d +peer-navigator.md ec0758b9982cfc499af622be244919305a3b480dc06aa13b3389634efed8ea32 +solo.md 6997ae38384ea4f84bbdc1422a668be089fbfd2709d412392dbbb4911c565596 +worker-chatty.md 6869901289dab1843e6bd47057c87a9d3178febb23c62521d626b39be402eebf +worker-deep.md e29e08042707741a603a51fc9e1d1e9c45f0d1fbad6c90f900f6dd8413656589 +worker-divergent.md 328ef5af5c200cc8a610b13c73c39426c92a056ff9a01018e0ba35c1979666f8 +worker.md 7b3332e3d3da384a7c242c1aa34e90a063e7152ddf6de69f8a4e7e4a0b710726 +``` + +## 6. Candidate future axes + +Not in the matrix above; each would need its own conditions. + +- **Flat versus threaded messaging.** Every persona here pins flat posting. A + threaded variant injects up to 12 prior channel messages into each recipient, + which should raise cost and may raise coordination quality. Cleanly testable + by changing one sentence in each persona. +- **Per-slot thinking effort.** Currently uniform medium + (`container_runtime.THINKING_EFFORT`). A high-effort lead over low-effort + workers is the obvious cost-efficiency play. Blocked on G2 — effort is a + module constant, not a manifest field. +- **Base prompt: trimmed.** A1n covers on/off. The untested third option is a + ~2KB benchmark-specific base via `BUZZ_ACP_BASE_PROMPT_FILE`, hashed into the + manifest — keeping the CLI reference and mention mechanics, dropping the + workspace, memory, and agent-creation sections. +- **Verification budget.** Every lead here verifies once. Verify-twice, or + verify-only-on-suspicion, is a distinct scheme. diff --git a/benchmarks/harbor-buzz-orchestra/manifests/lhtb-or-kimi-lead-2deepseek-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/lhtb-or-kimi-lead-2deepseek-high.yaml new file mode 100644 index 0000000000..c9ff7e4678 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/lhtb-or-kimi-lead-2deepseek-high.yaml @@ -0,0 +1,104 @@ +# OR3h -- LHTB team on OpenRouter: kimi-k3 lead, deepseek-v4-flash scout, +# deepseek-v4-flash worker. All three seats at reasoning effort `high`. +# +# Structurally this is tb-gt-sol-luna-terra-high with open-weight models +# substituted seat for seat: an expensive lead directing two cheap seats, one +# read-only scout and one that edits and builds. That cell is the reason to run +# this one -- it scored 0.695 against solo Sol's 0.591 on LHTB-46, and the +# open question is whether the long-horizon team win survives a 22x cheaper +# roster or was really about the specific models. +# +# NOTE THE ASYMMETRY WITH ITS OPENAI ANALOGUE. There, scout and worker were +# DIFFERENT models (luna scout, terra worker), which is what let that cell +# separate "cheap in the read-only seat" from "cheap in the editing seat". +# Here both cheap seats are the same model, so this cell cannot make that +# distinction -- it is the two-model analogue of tb-gt-sol-2terra-high, not of +# tb-gt-sol-luna-terra-high. Reading a seat-placement result out of it would be +# reading something it does not contain. +# +# ROSTER IDS ARE PROTOCOL, NOT LABELS. `lead`, `scout` and `worker` are read +# verbatim out of the "Your team" table by the gt personas. Renaming any of them +# breaks addressing silently: the @mention resolves to nobody, the send still +# reports success, and the trial stalls to its timeout. +# +# BOTH UPSTREAMS ARE PINNED (openrouter-live.json): moonshotai/mxfp4 for the +# lead, gmicloud/fp8 for both cheap seats. On a team cell the pin matters more +# than on a solo one -- unpinned, each seat independently lands on a different +# upstream per request, so "the composition" would not even be stable within a +# single trial. It is also what makes prompt caching work: measured 2026-08-01, +# pinned routes served a repeated prefix from cache on every call while the +# unpinned route managed one in three. A team re-sends more history than a solo +# agent does, so this cell is the one that would have been hurt worst. +# +# THIS RUNS UNDER THE PATCHED HARBOR. LHTB needs continue_until_timeout honored +# (patches/apply_continue_until_timeout.py); stock harbor ignores the flag and +# agents self-report DONE minutes into hour-long budgets. lhtb46.sh refuses to +# start on an unpatched harbor -- do not bypass that check. +schema_version: "1" +condition: lhtb-or-kimi-lead-2deepseek-high +roster: + - id: lead + kind: orchestrator + role: lead + count: 1 + # The EXPENSIVE model in the directing seat -- 22x the input rate of the + # seats it directs. Whether that is worth it is the cell's question. + endpoint: moonshotai/kimi-k3 + model_revision: moonshotai/kimi-k3-20260715 + prompt: + # Byte-identical to the file every other gt cell pins. + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + generation: + thinking_effort: high + + - id: scout + kind: worker + role: scout + count: 1 + endpoint: deepseek/deepseek-v4-flash-0731 + model_revision: deepseek/deepseek-v4-flash-20260731 + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + generation: + thinking_effort: high + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: deepseek/deepseek-v4-flash-0731 + model_revision: deepseek/deepseek-v4-flash-20260731 + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + generation: + thinking_effort: high + +prices: + # The rates of the PINNED endpoints, not the model ids' headline rates. Both + # rows are load-bearing: the whole point of the cell is the split between an + # expensive lead and cheap seats, so a wrong row moves the conclusion and not + # just the total. + moonshotai/kimi-k3: + input_per_million_usd: 3.00 + cached_input_per_million_usd: 0.30 + output_per_million_usd: 15.00 + cache_read_rate: 0.0 + deepseek/deepseek-v4-flash-0731: + input_per_million_usd: 0.133 + cached_input_per_million_usd: 0.0266 + output_per_million_usd: 0.266 + cache_read_rate: 0.0 +trial_budget: + # Identical to the LHTB cells it is meant to be read against. Does not bind + # on LHTB anyway -- Harbor enforces each task's own `[agent] timeout_sec` + # scaled by the run's --timeout-multiplier (3.0). + timeout_seconds: 36000 + +environment: + # Identical in every condition. override_cpus 4 with -n 8 is exactly 1:1 on a + # 32-vCPU m7a.8xlarge. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-cobol-sonnet-haiku.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-cobol-sonnet-haiku.yaml index 19289f1ca3..83eb7096d5 100644 --- a/benchmarks/harbor-buzz-orchestra/manifests/tb-cobol-sonnet-haiku.yaml +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-cobol-sonnet-haiku.yaml @@ -14,9 +14,7 @@ roster: prompt: path: personas/orchestrator-tb.md sha256: 206829331cdd85fb277266901b9489e190ba23097600a8bb69142d044b9a42e6 - generation: - max_output_tokens: 4096 - context_window_tokens: 200000 + - id: worker kind: worker role: implementer @@ -26,17 +24,30 @@ roster: prompt: path: personas/worker-tb.md sha256: 664a994ae3ba93f02ebbb8f61f781f6a59ea3327b21e26e7ffc2d06827ed3e45 - generation: - max_output_tokens: 4096 - context_window_tokens: 200000 + +# Cache-read rates are 0.0 on every Anthropic-route endpoint on purpose: +# buzz-agent never sends `cache_control`, and Anthropic prompt caching is +# opt-in, so no cache reads occur and no discount should be modelled. Revisit +# this the moment buzz-agent starts requesting caching, and set a measured rate +# for OpenAI-route endpoints (which cache automatically) rather than a guess. prices: claude-sonnet-4-6: input_per_million_usd: 3 cached_input_per_million_usd: 0.3 output_per_million_usd: 15 + cache_read_rate: 0.0 claude-haiku-4-5: input_per_million_usd: 1 cached_input_per_million_usd: 0.1 output_per_million_usd: 5 + cache_read_rate: 0.0 trial_budget: - timeout_seconds: 900 + # 36000 = the dataset's longest task (12000s) at the 3x timeout multiplier the + # study runs with (DEFAULT_TIMEOUT_MULTIPLIER in scripts/benchmark.py). The two + # are one setting in two files: benchmark.py refuses to start if this is the + # smaller of the pair, because the harness would then cut a task short of the + # deadline Harbor granted it and record a timeout the agent never hit. A flat + # 900s did exactly that before the multiplier existed — 39 of the 89 tasks + # allow more than 900s, and one live trial died at 905s against a task budget + # of 1800s. + timeout_seconds: 36000 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-3luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-3luna.yaml new file mode 100644 index 0000000000..e1d0a2f30a --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-3luna.yaml @@ -0,0 +1,109 @@ +# G0 -- the goosetown persona set at the cheap floor: a luna lead coordinating +# one read-only luna scout and one luna worker, all three direct from OpenAI. +# +# This is the CONTROL for the goosetown wave, and without it a G1s win is +# uninterpretable. G1s/G2s put an expensive model in the lead seat at the same +# time as they change the persona bytes, so a score move there could be either +# thing. G0 holds the personas fixed and puts the cheap model in the lead seat, +# so: +# +# G0 -> G1s varies the lead's model only (luna -> sol) +# G0 vs C1 varies the persona scheme (and the route -- see below) +# +# It is also the only cell in this wave that serves the study's actual headline. +# G1s/G2s/G1/G2 are all hybrid cells: they ask whether a better-orchestrated +# expensive lead earns its keep, which is doc 02 §1's *rival*, not its thesis. +# "Three cheap models talking to each other match one expensive model" needs a +# cheap-only team, and after B1 (0.536) and C1 (0.494) both came in *below* solo +# luna (0.545) this is the study's remaining shot at it. +# +# What is being tested is not prose. The old lead persona +# (personas/bench/lead-delegate.md:40) told the lead "do the work directly when +# that is the shorter path", which describes a solo agent that occasionally pays +# a round trip -- and that is exactly what C1/C3 measured (score at or below the +# matching solo, input tokens ~1.9x). personas/bench/gt/gt-lead.md inverts it +# into a boundary: reads are the lead's, every byte written is a teammate's, and +# there is no trivial-write exception. The scout seat is new to the study +# entirely -- a read-only role cannot collide with anyone, which is what makes +# parallel recon safe and is why gt-lead is allowed to send several assignments +# in one turn. +# +# ROUTE CAVEAT. This runs direct against api.openai.com, while A1/B1/C1 ran +# through the Databricks AI Gateway. A G0-vs-C1 comparison therefore varies the +# persona scheme *and* the route. Within the goosetown wave, G0/G1s/G2s share +# the route exactly, so those three comparisons are clean. Report the route with +# the number. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-gt-3luna +roster: + - id: lead + kind: orchestrator + # Read verbatim out of the "Your team" table by both delegate personas + # ("Your lead is the teammate whose Role column reads `lead`"). Renaming it + # means every report @mentions nobody, the sends all report success, and the + # trial stalls to its timeout with no error anywhere. + role: lead + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + # `scout` is read by gt-lead.md, which routes recon and verification here + # and edits away from here. A seat labelled anything else gets sent edits it + # is forbidden to make, and answers "this needs a worker" every time. + role: scout + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Identical to every other luna cell, deliberately: holding the rates equal is + # what makes the cost *ratio* against A1 and C1 sound. + gpt-5.6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only; superseded by the provider's measured split (doc 02 §2.1). + # Caching matters more for a team than for a solo agent, because a team's + # bill is driven by re-sent context and that is exactly what a prefix cache + # serves. + cache_read_rate: 0.0 +trial_budget: + # 12000s longest task x the study's 3x multiplier, identical to every cell in + # the comparison set. benchmark.py's check_budget_clears_clock refuses to + # start below that product. + timeout_seconds: 36000 + +environment: + # Identical in every condition, solos included -- see tb-solo-luna.yaml for + # the full reasoning. Note this roster is 3 agents in one container, the same + # headcount C1 ran at. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-2luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-2luna.yaml new file mode 100644 index 0000000000..ea43903589 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-2luna.yaml @@ -0,0 +1,111 @@ +# G1 -- goosetown personas with a claude-opus-5 lead over one read-only luna +# scout and one luna worker, through the Databricks AI Gateway. +# +# THIS IS THE CLEAN PERSONA A/B, and it is the only cell in the wave that is +# one. Against tb-team-opus-luna (C3, condition tb-team-opus-2luna) it holds +# constant every single thing the study can hold: +# +# same lead model databricks-claude-opus-5 +# same delegate model databricks-gpt-5-6-luna, 2 seats +# same headcount 3 agents in one container +# same route opus -> anthropic-messages, luna -> /responses +# same price sheet, same clock, same container, same effort (medium, unpinned) +# +# The only difference is the persona bytes. So the G1-vs-C3 delta is +# attributable to the prompt scheme and to nothing else -- which is what makes +# this cell worth running on the two-route gateway rather than moving it to the +# clean OpenAI path like G1s. Moving it would have bought route cleanliness and +# lost the controlled comparison, and G1s already provides the clean route. +# +# What C3 measured, and what this is trying to fix: +# +# C3 score 0.831 cost $68.00 input 51.6M $/solved 0.92 +# A3 score 0.795 cost $48.17 input 28.5M $/solved 0.69 (solo opus) +# A2 score 0.843 cost $41.52 input 25.5M $/solved 0.55 (solo sol) +# +# C3 bought +0.036 over solo opus for 1.81x its input tokens, and still lost to +# solo sol on both axes. The diagnosis is in the old persona's own words: +# personas/bench/lead-delegate.md:40 told the lead "do the work directly when +# that is the shorter path" and framed delegation as something that has to earn +# its round trip. That describes a solo agent with an expensive habit, and C3's +# numbers are what that produces -- score at solo-opus level, tokens at team +# level. +# +# personas/bench/gt/gt-lead.md replaces the trade-off with a boundary: reads are +# the lead's, every byte written is a teammate's, and there is deliberately no +# trivial-write exception, because the exception is the loophole the old persona +# fell through. Whether a lead that cannot type is better or worse than one that +# can is the question; a null here with C3's cost profile intact would be a +# genuine finding about hierarchy, not a failed rewrite. +# +# Endpoint names are exact Databricks serving-endpoint names and resolve via +# testbed/endpoints/databricks-live.json. buzz-agent's `databricks_v2` provider +# picks the route per model from the endpoint name; see tb-solo-opus.yaml for the +# detail. +schema_version: "1" +condition: tb-gt-opus-2luna +roster: + - id: lead + kind: orchestrator + role: lead + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: claude-opus-5 + prompt: + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + role: scout + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Byte-identical to tb-team-opus-luna.yaml. The G1-vs-C3 comparison is a cost + # comparison as much as a score one, so the rate sheet has to be the same + # sheet or the delta is partly a pricing edit. + # + # Opus is 5x luna on input and cache-read but ~4.17x on output; see + # tb-solo-opus.yaml for the cache-write (cache_creation) surcharge caveat, + # which lands hardest on the lead's high-context seat. See tb-solo-luna.yaml + # for why cache_read_rate is 0.0 (DB10). + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 25.0 + cache_read_rate: 0.0 + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-2terra.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-2terra.yaml new file mode 100644 index 0000000000..14d530b331 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-2terra.yaml @@ -0,0 +1,106 @@ +# G1ot -- goosetown personas with a Claude Opus 5 lead over one read-only terra +# scout and one terra worker. The terra-seat analogue of G1. +# +# READ THE ROUTE CAVEAT BEFORE QUOTING ANY NUMBER FROM THIS CELL. +# +# This cell is CROSS-PROVIDER by necessity, not by choice. Opus 5 is only served +# to us through Databricks (`databricks-claude-opus-5`); it is absent from +# openai-live.json, and anthropic-live.json carries only Sonnet 4.6 and Haiku +# 4.5. Terra is only served direct from OpenAI (`gpt-5.6-terra`); it is absent +# from databricks-live.json. So there is no single endpoint config that can serve +# both seats, and this manifest resolves against a MERGED config -- +# testbed/endpoints/mixed-opus-terra.json -- which needs BOTH DATABRICKS_TOKEN +# and OPENAI_API_KEY exported. Neither sweep.sh nor sweep-openai.sh does that; +# use sweep-mixed.sh. +# +# What that costs interpretively: +# +# * G1 -> G1ot moves the SEAT MODEL and the SEAT PROVIDER together (luna on +# Databricks -> terra on OpenAI). It is therefore NOT the clean seat-model +# read that G1s -> G1st is. If you want the one-variable answer to "do +# stronger seats rescue the delegation team", that is G1st, not this cell. +# * G1st -> G1ot moves the LEAD MODEL and the LEAD PROVIDER together +# (sol on OpenAI -> opus 5 on Databricks), so it is not a clean lead-model +# read either -- unlike G0 -> G1s, which was. +# +# Note this is a difference of degree rather than kind: every opus-led cell in the +# study (C3, G1, G2) is ALREADY two-route internally, because opus does not speak +# /responses and goes to /ai-gateway/anthropic/v1/messages while its luna seats +# take the Databricks /responses path. G1ot widens that split from two paths +# inside one vendor to two vendors. What it adds beyond G1 is a second set of +# rate limits and a second failure mode, not a new confound in kind. +# +# WHAT IT IS STILL GOOD FOR. The cost story survives the route caveat intact, +# because cost is not a paired-difference quantity: if an Opus-5-led team with +# the strongest available cheap seats still fails to beat solo sol (0.843) or +# solo opus (0.851), that is worth knowing whatever the routes were, and it is +# the last untested corner of the delegation hypothesis. +# +# Endpoint names resolve via testbed/endpoints/mixed-opus-terra.json. +schema_version: "1" +condition: tb-gt-opus-2terra +roster: + - id: lead + kind: orchestrator + # See tb-gt-3luna.yaml: `lead`, `scout` and `worker` are read verbatim out of + # the "Your team" table by the personas. Renaming any of them breaks + # addressing silently -- the @mention resolves to nobody, the send still + # reports success, and the trial stalls to its timeout. + role: lead + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: claude-opus-5 + prompt: + # Byte-identical to the file G0, G1s, G1, G1st, G2s and G2 pin. + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + role: scout + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: terra 2.5/0.25/15.0 -> 2.00/0.20/12.00 (-20%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Opus 5 output is 25.0, NOT sol's 30.0 -- do not copy the sol row by reflex. + # See tb-solo-luna.yaml for why cache_read_rate is 0.0 (DB10). + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 25.0 + cache_read_rate: 0.0 + gpt-5.6-terra: + input_per_million_usd: 2.0 + cached_input_per_million_usd: 0.2 + output_per_million_usd: 12.0 + cache_read_rate: 0.0 +trial_budget: + # Identical to the solo baselines -- see tb-gt-sol-2luna.yaml. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-3luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-3luna.yaml new file mode 100644 index 0000000000..5f2b546fe9 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-opus-3luna.yaml @@ -0,0 +1,99 @@ +# G2 -- G1 with a second read-only scout. A claude-opus-5 lead over two luna +# scouts and one luna worker, through the Databricks AI Gateway. +# +# The recon-width question asked on the opus route, so that the wave's headline +# comparison exists in both places. Against tb-gt-opus-2luna it varies exactly +# one thing -- one scout becomes two -- with the lead model, persona bytes, +# route, price sheet, clock and container all held. +# +# Why run it here as well as at G2s. If widening recon helps at all, the size of +# the help should depend on how good the synthesiser is: a lead that can hold +# two conflicting briefs and resolve them gets more out of a second scout than +# one that cannot. So the interesting number is not the G1->G2 delta or the +# G1s->G2s delta on its own, it is whether they are the same size. That is a +# question about where orchestration quality actually lives, and it needs both +# routes measured or it cannot be asked. +# +# The four cells therefore form a 2x2 -- lead model x recon width: +# +# 1 scout 2 scouts +# sol lead G1s (this wave) G2s +# opus lead G1 G2 +# +# with G0 hanging off the sol column as the cheap-lead control. Every cell in a +# column shares its route, so the two width deltas are each clean; the row +# comparison (sol vs opus lead) carries the route caveat that C3 has carried +# since it was written, because opus cannot speak /responses. +# +# FOUR agents in one container. See `environment` -- 4 vCPU and 8192 MB is the +# dataset maximum rather than a comfortable margin, so if this cell diverges +# from its 3-agent sibling in a way the score cannot explain, check `free -m` +# and container exit-137 before believing the coordination story. +# +# Endpoint names resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-gt-opus-3luna +roster: + - id: lead + kind: orchestrator + role: lead + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: claude-opus-5 + prompt: + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + role: scout + # The only line that differs from tb-gt-opus-2luna. Seats are named + # `{id}-{index}`, 1-indexed, so this yields `scout-1` and `scout-2` as + # separate rows in the "Your team" table -- which is what lets the lead give + # each a different angle instead of a smaller slice of the same one. + count: 2 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + # Still one, for the same reason as G2s: widening recon and implementation + # together would make the delta unattributable. + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Byte-identical to tb-gt-opus-2luna and tb-team-opus-luna. + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 25.0 + cache_read_rate: 0.0 + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2luna.yaml new file mode 100644 index 0000000000..2229b3afe2 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2luna.yaml @@ -0,0 +1,111 @@ +# G1s -- goosetown personas with a gpt-5.6-sol lead over one read-only luna +# scout and one luna worker. Single provider, single route, direct from OpenAI. +# +# This is the strongest version of the hybrid the study argues against, and it +# is a cleaner cell than the opus-led G1 for three separate reasons: +# +# * ONE ROUTE. sol and luna are both gpt-5.6 family and both take the OpenAI +# `/responses` path. Every opus-led cell in this study (C3, G1, G2) is +# inherently two-route -- opus goes to /ai-gateway/anthropic/v1/messages +# because it does not speak /responses -- so the lead's model and the lead's +# serving path move together and cannot be separated. Here they do not move. +# * IT IS THE BETTER STEELMAN. sol solo scored 0.843 against opus solo's +# 0.795. The expensive model that a hybrid should be built around is the one +# that wins alone, and on this dataset that is sol. +# * EXACT ARITHMETIC. sol is 5.00/0.50/30.00 against luna's 1.00/0.10/6.00 -- +# exactly 5x on input, cache-read *and* output. Every price ratio this cell +# reports is a clean 5x with no output-axis fudge, which is doc 02 §2's +# parity argument stated in its sharpest form. +# +# The comparison set: +# +# G0 -> G1s lead model only (luna -> sol); persona bytes, route, headcount, +# container and clock all held +# G1s -> G2s recon width only (1 scout -> 2) +# G1s vs A2 the whole question: does a well-orchestrated cheap team wrapped +# around one sol lead beat sol working alone (0.843, $41.52)? +# +# The mechanism under test is cost as much as score. C3 put an expensive lead in +# the most token-heavy seat and spent 51.6M input tokens to solo opus's 28.5M -- +# 1.81x -- for +0.036. gt-lead.md is written to attack exactly that: the lead +# reads structured briefs and verdicts instead of raw terminal output, and +# gt-scout.md/gt-worker.md are both told to send the decisive output rather than +# the transcript, because every pasted log line is re-sent on every subsequent +# round. Whether that actually shrinks the expensive seat is the measurement, +# not an assumption -- the lead also reads *more* reports than C3's did, and +# those two effects pull opposite ways. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-gt-sol-2luna +roster: + - id: lead + kind: orchestrator + # See tb-gt-3luna.yaml: `lead`, `scout` and `worker` are all read verbatim + # out of the "Your team" table by the personas. Renaming any of them breaks + # addressing silently -- the @mention resolves to nobody, the send still + # reports success, and the trial stalls to its timeout. + role: lead + count: 1 + endpoint: gpt-5.6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to the file G0, G2s, G1 and G2 pin, so the lead's + # instructions are not a variable anywhere in this wave -- only the lead's + # model and the number of scouts are. + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + role: scout + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Both rows are load-bearing: this is a mixed-model cell, so the headline + # $/solved-task depends on the split between the seats as well as on the + # totals. The figures are the same ones every other sol and luna cell carries. + gpt-5.6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + cache_read_rate: 0.0 + gpt-5.6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + # Identical to the solo baselines. A team genuinely needs longer than a solo + # agent -- every handoff is a round trip -- but giving it a larger budget would + # confound the comparison it exists to make, and it does not bind anyway: + # Harbor enforces each task's own `[agent] timeout_sec` scaled by 3x. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2terra-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2terra-high.yaml new file mode 100644 index 0000000000..a8d5f1510b --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2terra-high.yaml @@ -0,0 +1,119 @@ +# G1sth -- G1st with every seat pinned to `thinking_effort: high`. Goosetown +# personas, a gpt-5.6-sol lead over one read-only terra scout and one terra +# worker. Single provider, single route, direct from OpenAI. +# +# BUILT FOR LHTB, not Terminal-Bench. Nothing in the manifest names a dataset, +# but the reason this cell exists is docs/09-lhtb-benchmark.md: the long-horizon +# set is where extra deliberation per turn is most plausibly worth its price, +# and it is the one benchmark in the study where the team cells have not yet +# been given the effort dial. Run it with the patched +# `continue_until_timeout` (see docs/09) and `--timeout-multiplier 3.0`. +# +# WHAT THIS ISOLATES. Against tb-solo-sol-high.yaml -- same route, same lead +# model, same effort, same price sheet, same clock, same container -- the only +# thing that moves is headcount and the personas that coordinate it. That is the +# team-vs-solo question asked at the top of the effort ramp instead of at the +# provider default. +# +# EFFORT IS PINNED HERE, DELIBERATELY, AND THAT IS THE DIFFERENCE FROM +# tb-gt-sol-2terra.yaml. That manifest carries a long comment explaining why +# effort must stay unpinned: it belongs to the G wave, where no cell pins effort, +# and pinning it there would confound "stronger seats" with "more thinking on the +# seats". None of that applies to this cell. This one is not in the G wave and +# makes no claim about seat strength; it is a two-cell high-effort LHTB pair +# whose control (tb-solo-sol-high) pins the same value. Do NOT compare this cell +# against tb-gt-sol-2terra or any other unpinned cell and call the delta a team +# effect -- it would be an effort delta plus a headcount delta at once. +# +# All three seats are pinned, not just the lead. A lead thinking harder than the +# seats it delegates to is a third condition, not this one; if that question +# comes up it needs its own manifest. +# +# Effort support verified before this cell was scheduled: `gpt-5.6-sol` and +# `gpt-5.6-terra` both match the `gpt-5.6` family token in config.rs, whose +# supported set includes high. A2x already ran sol at `xhigh` and A5h/B3h ran +# terra at `high` -- neither value is rejected nor silently clamped on this +# route. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-gt-sol-2terra-high +roster: + - id: lead + kind: orchestrator + # See tb-gt-3luna.yaml: `lead`, `scout` and `worker` are all read verbatim + # out of the "Your team" table by the personas. Renaming any of them breaks + # addressing silently -- the @mention resolves to nobody, the send still + # reports success, and the trial stalls to its timeout. + role: lead + count: 1 + endpoint: gpt-5.6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to the file G0, G1s, G1, G1st, G2s and G2 pin, so the + # lead's instructions are not a variable anywhere in this wave. + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + generation: + thinking_effort: high + + - id: scout + kind: worker + role: scout + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + generation: + thinking_effort: high + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + generation: + thinking_effort: high + +prices: + # Repriced 2026-07-30: terra 2.5/0.25/15.0 -> 2.00/0.20/12.00 (-20%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Both rows are load-bearing: this is a mixed-model cell, so the headline + # $/solved-task depends on the split between the seats as well as on the + # totals. Same figures every other sol and terra cell in the study carries -- + # effort changes how many tokens get spent, not what a token costs. + gpt-5.6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + cache_read_rate: 0.0 + gpt-5.6-terra: + input_per_million_usd: 2.0 + cached_input_per_million_usd: 0.2 + output_per_million_usd: 12.0 + # Fallback only; the direct OpenAI Responses route reports the real split + # and the harness prefers it. See tb-solo-luna-high.yaml. + cache_read_rate: 0.0 +trial_budget: + # Identical to the solo baselines. A team genuinely needs longer than a solo + # agent -- every handoff is a round trip -- but giving it a larger budget would + # confound the comparison it exists to make, and it does not bind anyway: + # Harbor enforces each task's own `[agent] timeout_sec` scaled by 3x. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2terra.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2terra.yaml new file mode 100644 index 0000000000..8b17025376 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-2terra.yaml @@ -0,0 +1,124 @@ +# G1st -- goosetown personas with a gpt-5.6-sol lead over one read-only terra +# scout and one terra worker. Single provider, single route, direct from OpenAI. +# +# WHAT THIS ISOLATES. G1st is G1s with the two subordinate seats upgraded from +# luna to terra and *nothing else touched*: same lead model, same three persona +# files byte-for-byte, same headcount, same route, same container, same clock. +# So G1s -> G1st is a clean one-variable read on the question G1s leaves open -- +# was the goosetown team's deficit caused by the *cheap seats being too weak* to +# be worth delegating to? +# +# Why that is the live hypothesis. G0 -> G1s showed the LEAD model matters +# enormously (+0.216 for sol over luna, p = 0.000, 4:23 discordant). The obvious +# next question is whether the seats matter the same way. Every anti-team result +# so far used luna seats, and luna solo scores 0.545 -- barely half the tasks. A +# lead that cannot trust its subordinates' work has to redo it, which would show +# up as exactly the pattern observed: more tokens, no more solves. Terra solo +# scores 0.807 at high effort, so terra seats are a genuinely stronger hand. +# +# The comparison set: +# +# G1s -> G1st seat model only (luna -> terra); lead, personas, route, +# headcount, container and clock all held +# G1st vs A2 the whole question, with the strongest cheap seats available: +# does an orchestrated team beat sol working alone (0.843)? +# G1st vs A2r the same, fully route-matched once A2r lands +# G1st -> G1ot lead model only (sol -> opus 5), the terra-seat analogue of +# G1s -> G1. NOTE that pair is NOT single-route: opus 5 is +# Databricks-only and terra is OpenAI-only, so G1ot moves the +# lead's provider as well as its model. See tb-gt-opus-2terra.yaml. +# +# EFFORT IS DELIBERATELY UNPINNED. The terra cells that already exist (A5h, B3h) +# both pin `thinking_effort: high`, and it is tempting to copy that here since +# terra's published numbers come from the high-effort runs. Do not. No manifest +# in the G wave pins effort, so pinning it here would confound "stronger seats" +# with "more thinking on the seats" and destroy the one-variable property that is +# this cell's entire reason to exist. Unset means the harness omits the +# `reasoning` field and the provider default applies -- identical treatment to +# G0, G1s and G1. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-gt-sol-2terra +roster: + - id: lead + kind: orchestrator + # See tb-gt-3luna.yaml: `lead`, `scout` and `worker` are all read verbatim + # out of the "Your team" table by the personas. Renaming any of them breaks + # addressing silently -- the @mention resolves to nobody, the send still + # reports success, and the trial stalls to its timeout. + role: lead + count: 1 + endpoint: gpt-5.6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to the file G0, G1s, G1, G2s and G2 pin, so the lead's + # instructions are not a variable anywhere in this wave. + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + role: scout + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: terra 2.5/0.25/15.0 -> 2.00/0.20/12.00 (-20%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Both rows are load-bearing: this is a mixed-model cell, so the headline + # $/solved-task depends on the split between the seats as well as on the + # totals. Same figures every other sol and terra cell in the study carries. + # + # Note the seat/lead ratio: terra is 2.0/0.20/12.0 against sol's + # 5.00/0.50/30.00, exactly 40% on all three axes. Against G1s's luna seats + # (0.20/0.02/1.20) a terra seat is 10x the unit price, so this cell buys + # stronger seats at a real premium, and the token-tax comparison against G1s + # has to be read per-dollar as well as per-token. + # + # The 2026-07-30 drop widened that gap: it was 2.5x luna before (terra -20%, + # luna -80%), so any G1st-vs-G1s per-dollar reading taken before that date + # used a much flatter price ladder than this manifest now encodes. + gpt-5.6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + cache_read_rate: 0.0 + gpt-5.6-terra: + input_per_million_usd: 2.0 + cached_input_per_million_usd: 0.2 + # Fallback only; the direct OpenAI Responses route reports the real split + # and the harness prefers it. See tb-solo-luna-high.yaml. + output_per_million_usd: 12.0 + cache_read_rate: 0.0 +trial_budget: + # Identical to the solo baselines. A team genuinely needs longer than a solo + # agent -- every handoff is a round trip -- but giving it a larger budget would + # confound the comparison it exists to make, and it does not bind anyway: + # Harbor enforces each task's own `[agent] timeout_sec` scaled by 3x. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-3luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-3luna.yaml new file mode 100644 index 0000000000..2eb72d4cae --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-gt-sol-3luna.yaml @@ -0,0 +1,105 @@ +# G2s -- G1s with a second read-only scout. A gpt-5.6-sol lead over two luna +# scouts and one luna worker, single route, direct from OpenAI. +# +# Against tb-gt-sol-2luna this holds the lead model, the persona bytes, the +# route, the price sheet, the clock and the container fixed and varies exactly +# one thing: how wide the recon phase is. It is the cell that tests goosetown's +# central claim, which the study has never had a way to evaluate -- +# +# "Every non-trivial task starts with researchers, not workers. Map the +# territory before sending the teams -- surprises are cheaper when they're on +# paper." (goosetown-orchestrator/SKILL.md:97) +# +# -- and the reason it is testable here at all is that scouts are READ-ONLY. Two +# writing agents in one container have to be serialised by the lead or they +# corrupt each other's work, which is why every previous team cell in this study +# effectively ran in series and paid a round trip for it. Two scouts cannot +# collide, so gt-lead.md is free to wake both in the same turn (one message +# each, several messages per turn) and get genuinely parallel recon. Deliberate +# overlap between them costs tokens and nothing else. +# +# The honest null: a second scout buys nothing on Terminal-Bench because most +# tasks are single-threaded surgery with one obvious place to look, and the study +# pays ~1.3x the recon tokens to learn that. That is doc 02 §7.2's threat applied +# to the recon phase, and it is why the result should be read stratified by task +# length rather than in aggregate -- if parallel recon pays anywhere, it pays on +# the long tail. +# +# Note the roster is now FOUR agents in one container, more than anything the +# study has run. See `environment` below: 4 vCPU and 8192 MB is the dataset +# maximum, not a comfortable margin, and container exit-137 is the thing to watch +# if this cell behaves unlike its 3-agent sibling. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-gt-sol-3luna +roster: + - id: lead + kind: orchestrator + role: lead + count: 1 + endpoint: gpt-5.6-sol + model_revision: gpt-5.6-sol + prompt: + path: personas/bench/gt/gt-lead.md + sha256: 8a27e833dec0a6d1700c5ea3100f8a81c09e5090ef0a589b9583fac40b24c05c + + - id: scout + kind: worker + role: scout + # The only line that differs from tb-gt-sol-2luna. The provisioner names + # seats `{id}-{index}`, 1-indexed, so this yields `scout-1` and `scout-2` as + # two separate rows in every agent's "Your team" table -- which is what lets + # the lead address them individually and give each a different angle rather + # than a smaller slice of the same one. + count: 2 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-scout.md + sha256: 0359a957428dd09e56e57a7fd3fe455d860264d910705f0e4992337dde25e5a5 + + - id: worker + kind: worker + role: worker + # Deliberately still one. Widening recon and widening implementation at the + # same time would make the G1s -> G2s delta unattributable, and the write + # side is the one that needs serialising anyway. + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/gt/gt-worker.md + sha256: 7a529ae58f2a2ec635fdf65fb43284b30c09d2c9dbf31e70951fe021c3b2b766 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Byte-identical to tb-gt-sol-2luna. Holding the rates equal is what makes the + # recon-width cost delta a measurement rather than an artefact of pricing. + gpt-5.6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + cache_read_rate: 0.0 + gpt-5.6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml. These are the + # dataset's maxima (cpus 83x1/3x2/3x4; memory_mb 68x2048/13x4096/8x8192) and + # Harbor replaces rather than raises, so they cannot go higher without + # shrinking the tasks that already ask for 8 GB. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-meli-solo-baseten.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-meli-solo-baseten.yaml new file mode 100644 index 0000000000..711209e3b0 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-meli-solo-baseten.yaml @@ -0,0 +1,39 @@ +# Meli solo cell, take 2: same Tyler-picked persona (buzz-benchmarking thread +# 74a65a09) on deepseek/deepseek-v4-flash-0731 via OpenRouter, now pinned to +# the baseten/fp8 upstream (Tyler, event f7457e0b) instead of gmicloud/fp8. +# +# Identical to tb-meli-solo.yaml except the pinned provider and its listed +# rates. Runs on the post-merge harness (PR #5145 head), which carries the +# #4896 vision-input recovery and the #5136 explicit-mention task post that +# the first solo run predated. The pin itself lives in +# testbed/endpoints/openrouter-live.json (OPENROUTER_PROVIDER_ORDER). +schema_version: "1" +condition: tb-meli-solo-baseten +roster: + - id: meli + kind: orchestrator + role: solo + count: 1 + endpoint: deepseek/deepseek-v4-flash-0731 + model_revision: deepseek/deepseek-v4-flash-20260731 + prompt: + path: personas/bench/meli-solo.md + sha256: de6330190fe22cb4b58cd5c2ca1cb7fc080e9b7f97b298c8960e08854e7a7a8b + generation: + thinking_effort: high +prices: + deepseek/deepseek-v4-flash-0731: + # baseten/fp8 listed rates (OpenRouter endpoints API, 2026-08-07): + # prompt $0.13/M, completion $0.26/M, cache read $0.028/M. + input_per_million_usd: 0.13 + cached_input_per_million_usd: 0.028 + output_per_million_usd: 0.26 + cache_read_rate: 0.0 +trial_budget: + # Identical to every other TB solo cell. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-meli-solo.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-meli-solo.yaml new file mode 100644 index 0000000000..bbf2905de7 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-meli-solo.yaml @@ -0,0 +1,37 @@ +# Meli solo cell: Tyler-picked persona (buzz-benchmarking thread 74a65a09) +# on deepseek/deepseek-v4-flash-0731 via OpenRouter. +# +# Byte-identical to tb-solo-deepseek-v4-flash-high.yaml except the persona: +# Meli's production system prompt verbatim plus one appended unattended-mode +# line (authorized by Tyler, event 36d96837). Same pinned gmicloud/fp8 +# upstream, same measured prices, same effort pin -- see that manifest for the +# full rationale. The pair isolates one variable: the persona. +schema_version: "1" +condition: tb-meli-solo +roster: + - id: meli + kind: orchestrator + role: solo + count: 1 + endpoint: deepseek/deepseek-v4-flash-0731 + model_revision: deepseek/deepseek-v4-flash-20260731 + prompt: + path: personas/bench/meli-solo.md + sha256: de6330190fe22cb4b58cd5c2ca1cb7fc080e9b7f97b298c8960e08854e7a7a8b + generation: + thinking_effort: high +prices: + deepseek/deepseek-v4-flash-0731: + # gmicloud/fp8 endpoint rates, measured by Atish (tb-solo-deepseek manifest). + input_per_million_usd: 0.133 + cached_input_per_million_usd: 0.0266 + output_per_million_usd: 0.266 + cache_read_rate: 0.0 +trial_budget: + # Identical to every other TB solo cell. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2gemini.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2gemini.yaml new file mode 100644 index 0000000000..54df782714 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2gemini.yaml @@ -0,0 +1,82 @@ +# B2 peer pair: two gemini-3.6-flash agents on one channel, one driving and one +# navigating — tb-peer-2luna with the model swapped. +# +# Two comparisons hang off this manifest, and they are the reason it exists: +# * against tb-solo-gemini it isolates the pairing, holding the model fixed; +# * against tb-peer-2luna it isolates the model, holding the pairing fixed. +# Only the second is new information. The first is the same question B1 already +# answers for luna, asked of a model whose failure modes differ — a cheap fast +# model may pair better or worse than a slower one, and one data point on that +# is worth more than another luna variant. +# +# The driver is the orchestrator because the roster allows exactly one, and +# because the driver is the agent the harness must wake with the task and the +# only one permitted to publish `DONE:`. The navigator is read-only by persona, +# not by permission. +# +# Route caveat, and it matters more here than for the solo: this pair runs on the +# MLflow chat/completions gateway, where a *multi-turn* tool loop is what broke +# before the 2026-07-29 provider fixes. Gemini requires each tool call's +# `thoughtSignature` echoed back on replay, and two agents mean twice the replay +# surface. See docs/08-gemini-provider-fixes.md. +# +# Endpoint names are exact Databricks serving-endpoint names; provider/host/key +# resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-peer-2gemini +roster: + - id: driver + kind: orchestrator + role: driver + count: 1 + endpoint: databricks-gemini-3-6-flash + model_revision: gemini-3.6-flash + prompt: + # Byte-identical to tb-peer-2luna's driver — the pairing is the constant. + path: personas/bench/peer-driver.md + sha256: 1cf4d1cbf4093dd0d9b1a6a7ccd9a875c39621d0bab3ddec7e0bb23177761149 + + - id: navigator + kind: worker + # Read verbatim by both personas out of the "Your team" table — peer-driver + # addresses `navigator` and peer-navigator addresses `driver`. Renaming + # either role here silently breaks the pairing: the @mention resolves to + # nobody, the send still reports success, and the trial stalls to timeout. + role: navigator + count: 1 + endpoint: databricks-gemini-3-6-flash + model_revision: gemini-3.6-flash + prompt: + path: personas/bench/peer-navigator.md + sha256: dcf2ae226b5967067687060871438397125e54b8c412f69076eb2a8edd56d2c5 + +prices: + # PLACEHOLDER — NOT A MEASUREMENT. Identical zeroes to tb-solo-gemini, and + # they have to be identical for the same reason tb-peer-2luna copies + # tb-solo-luna's figures: holding the rate equal is what makes the + # solo-versus-pair cost *ratio* sound, whatever the rate turns out to be. + # + # Token counts are measured and recorded per trial regardless, so cost is + # recoverable afterwards. Fill these in before quoting any dollar figure. + databricks-gemini-3-6-flash: + input_per_million_usd: 0.0 + cached_input_per_million_usd: 0.0 + output_per_million_usd: 0.0 + # Caching is load-bearing for a pair in a way it is not for a solo agent: + # the pair's cost is driven by re-sent context, which is exactly what a + # prefix cache serves. Whether this route reports a cache split at all is + # itself a finding — read `accounting_cache_read_tokens_are_measured` off + # the first trial. + cache_read_rate: 0.0 +trial_budget: + # Matches every other condition exactly. A pair genuinely needs longer than a + # solo agent — every handoff is a round trip — but giving it a larger budget + # would confound the comparison, and this value does not bind anyway: Harbor + # enforces each task's own `[agent] timeout_sec`, scaled by the study's 3x + # multiplier. + timeout_seconds: 36000 + +environment: + # Identical in every condition — see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2luna-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2luna-high.yaml new file mode 100644 index 0000000000..a1afd15598 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2luna-high.yaml @@ -0,0 +1,88 @@ +# B1h -- two gpt-5.6-luna peers at reasoning effort `high`, direct from OpenAI. +# +# The pairing question asked again with the effort dial turned up. It exists +# because the two dials could interact, and the study has no way to see that +# from the solo cells alone: +# +# B1 (medium) scored 0.536 against A1's 0.545 -- pairing bought nothing at +# the cheap floor. One reading is that two luna agents simply cannot solve +# what one luna agent cannot. Another is that the pair spends its advantage +# on coordination it is not thinking hard enough to use well. Those predict +# different things at `high`: the first says B1h tracks A1h exactly, the +# second says the gap closes or inverts. +# +# Against tb-solo-luna-high it holds model, route, price sheet, persona bytes, +# effort, clock and container fixed and varies only headcount -- so the A1h/B1h +# delta is attributable to the pairing itself. Against tb-peer-2luna it varies +# effort *and* route together; see tb-solo-luna-high.yaml for why the route +# moved and why that comparison must be reported as a joint one. +# +# Expect this to be the most expensive cell of the four. Two agents at high +# effort, and every handoff re-sends context -- doc 02 §2's point that re-sent +# context is the entire bill applies with extra force when each of those tokens +# was more expensive to produce. That is the measurement, not a defect. +schema_version: "1" +condition: tb-peer-2luna-high +roster: + - id: driver + kind: orchestrator + role: driver + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/peer-driver.md + sha256: 1cf4d1cbf4093dd0d9b1a6a7ccd9a875c39621d0bab3ddec7e0bb23177761149 + generation: + # Both roster entries pin the same level. Raising effort for only one seat + # would be a different and more confusing experiment -- an asymmetric pair + # -- and it is not the one that was asked for. + thinking_effort: high + + - id: navigator + kind: worker + # Read verbatim by both personas out of the "Your team" table -- peer-driver + # addresses `navigator` and peer-navigator addresses `driver`. Renaming + # either role here silently breaks the pairing: the @mention resolves to + # nobody, the send still reports success, and the trial stalls to timeout. + role: navigator + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/peer-navigator.md + sha256: dcf2ae226b5967067687060871438397125e54b8c412f69076eb2a8edd56d2c5 + generation: + thinking_effort: high + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Identical to tb-solo-luna-high, deliberately -- holding the rates equal is + # what makes the solo-versus-pair cost *ratio* sound. + gpt-5.6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only; superseded by the provider's measured split. Caching is + # load-bearing for this condition in a way it is not for a solo one: the + # pair's cost is driven by re-sent context, which is exactly what a prefix + # cache serves. + cache_read_rate: 0.0 +trial_budget: + # Matches the solo cells exactly. A pair genuinely needs longer than a solo + # agent -- every handoff is a round trip -- but giving it a larger budget + # would confound the comparison, and it does not bind anyway: Harbor enforces + # each task's own `[agent] timeout_sec` scaled by the study's 3x multiplier. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2luna.yaml new file mode 100644 index 0000000000..9cb5837283 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2luna.yaml @@ -0,0 +1,107 @@ +# B-tier peer pair: two gpt-5.6-luna agents on one channel, one driving and one +# navigating, served by the same Databricks slate as tb-solo-luna. +# +# This is the controlled comparison the mixed-model team manifest cannot make. +# Against tb-solo-luna it holds the model, the serving path, the price sheet, the +# generation settings and the trial budget fixed, and varies exactly one thing: +# how many agents share the container. Any delta in score, cost, tokens or +# wall-clock is attributable to the pairing itself. +# +# The driver is the orchestrator because the roster allows exactly one, and +# because the driver is the agent the harness must wake with the task and the +# only one permitted to publish `DONE:`. The navigator is read-only by persona, +# not by permission — both agents get the same tools and the same filesystem, +# so the write boundary is instruction-enforced and worth checking in the +# transcripts rather than assuming. +# +# Expect this to cost more than solo per task even where it scores the same: +# two agents, and every handoff re-sends context. That is the measurement, not +# a defect. +# +# Endpoint names are exact Databricks serving-endpoint names; provider/host/key +# resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-peer-2luna +roster: + - id: driver + kind: orchestrator + role: driver + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/peer-driver.md + sha256: 1cf4d1cbf4093dd0d9b1a6a7ccd9a875c39621d0bab3ddec7e0bb23177761149 + + - id: navigator + kind: worker + # Read verbatim by both personas out of the "Your team" table — peer-driver + # addresses `navigator` and peer-navigator addresses `driver`. Renaming + # either role here silently breaks the pairing: the @mention resolves to + # nobody, the send still reports success, and the trial stalls to timeout. + role: navigator + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/peer-navigator.md + sha256: dcf2ae226b5967067687060871438397125e54b8c412f69076eb2a8edd56d2c5 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Operator-supplied list prices, doc 04 §7, identical to tb-solo-luna's luna + # figures — holding them equal is what makes the solo-versus-pair cost *ratio* + # sound. See tb-solo-luna.yaml for why cache_read_rate is 0.0 (DB10). + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only; superseded by the provider's measured cache split, with + # `accounting_cache_read_tokens_are_measured` recording which was used. 0.0 + # keeps an unmeasured run conservative. + # + # Caching is load-bearing for this condition in a way it is not for a solo + # one: the pair's cost is driven by re-sent context, which is exactly what a + # prefix cache serves. + cache_read_rate: 0.0 +trial_budget: + # Matches tb-solo-luna exactly. A pair genuinely needs longer than a solo agent + # — every handoff is a round trip — but giving it a larger budget would confound + # the comparison, and this value does not bind anyway: Harbor enforces each + # task's own `[agent] timeout_sec`, which tops out at 12000s across + # Terminal-Bench 2.1, scaled by the study's 3x timeout multiplier. Both + # conditions therefore run against the task's own clock, which is the only + # budget the comparison can honestly share. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2terra-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2terra-high.yaml new file mode 100644 index 0000000000..317c9f1032 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-2terra-high.yaml @@ -0,0 +1,75 @@ +# B3h -- two gpt-5.6-terra peers at reasoning effort `high`, direct from OpenAI. +# +# The fourth corner of the effort wave, and the one that speaks most directly +# to the study's headline. Doc 02 §2 states the parity arithmetic: a cheap team +# wins if it consumes less than the price ratio's worth of extra tokens. Terra +# sits at exactly 0.5x sol, so the parity line for a terra *pair* against solo +# sol is unusually easy to state and unusually easy to test: +# +# two terra agents cost less than one sol agent iff they consume less than +# 2x sol's tokens -- and a pair that re-sends context to a second seat is +# right at the edge of that. B1 measured 1.71x A1's cost at medium (13.67 vs +# 8.00 over the same tasks), so the pairing tax is close to, but under, the +# 2x line. Whether it stays under at `high` is the number this cell exists +# to produce. +# +# Against tb-solo-terra-high it varies only headcount. Against tb-peer-2luna-high +# it varies only model tier. Both are clean single-variable comparisons because +# every cell in this wave shares one route, one persona set, one clock and one +# container size. +# +# Same cache-write caveat as tb-solo-terra-high: terra's published $2.50/Mtok +# write rate has no field in this schema, so the reported cost is a slight +# under-estimate -- and slightly more so here, because a pair writes more cache +# than a solo agent does. +schema_version: "1" +condition: tb-peer-2terra-high +roster: + - id: driver + kind: orchestrator + role: driver + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/peer-driver.md + sha256: 1cf4d1cbf4093dd0d9b1a6a7ccd9a875c39621d0bab3ddec7e0bb23177761149 + generation: + thinking_effort: high + + - id: navigator + kind: worker + # See tb-peer-2luna-high.yaml: this string is resolved by persona text, not + # by the harness, and renaming it fails silently into a timeout. + role: navigator + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + path: personas/bench/peer-navigator.md + sha256: dcf2ae226b5967067687060871438397125e54b8c412f69076eb2a8edd56d2c5 + generation: + thinking_effort: high + +prices: + # Repriced 2026-07-30: terra 2.5/0.25/15.0 -> 2.00/0.20/12.00 (-20%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Identical to tb-solo-terra-high. Operator-supplied 2026-07-30. + gpt-5.6-terra: + input_per_million_usd: 2.0 + cached_input_per_million_usd: 0.2 + output_per_million_usd: 12.0 + cache_read_rate: 0.0 +trial_budget: + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-sol-opus.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-sol-opus.yaml new file mode 100644 index 0000000000..12782a32e3 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-peer-sol-opus.yaml @@ -0,0 +1,128 @@ +# D1 -- gpt-5.6-sol driving, claude-opus-5 navigating. The steelman for teams. +# +# WHY THIS CELL EXISTS. Every multi-agent cell in the study so far has a +# capability gradient: an expensive lead with cheap workers (C3, G1, G1s), or a +# uniform floor (B1, C1, C2, G0). That leaves the anti-team result one easy +# objection -- "of course coordination did not help, you paired a strong model +# with weak ones and the weak ones dragged it down." This cell removes the +# objection. On Terminal-Bench the two solos are statistically tied at the top +# (A3 opus 0.851, A2 sol 0.843), so this is two near-equal strong peers with no +# weak seat to blame. If a pair of the study's best models cannot beat either of +# them working alone, the finding is very hard to argue with. +# +# It is deliberately the most favourable configuration for the team thesis that +# this study can construct. That is the point: the result only means something +# if the setup was trying to succeed. +# +# THE CONTROLS ALREADY EXIST, WHICH IS WHY IT IS ON TERMINAL-BENCH AND BOTH +# SEATS ARE ON DATABRICKS. Against A2 (`tb-solo-sol`, databricks-gpt-5-6-sol, +# 0.843) and A3 (`tb-solo-opus`, databricks-claude-opus-5, 0.851) this holds +# route, effort, price sheet, clock, container and dataset fixed and varies only +# headcount. Comparing a mixed pair against only ONE solo would conflate "added +# an agent" with "added a better model" -- the joint-delta mistake tb-solo-luna-high +# made, that tb-solo-luna-openai had to be built to undo, and that the G1s-vs-A2 +# headline still carries. Here there is nothing left to caveat: it is the most +# tightly controlled cell in the study. +# +# The mixed-provider question was checked before writing this: a single endpoint +# config may name entries from different providers, because benchmark.py's +# `write_provisioner_config` resolves each entry's own `api_key_env` and each +# entry carries its own `provider` and `env`. It is not needed here -- both +# seats are Databricks, so `testbed/endpoints/databricks-live.json` covers them +# -- but it means the OpenAI-routed variant is possible without a harness change. +# +# WHICH MODEL DRIVES, AND WHY IT IS NOT ARBITRARY. Sol drives, Opus navigates. +# The only plausible mechanism by which a pair beats a strong solo is that a +# second strong reader catches what the implementer misses; two implementers in +# parallel is not what these personas do. Putting the better solo (opus, 0.851) +# in the reviewing seat tests that mechanism directly. The reverse assignment is +# a different experiment and would need its own cell -- do not read this one as +# covering both. +# +# COST WARNING. The pair axis has run ~1.76x its solo, and these are the two +# most expensive solos in the study ($41.75 and $61.75). Budget $85-110 and +# ~6h. There is no cheaper version of this question. +# +# SCHEDULING CONSTRAINT -- this is not optional. Opus on Databricks is capped on +# OUTPUT tokens per minute WORKSPACE-WIDE, so two boxes running opus collide +# invisibly. The collision is differentially harmful to exactly this comparison: +# a fatal 429 costs a team cell ~0.26 reward and costs a solo cell nothing, so +# throttling would damage this pair while leaving A2 and A3 -- already banked -- +# untouched. That is bias pointing the wrong way. Do not start this cell while +# G1 (`tb-gt-opus-2luna`) is running on another box. +# +# Endpoint names resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-peer-sol-opus +roster: + - id: driver + kind: orchestrator + role: driver + count: 1 + endpoint: databricks-gpt-5-6-sol + model_revision: databricks-gpt-5-6-sol + prompt: + # Byte-identical to B1/B1h/B3h, so this sits on the same pair axis rather + # than being a new persona experiment wearing a pair's clothes. + path: personas/bench/peer-driver.md + sha256: 1cf4d1cbf4093dd0d9b1a6a7ccd9a875c39621d0bab3ddec7e0bb23177761149 + + - id: navigator + kind: worker + # Read verbatim by both personas out of the "Your team" table -- peer-driver + # addresses `navigator` and peer-navigator addresses `driver`. Renaming + # either role here silently breaks the pairing: the @mention resolves to + # nobody, the send still reports success, and the trial stalls to timeout. + role: navigator + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: databricks-claude-opus-5 + prompt: + path: personas/bench/peer-navigator.md + sha256: dcf2ae226b5967067687060871438397125e54b8c412f69076eb2a8edd56d2c5 + +# NO `generation` BLOCK ON EITHER SEAT, deliberately. A2 and A3 do not pin +# `thinking_effort` either, so all three sit at the provider default and the +# comparison is effort-matched. Pinning a level here would look tidier and would +# add back exactly one difference against both controls. (Unset means the +# harness omits the `reasoning` field entirely -- see +# `responses_body_omits_reasoning_when_effort_none` in buzz-agent/src/llm.rs.) + +prices: + # Both rows copied verbatim from tb-solo-sol.yaml and tb-solo-opus.yaml. This + # is what makes the pair-versus-solo cost RATIO sound: the dollars move only + # because token volume moved, never because a rate was edited. + databricks-gpt-5-6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + # Fallback only; the agent reports the provider's measured cache split and + # the accounting path prices that, recording which it used + # (`accounting_cache_read_tokens_are_measured`). 0.0 keeps an unmeasured run + # conservative rather than inventing a discount. + cache_read_rate: 0.0 + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + # Note opus is NOT a clean 5x luna on every axis the way sol is: input and + # cache-read are 5x (5.0 vs 1.0, 0.5 vs 0.1) but output is ~4.17x. + output_per_million_usd: 25.0 + # Cache-write caveat, and it bites hardest in a team: the anthropic route + # reports `cache_creation_input_tokens` separately and Anthropic bills cache + # writes at ~1.25x input, but our accounting prices write tokens at the plain + # input rate (02 §7.5 residual). A pair re-sends context constantly, so write + # volume here is larger than in A3. Opus cost in this cell is understated by + # that margin -- confirm before publishing its dollar figure. + cache_read_rate: 0.0 + +trial_budget: + # Matches A2 and A3 exactly. A pair genuinely needs longer than a solo agent -- + # every handoff is a round trip -- but giving it a larger budget would confound + # the comparison, and it does not bind anyway: Harbor enforces each task's own + # `[agent] timeout_sec` scaled by the study's 3x multiplier. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-deepseek-v4-flash-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-deepseek-v4-flash-high.yaml new file mode 100644 index 0000000000..30ec5183e4 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-deepseek-v4-flash-high.yaml @@ -0,0 +1,67 @@ +# OR2h -- solo deepseek/deepseek-v4-flash-0731 at reasoning effort `high`, +# via OpenRouter. +# +# The cheap end of the OpenRouter pair. Against tb-solo-kimi-k3-high this is a +# tier comparison at a 22x input price ratio ($0.133 vs $3.00 per Mtok), which +# makes it the natural open-weight analogue of the luna -> sol ladder. +# +# THE UPSTREAM IS PINNED, AND THE CHOICE IS NOT THE OBVIOUS ONE. OpenRouter +# serves this model id from nine upstreams spanning fp4 to fp8 quantization, +# 262K to 1M context, and $0.09-$0.14 per Mtok. Cheapest is deepinfra/fp4 -- +# and it caches well -- but fp4 is a materially more degraded set of weights +# than the fp8 every other endpoint serves, so benchmarking it and labelling +# the row "deepseek-v4-flash-0731" would overstate what ran. This pins +# `gmicloud/fp8`: fp8, the full 1M context, 98.9% uptime, and the cheapest of +# the fp8 endpoints at $0.133/$0.266. +# +# Caching was MEASURED per endpoint, not read off the capability flag, because +# the flag is wrong. `/api/v1/models/.../endpoints` advertises +# `supports_implicit_caching: false` for gmicloud/fp8 -- which caches a repeated +# 21K prefix on every single call -- and `true` only for deepseek/fp8, which is +# not routable on this account at all (404 with allow_fallbacks off). Measured, +# three calls each: gmicloud/fp8 cached 21760 every time at $0.00059/call; +# siliconflow/fp8 cached 0 every time at $0.00306; unpinned cached 0, 0, 21760. +# So the pin buys ~5x on input cost as well as a fixed condition. +# +# EFFORT IS PINNED BUT ITS EFFECT ON THIS MODEL IS SMALL AND WORTH CHECKING IN +# THE RESULTS. deepseek-v4-flash-0731 advertises supported_efforts +# ["max","high","low"] and defaults to "high", so `high` here matches the +# provider default; it is pinned anyway so a later default change cannot +# silently redefine the cell. On the fp4 endpoint, reasoning length did not +# separate low from high at all (medians 1097 / 1067, within-level spread +# wider than between-level). If the effort cells land on top of each other, +# that measurement -- not the benchmark -- is the first thing to re-read. +schema_version: "1" +condition: tb-solo-deepseek-v4-flash-high +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: deepseek/deepseek-v4-flash-0731 + model_revision: deepseek/deepseek-v4-flash-20260731 + prompt: + # Byte-identical to every other solo cell in the study. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + thinking_effort: high +prices: + deepseek/deepseek-v4-flash-0731: + # The gmicloud/fp8 endpoint's own rates -- NOT the model id's headline + # $0.14/$0.28, which belongs to a different set of upstreams. Pinning is + # what makes this row the correct one. + input_per_million_usd: 0.133 + cached_input_per_million_usd: 0.0266 + output_per_million_usd: 0.266 + # Fallback only; the pinned route reports the real split and the harness + # prefers it. See tb-solo-kimi-k3-high.yaml. + cache_read_rate: 0.0 +trial_budget: + # Identical to every other TB solo cell. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-gemini.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-gemini.yaml new file mode 100644 index 0000000000..e5e9ae24b9 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-gemini.yaml @@ -0,0 +1,70 @@ +# A4 solo baseline: the same single-agent condition as tb-solo-luna, on +# gemini-3.6-flash. +# +# Third model on the solo axis, and the first one that is neither OpenAI- nor +# Anthropic-shaped. Roster shape, persona bytes, effort, budget and clock are +# identical to tb-solo-luna and tb-solo-sol — only the endpoint moves — so a +# score difference is attributable to the model. +# +# Unlike the other two, this one exercises a *third* serving route. Endpoint +# names containing neither `gpt-5` nor `claude` fall through +# `databricks_v2_route_for_model` to the MLflow chat/completions gateway, which +# has its own body builder and parser (`openai_body` / `parse_openai`). Two +# defects there made Gemini unusable until 2026-07-29 — dropped thought +# signatures and discarded block-array text — so a run of this manifest on a +# buzz-agent older than that fix scores near zero for reasons that have nothing +# to do with the model. See docs/08-gemini-provider-fixes.md. +# +# Endpoint names are exact Databricks serving-endpoint names: the runtime passes +# the manifest endpoint name to the gateway as the model. They resolve to +# provider/host/key via testbed/endpoints/databricks-live.json, which is +# deployment config and deliberately outside this manifest. +schema_version: "1" +condition: tb-solo-gemini +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-gemini-3-6-flash + model_revision: gemini-3.6-flash + prompt: + # Byte-identical to tb-solo-luna and tb-solo-sol. The persona is model- and + # provider-agnostic on purpose; a prompt tuned per model would make the + # three incomparable. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe +prices: + # PLACEHOLDER — NOT A MEASUREMENT. Doc 04 §7 carries operator-supplied list + # prices for the GPT-5.6 endpoints only; there is no Gemini row, and the + # serving-endpoint record exposes no rate. Zeroes are deliberate: a wrong + # non-zero rate would produce a plausible dollar figure that silently + # misreports, whereas $0.00 is obviously not a cost. + # + # This does NOT affect the run. Token counts are measured per trial and + # recorded in the receipts either way, so cost is recoverable afterwards as + # tokens x rate. Fill these in before quoting any dollar figure for A4/B2. + databricks-gemini-3-6-flash: + input_per_million_usd: 0.0 + cached_input_per_million_usd: 0.0 + output_per_million_usd: 0.0 + # See tb-solo-luna.yaml for why this is 0.0 (DB10): the agent reports the + # provider's measured cache split and the accounting path prices that, + # recording which it used in `accounting_cache_read_tokens_are_measured`. + cache_read_rate: 0.0 +trial_budget: + # Identical to the other solo baselines, and it has to be: the trial budget is + # a backstop above the longest task in the dataset (12000s) at the 3x timeout + # multiplier, so each task's own `[agent] timeout_sec` governs instead of a + # flat harness deadline. A different value here would mean the conditions ran + # on different clocks, and benchmark.py's check_budget_clears_clock would + # refuse to start if it were smaller than 12000 x 3. + timeout_seconds: 36000 + +environment: + # Identical in every condition — see tb-solo-luna.yaml for the full reasoning. + # 4 and 8192 are the maxima the dataset declares, and Harbor replaces rather + # than raises, so any smaller value would quietly shrink the tasks that ask + # for more. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-kimi-k3-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-kimi-k3-high.yaml new file mode 100644 index 0000000000..1887b8bc51 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-kimi-k3-high.yaml @@ -0,0 +1,67 @@ +# OR1h -- solo moonshotai/kimi-k3 at reasoning effort `high`, via OpenRouter. +# +# The first cell in the study on a route other than direct-OpenAI or Databricks, +# and the first on open-weight models. Read it against the OpenAI solo cells for +# a tier comparison only -- the ROUTE differs, so this is not a clean +# model-versus-model contrast in the way A5m -> A5h is a clean effort contrast. +# The same caveat that gated G1s versus A2 applies here and is larger. +# +# THE UPSTREAM IS PINNED, AND THAT IS LOAD-BEARING. `moonshotai/kimi-k3` is not +# one deployment: OpenRouter serves it from ten upstreams spanning mxfp4/fp8/ +# unknown quantization, 912K-1048K context, and $3.00-$4.50 per Mtok input. +# Unpinned, consecutive requests land on different ones and the cell measures a +# mixture whose composition moves with provider load. testbed/endpoints/ +# openrouter-live.json pins `moonshotai/mxfp4` -- Moonshot's own deployment, +# 100% uptime over the sampling window -- and buzz-agent sends it with +# `allow_fallbacks: false`, so an unavailable upstream fails loudly instead of +# quietly redefining the condition mid-run. +# +# The pin also decides whether prompt caching happens at all. Measured +# 2026-08-01 over a fixed 21K-token prefix, three calls: unpinned returned +# cached_tokens 0, 0, 20480; pinned to moonshotai/mxfp4 it returned 20480 every +# time, $0.00715 against $0.06245 per call. Nearly 9x on input cost, decided by +# routing luck. Do not remove the pin to "let OpenRouter optimise". +# +# EFFORT REACHES THE MODEL AS OpenRouter's `reasoning.effort`, not OpenAI's +# `reasoning_effort` -- see apply_openrouter_mutations in llm.rs. kimi-k3 +# advertises supported_efforts ["max","high","low"] and defaults to "max", so +# `high` here is a deliberate step DOWN from the provider default and must stay +# pinned: leaving it unset would silently run this cell at max. +# +# PRICES are the moonshotai/mxfp4 endpoint's own rates, which are also the +# headline rates for the model id. They are NOT the rates of the two premium +# kimi endpoints ($4.50/$22.50); the pin is what makes the cheaper row correct, +# which is another reason it cannot be dropped without restating cost. +schema_version: "1" +condition: tb-solo-kimi-k3-high +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: moonshotai/kimi-k3 + model_revision: moonshotai/kimi-k3-20260715 + prompt: + # Byte-identical to every other solo cell in the study. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + thinking_effort: high +prices: + moonshotai/kimi-k3: + input_per_million_usd: 3.00 + cached_input_per_million_usd: 0.30 + output_per_million_usd: 15.00 + # Fallback only. OpenRouter reports the split at + # usage.prompt_tokens_details.cached_tokens, which buzz-agent already reads + # (openai_chat_cached_tokens, llm.rs) -- verified live on this exact pinned + # route, not assumed. 0.0 keeps an unmeasured run conservative. + cache_read_rate: 0.0 +trial_budget: + # Identical to every other TB solo cell. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-high.yaml new file mode 100644 index 0000000000..394c421c54 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-high.yaml @@ -0,0 +1,93 @@ +# A1h -- solo luna at reasoning effort `high`, served direct from OpenAI. +# +# Part of the effort wave (doc 02 §3.2). A2x/A3x ask whether more thinking buys +# solved tasks at the *frontier*; this pair of luna cells asks it at the cheap +# floor, where the answer could plausibly differ: a weaker model has more +# headroom to gain from thinking, and A1's 0.545 leaves a lot of room above it. +# +# TWO THINGS CHANGE FROM tb-solo-luna, NOT ONE, and that is deliberate but must +# be stated wherever this cell is compared: +# +# 1. effort medium -> high +# 2. route Databricks AI Gateway -> api.openai.com directly +# +# The route moved because the Databricks staging gateway rate-limited the +# 2026-07-29 wave hard enough to distort results -- the Gemini cell breached on +# 100% of its trials -- and a cell whose purpose is to isolate one dial cannot +# also be absorbing 429 backoff. The cost is that an A1-vs-A1h delta is not a +# clean effort measurement; it is effort plus whatever the two serving paths do +# differently. Both models are the same `gpt-5.6-luna`, so the expected route +# effect is small, but "expected small" is not "measured zero" and the honest +# framing in the post is a joint one. +# +# RESOLVED 2026-07-29 -- it is now measured zero, so the post can say "effort" +# rather than "effort plus route". tb-solo-luna-openai (A1r) holds effort at the +# default and moves only the route: route alone is -0.023 at p = 0.815 (8:10 +# discordant, a null) while effort alone is -0.170 at p = 0.001 (2:17). The two +# sum to -0.193 against the joint -0.195. Route was inert in cost and tokens as +# well, within 3% on every column. Do not delete A1r; it is what licenses the +# single-variable framing here. +# +# The within-wave comparisons are clean, because all four effort-wave cells sit +# on this same route: A1h vs A5h (luna vs terra at high) and A1h vs B1h (solo vs +# pair at high) vary exactly one thing each. +# +# Endpoint names here are literal OpenAI model ids, because the runtime sets +# BUZZ_AGENT_MODEL from the manifest endpoint name. Resolution to +# provider/key lives in testbed/endpoints/openai-live.json, outside this +# manifest. No base-url override: the default lets OPENAI_COMPAT_API=auto pick +# the Responses API, which is the route these reasoning models need. +schema_version: "1" +condition: tb-solo-luna-high +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + # Byte-identical to tb-solo-luna, tb-solo-sol and tb-solo-terra-high. + # Tuning the persona for a higher effort level would confound the dial + # with a prompt change. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + # The line this condition exists for. `high` rather than `xhigh` because + # the operator asked for high on the luna/terra arm; note the live probe + # measured luna at 1.22x reasoning tokens for high and only 1.25x for + # xhigh, so high captures nearly all of the available travel anyway. + thinking_effort: high +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Identical to tb-solo-luna's luna figures. Effort changes token *volume*, not + # the rate card, so holding the rates equal is what makes the cost delta a + # measurement of what the extra thinking cost. + gpt-5.6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only. On the direct OpenAI Responses route the provider reports + # the split at `usage.prompt_tokens_details.cached_tokens` and the harness + # threads it through, stamping + # `accounting_cache_read_tokens_are_measured: true`. 0.0 keeps an + # unmeasured run conservative rather than optimistic. + cache_read_rate: 0.0 +trial_budget: + # Unchanged from tb-solo-luna. This is the field most tempting to raise for a + # higher-effort run, and raising it would be a mistake: if `high` needs a + # longer clock to finish the same work, that is a *result* about effort, not + # a budgeting problem to engineer away. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-nobase.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-nobase.yaml new file mode 100644 index 0000000000..d333568c3e --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-nobase.yaml @@ -0,0 +1,60 @@ +# A1 sensitivity check: the solo baseline with buzz-acp's [Base] platform +# section suppressed. +# +# Byte-identical to tb-solo-luna.yaml except for include_platform_prompt. That +# is the point — the pair isolates one variable, so any score, cost, or token +# delta between them is attributable to the ~12KB of production-workspace +# guidance buzz-acp prepends ahead of the pinned persona +# (crates/buzz-acp/src/base_prompt.md). +# +# Why it is worth a run: that section tells agents to sweep `buzz feed get` on +# startup, look for RESEARCH/ and PLANS/ and AGENTS.md, edit inside a git +# worktree, and that "publishing is optional and silence is usually correct". +# In a graded container the first three waste tool rounds and can write files +# the grader sees, and the fourth is how a multi-agent trial stalls. Every +# persona in personas/bench/ overrides all of it in prose. This condition is +# what turns that override from an assertion into a measurement. +# +# Not a headline condition. tb-solo-luna.yaml stays the baseline, because +# [Base] is what a real Buzz agent receives and the study is about Buzz. Run +# this once, at the same k, and report the delta as a caveat. +schema_version: "1" +condition: tb-solo-luna-nobase +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + include_platform_prompt: false + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Operator-supplied list prices, doc 04 §7. See tb-solo-luna.yaml for why + # cache_read_rate is 0.0 (DB10). + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + # 36000 = the dataset's longest task (12000s) at the 3x timeout multiplier the + # study runs with (DEFAULT_TIMEOUT_MULTIPLIER in scripts/benchmark.py). The two + # are one setting in two files: benchmark.py refuses to start if this is the + # smaller of the pair, because the harness would then cut a task short of the + # deadline Harbor granted it and record a timeout the agent never hit. A flat + # 900s did exactly that before the multiplier existed — 39 of the 89 tasks + # allow more than 900s, and one live trial died at 905s against a task budget + # of 1800s. + timeout_seconds: 36000 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-openai.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-openai.yaml new file mode 100644 index 0000000000..8b09a2581d --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna-openai.yaml @@ -0,0 +1,103 @@ +# A1r -- solo luna at the DEFAULT reasoning effort, served direct from OpenAI. +# +# The route control for the effort wave, and the cell that makes A1h readable. +# +# tb-solo-luna-high says plainly in its own header that it changes two things at +# once against tb-solo-luna: effort medium -> high AND route Databricks -> +# api.openai.com. That was a defensible trade at the time (the gateway was +# rate-limiting hard enough to distort a cell whose only job is to isolate one +# dial), but it left the resulting delta joint. The measured delta turned out to +# be large enough that the ambiguity matters: +# +# A1 tb-solo-luna medium, Databricks 0.552 $7.94 in 25.0M +# A1h tb-solo-luna-high high, OpenAI 0.747 $24.72 in 103.9M +# matched on the 87 tasks both scored: +0.195, discordant 18:1, p=0.000 +# +# A +0.195 at p=0.000 is the largest effect in the study so far, and as it stands +# it is attributable to "effort or route or both". This cell holds effort at the +# default and moves only the route, which splits that single number into two +# measurable ones: +# +# A1 -> A1r same effort, different route = the route effect, alone +# A1r -> A1h same route, different effort = the effort effect, alone +# +# If A1r lands near 0.552 the effort claim survives intact and the route is +# shown to be inert. If it lands near 0.747 then what was going to be published +# as an effort result is really a serving-path result, which would be a finding +# about the Databricks gateway rather than about reasoning -- and a much less +# interesting one to have gotten wrong in print. +# +# ANSWERED 2026-07-29. It landed at 0.568, i.e. on the A1 end. Paired McNemar on +# the shared task set: +# +# route alone A1 -> A1r -0.023 discordant 8:10 p = 0.815 NULL +# effort alone A1r -> A1h -0.170 discordant 2:17 p = 0.001 +# joint A1 -> A1h -0.195 discordant 1:18 p = 0.000 +# +# The two parts sum to -0.193 against a joint -0.195, so the decomposition is +# additive and the effort claim survives essentially whole. The route is inert +# in cost and tokens too, which is the stronger version of the result: over 87 +# shared tasks A1 spent $7.90 / 24.9M input and A1r spent $7.96 / 25.5M -- under +# 3% apart on every column. A1h spent $24.63 / 103.8M for its +0.170. +# +# Keep this cell in the matrix rather than folding it into A1. It is cheap, it +# is the only thing standing between "high effort is worth 3.1x cost" and "we +# changed two variables and guessed which one mattered", and if the gateway ever +# starts degrading output it is the tripwire that would show it. +# +# Byte-identical to tb-solo-luna-high except for the condition name and the +# absent `generation` block. In particular the persona hash, the price sheet, the +# trial budget and the environment overrides are all held, because every one of +# them would otherwise leak into the route comparison. +schema_version: "1" +condition: tb-solo-luna-openai +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + # The literal OpenAI model id; the runtime sets BUZZ_AGENT_MODEL from the + # endpoint name and resolution lives in testbed/endpoints/openai-live.json. + # Same string tb-solo-luna-high uses, which is the point. + endpoint: gpt-5.6-luna + model_revision: gpt-5.6-luna + prompt: + # Byte-identical to tb-solo-luna, tb-solo-luna-high, tb-solo-sol and + # tb-solo-terra-high. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + # NO `generation` block, and its absence is the whole cell. Omitting + # thinking_effort leaves the provider default (medium), which is exactly what + # tb-solo-luna ran at on the other route. Pinning `medium` explicitly would + # look tidier and would be wrong: it would test "effort explicitly set to + # medium" against A1's "effort unset", one more difference than intended. +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Identical to tb-solo-luna and tb-solo-luna-high. Rates are per-token, so the + # same sheet across all three cells is what makes the cost column a measurement + # of token volume rather than of a pricing edit. + gpt-5.6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only; on the direct OpenAI Responses route the provider reports + # the split at usage.prompt_tokens_details.cached_tokens and the harness + # stamps accounting_cache_read_tokens_are_measured: true. 0.0 keeps an + # unmeasured run conservative rather than optimistic. + cache_read_rate: 0.0 +trial_budget: + # Unchanged from both siblings. A route that needs a longer clock for the same + # work is a result, not a budgeting problem. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna.yaml new file mode 100644 index 0000000000..499cbb46cf --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-luna.yaml @@ -0,0 +1,83 @@ +# Terminal-Bench single-agent baseline on the Databricks bring-up slate. +# +# One agent, no workers, one Buzz channel. This is the condition every +# multi-agent result is measured against, so the lone agent gets byte-identical +# wiring to a worker — same binaries, same MCP toolset, same env. The only +# difference is the persona and the fact that there is nobody to delegate to. +# +# Endpoint names are exact Databricks serving-endpoint names: the runtime +# passes the manifest endpoint name to the gateway as the model. They resolve +# to provider/host/key via testbed/endpoints/databricks-live.json, which is +# deployment config and deliberately outside this manifest. +# +# context_window_tokens is 272000, not the model's full window, on purpose. +# 272k is the input boundary below which the short-context price table (doc 04 +# §7) applies, and it is what buzz-agent's compaction ceiling defaults to. Set +# to the true window instead and every long trial would silently cross into an +# unknown, higher rate tier (DB9). +schema_version: "1" +condition: tb-solo-luna +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Operator-supplied list prices, doc 04 §7. cache_read_rate stays 0.0 until + # DB10 is settled — these endpoints are on the OpenAI Responses route, which + # caches automatically, but the published cache-write line item implies + # opt-in caching that buzz-agent never requests. 0.0 over-reports by at most + # 10% of the input component, i.e. errs expensive. + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + # 36000 = the dataset's longest task (12000s) at the 3x timeout multiplier the + # study runs with (DEFAULT_TIMEOUT_MULTIPLIER in scripts/benchmark.py). The two + # are one setting in two files: benchmark.py refuses to start if this is the + # smaller of the pair, because the harness would then cut a task short of the + # deadline Harbor granted it and record a timeout the agent never hit. A flat + # 900s did exactly that before the multiplier existed — 39 of the 89 tasks + # allow more than 900s, and one live trial died at 905s against a task budget + # of 1800s. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-opus-xhigh.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-opus-xhigh.yaml new file mode 100644 index 0000000000..5797e81998 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-opus-xhigh.yaml @@ -0,0 +1,69 @@ +# A3x solo baseline: tb-solo-opus with reasoning effort raised medium -> xhigh. +# +# The effort axis (G2) on the frontier denominator. Everything except +# `thinking_effort` is byte-identical to tb-solo-opus — roster, persona bytes, +# endpoint, revision, prices, budget, clock, container resources — so the +# A3-vs-A3x delta isolates effort, in both score and cost. +# +# Paired with tb-solo-sol-xhigh, this asks the question the study actually cares +# about: what is *good enough*. Two axes, measured the same way — model tier +# (luna -> sol -> opus) and effort (medium -> xhigh) — and the interesting +# outcome is finding where each one stops paying. If A3x costs materially more +# than A3 and scores the same, then "buy a bigger model" and "buy more thinking" +# are both answers to a question most tasks were not asking. +# +# Effort reaches this endpoint in a different shape than it reaches sol, which +# matters when reading the two results side by side. On the Anthropic route the +# level selects a *thinking token budget*: medium is 8_192 and xhigh is 32_768 +# (config.rs:35), a 4x increase. On sol's Responses route it is a `reasoning. +# effort` string and the provider decides what that spends. So A3x's token +# growth is bounded by a number we can name and A2x's is not — do not read a +# larger delta on one as the model "trying harder" than the other. +# +# xhigh is genuinely available here rather than silently clamped: +# `databricks-claude-opus-5` strips to `claude-opus-5` (config.rs +# `strip_catalog_prefix` slices from the first `claude-`/`gpt-` token), which +# `anthropic_model_supports_xhigh` matches by prefix, so `clamp_adaptive_effort` +# passes it through. Worth stating explicitly because the failure mode is +# silent: an unsupported level clamps down to `high` with only a log warning, +# and the cell would report a null that was really a mislabelled `high` run. +# +# Endpoint names are exact Databricks serving-endpoint names; they resolve to +# provider/host/key via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-solo-opus-xhigh +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: claude-opus-5 + prompt: + # Byte-identical to every other solo cell. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + thinking_effort: xhigh +prices: + # Identical to tb-solo-opus. Effort moves token volume, not the rate card. + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 25.0 + cache_read_rate: 0.0 +trial_budget: + # Unchanged from tb-solo-opus — see tb-solo-sol-xhigh.yaml for why raising it + # for a higher-effort cell would convert a result into a budgeting decision. + # + # Watch this one: opus already produced 11 AgentTimeoutErrors in 89 trials at + # medium, and a 4x thinking budget can only push that up. Those trials are + # still scored on whatever reached disk, so they are results rather than + # losses — but if the count climbs sharply, "xhigh ran out of clock" is part + # of the finding and must be reported next to the score, not buried. + timeout_seconds: 36000 + +environment: + # Identical in every condition — see tb-solo-sol.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-opus.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-opus.yaml new file mode 100644 index 0000000000..414335023d --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-opus.yaml @@ -0,0 +1,98 @@ +# A3 solo baseline: the same single-agent condition as tb-solo-luna and +# tb-solo-sol, on claude-opus-5 — the study's headline frontier denominator. +# +# The question "can three cheap agents match one expensive agent" needs an +# "expensive agent" to name, and this is it. tb-solo-sol stays as a second, +# cheaper-frontier baseline, but Opus 5 is the model the headline is measured +# against. Roster shape, persona bytes, budget and clock are identical to the +# other two solos — only the endpoint moves — so any score or cost difference is +# attributable to the model. +# +# Provider routing note (this is the one thing that differs mechanically from the +# gpt-5.6 solos). Opus 5 does NOT speak the OpenAI `/responses` API that +# gpt-5.6-luna/sol use; the Databricks gateway serves it only on +# `mlflow/v1/chat/completions`, `anthropic/v1/messages` and +# `cursor/v1/chat/completions`. buzz-agent's `databricks_v2` provider already +# handles this: `databricks_v2_route_for_model` sends any model whose name +# contains `claude` to `/ai-gateway/anthropic/v1/messages` with an +# `anthropic_body` and `parse_anthropic`, so no code or manifest-schema change is +# needed — the endpoint name below is all that selects the route. Verified 200 on +# the staging box via the privatelink IP with a well-formed anthropic `usage` +# block (input / cache_read / cache_creation / output). This obsoletes the old +# "Opus needs a Bedrock endpoint" deferral in 02-benchmark-plan.md. +# +# Endpoint names are exact Databricks serving-endpoint names: the runtime passes +# the manifest endpoint name to the gateway as the model. They resolve to +# provider/host/key via testbed/endpoints/databricks-live.json, which is +# deployment config and deliberately outside this manifest. +schema_version: "1" +condition: tb-solo-opus +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: claude-opus-5 + prompt: + # Byte-identical to tb-solo-luna and tb-solo-sol. The persona is + # model- and provider-agnostic on purpose; a prompt tuned per model would + # make the baselines incomparable. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe +prices: + # Reference list prices, doc 02 §2.1 (`anthropic/claude-opus-5`). Note the + # ratio to luna is NOT a clean 5x on every axis the way sol is: input and + # cache-read are 5x luna (5.0 vs 1.0, 0.5 vs 0.1) but OUTPUT is ~4.17x + # (25.0 vs 6.0). The headline break-even math in 02 §2 must use these, not + # sol's, when Opus is the denominator. See tb-solo-luna.yaml for why + # cache_read_rate is 0.0 (DB10). + # + # Cache-write caveat, and it bites harder here than anywhere else in the study: + # the anthropic route reports `cache_creation_input_tokens` separately, and + # Anthropic bills cache writes at ~1.25x the input rate. Our accounting + # currently prices write tokens at the plain input rate (02 §7.5 residual), so + # Opus cost is understated by whatever the write volume is. Confirm before any + # published Opus dollar figure. + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 25.0 + # Fallback only; the agent reports the provider's measured cache split + # (anthropic `cache_read_input_tokens`) and the accounting path prices that, + # recording which it used (`accounting_cache_read_tokens_are_measured`). 0.0 + # keeps an unmeasured run conservative rather than inventing a discount. + cache_read_rate: 0.0 +trial_budget: + # Identical to tb-solo-luna and tb-solo-sol — the trial budget is a backstop + # above the dataset's longest task (12000s) at the 3x multiplier, so each + # task's own `[agent] timeout_sec` governs instead of a flat harness deadline. + # A different value here would mean the baselines ran on different clocks, and + # benchmark.py's check_budget_clears_clock would refuse to start if it were + # below 12000 x 3. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-high.yaml new file mode 100644 index 0000000000..ae6cc73a55 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-high.yaml @@ -0,0 +1,73 @@ +# Solo gpt-5.6-sol on the OpenAI-direct route at `thinking_effort: high` -- the +# matched control for tb-gt-sol-2terra-high. +# +# WHY THIS EXISTS. tb-gt-sol-2terra-high pins high effort on all three of its +# seats. The study already has a solo sol cell on this route +# (tb-solo-sol-openai) but it leaves effort unpinned, so scoring the team +# against it would move effort and headcount together -- the joint-delta mistake +# tb-solo-luna-openai had to be built to undo. This cell is the same solo agent +# with the same one thing changed that the team cell changed. +# +# What is held against tb-gt-sol-2terra-high: route, lead model, thinking +# effort, price sheet, trial budget, container CPU/memory, clock multiplier, +# dataset and concurrency. What moves: the two terra seats and the goosetown +# personas that coordinate them. That is the whole comparison -- at the top of +# the effort ramp, does wrapping a sol lead in two capable subordinates beat sol +# working alone? +# +# BUILT FOR LHTB, not Terminal-Bench. It runs fine on either -- nothing in the +# manifest names a dataset -- but see docs/09-lhtb-benchmark.md: run it with the +# patched `continue_until_timeout` and `--timeout-multiplier 3.0`, or the 24 +# long-horizon tasks execute single-shot and the number is not comparable to +# anything. +# +# One statistical note that does not apply to the Terminal-Bench cells: LHTB +# verifiers emit PARTIAL credit (they call `_partial_reward(details)` against a +# per-gate dict), so rewards are graded, not binary. The paired exact McNemar +# test the TB analysis rests on is the wrong tool here -- use means with a +# paired Wilcoxon signed-rank over the shared task set (`paired_w.py`). +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-solo-sol-high +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: gpt-5.6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to every other solo cell in the study. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + # Must match every seat in tb-gt-sol-2terra-high; that pairing is the only + # reason this cell exists. `gpt-5.6-sol` matches the `gpt-5.6` family + # token in config.rs, whose supported set includes high, and A2x already + # ran this endpoint one notch further at `xhigh` -- the value is neither + # rejected nor silently clamped here. + thinking_effort: high +prices: + # The same sol row tb-gt-sol-2terra-high carries, so the cost column measures + # token volume rather than a pricing edit. No terra row: there are no terra + # seats, and an unused price row is a claim the cell cannot support. + gpt-5.6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + # Fallback only. On the direct OpenAI Responses route the provider reports + # the split at usage.prompt_tokens_details.cached_tokens and the harness + # stamps accounting_cache_read_tokens_are_measured: true; 0.0 keeps an + # unmeasured run conservative rather than optimistic. + cache_read_rate: 0.0 +trial_budget: + # Identical to tb-gt-sol-2terra-high. Does not bind on LHTB anyway -- Harbor + # enforces each task's own `[agent] timeout_sec` scaled by the run's + # --timeout-multiplier. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-openai.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-openai.yaml new file mode 100644 index 0000000000..c25da833f3 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-openai.yaml @@ -0,0 +1,72 @@ +# Solo gpt-5.6-sol on the OpenAI-direct route -- the matched control for +# tb-gt-sol-2luna. +# +# WHY THIS EXISTS. tb-solo-sol is the study's sol baseline, but it runs on +# `databricks-gpt-5-6-sol`, and tb-gt-sol-2luna runs on `gpt-5.6-sol` direct +# from OpenAI. Comparing the team against tb-solo-sol would therefore move the +# lead's serving path at the same time as its headcount -- the same joint-delta +# mistake tb-solo-luna-high made and that tb-solo-luna-openai had to be built to +# undo. That one resolved to a null (route alone -0.023, p = 0.815), so the +# expectation here is also that route is inert, but "expected inert" is not +# "measured inert" and this cell costs about $30 to run. +# +# What is held against tb-gt-sol-2luna: route, lead model, price sheet, trial +# budget, container CPU/memory, clock multiplier and dataset. What moves: the +# two luna seats and the personas that coordinate them. That is the whole +# comparison -- does wrapping a sol lead in two cheap scouts beat sol alone? +# +# NO `generation` BLOCK, deliberately. None of the tb-gt-* manifests pin +# thinking_effort either, so both sides of this comparison sit at the provider +# default. Pinning `medium` here would look tidier and would add back exactly +# one difference. +# +# BUILT FOR LHTB, not Terminal-Bench. It runs fine on either -- nothing in the +# manifest names a dataset -- but the reason it exists is +# docs/09-lhtb-benchmark.md, where the team cell would otherwise produce a +# number with nothing to compare it to: LHTB's own leaderboard is off-limits +# because stock harbor ignores `continue_until_timeout` and runs the 24 +# long-horizon tasks single-shot. +# +# One statistical note that does not apply to the Terminal-Bench cells: LHTB +# verifiers emit PARTIAL credit (they call `_partial_reward(details)` against a +# per-gate dict), so rewards are graded, not binary. The paired exact McNemar +# test the TB analysis rests on is the wrong tool here -- use means with a +# paired Wilcoxon signed-rank over the shared task set. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-solo-sol-openai +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: gpt-5.6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to every other solo cell in the study. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe +prices: + # The same sol row tb-gt-sol-2luna carries, so the cost column measures token + # volume rather than a pricing edit. No luna row: there are no luna seats, and + # an unused price row is a claim the cell cannot support. + gpt-5.6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + # Fallback only. On the direct OpenAI Responses route the provider reports + # the split at usage.prompt_tokens_details.cached_tokens and the harness + # stamps accounting_cache_read_tokens_are_measured: true; 0.0 keeps an + # unmeasured run conservative rather than optimistic. + cache_read_rate: 0.0 +trial_budget: + # Identical to tb-gt-sol-2luna. Does not bind on LHTB anyway -- Harbor + # enforces each task's own `[agent] timeout_sec` scaled by the run's + # --timeout-multiplier. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-xhigh.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-xhigh.yaml new file mode 100644 index 0000000000..f5dc8f9f41 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol-xhigh.yaml @@ -0,0 +1,68 @@ +# A2x solo baseline: tb-solo-sol with reasoning effort raised medium -> xhigh. +# +# This is the effort axis (G2), and it is built the same way round as A1-vs-A2: +# change one thing. Roster shape, persona bytes, endpoint, model revision, +# prices, budget, clock and container resources are byte-identical to +# tb-solo-sol. The only difference in the whole file is `thinking_effort`, so a +# score difference between A2 and A2x is attributable to effort and nothing +# else, and a *cost* difference is attributable to the tokens effort bought. +# +# The question it answers is not "is xhigh better" but "is medium already good +# enough". Those have different shapes: the first wants a win, the second wants +# to know the size of the gap and what it cost. A null result here — xhigh +# scores the same and bills more — is the useful outcome, not a failed run, and +# is the reason the cell is worth its money. +# +# Why xhigh rather than high: the gap between medium and high is small enough +# that a null could be read as "the axis does not move anything". xhigh is the +# highest level sol supports, so if diminishing returns are real this is where +# they are most visible, and a null is hardest to dismiss. +# +# Effort genuinely reaches this endpoint. `databricks-gpt-5-6-sol` strips to +# `gpt-5-6-sol` (config.rs `strip_catalog_prefix`), matches the `gpt-5-6` family +# token, and that family's supported set includes xhigh and max — so +# `clamp_adaptive_effort` passes xhigh through untouched. This matters because +# an unsupported level is clamped down with only a log warning: the run would +# quietly execute at `high` while claiming xhigh, and the null result would be +# an artifact rather than a finding. +schema_version: "1" +condition: tb-solo-sol-xhigh +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-gpt-5-6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to tb-solo-sol and tb-solo-luna. Tuning the persona for a + # higher effort level would confound the axis with a prompt change. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + # The single line this condition exists for. Unset elsewhere, which means + # the harness default of `medium` (container_runtime.THINKING_EFFORT), so + # tb-solo-sol needs no edit. It also keeps its hash: `canonical_bytes` + # drops this field when unset, precisely so that opening the axis did not + # re-identify every condition that ignores it. + thinking_effort: xhigh +prices: + # Identical to tb-solo-sol, deliberately. Effort changes token *volume*, not + # the rate card, so holding the rates equal is what makes the A2-vs-A2x cost + # delta a pure measurement of what the extra thinking cost. + databricks-gpt-5-6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + cache_read_rate: 0.0 +trial_budget: + # Unchanged from tb-solo-sol. Note this is the field most tempting to raise + # for a higher-effort run, and raising it would be a mistake: if xhigh needs a + # longer clock to finish the same work, that is a *result* about effort, not a + # budgeting problem to engineer away. + timeout_seconds: 36000 + +environment: + # Identical in every condition — see tb-solo-sol.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol.yaml new file mode 100644 index 0000000000..465a8d0a78 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sol.yaml @@ -0,0 +1,77 @@ +# A2 solo baseline: the same single-agent condition as tb-solo-luna, on +# gpt-5.6-sol instead of gpt-5.6-luna. +# +# This is the one comparison in the study that isolates the model. Roster shape, +# persona bytes, effort, budget, clock and serving stack are all identical to +# tb-solo-luna — only the endpoint moves — so a score difference between the two +# is attributable to the model and nothing else. Every multi-agent condition is +# measured against these two, which is why they have to be built this way round: +# change one thing. +# +# Endpoint names are exact Databricks serving-endpoint names: the runtime passes +# the manifest endpoint name to the gateway as the model. They resolve to +# provider/host/key via testbed/endpoints/databricks-live.json, which is +# deployment config and deliberately outside this manifest. +schema_version: "1" +condition: tb-solo-sol +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: databricks-gpt-5-6-sol + model_revision: gpt-5.6-sol + prompt: + # Byte-identical to tb-solo-luna. The persona is model-agnostic on + # purpose; a prompt tuned per model would make the two incomparable. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe +prices: + # Operator-supplied list prices, doc 04 §7 (the same figures tb-team-sol-luna.yaml + # carries for sol). See tb-solo-luna.yaml for why cache_read_rate is 0.0 (DB10). + # + # Note sol is ~5x luna on input and ~5x on output at these rates, so the cost + # comparison between A1 and A2 is the number most sensitive to getting the real + # prices in — more so than either score. + databricks-gpt-5-6-sol: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 30.0 + # Fallback only; the agent reports the provider's measured cache split and + # the accounting path prices that, recording which it used + # (`accounting_cache_read_tokens_are_measured`). 0.0 keeps an unmeasured run + # conservative rather than inventing a discount. + cache_read_rate: 0.0 +trial_budget: + # Identical to tb-solo-luna, and it has to be: the trial budget is a backstop + # above the longest task in the dataset (12000s) at the 3x timeout multiplier, + # so each task's own `[agent] timeout_sec` governs instead of a flat harness + # deadline. A different value here would mean the two conditions ran on + # different clocks, and benchmark.py's check_budget_clears_clock would refuse + # to start if it were smaller than 12000 x 3. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sonnet.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sonnet.yaml new file mode 100644 index 0000000000..a624e9cfc7 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-sonnet.yaml @@ -0,0 +1,43 @@ +# Terminal-Bench single-agent baseline: one agent, no workers, one Buzz +# channel. This is the condition every multi-agent result is measured against, +# so the lone agent gets byte-identical wiring to a worker — same binaries, +# same MCP toolset, same env. The only difference is the persona and the fact +# that there is nobody to delegate to. +# +# Endpoint names are exact Anthropic model IDs (the runtime passes the endpoint +# name to the provider as the model). +# Prices: Anthropic list $/Mtok as of 2026-07 (Sonnet 3/15). +schema_version: "1" +condition: tb-solo-sonnet46 +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: claude-sonnet-4-6 + model_revision: claude-sonnet-4-6 + prompt: + path: personas/solo-tb.md + sha256: 5303e47d921820cc44cb8c029e41962be9649f273002c34398dd9145b224482d + +# Cache-read rates are 0.0 on every Anthropic-route endpoint on purpose: +# buzz-agent never sends `cache_control`, and Anthropic prompt caching is +# opt-in, so no cache reads occur and no discount should be modelled. Revisit +# this the moment buzz-agent starts requesting caching, and set a measured rate +# for OpenAI-route endpoints (which cache automatically) rather than a guess. +prices: + claude-sonnet-4-6: + input_per_million_usd: 3 + cached_input_per_million_usd: 0.3 + output_per_million_usd: 15 + cache_read_rate: 0.0 +trial_budget: + # 36000 = the dataset's longest task (12000s) at the 3x timeout multiplier the + # study runs with (DEFAULT_TIMEOUT_MULTIPLIER in scripts/benchmark.py). The two + # are one setting in two files: benchmark.py refuses to start if this is the + # smaller of the pair, because the harness would then cut a task short of the + # deadline Harbor granted it and record a timeout the agent never hit. A flat + # 900s did exactly that before the multiplier existed — 39 of the 89 tasks + # allow more than 900s, and one live trial died at 905s against a task budget + # of 1800s. + timeout_seconds: 36000 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-terra-high.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-terra-high.yaml new file mode 100644 index 0000000000..b567dfb91c --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-terra-high.yaml @@ -0,0 +1,82 @@ +# A5h -- solo gpt-5.6-terra at reasoning effort `high`, served direct from OpenAI. +# +# Terra is a third model tier for this study, which turns the price/performance +# curve from a line into a shape. If A1 (0.545) and A2 (0.843) bracket it, where +# terra falls says whether the tier return is roughly linear in price or whether +# one of the two steps carries nearly all of it. +# +# input cached output +# luna 0.20 0.02 1.20 (1x) +# terra 2.00 0.20 12.00 (10x luna, 0.4x sol) +# sol 5.00 0.50 30.00 (25x luna) +# +# WHEN THIS CELL RAN, terra's card was the exact midpoint of the other two on +# every axis -- luna 1.00/0.10/6.00, terra 2.50/0.25/15.00, sol 5.00/0.50/30.00 +# -- and the original text here leaned on that symmetry. The 2026-07-30 drop +# ended it: luna fell 80% and terra 20%, so the tiers are still ordered the same +# way but the gaps are now very uneven. The measurement is unchanged; only the +# overlay moved. Use the current numbers above for any ratio argument. +# +# NOTE ON THE PRICE TABLE: terra also publishes a **cache-write** rate of +# $2.50/Mtok, i.e. 1.25x its input rate. The manifest schema has no field for +# it and the accounting layer does not count write tokens separately, so every +# terra dollar figure this study reports is a slight *under*-estimate. Bounded: +# writes are the non-cached slice of input, ~23% at the 77% hit rate this fleet +# measures, so the understatement is at most ~0.25 x 0.23 = ~6% of the input +# component. Worth a footnote in the post, not worth a schema change mid-study. +# This is also the first hard confirmation of doc 02 §7.5's open question about +# whether cache writes carry a surcharge on this slate. They do. +# +# The effort axis for terra lives in tb-solo-terra-medium.yaml (A5m), which is +# this file with `thinking_effort: medium` and nothing else changed. Read A5m -> +# A5h as the effort delta and either against A1/A2 as the tier axis. See doc 02 +# §3.2. +schema_version: "1" +condition: tb-solo-terra-high +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + # Byte-identical to every other solo cell in the study. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + # Verified reaching the provider before this cell was scheduled: + # `gpt-5.6-terra` matches the `gpt-5.6` family token in config.rs, whose + # supported set includes high, and a live probe moved reasoning_tokens + # 73 -> 81 between medium and high. Effort is neither rejected nor + # silently clamped here. + thinking_effort: high +prices: + # Repriced 2026-07-30: terra 2.5/0.25/15.0 -> 2.00/0.20/12.00 (-20%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # See the cache-write note in the header: the published $2.50/Mtok write rate + # has no home in this schema. The 1.25x write-to-input ratio survived the + # drop, so the bound stated there is unchanged even though both figures moved. + gpt-5.6-terra: + input_per_million_usd: 2.0 + cached_input_per_million_usd: 0.2 + output_per_million_usd: 12.0 + # Fallback only; the direct OpenAI Responses route reports the real split + # and the harness prefers it. See tb-solo-luna-high.yaml. + cache_read_rate: 0.0 +trial_budget: + # Identical to every other cell. Terra is an unknown quantity on this + # benchmark and the temptation is to give it room; giving it more clock than + # luna or sol got would make the tier comparison meaningless. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-terra-medium.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-terra-medium.yaml new file mode 100644 index 0000000000..e1be342a21 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-solo-terra-medium.yaml @@ -0,0 +1,86 @@ +# A5m -- solo gpt-5.6-terra at reasoning effort `medium`, served direct from OpenAI. +# +# WHAT THIS ISOLATES. Byte-identical to tb-solo-terra-high in every field except +# `thinking_effort` -- same roster, same persona bytes, same endpoint, same +# revision, same prices, same budget, same container. So A5m -> A5h is a clean +# one-variable read on effort, for terra, and it is the piece +# tb-solo-terra-high's own header said was missing: +# +# "There is no terra-at-medium cell, so this manifest measures a *tier*, not +# an effort delta." +# +# Now there is. Read the pair as the effort axis; read either against A1/A2 as +# the tier axis. +# +# WHY IT MATTERS THAT THIS IS THE CHEAP END. The two effort pairs already +# measured both start from a strong model: A2 -> A2x (sol, medium -> xhigh) +# *lost* 0.006 while costing 2.6x, and A1h showed luna gaining 0.205 for 3.1x. +# So effort paid enormously at the bottom of the tier ladder and not at all at +# the top. Terra sits between them, which makes A5m -> A5h the datapoint that +# says whether that is a smooth curve or a cliff, and where the cliff is. A5h +# already scores 0.807 -- within noise of solo sol's 0.843 at a fraction of the +# price -- so if A5m lands close to A5h, the study's cheapest good answer gets +# cheaper still. +# +# EFFORT IS PINNED, NOT LEFT DEFAULT. `medium` may well be what the provider +# would pick anyway, but an unpinned cell records no effort at all in its +# manifest, so a later provider-side default change would silently redefine the +# condition and the hash would not move. Pinning costs nothing and makes the +# comparison legible from the manifest alone. +# +# The level reaches the provider rather than being clamped: `gpt-5.6-terra` +# matches the `gpt-5.6` family token in config.rs, and the live probe that +# qualified A5h moved reasoning_tokens 73 -> 81 between medium and high -- i.e. +# medium was the measured baseline of that probe, so it is known-good by +# construction. +# +# PRICES. The 2026-07-30 sheet, same as every other terra cell: +# +# input cached output +# luna 0.20 0.02 1.20 (1x) +# terra 2.00 0.20 12.00 (10x luna, 0.4x sol) +# sol 5.00 0.50 30.00 (25x luna) +# +# Note terra is no longer the tidy midpoint tb-solo-terra-high's header +# describes -- that text predates the drop, which cut luna 80% and terra 20%. +# The three tiers are still ordered the same way, but the gaps are now very +# uneven, so any "price/performance curve" reading has to use these numbers +# rather than the old ones. +# +# Endpoint names resolve via testbed/endpoints/openai-live.json. +schema_version: "1" +condition: tb-solo-terra-medium +roster: + - id: solo + kind: orchestrator + role: solo + count: 1 + endpoint: gpt-5.6-terra + model_revision: gpt-5.6-terra + prompt: + # Byte-identical to every other solo cell in the study. + path: personas/bench/solo.md + sha256: 61c9ac9dc47cbd3c987a7ee6ceb7397496834df8d3fc284c8fe6980e1d5b4ebe + generation: + thinking_effort: medium +prices: + # Identical to tb-solo-terra-high. Effort moves token volume, not the rate + # card. The published $2.50/Mtok cache-WRITE rate still has no home in this + # schema, so this cell is understated by the same bounded amount as A5h -- + # equally, so the effort comparison is unaffected. + gpt-5.6-terra: + input_per_million_usd: 2.0 + cached_input_per_million_usd: 0.2 + output_per_million_usd: 12.0 + # Fallback only; the direct OpenAI Responses route reports the real split + # and the harness prefers it. See tb-solo-luna-high.yaml. + cache_read_rate: 0.0 +trial_budget: + # Identical to A5h. A lower-effort cell needs less clock, not more, so this + # cannot bind here in any way it did not already bind there. + timeout_seconds: 36000 + +environment: + # Identical in every condition -- see tb-solo-luna.yaml for the full reasoning. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-team-3luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-team-3luna.yaml new file mode 100644 index 0000000000..fc7f59b37d --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-team-3luna.yaml @@ -0,0 +1,110 @@ +# C2 — three gpt-5.6-luna agents in a hierarchy: one delegating lead and two +# implementers. The topology control for tb-triad-3luna. +# +# This is the cleanest prompt-scheme experiment in the study. Against C1 it holds +# the model, the agent count, the serving path, the price sheet and the trial +# budget fixed and varies exactly one thing: how the three agents relate to each +# other. Peers who argue, versus a lead who assigns. Any score or cost delta +# between C1 and C2 is attributable to the persona bytes, which is a result the +# multi-agent literature mostly asserts rather than measures. +# +# It also answers the cheaper objection to C1 directly. If C1 beats the solo +# baselines and C2 does too, by a similar margin, then the win is headcount and +# the peer protocol is decoration. If C1 wins and C2 does not, the protocol is +# doing the work. +# +# Roster shape is deliberately the same as tb-team-sol-luna.yaml (lead + +# 2x implementer, count: 2) so the two are comparable across the model change: +# C2 is that condition with the expensive lead swapped for a cheap one. The lead +# accumulates the most context in a hierarchy — it reads every report — so +# putting a cheap model in that seat is the specific bet this condition tests. +# +# `count: 2` produces agent ids `worker-1` and `worker-2` (container_runtime +# splits ids on the trailing index), and both receive the same persona bytes. +# lead-delegate.md already handles this shape: its rule 4 assigns the task's own +# success check to the worker that did *not* do the work, which is a free +# verification pass that C1 pays for with a third persona. +# +# No critic here on purpose. Adding one would make C2 differ from C1 in both +# topology and headcount-of-verifiers, and then neither comparison isolates +# anything. +# +# Endpoint names are exact Databricks serving-endpoint names; provider/host/key +# resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-team-3luna +roster: + - id: orch + kind: orchestrator + # `lead` is read verbatim by worker.md: "Your lead is the teammate whose Role + # column reads `lead`". Renaming it means every report @mentions nobody, the + # sends all report success, and the trial stalls to its timeout with no error. + role: lead + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + # Byte-identical to the file tb-team-sol-luna.yaml pins, so the lead's + # instructions are not a variable between the two team conditions. + path: personas/bench/lead-delegate.md + sha256: 0dae4944765fab10cd5b1d3adcb1ef9c789384e52060acb140ab20f9c3f4686b + + - id: worker + kind: worker + role: implementer + count: 2 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/worker.md + sha256: 6c8525f48ac71ed875d3de3fc867cc1ddca885d433ad3fc0aba09b3eb2607b45 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Operator-supplied list prices, doc 04 §7, identical to every other luna + # condition. See tb-triad-3luna.yaml for why holding these equal keeps the + # cross-condition cost ratios sound, and why cache_read_rate is 0.0 (DB10). + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only; the agent reports the provider's measured cache split and + # the accounting path prices that instead, flagging which of the two it + # used. 0.0 keeps a fallback conservative rather than inventing a discount. + cache_read_rate: 0.0 +trial_budget: + # Identical to every other condition in the comparison set. See + # tb-triad-3luna.yaml. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-team-opus-luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-team-opus-luna.yaml new file mode 100644 index 0000000000..e8d7442f55 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-team-opus-luna.yaml @@ -0,0 +1,114 @@ +# C3 — the hybrid steelman: one claude-opus-5 lead delegating to two +# gpt-5.6-luna workers, over one Buzz channel. +# +# This is the topology most people actually build — "smart expensive brain, +# cheap hands" — so keeping one cell of it means the study argues against the +# real alternative, not a straw one. Against the all-luna team (tb-team-3luna) +# it isolates what the expensive lead buys; against solo Opus (tb-solo-opus) it +# asks whether wrapping the frontier model in a team of cheap workers beats just +# running the frontier model alone. +# +# The lead accumulates the most context in a hierarchy — it reads every worker +# report — so this puts the expensive model in the single most token-heavy seat. +# That is deliberate: it is the honest, unfavourable-to-our-thesis version of the +# hybrid, and if the all-luna team still competes with it on $/solved-task, that +# is the result worth publishing. +# +# Mixed-model, so both price rows are load-bearing and the two provider ROUTES +# differ: the luna workers go to the OpenAI `/responses` gateway path, the Opus +# lead to `/ai-gateway/anthropic/v1/messages` (Opus 5 does not speak +# `/responses`). buzz-agent's `databricks_v2` provider selects the route per +# model automatically from the endpoint name — see tb-solo-opus.yaml for the +# detail. Nothing here needs to know about it; the endpoint names do the work. +# +# Replaces the earlier Sol-lead hybrid (tb-team-sol-luna.yaml, condition +# tb-team-sol-2luna): the study now uses Opus 5 as the expensive model in every +# mixed setting, keeping Sol only as a solo baseline (tb-solo-sol). +# +# Endpoint names are exact Databricks serving-endpoint names; provider/host/key +# resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-team-opus-2luna +roster: + - id: orch + kind: orchestrator + # `lead` is read verbatim by worker.md: "Your lead is the teammate whose Role + # column reads `lead`". Renaming it means every report @mentions nobody, the + # sends all report success, and the trial stalls to its timeout with no error. + role: lead + count: 1 + endpoint: databricks-claude-opus-5 + model_revision: claude-opus-5 + prompt: + # Byte-identical to the file tb-team-3luna.yaml pins, so the lead's + # instructions are not a variable between the all-luna and hybrid team + # conditions — only the lead's model is. + path: personas/bench/lead-delegate.md + sha256: 0dae4944765fab10cd5b1d3adcb1ef9c789384e52060acb140ab20f9c3f4686b + + - id: worker + kind: worker + role: implementer + count: 2 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/worker.md + sha256: 6c8525f48ac71ed875d3de3fc867cc1ddca885d433ad3fc0aba09b3eb2607b45 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Reference list prices — Opus from doc 02 §2.1 (`anthropic/claude-opus-5`), + # luna the same figures every other luna condition carries. Opus is 5x luna on + # input and cache-read but ~4.17x on output; see tb-solo-opus.yaml for the + # cache-write (cache_creation) surcharge caveat, which matters most on the + # lead's high-context seat. See tb-solo-luna.yaml for why cache_read_rate is + # 0.0 (DB10). + databricks-claude-opus-5: + input_per_million_usd: 5.0 + cached_input_per_million_usd: 0.5 + output_per_million_usd: 25.0 + cache_read_rate: 0.0 + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + cache_read_rate: 0.0 +trial_budget: + # Identical to every other condition in the comparison set (12000s longest task + # x 3x multiplier). benchmark.py's check_budget_clears_clock refuses to start if + # this drops below that product; a different value would mean this condition ran + # on a different clock than the baselines it is measured against. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/manifests/tb-triad-3luna.yaml b/benchmarks/harbor-buzz-orchestra/manifests/tb-triad-3luna.yaml new file mode 100644 index 0000000000..c4443553c8 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/manifests/tb-triad-3luna.yaml @@ -0,0 +1,140 @@ +# C1 — three gpt-5.6-luna agents on one channel: two peers plus an independent +# critic. The study's headline team. +# +# The question this condition exists to answer: can three cheap agents talking to +# each other match one expensive agent working alone, for less money? Against +# tb-solo-sol it is the whole thesis; against tb-solo-luna it says what the extra +# two agents bought; against tb-peer-2luna it isolates the marginal value of the +# *third* agent, since B1 is this roster minus the critic and nothing else moves. +# +# Topology, and why this one. The driver owns the keyboard, the navigator argues +# before decisions are made, and the critic checks only finished work — once, +# after the navigator has already agreed. Splitting "argue" from "verify" across +# two agents is the point: a navigator that both plans and signs off has an +# incentive to approve its own reasoning, which is the failure this shape is +# built to avoid. +# +# The driver is the orchestrator because the roster allows exactly one, and +# because the driver is the agent the harness wakes with the task and the only +# one permitted to publish `DONE:`. +# +# Read-only on the navigator and the critic is enforced by persona, not by +# permission — all three agents get the same tools and the same filesystem. That +# boundary is therefore a finding to check in the transcripts, not an assumption +# to rely on. With three agents on one filesystem, a write from the wrong one is +# the most likely way this condition fails for a reason unrelated to the models. +# +# Expect it to cost more per task than solo luna: three agents, and every handoff +# re-sends context. That is the measurement. The break-even that matters is +# against sol, not against luna — at these prices sol is 5x luna on both axes, so +# this roster may burn up to 5x the tokens of tb-solo-sol and still win on cost. +# +# Endpoint names are exact Databricks serving-endpoint names; provider/host/key +# resolve via testbed/endpoints/databricks-live.json. +schema_version: "1" +condition: tb-triad-3luna +roster: + - id: driver + kind: orchestrator + # Read verbatim out of the composed "Your team" table by both other + # personas: triad-navigator addresses `driver`, and triad-driver tells the + # critic who woke it. Renaming any role in this file silently breaks the + # trial — the @mention resolves to nobody, the send still reports success, + # and the run stalls to its timeout with no error anywhere. + role: driver + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + # Derived from personas/bench/peer-driver.md rather than reusing it: that + # file says "You and one peer" and knows nothing about a critic, so a + # third agent would be invisible to it. Editing it in place would have + # rewritten tb-peer-2luna's hash and orphaned B1's results, so the triad + # gets its own file and the pair keeps its own. + path: personas/bench/triad-driver.md + sha256: b0bff7a8327e52c3ae4ec962b9619b6260e9da3aa2b86188068c732382ad2799 + + - id: navigator + kind: worker + role: navigator + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + path: personas/bench/triad-navigator.md + sha256: 02e0029f25fc80f47fefa12304dff7eb11e0561f9eadcd7a34b1188f11841dc5 + + - id: critic + kind: worker + role: critic + count: 1 + endpoint: databricks-gpt-5-6-luna + model_revision: gpt-5.6-luna + prompt: + # Byte-identical to the file tb-team-* conditions use. critic.md is + # already role-agnostic — it replies to "whoever asked for the review, the + # teammate named in the message that woke you" and never assumes a lead — + # so it drops into a peer topology unchanged. Keeping it identical means + # C1-versus-C2 varies the two peer personas and nothing else. + path: personas/bench/critic.md + sha256: 4a5c6273c2cbd2cae593ff1a46117334e0cca7bc913a51eedb5e1d39ea755743 + +prices: + # Repriced 2026-07-30: luna 1.0/0.1/6.0 -> 0.20/0.02/1.20 (-80%). + # Sol, Opus 5, Sonnet and Haiku did not move. This changes the + # condition hash, so a cell that ran before this date carries the old + # hash and the old rates in its receipts -- that mismatch is expected, + # not corruption. Restate a completed cell's cost with + # benchmark-runs/tools/reprice.py, which re-prices the measured tokens + # at a new sheet. Never edit a receipt: the tokens are the + # measurement, the price sheet is only an overlay on them. + # Operator-supplied list prices, doc 04 §7, identical to tb-solo-luna's and + # tb-peer-2luna's luna figures — holding them equal across all luna conditions + # keeps the solo/pair/triad cost *ratios* sound. The sol-versus-luna ratio is + # the single number the headline is most sensitive to. See tb-solo-luna.yaml + # for why cache_read_rate is 0.0 (DB10). + databricks-gpt-5-6-luna: + input_per_million_usd: 0.2 + cached_input_per_million_usd: 0.02 + output_per_million_usd: 1.2 + # Fallback only, and it should never fire on this condition. The agent reports + # the provider's real cache split on its usage line and the accounting path + # prices the measured count, recording + # `accounting_cache_read_tokens_are_measured: true`. Cache matters more here + # than in any solo condition, because a team's re-sent context is exactly what + # a prefix cache serves. + cache_read_rate: 0.0 +trial_budget: + # Matches tb-solo-luna, tb-solo-sol and tb-peer-2luna exactly. A triad genuinely + # needs longer than a solo agent — every handoff is a round trip — but granting + # it a larger budget would confound the comparison with the thing being + # measured. It does not bind anyway: Harbor enforces each task's own + # `[agent] timeout_sec`, which tops out at 12000s across Terminal-Bench 2.1, + # scaled by the study's 3x multiplier. All conditions therefore run against the + # task's own clock, which is the only budget they can honestly share. + timeout_seconds: 36000 + +environment: + # Identical in every condition, including the solo baselines. Harbor enforces + # a task's declared resources as hard Docker limits, and the Buzz stack runs + # *inside* that container, so a 3-agent roster would otherwise fit three + # buzz-acp/buzz-agent/buzz-dev-mcp process groups plus the task's own work + # into the 1 vCPU and 2 GB most Terminal-Bench tasks ask for. A team that + # thrashes there has been penalised for memory pressure rather than for + # coordinating badly — a confound in precisely what this study measures. + # + # Raising it only for the team tiers would trade that confound for a worse + # one, so the solos carry it too and every cell is measured on identical + # hardware. + # + # 4 and 8192 are the *maxima* the dataset declares (cpus: 83x1, 3x2, 3x4; + # memory_mb: 68x2048, 13x4096, 8x8192), and that is not a coincidence. Harbor + # replaces rather than raises — environments/base.py assigns + # `task_env_config.cpus = override_cpus` outright — so any smaller value would + # quietly shrink the 8 tasks that ask for 8 GB and the 3 that ask for 4 CPUs. + # Every task therefore gets at least what it requested, and most get more. + # + # storage_mb is deliberately absent: all 89 tasks already declare 10240, so an + # override would restate the status quo and add a knob with nothing to say. + override_cpus: 4 + override_memory_mb: 8192 diff --git a/benchmarks/harbor-buzz-orchestra/patches/apply_continue_until_timeout.py b/benchmarks/harbor-buzz-orchestra/patches/apply_continue_until_timeout.py new file mode 100644 index 0000000000..c071c55655 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/patches/apply_continue_until_timeout.py @@ -0,0 +1,704 @@ +#!/usr/bin/env python3 +"""Teach an installed Harbor 0.16.x to honor LHTB's ``continue_until_timeout``. + +Why this exists +--------------- +30 of LHTB's 46 tasks set ``continue_until_timeout = true`` in ``[agent]``. +Stock Harbor drops the field on the floor (Pydantic ignores unknown keys), so +those tasks run single-shot: the agent emits ``DONE:``/``task_complete``, the +trial ends, and whatever happens to be on disk is scored. Our first full EC2 +run showed exactly what that costs -- 24 of 24 trials self-reported success at +a mean of 7.8 minutes against a multi-hour budget, and scored 0 completions. + +With the flag honored, the trial runs an interim verifier after the agent +stops; if the reward is short of 1.0 the agent is re-invoked in the same +container with the verifier feedback appended, until it passes or the agent +timeout expires. + +Why a script and not ``git apply`` +---------------------------------- +The reference diff (``LHTB/harbor/patches/continue-until-timeout.patch``) is +written against Harbor 0.7.0 and does not apply to 0.16.1 -- the trial class +was split into ``Trial`` / ``SingleStepTrial`` / ``MultiStepTrial`` and the +agent invocation moved into ``Trial._run_agent_phase`` behind a stack of +context managers. More importantly the orchestra consumes Harbor as a *wheel* +from artifactory, not a checkout, so the thing that needs editing is +``site-packages`` on each benchmark box. This script finds that install, +applies both edits by exact-string replacement, and is safe to re-run. + +Where the loop lives +-------------------- +``SingleStepTrial._run_agent``, not ``Trial._run_agent_phase``. Patching the +phase helper would mean re-entering its network-policy / default-user / log +context managers per iteration and hand-merging the ``AgentContext`` it resets +on every call. Driving it from the caller keeps the loop readable and leaves +the phase helper's contract untouched. All 46 LHTB tasks are single-step with +a shared verifier, so ``MultiStepTrial`` is deliberately left alone. + +PER-PHASE RECEIPTS -- broken 2026-07-30, FIXED 2026-07-30 (and the first diagnosis +here was wrong) +---------------------------------------------------------------------------------- +Symptom: ``receipts.jsonl`` under-reported a multi-phase trial badly. Measured on +``lhtb46-team``: ``$1.76`` in receipts against ``$9.90`` in ``agent_result`` on +``grammar-fuzz-coverage-hunt`` (5.6x), and ``$0.03`` against ``$2.05`` on +``langchain-version-migration`` (63x). + +**The first diagnosis written here blamed ``_sync_agent_output``, and it was +wrong.** That function is only ``_download_agent_logs()`` + +``_populate_agent_context()``; it does not touch receipts, and it could not be +made to -- ``_download_agent_logs`` early-returns on +``self._are_agent_logs_downloaded``, so calling it per phase (the fix originally +prescribed here) would have been a no-op after phase 1 and would have "fixed" +nothing while looking correct. + +The actual writer is ``BuzzContainerRuntime.run()``, one level down. Its own +``finally`` calls ``bundle.write()`` -> ``accounting.write_receipts()``, which +opens ``receipts.jsonl`` with mode ``"w"``. ``run()`` is the agent invocation, so +it executes **once per phase**, truncating the same path every time: last phase +wins. The per-phase data was always being produced correctly and then thrown away +by the next phase, which is why the survivor is a plausible-looking small number +rather than an obviously broken one. + +The fix, in the loop below: move each phase's file aside to +``receipts.phase-NN.jsonl`` as soon as that phase returns (in a ``finally``, since +the deadline phase raises *after* its receipts are written), then rebuild +``receipts.jsonl`` in the trial's ``finally`` as all phases concatenated. +Concatenating rather than only numbering matters -- every analysis tool reads +``receipts.jsonl`` and sums its rows, so deleting it would turn this into a +silent ``$0``, and summing rows per seat is already how those tools aggregate. +``agent_result.metadata`` also gains ``receipts_phase_files`` and +``receipts_cost_usd`` so the two independent accountings can be asserted equal +rather than chosen between. + +**Runs made BEFORE this fix (the 46-task LHTB wave on h1/h2) still need the +inverted rule: score their cost and tokens from ``agent_result``.** Receipts there +hold one phase of many; the per-seat split is unrecoverable on those trials except +where a trial happened to run a single phase. On an *unpatched* run the original +rule still holds -- receipts are authoritative, because a timed-out trial has no +``agent_result`` at all. See docs/02 SS3.1 and docs/09. + +Two things measured on the real h1/h2 trials while fixing this, both of which +contradict a guess that was easy to make here: + +1. **Most trials leave this loop CLEANLY, with no exception.** The ``remaining <= + 0`` checks break the loop, so the last phase returned normally and its usage + *was* merged. 4 of 5 sampled team trials had no ``exception_info`` at all + despite running 15-194 phases. So ``agent_result`` really is complete on the + common path, and the "agent_result is missing its last phase" worry applies + only to the ``AgentTimeoutError`` minority. +2. **On that minority the final phase's usage exists NOWHERE, and receipts read + $0.00 rather than missing.** ``epidemic-inverse-control-audit``: 79 phases, + ``AgentTimeoutError``, ``agent_result`` $5.45, receipts $0.00 across 3 rows. + ``BuzzContainerRuntime._settle_usage`` runs only on its success path, so a + cancelled phase's tokens are never flushed before teardown. The loop below + therefore refuses to merge a zero-cost recovered phase and records it in + ``agent_result.metadata.unsettled_phases`` instead -- such a trial's cost is a + floor. Do NOT reconstruct these as ``agent_result + receipts``; on that path + the receipts are the zero, not the missing piece. + +Not fixed, and deliberately out of scope: ``summary.json`` beside the receipts is +still last-phase-only, for the same truncating reason. Nothing in the analysis +path reads it. Nor is the unsettled-usage hole itself -- closing it means flushing +usage on the cancellation path inside ``BuzzContainerRuntime.run``, which is a +change to the runtime rather than to this patch. + +Usage: python3 apply_continue_until_timeout.py [--check] [--harbor-root DIR] +""" + +from __future__ import annotations + +import argparse +import importlib.util +import sys +from pathlib import Path + +MARKER = "continue_until_timeout" + +# --- edit 1: the config field ------------------------------------------------- + +CONFIG_ANCHOR = """class AgentConfig(PhaseNetworkPolicyConfig): + timeout_sec: float | None = None + user: str | int | None = Field( + default=None, + description="Username or UID to run the agent as. None uses the environment's default USER (e.g., root).", + ) +""" + +CONFIG_PATCHED = ( + CONFIG_ANCHOR + + """ continue_until_timeout: bool = Field( + default=False, + description=( + "When true, the trial keeps restarting the agent after early " + "completion or failed interim verification until agent.timeout_sec " + "elapses or the verifier passes. An agent declaring itself done " + "does not end the trial." + ), + ) +""" +) + +# --- edit 2: imports on single_step.py --------------------------------------- + +IMPORT_ANCHOR = """from typing import override +import asyncio +""" + +IMPORT_PATCHED = """from typing import override +import asyncio +import json +import time +from pathlib import Path + +from harbor.models.agent.context import AgentContext +from harbor.models.verifier.result import VerifierResult +""" + +# --- edit 3: helpers + the phase loop ---------------------------------------- + +HELPERS = ''' + +def _merge_agent_contexts( + accumulated: "AgentContext | None", phase: "AgentContext" +) -> "AgentContext": + """Sum per-phase agent metrics into one context. + + ``Trial._run_agent_phase`` assigns a fresh ``AgentContext`` to the target on + every call, so without this each phase would clobber the last and a trial + that ran six phases would report the tokens and cost of only the sixth. + """ + if accumulated is None: + return phase.model_copy(deep=True) + + def _sum_int(a: int | None, b: int | None) -> int | None: + if a is None and b is None: + return None + return (a or 0) + (b or 0) + + def _sum_float(a: float | None, b: float | None) -> float | None: + if a is None and b is None: + return None + return (a or 0.0) + (b or 0.0) + + merged_rollouts: list = [] + if accumulated.rollout_details: + merged_rollouts.extend(accumulated.rollout_details) + if phase.rollout_details: + merged_rollouts.extend(phase.rollout_details) + + merged_metadata: dict = dict(accumulated.metadata or {}) + for key, value in (phase.metadata or {}).items(): + if key == "n_episodes" and isinstance(value, int): + prior = merged_metadata.get("n_episodes") + merged_metadata["n_episodes"] = ( + int(prior) + value if isinstance(prior, int) else value + ) + else: + merged_metadata[key] = value + + return AgentContext( + n_input_tokens=_sum_int(accumulated.n_input_tokens, phase.n_input_tokens), + n_cache_tokens=_sum_int(accumulated.n_cache_tokens, phase.n_cache_tokens), + n_output_tokens=_sum_int(accumulated.n_output_tokens, phase.n_output_tokens), + cost_usd=_sum_float(accumulated.cost_usd, phase.cost_usd), + rollout_details=merged_rollouts or None, + metadata=merged_metadata or None, + ) + + +def _verifier_passed(result: "VerifierResult | None") -> bool: + if result is None or not result.rewards: + return False + try: + return float(result.rewards.get("reward", 0)) >= 1.0 + except (TypeError, ValueError): + return False + + +def _context_from_receipts(path: Path) -> "AgentContext | None": + """Rebuild one phase's AgentContext from the receipts that phase wrote. + + Needed because a phase cancelled at the deadline never reports its usage. + ``_run_agent_phase`` hands the agent a fresh empty ``AgentContext`` and + ``BuzzOrchestraAgent.run`` copies totals into it only *after* + ``runtime.run()`` returns -- and ``asyncio.wait_for`` cancels that coroutine + on timeout, so the copy never happens. The spend is real and did get written, + by ``BuzzContainerRuntime.run``'s own ``finally``, to that phase's receipts. + Reading it back from there is the only way ``agent_result`` can include the + phase that spent the budget, which for this loop is the EXPECTED ending. + + Returns None if there is nothing usable, never a zeroed context: a zero would + merge in silently and read as "this phase was free". + """ + try: + rows = [ + json.loads(line) + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + except Exception: + return None + if not rows: + return None + + def total(key: str) -> int: + return sum(int(row.get(key) or 0) for row in rows) + + return AgentContext( + n_input_tokens=total("input_tokens"), + n_cache_tokens=total("estimated_cache_read_tokens"), + n_output_tokens=total("output_tokens"), + cost_usd=sum(float(row.get("cost_usd") or 0.0) for row in rows), + ) + + +def _sum_receipt_cost(path: Path) -> float | None: + """Total cost_usd across a receipts file, or None if it cannot be read. + + Used only to cross-check the rebuilt receipts against ``agent_result``. + Returns None rather than 0.0 on any failure: a zero here would be + indistinguishable from a genuinely free trial, and reading absent data as + zero is the exact bug this whole patch section exists to undo. + """ + try: + total = 0.0 + for line in path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + total += float(json.loads(line).get("cost_usd") or 0.0) + return total + except Exception: + return None + + +def _read_verifier_feedback(verifier_dir: Path) -> str: + """Best-effort human-readable reason the interim verifier was unhappy. + + LHTB verifiers emit graded partial credit, so the reward alone is real + signal to the agent -- it says how close the last attempt got. + """ + details_path = verifier_dir / "migration_details.json" + if details_path.exists(): + try: + data = json.loads(details_path.read_text()) + failure = data.get("failure") + if isinstance(failure, str) and failure.strip(): + return failure.strip() + return json.dumps(data, indent=2)[:4000] + except Exception: + pass + + reward_path = verifier_dir / "reward.txt" + if reward_path.exists(): + try: + return ( + f"Verifier reward so far: {reward_path.read_text().strip()} " + "(1.0 is a full pass)." + ) + except OSError: + pass + + return "Verification did not fully pass." + + +def _format_continue_instruction( + base_instruction: str, *, phase: int, remaining_sec: int, feedback: str +) -> str: + return ( + f"{base_instruction.rstrip()}\\n\\n" + "---\\n" + "## VERIFICATION DID NOT PASS - CONTINUE WORKING\\n\\n" + f"Your previous attempt did not fully pass verification (phase {phase}). " + f"You have approximately {remaining_sec} seconds left in this trial.\\n\\n" + "**Do not stop.** This trial ends only at timeout or full verifier " + "success. Your shell session and files are exactly as you left them - " + "keep improving the solution.\\n\\n" + "Verifier feedback:\\n" + f"{feedback.strip()}\\n" + ) + +''' + +RUN_AGENT_ANCHOR = """ async def _run_agent(self) -> None: + try: + await self._run_agent_phase( + target=self.result, + instruction=self.task.instruction, + timeout_sec=self._agent_timeout_sec, + user=self.task.config.agent.user, + ) + except (AgentTimeoutError, NonZeroAgentExitCodeError) as exc: + self._record_exception(exc) + finally: + await self._sync_agent_output(self.result) +""" + +RUN_AGENT_PATCHED = ''' async def _run_interim_verifier(self) -> VerifierResult: + """Run the verifier mid-trial purely to decide whether to continue.""" + mode = resolve_task_verifier_mode(self.task.config) + user = self.task.config.verifier.user + if mode == VerifierEnvironmentMode.SEPARATE: + return await self._run_separate_verifier( + key="interim", + timeout_sec=self._verifier_timeout_sec, + artifacts_dir=self.paths.artifacts_dir, + user=user, + ) + return await self._run_shared_verifier( + timeout_sec=self._verifier_timeout_sec, + user=user, + ) + + async def _hide_shared_verifier_tests(self) -> None: + """Delete the verifier tests before handing control back to the agent. + + In shared-environment mode the interim verifier uploads the real tests + into the same container the agent is about to resume in. Leaving them + there would let the agent read the grader and write to it, which is + cheating that we would then measure as skill. + """ + if resolve_task_verifier_mode(self.task.config) == ( + VerifierEnvironmentMode.SEPARATE + ): + return + try: + await self.agent_environment.reset_dirs( + remove_dirs=[self.agent_env_paths.tests_dir], + create_dirs=[], + ) + except Exception as exc: + self.logger.warning( + f"Failed to remove shared verifier tests before resuming: {exc}" + ) + + async def _run_agent(self) -> None: + agent_cfg = self.task.config.agent + continue_until = ( + getattr(agent_cfg, "continue_until_timeout", False) + and self._agent_timeout_sec is not None + ) + + if not continue_until: + try: + await self._run_agent_phase( + target=self.result, + instruction=self.task.instruction, + timeout_sec=self._agent_timeout_sec, + user=agent_cfg.user, + ) + except (AgentTimeoutError, NonZeroAgentExitCodeError) as exc: + self._record_exception(exc) + finally: + await self._sync_agent_output(self.result) + return + + # continue_until_timeout: loop agent -> verify -> agent until the + # verifier fully passes or the agent budget is spent. Reaching the + # deadline is the EXPECTED exit here, not a failure; the exception is + # still recorded (stock behaviour) but _run() goes on to score the + # trial normally, so a timed-out trial yields a reward, not a loss. + deadline = time.monotonic() + self._agent_timeout_sec + base_instruction = self.task.instruction + instruction = base_instruction + phase = 0 + aggregated: AgentContext | None = None + first_started_at = None + receipts_path = self.paths.agent_dir / "buzz" / "receipts.jsonl" + phase_receipts: list[Path] = [] + unsettled_phases: list[int] = [] + + def keep_phase_receipts(index: int) -> "Path | None": + """Move this phase's receipts aside before the next phase truncates them. + + BuzzContainerRuntime.run() ends EVERY phase by calling bundle.write(), + which calls accounting.write_receipts(), which opens receipts.jsonl + with mode "w". One truncating write per phase at one path: absent this, + the only receipts that survive a multi-phase trial are the last + phase's. Measured on lhtb46-team before the fix -- + grammar-fuzz-coverage-hunt reported $1.76 against agent_result's $9.90 + (5.6x low), langchain-version-migration $0.03 against $2.05 (63x). + """ + if not receipts_path.exists(): + return None + try: + kept = receipts_path.with_name(f"receipts.phase-{index:02d}.jsonl") + receipts_path.replace(kept) + phase_receipts.append(kept) + return kept + except OSError as exc: + # Never let bookkeeping fail a trial that actually ran. + self.logger.warning(f"could not keep phase {index} receipts: {exc}") + return None + + try: + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + + try: + await self._run_agent_phase( + target=self.result, + instruction=instruction, + timeout_sec=remaining, + user=agent_cfg.user, + ) + finally: + # In a finally, because the phase that runs out the deadline + # raises AFTER the runtime's own finally has already written + # its receipts -- and that phase is this loop's expected + # ending, so the exception path is the one that most needs + # them kept. + kept_path = keep_phase_receipts(phase + 1) + phase_ctx = self.result.agent_result + if kept_path is not None and ( + phase_ctx is None or phase_ctx.is_empty() + ): + # An empty context means this phase was cancelled before + # it could report -- see _context_from_receipts. Merged + # here rather than after the try, because the normal merge + # below is skipped by the exception. Guarded on is_empty() + # so a phase that DID report is never counted twice. + recovered = _context_from_receipts(kept_path) + if recovered is not None and (recovered.cost_usd or 0.0) > 0.0: + aggregated = _merge_agent_contexts(aggregated, recovered) + else: + # Receipts exist but price the phase at zero, which + # measured real on lhtb46-team/epidemic-inverse- + # control-audit: $5.45 in agent_result, $0.00 across + # 3 receipt rows. BuzzContainerRuntime._settle_usage + # runs only on its success path -- "a timeout has no + # usage to flush" -- so a cancelled phase's tokens + # were never flushed before teardown and exist + # NOWHERE. Merging that zero would quietly assert the + # phase was free; recording it instead marks the + # trial's cost as a floor. Do not "fix" this by + # summing agent_result + receipts: on this path the + # receipts are the zero, not the missing piece. + unsettled_phases.append(phase + 1) + # _run_agent_phase stamps a FRESH TimingInfo on every call, so + # the first phase's start has to be kept and put back at the + # end. Without it the trial reports only the last phase's + # duration -- a trial that worked for three hours can land in + # the results as 1.1 seconds, quietly destroying every + # agent-hours and $/agent-hour figure derived from it. + if first_started_at is None: + first_started_at = self.result.agent_execution.started_at + + aggregated = _merge_agent_contexts(aggregated, self.result.agent_result) + self.result.agent_result = aggregated + + remaining = deadline - time.monotonic() + if remaining <= 0: + break + + interim: VerifierResult | None = None + verifier_timed_out = False + try: + interim = await self._run_interim_verifier() + except asyncio.TimeoutError: + verifier_timed_out = True + finally: + await self._hide_shared_verifier_tests() + + if not verifier_timed_out and _verifier_passed(interim): + break + + phase += 1 + feedback = ( + "The interim verifier timed out. Keep improving the solution." + if verifier_timed_out + else _read_verifier_feedback(self.paths.verifier_dir) + ) + instruction = _format_continue_instruction( + base_instruction, + phase=phase, + remaining_sec=max(0, int(remaining)), + feedback=feedback, + ) + self.logger.info( + f"continue_until_timeout: phase {phase} starting, " + f"{remaining:.0f}s remaining" + ) + except (AgentTimeoutError, NonZeroAgentExitCodeError) as exc: + self._record_exception(exc) + finally: + # Restore the start here rather than per iteration: the phase that + # runs out the deadline raises *after* stamping its own TimingInfo, + # so a per-iteration restore leaves the last phase's start in the + # result -- which is the common case, since spending the whole + # budget is this loop's expected ending. + if first_started_at is not None and self.result.agent_execution: + self.result.agent_execution.started_at = first_started_at + + # Rebuild receipts.jsonl as every phase concatenated. Leaving only + # the numbered per-phase files would be WORSE than the bug this + # fixes: every analysis tool reads receipts.jsonl and sums its rows, + # so an absent file reads as "$0 spent" rather than "data missing" -- + # the same absent-means-zero trap as docs/02 SS3.1. Concatenation is + # the right shape because a seat's rows already sum across phases, + # which is how every one of those tools aggregates. + if phase_receipts: + try: + with receipts_path.open("w", encoding="utf-8") as handle: + for part in phase_receipts: + handle.write(part.read_text(encoding="utf-8")) + receipts_path.chmod(0o600) + except OSError as exc: + self.logger.warning(f"could not rebuild receipts.jsonl: {exc}") + + if aggregated is not None: + if aggregated.metadata is None: + aggregated.metadata = {} + aggregated.metadata["continue_until_timeout_phases"] = phase + # Recorded so a later analysis can ASSERT agreement instead of + # picking a favourite field. receipts and agent_result are built + # from different sources -- priced usage rows scraped from the + # container versus the context the agent itself returned -- so a + # wide gap means one of them is wrong and the run needs looking + # at, not averaging. A None cost means the file was unreadable. + aggregated.metadata["receipts_phase_files"] = len(phase_receipts) + aggregated.metadata["receipts_cost_usd"] = _sum_receipt_cost( + receipts_path + ) + # Non-empty means this trial's cost is a FLOOR, not a level: those + # phases ran and spent, and their usage was never flushed. An + # analysis that quotes the mean cost of a cell containing these + # should say so rather than average them in silently. + aggregated.metadata["unsettled_phases"] = unsettled_phases + self.result.agent_result = aggregated + await self._sync_agent_output(self.result) +''' + + +def find_harbor_root(explicit: str | None) -> Path: + if explicit: + root = Path(explicit) + if not (root / "trial" / "single_step.py").exists(): + raise SystemExit(f"not a harbor package dir: {root}") + return root + spec = importlib.util.find_spec("harbor") + if spec is None or not spec.origin: + raise SystemExit("harbor is not importable; pass --harbor-root") + return Path(spec.origin).parent + + +def write_unlinked(path: Path, text: str) -> None: + """Replace `path`'s contents without writing through a shared inode. + + uv populates a venv by hardlinking files out of its wheel cache, so a plain + write_text() into site-packages edits the cached wheel too. The patch then + survives `uv pip install --reinstall-package harbor` -- reinstall relinks + the same, already-modified inode -- and there is no way back to a pristine + tree short of `uv cache clean harbor`. Unlinking first gives this path a + fresh inode and leaves the cache untouched. + """ + path.unlink() + path.write_text(text) + + +def apply_edit(path: Path, anchor: str, patched: str, *, check: bool) -> str: + text = path.read_text() + if patched in text: + return "already-applied" + if anchor not in text: + raise SystemExit( + f"anchor text not found in {path} -- Harbor version drifted, " + "re-derive the patch by hand" + ) + if check: + return "would-apply" + write_unlinked(path, text.replace(anchor, patched, 1)) + return "applied" + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--check", action="store_true", help="report without writing") + ap.add_argument("--harbor-root", default=None) + args = ap.parse_args() + + root = find_harbor_root(args.harbor_root) + print(f"harbor package: {root}") + + config_py = root / "models" / "task" / "config.py" + single_py = root / "trial" / "single_step.py" + for p in (config_py, single_py): + if not p.exists(): + raise SystemExit(f"missing {p}") + + results = { + "config.py:AgentConfig.continue_until_timeout": apply_edit( + config_py, CONFIG_ANCHOR, CONFIG_PATCHED, check=args.check + ), + "single_step.py:imports": apply_edit( + single_py, IMPORT_ANCHOR, IMPORT_PATCHED, check=args.check + ), + "single_step.py:_run_agent loop": apply_edit( + single_py, RUN_AGENT_ANCHOR, RUN_AGENT_PATCHED, check=args.check + ), + } + + # Helpers are module-level, so they go in separately -- inserted ahead of + # the class rather than inside it. The sentinel must be the *definition*, + # not the bare name: the loop above calls _merge_agent_contexts, so a + # substring check on the name alone reports "already applied" and silently + # skips inserting the helpers. + text = single_py.read_text() + if "def _merge_agent_contexts" not in text: + if args.check: + results["single_step.py:module-helpers"] = "would-apply" + else: + class_anchor = "class SingleStepTrial(Trial):" + if class_anchor not in text: + raise SystemExit("could not locate SingleStepTrial class") + write_unlinked( + single_py, + text.replace( + class_anchor, HELPERS.strip("\n") + "\n\n" + class_anchor, 1 + ), + ) + results["single_step.py:module-helpers"] = "applied" + else: + results["single_step.py:module-helpers"] = "already-applied" + + for k, v in results.items(): + print(f" {v:16s} {k}") + + if not args.check: + import os + import subprocess + + # Import the patched copy specifically, which may not be the harbor on + # the default sys.path (scratch-dir dry runs, multiple venvs). + env = dict(os.environ) + env["PYTHONPATH"] = str(root.parent) + os.pathsep + env.get("PYTHONPATH", "") + rc = subprocess.run( + [ + sys.executable, + "-c", + "import harbor.trial.single_step as m; " + "from harbor.models.task.config import AgentConfig; " + "assert 'continue_until_timeout' in AgentConfig.model_fields, " + "'config field missing'; " + "assert hasattr(m, '_merge_agent_contexts'), 'helpers missing'; " + "import inspect; src = inspect.getsource(m.SingleStepTrial._run_agent); " + "assert 'continue_until_timeout_phases' in src, 'loop missing'; " + # Asserted separately from the loop marker: the loop and the + # per-phase receipts fix were added at different times, and a + # re-derived patch that drops the receipts half would otherwise + # verify green while quietly restoring the last-phase-only bug. + "assert 'receipts.phase-' in src, 'per-phase receipts fix missing'; " + "assert hasattr(m, '_sum_receipt_cost'), 'receipts cross-check missing'; " + "print('verify: OK -- field, helpers, phase loop and per-phase " + "receipts all present')", + ], + env=env, + ).returncode + if rc != 0: + return rc + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/harbor-buzz-orchestra/patches/gatecheck.py b/benchmarks/harbor-buzz-orchestra/patches/gatecheck.py new file mode 100755 index 0000000000..1734efc1e4 --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/patches/gatecheck.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""Decide whether an LHTB smoke run earned its full run. + +Usage: gatecheck.py [ ...] + +Prints one KEY=VALUE line per signal and exits 0 only if every gate passes. +Four gates, each of which has silently ruined a run before: + + seats every required seat id appears in receipts. A goosetown team whose + role name fails to resolve still reports every send as a success; + the trial burns its whole budget and reads as a bad score rather + than a broken manifest. + phases continue_until_timeout fired more than once. If harbor is running + unpatched the trial is single-shot and the whole point of the run + is gone -- and the applier's own --check cannot tell a stale patch + from a current one. + receipts metadata.receipts_phase_files is present. This is the first run + with the per-phase receipts fix; without it the bill is whatever + the last phase happened to cost, which under-reported a team cell + by 87% on the previous LHTB wave. + scored the verifier produced a reward (may be 0.0; None means the trial + never reached scoring). +""" + +import json +import pathlib +import re +import sys + +jobs = pathlib.Path(sys.argv[1]) +required = set(sys.argv[2:]) + + +def role(agent_id: str) -> str: + """Roster id behind a receipt's agent_id. + + Receipts carry the *instance* id -- `lead-1`, `scout-1`, `worker-1` -- while + the manifest roster and this script's arguments carry the role, because a + seat with `count: 2` produces `worker-1` and `worker-2`. Comparing the two + directly fails the gate on a cell where every seat spoke perfectly. + """ + return re.sub(r"-\d+$", "", agent_id) + + +seats: set[str] = set() +for f in jobs.glob("*/*/agent/buzz/receipts.jsonl"): + for line in f.read_text().splitlines(): + if line.strip(): + seats.add(role(json.loads(line).get("agent_id", "?"))) + +results = sorted(jobs.glob("*/*/result.json")) +phases, phased_receipts, rewards = [], [], [] +for r in results: + d = json.loads(r.read_text()) + md = ((d.get("agent_result") or {}).get("metadata")) or {} + phases.append(int(md.get("continue_until_timeout_phases") or 0)) + phased_receipts.append(md.get("receipts_phase_files") is not None) + vr = (d.get("verifier_result") or {}).get("rewards") or {} + rewards.append(vr.get("reward")) + +ok_seats = required <= seats +ok_phases = bool(phases) and max(phases) > 1 +ok_receipts = bool(phased_receipts) and all(phased_receipts) +ok_scored = bool(rewards) and all(r is not None for r in rewards) + +print(f"trials={len(results)}") +print(f"seats={','.join(sorted(seats)) or '(none)'} required={','.join(sorted(required))}") +print(f"phases={phases}") +print(f"receipts_phase_files_present={phased_receipts}") +print(f"rewards={rewards}") +print(f"GATE seats={ok_seats} phases={ok_phases} receipts={ok_receipts} scored={ok_scored}") + +sys.exit(0 if (ok_seats and ok_phases and ok_receipts and ok_scored) else 1) diff --git a/benchmarks/harbor-buzz-orchestra/patches/lhtb-high.sh b/benchmarks/harbor-buzz-orchestra/patches/lhtb-high.sh new file mode 100755 index 0000000000..ebd0c1513b --- /dev/null +++ b/benchmarks/harbor-buzz-orchestra/patches/lhtb-high.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# One high-effort LHTB-46 cell, smoke-gated. One cell per box, so the two cells +# of this pair run concurrently on separate hardware instead of serially. +# +# Usage: lhtb-high.sh