bench: tb21 harness carry + usage accounting, now synced with main (merge, no rebase)

benchmarks/harbor-buzz-orchestra/.gitignore

@@ -4,3 +4,4 @@ __pycache__/
 .pytest_cache/
 .benchmark/
 jobs/
+jobs-archive/

benchmarks/harbor-buzz-orchestra/README.md

@@ -36,6 +36,19 @@ roster:
 `endpoint_config` maps those endpoint names to providers, URLs, and API-key
 environment variables. The adapter contains no fixed roster or model.
 
+Exactly one orchestrator is required; **workers are optional**. A roster of one
+orchestrator and nothing else is the single-agent baseline — see
+`manifests/tb-solo-sonnet.yaml` and `personas/solo-tb.md`. The lone agent gets
+byte-identical wiring to a worker (same binaries, same MCP toolset, same env),
+because a handicapped baseline would flatter every multi-agent condition
+measured against it.
+
+`generation` has no `temperature`: `buzz-agent` exposes no temperature
+environment variable, so the value could never reach the provider. It used to
+be accepted, hashed into the condition identity, and silently discarded — with
+a default of `0.0` while the measured provider default is `1.0`. It is now
+rejected outright rather than quietly lying.
+
 ## Run
 
 With the production compose stack and model endpoints already running, execute
@@ -62,13 +75,237 @@ rather than deletes that channel, leaving the relay/Postgres event timeline
 and the per-agent acp/agent logs (downloaded into the trial's `buzz/`
 artifacts) available for analysis.
 
+## Trial artifacts
+
+Every trial writes a self-contained bundle to its `buzz/` artifact directory —
+the container is gone once the trial ends, so this is what survives:
+
+```
+manifest.json                the frozen condition and its sha256
+<agent-id>.system-prompt.md  the composed prompt as that agent saw it
+<agent-id>.stdout/.stderr.log
+receipts.jsonl               one priced usage row per agent
+endpoints.redacted.json      provider and env-var *names* only, never values
+summary.json                 the index to start from
+```
+
+The bundle exists so system prompts can be tuned after a sweep: the exact
+prompt bytes sit next to what that agent cost. It is written even when the
+trial fails or times out — a trial that burned tokens and then stalled still
+cost money, and dropping it would bias cost figures toward successes.
+
+Because every agent runs with a live provider token and a Nostr private key in
+its environment, the bundle is scanned for credential-shaped strings and the
+result is recorded in `summary.json` under `secret_scan`. Check it before
+sharing a bundle.
+
+### Container trust store
+
+Many Terminal-Bench images ship no `ca-certificates` package, so
+`/etc/ssl/certs/ca-certificates.crt` does not exist and **every https client in
+the container fails** — apt, curl, git, pip. `buzz-agent` is the exception: it
+gets `SSL_CERT_FILE=/opt/buzz/ca-certificates.crt`, which nothing else reads.
+
+That asymmetry cost real scores. On the A1 sweep an agent hit a transient apt
+error, rewrote `sources.list` from http to https to work around it, and left
+the container in a state where the **verifier's** own `apt-get update` could
+not validate a certificate. The verifier reported `E: Unable to locate package
+curl`, never installed pytest, and the task was recorded as reward 0.0 —
+identical in `result.json` to a model that got the answer wrong.
+
+So `_install_stack` copies the same bundle to `/etc/ssl/certs/ca-certificates.crt`
+before the agents launch. It is offline (the bundle is already uploaded), so it
+adds no egress dependency, and it **only writes when that path is missing or
+empty** — a task whose subject is certificate handling keeps whatever store its
+image shipped.
+
+The outcome is recorded per trial in `result.json` under
+`agent_result.metadata.container_trust_store`:
+
+| value | meaning |
+|---|---|
+| `present` | the image had its own store; we changed nothing |
+| `seeded` | the image had none and we installed ours |
+| `failed` | the image had none and we could not install ours |
+
+`failed` is deliberately not fatal — the agent still reaches its provider
+through `SSL_CERT_FILE`, and killing a runnable trial would trade a partial
+handicap for a total one. But **read a 0.0 on a `failed` trial as suspect**,
+not as a wrong answer.
+
+### Verifier dependencies
+
+81 of the 89 Terminal-Bench `test.sh` files begin with `apt-get install -y
+curl`, to fetch the uv installer that installs pytest. Only 3 of 12 sampled
+task images ship curl, so most trials do that install at scoring time — the one
+moment when a transient apt failure is unrecoverable and lands as a 0.0.
+
+So the runtime installs `VERIFIER_DEPS` itself, and the timing is the point:
+
+- **After the agents are stopped.** Installing before would hand the agent a
+  tool its task image chose not to ship, which changes the thing being
+  measured. Doing it in teardown changes only what the verifier finds.
+- **Before Harbor's verifier phase**, which is what needs it.
+
+The win is not saving the verifier's apt call — it still makes one. It is that
+with curl already in dpkg's status file, `apt-get install -y curl` resolves
+from the installed version and succeeds *even when `apt-get update` left the
+index empty*. That is exactly the failure that scored `compile-compcert` 0.0.
+
+Recorded as `agent_result.metadata.container_verifier_deps`: `present` (image
+shipped it) | `installed` | `unavailable`. Best-effort, like the trust store —
+measured at ~7s per trial on the EC2 runner, which at `--n-concurrent 16`
+is under a minute across a full 89-task sweep.
+
+The package list is deliberately just `curl`. The rest of the task set's apt
+requests are a long tail no blanket pre-install should chase: git 3, binutils
+1, everything else once.
+
+### Preflight
+
+`benchmark.py` refuses to start a sweep that cannot be scored. Two checks run
+before the stack, the binaries, and the money:
+
+- `check_tls_not_intercepted()` — the **host's** path to PyPI is not being
+  MITM'd. A Cloudflare WARP session once turned a live 89-task sweep into 0/89.
+- `check_container_can_be_scored()` — a **container** can seed its trust store
+  and install curl. The host reaching PyPI says nothing about this: the docker
+  proxy, the image's trust store and apt are a separate path, and it is the one
+  every verifier depends on.
+
+The container check runs the trials' own two setup commands —
+`seed_trust_store_command()` and `install_verifier_deps_command()`, imported
+from `container_runtime`, not reimplemented — against `ubuntu:24.04`, the base
+most task images derive from and which likewise ships neither `ca-certificates`
+nor curl. A preflight that proved something subtly different from what the
+trials do would be worse than none, because it would read as a clean bill of
+health. It costs ~7s.
+
+Both follow the same rule: a check that *cannot see* (no docker, no openssl, a
+pull that timed out) prints `skipping check` and lets the run proceed. Only a
+positive signal of breakage stops it.
+
+### What none of this covers
+
+An agent that switches apt to a mirror the network cannot reach at all.
+`compile-compcert` failed that way — the agent moved apt to
+`https://azure.archive.ubuntu.com`, whose Azure address is unreachable from the
+Square egress path at :443 regardless of trust anchors. Restoring
+`sources.list` before the verifier runs would close it; that is not implemented.
+
+`verifier_health.py` carries `unable to locate package` as the backstop, so a
+verifier that still cannot install its dependencies is reported as broken by
+`summarize.py` rather than averaged in as a zero.
+
+### Token and cost accounting
+
+`buzz-agent` reports cumulative token counts per turn, `buzz-acp` logs them
+under the `acp::usage` tracing target, and the runtime parses them out of the
+downloaded logs to price each agent against the manifest's `prices` table. The
+runtime sets `RUST_LOG` to enable that target; if an endpoint config sets its
+own `RUST_LOG`, the directive is appended rather than replaced, because losing
+it would present as a free trial rather than as an error.
+
+**Cache reads are estimated, not measured.** `LlmResponse.input_tokens` is an
+inclusive sum of plain, cache-read, and cache-write input and nothing
+downstream carries the split. Each endpoint's `Price` therefore declares a
+`cache_read_rate` (0.0–1.0); that fraction of input is billed at
+`cached_input_per_million_usd` and the rest at the full rate. Every receipt
+also carries `cost_usd_no_cache_discount`, so the assumption's leverage is
+visible rather than baked into one number.
+
+The rate lives in the manifest, not in code, so it is frozen into the condition
+hash — two cost figures are only comparable under the same assumption.
+
+Choosing a rate:
+
+- **Anthropic-route endpoints: `0.0`, and that is exact, not conservative.**
+  `buzz-agent` never sends `cache_control` and Anthropic prompt caching is
+  opt-in, so no cache reads occur. (It also means Buzz is leaving a real
+  discount on the table — worth fixing in the agent, not in the harness.)
+- **OpenAI-route endpoints: non-zero.** OpenAI caches automatically above ~1k
+  prompt tokens and reports `cached_tokens`, and an agent loop resends a
+  growing prefix every turn, so the true rate is substantial. Calibrate it from
+  the gateway's own `cached_tokens` on a real trial rather than guessing.
+
+Leaving every rate at 0.0 gives the old upper-bound behaviour, but it is not
+neutral across conditions: it over-charges exactly the endpoints that really do
+cache, which biases OpenAI-route workers against Anthropic-route ones.
+
+**Reasoning tokens are not separable, and this does not affect cost.** Providers
+bill thinking at the output rate, so reasoning tokens inside `output_tokens` are
+already priced correctly. Only the thinking-vs-answer split is unavailable —
+an analysis detail, not a cost error.
+
+**Auto-compaction is billed but unreported, so it is bounded instead.** When a
+session crosses its compaction threshold, `buzz-agent` summarises the history
+into a fresh context (a "handoff"). That summarisation is a real provider call,
+but `Llm::summarize` returns a bare `String` and throws its usage away, so those
+tokens never reach the counters above. On a long trial with the default
+`max_handoffs` of 80 the omission can exceed everything else we model.
+
+The runtime parses the handoff log lines — which carry the pre-handoff context
+size — and publishes a worst case beside the metered figure: each handoff is
+charged at most its pre-handoff context as input plus tokens of output, at
+full rate. So `cost_usd` is the floor, `cost_usd_including_handoff_bound` is the
+ceiling, and the truth is between them. Receipts keep `handoffs`,
+`handoff_input_tokens_upper_bound`, and `handoff_cost_usd_upper_bound` separate
+from the metered fields; nothing is silently merged. Getting the exact number
+requires a `buzz-agent` change (return usage from `summarize` and fold it into
+the turn counters).
+
+Handoffs do **not** fail reconciliation — the metered numbers are still genuine —
+but the reconciliation note says the total is a floor whenever any occurred.
+
+A separate signal to watch: if the handoff cap is reached or summarisation
+fails, the agent falls back to *truncating* its own history. The runtime counts
+those as `handoff_truncations` and raises a warning, because dropped context is
+a task-fidelity problem, not just a cost one.
+
+A trial whose agents ran but reported no tokens is marked
+`accounting_reconciled: false` with a reason. Such a trial is an
+instrumentation failure, not a free trial, and must not be averaged into a cost
+figure as though it were a real zero.
+
+### Compaction threshold
+
+`buzz-agent` fires a handoff at whichever comes first: a percentage of the
+context window (`BUZZ_AGENT_HANDOFF_PERCENT`, default 90) or an absolute
+ceiling (`BUZZ_AGENT_HANDOFF_AT_TOKENS`, default 272,000; `0` disables it). The
+ceiling is inert at a 200k window, where 90% (180k) binds first, and takes over
+on large ones — at 1M, 90% would mean summarising ~900k tokens of history in a
+single call, and every request would sit above OpenAI's 272k long-context
+pricing boundary.
+
+Two manifest fields pass straight through, both optional:
+
+```yaml
+generation:
+  max_output_tokens: 4096
+  context_window_tokens: 1000000
+  compact_at_percent: 30       # → BUZZ_AGENT_HANDOFF_PERCENT
+  compact_at_tokens: 272000    # → BUZZ_AGENT_HANDOFF_AT_TOKENS
+```
+
+Leave them unset to inherit the agent's defaults; the runtime then omits the
+variables entirely, so a variable's presence in a trial bundle means the
+condition really did pin it. `context_window_tokens` is always reported to the
+agent as the real window — the harness never bends it to move the trigger.
+
+`compact_at_tokens` is validated against `context_window_tokens - max_output_tokens`,
+since buzz-agent reserves room for the response and a target above that could
+never fire. A ceiling merely larger than the percentage is fine — that is the
+default at 200k. Because compaction cadence moves both cost and task
+performance, both fields are part of the condition hash.
+
 ## Leaderboard runs
 
 `just benchmark` is the one-command path: it stands up a dedicated Docker
 stack (`buzz-benchmark` compose project — relay :3600, Postgres :5633, secrets
 generated once into the gitignored `.benchmark/`), applies the benchmark
-schema, and defaults to leaderboard-eligible settings (Terminal-Bench 2.1,
-5 attempts per problem, the Sonnet+Haiku team). All selectors pass through:
+schema, and defaults to leaderboard-eligible settings
+(`terminal-bench/terminal-bench-2-1`, 5 attempts per problem, the Sonnet+Haiku
+team). All selectors pass through:
 
 ```bash
 just benchmark                                   # full TB 2.1, k=5
@@ -108,6 +345,14 @@ uv run --project benchmarks/harbor-buzz-orchestra/testbed \
     --provisioner-config <PROVISIONER.json>
 ```
 
+Harbor accepts two dataset forms, resolved from different places: `org/name[@ref]`
+from the hub package registry, and `name[@version]` from `registry.json`.
+Terminal-Bench 2.1 exists only as a package — `terminal-bench@2.0` in
+`registry.json` is the older 2.0 cut — which is why the default is the
+slash form. Whichever form is given is validated against the right source
+before the stack comes up, so a typo or an upstream rename fails in a second
+rather than after the first trial.
+
 `--path` replaces `--dataset` for local task directories; `--include-task` /
 `--exclude-task` filter by glob; `--dry-run` prints the underlying `harbor run`
 command. After the job finishes the script derives a `metadata.yaml` from the