M4.1 — [Tracking] Harden and release the benchmark

[GiggleLiu]

Background

New here? Read #9 first — it explains the project and defines every term below.

This is a tracking issue — a deliberately coarse placeholder for the final "make it solid and shippable" milestone. We keep far-off work coarse on purpose and split it into specific, individually-checkable issues only when its turn comes (after the public site, #6–#7, lands).

Objective

Take the working multi-model public benchmark to a maintainable, reproducible, documented release. To be split into specific issues when M3 is done.

Definition of done (what the split-out issues will cover)

• Docs — a README / contributor guide explaining the counterexample certificate format, how to add a model, how to run a session, and how to read the metrics.
• Reproducibility — archive each session's full trajectory + the pinned library version + model versions, so any leaderboard number can be re-validated offline from the saved artifacts.
• Calibration in CI — the checker's known-answer tests (from M1.2 — Build the bug-hunting session: explore → certify → verify, under budget #2 and M2.2 — Harden the verifier (robust equality) and add the novelty filter #5) run automatically in CI, so a future change that secretly weakens the checker fails the build.
• Observability — surface per-session cost, failure reasons, and AI error rates so we can debug the benchmark's health.

Verification (how a reviewer confirms this is done — for now)

This issue lists the four areas above, each with a one-line "done" criterion. When #6–#7 land, re-run the decomposition (the /infinite-bullets skill) to split it into individual, separately-verifiable issues. As a tracking issue, its job right now is to capture scope — not to be implemented.

Dependencies

Depends on M1–M3 (#1–#7).

Out of scope

Anything already covered by #1–#7.

[Ferrari-72]

M3 complete. Decomposed into 4 specific issues:

• M4.1a — README and contributor guide #10 M4.1a — README and contributor guide
• M4.1b — Calibration tests in CI #11 M4.1b — Calibration tests in CI
• M4.1c — Session reproducibility archive #12 M4.1c — Session reproducibility archive
• M4.1d — Observability: per-session cost and failure report #13 M4.1d — Observability: per-session cost and failure report