Never give a run a second chance at its own outcome

[MongLong0214]

CDEB-07. Depends on #523, #524, #521 (all merged). Pairs with #528 (CDEB-08).

The retry that is not a retry

Once a model has answered, that attempt is the observation. Re-running it and keeping the better result is selection on the outcome — and in code it does not look like cheating, it looks like a retry. That is why this cannot be a rule someone remembers.

A durable marker is written the moment a first model turn is observed, and every path afterwards is checked against it in both directions:

• an adapter reporting a pre-turn failure after the marker exists → integrity error
• a post-turn result arriving without a marker → integrity error
• an adapter that throws → interrupted, not retryable

That last one is deliberate. A process can produce a turn before its host reports one, so the safe reading of "I do not know" is that it did. This costs us real pre-turn infrastructure failures, charged as lost runs. That trade is stated in the Limit:.

The tree is the observation too

An evaluator retry runs against the frozen tree bound by object id — never the workspace. Otherwise an evaluator flake becomes a second chance at a differently derived candidate, which is a different experiment wearing the same logical run id.

Resume is the same defect wearing a recovery badge

Resume computes the missing set from the randomization manifest and what is durably on disk, and refuses to write over an id that already has a row.

Written for the kill at the worst moment

A row, an attempt and a final tree each land completely or not at all. A partial that was fsynced but never renamed is cleaned before resume rather than read. A tree archive without its metadata is now refused by the verifier instead of counting as a final tree.

A half-written artifact that verifies is worse than a missing one, because only one of the two is visible.

Progress cannot leak an outcome

Whoever watches a run must not learn which arm is winning while the run can still be stopped. The surface carries five lifecycle and count fields, is frozen, and is shaped so it cannot carry anything else. A shape that cannot express the value beats a rule saying not to print it.

One wiring gap, stated not hidden

Warn: #528's analyzer reads only row files the freeze names. So the freeze must publish the full set of canonical row paths, and this orchestrator must write rows at exactly those paths. The two are wired together through the freeze and nothing checks the pairing yet — that closes once both land.

91 cases pass across the orchestrator, verifier, isolation and ledger suites; both typechecks clean; two builds leave dist unchanged; both verifiers pass. No live study ran, so every agent and evaluator here is a recorded adapter rather than a container.

[github-actions]

CommitLore — record lint

Trailers: clean — 1 commit in origin/dev..5607dd5be161ae80bca2ba183fde017c72137d3c
Active constraints: 9 limits · 18 ruled-out · 9 warnings — from 8 records over 9 changed paths

Active constraints for the paths this PR touches

Limits (9)

• r-cdeb07or 5607dd5 — the state machine cannot distinguish an agent that produced no model turn from one whose host failed to report the turn it produced, so it treats both as interrupted and neither as retryable -- a real pre-turn infrastructure failure is therefore sometimes charged as a lost run
• r-cdeb07or 5607dd5 — atomicity rests on rename and fsync semantics of the underlying filesystem; a filesystem that reorders them can still present a partial the recovery pass reads as whole
• r-cdeb05lg 5d347b9 — the ledger observes only bytes emitted before the agent process closes -- terminal usage never written to the stream, external billing, and work outside that stream are all unavailable to it rather than estimated
• r-cdeb06ev e9d3c72 — a determined candidate can still spend its budget rather than forge -- the resource control bounds a spinning implementation but cannot distinguish slow honest work from deliberate exhaustion
• r-cdeb03rt f0ad091 — enforced by code — the capability gate refuses a missing or never-probed capability, the run spec cannot express host HOME or settings inheritance, provider credentials cross the container boundary only through a two-key allowlist, the captured stream is identity-checked before anything reads it as measurement, and the pin loader rejects hand-edited enforcement shapes. Still resting on the operator — building and pulling the image the freeze records, running preflight on the machine that runs the study, keeping the container runtime daemon itself trusted, and passing provider credentials into the preflight and run calls; the gate can verify what a container does, not that the host it runs on is honest
• r-v13fix f4b6859 — schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written
• r-cdebv13 e0bd61a — the calibration is measured on disposable tasks, so it estimates overhead on work that is not the corpus and a corpus with different injection density will sit off it
• r-cdebpres e01b482 — one repository, four tasks, six usable runs per arm and a local evaluator -- no claim gate is evaluable and preregistration §8 forbids any number here reaching a public surface
• r-cdebver01 ce7b278 — the schemas freeze protocol 1.2.0 constants -- thresholds, matrix size, category names -- so a protocol change is a schema change and CI notices

Ruled out (18)

• r-cdeb07or 5607dd5 — retrying an agent after a post-turn infrastructure failure | keeping whichever attempt survived selects on the outcome, and the surviving attempt is not a random draw
• r-cdeb07or 5607dd5 — re-deriving the candidate tree for an evaluator retry | it converts an evaluator flake into a second candidate, which is a different experiment
• r-cdeb05lg 5d347b9 — inferring terminal usage from the turns that did arrive | it would be a plausible number carrying no evidence, and plausible is exactly what makes it survive review
• r-cdeb05lg 5d347b9 — excluding an unavailable run from the denominator | that silently redefines the population the token claim describes
• r-cdeb06ev e9d3c72 — reading the candidate's own test command from its package manifest | that is the manifest the candidate wrote, and honouring it hands the verdict back to the author it is meant to judge
• r-cdeb06ev e9d3c72 — treating a passing candidate-owned suite as corroborating evidence | a forgery that passes its own tests is the expected shape of the attack, so agreement there carries no information
• r-cdeb03rt f0ad091 — degrading with a warning when the CLI lacks an isolation flag, as the legacy driver does | that is exactly the silent degradation this ticket exists to remove — a run that inherited the operator's settings measures one laptop, not the study
• r-cdeb03rt f0ad091 — container networking set to none with a unix-socket egress proxy | the agent's HTTP stack cannot be assumed to speak a socket proxy, so provider reachability would fail closed on every machine; an internal network plus allowlist proxy is enforceable and probe-verified instead
• r-cdeb03rt f0ad091 — filling the pin manifest's digest fields with values read from this machine | the digests must come from the freeze ceremony that builds the image; a hand-filled manifest is an unfrozen pin pretending, and the gate is built to refuse it
• r-v13fix f4b6859 — keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run
• r-v13fix f4b6859 — lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product
• r-cdebv13 e0bd61a — raising the 15% to a number that looked achievable | picking a threshold to be clearable is the same act as picking one after seeing results, and only the derivation makes it preregistered
• r-cdebv13 e0bd61a — keeping token efficiency in the conjunction with the new threshold | the objection was never the number, it was that one gate could erase a measured behaviour result while the report called it a three-metric study
• r-cdebv13 e0bd61a — widening the hook matcher so the two silent tasks would deliver | that redefines a delivery failure as a delivery success, and §9.5 exists to record exactly that failure
• r-cdebpres e01b482 — reporting the completed-only table as the pilot's result without the timed-out task | the exclusion is the registered stop_reason rule, and a reader who cannot see that a quarter of the study never finished cannot judge the rest
• r-cdebpres e01b482 — dropping the two zero-delivery tasks as broken | they are the intention-to-treat case the protocol is built around, and they are also how the qualification gap was found
• r-cdebver01 ce7b278 — default-in discovery over the CDEB tree like the legacy gate | that gate's enemy was the unregistered file, CDEB's is the unregistered input, and bench:m5 reads every jsonl in bench/results, so the registered analysis would run over M1, M4 and files marked non-citable #441 showed a glob passing a stopping rule on contamination
• r-cdebver01 ce7b278 — validating with the draft-07 Ajv entry and downgrading the schemas | the 2020-12 entry ships in the same package, and weakening six schemas to fit a meta-schema is backwards

Warnings (9)

• r-cdeb07or 5607dd5 (claim) — the analyzer reads only row files the freeze names, so the freeze must publish the full set of canonical row paths and this orchestrator must write rows at exactly those paths; the two are wired together by the freeze and nothing checks the pairing yet
• r-cdeb05lg 5d347b9 (claim) — one unavailable run makes the whole aggregate unavailable, which is the strict reading of §14.6 and will feel severe on a long matrix; the reasons are per run, so a report can say which ones and why
• r-cdeb05lg 5d347b9 (claim) — the artifact path uses node:zlib zstd, which exists from Node 22.15 -- above the package floor of 22, though this is bench-only and no shipped code imports it
• r-cdeb06ev e9d3c72 (claim) — one isolation case skips where the sandbox denies listen, so the socket-level egress path is unverified here and needs a machine that allows binding before the freeze treats it as tested
• r-cdeb03rt f0ad091 (claim) — the two socket-level egress proxy tests skip where the sandbox denies listen — run them where binding is allowed before the freeze treats the proxy as tested; and the probe's frozen wall-time anchors were measured on the host CLI, so re-measure on the pinned runtime before any freeze relies on the 0.6 screen
• r-v13fix f4b6859 (claim) — the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task
• r-cdebv13 e0bd61a (claim) — the 0.6 probe budget and the 1.15 calibration factor are judgement calls with no measurement behind them, unlike the three findings that forced this revision
• r-cdebpres e01b482 (claim) — the exposure counter counts deliveries rather than opportunities, so a zero cannot distinguish a hook that never fired from one that fired on a path with no records -- CDEB v1 §9.5 requires that split and CDEB-05 must build it
• r-cdebver01 ce7b278 (claim) — every failure fixture is the one valid row with a single mutation, so keep the valid-row builder in lockstep with result.schema.json -- a builder drift fails all eleven cases at once and reads as a broken verifier

the index is unavailable (the index has no baseline commit); answering with a full scan

git log --follow accepts exactly one pathspec, so renames are not followed for 9 paths; query one path at a time to follow its rename chain

_{Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.}