Skip to content

Run the whole chain against itself before trusting any of it - #532

Merged
MongLong0214 merged 1 commit into
devfrom
cdeb-09-smoke
Aug 11, 2026
Merged

Run the whole chain against itself before trusting any of it#532
MongLong0214 merged 1 commit into
devfrom
cdeb-09-smoke

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

CDEB-09. Depends on #529 and #528 (both merged). Closes the wiring gap #529 flagged.

Every earlier ticket tested its own component with its own fixtures. Components that each refuse correctly still compose into a pipeline that does not — a guarantee enforced in one module is worth nothing if the module that should invoke it never does, and no unit test can see that gap because it lives exactly between the things being tested.

This is the only place the composed path runs: freeze → orchestrate → evaluate → ledger → analyze → verify, with each attack injected into the running pipeline rather than into a stand-in for it.

The gap was real

#529's Warn: said the analyzer reads only row files the freeze names, the orchestrator writes by its own convention, and nothing checked the pairing. A full run produced rows the analyzer would have refused to read.

The writer moved, not the reader. The orchestrator now writes at the paths the freeze publishes. Relaxing the analyzer to accept rows discovered on disk would have closed the gap by deleting the guarantee that exists because of #441.

Every case names its refusal

A run that fails proves nothing on its own — it can fail for the wrong reason and look identical from outside. That is how a control that never fires passes for one that does. So the assertions name where the pipeline stopped:

attack what is asserted
model drift hard stop, no row and no final tree, while recording that a first turn was observed
evaluator tamper / network / secret a failing evaluation for every cell — not an error that could pass for infrastructure trouble
shipping proxy mutation caught
nested invalid result refused
evaluator retry bound to the patch-frozen tree

Stated

Limit: this exercises the attacks the design anticipated. An adversary who reads it will look for the ones it does not attempt — notably collusion between a candidate and an evaluator image, and any attack on the freeze itself before a run starts.

Warn: the container runtime was unavailable here, so network isolation, host secret and mount isolation, and resource limits are proven only against the local sealed evaluator. The OCI enforcement path needs a machine with a working daemon before the freeze may treat it as tested.

43 cases pass across the smoke, orchestrator, analyzer and verifier suites; both typechecks clean; two builds leave dist unchanged; both verifiers pass. No live provider or container ran.

Every earlier ticket tested its own component with its own fixtures, and
components that each refuse correctly still compose into a pipeline that does
not. A guarantee enforced in one module is worth nothing if the module that
should invoke it never does, and no unit test can see that gap -- it lives
exactly between the things being tested. This is the only place the composed
path runs: freeze, orchestrate, evaluate, ledger, analyze, verify, with each
attack injected into the running pipeline rather than into a stand-in for it.

The gap was real and this is where it showed. The analyzer reads only row files
the freeze names; the orchestrator was written before that rule existed and
wrote rows by its own convention. A full run therefore produced rows the
analyzer would have refused to read. The orchestrator now writes at the paths
the freeze publishes -- the writer moved, not the reader, because a reader that
accepts unnamed files is the defect #441 already cost this project once.

Each case has to show the pipeline stopping and where. A run that fails proves
nothing on its own: it can fail for the wrong reason and look identical from
outside, which is how a control that never fires passes for one that does. So
the assertions name the refusal -- the model identity hard stop leaves no row
and no final tree while recording that a first turn was observed, and a sealed
evaluator faced with tamper, network or secret access returns a failing
evaluation for every cell rather than an error that could be mistaken for
infrastructure trouble.

Limit: this exercises the attacks the design anticipated, and an adversary who reads it will look for the ones it does not attempt -- notably collusion between a candidate and an evaluator image, and any attack on the freeze itself before a run starts
Ruled-out: relaxing the analyzer to accept rows discovered on disk | it would have closed the wiring gap by removing the guarantee that closes #441
Warn: the container runtime was unavailable here, so network isolation, host secret and mount isolation, and resource limits are proven only against the local sealed evaluator; the OCI enforcement path needs a machine with a working daemon before the freeze may treat it as tested
Blast: module
Undo: easy
Certainty: firm
Verified: forty-three cases pass across the smoke, orchestrator, analyzer and verifier suites, covering evaluator tamper, network and secret attempts refused inside a running study, a mutated shipping proxy caught, model drift stopping the run with no row written, a nested invalid result refused, an evaluator retry bound to the patch-frozen tree, and a composed run whose rows the strict analyzer accepts; package and bench typechecks clean; two builds leave dist unchanged; both bench verifiers pass
Unverified: no live provider or container ran, so every agent, evaluator and stream here is a recorded or local stand-in
Provenance: authored
Record-Id: r-cdeb09sm
@github-actions

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 1 commit in origin/dev..74d9c620e7c5ed12867c9377d36596eecda2d85c
Active constraints: 7 limits · 11 ruled-out · 7 warnings — from 6 records over 5 changed paths

Active constraints for the paths this PR touches

Limits (7)

  • r-cdeb09sm 74d9c62 — this exercises the attacks the design anticipated, and an adversary who reads it will look for the ones it does not attempt -- notably collusion between a candidate and an evaluator image, and any attack on the freeze itself before a run starts
  • r-cdeb07or 5607dd5 — the state machine cannot distinguish an agent that produced no model turn from one whose host failed to report the turn it produced, so it treats both as interrupted and neither as retryable -- a real pre-turn infrastructure failure is therefore sometimes charged as a lost run
  • r-cdeb07or 5607dd5 — atomicity rests on rename and fsync semantics of the underlying filesystem; a filesystem that reorders them can still present a partial the recovery pass reads as whole
  • r-cdeb05lg 5d347b9 — the ledger observes only bytes emitted before the agent process closes -- terminal usage never written to the stream, external billing, and work outside that stream are all unavailable to it rather than estimated
  • r-v13fix f4b6859 — schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written
  • r-cdebpres e01b482 — one repository, four tasks, six usable runs per arm and a local evaluator -- no claim gate is evaluable and preregistration §8 forbids any number here reaching a public surface
  • r-cdebver01 ce7b278 — the schemas freeze protocol 1.2.0 constants -- thresholds, matrix size, category names -- so a protocol change is a schema change and CI notices

Ruled out (11)

  • r-cdeb09sm 74d9c62 — relaxing the analyzer to accept rows discovered on disk | it would have closed the wiring gap by removing the guarantee that closes bench:m5 reads every jsonl in bench/results, so the registered analysis would run over M1, M4 and files marked non-citable #441
  • r-cdeb07or 5607dd5 — retrying an agent after a post-turn infrastructure failure | keeping whichever attempt survived selects on the outcome, and the surviving attempt is not a random draw
  • r-cdeb07or 5607dd5 — re-deriving the candidate tree for an evaluator retry | it converts an evaluator flake into a second candidate, which is a different experiment
  • r-cdeb05lg 5d347b9 — inferring terminal usage from the turns that did arrive | it would be a plausible number carrying no evidence, and plausible is exactly what makes it survive review
  • r-cdeb05lg 5d347b9 — excluding an unavailable run from the denominator | that silently redefines the population the token claim describes
  • r-v13fix f4b6859 — keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run
  • r-v13fix f4b6859 — lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product
  • r-cdebpres e01b482 — reporting the completed-only table as the pilot's result without the timed-out task | the exclusion is the registered stop_reason rule, and a reader who cannot see that a quarter of the study never finished cannot judge the rest
  • r-cdebpres e01b482 — dropping the two zero-delivery tasks as broken | they are the intention-to-treat case the protocol is built around, and they are also how the qualification gap was found
  • r-cdebver01 ce7b278 — default-in discovery over the CDEB tree like the legacy gate | that gate's enemy was the unregistered file, CDEB's is the unregistered input, and bench:m5 reads every jsonl in bench/results, so the registered analysis would run over M1, M4 and files marked non-citable #441 showed a glob passing a stopping rule on contamination
  • r-cdebver01 ce7b278 — validating with the draft-07 Ajv entry and downgrading the schemas | the 2020-12 entry ships in the same package, and weakening six schemas to fit a meta-schema is backwards

Warnings (7)

  • r-cdeb09sm 74d9c62 (claim) — the container runtime was unavailable here, so network isolation, host secret and mount isolation, and resource limits are proven only against the local sealed evaluator; the OCI enforcement path needs a machine with a working daemon before the freeze may treat it as tested
  • r-cdeb07or 5607dd5 (claim) — the analyzer reads only row files the freeze names, so the freeze must publish the full set of canonical row paths and this orchestrator must write rows at exactly those paths; the two are wired together by the freeze and nothing checks the pairing yet
  • r-cdeb05lg 5d347b9 (claim) — one unavailable run makes the whole aggregate unavailable, which is the strict reading of §14.6 and will feel severe on a long matrix; the reasons are per run, so a report can say which ones and why
  • r-cdeb05lg 5d347b9 (claim) — the artifact path uses node:zlib zstd, which exists from Node 22.15 -- above the package floor of 22, though this is bench-only and no shipped code imports it
  • r-v13fix f4b6859 (claim) — the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task
  • r-cdebpres e01b482 (claim) — the exposure counter counts deliveries rather than opportunities, so a zero cannot distinguish a hook that never fired from one that fired on a path with no records -- CDEB v1 §9.5 requires that split and CDEB-05 must build it
  • r-cdebver01 ce7b278 (claim) — every failure fixture is the one valid row with a single mutation, so keep the valid-row builder in lockstep with result.schema.json -- a builder drift fails all eleven cases at once and reads as a broken verifier

the index is unavailable (the index has no baseline commit); answering with a full scan

git log --follow accepts exactly one pathspec, so renames are not followed for 5 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant