Run the whole chain against itself before trusting any of it - #532
Merged
Conversation
Every earlier ticket tested its own component with its own fixtures, and components that each refuse correctly still compose into a pipeline that does not. A guarantee enforced in one module is worth nothing if the module that should invoke it never does, and no unit test can see that gap -- it lives exactly between the things being tested. This is the only place the composed path runs: freeze, orchestrate, evaluate, ledger, analyze, verify, with each attack injected into the running pipeline rather than into a stand-in for it. The gap was real and this is where it showed. The analyzer reads only row files the freeze names; the orchestrator was written before that rule existed and wrote rows by its own convention. A full run therefore produced rows the analyzer would have refused to read. The orchestrator now writes at the paths the freeze publishes -- the writer moved, not the reader, because a reader that accepts unnamed files is the defect #441 already cost this project once. Each case has to show the pipeline stopping and where. A run that fails proves nothing on its own: it can fail for the wrong reason and look identical from outside, which is how a control that never fires passes for one that does. So the assertions name the refusal -- the model identity hard stop leaves no row and no final tree while recording that a first turn was observed, and a sealed evaluator faced with tamper, network or secret access returns a failing evaluation for every cell rather than an error that could be mistaken for infrastructure trouble. Limit: this exercises the attacks the design anticipated, and an adversary who reads it will look for the ones it does not attempt -- notably collusion between a candidate and an evaluator image, and any attack on the freeze itself before a run starts Ruled-out: relaxing the analyzer to accept rows discovered on disk | it would have closed the wiring gap by removing the guarantee that closes #441 Warn: the container runtime was unavailable here, so network isolation, host secret and mount isolation, and resource limits are proven only against the local sealed evaluator; the OCI enforcement path needs a machine with a working daemon before the freeze may treat it as tested Blast: module Undo: easy Certainty: firm Verified: forty-three cases pass across the smoke, orchestrator, analyzer and verifier suites, covering evaluator tamper, network and secret attempts refused inside a running study, a mutated shipping proxy caught, model drift stopping the run with no row written, a nested invalid result refused, an evaluator retry bound to the patch-frozen tree, and a composed run whose rows the strict analyzer accepts; package and bench typechecks clean; two builds leave dist unchanged; both bench verifiers pass Unverified: no live provider or container ran, so every agent, evaluator and stream here is a recorded or local stand-in Provenance: authored Record-Id: r-cdeb09sm
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (7)
Ruled out (11)
Warnings (7)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CDEB-09. Depends on #529 and #528 (both merged). Closes the wiring gap #529 flagged.
Every earlier ticket tested its own component with its own fixtures. Components that each refuse correctly still compose into a pipeline that does not — a guarantee enforced in one module is worth nothing if the module that should invoke it never does, and no unit test can see that gap because it lives exactly between the things being tested.
This is the only place the composed path runs: freeze → orchestrate → evaluate → ledger → analyze → verify, with each attack injected into the running pipeline rather than into a stand-in for it.
The gap was real
#529's
Warn:said the analyzer reads only row files the freeze names, the orchestrator writes by its own convention, and nothing checked the pairing. A full run produced rows the analyzer would have refused to read.The writer moved, not the reader. The orchestrator now writes at the paths the freeze publishes. Relaxing the analyzer to accept rows discovered on disk would have closed the gap by deleting the guarantee that exists because of #441.
Every case names its refusal
A run that fails proves nothing on its own — it can fail for the wrong reason and look identical from outside. That is how a control that never fires passes for one that does. So the assertions name where the pipeline stopped:
Stated
Limit:this exercises the attacks the design anticipated. An adversary who reads it will look for the ones it does not attempt — notably collusion between a candidate and an evaluator image, and any attack on the freeze itself before a run starts.Warn:the container runtime was unavailable here, so network isolation, host secret and mount isolation, and resource limits are proven only against the local sealed evaluator. The OCI enforcement path needs a machine with a working daemon before the freeze may treat it as tested.43 cases pass across the smoke, orchestrator, analyzer and verifier suites; both typechecks clean; two builds leave
distunchanged; both verifiers pass. No live provider or container ran.