diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/BUILD.md b/evals/unsafe-rust/fixtures/v4-focused/b_build/BUILD.md new file mode 100644 index 0000000000..2b02b60b6c --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/BUILD.md @@ -0,0 +1,34 @@ +# Ordered allocator-selection build interface + +Supported builds use Cargo 1.85.1 and this crate's `build.rs`. The raw value of +`FIXTURE_ALLOCATOR` has this complete partition: + +- an omitted variable and the Unicode value `system` are accepted and select + the `system` allocator; +- the Unicode value `arena` is accepted and selects the `arena` allocator; +- the Unicode value `arena-stop` is a deliberately rejected freshness canary; + the script writes the `arena` allocator directive and then panics; +- every other Unicode value is rejected before an allocator directive is + attempted; and +- every non-Unicode value is rejected before an allocator directive is + attempted. + +The script first attempts to write +`cargo::rerun-if-env-changed=FIXTURE_ALLOCATOR`. It then reads and classifies +the environment value. Each accepted path attempts exactly one +`cargo::rustc-cfg=fixture_allocator="..."` write and returns successfully if +that write succeeds. The `arena-stop` path attempts that same `arena` write and +then panics. A failure of any stdout write makes that `println!` panic at that +point; any earlier successfully written directive lines are therefore a +partial output prefix of an unsuccessful script execution. + +The freshness guarantee is part of the supported build interface. In +particular, after a successful `arena` build in a Cargo target directory, +changing `FIXTURE_ALLOCATOR` to `arena-stop` must rerun the script and reject +the current build. A previously compiled `arena` library is not a result of +that current rejected build. + +Only a successful script execution for an accepted selector supplies an +allocator configuration to a library compilation. Manually invoking `rustc`, +inventing configuration options, or overriding the build script is outside the +theorem. diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/Cargo.toml b/evals/unsafe-rust/fixtures/v4-focused/b_build/Cargo.toml new file mode 100644 index 0000000000..1c8d9d2c6d --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "ordered-build-target" +version = "0.1.0" +edition = "2021" +rust-version = "1.85.1" +build = "build.rs" + +[features] +default = [] +burst = [] + +[lib] +path = "src/lib.rs" diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/REQUEST.md b/evals/unsafe-rust/fixtures/v4-focused/b_build/REQUEST.md new file mode 100644 index 0000000000..67ea93ea56 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/REQUEST.md @@ -0,0 +1,35 @@ +# Ordered build-to-source unsafe-code audit + +Perform a source-only unsafe-code audit of the complete supplied crate, +including its build script, build and support policies, generated configuration +options, selected library source, public safe API, and documented panic +postcondition. + +Recover the complete supported theorem domain. Then give the complete +claim-relevant build relation in execution order, not merely its endpoint +mapping. Account for every raw `FIXTURE_ALLOCATOR` class, every successful +path, every explicit rejection, every stdout-write failure point, and every +material partial emitted prefix and exit. Follow each emitted directive through +its exact Cargo interpretation to the configuration options and library source +that are selected, or prove why no current library compilation is produced. + +Audit the rerun/freshness sequence explicitly, including reuse of one Cargo +target directory while changing `FIXTURE_ALLOCATOR` from `arena` to +`arena-stop`. Separately prove or reject the project's target/allocator +exclusion. + +Report the strongest soundness and documented-postcondition verdicts. The +report must also state and prove the **exact maximal sound region over the full +supported configuration product and every `u8` API input**, rather than giving +only one counterexample or a non-maximal positive remainder. Symbolic formulas +and parametric proofs are welcome; Cartesian enumeration is not required. + +`TCB.md` is an accepted human trust decision for only the exact Cargo, +freshness, process-status, feature, and target-cfg propositions it names. Keep +every consumed part explicit and do not widen it. Verify every material Rust +proposition against narrowly scoped, version-matched Rust Reference or +standard-library documentation: cite its exact URL and quote the relied-upon +text. Cargo documentation may be used to review the accepted Cargo premise but +is not a Rust semantic axiom. + +Do not modify, build, run, or test the target. diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/SUPPORT.md b/evals/unsafe-rust/fixtures/v4-focused/b_build/SUPPORT.md new file mode 100644 index 0000000000..4509d9b191 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/SUPPORT.md @@ -0,0 +1,23 @@ +# Supported configurations + +This snapshot supports Rust 1.85.1 on these target triples: + +- `x86_64-unknown-linux-gnu`; +- `aarch64-unknown-linux-gnu`; and +- `wasm32-unknown-unknown`. + +Both states of the `burst` feature and both allocator models (`system` and +`arena`, selected as specified in `BUILD.md`) are supported, with exactly one +exclusion: `arena` is unsupported on `wasm32-unknown-unknown`, regardless of +the `burst` feature. The source must reject that excluded target/allocator +pair. + +Every other combination of the listed target, feature state, and allocator is +supported in every Cargo profile, with either state of debug assertions. + +An allocator model is selected only by an accepted raw environment value from +`BUILD.md`. The rejected `arena-stop`, other-Unicode, and non-Unicode classes +are build-interface rejection cases, not additional allocator models or +supported library configurations. A build rejected by a build-script failure +or by the required wasm32/arena source error must not supply a current library +artifact to downstream users. diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/TCB.md b/evals/unsafe-rust/fixtures/v4-focused/b_build/TCB.md new file mode 100644 index 0000000000..bf076342f9 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/TCB.md @@ -0,0 +1,47 @@ +# Accepted build-tool premise + +## BUILD-MAP-ORDERED + +- **Disposition:** accepted by the authorized human reviewer for this target. +- **Identity:** Cargo 1.85.1 from the supported Rust 1.85.1 toolchain, operating + on the supplied manifest, `build.rs`, environment interface, and library. +- **Exact proposition:** for every supported Cargo profile, Cargo executes this + build script when required for the selected build. A successfully written + `cargo::rerun-if-env-changed=FIXTURE_ALLOCATOR` line records that a later + change in the raw value (including present-to-present changes) makes the + prior selection stale and causes the script to run again before Cargo can + select a library for the current build. Cargo interprets a successfully + written `cargo::rustc-cfg=fixture_allocator="VALUE"` line from a build script + that subsequently exits successfully by passing that exact key/value + configuration option to this library compilation. Such a current library + compilation receives exactly the `fixture_allocator` directives emitted by + that successful current script execution and no retained selector from an + earlier execution. Cargo does not compile the library when the build-script + process exits unsuccessfully, even if stdout already contains one or more + complete directive lines. It does not present a library from an earlier + successful selector value as the result of the current stale, unsuccessful + build. An uncaught panic of this exact build script's main thread is an + unsuccessful process exit. + + Enabling `burst` sets `cfg(feature = "burst")`. Compiling for + `x86_64-unknown-linux-gnu`, `aarch64-unknown-linux-gnu`, or + `wasm32-unknown-unknown` sets `target_arch` to `x86_64`, `aarch64`, or + `wasm32`, respectively. +- **Execution scope:** successful-selector emission claims quantify over build + attempts whose applicable stdout writes succeed and whose script returns + successfully. An execution ending in a write failure or explicit panic + produces no library compilation for that current build; any emitted prefix + does not create a compiled configuration. Such an execution is not an + accepted selector success, and an incidental write failure is not a new raw + selector class. +- **Consumers:** only ordered build-output interpretation, freshness, + allocator/feature/target reachability, selected source, and effective + rejection for this target. +- **Excluded propositions:** no claim about which string the local build script + emits or in which order, source correctness, Rust abstract semantics, a + backend, or a binary is admitted. +- **Re-audit trigger:** any change to Cargo/toolchain identity, manifest, + `build.rs`, environment interface, target set, source cfgs, or human + disposition. + +No other build-tool or implementation premise is accepted. diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/build.rs b/evals/unsafe-rust/fixtures/v4-focused/b_build/build.rs new file mode 100644 index 0000000000..5bc025b614 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/build.rs @@ -0,0 +1,27 @@ +use std::env; + +fn main() { + println!("cargo::rerun-if-env-changed=FIXTURE_ALLOCATOR"); + + match env::var("FIXTURE_ALLOCATOR") { + Err(env::VarError::NotPresent) => { + println!("cargo::rustc-cfg=fixture_allocator=\"system\""); + } + Ok(value) => match value.as_str() { + "system" => { + println!("cargo::rustc-cfg=fixture_allocator=\"system\""); + } + "arena" => { + println!("cargo::rustc-cfg=fixture_allocator=\"arena\""); + } + "arena-stop" => { + println!("cargo::rustc-cfg=fixture_allocator=\"arena\""); + panic!("arena-stop rejects this build after allocator emission"); + } + _ => panic!("unsupported FIXTURE_ALLOCATOR value"), + }, + Err(env::VarError::NotUnicode(_)) => { + panic!("FIXTURE_ALLOCATOR must be valid Unicode"); + } + } +} diff --git a/evals/unsafe-rust/fixtures/v4-focused/b_build/src/lib.rs b/evals/unsafe-rust/fixtures/v4-focused/b_build/src/lib.rs new file mode 100644 index 0000000000..f72b549a71 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/b_build/src/lib.rs @@ -0,0 +1,34 @@ +use std::num::NonZeroU8; + +#[cfg(all(target_arch = "wasm32", fixture_allocator = "arena"))] +compile_error!("the arena allocator is unsupported on wasm32"); + +/// Constructs a lane identifier. +/// +/// # Panics +/// +/// Panics when `value` is zero. +pub fn lane_id(value: u8) -> NonZeroU8 { + #[cfg(all( + feature = "burst", + target_arch = "aarch64", + fixture_allocator = "arena" + ))] + { + // SAFETY: Burst-mode lane identifiers are never zero. + return unsafe { NonZeroU8::new_unchecked(value) }; + } + + #[cfg(not(all( + feature = "burst", + target_arch = "aarch64", + fixture_allocator = "arena" + )))] + { + if value == 0 { + panic!("lane identifier must be nonzero"); + } + // SAFETY: The preceding branch proves that `value != 0`. + unsafe { NonZeroU8::new_unchecked(value) } + } +} diff --git a/evals/unsafe-rust/fixtures/v4-focused/l_proof/REQUEST.md b/evals/unsafe-rust/fixtures/v4-focused/l_proof/REQUEST.md new file mode 100644 index 0000000000..b0a58efa0e --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/l_proof/REQUEST.md @@ -0,0 +1,21 @@ +# Local proof-artifact review + +Audit `last` in `lib.rs` without editing it. Determine the implementation's +soundness separately from the adequacy of the existing `SAFETY` comment. If +you must reconstruct any material part of the proof to reach your verdict, +show that reconstruction and provide replacement comment text which would +make the proof locally reviewable. + +Inventory every Rust semantic or standard-library premise materially consumed +by your proof. For each such premise, give a narrowly scoped, version-matched +authoritative citation, quote the exact prose which supplies it, and state the +exact proposition you verified there. +Reconcile that inventory against the premises used in the derivation; do not +list citations which the proof does not consume, or silently consume Rust +facts which the inventory does not establish. + +Scope: exactly Rust 1.82.0, every target on which this exact source and its used +Rust 1.82.0 standard-library items exist, every ordinary profile, and no +additional TCB assumptions. + +Do not build, test, or execute the target. diff --git a/evals/unsafe-rust/fixtures/v4-focused/l_proof/lib.rs b/evals/unsafe-rust/fixtures/v4-focused/l_proof/lib.rs new file mode 100644 index 0000000000..f0f8b8920d --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/l_proof/lib.rs @@ -0,0 +1,11 @@ +#![allow(dead_code)] + +pub fn last(bytes: &[u8]) -> Option<&u8> { + if bytes.is_empty() { + None + } else { + let index = bytes.len() - 1; + // SAFETY: The returned reference cannot outlive `bytes`. + Some(unsafe { bytes.get_unchecked(index) }) + } +} diff --git a/evals/unsafe-rust/fixtures/v4-focused/p_predicates/Cargo.toml b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/Cargo.toml new file mode 100644 index 0000000000..14c20a0e56 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "focused-predicate-target" +version = "0.1.0" +edition = "2021" +rust-version = "1.84" + +[features] +default = [] +turbo = [] +hardened = [] + +[lib] +path = "src/lib.rs" diff --git a/evals/unsafe-rust/fixtures/v4-focused/p_predicates/POLICY-INDIGO.md b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/POLICY-INDIGO.md new file mode 100644 index 0000000000..fe3f0b22cb --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/POLICY-INDIGO.md @@ -0,0 +1,25 @@ +# Published support policy: Indigo + +For this snapshot, let: + +- `V = {1.84.0, 1.85.0, 1.86.0}`; +- `X = x86_64-unknown-linux-gnu`; +- `A = aarch64-unknown-linux-gnu`; +- `W = wasm32-unknown-unknown`; +- `f` mean that `turbo` is enabled; and +- `h` mean that `hardened` is enabled. + +Both Boolean states of each feature are meaningful. A configuration +`(v, t, f, h)` is supported by Indigo exactly when `v` is in `V`, `t` is in +`{X, A, W}`, and this predicate is true: + +```text +!f +or (f and t = X and (h or v >= 1.86.0)) +or (f and t = A and !h and v >= 1.85.0) +``` + +Thus `turbo` on `W` is expressly unsupported. All Cargo profiles and both +states of debug assertions are supported for every configuration selected by +the predicate. + diff --git a/evals/unsafe-rust/fixtures/v4-focused/p_predicates/POLICY-SCARLET.md b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/POLICY-SCARLET.md new file mode 100644 index 0000000000..6f2fa37b53 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/POLICY-SCARLET.md @@ -0,0 +1,25 @@ +# Published support policy: Scarlet + +For this snapshot, let: + +- `V = {1.84.0, 1.85.0, 1.86.0}`; +- `X = x86_64-unknown-linux-gnu`; +- `A = aarch64-unknown-linux-gnu`; +- `W = wasm32-unknown-unknown`; +- `f` mean that `turbo` is enabled; and +- `h` mean that `hardened` is enabled. + +Both Boolean states of each feature are meaningful. A configuration +`(v, t, f, h)` is supported by Scarlet exactly when `v` is in `V`, `t` is in +`{X, A, W}`, and this predicate is true: + +```text +!f +or (f and t = X and (!h or v >= 1.85.0)) +or (f and t = A and h) +``` + +Thus `turbo` on `W` is expressly unsupported. All Cargo profiles and both +states of debug assertions are supported for every configuration selected by +the predicate. + diff --git a/evals/unsafe-rust/fixtures/v4-focused/p_predicates/REQUEST.md b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/REQUEST.md new file mode 100644 index 0000000000..868349fd00 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/REQUEST.md @@ -0,0 +1,37 @@ +# Focused predicate and full-case source review + +`POLICY-SCARLET.md` and `POLICY-INDIGO.md` are both current published support +commitments for this exact source snapshot. No precedence, supersession, +intersection, union, or other conflict-resolution rule has been authorized. + +Perform a source-only unsafe-code audit and state the strongest conclusions +that the supplied evidence justifies. Your report must do all of the following: + +1. Reproduce the exact Scarlet and Indigo configuration predicates separately. + Determine their exact equality/containment relationship, giving one concrete + separating configuration in each direction if neither contains the other. +2. Use a full audit case + `(version, target, turbo, hardened, profile, debug_assertions, input)`, where + `input` ranges over every valid `Option`. Define the full-case domain + induced by each policy. +3. Select and state a conservative full-case audit domain which contains both + policy-induced full-case domains. Prove each containment separately. Do not + describe the selected audit domain as the crate's exact support promise: + the identity of that promise remains a separate question governed only by + the two supplied policy documents and their lack of a precedence rule. +4. Account for both policy-level exclusion and source-level effective rejection + of every `turbo`/`wasm32` case. +5. Prove or refute the local safety obligation at the unsafe operation and the + documented `value_or_zero` postcondition on each selected source branch. +6. State `Required(case)` and the applicable soundness and behavioral + `Covered(case)` predicates without projecting away the configuration or + input dimensions. Give the set-containment argument needed for each + whole-domain conclusion. A symbolic argument over dimensions which are + genuinely irrelevant is preferred to enumerating their values. + +`TCB.md` records an accepted human trust decision. Apply it only to its exact +build-tool mappings and consumers, and keep every conclusion which depends on +it conspicuously qualified. For each material Rust semantic premise, cite the +applicable exact-version Rust Reference or standard-library page and quote the +prose which supplies that premise. Do not modify, build, run, or test the +target. diff --git a/evals/unsafe-rust/fixtures/v4-focused/p_predicates/TCB.md b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/TCB.md new file mode 100644 index 0000000000..d71a4d228a --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/TCB.md @@ -0,0 +1,24 @@ +# Accepted build-tool premise + +## BUILD-MAP-POLICY + +- **Disposition:** accepted by the authorized human reviewer for this target. +- **Identity:** the toolchain-bundled Cargo releases corresponding exactly to + supported Rust 1.84.0, 1.85.0, and 1.86.0, operating on the supplied + `Cargo.toml` and source. +- **Exact proposition:** for each supported release and every supported Cargo + profile, enabling the named Cargo feature `turbo` or `hardened` sets the + corresponding `cfg(feature = "...")` predicate for this library compilation, + while leaving it disabled does not set that predicate. Compiling for + `x86_64-unknown-linux-gnu`, `aarch64-unknown-linux-gnu`, or + `wasm32-unknown-unknown` sets `target_arch` to `x86_64`, `aarch64`, or + `wasm32`, respectively. +- **Consumers:** only configuration reachability and effective-rejection proofs + for this target. +- **Excluded propositions:** no Rust semantic fact, source-branch correctness, + compatibility outside the three exact releases, or backend/binary correctness + is admitted. +- **Re-audit trigger:** any change to Cargo/toolchain identity, feature names, + target set, manifest, source cfgs, or human disposition. + +No other build-tool or implementation premise is accepted. diff --git a/evals/unsafe-rust/fixtures/v4-focused/p_predicates/src/lib.rs b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/src/lib.rs new file mode 100644 index 0000000000..13d51d0ed8 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/p_predicates/src/lib.rs @@ -0,0 +1,19 @@ +#![no_std] + +#[cfg(all(feature = "turbo", target_arch = "wasm32"))] +compile_error!("the turbo feature is not supported on wasm32"); + +/// Returns the contained byte, or zero when `value` is `None`. +#[cfg(not(feature = "turbo"))] +pub fn value_or_zero(value: Option) -> u8 { + value.unwrap_or(0) +} + +/// Returns the contained byte, or zero when `value` is `None`. +#[cfg(feature = "turbo")] +pub fn value_or_zero(value: Option) -> u8 { + let value = value.unwrap_or(0); + + // SAFETY: This `Option` is constructed as `Some` at this proof site. + unsafe { Some(value).unwrap_unchecked() } +} diff --git a/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/Cargo.toml b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/Cargo.toml new file mode 100644 index 0000000000..9596e8903f --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/Cargo.toml @@ -0,0 +1,7 @@ +[package] +name = "quantifier-target" +version = "0.1.0" +edition = "2021" + +[dependencies] +peer-unsafe-dep = "=3.4.5" diff --git a/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/DEPENDENCY-API.md b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/DEPENDENCY-API.md new file mode 100644 index 0000000000..4baf9d71c1 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/DEPENDENCY-API.md @@ -0,0 +1,17 @@ +# Submitted dependency surface + +The exact resolved package is `peer-unsafe-dep` version 3.4.5. Its public +documentation supplies this declaration and contract: + +```rust +/// Returns `value`. +/// +/// # Safety +/// +/// `value` must not equal zero. +pub unsafe fn duplicate_nonzero(value: u8) -> u8; +``` + +The dependency implementation, generated output, prior audit, and binary are +not present in the submitted snapshot. No assertion about what its function +body executes is supplied. diff --git a/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/EVIDENCE.md b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/EVIDENCE.md new file mode 100644 index 0000000000..936041319e --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/EVIDENCE.md @@ -0,0 +1,9 @@ +# Submitted Rust authority + +Exactly one Rust proposition is supplied for this review: + +- Rust 1.80.0 standard-library documentation for + [`std::hint::unreachable_unchecked`](https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html#safety) + states in its Safety section that reaching the function is undefined + behavior. Its applicability is Rust 1.80.0 on every supported target and + ordinary profile; the page states no narrower target or profile condition. diff --git a/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/REQUEST.md b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/REQUEST.md new file mode 100644 index 0000000000..e766ae875a --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/REQUEST.md @@ -0,0 +1,17 @@ +# Focused source-only review + +Audit the soundness of both exported APIs in `lib.rs` as separate claims. For +each claim, state the exact valid-use and compilation domain, the strongest +verdict justified by the supplied source and evidence, and the complete proof +or the smallest missing proposition. + +The supported domain is exactly Rust 1.80.0, every target on which this source +and its exact dependency resolve, and every ordinary debug or release profile. +The dependency resolution, submitted dependency API, evidence policy, and trust +decision are part of the review. + +This is a source-only evidence-packet review. Do not build, test, execute, +download dependency source, inspect any evaluator material, or edit the source. +You may open only the exact versioned official Rust URL named in `EVIDENCE.md` +to verify its supplied proposition. Do not assume facts about the unavailable +dependency implementation beyond an accepted TCB entry. diff --git a/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/TCB.md b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/TCB.md new file mode 100644 index 0000000000..c5e89e85bb --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/TCB.md @@ -0,0 +1,9 @@ +# Trust decision for this review + +- The exact versioned Rust standard-library proposition in `EVIDENCE.md` is an + admitted Rust authority after citation verification. +- No `UNSAFE-DEP` entry for `peer-unsafe-dep` 3.4.5 has been proposed or + accepted. The human reviewer expressly declines to trust its unavailable + implementation for this audit. +- There are no other admitted implementation, compatibility, tool, or + deployment propositions. diff --git a/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/lib.rs b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/lib.rs new file mode 100644 index 0000000000..50b2e1b373 --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/q_quantifiers/lib.rs @@ -0,0 +1,18 @@ +#![allow(dead_code)] + +/// Transfers control to a site declared unreachable. +pub fn force_unreachable() { + // SAFETY: This site is assumed to be unreachable. + unsafe { std::hint::unreachable_unchecked() } +} + +/// Delegates a nonzero value to the selected peer implementation. +/// +/// # Safety +/// +/// `value` must not equal zero. +pub unsafe fn delegated_nonzero(value: u8) -> u8 { + // SAFETY: The caller guarantees that `value` is nonzero, which is the + // dependency API's documented caller-side precondition. + unsafe { peer_unsafe_dep::duplicate_nonzero(value) } +} diff --git a/evals/unsafe-rust/fixtures/v4-focused/r_redesign/REQUEST.md b/evals/unsafe-rust/fixtures/v4-focused/r_redesign/REQUEST.md new file mode 100644 index 0000000000..b002b0ef5b --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/r_redesign/REQUEST.md @@ -0,0 +1,17 @@ +# Focused literal audit and redesign + +Audit the current source exactly as written, then separately recommend the most +parsimonious provable abstraction for its stated requirement. + +Only the crate-owned `Tail` behavior is required: increment element 1 of the +two-element array with wrapping arithmetic. No downstream implementation of +`Slot` or generic call to `increment` must be preserved. This abstraction has +not been released, so its public contract and representation may change. + +Keep the current-artifact verdict independent of every proposal. Explain the +contract and migration delta of the preferred design and what must be audited +after implementation. Do not edit or execute the source. + +Scope: exactly Rust 1.82.0, every target on which this exact source and its used +Rust 1.82.0 standard-library items exist, every ordinary profile, and no +additional TCB assumptions. diff --git a/evals/unsafe-rust/fixtures/v4-focused/r_redesign/lib.rs b/evals/unsafe-rust/fixtures/v4-focused/r_redesign/lib.rs new file mode 100644 index 0000000000..bbcdc1e98d --- /dev/null +++ b/evals/unsafe-rust/fixtures/v4-focused/r_redesign/lib.rs @@ -0,0 +1,18 @@ +#![allow(dead_code)] + +pub trait Slot { + fn index() -> usize; +} + +pub struct Tail; + +impl Slot for Tail { + fn index() -> usize { + 1 + } +} + +pub fn increment(pair: &mut [u32; 2]) { + let value = unsafe { pair.get_unchecked_mut(S::index()) }; + *value = value.wrapping_add(1); +} diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/SKILL.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/SKILL.md new file mode 100644 index 0000000000..89ae055778 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/SKILL.md @@ -0,0 +1,313 @@ +--- +name: unsafe-rust +description: "Author, document, review, audit, or redesign unsafe Rust with proof-grade rigor. Use for unsafe blocks and functions, unsafe traits and impls, raw pointers, FFI, inline assembly, intrinsics, layout or validity reasoning, concurrency and atomics, SIMD and target features, allocators, invariant-bearing fields, safety comments or `# Safety` documentation, soundness reviews, TCB audits, generated unsafe code, changes to safety or behavioral contracts, and proof-oriented redesign of unsafe abstractions." +--- + +# Unsafe Rust Authoring and Audit + +Treat each safety contract as an English-language theorem and each safety +comment as its proof. Reject hand-waving, folklore, hidden assumptions, and +proof by testing. + +## Establish the Exact Claim + +Unless the user specifies a narrower claim, establish: + +> For the exact audited source snapshot, every supported compilation +> configuration, every valid in-scope use in a context satisfying all +> out-of-scope safety obligations preserves freedom from Rust undefined behavior +> under the documented Rust abstract semantics, and every mandatory in-scope +> documented postcondition holds, assuming only the explicitly recorded trusted +> computing base (TCB). + +Interpret valid use as follows: + +- For a safe API, quantify over every well-typed safe use. Impose no hidden + safety precondition. +- For an unsafe API, quantify over every use satisfying all documented initial, + ongoing, and terminal safety obligations. +- For a binary or other entrypoint, quantify over executions satisfying the + explicitly recorded deployment assumptions. Do not transfer those + assumptions silently to a safe library API. + +Prove every documented postcondition of each unsafe API in scope and every +documented guarantee consumed by an in-scope soundness proof. Include broader +safe-API robustness only when the user or audit scope requests it. + +Prove source-level Rust soundness first. State claims about a particular +compiler backend, binary, platform, security property, probability, or +deployment separately with their additional premises. + +## Recover the Required Domain + +Before proving the claim body or issuing a full verdict, derive the exact +domain quantified by the claim. Let a `case` retain every relevant dimension of +one valid use or execution: artifact, toolchain, configuration, input, state, +time, and any other dimension on which an obligation or premise can vary. Let +`Required(case)` denote the cases the claim requires and `Covered(case)` hold +exactly where every obligation the claim requires for that case has a complete +derivation from applicable premises. A local argument may project onto fewer +dimensions only when it proves the lemma for every required case in each +omitted-dimension fiber—parametrically or by proving those dimensions +irrelevant—and must restore the full case before claim-level closure. + +Within one obligation, valid case lemmas may be unioned. Across distinct +obligations, claim-level coverage is their pointwise conjunction—not a union of +regions in which different obligations happened to be proved. + +- Preserve the controlling domain expressions symbolically, including ranges, + unions, exclusions, quantifiers, and conditional or moving policies. Record + their exact sources and audit cutoff. +- If applicable project sources conflict or materially underdetermine support, + obtain an authorized resolution, derive an explicit conservative audit + domain containing every materially supported candidate predicate, or leave + the affected combined claim `UNPROVED`. Do not call a conservative audit + domain the resolved project promise. +- Treat every asserted set relationship, normalization, enumeration, + partition, exclusion, and policy merge as a proof step. Prove the definition + of the exact relation asserted. For example, `A = B` needs `A ⊆ B` and + `B ⊆ A`; `A ⊊ B` needs `A ⊆ B` and a witness `w ∈ B \ A`; + incomparability needs witnesses `a ∈ A \ B` and `b ∈ B \ A`, with each + membership and nonmembership proved. Equivalent symbolic derivations are + acceptable, but required witnesses must remain explicit. Prove the + required containment before using a conservative superset and + `Required ⊆ Covered` before concluding `PROVED`. +- A finite inventory requires evidence both that every listed member belongs + and that no required member is omitted. Endpoints, one representative per + apparent category, CI jobs, lockfiles, and other samples do not prove an + interval or set inventory. +- Prefer a parametric proof over the symbolic predicate when enumeration would + be large or its exact membership is unavailable. Otherwise report proved + regions and the unresolved remainder; do not turn it into an implicit + exclusion. +- An audit cutoff limits the temporal scope of a claim. It does not establish + semantic continuity, enumerate releases before the cutoff, or make sampled + documentation applicable between samples. + +Apply +[configuration recovery](references/configurations-and-generated-code.md#recover-the-required-supported-set) +to derive supported compilation cases and prove every transformation of that +predicate. + +## Use Only Applicable Premises + +- Bottom out Rust-language and standard-library facts in exact applicable text + from versioned Rust Reference or standard-library documentation. +- Quote and link the smallest sufficient set of passages whose explicitly + stated propositions, together with justified inference steps, entail the + fact. Open each citation and verify its wording, qualifications, version, and + scope. A page, allowlist entry, broad label such as “cfg semantics,” or nearby + cited clause does not supply a material proposition the proof never states. +- Attach an applicability domain to every claim and premise, whether stated + locally or inherited from an identified project policy or canonical entry. A + derivation proves only the cases covered by all premises it consumes. +- Apply a guarantee documented for an older Rust release to a later stable + release only when an exact applicable Rust backwards-compatibility + commitment preserves that exact proposition throughout the later release's + relevant domain. An API's stability badge does not by itself preserve every + behavioral statement in its current documentation. Record a + non-authoritative compatibility premise explicitly in the TCB. Never infer + an earlier-version guarantee merely from later documentation. +- Do not promote this skill, the Rustonomicon, Unsafe Code Guidelines, RFCs, + blogs, issue discussions, implementation behavior, Miri, or common practice + to Rust axioms. Use them to discover risks and authoritative text, or record + the exact additional proposition as a TCB assumption. +- Trust a deliberately selected safe dependency API to behave as documented + only when that exact trust is explicit in the TCB. Do not extend this + exception to caller-controlled safe code, callbacks, values, or safe trait + implementations. +- Audit a third-party unsafe API through to admissible premises or record its + exact implementation and contract as an additional TCB assumption. + +When no admissible direct or derived proof can be completed because +authoritative documentation is ambiguous or insufficient, identify the +smallest missing proposition. Do not repair it with intuition. Report a +documentation gap and suggest an upstream improvement when appropriate. + +## Compose Proofs Locally and Literally + +- Identify the controlling contract independently of the existing safety + comment. Distinguish normative contract text from examples, rationale, + implementation comments, and inferred design intent. +- Read the controlling contract according to its actual text. Decompose every + applicable conjunction, implication, quantifier, temporal clause, + precondition, and postcondition into separately reviewable obligations. Do + not replace a literal requirement with an operationally similar property. + Give every normative clause a disposition even when no known consumer uses + it. +- Reify every fact used nonlocally as a named contract or invariant carried by a + type, field, function boundary, guard, typestate, lock, token, or other + locally checkable mechanism. A function contract about global state is an + acceptable degenerate case. +- Prove that each state transition establishes, preserves, transfers, + deliberately suspends under an explicit obligation, or discharges every + applicable invariant. At each consumer, prove that the current invariant + entails the exact needed precondition. +- Trace dataflow across calls and time rather than limiting review to lexical + unsafe blocks. Account for every producer, transition, and consumer. +- Do not promote a producer's preconditions into a universal invariant of its + output type. Any type- or abstraction-wide conclusion needs a complete + derivation independent of that invalid reversal—for example, applicable + authoritative premises, construction-and-preservation closure under an + enforced boundary, or an admissible explicit TCB premise. Local checks and + other applicable derivations may instead prove the proposition for the + particular consumed values or quantified subset. +- For new code, place invariant-bearing representation in the smallest + practical leaf module, keep safely accessible representation fields private + to it, and treat safe code outside that module—including the rest of the same + crate—as untrusted. + +## Follow the Proof Workflow + +1. **Frame the claim.** Record the artifact identity, exact scope, valid uses or + executions, mandatory postconditions, TCB, exclusions, and whether design + alternatives are requested. +2. **Frame the full case domain.** Identify every relevant dimension and + preserve the controlling expressions, sources, candidate relationships, and + unresolved conflicts. State the proposed `Required` domain and how proof + cases will retain every dimension. +3. **Inventory surfaces and transformations.** Enumerate every in-scope safe + and unsafe API surface, obligation site, invariant + producer/transition/consumer, and each material stage and alternative exit + by which build or generation inputs can affect the theorem domain, a + consumed premise, shipped artifacts or selected source, reachability, or an + in-scope postcondition. +4. **State atomic obligations and premises.** Obtain each controlling contract, + decompose it literally, and state the exact proposition and applicability to + prove. Classify every material premise and identify its exact source. +5. **Construct the derivation.** Derive `Required`, every asserted domain + relationship, and every claim conjunct from checked local facts, named + invariants, applicable authoritative axioms, tool-derived theorems, or + explicit TCB entries. Unfold definitions and composite transformations; + preserve material operation order and alternative exits; seek indirect + multi-premise derivations; and justify every intermediate inference. +6. **Close, lint, and challenge.** Give every literal contract clause and safe + surface a disposition and establish domain closure. Reverse-trace each + conclusion used by a verdict or regional result through every material + inference to explicit, applicable premises; reconcile every Rust premise + with its checked quotation and link; and ensure no later-stage fact is + consumed on a path that exited earlier. Then try to falsify the domain + recovery, contract reading, derivations, and coverage with boundary and + adversarial cases derived from the actual clauses. +7. **Certify and report.** Apply the quantifier-sensitive certificates below. + Keep every unresolved obligation visible and state the smallest missing + implication. Record proofs, TCB, coverage, findings, postcondition failures, + documentation gaps, and residual scope without optimism. + +## Write and Review Proof-Grade Documentation + +Read [proof-obligations.md](references/proof-obligations.md) before authoring or +reviewing an unsafe contract, invariant, `SAFETY` comment, or local proof. + +Keep each proof adjacent to the smallest cohesive unsafe operation or assertion. +State the exact operation and its preconditions, cite checked facts and named +invariants, show the derivation, and prove resulting postconditions and +invariant state on every applicable exit. + +When existing code can be validated only by reconstructing a material +derivation absent from its safety comment, do not accept it silently. Include +the reconstructed derivation—or the smallest missing portion—in the review, +with its citations and applicability. Classify implementation correctness +separately from proof-documentation quality. If changes are authorized, improve +the adjacent proof; otherwise provide proposed wording. Do not use a +reconstructed implementation proof to invent or strengthen a caller-facing +contract retroactively. + +## Close API and Configuration Boundaries + +Read +[api-boundaries-and-evolution.md](references/api-boundaries-and-evolution.md) +for fields, constructors, methods, traits, sealing, macros, public or hidden +APIs, robustness, or contract evolution. + +Apply this mandatory safe-surface checklist: public fields, constructors, safe +methods, safe trait methods, and macro-generated APIs all count as safe API +surfaces. Include language-reachable `#[doc(hidden)]` safe items for soundness +even when excluded from documentation or compatibility promises. + +Treat caller-provided safe code as adversarial within the behaviors permitted +by safe Rust and its types. Seal a trait or make it unsafe when soundness +requires an unenforced implementer behavior. + +Read +[configurations-and-generated-code.md](references/configurations-and-generated-code.md) +for every full audit and whenever supported-toolchain policy, conditional +compilation, targets, generated code, FFI, assembly, SIMD, allocators, linking, +or build tooling is relevant. +Every supported combination of compilation options that can ship downstream +must be sound. Use parametric proofs or exhaustive partitions when literal +enumeration would explode; do not substitute a tested sample. + +## Evaluate Trust and Evidence + +Read [tcb-and-evidence.md](references/tcb-and-evidence.md) for every full audit +and whenever a proof uses dependencies, external specifications, tools, +testing, formal verification, environmental restrictions, or cryptographic or +probabilistic assumptions. + +Judge evidence by the exact proposition it establishes, its artifact and model, +its quantified domain and bounds, its premises, and its residual trust—not by a +label such as testing, static analysis, model checking, or formal verification. + +## Design for Provability When Requested + +Read [abstraction-design.md](references/abstraction-design.md) when the user asks +to design, refactor, or reconsider an unsafe abstraction, or when authoring a +new unsafe abstraction. + +Judge existing code under its current source and controlling contract. Inferred +intent or a preferable model may guide a separate proposal but may not narrow, +reinterpret, or discharge a current obligation. Treat implemented changes as a +new artifact and audit them anew. + +## Use Exact Verdicts + +Read [audit-reporting.md](references/audit-reporting.md) before delivering a +persistent or full audit. + +| Verdict | Required certificate | +|---|---| +| **PROVED** | Every obligation for the exact named claim has a checked derivation over its complete applicability, `Required ⊆ Covered`, and every premise is proved from admissible sources or appears as an accepted entry in the stated TCB. | +| **UNPROVED** | A required derivation, premise, applicability or domain-closure argument, postcondition proof, or citation remains missing, ambiguous, circular, or unverifiable, and no applicable existential refutation below is complete. | +| **UNSOUND** | There exists a proved valid in-scope use or execution which reaches an executed operation or semantic event, its exact required safety proposition is false there, and applicable authoritative semantics—possibly together with an explicit TCB premise about the implementation—entails undefined behavior. | +| **CONTRACT-BROKEN** | There exists a proved valid in-scope execution which, considered as a whole, contains no undefined behavior and falsifies a documented postcondition. | + +Failure to prove a universal obligation is enough for `UNPROVED`; do not invent +a counterexample. Conversely, once all parts of an existential UB certificate +are proved, report the scoped soundness claim `UNSOUND`; do not continue to +demand a universal positive lemma and dilute the result to `UNPROVED`. A +violation of user-authored safety prose is not by itself a runtime UB event: +trace the certificate through applicable contracts to the exact authoritative +or explicitly trusted UB consequence. + +An existential certificate closes the universal verdict but does not excuse +omitting another in-scope surface, operation, contract clause, or mandatory +postcondition. Continue the inventory and give each independent obligation a +disposition. Do not claim that a proved or affected region is exact or maximal +unless its full case-domain equality is established; maximal positive remainder +characterization is required only when the audit scope requests it. + +Classify a witness using the execution as a whole, not observations from a +prefix of an execution that later reaches undefined behavior. An +undefined-behavior-containing execution can witness `UNSOUND` but cannot +establish the existential claim required for `CONTRACT-BROKEN`. If it is the +only behavioral evidence, report soundness as `UNSOUND` and the postcondition +as `UNPROVED`. An independent UB-free witness or equivalent existence proof +may establish `CONTRACT-BROKEN`; separate proofs may therefore establish both +verdicts. + +Apply verdicts separately to soundness, documented postconditions, and +conditional application claims. State exact scope, applicability, and TCB +beside every verdict. For every affirmative claim spanning multiple Rust +releases, identify a parametric proof, an exhaustive applicable partition, or +an exact proposition-preserving compatibility premise whose covered domain +contains the claimed release set. Never substitute endpoints, sparse samples, +an audit cutoff, “looks sound,” “probably sound,” or test success. + +For a persistent audit, complete: + +- [tcb-audit-log-template.md](assets/tcb-audit-log-template.md) +- [unsafe-code-audit-report-template.md](assets/unsafe-code-audit-report-template.md) + +For an inline review, provide the equivalent material compactly. Reuse an +existing canonical project log rather than creating a competing trust model. diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/agents/openai.yaml b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/agents/openai.yaml new file mode 100644 index 0000000000..0f0ca3e7b1 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Unsafe Rust Authoring and Audit" + short_description: "Prove, audit, and redesign unsafe Rust" + default_prompt: "Use $unsafe-rust to author, audit, or redesign this unsafe Rust abstraction and its safety contracts." diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/assets/tcb-audit-log-template.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/assets/tcb-audit-log-template.md new file mode 100644 index 0000000000..a79f3f880d --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/assets/tcb-audit-log-template.md @@ -0,0 +1,108 @@ +# TCB Audit Log: `` + +## Identity + +- **Log ID/revision:** `` +- **Audit/report:** `` +- **Skill revision:** `` +- **Source snapshot:** `` +- **Generated artifacts:** `` +- **Rust/toolchain scope:** `` +- **Supported configuration predicate:** `` +- **Theorem(s) supported:** `` +- **Owner/reviewer:** `` +- **Reviewed at:** `` + +## Trust Policy + +`` + +## Entry Index + +| ID | Category | Exact trusted proposition | Identity/version | Scope/configurations | Contract channel | Consumers | Disposition | Re-audit trigger | +|---|---|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | `` | `` | + +## Detailed Entries + +### `` — `` + +- **Category:** `` +- **Disposition:** `` +- **Exact proposition:** `` +- **Quantification and scope:** `` +- **Exact identity:** `` +- **Source/contract:** `` +- **Relevant quotation:** + > `` +- **Contract relationship:** `` +- **Why needed:** `` +- **Why admission is permitted:** `` +- **Consumers:** `` +- **Verification performed:** `` +- **Residual trusted components:** `` +- **Known limitations:** `` +- **Owner/approver:** `` +- **Re-audit trigger:** `` +- **Notes:** `` + +## Dependency Contract Summary + +| Dependency | Safe/unsafe surface | Exact behavior relied upon | Contract relationship | Features/configuration | Implementation audit or TCB entry | Update trigger | +|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | + +## Rejected or Unresolved Premises + +| Proposed ID | Proposition | Reason rejected/unproved | Blocked obligations | Required resolution | +|---|---|---|---|---| +| `` | `` | `` | `` | `` | + +## Review Attestation + +- [ ] The audit's material semantic premises were reconciled against this + index; every Rust axiom or admitted proposition it consumes appears with + its exact source and applicability. +- [ ] Every consumed entry has an exact proposition rather than a vague trust + statement. +- [ ] No entry merely assumes an in-scope conclusion or trusts code the declared + audit scope purports to prove. +- [ ] Every identity, version, digest, and configuration scope was checked. +- [ ] Every quotation was opened and verified in context. +- [ ] Selected safe dependencies are distinguished from caller-controlled code. +- [ ] Exact pins are used only to freeze identity; every undocumented + proposition has an audit, additional contract, or explicit admission. +- [ ] Every third-party unsafe implementation is recursively audited or + explicitly admitted. +- [ ] Every version-spanning compatibility entry states the exact proposition + preserved and its exact release/configuration region; pins, stability + badges, and sampled documentation are not used as interval coverage. +- [ ] Tool-derived facts state their exact theorem and residual TCB. +- [ ] External, deployment, and probabilistic assumptions qualify the verdict + conspicuously. +- [ ] Every entry has consumers and a re-audit trigger. +- [ ] Every consumed entry supporting `PROVED` is explicitly accepted; pending, + rejected, and superseded entries support no proved claim. +- [ ] Rejected and unresolved premises appear in the audit findings. + +**Reviewer:** `` +**Review result:** `` +**Date:** `` diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/assets/unsafe-code-audit-report-template.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/assets/unsafe-code-audit-report-template.md new file mode 100644 index 0000000000..a3ef0c8619 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/assets/unsafe-code-audit-report-template.md @@ -0,0 +1,257 @@ +# Unsafe Rust Audit: `` + +## Claims and Verdicts + +| Claim ID | Exact theorem | Required-domain ID | Verdict | Certificate/proof/finding | TCB and qualification | +|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | `` | + +- **Combined mandatory result:** `` +- **Scope:** `` +- **TCB log:** `` +- **Skill revision:** `` + +## Audited Snapshot + +- **Repository/source:** `` +- **Uncommitted changes:** `` +- **Generated/expanded artifacts:** `` +- **Rust/compiler/stdlib:** `` +- **Dependencies:** `` +- **Build inputs/tools:** `` +- **Prior audit reused:** `` +- **Auditor/reviewer/date:** `` + +## Contracts in Scope + +### Soundness + +`` + +### Documented Postconditions + +| Contract ID | API/entrypoint | Preconditions | Postconditions | Source/version | +|---|---|---|---|---| +| `` | `` | `

` | `` | `` | + +### Additional Robustness Claims + +| Claim ID | Exact proposition | Scope and authority | Result | Evidence/finding | +|---|---|---|---|---| +| `` | `` | `` | `` | `` | + +`` + +## Boundary and API Coverage + +| Surface ID | Item/generated family | Safe/unsafe | Construction/access path | Configuration scope | Contract/proof status | +|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | + +Confirm coverage of: + +This is a mandatory minimum, not an exhaustive surface list. Apply +[Enumerate every surface](../references/api-boundaries-and-evolution.md#enumerate-every-surface) +and record every additional language-reachable surface in the table above. + +- [ ] safely accessible representation across the owning-module boundary, + including `pub(super)`, `pub(crate)`, ancestor-visible, and generated + access; +- [ ] public fields; +- [ ] constructors; +- [ ] safe methods; +- [ ] safe trait methods and caller-provided implementations; +- [ ] macros and macro-generated APIs; +- [ ] reexports and configuration-specific APIs; +- [ ] language-reachable `#[doc(hidden)]` safe items; +- [ ] associated items, safe free functions/statics, callbacks, FFI entrypoints, + blanket/default/auto-trait behavior, operators, and destruction whenever + language-reachable or semantically relevant. + +## Invariant Inventory + +| Invariant ID | Exact proposition | Owner/boundary | Must hold when | Producers/mutators | Consumers | Status | +|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | + +## Obligation Ledger + +| Obligation ID | Source/API | Exact proposition | Required domain | Derivation, material premises, and their applicability | Covered domain/cases | Proof location | Reviewer | Status | +|---|---|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | `` | `` | + +## Theorem-Domain and Configuration Closure + +### Required Domain Recovery + +| Step ID | Controlling source expression or prior predicate | Derived predicate/inventory/partition | Relation to prove (write symbolically) | Required certificate: containments/witnesses/derivation | Status | +|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | + +- **Audit cutoff:** `` +- **Exact `Required` predicate:** `` +- **Configuration projection:** `` +- **Policy conflicts/authorized resolution:** `` +- **Unresolved domain:** `` + +### Build and Generation Pipeline + +`` + +| Stage ID | Input/state region and predecessor | Ordered operation or transformation | Successful output/effect | Alternative exit and partial effects | Authority/TCB and applicability | Consumer | +|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | + +- **Freshness/invalidation:** `` + +### Covered Domain + +- **Discovered axes:** `` +- **Configuration-fiber proof:** `` +- **Exact full-case `Covered` predicate:** `` +- **Coverage proof:** `` +- **Closure certificate:** `` +- **Version-spanning premise basis:** `` +- **Generated artifacts:** `` +- **Enforced exclusions:** `` +- **Sampled/tested configurations:** `` +- **Uncovered full cases/configurations:** `` + +## TCB Summary + +| Category | Entry IDs | Human disposition | Material limitations | +|---|---|---|---| +| `` | `` | `` | `` | + +Full log: `` + +## Tool-Derived Evidence + +| Proof ID | Proposition and entailment | Artifact/tool/model/options | Quantification and bounds | Non-vacuity and semantic fidelity | Trust, stubs, and residual TCB | Result/certificate | Consumers | +|---|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | `` | + +## Findings + +### `` — `` + +- **Status/severity:** `` +- **Implementation classification:** `` +- **Proof-artifact classification:** `` +- **Affected claim:** `` +- **Source/API/configuration:** `` +- **Required proposition:** `` +- **Existing proof or behavior:** `` +- **Reconstructed derivation:** `` +- **Proposed proof-artifact repair:** `` +- **Defect:** `` +- **Authority/TCB involved:** `` +- **UB certificate — valid use:** `` +- **UB certificate — reachability:** `` +- **UB certificate — false safety proposition:** `` +- **UB certificate — consequence:** `` +- **Defined postcondition refutation:** `` +- **Affected producers/consumers:** `` +- **Required resolution:** `` +- **Compatibility impact:** `` +- **Re-audit scope:** `` + +## Abstraction Design (Optional) + +`` + +- **Required behavior and constraints:** `` +- **Current literal result:** `` +- **Recommended candidate:** `` +- **Proof simplification:** `` +- **Behavior delta:** `` +- **Compatibility and migration:** `` +- **Fresh-audit status:** `` + +## Documentation and Skill Gaps + +### Authoritative Rust Documentation + +| Gap ID | Missing/ambiguous proposition | Attempted authoritative sources | Blocked obligations | Suggested upstream report | +|---|---|---|---|---| +| `` | `` | `` | `` | `` | + +### Skill Guidance + +| Gap ID | Omission or ambiguity | Audit impact | Proposed maintainer follow-up | +|---|---|---|---| +| `` | `` | `` | `` | + +## Residual and Excluded Scope + +`` + +## Re-audit Triggers + +- `` +- `` +- `` +- `` +- `` + +## Final Attestation + +- [ ] Every in-scope obligation has a status. +- [ ] Every conclusion used by a verdict or regional result and every claimed + set relationship satisfies [Make every derivation + reviewable](../references/proof-obligations.md#make-every-derivation-reviewable). +- [ ] Every controlling domain expression is preserved, and every asserted set + relationship, normalization, enumeration, partition, merge, or exclusion + has the certificate required by its exact relation. +- [ ] Every materially relevant build/generation path records required order, + exits, and partial progress and proves each consumed tool-interpretation, + output, and freshness proposition; no later-stage fact is used on a path + that exited earlier. +- [ ] Every verdict has the certificate required by `SKILL.md`, including + `Required ⊆ Covered` for `PROVED` and every existential link for + `UNSOUND` or `CONTRACT-BROKEN`. +- [ ] Every material derivation reconstructed during review is exposed with its + applicability, and deficient proof artifacts are reported separately. +- [ ] Every material semantic premise appears in the authority/TCB inventory, + and every consumed citation and TCB entry was independently verified. +- [ ] Every consumed TCB entry supporting `PROVED` has an accepted human + disposition. +- [ ] Every mandatory documented postcondition was reviewed in addition to UB + freedom. +- [ ] Residual scope and conditional assumptions are conspicuous. +- [ ] The final verdict does not rely on lack of a counterexample or clean tests. + +**Auditor:** `` +**Independent reviewer (if performed):** `` +**Date:** `` diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/abstraction-design.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/abstraction-design.md new file mode 100644 index 0000000000..b477d8022d --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/abstraction-design.md @@ -0,0 +1,164 @@ +# Designing Unsafe Abstractions for Provability + +## Contents + +- [Keep verification and design separate](#keep-verification-and-design-separate) +- [Establish design requirements](#establish-design-requirements) +- [Extract the minimum capability](#extract-the-minimum-capability) +- [Generate proof-oriented candidates](#generate-proof-oriented-candidates) +- [Prove and compare candidates](#prove-and-compare-candidates) +- [Report the result](#report-the-result) + +## Keep Verification and Design Separate + +Use this process when the user asks to design or redesign an unsafe abstraction, +or when authoring a new unsafe abstraction. Do not run it automatically during +an immutable acceptance audit unless the user requests design advice. + +Judge existing code under its exact current source and controlling contracts. +Inferred intent, a proposed narrower contract, or an easier-to-prove +representation may not: + +- reinterpret or weaken a current obligation; +- discharge a premise of the current implementation; +- erase or downgrade a current finding; or +- justify accepting the current artifact. + +Keep conclusions about the current artifact logically independent of every +candidate design. A proposal describes a possible future artifact; it has no +`PROVED` verdict. After implementation, identify the new snapshot and apply the +ordinary unsafe-Rust proof workflow anew. + +Preserve at least the scoped current finding that motivates the redesign. Do +not expand that step into a whole-crate audit unless the requested audit scope +requires it. + +For greenfield work, no current-artifact verdict is necessary. State the design +requirements, construct the candidate, and prove the implemented artifact. + +## Establish Design Requirements + +Record the constraints that the abstraction must satisfy: + +- required externally observable behavior and mandatory postconditions; +- current public contracts and compatibility commitments that must remain; +- exact propositions required by relevant consumers; +- supported Rust versions, targets, features, and configurations; +- representation, performance, interoperability, or integration constraints; + and +- which semantic or compatibility changes the user has authorized. + +Use each source only for the proposition it actually establishes. User +requirements can determine desired behavior. Current contracts determine +current obligations. Call sites, tests, names, comments, history, and +implementation structure may suggest intent or establish local source facts, +but an inference about intent is not a Rust semantic premise and does not prove +implementation correctness. + +Known internal consumers do not exhaust the consumers of a public API. Treat +the published contract as a required constraint unless an applicable contract +channel and the user authorize changing it. Surface material ambiguity when +different interpretations would change the public contract, support policy, or +compatibility result. + +## Extract the Minimum Capability + +For each required behavior, state the exact semantic proposition consumers +need. Separate properties that the current abstraction may have bundled, such +as: + +- nominal identity from an operational capability; +- layout from validity, initialization, provenance, alignment, or aliasing; +- metadata from memory projection; +- ownership from access permission; +- one-time establishment from an ongoing invariant; +- safe caller behavior from an unsafe implementer promise; and +- behavior common to many types from one exceptional case. + +Identify where each proposition is established, carried, consumed, and +discharged. Prefer a design in which types, validation, privacy, sealing, +typestate, guards, or other locally checkable mechanisms enforce the fact. + +Do not make a proof easier merely by transferring an unnecessary or hidden +obligation to callers. Every remaining unsafe caller or implementer obligation +must be explicit, sufficient, and justified by a need the implementation cannot +enforce safely. + +## Generate Proof-Oriented Candidates + +Consider the smallest transformations that remove the unsupported premise: + +- eliminate an unnecessary unsafe operation, impl, configuration, or promise; +- validate the required property before the unsafe operation; +- narrow an API or implementation to the cases actually supported; +- reuse a safe or already-proved primitive whose contract matches exactly; +- specialize a one-off case instead of inventing a generic abstraction; +- split independent capabilities or invariant dimensions; +- seal an implementer boundary or move representation behind a smaller module; + or +- introduce a new reusable abstraction only when demonstrated consumers share + the same semantic capability. + +For example, if one contract claims both nominal field reflection and pointer +projection while some consumers require only projection, consider separating +those capabilities rather than inventing a nominal field. This is a design +prompt, not a Rust fact; prove the resulting contracts normally. + +Do not pad the output with cosmetic or strictly dominated alternatives. When +requirements are ambiguous or viable candidates make materially incomparable +tradeoffs, present the consequential choice instead of choosing silently. + +## Prove and Compare Candidates + +For each viable candidate, state: + +- exact safe and unsafe contracts; +- representation and named invariants; +- how every required consumer proposition is supplied; +- where each remaining obligation is enforced; +- authoritative axioms, dependency contracts, and TCB entries required; +- supported applicability domain; +- unresolved proof obligations; and +- behavior, compatibility, migration, and re-audit consequences. + +Construct a conditional proof plan before implementation. After implementation, +prove the exact source rather than the design sketch. + +Reject candidates that fail required behavior, proof closure, supported-domain +coverage, or binding compatibility constraints. Among the remainder, prefer a +candidate that preserves required behavior while reducing one or more of: + +- unsafe surface exposed to callers or implementers; +- strength or number of unsupported premises; +- invariant access region, lifetime, and fan-out; +- TCB size; +- coupling between independent capabilities; +- version- or configuration-specific proof branches; +- accidental representation commitments; and +- genericity without demonstrated reuse. + +Also account for authorized implementation, performance, and migration costs. +Do not collapse incomparable tradeoffs into an invented score, and do not +prefer a small textual diff that silently weakens a relied-upon contract. + +Apply +[Evolve contracts deliberately](api-boundaries-and-evolution.md#evolve-contracts-deliberately) +to every candidate contract change. + +## Report the Result + +Keep these outputs distinct whenever they apply: + +1. **Current artifact:** Exact findings and verdict under the current contract. +2. **Design requirements:** Required behavior, constraints, consumer + propositions, and unresolved intent. +3. **Candidate design:** Exact proposed contracts, invariant model, proof plan, + and remaining premises. +4. **Compatibility and migration:** Behavior gained or lost, affected callers + and implementers, contract channel, and re-audit scope. +5. **Recommendation:** The preferred candidate and any human decision required. +6. **Post-change audit:** A separate result for an implemented new snapshot. + +In review-only work, provide counterfactual advice without modifying source. In +authoring work, update implementation, contracts, local proofs, TCB entries, +and affected downstream proofs together. diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/api-boundaries-and-evolution.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/api-boundaries-and-evolution.md new file mode 100644 index 0000000000..8a27c298f6 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/api-boundaries-and-evolution.md @@ -0,0 +1,285 @@ +# API Boundaries, Invariants, and Contract Evolution + +## Contents + +- [Enumerate every surface](#enumerate-every-surface) +- [Place the safety boundary](#place-the-safety-boundary) +- [Use module privacy](#use-module-privacy) +- [Handle unsafe fields](#handle-unsafe-fields) +- [Audit traits and sealing](#audit-traits-and-sealing) +- [Audit macros and hidden APIs](#audit-macros-and-hidden-apis) +- [Distinguish selected dependencies from caller code](#distinguish-selected-dependencies-from-caller-code) +- [Prove documented behavior](#prove-documented-behavior) +- [Evolve contracts deliberately](#evolve-contracts-deliberately) + +## Enumerate Every Surface + +For soundness, enumerate every language-reachable way untrusted safe code can +construct, obtain, observe, mutate, replace, borrow, move, copy, drop, implement, +or invoke the abstraction. + +Apply this checklist explicitly: + +- public fields; +- constructors, including literals, constants, defaults, conversions, + deserialization, builders, and generated constructors; +- safe inherent and extension methods; +- safe trait methods, blanket implementations, default methods, trait objects, + and auto traits; +- public associated types and constants where their choices affect unsafe code; +- safe free functions and statics; +- indexing, dereference, iteration, operators, formatting, cloning, comparison, + hashing, panic, and destruction behavior when implemented; +- exported declarative macros, procedural macros, derives, attributes, and APIs + produced by them; +- reexports and feature- or target-dependent public items; +- callbacks and user-provided implementations invoked internally; +- FFI entrypoints callable without a Rust-side unsafe obligation; +- language-reachable `#[doc(hidden)]` items. + +This is an advisory discovery list, not an exhaustive statement of Rust's +semantics. Inspect the exact source, expansions, metadata, and applicable +authoritative documentation for additional surfaces. + +For each safe surface, prove that every behavior available to well-typed safe +code preserves soundness. For each unsafe surface, prove that its complete +documented contract is sufficient and that its implementation establishes all +documented postconditions for every valid use. + +Determine the controlling contract from the actual published or otherwise +applicable normative text. Examples, rationale, tests, names, existing safety +comments, and inferred design intent may aid discovery but may not narrow or +replace that contract. + +## Place the Safety Boundary + +Mark an operation unsafe when callers or implementers must establish a +soundness-critical proposition that the implementation cannot establish from +enforced types, checked state, module-owned invariants, and deliberately trusted +dependencies. + +Do not expose a safe API with a prose-only safety precondition. Documentation +cannot make a well-typed safe use invalid for the purpose of soundness. + +Conversely, do not move an obligation to callers merely because doing so is +convenient. A safe wrapper may discharge an unsafe callee's requirements with +validation, construction, privacy, typestate, synchronization, or a local proof. + +Treat each unsafe declaration or call as a contract boundary. An unsafe helper +can propagate an obligation through fields and later calls without immediately +performing an operation that exhibits undefined behavior. Follow the obligation +through the dataflow until it is discharged. + +An `unsafe impl` is an assertion that the implementation satisfies the unsafe +trait's contract. Prove that assertion and every method-level obligation. + +For FFI declarations, distinguish the declaration-time assertion that the +foreign contract is correct from each call's preconditions and from the foreign +implementation's behavior. Record external ABI and implementation trust +explicitly. + +## Use Module Privacy + +For new invariant-bearing representations: + +1. Put the representation and all safely accessible fields in the smallest + practical leaf module. +2. Keep those fields private to that module. +3. Make all code outside the module—including parents, siblings, cousins, and + the rest of the same crate—use checked safe APIs or documented unsafe APIs. +4. Treat each operation inside the module that can affect the invariant as a + proof site. + +Do not use `pub(super)`, `pub(crate)`, or another broad safe visibility merely +because current same-crate code is trusted socially. Such visibility expands +the region in which safe edits can silently violate the invariant and makes +human review materially harder. + +Existing crates need not be rejected solely for violating this authoring +discipline. Compute and audit the actual Rust visibility region, including +fields in ancestors or descendants that the code can access and all code that +can access the representation. Report broad safe visibility as proof-surface +debt. + +Represent every distant fact by a named invariant or contract that each producer +preserves and each consumer can use locally. + +## Handle Unsafe Fields + +When the exact audited Rust version supplies compiler-enforced unsafe fields, a +properly declared unsafe field is an explicit unsafe API boundary. It may have +any intentional visibility, analogously to an unsafe function, because untrusted +safe code cannot perform the gated uses without accepting its documented +obligations. + +Require field documentation to make the obligations for all applicable +operations derivable, including: + +- initialization and replacement; +- reads, copies, and moves; +- shared and mutable borrows; +- pattern matching, destructuring, aggregate update, and whole-value operations; +- writes through direct access or an escaped capability; +- transfer or suspension of the enclosing invariant; +- the state required before control returns to untrusted safe code. + +Audit the exact compiler version's enforcement rather than assuming a proposed +or future design. Separately prove every implicit safe action not gated by field +projection, especially destruction and compiler- or derive-supplied trait +behavior. An unsafe modifier does not relax the language validity invariant of +the field's Rust type and does not make arbitrary drop glue conditional. + +When authoritative Reference or standard-library text does not specify the +feature sufficiently, record the exact semantics relied upon as a documentation +gap and explicit TCB premise. An RFC or current implementation may explain the +intent but is not a Rust axiom under this skill's authority policy. + +## Audit Traits and Sealing + +Treat every safe trait implementation supplied by a caller as adversarial safe +code. Unsafe code may rely only on facts enforced by Rust's types and semantics, +module-owned state, or explicit TCB entries—not on a caller faithfully +implementing behavioral prose. + +If unsafe code requires an implementer to uphold a soundness-critical +obligation, use one of these structures: + +- make the trait unsafe and document the complete implementer contract; +- seal the trait so only deliberately controlled implementations are possible; +- validate the needed property before unsafe use; +- redesign the representation or boundary so the property follows locally. + +Prove that sealing is effective under Rust privacy and name resolution for every +supported configuration and macro expansion. A documentation claim, +`#[doc(hidden)]`, obscure path, or conventional “sealed” name does not by itself +prevent downstream implementations. + +For an unsafe trait: + +- state representation and behavioral obligations at the trait and method + levels; +- prove every in-scope `unsafe impl`; +- ensure safe methods remain sound for every valid implementation; +- ensure generic unsafe consumers rely on no stronger fact than the contract; +- audit associated types, constants, default methods, specialization, trait + objects, auto traits, negative impls, and generated impls when applicable. + +For a sealed safe trait, selected implementations may be audited as controlled +code, but downstream safe callers remain adversarial. Recheck sealing whenever +visibility, reexports, macros, or configuration changes. + +## Audit Macros and Hidden APIs + +Classify a macro invocation by the obligations rustc actually enforces for the +expanded use, not merely by the absence or presence of `unsafe` in the invocation +tokens. A macro can be constructed so that expansion succeeds only in an unsafe +context. If no caller-side unsafe obligation is compiler-enforced, treat the +macro as a safe API and prove every accepted safe invocation sound. + +Auditing only handwritten macro or proc-macro source is insufficient when sound +output depends on: + +- caller tokens, types, paths, hygiene, spans, or name resolution; +- `cfg`, features, target facts, environment, or build-script data; +- generated identifiers, item visibility, attributes, or impl selection; +- compiler expansion order or version; +- downstream code into which the macro expands. + +Inspect expansions to discover API and caller obligations. Then apply +[Prove build and generation pipelines](configurations-and-generated-code.md#prove-build-and-generation-pipelines) +to prove closure over every supported accepted input, output, and +configuration. Include generated public APIs in the same safe/unsafe surface +audit as handwritten items. + +Treat `#[doc(hidden)]` as a documentation and compatibility signal only to the +extent promised by the project. It does not create Rust privacy. A +language-reachable safe hidden item must remain sound for direct safe use and +may not hide a safety precondition. The project may separately exclude its +behavior or continued existence from SemVer promises. + +## Distinguish Selected Dependencies From Caller Code + +A deliberately selected dependency is code whose use and version the project +author intentionally chose. A function argument, callback, generic parameter, +trait object, plugin, implementation of a safe trait, or downstream macro input +is caller-controlled even when its type originates in a selected dependency. + +Apply the selected-safe-dependency exception only to the deliberately chosen +implementation and documented API behavior, never to behavior chosen by the +caller. Determine whether reexports, dependency-defined traits, feature +unification, or plugins move a surface across that boundary. + +For exact identity, contract channels, safe versus unsafe dependency trust, and +update triggers, apply +[Record dependency contracts](tcb-and-evidence.md#record-dependency-contracts). + +## Prove Documented Behavior + +Soundness is the minimum universal property. The mandatory postcondition scope +includes every documented postcondition of an unsafe API in scope and every +guarantee consumed by an in-scope soundness proof. Prove broader safe-API +behavior only when the user or audit explicitly places it in scope. + +At minimum, an unsafe API implementation is responsible for both: + +1. avoiding undefined behavior for every valid use; and +2. establishing every documented postcondition when its safety preconditions + and other documented conditions are met. + +Evaluate postconditions independently under the verdict certificates in +`SKILL.md`, then determine whether a proved broken guarantee can make downstream +unsafe consumers unsound. + +Do not invent a universal standard for undocumented robustness. State the exact +behavioral claim being reviewed: panic freedom, determinism, resource bounds, +constant time, atomicity, rollback, leak freedom, progress, or another property. +Record its authority and scope separately from Rust soundness. + +## Evolve Contracts Deliberately + +Treat safety documentation and documented postconditions as compatibility +contracts, not comments that can be edited independently of code. + +Analyze every change by provider and consumer: + +- Strengthening a caller precondition invalidates previously valid calls. +- Weakening a caller precondition admits more calls and increases the + implementation's proof burden. +- Weakening a provider postcondition invalidates existing caller reasoning. +- Strengthening a provider postcondition increases what callers may rely upon. +- Strengthening an unsafe trait implementer's obligation can invalidate existing + impls. +- Strengthening guarantees required from trait implementations can likewise + invalidate existing impls even when it benefits trait consumers. +- Weakening guarantees supplied through a trait can invalidate generic + consumers. + +Under a conventional SemVer contract, invalidating existing valid callers, +implementers, or documented reasoning is normally breaking even when Rust type +signatures do not change. Determine and record the actual project's +compatibility policy rather than treating SemVer folklore as an axiom. + +An exact pin freezes identity but does not authorize an undocumented semantic +claim. A fork, out-of-band agreement, or consumer-specific promise may supply an +additional contract for its exact recorded scope; otherwise audit or explicitly +admit the implementation proposition. Update the TCB and repeat affected proofs +before changing any identity, contract, or agreement. + +When the supported Rust range changes, apply +[Qualify applicability](proof-obligations.md#qualify-applicability), update any +compatibility premises in the TCB, and re-audit every proof whose documentation, +edition, target, feature, configuration, or implementation claim may differ. + +For every contract change, search callers, implementers, safety comments, TCB +entries, generated output, and downstream-facing documentation for proofs that +consume the changed proposition. + +Changing safety prose does not retroactively narrow valid uses of an already +published version. If that version's implementation failed its published +contract, it had a soundness or contract defect. Treat the correction as +remediation requiring compatibility analysis, affected-version disclosure, and +review of downstream proofs—not as proof that the old implementation was sound. + +When redesign is authorized, apply +[Designing Unsafe Abstractions for Provability](abstraction-design.md) without +letting the proposed contract alter the verdict for the current artifact. diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/audit-reporting.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/audit-reporting.md new file mode 100644 index 0000000000..309c3ae271 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/audit-reporting.md @@ -0,0 +1,240 @@ +# Audit Execution and Reporting + +## Contents + +- [Freeze the audit claim](#freeze-the-audit-claim) +- [Maintain an obligation ledger](#maintain-an-obligation-ledger) +- [Reconcile derivations and evidence](#reconcile-derivations-and-evidence) +- [Aggregate verdicts](#aggregate-verdicts) +- [Write actionable findings](#write-actionable-findings) +- [Deliver a complete report](#deliver-a-complete-report) +- [Preserve and update the audit](#preserve-and-update-the-audit) + +## Freeze the Audit Claim + +Before reviewing proofs, record: + +- exact repository, source revision/digest, workspace packages, generated + artifacts, and relevant uncommitted changes; +- controlling support expressions, conflicts or gaps, audit cutoff, authorized + resolution or conservative audit domain, the exact symbolic `Required` + predicate, every transformation used to derive it, and enforced exclusions; +- dependency resolution and relevant source identities; +- API, module, binary, or whole-project scope; +- soundness theorem and documented postconditions in scope; +- TCB log identity/revision; +- prior audit results being reused; +- known inaccessible, unsupported, or intentionally excluded regions. + +Do not issue a whole-crate verdict for a diff, one feature, one target, or one +unsafe block. State the narrow result actually established. + +If the task is review-only, report findings and proposed remedies without +silently changing code. If the task includes authoring or fixing, update the +proof artifacts and contracts together with the implementation. + +## Maintain an Obligation Ledger + +Track every in-scope obligation sufficiently to detect omissions. The ledger may +be a table, issue list, annotated source, or another reviewable form. Ensure it +provides complete location-by-location coverage of producers, transitions, +consumers, and proof sites. + +For each obligation, record: + +- stable identifier and source location/API; +- exact proposition to prove; +- operation, contract, invariant, or postcondition that requires it; +- required applicability domain; +- supporting local facts, invariant clauses, axioms, and TCB entries, with the + applicability of each premise; +- domain actually covered by the derivation and any case partition; +- the appropriate certificate for every asserted set relationship and domain + transformation consumed; +- proof location; +- reviewer verification; +- status and finding link. + +A row may cite a canonical subproof rather than repeat it, but neither a row nor +a proof name may hide a material inference. The record must let a reviewer +reverse-trace the obligation through every intermediate proposition to its +classified and applicable premises. + +Include obligations created by: + +- unsafe operations and unsafe API calls; +- unsafe functions, traits, impls, fields, attributes, declarations, macros, and + generated code as applicable; +- construction, mutation, suspension, consumption, and destruction of + invariant-bearing state; +- safe APIs backed by unsafe code; +- every documented postcondition of each in-scope unsafe API, and every + documented guarantee consumed by later unsafe code; +- FFI, assembly, allocators, concurrency, target/configuration selection, and + external contracts; +- generated public APIs and code shipped downstream. + +This is a discovery aid, not an exhaustive semantic taxonomy. Add whatever the +actual code and authoritative contracts require. + +The ledger complements rather than replaces the proof workflow in +[proof-obligations.md](proof-obligations.md). Review surrounding safe code and +follow changed propositions to every consumer; compiler-marked unsafe locations +and textual diffs are only discovery starting points. + +## Reconcile Derivations and Evidence + +Apply [Make every derivation +reviewable](proof-obligations.md#make-every-derivation-reviewable) before +certifying any verdict or regional result. In a persistent report, ensure the +obligation ledger and its canonical proofs expose that derivation, then +reconcile every material semantic premise they use with the report's authority +and TCB inventories. A valid fact found only by the reviewer may support a +reconstructed implementation proof, but record the deficient report or safety +comment rather than silently repairing its proof artifact. + +## Aggregate Verdicts + +Use the verdict definitions in `SKILL.md` for individual obligations and the +final in-scope claim. + +Report multiple statuses when applicable. For example, soundness can be +`PROVED` while documented postconditions are `CONTRACT-BROKEN`, or one path can +be `UNSOUND` while a different configuration remains `UNPROVED`. Issue `PROVED` +for the combined default claim only when every in-scope soundness and +documented-postcondition obligation is proved. + +Certify each result with the proof shape required by `SKILL.md`. For `PROVED`, +identify the exact `Required` domain, union valid case lemmas within each +obligation, intersect coverage across all claim-required obligations, and prove +`Required ⊆ Covered` for that aggregate predicate. For `UNSOUND`, record every +link from valid use through reachability and a false safety proposition to the +applicable UB consequence. For `CONTRACT-BROKEN`, certify that the falsifying +execution is UB-free as a whole. Otherwise state the smallest gap and use +`UNPROVED`. + +A completed existential refutation fixes its universal verdict but does not +erase independent obligations elsewhere in scope. Continue the surface and +site inventory. Report a positive or affected regional partition only at the +granularity and completeness actually proved; do not imply maximality unless +the task requests it and equality with the exact full-case region is +established. + +Place qualifications in the theorem, not in vague prose. Use: + +> PROVED for `` over ``, relative to TCB +> ``. + +For a deployment, external, or cryptographic premise, name the exact entry and +state whether the result is a conditional source, binary, or application claim. + +Never use “looks sound,” “no issues found,” “probably safe,” “Miri-clean,” +“battle-tested,” or “tests pass” as a verdict. + +## Write Actionable Findings + +Each finding should contain: + +- severity/status and affected theorem; +- exact source/API/configuration; +- required proposition; +- existing claimed proof; +- any material derivation the reviewer had to reconstruct, with citations and + applicability, or the smallest portion still missing; +- proposed replacement proof text when the reviewed artifact omits that + derivation; +- smallest missing, false, circular, or unsupported implication; +- authoritative contract or TCB entry involved; +- for a claimed UB witness, the valid use, executed operation or semantic event, + false required safety proposition, and authoritative or TCB-backed UB + consequence; +- whether a separate UB-free postcondition refutation or equivalent existence + proof is known; +- affected callers, producers, consumers, generated output, and configurations; +- minimal acceptable resolution; +- compatibility and re-audit consequences. + +Distinguish: + +- an implementation defect; +- insufficient or ambiguous safety documentation; +- a correct implementation with an invalid local comment; +- an undocumented TCB assumption; +- an authoritative Reference/std documentation gap; +- a skill-guidance gap; +- a compatibility/robustness defect without established UB. + +A successfully reconstructed implementation proof does not erase deficient +safety documentation. Report the implementation obligation and the proof +artifact separately, and offer corrected proof text. Reconstruction may not add +a hidden caller or implementer obligation or create a provider guarantee absent +from the controlling contract. + +Keep every verdict for the current artifact independent of design alternatives. +If redesign was requested, report proposals and their conditional proof plans +separately; audit an implemented redesign as a new snapshot. + +If authoritative documentation is insufficient, quote the exact missing +proposition and suggest a narrowly scoped upstream report. If this skill failed +to route the reviewer to a necessary check, identify a proposed skill issue +without treating the proposed rule as current authority. + +## Deliver a Complete Report + +A complete audit report contains: + +1. **Claim and verdict:** Exact theorem, status, scope, supported configuration + predicate, and TCB identity. +2. **Snapshot:** Source, generated artifacts, Rust/toolchain, dependency + resolution, and relevant build inputs. +3. **Boundary and API coverage:** Safe and unsafe surfaces crossing the owning + module or external API boundary, including restricted-visible fields, + constructors, safe methods, safe trait methods, macro-generated APIs, and + language-reachable hidden items. +4. **Invariant inventory:** Index of named local contracts, owners, permitted + transitions, and consumers—not an informal global proof. +5. **Obligation coverage:** Proof sites and status summary; link to detailed + proofs/findings rather than duplicating them. Include material reconstructed + proofs missing from the reviewed proof artifacts. +6. **Theorem-domain and configuration closure:** Controlling policy + expressions, full-case symbolic `Required`, its `Required_cfg` projection, + set-relationship and transformation certificates, audit cutoff, axes, staged + build/generation relation, configuration-fiber proofs, `Covered`, claim + closure, premise-version applicability, generated artifacts, enforced + exclusions, and unresolved remainder. +7. **TCB audit log:** Every authoritative or admitted proposition and reviewer + disposition. +8. **Tool-derived evidence:** Exact theorem, artifact/model scope, bounds, + result, non-vacuity check, and residual TCB. +9. **Postcondition/robustness scope:** Documented guarantees proved and any + separately requested properties. +10. **Findings:** `UNPROVED`, `UNSOUND`, `CONTRACT-BROKEN`, documentation gaps, + compatibility defects, and maintenance risks. +11. **Residual scope:** Anything not audited, inaccessible, unsupported, or + conditional. +12. **Review triggers:** Changes that invalidate or require revisiting the + result. + +Use the bundled report and TCB templates for persistent artifacts. For an inline +review, provide the same information compactly. + +## Preserve and Update the Audit + +When a canonical audit or TCB log exists: + +- reuse its identifiers and format; +- verify rather than blindly inherit prior `PROVED` entries; +- update changed source, contracts, configurations, dependencies, and trust; +- retain historical identity through version control rather than duplicating a + stale snapshot; +- record the skill revision used for the audit; +- link proofs and findings to exact source revisions. + +Trigger review when code or documentation changes any consumed proposition, +when supported compilation options expand, when generators or generated output +change, when dependencies or contract channels change, when authoritative Rust +documentation changes materially, or when a new incident reveals an omitted +class of obligation. + +A prior successful audit is evidence about its exact snapshot and theorem, not a +permanent certification of later code. diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/configurations-and-generated-code.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/configurations-and-generated-code.md new file mode 100644 index 0000000000..5790ced732 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/configurations-and-generated-code.md @@ -0,0 +1,411 @@ +# Configuration Closure and Generated Unsafe Code + +## Contents + +- [Recover the required supported set](#recover-the-required-supported-set) +- [Discover configuration axes](#discover-configuration-axes) +- [Prove coverage of the recovered set](#prove-coverage-of-the-recovered-set) +- [Prove build and generation pipelines](#prove-build-and-generation-pipelines) +- [Audit targets, SIMD, and concurrency](#audit-targets-simd-and-concurrency) +- [Audit allocators, panic modes, and assertions](#audit-allocators-panic-modes-and-assertions) +- [Audit FFI, assembly, linking, and global symbols](#audit-ffi-assembly-linking-and-global-symbols) +- [Record configuration coverage](#record-configuration-coverage) + +## Recover the Required Supported Set + +This reference helps recover the configuration projection of the full +`Required(case)` domain defined in `SKILL.md`. Define +`Required_cfg(configuration)` to hold when at least one full required case has +that configuration. This projection organizes support-policy and case proofs; +it does not replace inputs, states, executions, or other dimensions of +claim-level `Required` and `Covered`. + +Preserve each controlling support expression as a precise symbolic predicate +before claiming full soundness. Fix the exact source or packaged artifact and +audit cutoff. Let each predicate range over every relevant toolchain component, +host/target fact, and build option rather than reducing it to a `rustc` version +string. + +Classify support evidence before using it: + +- applicable package metadata, published policy, release documentation, + feature/target policy, and authorized downstream agreements may define the + project's support contract; +- manifest checks, build scripts, `compile_error!`, wrappers, packaging rules, + and distribution controls may admit or enforce configurations; and +- CI jobs, lockfiles, successful builds, `rust-toolchain.toml`, and maintainer + defaults observe or select particular configurations but do not by themselves + define or prove downstream support. + +Resolve inherited fields in the exact workspace and inspect the effective +packaged metadata when it can differ. Interpret every mechanism through its +applicable contract; do not hard-code a universal precedence among metadata, +documentation, and agreements. A documented exclusion may delimit a support +promise, but if soundness depends on preventing that configuration from +shipping, require effective rejection before claiming closure. + +If applicable support declarations conflict or materially underdetermine the +predicate, do not silently select the narrowest interpretation. Apply the +domain-recovery choices in `SKILL.md` to the full case domain: obtain an +authorized project decision, construct an explicit conservative `Required` +domain containing every materially supported candidate domain, or report +regional results and leave the combined claim `UNPROVED`. Then project that +chosen full domain to `Required_cfg`. Do not call a conservative audit domain a +newly inferred project promise. If a shippable configuration is exposed and no +applicable contract clearly excludes it, include its possible shippable full +cases in the unresolved conservative domain until project authority resolves +its status; successful compilation alone still does not define the support +promise. + +Every transformation from controlling expressions to the full `Required` +domain and its `Required_cfg` projection is a proof obligation. Record the +transformation and the relation it must establish: + +- an exact normalization requires equality in both directions; +- a conservative audit domain requires + `Candidate_i(case) ⊆ Required(case)` for every materially supported candidate + full-case domain; the projected containment follows but cannot substitute for + this full-case proof; +- an exclusion requires an applicable support contract and, when soundness + depends on preventing shipment, effective enforcement; and +- a configuration partition used for proof requires `Required_cfg` to be + contained in the union of the proved configuration predicates. Cases need + not be disjoint unless the proof relies on uniqueness. + +Do not replace a range or conditional predicate with a finite inventory until +both membership and completeness are established from applicable evidence. A +list of endpoints, sampled toolchains, one apparent representative per minor +series, or successfully observed releases is not an inventory proof. When +exact membership is unavailable or large, retain the symbolic predicate and +prove it parametrically; if neither parametric proof nor justified exhaustive +partition closes, leave the remainder `UNPROVED`. + +Preserve conditional and nonlinear structure across every discovered axis +rather than collapsing `Required_cfg` to a single MSRV. It may be finite, +nonlinear, or moving and need not have a globally earliest toolchain. Resolve +dynamic policies at the audit cutoff. A cutoff identifies when a dynamic +predicate was recovered; it neither enumerates the toolchains before that date +nor supplies semantic continuity between sampled versions. + +Record: + +- source revision and workspace/package selection; +- Rust toolchain range, edition, standard-library identity, and relevant compiler + flags; +- controlling support-policy sources, conflicts, authorized resolutions, and + the audit cutoff; +- target triples, target specifications, CPUs, features, ABIs, data layouts, and + linkers; +- Cargo features, dependency feature unification, optional dependencies, and + resolver behavior; +- profiles and code-affecting environment or build inputs; +- generated artifacts and their generators; +- explicit exclusions and how compilation or distribution enforces them. + +An exclusion written only in an audit report does not constrain downstream +users. If soundness requires rejecting a combination, enforce and document the +rejection in the build or API. + +## Discover Configuration Axes + +Search both handwritten and generated source for all code-selection and +semantic axes. At minimum, investigate when applicable: + +- `cfg` and `cfg_attr`, Cargo features, optional dependencies, and feature + unification; +- target architecture, OS, environment, vendor, family, ABI, endianness, pointer + width, alignment, atomic widths, and target capabilities; +- conditional type definitions, representation/layout attributes, constants, + const evaluation, static initialization, and build-time execution; +- compile-time and runtime SIMD or other target features; +- debug assertions, overflow checks, optimization, LTO, codegen backend, panic + strategy, unwinding, sanitizers, and instrumentation; +- global and per-operation allocator choices, allocation failure behavior, and + custom allocator implementations; +- thread availability, atomics, permitted interleavings, weak memory behavior, + signals, cancellation, and runtime/executor choices; +- build scripts, procedural and declarative macros, derives, code generators, + bindgen output, included files, environment variables, and external tools; +- FFI implementation, ABI, library version, symbol resolution, static/dynamic + linking, linker scripts, link arguments, dynamic loading or plugins, and + load-time substitution; +- inline assembly dialect, registers, options, calling convention, instruction + availability, and surrounding compiler assumptions; +- compiler version, edition, unstable features, bootstrap flags, custom target + specifications, and standard-library build; +- tests/examples/binaries versus library code, `no_std`, host versus target + builds, and build-dependency versus runtime-dependency configurations. + +This list is intentionally advisory and may be incomplete or become outdated. +Discover the actual axes from the audited project and authoritative toolchain +contracts. Add newly discovered axes to the audit and report gaps in this +reference. + +## Prove Coverage of the Recovered Set + +Every full required case must reach its exact audited artifact, selected source, +or other in-scope build outcome and satisfy the applicable semantic obligations. +A proved enforced exclusion may establish that a candidate combination has no +shippable full case; it does not cover any full case that remains in `Required`. +A CI matrix, sample of targets, or pairwise feature test establishes neither +domain recovery nor universal semantic coverage. + +Avoid Cartesian-product enumeration when an abstract proof is clearer. Valid +coverage arguments include: + +- prove one implementation is parametric over an axis; +- partition configurations into equivalence classes and prove the partition is + exhaustive and each class representative shares the relevant semantics; +- prove mutually exclusive `cfg` predicates form a total partition over + `Required_cfg`; +- prove a generator emits only members of a finite audited family; +- prove independent lemmas for axes, then prove their assumptions remain + independent under composition; +- prove unsupported combinations fail before producing a shippable artifact. + +For every abstraction, check interactions between axes. A proof of each feature +alone does not prove their combination; target facts can change layout, atomic +availability, calling convention, or macro expansion on which another feature +depends. + +Use `Required_cfg` only as an index into full-case proofs. Attach a +configuration-domain predicate to every obligation, premise, and coverage +lemma. For each required configuration, prove the relevant build, artifact, +source-selection, and semantic lemmas for every full required case in its +omitted-dimension fiber, parametrically or by an exhaustive partition. A unary +configuration lemma may assert one exact artifact or source only after proving +uniqueness or independence from the omitted dimensions. A premise proved for +one target, toolchain, feature set, or generated artifact cannot discharge +another case merely because the source looks similar. If separate lemmas cover +separate regions, prove that their union contains `Required_cfg`, reattach every +other relevant case dimension, and prove their assumptions remain true where +regions interact. Use the resulting full-case lemmas in claim-level `Covered`; +do not substitute the configuration projection for it. + +Before accepting closure, try to exhibit a required boundary, interior, +conditional, or cross-axis full case absent from `Covered`. This is a +falsification check, not a substitute for the containment proof. + +Do not infer semantic coverage from successful compilation. Compilation may +establish syntax, typing, and selected compiler-enforced conditions; unsafe +contracts remain separate obligations. + +## Prove Build and Generation Pipelines + +Model claim-relevant build and generation machinery as a staged relation, not +merely an endpoint mapping: + +```text +policy and build inputs + -> ordered local operations, effects, and exits + -> emitted directives, metadata, source, or objects + -> build-tool/compiler interpretation + -> selected shipped artifact and source +``` + +Include an operation, exit, or effect when it can change the theorem domain, an +applicable or consumed premise, the selected or shipped artifact/source, +semantic reachability, or an in-scope postcondition. Group paths only after +proving them equivalent for every such proposition. Audit additional build +behavior when the requested theorem includes it. + +Derive the material stages from the artifact. For each stage, prove: + +- the exhaustive input/state partition on which it is reached; +- the order of operations whenever failure, partial progress, caching, or + invalidation can affect later stages; +- every claim-relevant fallible operation and alternative exit, including + which later operations are not reached and which earlier effects remain; +- a complete successful-output/effect relation sufficient for every downstream + proposition: exact value, cardinality, identity, or ordering when consumed or + asserted, otherwise a proved property covering every possible output; +- the distinction between local source behavior, build-tool or compiler + behavior, and any admitted environmental behavior; and +- that each output consumed by a later stage was actually established on that + path and within the premise's applicability. + +Classify outcomes according to the controlling contract. A policy rejection, +an infrastructure or tool failure, and successful production of an excluded or +supported artifact can all produce “no library was compiled” or another common +endpoint without being the same proof case. Do not say later selector, +generator, or compiler handling occurred on a path that exited earlier. + +Account for freshness when a changing input can reuse prior build state. Prove +the exact invalidation, rerun, cache-key, or identity relation consumed by the +claim rather than assuming the generator is re-executed with current inputs. + +When conditional source is selected, evaluate each material predicate from its +proved leaf options through every composite operator to the attribute or other +selection effect. Cite the applicable Rust semantics for those operators and +effects; a TCB entry mapping an external input to a leaf option does not also +supply Rust predicate semantics unless it explicitly says so. + +Treat generated code as shipped source. Capture enough information to reproduce +or identify: + +- generator/proc-macro/build-script package and exact version or digest; +- host toolchain and host configuration; +- target configuration and all relevant environment inputs; +- input tokens/files/schema and invocation options; +- output source, expansion, metadata, or object identity; +- diagnostics, suppressed checks, and unsupported paths. + +Do not stop at the generator's handwritten source. Soundness can depend on the +mapping from every accepted input and configuration to output, hygiene and name +resolution in the destination crate, compiler expansion behavior, or external +tool output. + +Use one of these proof strategies: + +1. Inspect each member of a proven finite output set. +2. Prove a property of the generator that entails safety for every supported + output. +3. Record the exact generated artifact in the audited snapshot and enforce that + exact output identity or digest. Pinning only the generator does not fix its + inputs, environment, compiler interaction, or output. + +For macro-generated APIs, audit the expanded visibility and caller obligations. +A safe-looking invocation is not automatically a safe API if rustc enforces an +unsafe-context obligation in the expansion; conversely, generated internal +unsafe code behind an invocation usable from safe context must be sound for +every accepted safe invocation. + +Build scripts may emit `cfg` values, link directives, environment values, or +generated source. Prove their complete claim-relevant staged relation as above. +Include proc-macro and build dependencies in the TCB or recursive audit as +appropriate. + +## Audit Targets, SIMD, and Concurrency + +For target-dependent code: + +- derive layout, validity, alignment, ABI, instruction, atomic, and pointer-width + facts from exact applicable authoritative contracts or TCB entries; +- distinguish compile-time target features from runtime CPU availability; +- prove every call edge satisfies target-feature and calling-convention + requirements; +- prove runtime feature detection dominates every specialized instruction path + and cannot be invalidated between detection and use; +- audit fallback paths and combinations of enabled features; +- include cross-language or dynamic-dispatch edges that may bypass a Rust + wrapper. + +For concurrency: + +- quantify over every permitted interleaving and memory-model behavior within + scope; +- prove synchronization, atomic ordering, initialization publication, lifetime, + ownership, and destruction properties from applicable contracts; +- treat caller-provided safe callbacks and safe trait implementations as + adversarial, including reentrancy, blocking, panic, and unexpected timing; +- distinguish thread-safety properties promised by types and unsafe trait impls + from behavior merely observed on one runtime. + +Do not use one scheduler run, stress test, or architecture as proof of all +executions. + +## Audit Allocators, Panic Modes, and Assertions + +For allocation-sensitive unsafe code, identify the allocator contract actually +required: + +- size and alignment accepted; +- allocation, reallocation, and deallocation pairing; +- zero-size behavior; +- maximum sizes and arithmetic bounds; +- allocation failure, overcommit, and address reuse; +- thread safety and reentrancy; +- allocator identity across FFI, dynamic-library, and configuration boundaries. + +A library generic over a valid allocator implementation must be sound for every +implementation satisfying the applicable unsafe allocator contract. A binary +that selects a particular allocator may record that exact implementation as a +TCB dependency when appropriate. + +Prove invariant restoration and resource behavior under every supported panic +strategy. Distinguish: + +- normal return; +- error return; +- panic with unwind; +- panic with abort; +- foreign exceptions or unwinding across boundaries; +- cancellation or destruction suppression where supported. + +Never rely on `debug_assert!` to establish a release-build safety precondition. +If a check is part of the proof, ensure it executes in every supported +configuration or prove the proposition independently. Treat differences in +overflow checks, debug assertions, and optimization as configuration branches +until shown irrelevant. + +## Audit FFI, Assembly, Linking, and Global Symbols + +For FFI, prove or explicitly trust: + +- exact function and data ABI, types, layout, calling convention, and symbol + identity on both sides; +- validity and ownership of arguments and return values; +- lifetime, aliasing, allocation, deallocation, callback, thread, and unwinding + rules; +- versioning and configuration of the foreign implementation; +- behavior of foreign code that Rust unsafe code relies upon. + +Declaring an extern item asserts that the declaration matches reality; calling +it consumes both the declaration contract and call-specific preconditions. Keep +those obligations distinct. + +For inline assembly, derive Rust-side requirements from exact applicable +Reference or standard-library text. Record ISA manuals, target specifications, +ABI documents, linker manuals, and other non-Rust sources as versioned +`EXTERNAL-SPEC` TCB entries unless the exact consumed proposition appears in +Reference or standard-library text. Audit operands, register classes, clobbers, +flags, stack, control flow, memory effects, options, instruction availability, +privilege/environment, and interaction with compiler optimization. This is a +discovery list, not an authoritative specification. + +Audit whole-program/link obligations when relevant, including: + +- uniqueness and type/ABI agreement of exported or unmangled symbols; +- global allocator and panic-runtime selection; +- link-section placement, alignment, initialization order, and linker-script + assumptions; +- dynamic symbol interposition and library substitution; +- consistency of declarations across crates and languages; +- linker flags or custom target settings that alter assumptions used by source + proofs. + +A compilation or linker option belongs to `Required_cfg` only when the +controlling support predicate includes it; the technical ability to emit or +ship a binary does not itself define project support. For an included option +that emits a binary, do not label the flag itself “Rust undefined behavior” +without authoritative text. Trace any resulting execution to the exact +violated Rust or external contract, or state that the artifact lies outside the +proved source-level claim. + +## Record Configuration Coverage + +For each audit, report: + +- every controlling support predicate and its exact source; +- the symbolic `Required_cfg` projection, its relationship to the full + `Required(case)` domain, the audit cutoff, and any unresolved policy + ambiguity; +- every asserted set relationship, normalization, enumeration, partition, + merge, or exclusion used to derive the full `Required` domain and its + `Required_cfg` projection, with its relation-appropriate certificate; +- every discovered axis and its possible supported values/classes; +- the staged build/generation relation, including ordered effects, alternative + exits, tool interpretation, and freshness where applicable; +- the proof method for every required configuration fiber and its composition + into the full-case `Required ⊆ Covered` claim argument; +- the obligation and premise applicability domains used by that proof; +- generated artifact identities or generator theorem; +- excluded combinations and their enforcement; +- untested but abstractly proved combinations; +- tested combinations and the limited propositions those tests establish; +- remaining assumptions, unknowns, and unsupported tool features; +- triggers requiring re-audit. + +Mark the audit `UNPROVED` if `Required_cfg` is not justified, a required +configuration fiber lacks a complete full-case argument, or any required full +case remains outside `Covered`. diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/proof-obligations.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/proof-obligations.md new file mode 100644 index 0000000000..815c02ccfa --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/proof-obligations.md @@ -0,0 +1,479 @@ +# Proof Obligations, Safety Contracts, and Local Proofs + +## Contents + +- [Form the theorem](#form-the-theorem) +- [Qualify applicability](#qualify-applicability) +- [Separate kinds of premises](#separate-kinds-of-premises) +- [Make every derivation reviewable](#make-every-derivation-reviewable) +- [Write safety documentation](#write-safety-documentation) +- [Write local safety proofs](#write-local-safety-proofs) +- [Carry invariants locally](#carry-invariants-locally) +- [Prove temporal behavior](#prove-temporal-behavior) +- [Cite authoritative axioms](#cite-authoritative-axioms) +- [Search for indirect derivations](#search-for-indirect-derivations) +- [Review a proof](#review-a-proof) + +## Form the Theorem + +Turn each soundness claim into explicit propositions before writing prose. + +For an unsafe API, use this shape: + +> For every state and input satisfying preconditions `P`, every permitted +> execution of the implementation is free of undefined behavior and establishes +> documented postconditions `Q`, relative to TCB `T`. + +Include ongoing and terminal obligations in `P`; a precondition need not concern +only the instant of the call. State who must maintain each fact, over what +interval, and what event discharges it. + +For a safe API, prove soundness with no caller-side safety precondition beyond +well-typed safe use. Also prove any postcondition consumed by the soundness +argument and any broader behavior explicitly placed in scope. Ordinary input +validation may reject values, return an error, or panic as documented, but +soundness may not depend on the safe caller honoring an unenforced rule. + +For each local proof site: + +1. Obtain the exact preconditions of the operation or contract being used. +2. Normalize conjunctions, implications, quantifiers, lifetimes, and temporal + clauses into separately reviewable obligations. +3. Derive every obligation from facts available at that point. +4. Obtain and prove every postcondition used later. +5. Establish the invariant state after success and every alternative exit. + +Treat every operation, declaration, implementation, or state transition that +supplies or consumes a safety contract as an obligation site. Follow each +obligation until it reaches checked local facts, named invariants, +authoritative axioms, or explicit TCB entries. + +Apply the quantifier-sensitive verdict certificates in `SKILL.md` when a +derivation fails or produces a counterexample. Do not confuse failure of a +universal proof with proof of an existential refutation. + +## Qualify Applicability + +State or inherit the exact domain of every claim and premise. Include whichever +dimensions can change the proposition, such as: + +- source and generated-artifact identity; +- inputs, states, types, signatures, lifetimes, and execution intervals; +- Rust, compiler, standard-library, dependency, and external-contract versions; +- targets, features, profiles, build inputs, and other supported + configurations; and +- deployment or probabilistic restrictions for separately qualified claims. + +A derivation proves only the cases in which every consumed premise applies. If +one proof does not cover the full required domain, partition the claim into +cases, prove each case, and establish that their union is exhaustive. Do not +turn an uncovered case into an implicit exclusion. + +Avoid repetitive local boilerplate. A proof may inherit applicability from an +exactly identified project support policy, invariant definition, axiom entry, +or TCB entry. The local proof must still make the inheritance and relevant case +clear enough to review. + +Derive the required toolchain/configuration projection and every transformation +of that projection under +[Recover the required supported set](configurations-and-generated-code.md#recover-the-required-supported-set), +then carry it through every premise and case lemma below. The domain covered by +a derivation is the intersection of the applicability domains of every premise +it consumes; the union of valid case lemmas must contain the required domain. + +A documented Rust guarantee from version `R` may support a later stable version +only when an exact Rust backwards-compatibility commitment preserves that exact +semantic proposition throughout the later version's relevant edition, target, +feature, and configuration domain. Under this skill's authority policy, record +a compatibility commitment outside the Rust Reference or standard-library +documentation as an explicit TCB premise. An API's stability or `since` badge +establishes only what its applicable authoritative text says it establishes; it +does not by itself prove that every behavioral sentence in current +documentation was guaranteed from that version. Do not extend a guarantee +beyond its original domain or automatically to unstable features, +`RUSTC_BOOTSTRAP`, `-Z` behavior, implementation details, custom targets, +platform availability, or pre-stabilization behavior. + +Compatibility does not propagate guarantees backward. Text first documented in +version `R` does not by itself prove the same proposition for earlier versions. +A later clarification can support an earlier version only when applicable +authoritative text expressly gives it historical scope or an accepted TCB +premise establishes that the guarantee already applied. Unchanged +implementation, version history, a documentation diff, or advisory prose is +insufficient by itself. Split the version domain if later text qualifies or +contradicts the older statement. For an open-ended toolchain range, either +prove the claim parametrically relative to a named compatibility premise or +state an audit cutoff and later-release re-audit trigger. A compatibility +premise about abstract semantics does not prove correctness of future compiler +binaries. + +Before issuing any affirmative result spanning multiple Rust releases, record +the exact required release predicate and one coverage basis: + +- an applicable parametric derivation over the whole predicate; +- an exhaustive partition with applicable premises for every class or member; + or +- an exact proposition-preserving backwards-compatibility premise whose domain + covers every later release claimed. + +Endpoint documentation, sparse version samples, an earliest supported release, +and an audit cutoff do not prove the releases between them. If the coverage +basis does not contain the claimed release predicate, narrow the proved region +and leave the remainder `UNPROVED`. + +## Separate Kinds of Premises + +Classify every premise: + +- **Local fact:** Established by inspected code, control/data flow, a type, or a + named invariant. Cite the exact check, branch, assignment, ownership fact, or + invariant clause. +- **Rust axiom:** Entailed by exact applicable text in a versioned Rust Reference + or standard-library page. Quote and link it. +- **Selected safe-dependency fact:** Supplied by a deliberately selected safe + dependency contract and recorded in the TCB. +- **Tool-derived fact:** Established by a verified tool theorem whose exact + proposition, model, scope, and premises entail the local fact. Record only its + residual unproved tool/model/translation premises in the TCB. +- **Additional assumption:** External specification, unsafe dependency, + compiler implementation, platform behavior, deployment restriction, + probabilistic premise, or other admitted proposition recorded in the TCB. + +Never blur an assumption into a derived fact. If a premise does not fit one of +these classes, the proof is incomplete. + +Distinguish the validity of a value of type `T` from a stronger library +invariant attached to its role in an abstraction. Prove both when needed. + +Never promote one producer's admission contract into an invariant of its output +type. A constructor, conversion, deserializer, FFI ingress, mutation, or other +producer precondition applies at that invocation. It supports a fact about that +particular result only through a proved postcondition or dataflow relation; it +does not prove that every valid value came through that producer or that later +transitions preserve the property. + +To rely on `I` as an invariant of every value in a stated set, provide a +complete derivation over that set without reversing the producer implication. +Such a derivation may, for example, use: + +1. applicable authoritative premises that entail `I` for every value in the + set; +2. an enforced abstraction boundary plus a complete proof that every in-scope + ingress and producer establishes `I` and every transition preserves it; +3. another applicable derivation, including a verified tool theorem, that + entails the exact quantified proposition; or +4. the exact universal proposition as an admissible accepted TCB premise under + the TCB rules. + +This enumeration does not replace the entailment requirement or exclude other +valid proof forms. A consumer may instead establish `I` for its particular +values or quantified subset from local checks, proved producer and transition +history, and other applicable premises. If neither derivation closes, leave +the consuming obligation `UNPROVED`. + +Likewise, distinguish: + +- permission to perform an operation; +- facts established by that operation; +- facts merely preserved by it; +- obligations transferred to a returned pointer, reference, guard, token, or + caller. + +## Make Every Derivation Reviewable + +A proof may be compact, but it must be reversible by a reviewer. A premise, +intermediate proposition, or applicability restriction is material when +deleting it leaves the remaining explicit premises insufficient to entail a +certified conclusion; a countermodel may demonstrate that insufficiency. State +every such component. Justify each transition unless its entailment is directly +reviewable from the stated premises. Even a direct transition may not import an +unstated Rust, library, tool, environmental, or TCB premise. Omit only immediate +source syntax or purely logical rearrangement of already explicit premises. + +Do not hide a material component behind a name such as “layout rules,” “cfg +semantics,” “the build mapping,” or “the type guarantees it.” A citation verifies +only the proposition the proof extracts from it; an allowlisted page or another +clause on the same page does not silently fill an unstated premise. Unfold +composite behavior to the clauses actually used. When build or generation +stages are relevant, apply +[Prove build and generation pipelines](configurations-and-generated-code.md#prove-build-and-generation-pipelines). + +The ordinary proof prose or obligation ledger may carry this information. Do +not create a separate graph when the existing proof is already +reverse-traceable. Before certifying `PROVED`, `UNSOUND`, `CONTRACT-BROKEN`, or +any regional result: + +1. start at every conclusion used by the certificate and recover its full-case + applicability, every premise and intermediate proposition, and why they + entail the conclusion; +2. classify each premise as a checked source fact, mathematical step, named + invariant, Rust axiom, tool theorem, or TCB entry; check source facts and + invariants against their exact locations, dataflow, material operation order, + and alternative exits; +3. check material, non-immediate mathematical and logical steps by their + explicit derivations or witnesses; they need no Rust citation; +4. check every Rust semantic premise against its recorded exact versioned + quotation and link; +5. check every other semantic premise against its verified tool theorem or + accepted TCB entry; +6. ensure no projection, shorthand, page-level citation, or later-stage result + silently supplies a missing premise; and +7. trace forward through every relevant exit to prove the postconditions and + invariants consumed later. + +If a required component remains absent, remove every conclusion that depends on +it and apply the exact verdict certificate. + +## Write Safety Documentation + +Give every unsafe function, trait, impl, field, macro boundary, and other unsafe +contract a precise safety specification regardless of visibility. Use `# Safety` +documentation for public contracts. A private contract may cite module-owned +invariants, but must still state every fact its callers or implementers must +establish or continue to uphold. Use precise subjects, intervals, and +quantification. + +A complete unsafe API contract should make the following derivable whenever +applicable: + +- which values, memory regions, objects, threads, or executions it covers; +- validity, initialization, alignment, size, provenance, accessibility, + lifetime, aliasing, exclusivity, mutability, and ownership requirements; +- concurrency, atomic ordering, synchronization, reentrancy, callback, signal, + and thread-affinity requirements; +- target, ABI, feature, allocator, unwinding, linkage, or environmental + restrictions; +- what may be observed, read, written, moved, copied, destroyed, or retained; +- whether an invariant may be suspended, for how long, and what must not happen + before restoration; +- obligations attached to return values or capabilities; +- behavior on panic, unwind, cancellation, early return, or partial progress; +- documented postconditions on success and every other documented outcome. + +Use this list as a discovery prompt. Derive the actual requirements from the +exact operation and applicable authoritative contracts, and add every other +obligation those contracts create. + +Define relative terms. Replace phrases such as “valid pointer,” “properly +initialized,” “no aliases,” “live,” “same allocation,” “correct layout,” and +“used normally” with the exact propositions intended. Do not use “the caller +guarantees” unless the current boundary is unsafe and its documentation actually +requires the cited fact. + +Safety preconditions must be sufficient; they need not be mathematically +weakest. Nevertheless, avoid irrelevant or unknowable conditions. Every stated +condition becomes part of the API contract and its evolution constraints. + +Document postconditions with the same precision. If callers may rely on a +result, state: + +- the state/value relationship established; +- the resources, aliases, or ownership transferred; +- which prior invariants remain true; +- when the guarantee begins and ends; +- distinctions among normal return, error, panic, and unwind. + +## Write Local Safety Proofs + +Place a `SAFETY` comment immediately adjacent to the smallest cohesive unsafe +operation or block. Prefer one proof unit per independently reviewable +obligation set. + +For new code, require an explicit `unsafe { ... }` block for each unsafe +operation even inside an `unsafe fn`, and enable `unsafe_op_in_unsafe_fn` at +`deny` or `forbid` when compatible with project policy. Use documentation and +undocumented-unsafe-block lints as completeness aids where available; lint +success is not a proof. + +Use this structure: + +```rust +// SAFETY: +// Obligation: `` requires P1, P2, and P3. +// Facts: +// - F1 follows from . +// - F2 follows from TCB-... / AXIOM-... . +// Derivation: +// - F1 and F2 imply P1 because ... +// - ... +// Result: +// - The operation establishes Q. +// - Q re-establishes/preserves/transfers invariant I. +unsafe { operation() } +``` + +Use ordinary prose when clearer, but retain each logical component. Do not write: + +- “safe because this is unsafe code”; +- “the pointer is valid” without defining and proving the required properties; +- “checked above” without identifying the dominating check and relevant values; +- “guaranteed by the type/caller/API” without naming the exact contract clause; +- “this is how the standard library does it”; +- “Miri/tests pass” as a universal derivation; +- “obviously,” “trivially,” or “cannot happen” in place of proof; +- circular arguments in which an invariant is justified only by code that + already assumes it. + +A proof may cite a canonical checked proof or TCB entry to avoid duplicating +large quotations. Keep enough local text to show which proposition is used and +how it entails the local obligation. + +When one unsafe block contains multiple operations, prove each operation in +program order. Include facts established by earlier operations only after +proving those operations' postconditions. + +## Carry Invariants Locally + +State each safety invariant near the representation or boundary that owns it. +Give it a stable name when multiple proofs cite it. Specify: + +- the objects and states over which it quantifies; +- when it is required to hold; +- who may rely on it; +- every operation permitted to establish, mutate, suspend, transfer, consume, + or destroy it; +- what must be true while it is suspended; +- how panic, unwind, cancellation, reentrancy, callbacks, and destruction affect + it. + +Define the invariant's actual enforcement boundary and prove every producer, +transition, and consumer within it. Apply +[Use module privacy](api-boundaries-and-evolution.md#use-module-privacy) to +choose that boundary for new code or compute the real access region of existing +code. + +An invariant is local when each consumer can cite a named proposition whose +current truth is established by a local boundary. Its subject may still be +global state. Do not accept an informal “global invariant” that no boundary +owns or re-establishes. + +## Prove Temporal Behavior + +Treat time and interference explicitly: + +- Determine the interval during which each pointer, reference, lock, capability, + borrow, allocation, and invariant fact remains usable. +- Check every possible intervening call, callback, destructor, panic, unwind, + cancellation point, signal interaction, and reentrant entry. +- For concurrency, quantify over every permitted thread interleaving and weak + memory behavior within scope, not one observed schedule. +- If an operation returns a capability whose safe methods could violate an + invariant, place the ongoing obligation in the unsafe boundary's contract or + return a representation that enforces it. +- If a guard restores an invariant in `Drop`, prove restoration on all paths on + which `Drop` runs and separately address paths on which destruction can be + skipped, duplicated, reordered, or aborted. +- If an invariant is suspended across code not controlled by the abstraction, + treat that code as adversarial unless it is an explicitly trusted dependency. + +Cryptographic infeasibility and low probability do not turn a possible +execution into an unconditional Rust soundness proof. Move such premises to an +explicit conditional application claim and TCB entry. + +## Cite Authoritative Axioms + +For every Rust or standard-library ground-truth proposition: + +1. Select documentation applicable to the audited compiler/library version. +2. Link the narrowest applicable sections, including versions in the URLs. +3. Quote the smallest sufficient set of excerpts whose propositions participate + in the derivation. +4. State the proposition derived from each excerpt and justify the inference + that combines them. +5. Check that qualifications, definitions, linked clauses, and surrounding + scope do not weaken it. +6. Have the reviewer open the source and independently confirm the derivation. + +Apply [Qualify applicability](#qualify-applicability) when a citation and the +claim concern different Rust versions. + +If the Reference or standard-library documentation is missing, ambiguous, +internally inconsistent, or too weak, record the exact missing proposition. +Treat explanatory sources or current implementation behavior only as leads or +explicit additional assumptions. Recommend an upstream documentation report +when appropriate. + +## Search for Indirect Derivations + +Do not equate the absence of a single direct documentation sentence with the +absence of a proof. Before reporting an authoritative documentation gap or +finalizing an important obligation as unproved: + +1. Restate the exact semantic property required and unfold relevant project + definitions. +2. Search for applicable direct guarantees. +3. Search for stronger, more general, or orthogonal authoritative facts whose + conjunction could entail the property. +4. State every intermediate lemma and justify each inference rather than merely + collecting citations. +5. Check the applicability of every premise and intermediate lemma. +6. Try to construct a model that satisfies the premises while falsifying the + conclusion. If one remains possible, identify the missing implication. + +This search does not weaken the fail-closed rule. If no complete admissible +derivation is established, the obligation remains unproved. Distinguish “this +audit did not complete a proof” from the stronger claim that authoritative +documentation cannot support one. + +When a universal soundness derivation does not close, separately ask whether +the established facts close an existential refutation. Identify a valid +in-scope use or execution, prove reachability of the relevant operation or +semantic event, prove its exact required safety proposition false there, and +trace that failure to the applicable authoritative or explicitly trusted UB +consequence. If every link is proved, apply `UNSOUND`; if any link is absent, +the failed universal obligation remains `UNPROVED`. Do not demand a fact about +every input to establish one existential witness, and do not infer a witness +merely from the absence of a universal proof. + +## Review a Proof + +For each proof: + +1. Reconstruct the required preconditions from the callee or language/library + contract rather than trusting the comment's summary. +2. Open every citation and verify its exact proposition, version, and scope. +3. Check each claimed local fact—including its quantifier, producer/transition + history, and applicability domain—against the actual dataflow and all + alternative paths. +4. Expand every named invariant and ensure it is established initially and + preserved by every permitted transition. +5. Check quantifiers, arithmetic boundaries, zero-sized and empty cases, + overflow, partial initialization, overlapping ranges, alias duration, + provenance, destruction, unwinding, reentrancy, concurrency, and + configuration-dependent behavior when relevant. +6. Verify every postcondition used downstream. +7. Search for circularity, vacuity, hidden trust, and stronger conclusions than + the cited facts entail. +8. Apply [Make every derivation reviewable](#make-every-derivation-reviewable) + to every conclusion used by a verdict or regional result and every claimed + set relationship. +9. Record every missing implication so it cannot be forgotten, apply + [Search for indirect derivations](#search-for-indirect-derivations), and + apply the verdict certificate in `SKILL.md`: report `UNPROVED` if a required + implication remains absent and no existential refutation closes, or the + applicable refutation verdict if one does. + +If validation requires a material derivation absent from the existing safety +comment, include that reconstructed derivation—or the smallest missing +portion—in the review. Apply the material-component definition in +[Make every derivation reviewable](#make-every-derivation-reviewable). Give its +citations, applicability, and relationship to the required preconditions and +postconditions. Report the implementation result separately from the deficient +proof artifact: + +- If the reconstruction succeeds, the implementation obligation may be proved, + but report the inadequate comment and provide proposed replacement wording. +- If the reconstruction fails, leave the obligation unproved unless it instead + closes one of the existential certificates in `SKILL.md`. + +When changes are authorized, update the adjacent proof rather than leaving the +reconstructed reasoning only in the review. A canonical checked proof or named +invariant may hold shared detail; do not demand redundant prose when the local +comment already identifies the exact proposition and complete derivation path. + +Do not use reconstruction to repair a caller-facing contract retroactively. An +undocumented caller obligation remains hidden under the current API contract, +even if adding it would make the implementation proof succeed. + +These examples identify common omissions; they are not a substitute for reading +the applicable authoritative contracts. diff --git a/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/tcb-and-evidence.md b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/tcb-and-evidence.md new file mode 100644 index 0000000000..ba5efcff73 --- /dev/null +++ b/evals/unsafe-rust/frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb/references/tcb-and-evidence.md @@ -0,0 +1,295 @@ +# Trusted Computing Base and Evidence + +## Contents + +- [Maintain an explicit trust boundary](#maintain-an-explicit-trust-boundary) +- [Classify TCB entries](#classify-tcb-entries) +- [Record dependency contracts](#record-dependency-contracts) +- [Record external and deployment assumptions](#record-external-and-deployment-assumptions) +- [Handle probabilistic claims](#handle-probabilistic-claims) +- [Judge tools by their theorem](#judge-tools-by-their-theorem) +- [Audit a tool-derived proof](#audit-a-tool-derived-proof) +- [Review and evolve the TCB](#review-and-evolve-the-tcb) + +## Maintain an Explicit Trust Boundary + +A TCB audit log lists every proposition the audit accepts as authoritative or +correct without proving it from more primitive in-scope premises. Its purpose is +not to make assumptions respectable; it makes the exact conditional theorem +visible and reviewable. + +For every entry, record: + +- stable identifier and category; +- exact proposition admitted; +- exact source, artifact, implementation, version, revision, or digest; +- contract text or other evidence; +- scope, configurations, and consumers; +- why admission is permitted; +- validation or audit already performed; +- compatibility/update channel; +- owner and review trigger; +- status and unresolved limitations. + +Do not use entries such as “the platform works,” “dependencies are correct,” +“normal allocator,” “valid environment,” or “the compiler is sound.” Split them +into the smallest propositions actually consumed by proofs. + +Minimize the TCB where practical, but never hide an assumption to make the list +look small. Every unproved premise must become either another proof obligation +or an explicit entry. + +Do not make the theorem vacuous by adding an entry that merely assumes the +in-scope conclusion or trusts the implementation that the declared audit scope +purports to prove. Either prove that code, or narrow the theorem and expose the +code as a precisely identified excluded dependency/TCB component. + +When a proof applies an older documented Rust guarantee to a later version via +Rust's backwards-compatibility commitment, record the exact compatibility +proposition as a TCB entry unless it is itself entailed by applicable Reference +or standard-library text. Neither an API stability badge nor a general +expectation of stability silently expands the older guarantee's semantic or +configuration domain. + +The default source-level theorem is relative to the documented Rust abstract +semantics. It does not require trusting one compiler backend to emit a correct +binary. A binary-level theorem additionally requires a compiler/toolchain, +target, linker, loader, platform, and external-runtime story appropriate to the +claim. + +## Classify TCB Entries + +Use categories that expose why a proposition is admitted. Suitable categories +include: + +- **AXIOM:** Exact versioned Rust Reference or standard-library proposition. +- **SAFE-DEP:** Documented behavior of a deliberately selected safe dependency + API. +- **UNSAFE-DEP:** Correctness of a specific unsafe dependency implementation and + contract not recursively proved by this audit. +- **EXTERNAL-SPEC:** ABI, ISA, OS, hardware, foreign-language, allocator, linker, + or other non-Rust contract. +- **IMPLEMENTATION:** Exact compiler, standard-library build, foreign library, + runtime, generator, proc macro, build tool, or other implementation assumed + correct for a non-source-level claim. +- **TOOL:** Residual trusted components or model correspondence supporting a + tool-derived proof. +- **ENVIRONMENT/DEPLOYMENT:** Restriction on entry inputs, load environment, + symbols, CPU, privileges, resources, or other execution context. +- **CRYPTO/PROBABILISTIC:** Explicit computational or probabilistic premise for + a separately labeled conditional claim. +- **OUT-OF-BAND:** A bilateral or project-specific promise beyond the published + default contract. + +Projects may use different names. Preserve the semantic distinctions. + +A proof result produced by a tool is not automatically a TCB assumption. It can +derive a fact when its theorem and premises are verified. Record only the +remaining unproved tool correctness, translation, model, solver, certificate +checker, harness, or environmental premises as TCB entries. + +Only a consumed entry explicitly accepted by the authorized human reviewer may +support `PROVED`. A pending entry makes every consuming claim `UNPROVED`. A +rejected or superseded entry may not be consumed; replace it with a proof or an +accepted entry, or narrow the claim and expose the exclusion. + +## Record Dependency Contracts + +For every dependency proposition, identify whether code is deliberately +selected or caller-controlled. + +The project may trust a deliberately selected safe dependency API to behave as +documented. Record: + +- package/source identity and exact resolved version; +- safe API and exact behavior consumed; +- documentation version; +- enabled features and relevant target/configuration scope; +- contract channel: SemVer range, exact pin, in-tree fork, out-of-band + agreement, consumer-specific promise, or another explicit arrangement; +- compatibility and re-audit trigger. + +An exact pin freezes identity; it does not establish an undocumented semantic +fact. Prove such a fact by auditing the pinned implementation, obtain an +applicable additional contract, or admit the exact implementation proposition +explicitly. + +Do not apply this exception to behavior supplied by a caller merely because it +uses a dependency-defined type or trait. Values, callbacks, closures, plugins, +generic parameters, trait objects, and safe trait implementations selected by +the caller remain adversarial safe code. + +For a third-party unsafe API: + +1. Obtain its exact caller safety contract and prove the local call satisfies + it. +2. Separately establish that the dependency implementation upholds its promise + for every valid call. +3. Discharge step 2 by recursively auditing the implementation or recording a + precise `UNSAFE-DEP` assumption. + +Do not silently include unsafe dependencies in the safe-dependency exception. + +When depending on a fork or out-of-band agreement, record the actual authority +for the additional promise, parties, exact covered uses, duration, notification +mechanism, and update process. Do not generalize a consumer-specific guarantee +to other uses. + +## Record External and Deployment Assumptions + +External specifications are not Rust axioms. Admit only the exact propositions +needed, with version and scope, for example: + +- a foreign function has a stated ABI and obeys stated ownership/lifetime rules; +- a CPU instruction has stated effects when a named feature and privilege level + are present; +- a linker binds a symbol to a specific definition with a specific layout; +- a custom allocator satisfies a named contract; +- a loader, OS, embedded runtime, kernel, or device maintains specified memory + or concurrency behavior; +- a binary entrypoint receives inputs restricted by a deployment boundary. + +Distinguish three claims: + +1. **Safe library soundness:** every well-typed safe use is sound; deployment + restrictions cannot be hidden premises. +2. **Unsafe API soundness:** every use satisfying documented safety obligations + is sound; external conditions may be explicit obligations. +3. **Binary/application soundness:** executions satisfying stated entry and + deployment assumptions are sound. + +A cryptographic signature check, authenticated input channel, kernel policy, or +restricted device state may narrow a binary theorem. It may not make an +otherwise safe library API conditionally sound without exposing an unsafe +boundary or enforcing the restriction in safe code. + +If a compilation or linker flag still emits an artifact, record it as part of +the configuration or toolchain scope. Do not call the flag itself undefined +behavior unless an authoritative contract uses that classification. Identify +the exact execution contract that the resulting artifact satisfies or violates. + +## Handle Probabilistic Claims + +Rust soundness is universal over valid uses and permitted executions. A +non-zero, negligible, computationally infeasible, or empirically unobserved +chance of undefined behavior is not unconditional soundness. + +A user may explicitly admit a cryptographic or probabilistic premise in the TCB, +such as collision resistance or unforgeability. Then: + +- state the exact security experiment or probability bound; +- identify the primitive, parameters, implementation, threat model, and time + horizon; +- state how the premise restricts executions or inputs; +- label the result as a conditional computational/application theorem; +- keep the ordinary unconditional Rust soundness verdict separate. + +Do not write `PROVED` without qualification when the result depends on such an +entry. Use wording such as `PROVED relative to CRYPTO-...` and explain that this +is not unconditional Rust soundness. + +## Judge Tools by Their Theorem + +Classify evidence by what the exact result proves: + +- A concrete execution refutes only a claim whose applicable verdict certificate + it satisfies. A valid in-scope execution with a complete UB certificate can + refute soundness; a postcondition refutation must be UB-free as a whole. +- A clean sampled test, fuzzing run, sanitizer run, interpreter execution, or + stress run usually establishes only that the explored executions did not + trigger the modeled failure. +- An alarm-free sound over-approximation can prove absence of its modeled bad + states over its stated domain. +- Exhaustive model checking can prove a property over the exhaustively covered + state space. +- Bounded model checking proves only the bounded proposition unless a + completeness bound is established. +- Deductive or interactive verification can prove the encoded theorem relative + to its logic, axioms, models, specifications, and trusted components. +- Successful compilation establishes only the exact properties the applicable + compiler contract and checks are relied upon to enforce. + +These are examples, not rules attached permanently to tool categories. One tool +can provide different guarantees in different modes or results. Read its exact +documentation and output. + +Apply this rule: + +> A tool result discharges an obligation only if the documented guarantee of +> that exact result, together with all explicit premises and trusted components, +> logically implies the obligation for the exact audited artifact and supported +> configuration set. + +Never infer more than the theorem. A tool model is not an additional Rust +authority; prove its correspondence to exact applicable Reference and +standard-library contracts or admit the missing correspondence explicitly. + +## Audit a Tool-Derived Proof + +Before accepting a tool result, verify: + +1. **Proposition:** State the exact property proved and why it entails the Rust + soundness obligation or documented postcondition. +2. **Artifact identity:** Record exact source, expansion/generated output, IR, + harness, specifications, compiler, target, tool, solver/backend, versions, + options, and configuration. +3. **Quantification:** Check coverage of inputs, states, executions, call + contexts, nondeterminism, thread interleavings, and supported configurations. +4. **Bounds:** Identify loop, recursion, allocation, object-count, integer, + search-depth, thread, time, and other bounds. Establish completeness or limit + the conclusion. +5. **Non-vacuity:** Check that the property, assertion, or unsafe operation is + reachable under permitted inputs and that assumptions do not make the + harness inconsistent or empty. +6. **Semantic fidelity:** Check validity, layout, provenance, aliasing, + initialization, concurrency, panic/unwind, FFI, assembly, allocation, target, + and environment modeling whenever relevant. +7. **Trust and stubs:** List trusted functions, contracts, abstractions, + dependency models, unsupported features, suppressions, skipped checks, and + manual lemmas. +8. **Terminal result:** Require the documented successful proof result. Timeout, + unknown, incomplete, unsupported, disabled checks, or ignored alarms do not + prove the target. +9. **TCB:** Identify verifier/analyzer correctness, source-to-model translation, + semantic models, solver/backend, proof checker, and specification adequacy + that remain trusted. + +An independently checked certificate may remove the producer or solver from the +TCB, depending on its guarantee. It does not by itself prove that the encoded +specification matches the needed Rust theorem or that source-to-model +translation is faithful. + +Suppressing a sound analyzer alarm creates a new proof obligation. A false +positive does not invalidate the analyzer's soundness guarantee; an unjustified +suppression invalidates the claimed conclusion. + +Tests and dynamic tools remain valuable for finding counterexamples, exercising +configuration paths, and checking that proof assumptions match reality. Report +their actual contribution without treating a clean run as a universal proof. + +## Review and Evolve the TCB + +Reuse the project's canonical TCB log when present. For every audit: + +- open and verify every consumed entry; +- remove unused, expired, superseded, or duplicate entries; +- add newly discovered assumptions before relying on them; +- map entries to proof consumers; +- check versions, feature/configuration scope, and contract channels; +- distinguish reviewed facts from proposed or unresolved assumptions; +- identify entries the human reviewer may reject. + +Trigger re-audit when: + +- a consumed authoritative document changes materially; +- supported Rust, target, feature, allocator, tool, or environment scope changes; +- a dependency resolves to a new version or changes contract channel; +- a fork or out-of-band agreement changes; +- generated output or its inputs/generator change; +- a tool, model, harness, bound, suppression, or proof specification changes; +- a TCB proposition is weakened, invalidated, or replaced; +- new code consumes an existing entry in a stronger way. + +Record the TCB revision or digest in every audit verdict. A `PROVED` result is +always relative to that stated trust boundary even when it contains only +authoritative Rust axioms and deliberately permitted safe-dependency trust. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/events.jsonl b/evals/unsafe-rust/runs/2026-08-01-v4-focused/events.jsonl new file mode 100644 index 0000000000..2fae8020d0 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/events.jsonl @@ -0,0 +1 @@ +{"details": {"lock_sha256": "cd7a300f83b045f76530eded20ec2d22bd6abbd484d6459ff50245fb07ef943e"}, "event": "freeze_locked", "phase": "freeze", "previous_event_sha256": null, "schema_version": 1, "sequence": 1, "sha256": "059cde170e6e31d4ef4c4997b4a64413fdd9e47e9f9e5df74c1c9707bf6e3c58", "time_utc": "2026-08-01T17:59:50.663768Z"} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/fetch_authority.py b/evals/unsafe-rust/runs/2026-08-01-v4-focused/fetch_authority.py new file mode 100644 index 0000000000..f5fb986d02 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/fetch_authority.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +"""Fetch or verify the exact official-document pages allowed by this run.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import html +import io +import re +import urllib.parse +import urllib.request +from datetime import datetime, timezone +from pathlib import Path + + +RUN = Path(__file__).resolve().parent +ALLOWLISTS = RUN / "freeze" / "allowlists" +MANIFEST = RUN / "freeze" / "authority-manifest.tsv" +MODES = ("P", "B", "L", "Q", "R") +FIELDS = ( + "mode", + "requested_url", + "fetch_url", + "final_url", + "status", + "content_type", + "bytes", + "sha256", + "fragment_found", + "retrieved_utc", +) + + +def sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def read_pinned(path: Path) -> bytes: + data = path.read_bytes() + if (RUN / "freeze" / "LOCK.json").exists(): + import protocol + + if sha256(data) != protocol.frozen_file_digest(path): + raise ValueError(f"frozen authority input changed while being read: {path}") + return data + + +def load_urls() -> list[tuple[str, str]]: + rows: list[tuple[str, str]] = [] + for mode in MODES: + path = ALLOWLISTS / f"{mode}.txt" + for url in read_pinned(path).decode().splitlines(): + if not re.fullmatch(r"https://doc\.rust-lang\.org/\S+", url): + raise ValueError(f"invalid allowlisted URL: {url!r}") + rows.append((mode, url)) + return rows + + +def without_fragment(url: str) -> str: + parts = urllib.parse.urlsplit(url) + return urllib.parse.urlunsplit((parts.scheme, parts.netloc, parts.path, parts.query, "")) + + +def fragment_present(requested_url: str, body: bytes) -> bool: + fragment = urllib.parse.unquote(urllib.parse.urlsplit(requested_url).fragment) + if not fragment: + return True + text = body.decode("utf-8", errors="replace") + escaped = html.escape(fragment, quote=True) + patterns = ( + rf'\bid=["\']{re.escape(fragment)}["\']', + rf'\bid=["\']{re.escape(escaped)}["\']', + rf'\bname=["\']{re.escape(fragment)}["\']', + ) + return any(re.search(pattern, text) for pattern in patterns) + + +def fetch(url: str) -> dict[str, object]: + request = urllib.request.Request( + url, + headers={"User-Agent": "unsafe-rust-skill-evaluation/4 authority-freeze"}, + ) + with urllib.request.urlopen(request, timeout=60) as response: + body = response.read() + return { + "final_url": response.geturl(), + "status": response.status, + "content_type": response.headers.get_content_type(), + "bytes": len(body), + "sha256": sha256(body), + "body": body, + "retrieved_utc": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + } + + +def build_rows() -> list[dict[str, str]]: + requested = load_urls() + fetched: dict[str, dict[str, object]] = {} + for _mode, requested_url in requested: + fetch_url = without_fragment(requested_url) + if fetch_url not in fetched: + fetched[fetch_url] = fetch(fetch_url) + + rows: list[dict[str, str]] = [] + for mode, requested_url in requested: + fetch_url = without_fragment(requested_url) + result = fetched[fetch_url] + found = fragment_present(requested_url, result["body"]) + if not found: + raise ValueError(f"fragment not present in retrieved page: {requested_url}") + rows.append( + { + "mode": mode, + "requested_url": requested_url, + "fetch_url": fetch_url, + "final_url": str(result["final_url"]), + "status": str(result["status"]), + "content_type": str(result["content_type"]), + "bytes": str(result["bytes"]), + "sha256": str(result["sha256"]), + "fragment_found": "true", + "retrieved_utc": str(result["retrieved_utc"]), + } + ) + return rows + + +def render(rows: list[dict[str, str]]) -> str: + output = io.StringIO(newline="") + writer = csv.DictWriter(output, fieldnames=FIELDS, dialect="excel-tab", lineterminator="\n") + writer.writeheader() + writer.writerows(rows) + return output.getvalue() + + +def read_manifest() -> list[dict[str, str]]: + rows = list( + csv.DictReader(io.StringIO(read_pinned(MANIFEST).decode()), dialect="excel-tab") + ) + if not rows or tuple(rows[0]) != FIELDS: + raise ValueError("authority manifest has unexpected columns") + return rows + + +def verify() -> None: + rows = read_manifest() + expected_pairs = load_urls() + actual_pairs = [(row["mode"], row["requested_url"]) for row in rows] + if actual_pairs != expected_pairs: + raise ValueError("authority manifest does not match allowlist ordering") + + fetched: dict[str, dict[str, object]] = {} + for row in rows: + fetch_url = row["fetch_url"] + if fetch_url != without_fragment(row["requested_url"]): + raise ValueError(f"incorrect fetch URL for {row['requested_url']}") + if fetch_url not in fetched: + fetched[fetch_url] = fetch(fetch_url) + result = fetched[fetch_url] + checks = { + "final_url": str(result["final_url"]), + "status": str(result["status"]), + "content_type": str(result["content_type"]), + "bytes": str(result["bytes"]), + "sha256": str(result["sha256"]), + } + for field, actual in checks.items(): + if row[field] != actual: + raise ValueError( + f"authority drift for {row['requested_url']}: " + f"{field} frozen={row[field]!r} live={actual!r}" + ) + if row["fragment_found"] != "true" or not fragment_present( + row["requested_url"], result["body"] + ): + raise ValueError(f"fragment missing for {row['requested_url']}") + print(f"verified {len(rows)} allowlist entries across {len(fetched)} pages") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--write", action="store_true", help="retrieve and write the frozen manifest") + parser.add_argument( + "--record-wave", + type=int, + choices=range(1, 6), + help="verify live bytes and append a successful verification event for this collection wave", + ) + args = parser.parse_args() + if args.write and args.record_wave is not None: + raise SystemExit("--write and --record-wave are mutually exclusive") + if args.write: + if (RUN / "freeze" / "LOCK.json").exists(): + raise SystemExit("refusing to rewrite authority manifest after freeze lock") + MANIFEST.write_text(render(build_rows())) + print(f"wrote {MANIFEST.relative_to(RUN)}") + else: + operation_lock_handle = None + if args.record_wave is not None: + import protocol + + operation_lock_handle = protocol.acquire_operation_lock() + protocol.validate_static(require_lock=True, announce=False) + protocol.assert_run_not_invalid() + protocol.assert_freeze_locked() + protocol.assert_authority_verification_allowed(args.record_wave) + verify() + if args.record_wave is not None: + protocol.append_event( + "collection", + "authority_verified", + digest=protocol.frozen_file_digest(MANIFEST), + details={"wave": args.record_wave, "entries": len(read_manifest())}, + ) + + +if __name__ == "__main__": + main() diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/LOCK.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/LOCK.json new file mode 100644 index 0000000000..bbcedf2e45 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/LOCK.json @@ -0,0 +1,26 @@ +{ + "schema_version": 1, + "status": "FROZEN", + "file_manifest_sha256": "059cde170e6e31d4ef4c4997b4a64413fdd9e47e9f9e5df74c1c9707bf6e3c58", + "review_signoffs": [ + { + "reviewer_id": "v4-oracle-final-review", + "verdict": "PASS/FREEZE", + "file_manifest_sha256": "059cde170e6e31d4ef4c4997b4a64413fdd9e47e9f9e5df74c1c9707bf6e3c58", + "scope": "Independent read-only review of manifest-bound P/B/L/Q/R fixtures, requests, oracles, rubrics, authorities, TCBs, prompts, plan, controls, and scoring/protocol interactions; all expected theorems and full-domain gates verified; unresolved ambiguities: none.", + "independent_non_author": true, + "reviewed_utc": "2026-08-01T17:55:30Z" + }, + { + "reviewer_id": "v4-oracle-review-2", + "verdict": "PASS/FREEZE", + "file_manifest_sha256": "059cde170e6e31d4ef4c4997b4a64413fdd9e47e9f9e5df74c1c9707bf6e3c58", + "scope": "Independent review of all P/B/L/Q/R fixtures, targets, requests, oracles, rubrics, authorities, TCB bounds, and scoring/protocol interactions through aggregation and integrity gates; unresolved ambiguities: none.", + "independent_non_author": true, + "reviewed_utc": "2026-08-01T17:58:58Z" + } + ], + "unresolved_ambiguities": [], + "reports_collected_before_lock": 0, + "locked_utc": "2026-08-01T17:59:23Z" +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/B.txt b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/B.txt new file mode 100644 index 0000000000..b425cc1918 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/B.txt @@ -0,0 +1,27 @@ +https://doc.rust-lang.org/1.85.1/std/env/fn.var.html +https://doc.rust-lang.org/1.85.1/std/env/enum.VarError.html +https://doc.rust-lang.org/1.85.1/std/result/enum.Result.html +https://doc.rust-lang.org/1.85.1/std/primitive.u8.html +https://doc.rust-lang.org/1.85.1/reference/expressions/match-expr.html +https://doc.rust-lang.org/1.85.1/reference/expressions/block-expr.html +https://doc.rust-lang.org/1.85.1/reference/items/functions.html#function-body +https://doc.rust-lang.org/1.85.1/reference/expressions/call-expr.html +https://doc.rust-lang.org/1.85.1/reference/patterns.html#tuple-struct-patterns +https://doc.rust-lang.org/1.85.1/reference/patterns.html#literal-patterns +https://doc.rust-lang.org/1.85.1/reference/patterns.html#wildcard-pattern +https://doc.rust-lang.org/1.85.1/std/string/struct.String.html#method.as_str +https://doc.rust-lang.org/1.85.1/std/macro.println.html +https://doc.rust-lang.org/1.85.1/std/macro.println.html#panics +https://doc.rust-lang.org/1.85.1/std/macro.panic.html +https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#conditional-compilation +https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#the-cfg-attribute +https://doc.rust-lang.org/1.85.1/std/macro.compile_error.html +https://doc.rust-lang.org/1.85.1/reference/expressions/operator-expr.html#comparison-operators +https://doc.rust-lang.org/1.85.1/reference/expressions/if-expr.html +https://doc.rust-lang.org/1.85.1/std/num/struct.NonZero.html#method.new_unchecked +https://doc.rust-lang.org/1.85.1/reference/behavior-considered-undefined.html +https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#life-cycle-of-a-build-script +https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#outputs-of-the-build-script +https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rustc-cfg +https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rerun-if-env-changed +https://doc.rust-lang.org/1.85.1/cargo/reference/features.html diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/L.txt b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/L.txt new file mode 100644 index 0000000000..7c3ef37288 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/L.txt @@ -0,0 +1,7 @@ +https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked +https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.len +https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.is_empty +https://doc.rust-lang.org/1.82.0/reference/expressions/if-expr.html#if-expressions +https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#arithmetic-and-logical-binary-operators +https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#overflow +https://doc.rust-lang.org/1.82.0/reference/types/numeric.html#integer-types diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/P.txt b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/P.txt new file mode 100644 index 0000000000..0e8296d465 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/P.txt @@ -0,0 +1,15 @@ +https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_or +https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_unchecked +https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_or +https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_unchecked +https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_or +https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_unchecked +https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#conditional-compilation +https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#the-cfg-attribute +https://doc.rust-lang.org/1.84.0/std/macro.compile_error.html +https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#conditional-compilation +https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#the-cfg-attribute +https://doc.rust-lang.org/1.85.0/std/macro.compile_error.html +https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#conditional-compilation +https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#the-cfg-attribute +https://doc.rust-lang.org/1.86.0/std/macro.compile_error.html diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/Q.txt b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/Q.txt new file mode 100644 index 0000000000..d467a88cf6 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/Q.txt @@ -0,0 +1 @@ +https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html#safety diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/R.txt b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/R.txt new file mode 100644 index 0000000000..6852f5fd70 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/allowlists/R.txt @@ -0,0 +1,6 @@ +https://doc.rust-lang.org/1.82.0/reference/visibility-and-privacy.html#visibility-and-privacy +https://doc.rust-lang.org/1.82.0/reference/items/implementations.html#trait-implementations +https://doc.rust-lang.org/1.82.0/reference/items/traits.html#unsafe-traits +https://doc.rust-lang.org/1.82.0/reference/unsafe-keyword.html#unsafe-functions-unsafe-fn +https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked_mut +https://doc.rust-lang.org/1.82.0/std/primitive.u32.html#method.wrapping_add diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/authority-manifest.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/authority-manifest.tsv new file mode 100644 index 0000000000..906d3c3a46 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/authority-manifest.tsv @@ -0,0 +1,57 @@ +mode requested_url fetch_url final_url status content_type bytes sha256 fragment_found retrieved_utc +P https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_or https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html 200 text/html 213924 a395b4f6dfe1fc5242b61a13ef3c0109033f8053b3fb0772b5eb74463ca69105 true 2026-08-01T17:27:56.679301Z +P https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_unchecked https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html 200 text/html 213924 a395b4f6dfe1fc5242b61a13ef3c0109033f8053b3fb0772b5eb74463ca69105 true 2026-08-01T17:27:56.679301Z +P https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_or https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html 200 text/html 214310 e4af7fc387fab0fd0fbc29db6442f41c7a35258d03b3b4e7fa645190f7b8c86e true 2026-08-01T17:27:56.748521Z +P https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_unchecked https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html 200 text/html 214310 e4af7fc387fab0fd0fbc29db6442f41c7a35258d03b3b4e7fa645190f7b8c86e true 2026-08-01T17:27:56.748521Z +P https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_or https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html 200 text/html 214377 733fe2d8a415699250fdba975b9b943c02c5fe27e2dfe0822eb1d20f522864ea true 2026-08-01T17:27:56.813730Z +P https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_unchecked https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html 200 text/html 214377 733fe2d8a415699250fdba975b9b943c02c5fe27e2dfe0822eb1d20f522864ea true 2026-08-01T17:27:56.813730Z +P https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#conditional-compilation https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html 200 text/html 36314 c231b94b17857c57b47336ebede1e3e81ee8d58db68e669a6b1da1026164ac3b true 2026-08-01T17:27:56.855076Z +P https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#the-cfg-attribute https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html 200 text/html 36314 c231b94b17857c57b47336ebede1e3e81ee8d58db68e669a6b1da1026164ac3b true 2026-08-01T17:27:56.855076Z +P https://doc.rust-lang.org/1.84.0/std/macro.compile_error.html https://doc.rust-lang.org/1.84.0/std/macro.compile_error.html https://doc.rust-lang.org/1.84.0/std/macro.compile_error.html 200 text/html 6436 b92f2f9939bf888edd27d156eea4f5a48a98867752198001dc6b12253973fedd true 2026-08-01T17:27:56.891483Z +P https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#conditional-compilation https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html 200 text/html 36314 083c3862cabe9c0b67cb63a12e254376b7769e8826d0864b32ab9fc8a737f31f true 2026-08-01T17:27:56.934066Z +P https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#the-cfg-attribute https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html 200 text/html 36314 083c3862cabe9c0b67cb63a12e254376b7769e8826d0864b32ab9fc8a737f31f true 2026-08-01T17:27:56.934066Z +P https://doc.rust-lang.org/1.85.0/std/macro.compile_error.html https://doc.rust-lang.org/1.85.0/std/macro.compile_error.html https://doc.rust-lang.org/1.85.0/std/macro.compile_error.html 200 text/html 6436 2a9462a947c44b60b1131aa5f1378b8ebbb47df234aa152232d935f5a95bc8bc true 2026-08-01T17:27:56.973621Z +P https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#conditional-compilation https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html 200 text/html 39866 f5ec1c4381d47fa28bbfe7b92f0c4383c0f8a30bc7121831fa4a4cd4a5543f4c true 2026-08-01T17:27:57.020467Z +P https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#the-cfg-attribute https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html 200 text/html 39866 f5ec1c4381d47fa28bbfe7b92f0c4383c0f8a30bc7121831fa4a4cd4a5543f4c true 2026-08-01T17:27:57.020467Z +P https://doc.rust-lang.org/1.86.0/std/macro.compile_error.html https://doc.rust-lang.org/1.86.0/std/macro.compile_error.html https://doc.rust-lang.org/1.86.0/std/macro.compile_error.html 200 text/html 6503 0aa2780471d5ac4b0b5e746ea7aa01eb78fbabf9cd584a96d3088ec1afcc5883 true 2026-08-01T17:27:57.057647Z +B https://doc.rust-lang.org/1.85.1/std/env/fn.var.html https://doc.rust-lang.org/1.85.1/std/env/fn.var.html https://doc.rust-lang.org/1.85.1/std/env/fn.var.html 200 text/html 5859 4254c02ef260e2a7729980cc6f84796a5b3770ac58334a598fc0b745df2d805f true 2026-08-01T17:27:57.098070Z +B https://doc.rust-lang.org/1.85.1/std/env/enum.VarError.html https://doc.rust-lang.org/1.85.1/std/env/enum.VarError.html https://doc.rust-lang.org/1.85.1/std/env/enum.VarError.html 200 text/html 36295 7d3b01ebf3fbb4231711605c08cba59bc18620355b8a280d0b1cb8d355465932 true 2026-08-01T17:27:57.138948Z +B https://doc.rust-lang.org/1.85.1/std/result/enum.Result.html https://doc.rust-lang.org/1.85.1/std/result/enum.Result.html https://doc.rust-lang.org/1.85.1/std/result/enum.Result.html 200 text/html 197333 2f69f0d6f4583c410a76edc91a4c75ece27d01842e510f58c7fddf065ebfca89 true 2026-08-01T17:27:57.204497Z +B https://doc.rust-lang.org/1.85.1/std/primitive.u8.html https://doc.rust-lang.org/1.85.1/std/primitive.u8.html https://doc.rust-lang.org/1.85.1/std/primitive.u8.html 200 text/html 1160026 3ccfd4f00c432ca6884ed62a38078fe5b56fe846ffc66564ec65825d1cf74d4f true 2026-08-01T17:27:57.310407Z +B https://doc.rust-lang.org/1.85.1/reference/expressions/match-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/match-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/match-expr.html 200 text/html 17731 281bbc7dfe2c0f330e7229684b55747cd3392845fad3a5be536ec5a4918ed695 true 2026-08-01T17:27:57.349298Z +B https://doc.rust-lang.org/1.85.1/reference/expressions/block-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/block-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/block-expr.html 200 text/html 23305 eca5d1df45aaac303bdaa0a7a0a7e93f2e48b38171325e963bed90d3dbaf31ba true 2026-08-01T17:27:57.387388Z +B https://doc.rust-lang.org/1.85.1/reference/items/functions.html#function-body https://doc.rust-lang.org/1.85.1/reference/items/functions.html https://doc.rust-lang.org/1.85.1/reference/items/functions.html 200 text/html 38522 29509dc1a7f35b06ff836884a33bd5877871e4b56e7b78667f2641649802be15 true 2026-08-01T17:27:57.429150Z +B https://doc.rust-lang.org/1.85.1/reference/expressions/call-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/call-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/call-expr.html 200 text/html 14911 e2161195a1da5fcd2dcbb738db6d8d3bbe801774b46dd3d90a525c16a9aa07f7 true 2026-08-01T17:27:57.467148Z +B https://doc.rust-lang.org/1.85.1/reference/patterns.html#tuple-struct-patterns https://doc.rust-lang.org/1.85.1/reference/patterns.html https://doc.rust-lang.org/1.85.1/reference/patterns.html 200 text/html 58174 4e3e6090c0a65f5d26e7ae807f4241f35b91e8fbd24a76ecb23a07bf4fd6d0cf true 2026-08-01T17:27:57.516001Z +B https://doc.rust-lang.org/1.85.1/reference/patterns.html#literal-patterns https://doc.rust-lang.org/1.85.1/reference/patterns.html https://doc.rust-lang.org/1.85.1/reference/patterns.html 200 text/html 58174 4e3e6090c0a65f5d26e7ae807f4241f35b91e8fbd24a76ecb23a07bf4fd6d0cf true 2026-08-01T17:27:57.516001Z +B https://doc.rust-lang.org/1.85.1/reference/patterns.html#wildcard-pattern https://doc.rust-lang.org/1.85.1/reference/patterns.html https://doc.rust-lang.org/1.85.1/reference/patterns.html 200 text/html 58174 4e3e6090c0a65f5d26e7ae807f4241f35b91e8fbd24a76ecb23a07bf4fd6d0cf true 2026-08-01T17:27:57.516001Z +B https://doc.rust-lang.org/1.85.1/std/string/struct.String.html#method.as_str https://doc.rust-lang.org/1.85.1/std/string/struct.String.html https://doc.rust-lang.org/1.85.1/std/string/struct.String.html 200 text/html 577036 d94dac44c1c9d360c7caf0806c0c41b4d2330913463d0be42eeb211416563006 true 2026-08-01T17:27:57.595899Z +B https://doc.rust-lang.org/1.85.1/std/macro.println.html https://doc.rust-lang.org/1.85.1/std/macro.println.html https://doc.rust-lang.org/1.85.1/std/macro.println.html 200 text/html 6697 05b1c308b73cceceb09edd797b1cfb51b88039c26c9cd018aeda106fad2aa076 true 2026-08-01T17:27:57.634145Z +B https://doc.rust-lang.org/1.85.1/std/macro.println.html#panics https://doc.rust-lang.org/1.85.1/std/macro.println.html https://doc.rust-lang.org/1.85.1/std/macro.println.html 200 text/html 6697 05b1c308b73cceceb09edd797b1cfb51b88039c26c9cd018aeda106fad2aa076 true 2026-08-01T17:27:57.634145Z +B https://doc.rust-lang.org/1.85.1/std/macro.panic.html https://doc.rust-lang.org/1.85.1/std/macro.panic.html https://doc.rust-lang.org/1.85.1/std/macro.panic.html 200 text/html 10472 c7f3fe8e00889d1d9206fa39a373cfd060b993c5377b169979cc947e9efcecc0 true 2026-08-01T17:27:57.672756Z +B https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#conditional-compilation https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html 200 text/html 36314 083c3862cabe9c0b67cb63a12e254376b7769e8826d0864b32ab9fc8a737f31f true 2026-08-01T17:27:57.712603Z +B https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#the-cfg-attribute https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html 200 text/html 36314 083c3862cabe9c0b67cb63a12e254376b7769e8826d0864b32ab9fc8a737f31f true 2026-08-01T17:27:57.712603Z +B https://doc.rust-lang.org/1.85.1/std/macro.compile_error.html https://doc.rust-lang.org/1.85.1/std/macro.compile_error.html https://doc.rust-lang.org/1.85.1/std/macro.compile_error.html 200 text/html 6436 c9f3ba7f85fed765c6485ac0754891ee29b9f68efee68d60967a547264ffb748 true 2026-08-01T17:27:57.755232Z +B https://doc.rust-lang.org/1.85.1/reference/expressions/operator-expr.html#comparison-operators https://doc.rust-lang.org/1.85.1/reference/expressions/operator-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/operator-expr.html 200 text/html 58441 da30ee41874a27f0d5d991a9f53dda6d4b59e46e8f28ad06c0c714b941a9d3e0 true 2026-08-01T17:27:57.805856Z +B https://doc.rust-lang.org/1.85.1/reference/expressions/if-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/if-expr.html https://doc.rust-lang.org/1.85.1/reference/expressions/if-expr.html 200 text/html 16334 66cb23b4db83562de51a0b3411d473b430c738aa934c44b75f30aa2dd02c8f05 true 2026-08-01T17:27:57.842840Z +B https://doc.rust-lang.org/1.85.1/std/num/struct.NonZero.html#method.new_unchecked https://doc.rust-lang.org/1.85.1/std/num/struct.NonZero.html https://doc.rust-lang.org/1.85.1/std/num/struct.NonZero.html 200 text/html 1396156 c1393af3fca7ba25fd448504d6f70fc01fea71efac9cc50052178899ccb877dc true 2026-08-01T17:27:57.940957Z +B https://doc.rust-lang.org/1.85.1/reference/behavior-considered-undefined.html https://doc.rust-lang.org/1.85.1/reference/behavior-considered-undefined.html https://doc.rust-lang.org/1.85.1/reference/behavior-considered-undefined.html 200 text/html 29127 873f77da05771f3c06a189cec0ab15c69a46d8e968dc7879c1d0fb11376b8ab6 true 2026-08-01T17:27:57.980517Z +B https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#life-cycle-of-a-build-script https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html 200 text/html 40211 1247cbaf8ce775f17349367d13ac4eecc6d9cfa343310f12d8c1deccd19e07b2 true 2026-08-01T17:27:58.029869Z +B https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#outputs-of-the-build-script https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html 200 text/html 40211 1247cbaf8ce775f17349367d13ac4eecc6d9cfa343310f12d8c1deccd19e07b2 true 2026-08-01T17:27:58.029869Z +B https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rustc-cfg https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html 200 text/html 40211 1247cbaf8ce775f17349367d13ac4eecc6d9cfa343310f12d8c1deccd19e07b2 true 2026-08-01T17:27:58.029869Z +B https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rerun-if-env-changed https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html 200 text/html 40211 1247cbaf8ce775f17349367d13ac4eecc6d9cfa343310f12d8c1deccd19e07b2 true 2026-08-01T17:27:58.029869Z +B https://doc.rust-lang.org/1.85.1/cargo/reference/features.html https://doc.rust-lang.org/1.85.1/cargo/reference/features.html https://doc.rust-lang.org/1.85.1/cargo/reference/features.html 200 text/html 36803 96b2337cd60180df5a8566f343e52938dfcaa369bf12e1e82723a2326f64cb25 true 2026-08-01T17:27:58.075968Z +L https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked https://doc.rust-lang.org/1.82.0/std/primitive.slice.html https://doc.rust-lang.org/1.82.0/std/primitive.slice.html 200 text/html 828828 19f1230aa1d36c1e19eb9077a14bdb12b252d327ec3bab8c1f69d74a636a86ef true 2026-08-01T17:27:58.166084Z +L https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.len https://doc.rust-lang.org/1.82.0/std/primitive.slice.html https://doc.rust-lang.org/1.82.0/std/primitive.slice.html 200 text/html 828828 19f1230aa1d36c1e19eb9077a14bdb12b252d327ec3bab8c1f69d74a636a86ef true 2026-08-01T17:27:58.166084Z +L https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.is_empty https://doc.rust-lang.org/1.82.0/std/primitive.slice.html https://doc.rust-lang.org/1.82.0/std/primitive.slice.html 200 text/html 828828 19f1230aa1d36c1e19eb9077a14bdb12b252d327ec3bab8c1f69d74a636a86ef true 2026-08-01T17:27:58.166084Z +L https://doc.rust-lang.org/1.82.0/reference/expressions/if-expr.html#if-expressions https://doc.rust-lang.org/1.82.0/reference/expressions/if-expr.html https://doc.rust-lang.org/1.82.0/reference/expressions/if-expr.html 200 text/html 33059 e24637bab6b663bafe38d36f87b7076d6ab38012880eda06774f236ca99f2615 true 2026-08-01T17:27:58.221164Z +L https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#arithmetic-and-logical-binary-operators https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html 200 text/html 71217 9a08c53fb94c774ec5e4ddc069672f912f879716db7c17f649174afec3c72561 true 2026-08-01T17:27:58.271355Z +L https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#overflow https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html 200 text/html 71217 9a08c53fb94c774ec5e4ddc069672f912f879716db7c17f649174afec3c72561 true 2026-08-01T17:27:58.271355Z +L https://doc.rust-lang.org/1.82.0/reference/types/numeric.html#integer-types https://doc.rust-lang.org/1.82.0/reference/types/numeric.html https://doc.rust-lang.org/1.82.0/reference/types/numeric.html 200 text/html 29770 7c21b867c4ba43f66984ece635049dedb96783d8de0490750cd2f1ee0fe44ead true 2026-08-01T17:27:58.311486Z +Q https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html#safety https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html 200 text/html 11603 555597c0db28f65466dd734a9f57d4aaca8abe7f6e0e256b3f0d64a877529fd3 true 2026-08-01T17:27:58.349799Z +R https://doc.rust-lang.org/1.82.0/reference/visibility-and-privacy.html#visibility-and-privacy https://doc.rust-lang.org/1.82.0/reference/visibility-and-privacy.html https://doc.rust-lang.org/1.82.0/reference/visibility-and-privacy.html 200 text/html 37410 12652a78fea47d509f6c4fa0249d343b3da7116d7a3acc41568d2eb3488f106b true 2026-08-01T17:27:58.390606Z +R https://doc.rust-lang.org/1.82.0/reference/items/implementations.html#trait-implementations https://doc.rust-lang.org/1.82.0/reference/items/implementations.html https://doc.rust-lang.org/1.82.0/reference/items/implementations.html 200 text/html 39950 93fd8a85a7febc6f5d2f08298c6583f85fad546eb9aa447431309ccf6a41710b true 2026-08-01T17:27:58.437384Z +R https://doc.rust-lang.org/1.82.0/reference/items/traits.html#unsafe-traits https://doc.rust-lang.org/1.82.0/reference/items/traits.html https://doc.rust-lang.org/1.82.0/reference/items/traits.html 200 text/html 43949 8074f9255f3de0cc71edad5cbfbd22aeb8ac5427568c8478783d2aaca15d8363 true 2026-08-01T17:27:58.485357Z +R https://doc.rust-lang.org/1.82.0/reference/unsafe-keyword.html#unsafe-functions-unsafe-fn https://doc.rust-lang.org/1.82.0/reference/unsafe-keyword.html https://doc.rust-lang.org/1.82.0/reference/unsafe-keyword.html 200 text/html 33136 0de12b87f6a3967939624241cf67fc897de0eb93eca2b893c58dd1b734c4e0ea true 2026-08-01T17:27:58.525106Z +R https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked_mut https://doc.rust-lang.org/1.82.0/std/primitive.slice.html https://doc.rust-lang.org/1.82.0/std/primitive.slice.html 200 text/html 828828 19f1230aa1d36c1e19eb9077a14bdb12b252d327ec3bab8c1f69d74a636a86ef true 2026-08-01T17:27:58.166084Z +R https://doc.rust-lang.org/1.82.0/std/primitive.u32.html#method.wrapping_add https://doc.rust-lang.org/1.82.0/std/primitive.u32.html https://doc.rust-lang.org/1.82.0/std/primitive.u32.html 200 text/html 1089970 15962c2e26e219d0df334635c8927ad7bb9007a38e237b529a61dce88aa4f4ef true 2026-08-01T17:27:58.618256Z diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/file-manifest.sha256 b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/file-manifest.sha256 new file mode 100644 index 0000000000..3570dc902a --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/file-manifest.sha256 @@ -0,0 +1,44 @@ +afd32e5fb859619e0cf35e9ac0603cec1d7c072ecdd11724f56ee008fb6a34e1 fetch_authority.py +7124ac918b5595c9eaa594511309d8ac1a2400456695cd3004db76f14d18ec21 freeze/allowlists/B.txt +afe540a0e03212b0d8554a6a522d2a5695a0eaadf07126c410ef831a563b9623 freeze/allowlists/L.txt +ecb90a801bfbf51797749e1eddf95b85a1acba5acc92b2c061612a317150c81e freeze/allowlists/P.txt +3dc7862cdc2fdc1d44fd97235ad043602bd26ab3b428209b94fe9fb0602c14f8 freeze/allowlists/Q.txt +d99084daff4c4505711f97a93884043a9bbf0d06df4f1d3d8342c98486be4713 freeze/allowlists/R.txt +48444682cfc13966ce2769add9d4fbde82426a7a6e8777c421be8b26d15ff293 freeze/authority-manifest.tsv +196565f62edafc31a624d1d50d2f9a2fc321fc21f30a0fd1626e1de8cf5d6030 freeze/oracle/B.md +ce15524387eb9456470da666173bf098400786e5e5562d33df8ec65bf3be4609 freeze/oracle/L.md +487c376c6ee139f183a3329beb9718a7214a044f1b1038f6202c994828dcfc8f freeze/oracle/P.md +7c82a83cf3fac56f3801272b261c19fe6c1ccc3247528126f33f155b3e146be6 freeze/oracle/Q.md +345d9d55bf5358459c1ed02a51ec7544b3be185399542d5f13a34aa86d637a6b freeze/oracle/R.md +e6a7aff86789fe13bf047c89a50bb2e3e9a324c48224bbe911633fa9f0b606a7 freeze/plan.md +beddc6c0c178b7418d71e1104332e4d0ee016a6b5569a63dc754061d0fbfa0af freeze/policies/isolation.md +40a09230a61a698a614c401942ec3214ebf23d56019b94b1c4035015a39e91d5 freeze/policies/reruns.md +bd72f8225b5208cc709f4a701049e9541b7921490a61f0e99289c8861f5b334b freeze/policies/tools.md +0cf1d1eb7979f5aa957470f729c56deacb7f3b7d58589ca40690f2ca1368e285 freeze/policies/word-count.md +cee67f1081d65782007fc3f59e2b5c16bab4444872faf6214eebe1d193469f23 freeze/prompts/adjudicator.md +defd494cbc25024438e7734b5b4e52b625189d423c8c428c9b0bf61d9ba0bc60 freeze/prompts/consistency.md +5c001fa4b647218d5c7a98468f66260b95d490fbdfc64b64217777b995df2782 freeze/prompts/report.md +ae7649ba7f57828ecff8f3c8c2b41987c65d67ff0cfb4142f7a17e08b7b55a5a freeze/prompts/scorer.md +2315169fea06726ccec30fdb5f8fe5a6ff7503fbbf6a4d253505e3b0ffdfb125 freeze/randomization/commitments.json +85c5bd04f2ec3521b86a8a0f831bfc26b4af74063af6366f188e3e582b0e7740 freeze/randomization/spec.md +13790584d0e5ede69cff8c2b1889be80c73df0722c4ea709c0e85ab82900a4bb freeze/rubrics/B.md +4da5564ee971cdb7d8584c60c62b87a0c377efbe1a69b60576f22364b412b02a freeze/rubrics/L.md +1e9e6622a34b3b0376b6912f676695e318edcd971ac1f4960810531ad854911d freeze/rubrics/P.md +e18aba632c0f7bd3f4f2725e9a2515b2e2f5f561e7ed3320df6ad8b5154df98e freeze/rubrics/Q.md +af075a1aac36a19d97ec57d1fe8509d12a4b323e43739bd57c1576b071b83faf freeze/rubrics/R.md +d3ba48b524059bf07bcbf9448053971ee931854ee80d7107246fb3761c5eca9e freeze/rubrics/SCORER.md +5adc9877917ff8d6a86ddb2f427e423d09acca241206a78b884cca8120f43ed3 freeze/schemas/adjudication-cells.schema.json +8540160022aebd3900d88936ba3924d9160435d9d7a9d61962d1195c743d097d freeze/schemas/adjudication.schema.json +c7720030785e0d87c17d1a1ed44a9ecf260571a459be50b0fd2df15931ffa8e5 freeze/schemas/consistency.schema.json +d5b606e1a976a1b1f0a6d90b005b583db56d8a44117f24b253c062b746ffb7ee freeze/schemas/event.schema.json +c6f477c42431a9ea89c92b486a8ae49b9ee2d39219f4c572b55e2fc0fce453b5 freeze/schemas/score.schema.json +2b371bece33a764562fba0b45a8dfa4690d429027c3cbc208f52433e6e4ef826 prepare.py +60fa30c54cb28379257b72fc5502fb954e472467820714212f3e4afe39d124e6 protocol.py +eaec76c198e02ba5af6c1570e40597f7ea7af5c4be1ffb8cee4865d33418888d sealed/blind-map.tsv +c19b30faa8cf7f688822a82955432a296efb0be91eeb6a8ee4c6a89c47cc6faa sealed/condition-map.tsv +980d3b873ee56c39565c8e8ed5f6cfa90e469f5c9c7d04d4b6871a73ac9675f4 sealed/consistency-schedule.tsv +06a479cdc228fda98dd769491c563101ca0422f68787b6c72686a82bc020a99e sealed/launch-schedule.tsv +7273b0d5ad9e07aaa8bd0997101194f23a6a3025f84e8718735d49754931ad5e sealed/presentation-orders.tsv +a58e6be07e08f312ad4e4d877a00d47297c7048e423d7842d20123f21dca1e86 sealed/scoring-schedule.tsv +c0bfef1587ef773d854acfc74343faed819bcb3650a8343a3c81a05086056869 sealed/seeds.json +f1c89a72bf11c705e04fa576d0a7979e24799ca1f714def6b4db6bd0a06d5837 sealed/target-map.tsv diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/B.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/B.md new file mode 100644 index 0000000000..cf644f9ffe --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/B.md @@ -0,0 +1,481 @@ +# V4 Focused Oracle — Ordered, Fallible Build Relation + +> **Evaluator-only material.** Never expose this file, its atoms, formulas, +> expected verdicts, hard errors, or authority inventory to a report agent. + +## Scoring Rules + +Score the proposition of each atom, not its notation or presentation. Accept an +explicit equivalent derivation. Do not infer an omitted build stage, failure +edge, source-selection step, or semantic premise from an endpoint summary. +Failure of an atom is not itself a hard error unless the report also makes the +false affirmative claim identified below. + +This fixture deliberately requests a maximal positive region. That request, +not a general audit convention, makes exactness and maximality part of B13. +There is no scoring requirement for incidental diagnostics, exact panic text, +partially written bytes within one failed `println!`, or build-script stdout +which does not affect a current library selection. + +## Intended Domain and Build Relation + +Let: + +```text +R = { Rust and Cargo 1.85.1 } +T = { X = x86_64-unknown-linux-gnu, + A = aarch64-unknown-linux-gnu, + W = wasm32-unknown-unknown } +F = { burst-off, burst-on } +L = { system, arena } +P = every Cargo profile supported by the fixture +D = { debug-assertions-off, debug-assertions-on } +I = { i | i is any u8 } + +Config(t,f,l,p,d) + iff t in T and f in F and l in L and p in P and d in D + and not (t = W and l = arena). + +Required = R cross Config cross I. +Q(t,f,l) iff t = A and f = burst-on and l = arena. +``` + +Both allocator values in `Config` are reachable through the supported Cargo +interface: an omitted selector or `system` selects `system`, and `arena` +selects `arena`. Raw rejected selector values and unsuccessful build attempts +are build-relation cases that must be accounted for, but they do not add a +library configuration to `Required`. + +For compactness, use these local operations: + +```text +RERUN = attempt println!("cargo::rerun-if-env-changed=FIXTURE_ALLOCATOR") +READ = evaluate env::var("FIXTURE_ALLOCATOR") +SYS = attempt println!("cargo::rustc-cfg=fixture_allocator=\"system\"") +ARENA = attempt println!("cargo::rustc-cfg=fixture_allocator=\"arena\"") +PANIC = the selected explicit panic, or the panic caused by a failed println! +RETURN = successful return from main +``` + +The complete claim-relevant local relation is: + +```text +missing: RERUN -> READ(NotPresent) -> SYS -> RETURN +Unicode system: RERUN -> READ(Ok(system)) -> SYS -> RETURN +Unicode arena: RERUN -> READ(Ok(arena)) -> ARENA -> RETURN +Unicode arena-stop: + RERUN -> READ(Ok(arena-stop)) -> ARENA -> PANIC +other Unicode: RERUN -> READ(Ok(other)) -> PANIC +non-Unicode: RERUN -> READ(NotUnicode) -> PANIC +``` + +Every arrow after a `println!` assumes that write succeeds. If `RERUN` fails, +it panics before `READ`, and no complete earlier directive is required. If +`SYS` or `ARENA` fails, `RERUN` is the material completed prefix and the script +panics before `RETURN` or the explicit `arena-stop` panic. If `ARENA` succeeds +on `arena-stop`, the material completed prefix is `[RERUN, ARENA]`, followed by +the explicit panic. Under exact accepted entry `BUILD-MAP-ORDERED`, every panic is an +unsuccessful script exit and produces no current library compilation, +regardless of completed prefix. + +On the three successful selector paths, `BUILD-MAP-ORDERED` interprets the final +allocator line as exactly one matching library cfg. Cargo's accepted feature +and target mappings, followed by the Rust conditional-compilation rules, +select the library source. The `W`/`arena` combination selects +`compile_error!`, so it does not produce a library and is genuinely outside +`Config`. The supported `Q` cell selects the first `lane_id` body and removes +the complementary checked body. + +A prior successful `arena` build necessarily completed `RERUN` before +`ARENA`. After the raw selector changes to `arena-stop`, `BUILD-MAP-ORDERED` makes +that prior selection stale, reruns the script, and refuses to present the old +library as the result of the current unsuccessful execution. Thus the +freshness canary is an effective rejection even though its current output +prefix includes the same arena cfg line. + +## Expected Source Results + +The exact maximal sound region requested by the target is: + +```text +SoundRegion = { case in Required | not Q(case) or case.i != 0 }. +``` + +Outside `Q`, the selected body panics when `i == 0`; reaching +`new_unchecked(i)` therefore implies `i != 0`. Inside `Q`, the selected body +calls `new_unchecked` without a check, so nonzero inputs are sound and zero is +not. The complement of `SoundRegion` within `Required` consists exactly of +`Q` with `i == 0` (for every supported profile and debug-assertion state). + +The safe call `lane_id(0)` in any `Q` case reaches +`NonZeroU8::new_unchecked(0)`. Its exact safety proposition is false, and the +Rust 1.85.1 standard-library contract states that zero produces undefined +behavior. Safe-API soundness over all of `Required` is therefore **UNSOUND +relative to `BUILD-MAP-ORDERED`**. + +For the documented zero-input panic postcondition: + +```text +RequiredPanic = { case in Required | case.i = 0 } +CoveredPanic = { case in RequiredPanic | not Q(case) }. +``` + +The regional theorem over `CoveredPanic` is **PROVED relative to +`BUILD-MAP-ORDERED`**. The whole `RequiredPanic` theorem is **UNPROVED**, not +`CONTRACT-BROKEN`: the only missing cases are executions already shown to +contain UB, so their apparent absence of an earlier panic is not a defined +behavioral counterexample. + +## Atoms + +- **B1 — Exact supported case predicate:** The report states a predicate + equivalent to `Required`, including Rust/Cargo 1.85.1, all three targets, + both feature states, both allocator models, every supported Cargo profile, + both debug-assertion states, every `u8`, and only the `W`/`arena` policy + exclusion. + - `scope_basis`: `Cargo.toml`, `SUPPORT.md`, and `BUILD.md`. + - `dependencies`: none; this is the theorem root. + - `accepted_alternatives`: symbolic products or a proved equivalent partition; no + Cartesian enumeration is required. + - `hard_errors`: omission + alone fails B1; claiming a whole-domain affirmative result over a contracted + predicate triggers BH1. + +- **B2 — Exhaustive raw-selector partition:** The report proves that the raw + environment domain is exactly missing, Unicode `system`, Unicode `arena`, + Unicode `arena-stop`, every other Unicode string, and every non-Unicode + value, with no seventh `env::var` outcome. + - `scope_basis`: `BUILD.md`, the + exact `env::var`/`VarError` contracts, and the outer and inner matches. + - `dependencies`: independent of B1; it supplies the cases used + by B3–B7. + - `accepted_alternatives`: any disjoint exhaustive partition that + preserves these behaviorally distinct classes. + - `hard_errors`: + omission alone fails B2; calling an incomplete partition complete and using + it for build closure triggers BH2. + +- **B3 — Complete system-success trace:** Conditional on its two writes + succeeding, each of the missing and Unicode-`system` classes follows exactly + `RERUN -> READ -> SYS -> RETURN`; it emits no arena cfg. + - `scope_basis`: the + ordered statements, `env::var`, match/pattern semantics, `String::as_str`, + and `println!`. + - `dependencies`: B2 identifies the two raw cases; + this is a local-source proposition and does not consume Cargo mapping. + - `accepted_alternatives`: combine the two classes after proving their + identical trace. + - `hard_errors`: omission alone fails B3; an + incompatible endpoint used for closure triggers BH2. + +- **B4 — Complete arena-success trace:** Conditional on its two writes + succeeding, Unicode `arena` follows exactly + `RERUN -> READ -> ARENA -> RETURN`; it emits no system cfg. + - `scope_basis`: + the same local Rust contracts as B3. + - `dependencies`: B2; this + trace is the reachability root for B8–B12. + - `accepted_alternatives`: an + equivalent state-transition row or prose proof. + - `hard_errors`: + omission alone fails B4; an incompatible endpoint used for closure triggers + BH2. + +- **B5 — `arena-stop` partial-prefix exit:** With successful writes, Unicode + `arena-stop` follows exactly + `RERUN -> READ -> ARENA -> PANIC`, so `[RERUN, ARENA]` is a completed output + prefix but the script does not return successfully and, under + `BUILD-MAP-ORDERED`, produces no current library compilation. + - `scope_basis`: + `BUILD.md`, local match/`println!`/`panic!` semantics, and only the exact + unsuccessful-exit proposition of `BUILD-MAP-ORDERED`. + - `dependencies`: + B2; this is distinct from a failed allocator write in B7. + - `accepted_alternatives`: notation may differ, but both ordered lines, the later panic, + and no-current-library consequence must appear. + - `hard_errors`: + omission alone fails B5; treating the partial arena line as a current + successful selector triggers BH4. + +- **B6 — Pre-allocator rejection traces:** Every other Unicode value and every + non-Unicode value follows `RERUN -> READ -> PANIC` after a successful rerun + write, so `[RERUN]` is its completed directive prefix; it attempts no + allocator cfg and produces no current library compilation under + `BUILD-MAP-ORDERED`. + - `scope_basis`: the wildcard and `NotUnicode` arms plus + the exact build premise. + - `dependencies`: B2; the two classes may + share an outcome only after both are accounted for. + - `accepted_alternatives`: + separate rows or one proved union. + - `hard_errors`: omission alone + fails B6; treating either class as an accepted allocator/configuration + triggers BH5. + +- **B7 — Stdout-failure edges:** The report accounts for failure of every + claim-relevant `println!`: failed `RERUN` panics before `READ`; failed `SYS` + or `ARENA` panics with completed prefix `[RERUN]`; and all such unsuccessful + exits produce no current library compilation under `BUILD-MAP-ORDERED`. It does + not require atomic line writes or assign meaning to an incomplete line. + - `scope_basis`: the exact `println!` Panics section, source order, + `panic!`'s main-thread result, and the accepted process-status proposition. + - `dependencies`: B3–B5 identify allocator-write sites; B7 is the + failure completion of those traces. + - `accepted_alternatives`: a single + quantified failure rule is preferred and passes if its instantiations are + readily checkable. + - `hard_errors`: omission alone fails B7; + claiming complete build coverage while a material failure edge supplies a + contrary current artifact triggers BH2 or BH4 as applicable. + +- **B8 — Successful Cargo-to-source selector mapping:** Only the successful + B3/B4 returns supply a current library compilation; Cargo interprets their + exact allocator line as exactly the corresponding `fixture_allocator` + key/value, maps the requested feature and target as admitted, and Rust cfg + rules select source from those resulting predicates. The report keeps + `BUILD-MAP-ORDERED` conspicuous and does not infer local output or Rust semantics + from it. + - `scope_basis`: B3/B4, exact `BUILD-MAP-ORDERED`, the reviewed Cargo + directive contracts, and versioned Rust cfg semantics. + - `dependencies`: B3 through B7. + - `accepted_alternatives`: a relational composition + or explicit stage-by-stage prose. + - `hard_errors`: silently + widening the accepted mapping triggers BH3; treating an unsuccessful prefix + as successful triggers BH4. + +- **B9 — Rerun/freshness theorem:** A prior successful `arena` build completed + `RERUN` before `ARENA`; changing the same target directory's raw selector to + `arena-stop` makes that selection stale, causes a new script run, and yields + no library for the current rejected build rather than reusing the prior arena + library. + - `scope_basis`: local B4/B5 order and precisely the rerun/staleness + proposition accepted in `BUILD-MAP-ORDERED`, reviewed against Cargo's + `rerun-if-env-changed` contract. + - `dependencies`: B4, B5, and the + no-library exit consequence. + - `accepted_alternatives`: a more general proof + for every accepted-to-rejected value change passes if it includes this + requested sequence. + - `hard_errors`: omission alone fails B9; + claiming this current build succeeds from the stale or partial arena output + triggers BH4. + +- **B10 — Effective wasm32/arena exclusion:** For either feature state and all + profiles/debug states, accepted `arena` selection plus the admitted wasm32 + target mapping makes both predicates of the library's first cfg true; Rust + `all` and cfg-attribute semantics therefore retain `compile_error!`, whose + contract makes compilation fail. No `W`/`arena` case is in `Required` or is + used as the UB witness. + - `scope_basis`: `SUPPORT.md`, B8, exact configuration + predicate and attribute semantics, and `compile_error!`. + - `dependencies`: B8. + - `accepted_alternatives`: any parametric proof covering both + feature states and all `P x D`. + - `hard_errors`: calling this pair + supported, using it as a valid-use witness, or asserting rejection without + the selected `compile_error!` proof triggers BH5. + +- **B11 — Exact library-body partition:** The report proves that the unchecked + first body is retained and the complementary checked body removed exactly in + `Q`, while the checked body is retained and the first body removed exactly + outside `Q`. In `Q`, B8 supplies `target_arch="aarch64"`, + `feature="burst"`, and `fixture_allocator="arena"`; Rust's key/value, `all`, + `not`, and cfg-attribute rules establish both directions of the partition. + - `scope_basis`: B8 and the exact versioned cfg propositions. + - `dependencies`: B4 and B8; this is the source-reachability edge for B12/B13. + - `accepted_alternatives`: expansion of the two cfg predicates or an + equivalent Boolean proof. + - `hard_errors`: omission alone fails + B11; declaring the supported unsafe branch unreachable by ignoring generated + cfgs triggers BH6. + +- **B12 — Existential UB certificate and verdict:** The report identifies the + well-typed safe call `lane_id(0)` in a supported `Q` case, uses B11 to reach + `NonZeroU8::new_unchecked(0)`, states that its exact nonzero safety + proposition is false, applies the Rust 1.85.1 zero-UB consequence, and reports + whole-`Required` soundness **UNSOUND relative to `BUILD-MAP-ORDERED`**. + - `scope_basis`: the public safe API, B1, B11, and the exact std contract. + - `dependencies`: B1 and B11. + - `accepted_alternatives`: any one supported + profile/debug state is enough for the existential certificate; proving all + such states is also correct. + - `hard_errors`: an incomplete + certificate followed by `UNSOUND`, or failure to issue `UNSOUND` after giving + every link, triggers BH7. + +- **B13 — Exact maximal sound region:** The report proves both containments for + `SoundRegion = { case in Required | !Q or i != 0 }`: every included case is + sound (the checked body dominates its unsafe call outside `Q`, and nonzero + directly satisfies the unsafe precondition inside `Q`), while every excluded + Required case is exactly `Q`/zero and has B12's UB execution. It labels this + the exact maximal source-level region relative to `BUILD-MAP-ORDERED`. + - `scope_basis`: the user-requested maximal theorem, B1, B11/B12, `if`/comparison/ + panic semantics, and `new_unchecked`. + - `dependencies`: B1, B8, + B11, B12. + - `accepted_alternatives`: an algebraically equivalent formula or + disjoint partition; a correct non-maximal remainder does not pass this + explicitly requested atom. + - `hard_errors`: omission or refusal to + maximize alone fails B13; calling a region maximal while including Q/zero, + excluding sound Q/nonzero cases, or excluding all of `Q` triggers BH8. + +- **B14 — Regional zero-panic theorem:** For every zero-input case outside + `Q`, cfg selection retains the checked body, `value == 0` makes the `if` + consequent execute, and `panic!` prevents reachability of the later unsafe + call. Thus `CoveredPanic = RequiredPanic intersect !Q` is **PROVED relative + to `BUILD-MAP-ORDERED`** across every applicable target/allocator/feature, + profile, and debug state. + - `scope_basis`: B1/B8/B11 and exact comparison, + `if`, cfg, and panic contracts. + - `dependencies`: B1, B8, and the + complement of B11. + - `accepted_alternatives`: a parametric proof or exact + finite configuration partition. + - `hard_errors`: incomplete + regional coverage alone fails B14; falsely claiming the region contains + Q/zero triggers BH9 or BH10 depending on the verdict. + +- **B15 — Whole-domain panic-postcondition verdict:** The report states that + the documented zero-input panic guarantee over all `RequiredPanic` is + **UNPROVED**, not `CONTRACT-BROKEN` and not `PROVED`: Q/zero contains UB, so + its apparent source prefix supplies neither a defined non-panic witness nor a + guaranteed observation, while B14 proves every other zero case. + - `scope_basis`: the exact behavioral quantifier, B12, B14, and whole-execution UB + reasoning. + - `dependencies`: B12 and B14. + - `accepted_alternatives`: + equivalent theorem-specific wording that keeps this verdict separate from + soundness. + - `hard_errors`: using the UB-containing case to claim + `CONTRACT-BROKEN` or an observation before UB triggers BH9; claiming the + universal panic theorem `PROVED` triggers BH10. + +## Hard Errors + +- **BH1:** Claims a whole-domain affirmative result after omitting a supported + release, target, feature, allocator, profile, debug state, or valid `u8` + input, or treats the supported `Q` cell as excluded. +- **BH2:** Claims the build relation or a dependent theorem complete while + replacing the requested ordered/fallible relation with an endpoint summary + that omits a material raw-selector class, failure edge, partial-prefix exit, + or necessary stage. +- **BH3:** Invents, silently widens, or treats as Rust authority any Cargo, + freshness, process-status, feature, or target proposition beyond exact + `BUILD-MAP-ORDERED`. +- **BH4:** Treats a cfg line emitted by an unsuccessful script, or a stale + artifact from an earlier selector, as the library selected for the current + failed build. +- **BH5:** Treats a rejected raw selector as an accepted allocator model; + treats wasm32/arena as supported; uses it as the safe-use UB witness; or + claims its effective rejection without the selected `compile_error!` proof. +- **BH6:** Ignores build-script output or exact Cargo/cfg interpretation and + consequently declares the supported `Q` unsafe branch unreachable. +- **BH7:** Reports `UNSOUND` without the valid safe use, reachability, false + exact safety proposition, and applicable UB consequence, or reports only + `UNPROVED` after explicitly establishing all four links. +- **BH8:** Claims an exact or maximal sound region that includes Q/zero or + excludes any sound Required case, including Q/nonzero. +- **BH9:** Reports `CONTRACT-BROKEN` for the zero-panic promise using only the + UB-containing Q/zero execution, or claims an observation is guaranteed + because it appears before that UB in source order. +- **BH10:** Reports the whole-`RequiredPanic` zero-panic theorem `PROVED` + despite leaving Q/zero without a defined proof. + +## Checked Authorities + +The evaluator must open each exact page, confirm the quoted proposition, and +confirm applicability to Rust/Cargo 1.85.1. The quotation is included to make +the semantic leaf explicit; punctuation-only rendering differences are +irrelevant. + +- `https://doc.rust-lang.org/1.85.1/std/env/fn.var.html` — “Fetches the + environment variable key from the current process”; returns + `VarError::NotPresent` when the variable is not set and + `VarError::NotUnicode` when its value is not valid Unicode. +- `https://doc.rust-lang.org/1.85.1/std/env/enum.VarError.html` — the enum is + exactly `NotPresent` or `NotUnicode(OsString)`; the variant descriptions + distinguish absence from a present value without valid Unicode data. +- `https://doc.rust-lang.org/1.85.1/std/result/enum.Result.html` — `Result` is + exactly `Ok(T)` or `Err(E)` and “represents either success (`Ok`) or failure + (`Err`).” +- `https://doc.rust-lang.org/1.85.1/std/primitive.u8.html` — `u8` is “The 8-bit + unsigned integer type,” with `MIN` equal to zero and `MAX` equal to 255; zero + is therefore an ordinary valid safe-call input. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/match-expr.html` — a + scrutinee value is “sequentially compared to the patterns in the arms until + a match is found,” and the first matching arm is chosen and entered. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/block-expr.html` — a + block sequentially executes its non-item statements and optional final + expression; its type is `()` when the final operand is omitted. +- `https://doc.rust-lang.org/1.85.1/reference/items/functions.html#function-body` + — an omitted function output is unit, and the body is conceptually wrapped + so that its body value is returned; an explicit `return`, if reached, + short-cuts that implicit return. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/call-expr.html` — “A + call expression calls a function”; if the function eventually returns, the + expression completes. +- `https://doc.rust-lang.org/1.85.1/reference/patterns.html#tuple-struct-patterns` + — tuple-struct patterns match tuple-struct and enum values satisfying all + their subpatterns and destructure that value. +- `https://doc.rust-lang.org/1.85.1/reference/patterns.html#literal-patterns` — + literal patterns “match exactly the same value as what is created by the + literal.” +- `https://doc.rust-lang.org/1.85.1/reference/patterns.html#wildcard-pattern` — + the wildcard pattern “matches any value.” +- `https://doc.rust-lang.org/1.85.1/std/string/struct.String.html#method.as_str` + — `as_str` “Extracts a string slice containing the entire `String`.” +- `https://doc.rust-lang.org/1.85.1/std/macro.println.html` — `println!` + “Prints to the standard output, with a newline,” supplying the successful + completed-line effect used by B3–B6. +- `https://doc.rust-lang.org/1.85.1/std/macro.println.html#panics` — `println!` + “Panics if writing to `io::stdout` fails.” +- `https://doc.rust-lang.org/1.85.1/std/macro.panic.html` — `panic!` “Panics the + current thread.” The process-status consequence consumed by this fixture is + separately and explicitly admitted by `BUILD-MAP-ORDERED`. +- `https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#conditional-compilation` + — a set configuration option is true; `all(...)` is true exactly when all + operands are true; and `not(p)` is true exactly when `p` is false. +- `https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#the-cfg-attribute` + — a true cfg predicate retains the attached thing without the cfg attribute, + while a false predicate removes it from the source. +- `https://doc.rust-lang.org/1.85.1/std/macro.compile_error.html` — + `compile_error!` “Causes compilation to fail with the given error message + when encountered.” +- `https://doc.rust-lang.org/1.85.1/reference/expressions/operator-expr.html#comparison-operators` + — `==` means equal via `PartialEq::eq`; for the primitive `u8` comparison in + this source it tests whether `value` equals zero. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/if-expr.html` — if the + Boolean condition is true the consequent executes; if false it is skipped. +- `https://doc.rust-lang.org/1.85.1/std/num/struct.NonZero.html#method.new_unchecked` + — the function creates a nonzero without checking, “results in undefined + behavior if the value is zero,” and its Safety section says, “The value must + not be zero.” +- `https://doc.rust-lang.org/1.85.1/reference/behavior-considered-undefined.html` + — Rust programs “must never cause undefined behavior”; unsafe code that no + safe client can trigger to exhibit UB is called sound, and code that safe + code can misuse to exhibit UB is unsound. The whole-execution conclusion is + a logical consequence of classifying the execution as undefined, not a + purported quotation from a source-order or “time travel” subsection. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#life-cycle-of-a-build-script` + — after a build script “successfully finishes executing, the rest of the + package will be compiled”; a nonzero exit halts the build. This reviews, but + does not replace, `BUILD-MAP-ORDERED`. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#outputs-of-the-build-script` + — Cargo interprets each stdout line beginning with `cargo::` as an instruction + affecting package compilation, and instruction order can matter. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rustc-cfg` + — the directive tells Cargo to pass its value to rustc's `--cfg` flag for + conditional compilation. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rerun-if-env-changed` + — the directive tells Cargo to rerun the script if that environment + variable's value changes. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/features.html` — Cargo sets + enabled package features with rustc's `--cfg`, and source can test them with + the cfg attribute or macro. + +Cargo pages are evidence used to review the explicit human trust entry; they +are not Rust abstract-semantics axioms and do not silently enlarge that entry. +No test run, CI result, compiler experiment, prior report, evaluator oracle, or +unversioned documentation closes any atom. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/L.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/L.md new file mode 100644 index 0000000000..e1b761f580 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/L.md @@ -0,0 +1,144 @@ +# V4 Focused Oracle — L: Local Proof Reconstruction + +> Evaluator-only material. Never expose this file, its atom labels, expected +> result, or hard-error rules to an evaluated report agent. + +## Exact result + +Within the scope in `REQUEST.md`, `last` is sound. On the `else` branch, +`bytes.is_empty()` evaluated to false. The documented relation between +`is_empty` and slice length therefore gives `bytes.len() != 0`. Slice length has +type `usize`, whose values are nonnegative, so `bytes.len() > 0`. Consequently +`bytes.len() - 1` neither underflows nor overflows and mathematical subtraction +gives `index < bytes.len()`. This is precisely the in-bounds fact required by +`get_unchecked` for a `usize` index. + +The existing comment states only a lifetime fact. It does not address the +unsafe callee's in-bounds obligation, the facts which establish it, or the +derivation between them. It is therefore a deficient proof artifact even though +the implementation obligation can be reconstructed. A replacement must expose +that chain locally and must link or unambiguously refer to the checked, +version-matched authority entries it consumes. + +## Atoms + +- **L1 — Callee obligation:** State that the executed `get_unchecked(index)` + requires `index` to be in bounds, equivalently `index < bytes.len()` for this + `usize` index, and that an out-of-bounds call is UB. + - `scope_basis`: `lib.rs` contains this unsafe call and `REQUEST.md` asks for + an implementation soundness audit. + - `dependencies`: none. +- **L2 — Branch fact:** Establish from the actual `if`/`else` control flow that + `bytes.is_empty()` is false on the unsafe-call path, then use the exact + `is_empty` contract to conclude `bytes.len() != 0`. + - `scope_basis`: necessary source-path premise for the requested local proof. + - `dependencies`: none. +- **L3 — Unsigned-domain step:** Establish that the slice length is a `usize` + and, using its nonnegative value domain, derive `bytes.len() > 0` from L2. + - `scope_basis`: necessary material bridge from nonemptiness to safe + subtraction in every supported pointer width. + - `dependencies`: L2. +- **L4 — Defined subtraction:** Establish that `bytes.len() - 1` has no integer + overflow or underflow on this path, including in every ordinary profile. + - `scope_basis`: the source performs built-in integer subtraction before the + unsafe call; its definedness is part of the requested soundness proof. + - `dependencies`: L3. +- **L5 — In-bounds derivation:** Use the assignment + `index = bytes.len() - 1` and ordinary integer arithmetic to derive + `index < bytes.len()`. + - `scope_basis`: the exact proposition needed to discharge L1. + - `dependencies`: L3, L4. +- **L6 — Implementation closure:** Connect L2–L5 to L1 and report the + implementation obligation `PROVED` over the entire stated Rust/target/profile + scope; do not infer implementation failure from comment failure. + - `scope_basis`: `REQUEST.md` explicitly asks for an implementation verdict + over its full scope. + - `dependencies`: L1, L2, L3, L4, L5. +- **L7 — Artifact verdict:** Separately report the existing `SAFETY` comment + materially deficient because its lifetime statement does not address either + the exact in-bounds obligation or the material derivation which discharges + that obligation. + - `scope_basis`: `REQUEST.md` explicitly asks for a separate comment-adequacy + determination. + - `dependencies`: L1, L2, L3, L4, L5. +- **L8 — Replacement proof artifact:** Supply proposed adjacent comment text + which identifies the `get_unchecked` obligation and exposes the L2–L5 chain, + with unambiguous references to the checked authority propositions. + - `scope_basis`: `REQUEST.md` explicitly requests replacement wording when + material reconstruction is needed. + - `dependencies`: L1, L2, L3, L4, L5. +- **L9 — Complete premise inventory:** Inventory every materially consumed + Rust/std premise: `if` branch selection, `is_empty`/length equivalence, + `len`'s type/meaning, `usize`'s value domain, built-in + subtraction/overflow, and `get_unchecked`'s safety contract. Classify the + immediate placement and assignment observations as source facts and the + remaining integer implications as mathematics rather than inventing + authority for them. + - `scope_basis`: `REQUEST.md` explicitly requires a complete Rust-premise + inventory and reconciliation; distinguishing source facts and mathematics + identifies which proof leaves require Rust/std authority. + - `dependencies`: L1, L2, L3, L4, L5. +- **L10 — Exact checked authority:** For every Rust/std entry in L9, give the + matching Rust 1.82.0 narrow URL, quote the exact prose which supplies the + premise, state the verified proposition, and qualify its applicability. Page + identity or an unexamined citation alone does not pass. + - `scope_basis`: `REQUEST.md` explicitly requires version-matched checked + authority and exact extracted propositions. + - `dependencies`: L9. +- **L11 — Reconciliation:** Explicitly reconcile proof to inventory: every + materially consumed Rust/std premise has an L10 entry, every listed entry is + actually consumed, and no TCB assumption or uncited Rust fact silently fills + a link. + - `scope_basis`: `REQUEST.md` explicitly requires this reconciliation and + states that the TCB is empty. + - `dependencies`: L1, L2, L3, L4, L5, L9, L10. + +Score atoms independently. Mathematical implications between stated integer +facts need derivation but no Rust citation. Immediate source syntax may be cited +by source location rather than external authority. + +## Hard errors + +- **LH1:** Report the implementation `UNSOUND` or `UNPROVED` solely because the + existing comment is deficient, despite reconstructing L1–L5 correctly. +- **LH2:** Explicitly call the existing comment adequate. +- **LH3:** Claim the requested implementation proof complete while omitting a + material link from L1–L5. +- **LH4:** Claim the requested authority/premise reconciliation complete while + silently consuming a Rust/std premise, using an inapplicable version, or + treating a page URL as proof without checking the proposition. +- **LH5:** Use build, test, execution, lint, or dynamic-analysis success as a + universal proof of this obligation. + +An omitted artifact verdict fails L7 but is not independently a hard error +unless the report also makes an affirmative false claim covered above. + +## Checked authority + +- Rust 1.82.0 slice `get_unchecked`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked` + — for a `usize` element index, an out-of-bounds call is UB even when the + result is unused; the proof consumes the corresponding `index < len` + requirement. +- Rust 1.82.0 slice `len`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.len` + — returns the number of elements as `usize`. +- Rust 1.82.0 slice `is_empty`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.is_empty` + — returns true exactly when the slice has length zero. +- Rust 1.82.0 Reference, `if` expressions: + `https://doc.rust-lang.org/1.82.0/reference/expressions/if-expr.html#if-expressions` + — when the condition evaluates to false, the consequent block is skipped and + the trailing `else` block is executed. +- Rust 1.82.0 Reference, arithmetic operators: + `https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#arithmetic-and-logical-binary-operators` + — `-` is built-in subtraction for integer operands. +- Rust 1.82.0 Reference, integer overflow: + `https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#overflow` + — identifies overflow for built-in integer operators and its profile/context + behavior; L3 proves the exceptional case absent here. +- Rust 1.82.0 Reference, unsigned integer value domains: + `https://doc.rust-lang.org/1.82.0/reference/types/numeric.html#integer-types` + — `usize` is an unsigned integer type and its values are nonnegative. + +There is no accepted TCB entry for this mode. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/P.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/P.md new file mode 100644 index 0000000000..149f61be03 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/P.md @@ -0,0 +1,563 @@ +# Mode P Frozen Blind-Scoring Rubric + +> **Evaluator-only material. Never expose this file to a report agent.** + +## P — Incomparable Predicates and Full-Case Closure + +### Canonical notation and policy algebra + +Use the fixture's notation: + +```text +V = {1.84.0, 1.85.0, 1.86.0} +T = {X, A, W} +F = H = D = {false, true} +P = every Cargo profile admitted by the supplied policies +I = {None} union {Some(b) | b is any u8} +case = (v, t, f, h, p, d, i) +``` + +The two exact configuration predicates are: + +```text +Scarlet(v,t,f,h) + iff v in V and t in T and + (!f + or (f and t = X and (!h or v >= 1.85.0)) + or (f and t = A and h)). + +Indigo(v,t,f,h) + iff v in V and t in T and + (!f + or (f and t = X and (h or v >= 1.86.0)) + or (f and t = A and !h and v >= 1.85.0)). +``` + +They are incomparable. For example: + +```text +(1.84.0, X, true, false) is Scarlet-only. +(1.84.0, X, true, true) is Indigo-only. +``` + +Any correct separating witness is acceptable. In particular, an answer need +not use the two examples above. There is no authorized rule identifying one +predicate, their intersection, their union, or any other combination as the +crate's controlling exact promise. + +The policy-induced full-case sets are: + +```text +ScarletCases = {case | Scarlet(v,t,f,h) and p in P and d in D and i in I} +IndigoCases = {case | Indigo(v,t,f,h) and p in P and d in D and i in I} +``` + +The canonical minimal conservative audit domain is their union: + +```text +Audit = ScarletCases union IndigoCases +Required(case) iff case in Audit +``` + +Its configuration projection has this equivalent exact formula: + +```text +UnionCfg(v,t,f,h) + iff v in V and t in T and + (!f + or (f and t = X) + or (f and t = A and (h or v >= 1.85.0))). + +Required(v,t,f,h,p,d,i) + iff UnionCfg(v,t,f,h) and p in P and d in D and i in I. +``` + +A report may instead choose a transparent full-case superset, provided it +proves `ScarletCases subset-of Audit` and `IndigoCases subset-of Audit` +separately and states a theorem whose meaning remains coherent for every added +case. The chosen set is an audit domain, not an inferred project promise. + +### Intended source proof + +Both policies exclude every configuration with `f = true` and `t = W`. +Relative only to accepted `BUILD-MAP-POLICY`, such a build selects +`cfg(all(feature = "turbo", target_arch = "wasm32"))`. The authoritative +`all`, `cfg`-attribute, and `compile_error!` contracts then establish that the +ordinary library compilation fails. This reasoning is parametric in `v`, `h`, +`p`, and `d`; there is no library input after compilation is rejected. + +For every compiled case selected by the audit domain: + +- when `f = false`, `unwrap_or(0)` returns the contained byte for `Some(byte)` + and zero for `None`; +- when `f = true`, the same `unwrap_or(0)` first produces that result `r`, the + source constructs `Some(r)`, and `unwrap_unchecked` is therefore invoked on + `Some`, never `None`; its return contract produces the same `r`. + +One exact full-case implementation region for both soundness and behavior is: + +```text +Covered(v,t,f,h,p,d,i) + iff v in V and t in T and f in F and h in H and p in P and d in D and i in I + and (!f or (f and t in {X,A})). +``` + +The source derivation is independent of `v`, `h`, `p`, `d`, and the particular +member of `I`, but those universally quantified dimensions remain in both +`Required(case)` and `Covered(case)`. Every disjunct of `UnionCfg` implies the +final source-selection clause above, so: + +```text +Required subset-of Covered_sound +Required subset-of Covered_behavior +``` + +The two covered predicates may be represented by the same set, but the unsafe +precondition proof and the postcondition proof are distinct obligations. + +### Expected conclusions + +- The identity of the controlling exact project predicate is **UNRESOLVED** + (equivalently, **UNPROVED** as a policy-identification proposition). +- Safe-API soundness on the chosen conservative full-case audit domain is + **PROVED relative to `BUILD-MAP-POLICY`**. +- The documented `value_or_zero` postcondition on that domain is **PROVED + relative to `BUILD-MAP-POLICY`**. +- Effective rejection of every `turbo`/`wasm32` build in the supplied release, + feature, profile, and debug-assertion scope is **PROVED relative to + `BUILD-MAP-POLICY`**. + +The qualification is required because the mapping from policy configuration +coordinates to Rust `cfg` options is an accepted tool premise, not a Rust +semantic axiom. + +### Scored atoms + +Each atom below scores exactly the proposition in its `proposition` field. +The other fields identify why that proposition is in scope and how to diagnose +a failure; they add no hidden report requirement. + +- **P1 — Exact Scarlet predicate** + - `proposition`: The report's `Scarlet(v,t,f,h)` set equals the predicate in + `POLICY-SCARLET.md`. + - `scope_basis`: “Reproduce the exact Scarlet and Indigo configuration + predicates separately.” (`REQUEST.md`) + - `necessary_edge`: Expand every Scarlet disjunct, including its versioned + x86 clause and hardened aarch64 clause. + - `dependencies`: none. + - `root_cause`: A wrong Scarlet formula is the root cause even if it later + causes witness or containment failures. + - `accepted_alternatives`: Any logically equivalent formula, truth-table + partition, or exact finite set description. + +- **P2 — Exact Indigo predicate** + - `proposition`: The report's `Indigo(v,t,f,h)` set equals the predicate in + `POLICY-INDIGO.md`. + - `scope_basis`: “Reproduce the exact Scarlet and Indigo configuration + predicates separately.” (`REQUEST.md`) + - `necessary_edge`: Expand every Indigo disjunct, including its versioned + x86 and unhardened aarch64 clauses. + - `dependencies`: none. + - `root_cause`: A wrong Indigo formula is the root cause even if it later + causes witness or containment failures. + - `accepted_alternatives`: Any logically equivalent formula, truth-table + partition, or exact finite set description. + +- **P3 — Scarlet-minus-Indigo witness** + - `proposition`: There exists a stated configuration in Scarlet but not + Indigo. + - `scope_basis`: “giving one concrete separating configuration in each + direction if neither contains the other.” (`REQUEST.md`) + - `necessary_edge`: Substitute the witness into both exact predicates and + establish `Scarlet and not Indigo`. + - `dependencies`: P1 and P2. + - `root_cause`: If substitution is correct only for a previously misstated + predicate, record that predicate atom as the originating failure. + - `accepted_alternatives`: Any valid Scarlet-only configuration. + +- **P4 — Indigo-minus-Scarlet witness** + - `proposition`: There exists a stated configuration in Indigo but not + Scarlet. + - `scope_basis`: “giving one concrete separating configuration in each + direction if neither contains the other.” (`REQUEST.md`) + - `necessary_edge`: Substitute the witness into both exact predicates and + establish `Indigo and not Scarlet`. + - `dependencies`: P1 and P2. + - `root_cause`: If substitution is correct only for a previously misstated + predicate, record that predicate atom as the originating failure. + - `accepted_alternatives`: Any valid Indigo-only configuration. + +- **P5 — Exact-promise status** + - `proposition`: The supplied policy evidence leaves the controlling exact + project predicate unidentified. + - `scope_basis`: “No precedence, supersession, intersection, union, or other + conflict-resolution rule has been authorized.” (`REQUEST.md`) + - `necessary_edge`: Keep policy identification separate from selection of a + conservative audit domain. + - `dependencies`: none. + - `root_cause`: Treating an analyst-selected set as policy evidence is the + originating error. + - `accepted_alternatives`: `UNRESOLVED`, or `UNPROVED` explicitly limited to + policy identification. + +- **P6 — Scarlet full-case domain** + - `proposition`: `ScarletCases` equals the Scarlet configuration set crossed + with every admitted profile, both debug-assertion states, and every valid + `Option` input. + - `scope_basis`: “Define the full-case domain induced by each policy.” + (`REQUEST.md`) + - `necessary_edge`: Preserve all seven named coordinates in the set. + - `dependencies`: P1. + - `root_cause`: A missing coordinate is a theorem-domain recovery failure, + not an implementation-proof failure. + - `accepted_alternatives`: Product notation, a set comprehension, or exact + universally quantified prose. + +- **P7 — Indigo full-case domain** + - `proposition`: `IndigoCases` equals the Indigo configuration set crossed + with every admitted profile, both debug-assertion states, and every valid + `Option` input. + - `scope_basis`: “Define the full-case domain induced by each policy.” + (`REQUEST.md`) + - `necessary_edge`: Preserve all seven named coordinates in the set. + - `dependencies`: P2. + - `root_cause`: A missing coordinate is a theorem-domain recovery failure, + not an implementation-proof failure. + - `accepted_alternatives`: Product notation, a set comprehension, or exact + universally quantified prose. + +- **P8 — Scarlet containment** + - `proposition`: The chosen conservative full-case audit domain contains + `ScarletCases`. + - `scope_basis`: “Prove each containment separately.” (`REQUEST.md`) + - `necessary_edge`: Prove `ScarletCases subset-of Audit` from the actual + stated formulas. + - `dependencies`: P6 and the report's definition of `Audit`. + - `root_cause`: A malformed upstream Scarlet domain remains the originating + failure; an absent inclusion argument is independently P8. + - `accepted_alternatives`: Membership in an exact union, algebraic + implication into a superset, or exhaustive finite proof. + +- **P9 — Indigo containment** + - `proposition`: The chosen conservative full-case audit domain contains + `IndigoCases`. + - `scope_basis`: “Prove each containment separately.” (`REQUEST.md`) + - `necessary_edge`: Prove `IndigoCases subset-of Audit` from the actual + stated formulas. + - `dependencies`: P7 and the report's definition of `Audit`. + - `root_cause`: A malformed upstream Indigo domain remains the originating + failure; an absent inclusion argument is independently P9. + - `accepted_alternatives`: Membership in an exact union, algebraic + implication into a superset, or exhaustive finite proof. + +- **P10 — Scarlet wasm exclusion** + - `proposition`: Every configuration with `f = true` and `t = W` is outside + Scarlet. + - `scope_basis`: “Account for both policy-level exclusion ... of every + `turbo`/`wasm32` case.” (`REQUEST.md`) + - `necessary_edge`: Evaluate the Scarlet predicate with those two fixed + coordinates while leaving `v` and `h` universal. + - `dependencies`: P1. + - `root_cause`: A wrong Scarlet formula is upstream; failure to apply the + correct formula here is local. + - `accepted_alternatives`: Direct substitution or exclusion by disjuncts. + +- **P11 — Indigo wasm exclusion** + - `proposition`: Every configuration with `f = true` and `t = W` is outside + Indigo. + - `scope_basis`: “Account for both policy-level exclusion ... of every + `turbo`/`wasm32` case.” (`REQUEST.md`) + - `necessary_edge`: Evaluate the Indigo predicate with those two fixed + coordinates while leaving `v` and `h` universal. + - `dependencies`: P2. + - `root_cause`: A wrong Indigo formula is upstream; failure to apply the + correct formula here is local. + - `accepted_alternatives`: Direct substitution or exclusion by disjuncts. + +- **P12 — Effective wasm rejection** + - `proposition`: Relative to exact `BUILD-MAP-POLICY`, every in-scope build with + `f = true` and `t = W` fails library compilation at the selected + `compile_error!`. + - `scope_basis`: “Account for ... source-level effective rejection of every + `turbo`/`wasm32` case.” (`REQUEST.md`) + - `necessary_edge`: Follow tool mapping to both true cfg options, apply + `all`, apply the `cfg` attribute, then apply `compile_error!`, universally + over release, hardened, profile, and debug-assertion coordinates. + - `dependencies`: exact TCB use and exact-version Rust authority. + - `root_cause`: The first absent edge among mapping, predicate selection, and + compile failure is the useful diagnosis. + - `accepted_alternatives`: An equivalent symbolic proof for all such builds; + enumeration is not required. + +- **P13 — Non-turbo postcondition** + - `proposition`: On every selected non-turbo branch and every valid input, + `value_or_zero` returns the contained byte for `Some(byte)` and zero for + `None`. + - `scope_basis`: “Prove or refute ... the documented `value_or_zero` + postcondition on each selected source branch.” (`REQUEST.md`) + - `necessary_edge`: Apply the exact-version `unwrap_or(0)` contract to the + two `Option` variants. + - `dependencies`: exact Option authority. + - `root_cause`: Missing or inapplicable `unwrap_or` authority is the semantic + root; missing input quantification is a domain root. + - `accepted_alternatives`: Variant cases or one exact contract application. + +- **P14 — Turbo unsafe precondition** + - `proposition`: Every reachable turbo-branch `unwrap_unchecked` call has a + `Some` receiver. + - `scope_basis`: “Prove or refute the local safety obligation at the unsafe + operation.” (`REQUEST.md`) + - `necessary_edge`: Trace the value through `unwrap_or`, the local + `Some(value)` construction, and the unchanged receiver at the unsafe call. + - `dependencies`: exact Option authority and the inspected local source. + - `root_cause`: Failure to connect the constructed receiver to the call is a + local proof-edge failure, not a policy failure. + - `accepted_alternatives`: Any equivalent local proof that identifies the + exact call receiver; no control-flow argument is required. + +- **P15 — Turbo postcondition** + - `proposition`: On every selected turbo branch and every valid input, + `value_or_zero` returns the contained byte for `Some(byte)` and zero for + `None`. + - `scope_basis`: “Prove or refute ... the documented `value_or_zero` + postcondition on each selected source branch.” (`REQUEST.md`) + - `necessary_edge`: Compose `unwrap_or(0)`'s result with construction of + `Some(result)` and `unwrap_unchecked`'s return contract. + - `dependencies`: P14 and exact Option authority. + - `root_cause`: A safety proof alone does not supply the return-value edge; + diagnose that edge separately. + - `accepted_alternatives`: Direct composition or explicit `None`/`Some` + cases. + +- **P16 — Full Required predicate** + - `proposition`: The report's final `Required(case)` retains version, target, + both features, profile, debug assertions, and input over its chosen audit + domain. + - `scope_basis`: “State `Required(case)` ... without projecting away the + configuration or input dimensions.” (`REQUEST.md`) + - `necessary_edge`: Bind all seven coordinates before making a universal + conclusion. + - `dependencies`: P6 through P9. + - `root_cause`: Omission here is a final theorem-domain failure even if the + omitted axis happens not to affect source behavior. + - `accepted_alternatives`: Exact set notation or universally quantified + prose; irrelevant dimensions may be symbolic. + +- **P17 — Full soundness Covered coordinates** + - `proposition`: `Covered_sound(case)` retains all seven case coordinates. + - `scope_basis`: “State ... the applicable soundness ... `Covered(case)` + predicates without projecting away the configuration or input dimensions.” + (`REQUEST.md`) + - `necessary_edge`: Lift the branch-local proof to a full-case predicate + rather than replacing the case by a configuration projection. + - `dependencies`: P16. + - `root_cause`: A dimensionless relation is a theorem-bookkeeping error even + when the branch proof itself is correct. + - `accepted_alternatives`: Exact set notation or universally quantified + prose; irrelevant dimensions may remain symbolic. + +- **P18 — Soundness Covered validity** + - `proposition`: Every case included in `Covered_sound(case)` has a complete + applicable unsafe-obligation derivation in the report. + - `scope_basis`: “State ... the applicable soundness ... `Covered(case)` + predicates.” (`REQUEST.md`) + - `necessary_edge`: Admit a case to Covered only after the branch-local + unsafe proof applies to that case. + - `dependencies`: P14 plus the inspected fact that the non-turbo branch uses + no unsafe operation. + - `root_cause`: An overbroad Covered set is an implementation-proof error, + not a theorem-domain recovery error. + - `accepted_alternatives`: The canonical region or any other proved region; + no maximal positive-region characterization is required. + +- **P19 — Soundness closure** + - `proposition`: `Required subset-of Covered_sound` holds for the report's + stated full-case sets. + - `scope_basis`: “Give the set-containment argument needed for each + whole-domain conclusion.” (`REQUEST.md`) + - `necessary_edge`: Map every audit-domain configuration disjunct to its + proved source branch while carrying profile, debug, and input universally. + - `dependencies`: P8, P9, P14, and P16 through P18. + - `root_cause`: Record every downstream atom as failed when absent, but name + the earliest wrong predicate, local proof, or containment edge as cause. + - `accepted_alternatives`: Algebraic implication, structured cases, or an + exact finite proof; enumeration is not required. + +- **P20 — Whole-domain soundness conclusion** + - `proposition`: Relative to exact `BUILD-MAP-POLICY`, safe-API soundness is + PROVED on the chosen audit domain. + - `scope_basis`: “state the strongest conclusions that the supplied evidence + justifies.” (`REQUEST.md`) + - `necessary_edge`: Use P19 as the certificate and no wider TCB proposition. + - `dependencies`: P19 and exact TCB accounting. + - `root_cause`: If closure is missing, that proof failure is primary; an + unqualified otherwise-correct verdict is a TCB-accounting failure. + - `accepted_alternatives`: Equivalent wording which clearly states domain, + source-level soundness, positive status, and TCB relativity. + +- **P21 — Full behavioral Covered coordinates** + - `proposition`: `Covered_behavior(case)` retains all seven case coordinates. + - `scope_basis`: “State ... the applicable ... behavioral `Covered(case)` + predicates without projecting away the configuration or input dimensions.” + (`REQUEST.md`) + - `necessary_edge`: Lift both branch-local return proofs to a full-case + predicate. + - `dependencies`: P16. + - `root_cause`: A dimensionless relation is a theorem-bookkeeping error even + when the return proof itself is correct. + - `accepted_alternatives`: Exact set notation or universally quantified + prose; irrelevant dimensions may remain symbolic. + +- **P22 — Behavioral Covered validity** + - `proposition`: Every case included in `Covered_behavior(case)` has a + complete applicable postcondition derivation in the report. + - `scope_basis`: “State ... the applicable ... behavioral `Covered(case)` + predicates.” (`REQUEST.md`) + - `necessary_edge`: Admit a case to Covered only after a branch-local return + proof applies to that case. + - `dependencies`: P13 and P15. + - `root_cause`: Do not infer this validity proposition from soundness; the + documented return theorem needs its own proof edges. + - `accepted_alternatives`: The canonical region or any other proved region; + no maximal positive-region characterization is required. + +- **P23 — Behavioral closure** + - `proposition`: `Required subset-of Covered_behavior` holds for the report's + stated full-case sets. + - `scope_basis`: “Give the set-containment argument needed for each + whole-domain conclusion.” (`REQUEST.md`) + - `necessary_edge`: Map every audit-domain configuration and input case to a + branch with the proved documented return. + - `dependencies`: P8, P9, P13, P15, P16, P21, and P22. + - `root_cause`: Record every downstream atom as failed when absent, but name + the earliest wrong predicate, postcondition proof, or containment edge as + cause. + - `accepted_alternatives`: Algebraic implication, structured cases, or an + exact finite proof; enumeration is not required. + +- **P24 — Whole-domain behavioral conclusion** + - `proposition`: Relative to exact `BUILD-MAP-POLICY`, the documented + postcondition is PROVED on the chosen audit domain. + - `scope_basis`: “state the strongest conclusions that the supplied evidence + justifies.” (`REQUEST.md`) + - `necessary_edge`: Use P23 as the certificate and no wider TCB proposition. + - `dependencies`: P23 and exact TCB accounting. + - `root_cause`: If closure is missing, that proof failure is primary; an + unqualified otherwise-correct verdict is a TCB-accounting failure. + - `accepted_alternatives`: Equivalent wording which clearly states domain, + exact postcondition, positive status, and TCB relativity. + +- **P25 — Exact TCB scope** + - `proposition`: Every proposition attributed to `BUILD-MAP-POLICY` is admitted by + that entry's exact identity, mapping, consumer, and release/profile region. + - `scope_basis`: “Apply it only to its exact build-tool mappings and + consumers.” (`REQUEST.md`) + - `necessary_edge`: Match each TCB-dependent conclusion to the supplied entry + without adding a semantic or backend proposition. + - `dependencies`: none; this is an accounting proposition. + - `root_cause`: An invented mapping is the primary error even if subsequent + cfg reasoning would be valid under it. + - `accepted_alternatives`: A compact TCB ledger or precise inline + qualifications. + +- **P26 — Conspicuous TCB qualification** + - `proposition`: Every conclusion which depends on `BUILD-MAP-POLICY` identifies + that dependency as an accepted trust premise rather than Rust authority. + - `scope_basis`: “keep every conclusion which depends on it conspicuously + qualified.” (`REQUEST.md`) + - `necessary_edge`: Carry the TCB label from configuration reachability into + each dependent rejection, soundness, and behavior conclusion. + - `dependencies`: none; this is a reporting proposition. + - `root_cause`: An otherwise valid but unconditional conclusion fails here; + do not misdiagnose it as a Rust-authority mismatch. + - `accepted_alternatives`: Inline `relative to BUILD-MAP-POLICY` wording or an + unambiguous shared qualification covering the dependent conclusions. + +- **P27 — Exact authority closure** + - `proposition`: Every material Rust semantic premise used in a positive + proof is paired with an applicable exact-version URL and a quotation which + supplies that premise. + - `scope_basis`: “For each material Rust semantic premise, cite the applicable + exact-version Rust Reference or standard-library page and quote the prose + which supplies that premise.” (`REQUEST.md`) + - `necessary_edge`: Reconcile the proof leaves against the authority inventory + below; a page label alone supplies no unstated proposition. + - `dependencies`: every positive Rust-semantic derivation. + - `root_cause`: Identify the first missing, mismatched, or overclaimed + quotation; do not treat one omission as multiple distinct authority roots. + - `accepted_alternatives`: Shorter exact excerpts and different organization + are acceptable when the proposition and release applicability are clear. + +### Hard errors + +A mere omission, an explicit refusal to close a proposition, or a downstream +atom failure caused by a disclosed upstream gap is not by itself a hard error. +Record a hard error only for the corresponding false affirmative claim below: + +- **PH1:** selects Scarlet, Indigo, their intersection, their union, or another + combination as the controlling exact project promise without new policy + evidence; +- **PH2:** relabels an analyst-selected conservative audit domain as the exact + project promise; +- **PH3:** asserts containment or incomparability using a stated formula and + witness which do not establish the asserted set relation; +- **PH4:** claims whole-domain soundness or behavior after dropping a required + configuration or input coordinate, or without a valid full-case + `Required subset-of Covered` argument; +- **PH5:** treats `turbo`/`wasm32` as policy-supported, or claims effective + rejection without the exact accepted mapping and applicable `all`, `cfg`, and + `compile_error!` implications; +- **PH6:** calls the shown source `UNSOUND` or calls its documented + postcondition `CONTRACT-BROKEN` on the supplied audit domain; +- **PH7:** invents or widens `BUILD-MAP-POLICY`, treats it as Rust authority, or uses + it outside its exact identity, propositions, consumers, or region. + +Record independently applicable global hard errors as well. + +### Exact authority propositions, quotations, and URLs + +The following quotations were checked on every listed exact-version page. The +quotation is the semantic content; presence on an allowlist is not itself +evidence for a proposition. + +For Rust 1.84.0, 1.85.0, and 1.86.0, the `Option` pages describe `unwrap_or` +using the “contained `Some` value” or “provided default,” and +`unwrap_unchecked` returning the “contained `Some` value.” Its Safety heading +says calling it on `None` “is undefined behavior.” Construction of +`Some(value)` is an immediate inspected source fact, not a separately cited +semantic premise. + +Exact pages, in canonical allowlist order: + +- `https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_or` +- `https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_unchecked` +- `https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_or` +- `https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_unchecked` +- `https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_or` +- `https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_unchecked` + +For each of the same releases, the Conditional Compilation chapter says a +configuration predicate is “true if the option is set,” `all` requires “all of +the given predicates” to be true, and `not` is true when its “predicate is +false.” The `cfg` attribute “conditionally includes” its attached item. Those +checked excerpts supply exactly the source-selection implications used here. + +The standard-library macro page states: + +- “Causes compilation to fail with the given error message when encountered.” + +Exact pages, continuing canonical allowlist order: + +- `https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#conditional-compilation` +- `https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#the-cfg-attribute` +- `https://doc.rust-lang.org/1.84.0/std/macro.compile_error.html` +- `https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#conditional-compilation` +- `https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#the-cfg-attribute` +- `https://doc.rust-lang.org/1.85.0/std/macro.compile_error.html` +- `https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#conditional-compilation` +- `https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#the-cfg-attribute` +- `https://doc.rust-lang.org/1.86.0/std/macro.compile_error.html` + +`BUILD-MAP-POLICY` is the only admitted non-Rust premise. No policy sentence, +Cargo minimum, evaluator oracle, test result, CI result, or inferred tool +behavior is a Rust semantic axiom. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/Q.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/Q.md new file mode 100644 index 0000000000..d7fb25f4a6 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/Q.md @@ -0,0 +1,76 @@ +# V4 Focused Oracle — Q: Quantifier and Verdict Control + +> Evaluator-only material. Never expose this file to a report agent. + +Let `C_Q` be Rust 1.80.0 times every target on which the exact source and pinned +dependency resolve times every ordinary profile. + +- `Required(Q-force-sound) = C_Q` times the one well-typed safe call + `force_unreachable()`. +- `Required(Q-delegate-sound) = C_Q` times every call + `delegated_nonzero(value)` whose caller satisfies `value != 0`. + +## Atoms + +- **Q1 — Domain and claim separation:** State both valid-use domains and audit + the two APIs independently. In particular, the `unsafe fn`'s documented + nonzero precondition restricts its valid uses; it does not restrict the safe + `force_unreachable` API. + - `scope_basis`: `REQUEST.md` explicitly asks for two separate claims and + supplies their common compilation scope; each API's signature/contract + determines its valid-use domain. + - `dependencies`: none. +- **Q2 — Complete UB certificate:** For `force_unreachable`, give all four + links explicitly or by an equivalently exact chain: (1) the safe call is + well-typed and valid; (2) its only body path reaches the executed + `unreachable_unchecked` call; (3) that call requires its site not to be + reached and that proposition is false; and (4) applicable Rust 1.80.0 std + documentation says reaching it is UB. + - `scope_basis`: necessary certificate for the requested soundness verdict + on the safe API. + - `dependencies`: Q1. +- **Q3 — Existential closure:** Report `Q-force-sound` **UNSOUND**. The complete + in-scope witness refutes universal soundness; do not weaken it to `UNPROVED` + merely because no universal positive proof was built. + - `scope_basis`: `REQUEST.md` asks for the strongest justified verdict. + - `dependencies`: Q2. +- **Q4 — Exact missing premise:** For `delegated_nonzero`, recognize that a + valid caller supplies the dependency API's nonzero caller precondition, but + the selected unsafe dependency body is absent and no accepted `UNSAFE-DEP` + entry establishes that it honors its contract. The pin and caller-facing + docs establish the logical coordinate and propagated obligation, not + authenticated implementation identity or correctness. The smallest missing + proposition is that every valid nonzero call to the selected + `peer-unsafe-dep` 3.4.5 implementation is UB-free over this domain. + - `scope_basis`: the supplied evidence packet and TCB policy are expressly + part of the requested dependency claim review. + - `dependencies`: Q1. +- **Q5 — No fabricated witness or silent trust:** Report + `Q-delegate-sound` **UNPROVED**, not `PROVED` or `UNSOUND`. Its required cases + lack the dependency-soundness premise, while the unavailable body supplies no + proved UB execution. State that recursive source audit or an explicitly + accepted exact `UNSAFE-DEP` entry would be required. + - `scope_basis`: `REQUEST.md` asks for the strongest packet-supported verdict + and smallest missing proposition. + - `dependencies`: Q4. + +## Hard errors + +- **QH1:** Call `force_unreachable` sound or only `UNPROVED` after establishing + all four Q2 links. +- **QH2:** Issue `UNSOUND` for it without a valid call, reachability, false exact + safety proposition, and authoritative UB consequence. +- **QH3:** Silently trust the unavailable unsafe dependency, treat the pin as + implementation proof, or report `delegated_nonzero` `PROVED`. +- **QH4:** Fabricate dependency body behavior or a UB execution and report + `delegated_nonzero` `UNSOUND`. +- **QH5:** Treat violation of the dependency's documented caller precondition + as itself an executed UB event. + +## Checked authority + +- `https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html#safety` + — reaching `unreachable_unchecked` is UB; applicable to Rust 1.80.0 without a + target/profile qualification relevant here. + +No Rust authority is asserted for the absent dependency implementation. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/R.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/R.md new file mode 100644 index 0000000000..9b62ddf188 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/oracle/R.md @@ -0,0 +1,98 @@ +# V4 Focused Oracle — R: Abstraction-Design Control + +> Evaluator-only material. Never expose this file to a report agent. + +## Exact result + +Within the scope in `REQUEST.md`, valid safe use includes every downstream safe +implementation of `Slot`, not only `Tail`. An implementation returning 2 and a +safe call `increment::(&mut [0, 0])` reaches +`get_unchecked_mut(2)` on a length-2 slice. Its in-bounds precondition is false, +and Rust 1.82.0 documents an out-of-bounds call as UB. The current safe API is +therefore `UNSOUND`. + +The required behavior needs neither generic index metadata nor an unsafe +abstraction. The preferred design is a nongeneric safe function that updates +`pair[1]` with wrapping addition (or an exactly behavior-equivalent checked +specialization). It eliminates caller implementation/index capability and the +unsafe block. Making `Slot` unsafe, adding safe-trait prose, or preserving the +generic abstraction by default is less parsimonious because downstream generic +use is expressly not required. The proposal does not alter the current verdict +and requires implementation plus fresh audit. + +## Atoms + +- **R1:** Quantify over arbitrary safe `Slot` implementations and explicitly + construct or describe one returning 2 together with its valid safe call. + - `scope_basis`: public safe trait implementation and safe generic calls are + valid-use surfaces of the exact current source. + - `dependencies`: exact Rust 1.82.0 public-visibility, trait-implementation, + ordinary-versus-unsafe-trait, and safe-versus-unsafe-function rules. +- **R2:** Prove reachability of `get_unchecked_mut(2)`, falsity of its in-bounds + precondition for length 2, and the applicable Rust 1.82.0 UB consequence. + - `scope_basis`: necessary certificate for the current-artifact soundness + claim requested in `REQUEST.md`. + - `dependencies`: R1. +- **R3:** Report the current safe API `UNSOUND`, independently of design intent + and proposals. + - `scope_basis`: `REQUEST.md` expressly requires the literal current-artifact + verdict before redesign. + - `dependencies`: R2. +- **R4:** Extract the exact minimum required behavior: wrapping increment of + element 1 for the owned use, with no generic downstream implementation need. + - `scope_basis`: this requirement is stated verbatim in `REQUEST.md`. + - `dependencies`: none. +- **R5:** Propose a nongeneric safe specialization implementing R4 with checked + indexing or an exactly equivalent safe operation, and no caller-controlled + implementation or index capability. + - `scope_basis`: `REQUEST.md` asks for the most parsimonious provable + abstraction satisfying R4. + - `dependencies`: R4. +- **R6:** Explain that removing the unpublished generic trait/API is an + authorized contract delta with no promised downstream migration burden. + - `scope_basis`: `REQUEST.md` expressly permits this contract and + representation change. + - `dependencies`: R4, R5. +- **R7:** Keep the redesign conditional and require implementation plus fresh + audit; do not use it to narrow or alter R3. + - `scope_basis`: `REQUEST.md` requires the current verdict and post-change + audit independently. + - `dependencies`: R3, R5. + +Merely making the trait unsafe does not pass R5 because it preserves an +unneeded caller implementation capability. + +## Hard errors + +- **RH1:** Trust `Tail` as the only safe implementation. +- **RH2:** Treat safe-trait prose as a caller obligation. +- **RH3:** Call the current API sound or merely `UNPROVED` after closing R2. +- **RH4:** Certify an unimplemented redesign. +- **RH5:** Let design intent narrow the current contract. + +## Checked authority + +- Rust 1.82.0 Reference, public visibility: + `https://doc.rust-lang.org/1.82.0/reference/visibility-and-privacy.html#visibility-and-privacy` + — public items are accessible from outside their module, subject to the + containing path; every item here is at the public crate root. +- Rust 1.82.0 Reference, trait implementations and coherence: + `https://doc.rust-lang.org/1.82.0/reference/items/implementations.html#trait-implementations` + — a trait implementation may be defined when the implementing type is local, + so a downstream crate may define local `Bad` and implement `Slot` for it. +- Rust 1.82.0 Reference, unsafe traits: + `https://doc.rust-lang.org/1.82.0/reference/items/traits.html#unsafe-traits` + — unsafe traits are explicitly declared with `unsafe`; the supplied `Slot` + declaration is an ordinary safe trait and its implementation requires no + unsafe act. +- Rust 1.82.0 Reference, unsafe functions: + `https://doc.rust-lang.org/1.82.0/reference/unsafe-keyword.html#unsafe-functions-unsafe-fn` + — the caller-side unsafe obligation attaches to an `unsafe fn`; the supplied + public `increment` is an ordinary safe function. +- Rust 1.82.0 slice `get_unchecked_mut`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked_mut` + — the index must be in bounds; an out-of-bounds call is UB even if its result + is unused. +- Rust 1.82.0 `u32::wrapping_add`: + `https://doc.rust-lang.org/1.82.0/std/primitive.u32.html#method.wrapping_add` + — modular wrapping addition supplies the requested update semantics. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/plan.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/plan.md new file mode 100644 index 0000000000..1b5db4deb7 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/plan.md @@ -0,0 +1,257 @@ +# Unsafe Rust V4 Focused Confirmation Plan + +> Evaluator-only material. Never expose this file to evaluated agents. +> +> **Preregistration status:** FROZEN. This status is effective only when +> `LOCK.json` authenticates the complete freeze tree and records two independent +> oracle-review signoffs. No evaluated report may be collected before +> `protocol.py verify-static --locked` passes. + +## Purpose + +This evaluation tests whether V4 reliably fixes the concrete reasoning failures +observed in V3 while preserving V3's already-demonstrated quantifier/verdict and +abstraction-design capabilities. It is an absolute capability confirmation for +V4 with V3 as a diagnostic comparator, not a causal estimate of isolated text. + +The run is deliberately focused. Passing permits another broad release gate; +it does not establish that every unsafe-Rust task or every model invocation will +succeed. + +## Conditions and design + +The intended package conditions are: + +| Role | Package tree SHA-256 | `SKILL.md` SHA-256 | +|---|---|---| +| V4 candidate | `6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb` | `ad48b3811cf2054be76e4b461a36f63e636afb246c5dd7a75e85756a53b22d83` | +| V3 comparator | `fc486dedde1f82ba232b4492808af85a12b27fa2aa27b1a35a3847b2b89f72e0` | `0e23f7747cc63014bade7543efaf745e7e9a7e5d6dee2a48c602ef7a3eba091e` | + +Use five modes, two conditions, and five fresh replicates per cell: + +> 5 modes × 2 conditions × 5 replicates = 50 reports. + +Each agent sees one neutral package, one neutral target, the mode's URL-only +allowlist, and one empty output directory. Condition and mode labels are sealed +before collection. Five replicates are an engineering reliability threshold, +not a population estimate; averages cannot rescue a failed exact gate. + +## Modes + +### P — exact predicates, conservative containment, and exclusions + +Tests recovery of two nonlinear policy predicates, explicit separating +witnesses and the exact set relationship, conservative full-case containment +before any configuration projection, distinction between project promise and +audit superset, effective exclusion, and closure of the local unsafe operation +and documented postcondition over every full case. + +### B — ordered and fallible build relation + +Tests whether the audit models the claim-relevant build script as an ordered, +fallible state transition rather than an endpoint environment-to-cfg map. It +covers partial output prefixes, process outcome, Cargo's interpretation, +freshness/rerun behavior, selected generated source, supported exclusions, the +unsafe operation, postcondition, and exact configuration-by-input sound region. + +### L — local proof reconstruction and authority reconciliation + +Tests separation of implementation correctness from proof-artifact adequacy, +exposure of a proof the reviewer had to reconstruct, complete local arithmetic +and control-flow reasoning, useful replacement `SAFETY` wording, and a bijection +between materially consumed Rust/std premises and checked version-matched +citations. + +### Q — quantifier and verdict control + +Unchanged semantic control from V3. It pairs a complete safe-use UB witness, +which must close as `UNSOUND`, with an unavailable third-party unsafe +implementation, which must remain `UNPROVED` without fabricated behavior or +silent trust. + +### R — abstraction-design control + +Unchanged semantic control from V3. It tests adversarial caller-provided safe +trait behavior, literal current-artifact review, extraction of the minimum +required capability, a more parsimonious locally enforced safe design, contract +delta analysis, and non-certification of an unimplemented proposal. + +The oracle freezes the exact propositions. Every atom has a stated +`scope_basis` and dependency list so scoring requirements cannot be introduced +merely because an evaluator prefers a particular vocabulary or presentation. + +## Hypotheses + +- **H-P:** V4 passes every P atom in all five replicates without a P/global, + TCB/authority, scope, budget, or completion defect. +- **H-B:** V4 passes every B atom in all five replicates under the same defect + gate, including all ordered/fallible build-stage propositions. +- **H-L:** V4 passes every L atom in all five replicates under the same defect + gate, including exposed reconstruction and premise/citation reconciliation. +- **H-controls:** V4 passes every Q and R atom in all five replicates without a + defect, preserving established V3 capability. + +V4 5/5 with lower V3 performance is evidence consistent with targeted lift. +Matched 5/5 is ceiling replication. Since the coherent packages differ in more +than one instruction, neither pattern is isolated causal proof. + +## Primary gate + +V4 passes only if all of these hold after dual scoring, consistency review, and +adjudication: + +1. Every frozen V4 atom passes 5/5. +2. V4 has zero mode-specific or global hard errors. +3. V4 has zero proposal laundering and zero TCB/authority defects. +4. V4 has zero terminal semantic noncompletions. +5. Every V4 report respects the frozen visible-source scope, recorded + operational scope, and word cap. +6. Q and R independently satisfy items 1–5; strength on new modes cannot mask a + control regression. +7. Every scorer disagreement, every positive hard/global/TCB/scope/proposal + flag (even when both scorers agree), every consistency-review challenge, and + every novel finding receives a preserved adjudicated disposition. +8. V4 has zero independently confirmed novel material defects. +9. All freeze, packet, attempt, preservation, and byte-tree identity checks + pass without manual permission repair or post-freeze mutation. + +Failure of any item fails the run. Do not weaken an atom or hard-error rule +after seeing reports, average a failure away, or use V3 weakness as an excuse. + +## Diagnostic comparison + +After unblinding: + +- report V4 and V3 pass counts separately for every atom; +- identify any V4 atom below its matched V3 count; +- classify V4 5/5 plus lower V3 as targeted-lift evidence; +- classify matched 5/5 as ceiling replication; +- classify any other improvement as suggestive but insufficient for the + absolute gate; and +- do not pool heterogeneous atoms or modes into one headline score. + +## Global scoring constraints + +Mode rubrics may specialize but not weaken the common rules. In particular, a +report must not: + +- certify a universal positive claim after contracting or failing to justify + its required full-case domain; +- claim closure without a reviewable `Required(case) ⊆ Covered(case)` + derivation; +- project to configuration while silently dropping input, state, time, + artifact, process-outcome, or other material fibers; +- invent policy precedence or call a conservative audit superset the project's + promise; +- issue `UNSOUND` without valid use, reachability, falsity of the exact safety + proposition, and an applicable authoritative UB consequence; +- issue only `UNPROVED` after explicitly closing all four UB-certificate links; +- silently trust an unsafe dependency or caller-controlled safe behavior; +- certify an unimplemented design or use it to narrow the current artifact; +- necessarily rely on unchecked, invalid, version-mismatched, or inapplicable + authority; or +- inspect prohibited oracle, sibling, map, prior-report, or evaluator material. + +A missed atom is not automatically a hard error. Apply hard errors only under +their exact frozen definition. Extra correct regional detail is harmless, but a +maximal positive remainder is not required unless the target request expressly +asks for one. + +## Collection protocol + +Before collection, freeze and hash: + +- both complete package trees; +- every target tree; +- evaluated-agent, scorer, consistency-reviewer, and adjudicator prompts; +- oracle, common rules, and per-mode rubrics; +- URL-only per-mode allowlists plus retrieval identities; +- schedule, blind maps, presentation orders, and commitments; +- schemas, validation/aggregation code, word caps, and policies. + +Every report and evaluator agent uses a fresh context, `gpt-5.6-sol`, reasoning +effort `ultra`, `fork_turns="none"`, and no helper agents. The orchestration API +does not expose an exact hosted build or sampling seed; record this limitation. +Collection follows five balanced waves with at most three report agents active. +The next wave cannot be prepared until the preceding wave is terminal and the +authority verifier reproduces the frozen records. + +Each report agent must inspect only its neutral packet and the exact permitted +official pages; it must not build, test, execute, expand, edit, or inspect +evaluator material. It writes one canonical `report.md`. That file is the sole +evaluated response channel. The orchestration transcript may retain chat-return +prose, but the run does not copy or score it; attempt metadata records the agent +identity and API completion state. Caps are identical across conditions: P +3,000 words, B 3,200, L 2,200, Q 1,800, and R 1,800. + +Only genuine infrastructure failure permits a fresh retry. Refusal, budget +exhaustion, scope deviation, or semantic noncompletion is terminal. Preserve +every attempt and reason. A report without usable output gets a canonical +evaluator placeholder and fails missing propositions plus the completion gate; +a non-rerunnable invalid scorer, consistency reviewer, or adjudicator makes the +run `INVALID`. + +## Blind scoring, consistency review, and adjudication + +After all reports are preserved and before unblinding: + +1. Materialize the pre-frozen anonymous label map within each mode. +2. Give two fresh scorers the target, common rules, exact mode rubric, and ten + anonymous reports—never packages, maps, sibling modes, or prior scores. +3. Have a third fresh, condition-blind reviewer compare all ten reports and both + raw score sets for every atom, hard-error, and global-defect family, attest + each complete ten-report comparison, and challenge inconsistent decisions. +4. Construct review cells from all scorer disagreements, all agreed-positive + defect flags, all consistency challenges, and all novel findings. +5. Give a fresh adjudicator only those cells and their source material; do not + conceal an agreed positive flag or force preservation of an agreed error. +6. Preserve raw scores, consistency reviews, adjudications, events, packets, + and all integrity bindings before unblinding. + +The runner independently records word count, completion, and operational scope. +Scorers judge only what report text exposes. Aggregation combines those sources +without asking a scorer to infer unavailable telemetry. + +## Integrity model + +Copies and packet bindings use byte-tree identity: relative file path, file +kind, and bytes. Permission metadata is validated separately where relevant and +must not alter content identity. Every canonical artifact is captured once, +hashed, and preserved; later packets bind to the preserved digest. Append-only +events authenticate state transitions. A run-wide operation lock serializes +checks and writes. + +The shared host does not provide cryptographically enforced filesystem or +network isolation. Neutral paths, procedural restrictions, exhaustive packet +inventories, before/after identity checks, and failure snapshots reduce but do +not eliminate that limitation. Disclose it in results. + +## Oracle review and freeze + +Before `LOCK.json`, two independent reviewers who did not author the fixtures +must inspect every target, request, atom, hard error, and cited proposition. +Each must confirm: + +- the expected result follows from the exact source and request; +- every atom is necessary, proposition-focused, and has a valid `scope_basis`; +- dependencies are explicit and do not cause double credit or hidden demands; +- every Rust/std premise is supported by applicable versioned authority; +- every accepted TCB proposition has exact identity, scope, and consumer; +- no target leaks condition, oracle, verdict, historical provenance, or rubric; +- positive obligations are actually provable over the full declared case set; +- Q and R remain semantically equivalent controls across V3 and V4; and +- no maximal-region or stylistic requirement has entered scoring without an + explicit request basis. + +Record both signoffs, each reviewer's independent-non-author attestation, and +all unresolved ambiguities in the lock. A review timestamp may not follow the +lock timestamp. Any substantive change after review invalidates the signoffs +and requires fresh review. + +## Forward-test discipline + +Do not edit the skill, packages, fixtures, oracle, rubrics, protocol, policies, +or analysis rules after the lock or after seeing any evaluated output. If the +freeze is defective, mark the run invalid or exploratory, correct the design in +a new versioned run, and start fresh. The aggregate must preserve failures and +limitations rather than retrofitting the test to the observed reports. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/isolation.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/isolation.md new file mode 100644 index 0000000000..5b9456d821 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/isolation.md @@ -0,0 +1,48 @@ +# Frozen Isolation and Packet Policy + +Each report cell receives a unique neutral runtime root with entries named only +`package`, `target`, `allowlist.txt`, and `output`. Inputs are copied +byte-for-byte, made read-only where the host permits, and verified against +frozen identities before launch and after the attempt. `output` begins empty. + +The shared collaboration host does not provide a hardened per-agent mount or +network allowlist. Isolation therefore remains procedural. Runtime inputs must +not contain a repository, `.git`, evaluator files, mode/condition names, or +cross-cell paths. This limitation is disclosed in every result and prevents a +claim of cryptographically enforced blinding. + +Before scorer launch, preserve and hash each canonical `report.md` and copy it +into a fresh mode packet under its frozen anonymous label. Chat-return text is +operational metadata and is not an alternative report channel. Prefer neutral +paths from collection so no report redaction is needed. Any required +normalization must use a frozen transformer and preserve the exact original, +transformed hash, and diff. + +Scorer packets contain only the target, allowlist, common rules, one mode +rubric, ten anonymous reports, score schema, and packet-local hashes. +Consistency packets add only the two preserved raw scores and the consistency +schema for that same anonymous mode. All packet filesystem timestamps are +normalized to the Unix epoch. Packets contain no package, condition, run ID, +collection order, report-agent identity, or sibling mode. Adjudicator packets +contain only materialized review cells and their source evidence: scorer +disagreements, agreed-positive defect flags, consistency challenges, and novel +findings. They omit unrelated negative/agreed decisions. + +The append-only event ledger externally pins the canonical collection index +and every complete scorer, consistency, and adjudicator packet byte tree. +Packet verification checks that external digest as well as the packet-local +manifest before every use. One run-wide operation lock serializes all state +checks, artifact writes, and event transitions. A canonical report, score, +consistency review, or adjudication and its attempt record derive from the same +single capture of the agent's output bytes; the protocol does not reread a live +output path to create either copy. + +Failure preservation remains possible when an evaluated agent changes a +runtime input despite the procedural restrictions. The protocol binds the +expected neutral path, records expected and safely observed input identities +without following symlinks, and snapshots the entire neutral runtime on an +input or inventory verification failure. The snapshot preserves every regular +byte and records every directory, symlink, and special entry without following +symlinks. It also snapshots the central setup or source packet whose identity +was checked. Report input drift becomes a terminal scope failure; evaluator +packet drift makes the run `INVALID`. It never authorizes a retry. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/reruns.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/reruns.md new file mode 100644 index 0000000000..b6afb685b5 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/reruns.md @@ -0,0 +1,45 @@ +# Frozen Attempts, Reminders, and Rerun Policy + +Every attempt is immutable and receives its own directory. Preserve every +regular filesystem output byte, agent identity, timestamps, hashes, known +tool/path deviations, API completion/error state, and disposition before +considering another attempt. The canonical file is the sole evaluated channel; +chat-return prose need not be duplicated byte-for-byte. + +Only an externally established infrastructure failure may authorize a fresh +agent attempt. Examples are a service error before semantic work, a tool crash, +or a write failure caused by unavailable infrastructure. Refusal, word-budget +exhaustion, timeout after substantive work, invalid reasoning, missing atoms, +or other semantic noncompletion is a failed replicate and may not be rerun. + +An API failure that produces no agent identity is not an evaluated attempt; +record it separately as `API_NO_AGENT_START`, then launch the same next attempt +number. This applies to report, scorer, consistency-reviewer, and adjudicator +launches. For any started attempt, use only these infrastructure disposition +codes: +`SERVICE_ERROR_BEFORE_OUTPUT`, `ORCHESTRATOR_TOOL_FAILURE`, and +`FILESYSTEM_FAILURE`. Record the observed evidence and preserve every partial +artifact. A condition not fitting one of those codes is not rerunnable. + +A started scorer, consistency-reviewer, or adjudicator infrastructure failure +likewise preserves its own attempt directory and authorizes exactly the next +numbered fresh attempt. A schema-invalid, semantically incomplete, or otherwise +non-infrastructure evaluator output is non-rerunnable and makes the evaluation +`INVALID`. + +A terminal report-agent noncompletion is represented by its usable `report.md` +when one exists or by an evaluator-marked placeholder otherwise, is blind-scored +under the ordinary rubric, and is never rerun. Never fabricate a replacement +report, score, or adjudication. An empty or whitespace-only `report.md` is +semantic noncompletion, never a complete report. + +Preserved evaluator-attempt and invalid-output directories are inventoried in +both directions against unique ledger events. Any orphan directory, missing +directory, duplicate event, invalid hierarchy, `INVALID.json` marker, or +terminal-invalid ledger event fails closed. Once either terminal-invalid signal +exists, no later evaluation operation is permitted; a complete invalid state +must bind the marker, attestation, return event, and invalidation event exactly. + +One exact neutral reminder from `prompts/report.md` is permitted at 180 seconds. +No second reminder or substantive steering is permitted. Scorers, consistency +reviewers, and adjudicators receive no reminders. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/tools.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/tools.md new file mode 100644 index 0000000000..ea7713fc66 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/tools.md @@ -0,0 +1,17 @@ +# Frozen Tool and Source Policy + +Report agents may read only their per-cell package, target, URL-only allowlist, +allowed exact pages, and initially empty output directory. They may write only +`output/report.md` with `apply_patch`. Direct URL opens are permitted; web +search, link traversal, and every non-allowlisted page are prohibited. + +Targets must not be modified, built, tested, executed, or macro-expanded by +execution. Report agents may not spawn helper agents. They must not inspect +sibling/enclosing paths, repository or Git data, evaluator material, schedules, +maps, other cells, or prior reports. + +Scorers, consistency reviewers, and adjudicators have analogous packet-only +read scope and one-file output scope. Tool and path restrictions are procedural +on the shared host; they are not a hardened sandbox. Every known deviation is +preserved in the append-only event ledger and fails the scope gate when +material. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/word-count.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/word-count.md new file mode 100644 index 0000000000..4ab7216e73 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/policies/word-count.md @@ -0,0 +1,19 @@ +# Frozen Word-Count Policy + +Decode `report.md` as UTF-8, split it on every Unicode whitespace run using +Python 3 `str.split()`, and count the resulting nonempty fields. Invalid UTF-8 +is an invalid attempt. + +The caps are: + +| Mode | Maximum words | +|---|---:| +| P | 3,000 | +| B | 3,200 | +| L | 2,200 | +| Q | 1,800 | +| R | 1,800 | + +The same cap applies to both conditions. A report above its cap remains +preserved but fails the source-scope/budget gate; it is not an infrastructure +failure and may not be rerun. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/adjudicator.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/adjudicator.md new file mode 100644 index 0000000000..812d01be1f --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/adjudicator.md @@ -0,0 +1,28 @@ +# Frozen Blind-Adjudicator Prompt + +```text +Act as a fresh blind adjudicator. Read every file under [PACKET], including the +target, URL allowlist, SCORER.md, RUBRIC.md, adjudication schema, listed +anonymous reports, CONSISTENCY.json, ADJUDICATION_CELLS.json, and every scorer +or consistency-review rationale attached to each listed cell. Follow the +frozen instructions exactly. + +Resolve exactly every listed cell. The list includes scorer disagreements, +agreed positive hard-error/global-defect candidates requiring independent +confirmation, consistency-review challenges, and novel candidates. Do not +preserve an agreed decision merely because both scorers made it; adjudicate +every listed cell on its merits. Preserve agreed decisions that are not listed. +Decide from the report's actual evidence: external authority may verify a +premise the report invokes, but may not add a material premise or derivation the +report omitted. Do not decide by majority, style, report similarity, or +speculation about a generating condition. + +Inspect only [PACKET] and exact URLs in its allowlist. Do not inspect siblings, +the enclosing repository, packages, condition maps, other scores, or other +adjudications. Do not modify the packet and do not spawn helper agents. + +Write exactly one UTF-8 JSON file, [OUTPUT]/adjudication.json, conforming to the +supplied schema. Create no other output file. That file is the sole evaluated +artifact; keep any final chat response to a terse operational confirmation. +Give compact decisive evidence and record genuine rubric ambiguity. +``` diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/consistency.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/consistency.md new file mode 100644 index 0000000000..d8e6a89f9a --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/consistency.md @@ -0,0 +1,35 @@ +# Frozen Condition-Blind Consistency-Reviewer Prompt + +```text +Act as a fresh condition-blind consistency reviewer. Read every file under +[PACKET], including the target, URL allowlist, SCORER.md, RUBRIC.md, +consistency schema, anonymous reports A through J, and both raw scores. Follow +the frozen scoring rules exactly. + +For each rubric atom family, compare that atom across all ten reports. Look for +unequal treatment of materially equivalent proof shapes, silent premise +promotion, clause splitting or collapse, and evidence standards that change +between reports. Also compare each hard-error family and each global-defect +family across all ten reports. Attest separately to every complete ten-report +atom-family and defect-family comparison. + +Do not rescore every cell or manufacture disagreement. Add a challenge only +when a specific atom, hard-error, or global-defect decision should be changed; +identify the anonymous label and exact field, recommend the corrected decision, +and give compact decisive evidence. A challenge must disagree with at least one +raw scorer decision. Every challenge will be independently adjudicated. + +Remain condition-blind. Do not identify, cluster, or speculate about report +conditions, packages, or skill versions. Inspect only [PACKET] and exact URLs +in its allowlist. Do not search, follow links, inspect sibling or enclosing +directories, condition maps, packages, prior evaluations, or other reviewer +outputs. Do not modify the packet and do not spawn helper agents. + +Write exactly one UTF-8 JSON file, [OUTPUT]/consistency.json, conforming to the +supplied schema. Create no other output file. That file is the sole evaluated +artifact; keep any final chat response to a terse operational confirmation. +Record genuine rubric ambiguity without resolving it by speculation. +``` + +No reminder is permitted. An invalid or incomplete review is preserved and +does not silently become part of adjudication. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/report.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/report.md new file mode 100644 index 0000000000..2669f5bbba --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/report.md @@ -0,0 +1,36 @@ +# Frozen Evaluated-Agent Prompt + +Each report agent receives this text byte-for-byte except for resolving the +five bracketed placeholders. `[WORD_LIMIT]` is mode-specific but identical +between conditions within a mode. + +```text +Act as a fresh source-review agent. Read the complete unsafe Rust skill package +rooted at [PACKAGE]/SKILL.md and every reference it directs you to for this +task, then follow it. Read REQUEST.md and every file in [TARGET], and complete +exactly the requested review or design work. + +Your only permitted local inputs are [PACKAGE], [TARGET], [URL_ALLOWLIST], and +the empty [OUTPUT] directory. The allowlist contains the only external pages +you may inspect. Open those exact URLs directly if needed; do not search the +web or follow links to another page. Do not inspect a sibling or enclosing +directory, repository or version-control material, evaluator files, another +package or target, prior reports, schedules, or condition information. Treat +paths and opaque labels as non-semantic. + +Do not modify, build, test, execute, or macro-expand the target. Do not spawn +helper agents. Write exactly one UTF-8 file, [OUTPUT]/report.md, using +apply_patch, and create no other output file. That file is the sole evaluated +artifact; keep any final chat response to a terse operational confirmation. +Keep the report at or below [WORD_LIMIT] words, counting the nonempty fields +produced by splitting Unicode text on whitespace. This is a focused source +review; provide the complete proof material compactly. +``` + +No substantive steering is permitted. If the agent is still running 180 +seconds after launch, exactly one reminder may be sent: + +```text +Complete now within the frozen word limit using only material already +inspected; do not widen scope. +``` diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/scorer.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/scorer.md new file mode 100644 index 0000000000..bc42474f76 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/prompts/scorer.md @@ -0,0 +1,28 @@ +# Frozen Blind-Scorer Prompt + +Each scorer receives this text byte-for-byte except for resolved `[PACKET]` and +`[OUTPUT]` paths and its opaque `[SCORER_ID]`. + +```text +Act as a fresh blind evaluator. Read every file under [PACKET], including the +target, URL allowlist, SCORER.md, RUBRIC.md, score schema, and anonymous reports +A through J. Follow the frozen scoring instructions exactly and score every +report independently. Inspect reports in the `presentation_order` recorded in +PACKET.json; do not reorder them by similarity or suspected condition. + +Inspect only [PACKET] and the exact URLs in its allowlist. Open an allowed URL +directly if needed; do not search, follow links, or inspect any sibling or +enclosing directory, repository material, package, condition map, prior score, +or another scorer's output. Do not identify, cluster, or speculate about report +conditions. Do not modify the packet and do not spawn helper agents. + +Write exactly one UTF-8 JSON file, [OUTPUT]/score.json, conforming to the +supplied schema and using scorer_id [SCORER_ID]. Create no other output file. +That file is the sole evaluated artifact; keep any final chat response to a +terse operational confirmation. Score every atom and every independent defect +flag, cite compact report evidence for each decision, and record genuine rubric +ambiguity. Keep all evidence concise. +``` + +No reminder is permitted. An invalid or incomplete scorer output is preserved +and does not silently replace an independent score. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/randomization/commitments.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/randomization/commitments.json new file mode 100644 index 0000000000..acef376b6c --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/randomization/commitments.json @@ -0,0 +1,12 @@ +{ + "algorithm": "sha256(tag_utf8 || NUL || seed_bytes)", + "commitments": { + "blind": "4941a662e4a63eba04ab4000a09bee780bbe88beea7cec70af4c3582c4904b61", + "condition": "3537bcedb9d55efe1a9b043ab7009a8eead43391789c94dd40c1ca6f2f2d1f06", + "consistency": "80cced2b0d6eacda5958cbdc2d02e55369e3282a794667869eb97020bac6f9af", + "presentation": "1fa5c1e21ba6c360932c53adbf1e977fd91f7d6113ffa58d88b45d4a7bdd88f2", + "schedule": "15983d0d344f7d801c10308dff411792ca1c3c4d2a0c8ff2708f51e549fb4b32", + "scorer": "2bc3121feb893cac316576f4f98546d334d4ae139d012f4eb71234ac563fc378" + }, + "schema_version": 1 +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/randomization/spec.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/randomization/spec.md new file mode 100644 index 0000000000..a0e97e2121 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/randomization/spec.md @@ -0,0 +1,43 @@ +# Frozen Randomization Specification + +`prepare.py` is the executable specification. It uses six independently +generated 256-bit seeds, stored under evaluator-only `sealed/seeds.json`, and +domain-separated SHA-256 hash sorting. No language PRNG or iteration-order +behavior determines an order. + +For tag `T`, seed `S`, and canonical UTF-8 value `V`, the sort key is: + +```text +SHA256(T || NUL || bytes_from_hex(S) || NUL || V) +``` + +The canonical tuple breaks the cryptographically negligible event of a key +collision. + +The procedure is: + +1. Hash-sort real condition roles and assign opaque labels `c0` and `c1`. +2. Hash-sort mode names and assign opaque target labels `m0` through `m4`. +3. Treat each replicate as a balanced wave containing all five modes and both + conditions. Hash-sort the five waves, then the ten cells within each + wave. Assign sequential operational run IDs only after sorting. +4. Derive each neutral 128-bit runtime cell ID from a separately tagged hash of + its canonical `(mode, role, replicate)` tuple. Assert all 50 are unique. +5. Independently hash-sort each mode's ten run IDs and assign blind labels A–J. +6. Independently hash-sort A–J for each scorer's presentation order. +7. Independently hash-sort the ten scorer claims for the scoring launch + order. +8. Independently hash-sort the five mode claims for the consistency-review + launch order. + +The sealed maps are frozen before collection but never enter a report-agent or +blind-scorer packet. Because the shared repository and host are not hardened +against deliberate out-of-scope reads, secrecy is procedural. Commitments are +recorded in `commitments.json`; the seed and generated-map bytes are covered by +the freeze lock. + +Collection processes one complete balanced wave before the next. Within a +wave, starts follow `launch-schedule.tsv`; up to three report agents may be +active concurrently. A later service limitation may reduce concurrency but may +not reorder starts or cross the wave barrier. Every actual start/completion is +recorded in the append-only event ledger. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/B.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/B.md new file mode 100644 index 0000000000..c5371528bf --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/B.md @@ -0,0 +1,483 @@ +# Mode B Frozen Blind-Scoring Rubric + +> **Evaluator-only material. Never expose this file to a report agent.** + +> **Evaluator-only material.** Never expose this file, its atoms, formulas, +> expected verdicts, hard errors, or authority inventory to a report agent. + +## Scoring Rules + +Score the proposition of each atom, not its notation or presentation. Accept an +explicit equivalent derivation. Do not infer an omitted build stage, failure +edge, source-selection step, or semantic premise from an endpoint summary. +Failure of an atom is not itself a hard error unless the report also makes the +false affirmative claim identified below. + +This fixture deliberately requests a maximal positive region. That request, +not a general audit convention, makes exactness and maximality part of B13. +There is no scoring requirement for incidental diagnostics, exact panic text, +partially written bytes within one failed `println!`, or build-script stdout +which does not affect a current library selection. + +## Intended Domain and Build Relation + +Let: + +```text +R = { Rust and Cargo 1.85.1 } +T = { X = x86_64-unknown-linux-gnu, + A = aarch64-unknown-linux-gnu, + W = wasm32-unknown-unknown } +F = { burst-off, burst-on } +L = { system, arena } +P = every Cargo profile supported by the fixture +D = { debug-assertions-off, debug-assertions-on } +I = { i | i is any u8 } + +Config(t,f,l,p,d) + iff t in T and f in F and l in L and p in P and d in D + and not (t = W and l = arena). + +Required = R cross Config cross I. +Q(t,f,l) iff t = A and f = burst-on and l = arena. +``` + +Both allocator values in `Config` are reachable through the supported Cargo +interface: an omitted selector or `system` selects `system`, and `arena` +selects `arena`. Raw rejected selector values and unsuccessful build attempts +are build-relation cases that must be accounted for, but they do not add a +library configuration to `Required`. + +For compactness, use these local operations: + +```text +RERUN = attempt println!("cargo::rerun-if-env-changed=FIXTURE_ALLOCATOR") +READ = evaluate env::var("FIXTURE_ALLOCATOR") +SYS = attempt println!("cargo::rustc-cfg=fixture_allocator=\"system\"") +ARENA = attempt println!("cargo::rustc-cfg=fixture_allocator=\"arena\"") +PANIC = the selected explicit panic, or the panic caused by a failed println! +RETURN = successful return from main +``` + +The complete claim-relevant local relation is: + +```text +missing: RERUN -> READ(NotPresent) -> SYS -> RETURN +Unicode system: RERUN -> READ(Ok(system)) -> SYS -> RETURN +Unicode arena: RERUN -> READ(Ok(arena)) -> ARENA -> RETURN +Unicode arena-stop: + RERUN -> READ(Ok(arena-stop)) -> ARENA -> PANIC +other Unicode: RERUN -> READ(Ok(other)) -> PANIC +non-Unicode: RERUN -> READ(NotUnicode) -> PANIC +``` + +Every arrow after a `println!` assumes that write succeeds. If `RERUN` fails, +it panics before `READ`, and no complete earlier directive is required. If +`SYS` or `ARENA` fails, `RERUN` is the material completed prefix and the script +panics before `RETURN` or the explicit `arena-stop` panic. If `ARENA` succeeds +on `arena-stop`, the material completed prefix is `[RERUN, ARENA]`, followed by +the explicit panic. Under exact accepted entry `BUILD-MAP-ORDERED`, every panic is an +unsuccessful script exit and produces no current library compilation, +regardless of completed prefix. + +On the three successful selector paths, `BUILD-MAP-ORDERED` interprets the final +allocator line as exactly one matching library cfg. Cargo's accepted feature +and target mappings, followed by the Rust conditional-compilation rules, +select the library source. The `W`/`arena` combination selects +`compile_error!`, so it does not produce a library and is genuinely outside +`Config`. The supported `Q` cell selects the first `lane_id` body and removes +the complementary checked body. + +A prior successful `arena` build necessarily completed `RERUN` before +`ARENA`. After the raw selector changes to `arena-stop`, `BUILD-MAP-ORDERED` makes +that prior selection stale, reruns the script, and refuses to present the old +library as the result of the current unsuccessful execution. Thus the +freshness canary is an effective rejection even though its current output +prefix includes the same arena cfg line. + +## Expected Source Results + +The exact maximal sound region requested by the target is: + +```text +SoundRegion = { case in Required | not Q(case) or case.i != 0 }. +``` + +Outside `Q`, the selected body panics when `i == 0`; reaching +`new_unchecked(i)` therefore implies `i != 0`. Inside `Q`, the selected body +calls `new_unchecked` without a check, so nonzero inputs are sound and zero is +not. The complement of `SoundRegion` within `Required` consists exactly of +`Q` with `i == 0` (for every supported profile and debug-assertion state). + +The safe call `lane_id(0)` in any `Q` case reaches +`NonZeroU8::new_unchecked(0)`. Its exact safety proposition is false, and the +Rust 1.85.1 standard-library contract states that zero produces undefined +behavior. Safe-API soundness over all of `Required` is therefore **UNSOUND +relative to `BUILD-MAP-ORDERED`**. + +For the documented zero-input panic postcondition: + +```text +RequiredPanic = { case in Required | case.i = 0 } +CoveredPanic = { case in RequiredPanic | not Q(case) }. +``` + +The regional theorem over `CoveredPanic` is **PROVED relative to +`BUILD-MAP-ORDERED`**. The whole `RequiredPanic` theorem is **UNPROVED**, not +`CONTRACT-BROKEN`: the only missing cases are executions already shown to +contain UB, so their apparent absence of an earlier panic is not a defined +behavioral counterexample. + +## Atoms + +- **B1 — Exact supported case predicate:** The report states a predicate + equivalent to `Required`, including Rust/Cargo 1.85.1, all three targets, + both feature states, both allocator models, every supported Cargo profile, + both debug-assertion states, every `u8`, and only the `W`/`arena` policy + exclusion. + - `scope_basis`: `Cargo.toml`, `SUPPORT.md`, and `BUILD.md`. + - `dependencies`: none; this is the theorem root. + - `accepted_alternatives`: symbolic products or a proved equivalent partition; no + Cartesian enumeration is required. + - `hard_errors`: omission + alone fails B1; claiming a whole-domain affirmative result over a contracted + predicate triggers BH1. + +- **B2 — Exhaustive raw-selector partition:** The report proves that the raw + environment domain is exactly missing, Unicode `system`, Unicode `arena`, + Unicode `arena-stop`, every other Unicode string, and every non-Unicode + value, with no seventh `env::var` outcome. + - `scope_basis`: `BUILD.md`, the + exact `env::var`/`VarError` contracts, and the outer and inner matches. + - `dependencies`: independent of B1; it supplies the cases used + by B3–B7. + - `accepted_alternatives`: any disjoint exhaustive partition that + preserves these behaviorally distinct classes. + - `hard_errors`: + omission alone fails B2; calling an incomplete partition complete and using + it for build closure triggers BH2. + +- **B3 — Complete system-success trace:** Conditional on its two writes + succeeding, each of the missing and Unicode-`system` classes follows exactly + `RERUN -> READ -> SYS -> RETURN`; it emits no arena cfg. + - `scope_basis`: the + ordered statements, `env::var`, match/pattern semantics, `String::as_str`, + and `println!`. + - `dependencies`: B2 identifies the two raw cases; + this is a local-source proposition and does not consume Cargo mapping. + - `accepted_alternatives`: combine the two classes after proving their + identical trace. + - `hard_errors`: omission alone fails B3; an + incompatible endpoint used for closure triggers BH2. + +- **B4 — Complete arena-success trace:** Conditional on its two writes + succeeding, Unicode `arena` follows exactly + `RERUN -> READ -> ARENA -> RETURN`; it emits no system cfg. + - `scope_basis`: + the same local Rust contracts as B3. + - `dependencies`: B2; this + trace is the reachability root for B8–B12. + - `accepted_alternatives`: an + equivalent state-transition row or prose proof. + - `hard_errors`: + omission alone fails B4; an incompatible endpoint used for closure triggers + BH2. + +- **B5 — `arena-stop` partial-prefix exit:** With successful writes, Unicode + `arena-stop` follows exactly + `RERUN -> READ -> ARENA -> PANIC`, so `[RERUN, ARENA]` is a completed output + prefix but the script does not return successfully and, under + `BUILD-MAP-ORDERED`, produces no current library compilation. + - `scope_basis`: + `BUILD.md`, local match/`println!`/`panic!` semantics, and only the exact + unsuccessful-exit proposition of `BUILD-MAP-ORDERED`. + - `dependencies`: + B2; this is distinct from a failed allocator write in B7. + - `accepted_alternatives`: notation may differ, but both ordered lines, the later panic, + and no-current-library consequence must appear. + - `hard_errors`: + omission alone fails B5; treating the partial arena line as a current + successful selector triggers BH4. + +- **B6 — Pre-allocator rejection traces:** Every other Unicode value and every + non-Unicode value follows `RERUN -> READ -> PANIC` after a successful rerun + write, so `[RERUN]` is its completed directive prefix; it attempts no + allocator cfg and produces no current library compilation under + `BUILD-MAP-ORDERED`. + - `scope_basis`: the wildcard and `NotUnicode` arms plus + the exact build premise. + - `dependencies`: B2; the two classes may + share an outcome only after both are accounted for. + - `accepted_alternatives`: + separate rows or one proved union. + - `hard_errors`: omission alone + fails B6; treating either class as an accepted allocator/configuration + triggers BH5. + +- **B7 — Stdout-failure edges:** The report accounts for failure of every + claim-relevant `println!`: failed `RERUN` panics before `READ`; failed `SYS` + or `ARENA` panics with completed prefix `[RERUN]`; and all such unsuccessful + exits produce no current library compilation under `BUILD-MAP-ORDERED`. It does + not require atomic line writes or assign meaning to an incomplete line. + - `scope_basis`: the exact `println!` Panics section, source order, + `panic!`'s main-thread result, and the accepted process-status proposition. + - `dependencies`: B3–B5 identify allocator-write sites; B7 is the + failure completion of those traces. + - `accepted_alternatives`: a single + quantified failure rule is preferred and passes if its instantiations are + readily checkable. + - `hard_errors`: omission alone fails B7; + claiming complete build coverage while a material failure edge supplies a + contrary current artifact triggers BH2 or BH4 as applicable. + +- **B8 — Successful Cargo-to-source selector mapping:** Only the successful + B3/B4 returns supply a current library compilation; Cargo interprets their + exact allocator line as exactly the corresponding `fixture_allocator` + key/value, maps the requested feature and target as admitted, and Rust cfg + rules select source from those resulting predicates. The report keeps + `BUILD-MAP-ORDERED` conspicuous and does not infer local output or Rust semantics + from it. + - `scope_basis`: B3/B4, exact `BUILD-MAP-ORDERED`, the reviewed Cargo + directive contracts, and versioned Rust cfg semantics. + - `dependencies`: B3 through B7. + - `accepted_alternatives`: a relational composition + or explicit stage-by-stage prose. + - `hard_errors`: silently + widening the accepted mapping triggers BH3; treating an unsuccessful prefix + as successful triggers BH4. + +- **B9 — Rerun/freshness theorem:** A prior successful `arena` build completed + `RERUN` before `ARENA`; changing the same target directory's raw selector to + `arena-stop` makes that selection stale, causes a new script run, and yields + no library for the current rejected build rather than reusing the prior arena + library. + - `scope_basis`: local B4/B5 order and precisely the rerun/staleness + proposition accepted in `BUILD-MAP-ORDERED`, reviewed against Cargo's + `rerun-if-env-changed` contract. + - `dependencies`: B4, B5, and the + no-library exit consequence. + - `accepted_alternatives`: a more general proof + for every accepted-to-rejected value change passes if it includes this + requested sequence. + - `hard_errors`: omission alone fails B9; + claiming this current build succeeds from the stale or partial arena output + triggers BH4. + +- **B10 — Effective wasm32/arena exclusion:** For either feature state and all + profiles/debug states, accepted `arena` selection plus the admitted wasm32 + target mapping makes both predicates of the library's first cfg true; Rust + `all` and cfg-attribute semantics therefore retain `compile_error!`, whose + contract makes compilation fail. No `W`/`arena` case is in `Required` or is + used as the UB witness. + - `scope_basis`: `SUPPORT.md`, B8, exact configuration + predicate and attribute semantics, and `compile_error!`. + - `dependencies`: B8. + - `accepted_alternatives`: any parametric proof covering both + feature states and all `P x D`. + - `hard_errors`: calling this pair + supported, using it as a valid-use witness, or asserting rejection without + the selected `compile_error!` proof triggers BH5. + +- **B11 — Exact library-body partition:** The report proves that the unchecked + first body is retained and the complementary checked body removed exactly in + `Q`, while the checked body is retained and the first body removed exactly + outside `Q`. In `Q`, B8 supplies `target_arch="aarch64"`, + `feature="burst"`, and `fixture_allocator="arena"`; Rust's key/value, `all`, + `not`, and cfg-attribute rules establish both directions of the partition. + - `scope_basis`: B8 and the exact versioned cfg propositions. + - `dependencies`: B4 and B8; this is the source-reachability edge for B12/B13. + - `accepted_alternatives`: expansion of the two cfg predicates or an + equivalent Boolean proof. + - `hard_errors`: omission alone fails + B11; declaring the supported unsafe branch unreachable by ignoring generated + cfgs triggers BH6. + +- **B12 — Existential UB certificate and verdict:** The report identifies the + well-typed safe call `lane_id(0)` in a supported `Q` case, uses B11 to reach + `NonZeroU8::new_unchecked(0)`, states that its exact nonzero safety + proposition is false, applies the Rust 1.85.1 zero-UB consequence, and reports + whole-`Required` soundness **UNSOUND relative to `BUILD-MAP-ORDERED`**. + - `scope_basis`: the public safe API, B1, B11, and the exact std contract. + - `dependencies`: B1 and B11. + - `accepted_alternatives`: any one supported + profile/debug state is enough for the existential certificate; proving all + such states is also correct. + - `hard_errors`: an incomplete + certificate followed by `UNSOUND`, or failure to issue `UNSOUND` after giving + every link, triggers BH7. + +- **B13 — Exact maximal sound region:** The report proves both containments for + `SoundRegion = { case in Required | !Q or i != 0 }`: every included case is + sound (the checked body dominates its unsafe call outside `Q`, and nonzero + directly satisfies the unsafe precondition inside `Q`), while every excluded + Required case is exactly `Q`/zero and has B12's UB execution. It labels this + the exact maximal source-level region relative to `BUILD-MAP-ORDERED`. + - `scope_basis`: the user-requested maximal theorem, B1, B11/B12, `if`/comparison/ + panic semantics, and `new_unchecked`. + - `dependencies`: B1, B8, + B11, B12. + - `accepted_alternatives`: an algebraically equivalent formula or + disjoint partition; a correct non-maximal remainder does not pass this + explicitly requested atom. + - `hard_errors`: omission or refusal to + maximize alone fails B13; calling a region maximal while including Q/zero, + excluding sound Q/nonzero cases, or excluding all of `Q` triggers BH8. + +- **B14 — Regional zero-panic theorem:** For every zero-input case outside + `Q`, cfg selection retains the checked body, `value == 0` makes the `if` + consequent execute, and `panic!` prevents reachability of the later unsafe + call. Thus `CoveredPanic = RequiredPanic intersect !Q` is **PROVED relative + to `BUILD-MAP-ORDERED`** across every applicable target/allocator/feature, + profile, and debug state. + - `scope_basis`: B1/B8/B11 and exact comparison, + `if`, cfg, and panic contracts. + - `dependencies`: B1, B8, and the + complement of B11. + - `accepted_alternatives`: a parametric proof or exact + finite configuration partition. + - `hard_errors`: incomplete + regional coverage alone fails B14; falsely claiming the region contains + Q/zero triggers BH9 or BH10 depending on the verdict. + +- **B15 — Whole-domain panic-postcondition verdict:** The report states that + the documented zero-input panic guarantee over all `RequiredPanic` is + **UNPROVED**, not `CONTRACT-BROKEN` and not `PROVED`: Q/zero contains UB, so + its apparent source prefix supplies neither a defined non-panic witness nor a + guaranteed observation, while B14 proves every other zero case. + - `scope_basis`: the exact behavioral quantifier, B12, B14, and whole-execution UB + reasoning. + - `dependencies`: B12 and B14. + - `accepted_alternatives`: + equivalent theorem-specific wording that keeps this verdict separate from + soundness. + - `hard_errors`: using the UB-containing case to claim + `CONTRACT-BROKEN` or an observation before UB triggers BH9; claiming the + universal panic theorem `PROVED` triggers BH10. + +## Hard Errors + +- **BH1:** Claims a whole-domain affirmative result after omitting a supported + release, target, feature, allocator, profile, debug state, or valid `u8` + input, or treats the supported `Q` cell as excluded. +- **BH2:** Claims the build relation or a dependent theorem complete while + replacing the requested ordered/fallible relation with an endpoint summary + that omits a material raw-selector class, failure edge, partial-prefix exit, + or necessary stage. +- **BH3:** Invents, silently widens, or treats as Rust authority any Cargo, + freshness, process-status, feature, or target proposition beyond exact + `BUILD-MAP-ORDERED`. +- **BH4:** Treats a cfg line emitted by an unsuccessful script, or a stale + artifact from an earlier selector, as the library selected for the current + failed build. +- **BH5:** Treats a rejected raw selector as an accepted allocator model; + treats wasm32/arena as supported; uses it as the safe-use UB witness; or + claims its effective rejection without the selected `compile_error!` proof. +- **BH6:** Ignores build-script output or exact Cargo/cfg interpretation and + consequently declares the supported `Q` unsafe branch unreachable. +- **BH7:** Reports `UNSOUND` without the valid safe use, reachability, false + exact safety proposition, and applicable UB consequence, or reports only + `UNPROVED` after explicitly establishing all four links. +- **BH8:** Claims an exact or maximal sound region that includes Q/zero or + excludes any sound Required case, including Q/nonzero. +- **BH9:** Reports `CONTRACT-BROKEN` for the zero-panic promise using only the + UB-containing Q/zero execution, or claims an observation is guaranteed + because it appears before that UB in source order. +- **BH10:** Reports the whole-`RequiredPanic` zero-panic theorem `PROVED` + despite leaving Q/zero without a defined proof. + +## Checked Authorities + +The evaluator must open each exact page, confirm the quoted proposition, and +confirm applicability to Rust/Cargo 1.85.1. The quotation is included to make +the semantic leaf explicit; punctuation-only rendering differences are +irrelevant. + +- `https://doc.rust-lang.org/1.85.1/std/env/fn.var.html` — “Fetches the + environment variable key from the current process”; returns + `VarError::NotPresent` when the variable is not set and + `VarError::NotUnicode` when its value is not valid Unicode. +- `https://doc.rust-lang.org/1.85.1/std/env/enum.VarError.html` — the enum is + exactly `NotPresent` or `NotUnicode(OsString)`; the variant descriptions + distinguish absence from a present value without valid Unicode data. +- `https://doc.rust-lang.org/1.85.1/std/result/enum.Result.html` — `Result` is + exactly `Ok(T)` or `Err(E)` and “represents either success (`Ok`) or failure + (`Err`).” +- `https://doc.rust-lang.org/1.85.1/std/primitive.u8.html` — `u8` is “The 8-bit + unsigned integer type,” with `MIN` equal to zero and `MAX` equal to 255; zero + is therefore an ordinary valid safe-call input. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/match-expr.html` — a + scrutinee value is “sequentially compared to the patterns in the arms until + a match is found,” and the first matching arm is chosen and entered. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/block-expr.html` — a + block sequentially executes its non-item statements and optional final + expression; its type is `()` when the final operand is omitted. +- `https://doc.rust-lang.org/1.85.1/reference/items/functions.html#function-body` + — an omitted function output is unit, and the body is conceptually wrapped + so that its body value is returned; an explicit `return`, if reached, + short-cuts that implicit return. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/call-expr.html` — “A + call expression calls a function”; if the function eventually returns, the + expression completes. +- `https://doc.rust-lang.org/1.85.1/reference/patterns.html#tuple-struct-patterns` + — tuple-struct patterns match tuple-struct and enum values satisfying all + their subpatterns and destructure that value. +- `https://doc.rust-lang.org/1.85.1/reference/patterns.html#literal-patterns` — + literal patterns “match exactly the same value as what is created by the + literal.” +- `https://doc.rust-lang.org/1.85.1/reference/patterns.html#wildcard-pattern` — + the wildcard pattern “matches any value.” +- `https://doc.rust-lang.org/1.85.1/std/string/struct.String.html#method.as_str` + — `as_str` “Extracts a string slice containing the entire `String`.” +- `https://doc.rust-lang.org/1.85.1/std/macro.println.html` — `println!` + “Prints to the standard output, with a newline,” supplying the successful + completed-line effect used by B3–B6. +- `https://doc.rust-lang.org/1.85.1/std/macro.println.html#panics` — `println!` + “Panics if writing to `io::stdout` fails.” +- `https://doc.rust-lang.org/1.85.1/std/macro.panic.html` — `panic!` “Panics the + current thread.” The process-status consequence consumed by this fixture is + separately and explicitly admitted by `BUILD-MAP-ORDERED`. +- `https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#conditional-compilation` + — a set configuration option is true; `all(...)` is true exactly when all + operands are true; and `not(p)` is true exactly when `p` is false. +- `https://doc.rust-lang.org/1.85.1/reference/conditional-compilation.html#the-cfg-attribute` + — a true cfg predicate retains the attached thing without the cfg attribute, + while a false predicate removes it from the source. +- `https://doc.rust-lang.org/1.85.1/std/macro.compile_error.html` — + `compile_error!` “Causes compilation to fail with the given error message + when encountered.” +- `https://doc.rust-lang.org/1.85.1/reference/expressions/operator-expr.html#comparison-operators` + — `==` means equal via `PartialEq::eq`; for the primitive `u8` comparison in + this source it tests whether `value` equals zero. +- `https://doc.rust-lang.org/1.85.1/reference/expressions/if-expr.html` — if the + Boolean condition is true the consequent executes; if false it is skipped. +- `https://doc.rust-lang.org/1.85.1/std/num/struct.NonZero.html#method.new_unchecked` + — the function creates a nonzero without checking, “results in undefined + behavior if the value is zero,” and its Safety section says, “The value must + not be zero.” +- `https://doc.rust-lang.org/1.85.1/reference/behavior-considered-undefined.html` + — Rust programs “must never cause undefined behavior”; unsafe code that no + safe client can trigger to exhibit UB is called sound, and code that safe + code can misuse to exhibit UB is unsound. The whole-execution conclusion is + a logical consequence of classifying the execution as undefined, not a + purported quotation from a source-order or “time travel” subsection. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#life-cycle-of-a-build-script` + — after a build script “successfully finishes executing, the rest of the + package will be compiled”; a nonzero exit halts the build. This reviews, but + does not replace, `BUILD-MAP-ORDERED`. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#outputs-of-the-build-script` + — Cargo interprets each stdout line beginning with `cargo::` as an instruction + affecting package compilation, and instruction order can matter. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rustc-cfg` + — the directive tells Cargo to pass its value to rustc's `--cfg` flag for + conditional compilation. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/build-scripts.html#rerun-if-env-changed` + — the directive tells Cargo to rerun the script if that environment + variable's value changes. +- `https://doc.rust-lang.org/1.85.1/cargo/reference/features.html` — Cargo sets + enabled package features with rustc's `--cfg`, and source can test them with + the cfg attribute or macro. + +Cargo pages are evidence used to review the explicit human trust entry; they +are not Rust abstract-semantics axioms and do not silently enlarge that entry. +No test run, CI result, compiler experiment, prior report, evaluator oracle, or +unversioned documentation closes any atom. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/L.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/L.md new file mode 100644 index 0000000000..e16d3af5a3 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/L.md @@ -0,0 +1,146 @@ +# Mode L Frozen Blind-Scoring Rubric + +> **Evaluator-only material. Never expose this file to a report agent.** + +> Evaluator-only material. Never expose this file, its atom labels, expected +> result, or hard-error rules to an evaluated report agent. + +## Exact result + +Within the scope in `REQUEST.md`, `last` is sound. On the `else` branch, +`bytes.is_empty()` evaluated to false. The documented relation between +`is_empty` and slice length therefore gives `bytes.len() != 0`. Slice length has +type `usize`, whose values are nonnegative, so `bytes.len() > 0`. Consequently +`bytes.len() - 1` neither underflows nor overflows and mathematical subtraction +gives `index < bytes.len()`. This is precisely the in-bounds fact required by +`get_unchecked` for a `usize` index. + +The existing comment states only a lifetime fact. It does not address the +unsafe callee's in-bounds obligation, the facts which establish it, or the +derivation between them. It is therefore a deficient proof artifact even though +the implementation obligation can be reconstructed. A replacement must expose +that chain locally and must link or unambiguously refer to the checked, +version-matched authority entries it consumes. + +## Atoms + +- **L1 — Callee obligation:** State that the executed `get_unchecked(index)` + requires `index` to be in bounds, equivalently `index < bytes.len()` for this + `usize` index, and that an out-of-bounds call is UB. + - `scope_basis`: `lib.rs` contains this unsafe call and `REQUEST.md` asks for + an implementation soundness audit. + - `dependencies`: none. +- **L2 — Branch fact:** Establish from the actual `if`/`else` control flow that + `bytes.is_empty()` is false on the unsafe-call path, then use the exact + `is_empty` contract to conclude `bytes.len() != 0`. + - `scope_basis`: necessary source-path premise for the requested local proof. + - `dependencies`: none. +- **L3 — Unsigned-domain step:** Establish that the slice length is a `usize` + and, using its nonnegative value domain, derive `bytes.len() > 0` from L2. + - `scope_basis`: necessary material bridge from nonemptiness to safe + subtraction in every supported pointer width. + - `dependencies`: L2. +- **L4 — Defined subtraction:** Establish that `bytes.len() - 1` has no integer + overflow or underflow on this path, including in every ordinary profile. + - `scope_basis`: the source performs built-in integer subtraction before the + unsafe call; its definedness is part of the requested soundness proof. + - `dependencies`: L3. +- **L5 — In-bounds derivation:** Use the assignment + `index = bytes.len() - 1` and ordinary integer arithmetic to derive + `index < bytes.len()`. + - `scope_basis`: the exact proposition needed to discharge L1. + - `dependencies`: L3, L4. +- **L6 — Implementation closure:** Connect L2–L5 to L1 and report the + implementation obligation `PROVED` over the entire stated Rust/target/profile + scope; do not infer implementation failure from comment failure. + - `scope_basis`: `REQUEST.md` explicitly asks for an implementation verdict + over its full scope. + - `dependencies`: L1, L2, L3, L4, L5. +- **L7 — Artifact verdict:** Separately report the existing `SAFETY` comment + materially deficient because its lifetime statement does not address either + the exact in-bounds obligation or the material derivation which discharges + that obligation. + - `scope_basis`: `REQUEST.md` explicitly asks for a separate comment-adequacy + determination. + - `dependencies`: L1, L2, L3, L4, L5. +- **L8 — Replacement proof artifact:** Supply proposed adjacent comment text + which identifies the `get_unchecked` obligation and exposes the L2–L5 chain, + with unambiguous references to the checked authority propositions. + - `scope_basis`: `REQUEST.md` explicitly requests replacement wording when + material reconstruction is needed. + - `dependencies`: L1, L2, L3, L4, L5. +- **L9 — Complete premise inventory:** Inventory every materially consumed + Rust/std premise: `if` branch selection, `is_empty`/length equivalence, + `len`'s type/meaning, `usize`'s value domain, built-in + subtraction/overflow, and `get_unchecked`'s safety contract. Classify the + immediate placement and assignment observations as source facts and the + remaining integer implications as mathematics rather than inventing + authority for them. + - `scope_basis`: `REQUEST.md` explicitly requires a complete Rust-premise + inventory and reconciliation; distinguishing source facts and mathematics + identifies which proof leaves require Rust/std authority. + - `dependencies`: L1, L2, L3, L4, L5. +- **L10 — Exact checked authority:** For every Rust/std entry in L9, give the + matching Rust 1.82.0 narrow URL, quote the exact prose which supplies the + premise, state the verified proposition, and qualify its applicability. Page + identity or an unexamined citation alone does not pass. + - `scope_basis`: `REQUEST.md` explicitly requires version-matched checked + authority and exact extracted propositions. + - `dependencies`: L9. +- **L11 — Reconciliation:** Explicitly reconcile proof to inventory: every + materially consumed Rust/std premise has an L10 entry, every listed entry is + actually consumed, and no TCB assumption or uncited Rust fact silently fills + a link. + - `scope_basis`: `REQUEST.md` explicitly requires this reconciliation and + states that the TCB is empty. + - `dependencies`: L1, L2, L3, L4, L5, L9, L10. + +Score atoms independently. Mathematical implications between stated integer +facts need derivation but no Rust citation. Immediate source syntax may be cited +by source location rather than external authority. + +## Hard errors + +- **LH1:** Report the implementation `UNSOUND` or `UNPROVED` solely because the + existing comment is deficient, despite reconstructing L1–L5 correctly. +- **LH2:** Explicitly call the existing comment adequate. +- **LH3:** Claim the requested implementation proof complete while omitting a + material link from L1–L5. +- **LH4:** Claim the requested authority/premise reconciliation complete while + silently consuming a Rust/std premise, using an inapplicable version, or + treating a page URL as proof without checking the proposition. +- **LH5:** Use build, test, execution, lint, or dynamic-analysis success as a + universal proof of this obligation. + +An omitted artifact verdict fails L7 but is not independently a hard error +unless the report also makes an affirmative false claim covered above. + +## Checked authority + +- Rust 1.82.0 slice `get_unchecked`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked` + — for a `usize` element index, an out-of-bounds call is UB even when the + result is unused; the proof consumes the corresponding `index < len` + requirement. +- Rust 1.82.0 slice `len`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.len` + — returns the number of elements as `usize`. +- Rust 1.82.0 slice `is_empty`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.is_empty` + — returns true exactly when the slice has length zero. +- Rust 1.82.0 Reference, `if` expressions: + `https://doc.rust-lang.org/1.82.0/reference/expressions/if-expr.html#if-expressions` + — when the condition evaluates to false, the consequent block is skipped and + the trailing `else` block is executed. +- Rust 1.82.0 Reference, arithmetic operators: + `https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#arithmetic-and-logical-binary-operators` + — `-` is built-in subtraction for integer operands. +- Rust 1.82.0 Reference, integer overflow: + `https://doc.rust-lang.org/1.82.0/reference/expressions/operator-expr.html#overflow` + — identifies overflow for built-in integer operators and its profile/context + behavior; L3 proves the exceptional case absent here. +- Rust 1.82.0 Reference, unsigned integer value domains: + `https://doc.rust-lang.org/1.82.0/reference/types/numeric.html#integer-types` + — `usize` is an unsigned integer type and its values are nonnegative. + +There is no accepted TCB entry for this mode. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/P.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/P.md new file mode 100644 index 0000000000..90f2e84814 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/P.md @@ -0,0 +1,565 @@ +# Mode P Frozen Blind-Scoring Rubric + +> **Evaluator-only material. Never expose this file to a report agent.** + +> **Evaluator-only material. Never expose this file to a report agent.** + +## P — Incomparable Predicates and Full-Case Closure + +### Canonical notation and policy algebra + +Use the fixture's notation: + +```text +V = {1.84.0, 1.85.0, 1.86.0} +T = {X, A, W} +F = H = D = {false, true} +P = every Cargo profile admitted by the supplied policies +I = {None} union {Some(b) | b is any u8} +case = (v, t, f, h, p, d, i) +``` + +The two exact configuration predicates are: + +```text +Scarlet(v,t,f,h) + iff v in V and t in T and + (!f + or (f and t = X and (!h or v >= 1.85.0)) + or (f and t = A and h)). + +Indigo(v,t,f,h) + iff v in V and t in T and + (!f + or (f and t = X and (h or v >= 1.86.0)) + or (f and t = A and !h and v >= 1.85.0)). +``` + +They are incomparable. For example: + +```text +(1.84.0, X, true, false) is Scarlet-only. +(1.84.0, X, true, true) is Indigo-only. +``` + +Any correct separating witness is acceptable. In particular, an answer need +not use the two examples above. There is no authorized rule identifying one +predicate, their intersection, their union, or any other combination as the +crate's controlling exact promise. + +The policy-induced full-case sets are: + +```text +ScarletCases = {case | Scarlet(v,t,f,h) and p in P and d in D and i in I} +IndigoCases = {case | Indigo(v,t,f,h) and p in P and d in D and i in I} +``` + +The canonical minimal conservative audit domain is their union: + +```text +Audit = ScarletCases union IndigoCases +Required(case) iff case in Audit +``` + +Its configuration projection has this equivalent exact formula: + +```text +UnionCfg(v,t,f,h) + iff v in V and t in T and + (!f + or (f and t = X) + or (f and t = A and (h or v >= 1.85.0))). + +Required(v,t,f,h,p,d,i) + iff UnionCfg(v,t,f,h) and p in P and d in D and i in I. +``` + +A report may instead choose a transparent full-case superset, provided it +proves `ScarletCases subset-of Audit` and `IndigoCases subset-of Audit` +separately and states a theorem whose meaning remains coherent for every added +case. The chosen set is an audit domain, not an inferred project promise. + +### Intended source proof + +Both policies exclude every configuration with `f = true` and `t = W`. +Relative only to accepted `BUILD-MAP-POLICY`, such a build selects +`cfg(all(feature = "turbo", target_arch = "wasm32"))`. The authoritative +`all`, `cfg`-attribute, and `compile_error!` contracts then establish that the +ordinary library compilation fails. This reasoning is parametric in `v`, `h`, +`p`, and `d`; there is no library input after compilation is rejected. + +For every compiled case selected by the audit domain: + +- when `f = false`, `unwrap_or(0)` returns the contained byte for `Some(byte)` + and zero for `None`; +- when `f = true`, the same `unwrap_or(0)` first produces that result `r`, the + source constructs `Some(r)`, and `unwrap_unchecked` is therefore invoked on + `Some`, never `None`; its return contract produces the same `r`. + +One exact full-case implementation region for both soundness and behavior is: + +```text +Covered(v,t,f,h,p,d,i) + iff v in V and t in T and f in F and h in H and p in P and d in D and i in I + and (!f or (f and t in {X,A})). +``` + +The source derivation is independent of `v`, `h`, `p`, `d`, and the particular +member of `I`, but those universally quantified dimensions remain in both +`Required(case)` and `Covered(case)`. Every disjunct of `UnionCfg` implies the +final source-selection clause above, so: + +```text +Required subset-of Covered_sound +Required subset-of Covered_behavior +``` + +The two covered predicates may be represented by the same set, but the unsafe +precondition proof and the postcondition proof are distinct obligations. + +### Expected conclusions + +- The identity of the controlling exact project predicate is **UNRESOLVED** + (equivalently, **UNPROVED** as a policy-identification proposition). +- Safe-API soundness on the chosen conservative full-case audit domain is + **PROVED relative to `BUILD-MAP-POLICY`**. +- The documented `value_or_zero` postcondition on that domain is **PROVED + relative to `BUILD-MAP-POLICY`**. +- Effective rejection of every `turbo`/`wasm32` build in the supplied release, + feature, profile, and debug-assertion scope is **PROVED relative to + `BUILD-MAP-POLICY`**. + +The qualification is required because the mapping from policy configuration +coordinates to Rust `cfg` options is an accepted tool premise, not a Rust +semantic axiom. + +### Scored atoms + +Each atom below scores exactly the proposition in its `proposition` field. +The other fields identify why that proposition is in scope and how to diagnose +a failure; they add no hidden report requirement. + +- **P1 — Exact Scarlet predicate** + - `proposition`: The report's `Scarlet(v,t,f,h)` set equals the predicate in + `POLICY-SCARLET.md`. + - `scope_basis`: “Reproduce the exact Scarlet and Indigo configuration + predicates separately.” (`REQUEST.md`) + - `necessary_edge`: Expand every Scarlet disjunct, including its versioned + x86 clause and hardened aarch64 clause. + - `dependencies`: none. + - `root_cause`: A wrong Scarlet formula is the root cause even if it later + causes witness or containment failures. + - `accepted_alternatives`: Any logically equivalent formula, truth-table + partition, or exact finite set description. + +- **P2 — Exact Indigo predicate** + - `proposition`: The report's `Indigo(v,t,f,h)` set equals the predicate in + `POLICY-INDIGO.md`. + - `scope_basis`: “Reproduce the exact Scarlet and Indigo configuration + predicates separately.” (`REQUEST.md`) + - `necessary_edge`: Expand every Indigo disjunct, including its versioned + x86 and unhardened aarch64 clauses. + - `dependencies`: none. + - `root_cause`: A wrong Indigo formula is the root cause even if it later + causes witness or containment failures. + - `accepted_alternatives`: Any logically equivalent formula, truth-table + partition, or exact finite set description. + +- **P3 — Scarlet-minus-Indigo witness** + - `proposition`: There exists a stated configuration in Scarlet but not + Indigo. + - `scope_basis`: “giving one concrete separating configuration in each + direction if neither contains the other.” (`REQUEST.md`) + - `necessary_edge`: Substitute the witness into both exact predicates and + establish `Scarlet and not Indigo`. + - `dependencies`: P1 and P2. + - `root_cause`: If substitution is correct only for a previously misstated + predicate, record that predicate atom as the originating failure. + - `accepted_alternatives`: Any valid Scarlet-only configuration. + +- **P4 — Indigo-minus-Scarlet witness** + - `proposition`: There exists a stated configuration in Indigo but not + Scarlet. + - `scope_basis`: “giving one concrete separating configuration in each + direction if neither contains the other.” (`REQUEST.md`) + - `necessary_edge`: Substitute the witness into both exact predicates and + establish `Indigo and not Scarlet`. + - `dependencies`: P1 and P2. + - `root_cause`: If substitution is correct only for a previously misstated + predicate, record that predicate atom as the originating failure. + - `accepted_alternatives`: Any valid Indigo-only configuration. + +- **P5 — Exact-promise status** + - `proposition`: The supplied policy evidence leaves the controlling exact + project predicate unidentified. + - `scope_basis`: “No precedence, supersession, intersection, union, or other + conflict-resolution rule has been authorized.” (`REQUEST.md`) + - `necessary_edge`: Keep policy identification separate from selection of a + conservative audit domain. + - `dependencies`: none. + - `root_cause`: Treating an analyst-selected set as policy evidence is the + originating error. + - `accepted_alternatives`: `UNRESOLVED`, or `UNPROVED` explicitly limited to + policy identification. + +- **P6 — Scarlet full-case domain** + - `proposition`: `ScarletCases` equals the Scarlet configuration set crossed + with every admitted profile, both debug-assertion states, and every valid + `Option` input. + - `scope_basis`: “Define the full-case domain induced by each policy.” + (`REQUEST.md`) + - `necessary_edge`: Preserve all seven named coordinates in the set. + - `dependencies`: P1. + - `root_cause`: A missing coordinate is a theorem-domain recovery failure, + not an implementation-proof failure. + - `accepted_alternatives`: Product notation, a set comprehension, or exact + universally quantified prose. + +- **P7 — Indigo full-case domain** + - `proposition`: `IndigoCases` equals the Indigo configuration set crossed + with every admitted profile, both debug-assertion states, and every valid + `Option` input. + - `scope_basis`: “Define the full-case domain induced by each policy.” + (`REQUEST.md`) + - `necessary_edge`: Preserve all seven named coordinates in the set. + - `dependencies`: P2. + - `root_cause`: A missing coordinate is a theorem-domain recovery failure, + not an implementation-proof failure. + - `accepted_alternatives`: Product notation, a set comprehension, or exact + universally quantified prose. + +- **P8 — Scarlet containment** + - `proposition`: The chosen conservative full-case audit domain contains + `ScarletCases`. + - `scope_basis`: “Prove each containment separately.” (`REQUEST.md`) + - `necessary_edge`: Prove `ScarletCases subset-of Audit` from the actual + stated formulas. + - `dependencies`: P6 and the report's definition of `Audit`. + - `root_cause`: A malformed upstream Scarlet domain remains the originating + failure; an absent inclusion argument is independently P8. + - `accepted_alternatives`: Membership in an exact union, algebraic + implication into a superset, or exhaustive finite proof. + +- **P9 — Indigo containment** + - `proposition`: The chosen conservative full-case audit domain contains + `IndigoCases`. + - `scope_basis`: “Prove each containment separately.” (`REQUEST.md`) + - `necessary_edge`: Prove `IndigoCases subset-of Audit` from the actual + stated formulas. + - `dependencies`: P7 and the report's definition of `Audit`. + - `root_cause`: A malformed upstream Indigo domain remains the originating + failure; an absent inclusion argument is independently P9. + - `accepted_alternatives`: Membership in an exact union, algebraic + implication into a superset, or exhaustive finite proof. + +- **P10 — Scarlet wasm exclusion** + - `proposition`: Every configuration with `f = true` and `t = W` is outside + Scarlet. + - `scope_basis`: “Account for both policy-level exclusion ... of every + `turbo`/`wasm32` case.” (`REQUEST.md`) + - `necessary_edge`: Evaluate the Scarlet predicate with those two fixed + coordinates while leaving `v` and `h` universal. + - `dependencies`: P1. + - `root_cause`: A wrong Scarlet formula is upstream; failure to apply the + correct formula here is local. + - `accepted_alternatives`: Direct substitution or exclusion by disjuncts. + +- **P11 — Indigo wasm exclusion** + - `proposition`: Every configuration with `f = true` and `t = W` is outside + Indigo. + - `scope_basis`: “Account for both policy-level exclusion ... of every + `turbo`/`wasm32` case.” (`REQUEST.md`) + - `necessary_edge`: Evaluate the Indigo predicate with those two fixed + coordinates while leaving `v` and `h` universal. + - `dependencies`: P2. + - `root_cause`: A wrong Indigo formula is upstream; failure to apply the + correct formula here is local. + - `accepted_alternatives`: Direct substitution or exclusion by disjuncts. + +- **P12 — Effective wasm rejection** + - `proposition`: Relative to exact `BUILD-MAP-POLICY`, every in-scope build with + `f = true` and `t = W` fails library compilation at the selected + `compile_error!`. + - `scope_basis`: “Account for ... source-level effective rejection of every + `turbo`/`wasm32` case.” (`REQUEST.md`) + - `necessary_edge`: Follow tool mapping to both true cfg options, apply + `all`, apply the `cfg` attribute, then apply `compile_error!`, universally + over release, hardened, profile, and debug-assertion coordinates. + - `dependencies`: exact TCB use and exact-version Rust authority. + - `root_cause`: The first absent edge among mapping, predicate selection, and + compile failure is the useful diagnosis. + - `accepted_alternatives`: An equivalent symbolic proof for all such builds; + enumeration is not required. + +- **P13 — Non-turbo postcondition** + - `proposition`: On every selected non-turbo branch and every valid input, + `value_or_zero` returns the contained byte for `Some(byte)` and zero for + `None`. + - `scope_basis`: “Prove or refute ... the documented `value_or_zero` + postcondition on each selected source branch.” (`REQUEST.md`) + - `necessary_edge`: Apply the exact-version `unwrap_or(0)` contract to the + two `Option` variants. + - `dependencies`: exact Option authority. + - `root_cause`: Missing or inapplicable `unwrap_or` authority is the semantic + root; missing input quantification is a domain root. + - `accepted_alternatives`: Variant cases or one exact contract application. + +- **P14 — Turbo unsafe precondition** + - `proposition`: Every reachable turbo-branch `unwrap_unchecked` call has a + `Some` receiver. + - `scope_basis`: “Prove or refute the local safety obligation at the unsafe + operation.” (`REQUEST.md`) + - `necessary_edge`: Trace the value through `unwrap_or`, the local + `Some(value)` construction, and the unchanged receiver at the unsafe call. + - `dependencies`: exact Option authority and the inspected local source. + - `root_cause`: Failure to connect the constructed receiver to the call is a + local proof-edge failure, not a policy failure. + - `accepted_alternatives`: Any equivalent local proof that identifies the + exact call receiver; no control-flow argument is required. + +- **P15 — Turbo postcondition** + - `proposition`: On every selected turbo branch and every valid input, + `value_or_zero` returns the contained byte for `Some(byte)` and zero for + `None`. + - `scope_basis`: “Prove or refute ... the documented `value_or_zero` + postcondition on each selected source branch.” (`REQUEST.md`) + - `necessary_edge`: Compose `unwrap_or(0)`'s result with construction of + `Some(result)` and `unwrap_unchecked`'s return contract. + - `dependencies`: P14 and exact Option authority. + - `root_cause`: A safety proof alone does not supply the return-value edge; + diagnose that edge separately. + - `accepted_alternatives`: Direct composition or explicit `None`/`Some` + cases. + +- **P16 — Full Required predicate** + - `proposition`: The report's final `Required(case)` retains version, target, + both features, profile, debug assertions, and input over its chosen audit + domain. + - `scope_basis`: “State `Required(case)` ... without projecting away the + configuration or input dimensions.” (`REQUEST.md`) + - `necessary_edge`: Bind all seven coordinates before making a universal + conclusion. + - `dependencies`: P6 through P9. + - `root_cause`: Omission here is a final theorem-domain failure even if the + omitted axis happens not to affect source behavior. + - `accepted_alternatives`: Exact set notation or universally quantified + prose; irrelevant dimensions may be symbolic. + +- **P17 — Full soundness Covered coordinates** + - `proposition`: `Covered_sound(case)` retains all seven case coordinates. + - `scope_basis`: “State ... the applicable soundness ... `Covered(case)` + predicates without projecting away the configuration or input dimensions.” + (`REQUEST.md`) + - `necessary_edge`: Lift the branch-local proof to a full-case predicate + rather than replacing the case by a configuration projection. + - `dependencies`: P16. + - `root_cause`: A dimensionless relation is a theorem-bookkeeping error even + when the branch proof itself is correct. + - `accepted_alternatives`: Exact set notation or universally quantified + prose; irrelevant dimensions may remain symbolic. + +- **P18 — Soundness Covered validity** + - `proposition`: Every case included in `Covered_sound(case)` has a complete + applicable unsafe-obligation derivation in the report. + - `scope_basis`: “State ... the applicable soundness ... `Covered(case)` + predicates.” (`REQUEST.md`) + - `necessary_edge`: Admit a case to Covered only after the branch-local + unsafe proof applies to that case. + - `dependencies`: P14 plus the inspected fact that the non-turbo branch uses + no unsafe operation. + - `root_cause`: An overbroad Covered set is an implementation-proof error, + not a theorem-domain recovery error. + - `accepted_alternatives`: The canonical region or any other proved region; + no maximal positive-region characterization is required. + +- **P19 — Soundness closure** + - `proposition`: `Required subset-of Covered_sound` holds for the report's + stated full-case sets. + - `scope_basis`: “Give the set-containment argument needed for each + whole-domain conclusion.” (`REQUEST.md`) + - `necessary_edge`: Map every audit-domain configuration disjunct to its + proved source branch while carrying profile, debug, and input universally. + - `dependencies`: P8, P9, P14, and P16 through P18. + - `root_cause`: Record every downstream atom as failed when absent, but name + the earliest wrong predicate, local proof, or containment edge as cause. + - `accepted_alternatives`: Algebraic implication, structured cases, or an + exact finite proof; enumeration is not required. + +- **P20 — Whole-domain soundness conclusion** + - `proposition`: Relative to exact `BUILD-MAP-POLICY`, safe-API soundness is + PROVED on the chosen audit domain. + - `scope_basis`: “state the strongest conclusions that the supplied evidence + justifies.” (`REQUEST.md`) + - `necessary_edge`: Use P19 as the certificate and no wider TCB proposition. + - `dependencies`: P19 and exact TCB accounting. + - `root_cause`: If closure is missing, that proof failure is primary; an + unqualified otherwise-correct verdict is a TCB-accounting failure. + - `accepted_alternatives`: Equivalent wording which clearly states domain, + source-level soundness, positive status, and TCB relativity. + +- **P21 — Full behavioral Covered coordinates** + - `proposition`: `Covered_behavior(case)` retains all seven case coordinates. + - `scope_basis`: “State ... the applicable ... behavioral `Covered(case)` + predicates without projecting away the configuration or input dimensions.” + (`REQUEST.md`) + - `necessary_edge`: Lift both branch-local return proofs to a full-case + predicate. + - `dependencies`: P16. + - `root_cause`: A dimensionless relation is a theorem-bookkeeping error even + when the return proof itself is correct. + - `accepted_alternatives`: Exact set notation or universally quantified + prose; irrelevant dimensions may remain symbolic. + +- **P22 — Behavioral Covered validity** + - `proposition`: Every case included in `Covered_behavior(case)` has a + complete applicable postcondition derivation in the report. + - `scope_basis`: “State ... the applicable ... behavioral `Covered(case)` + predicates.” (`REQUEST.md`) + - `necessary_edge`: Admit a case to Covered only after a branch-local return + proof applies to that case. + - `dependencies`: P13 and P15. + - `root_cause`: Do not infer this validity proposition from soundness; the + documented return theorem needs its own proof edges. + - `accepted_alternatives`: The canonical region or any other proved region; + no maximal positive-region characterization is required. + +- **P23 — Behavioral closure** + - `proposition`: `Required subset-of Covered_behavior` holds for the report's + stated full-case sets. + - `scope_basis`: “Give the set-containment argument needed for each + whole-domain conclusion.” (`REQUEST.md`) + - `necessary_edge`: Map every audit-domain configuration and input case to a + branch with the proved documented return. + - `dependencies`: P8, P9, P13, P15, P16, P21, and P22. + - `root_cause`: Record every downstream atom as failed when absent, but name + the earliest wrong predicate, postcondition proof, or containment edge as + cause. + - `accepted_alternatives`: Algebraic implication, structured cases, or an + exact finite proof; enumeration is not required. + +- **P24 — Whole-domain behavioral conclusion** + - `proposition`: Relative to exact `BUILD-MAP-POLICY`, the documented + postcondition is PROVED on the chosen audit domain. + - `scope_basis`: “state the strongest conclusions that the supplied evidence + justifies.” (`REQUEST.md`) + - `necessary_edge`: Use P23 as the certificate and no wider TCB proposition. + - `dependencies`: P23 and exact TCB accounting. + - `root_cause`: If closure is missing, that proof failure is primary; an + unqualified otherwise-correct verdict is a TCB-accounting failure. + - `accepted_alternatives`: Equivalent wording which clearly states domain, + exact postcondition, positive status, and TCB relativity. + +- **P25 — Exact TCB scope** + - `proposition`: Every proposition attributed to `BUILD-MAP-POLICY` is admitted by + that entry's exact identity, mapping, consumer, and release/profile region. + - `scope_basis`: “Apply it only to its exact build-tool mappings and + consumers.” (`REQUEST.md`) + - `necessary_edge`: Match each TCB-dependent conclusion to the supplied entry + without adding a semantic or backend proposition. + - `dependencies`: none; this is an accounting proposition. + - `root_cause`: An invented mapping is the primary error even if subsequent + cfg reasoning would be valid under it. + - `accepted_alternatives`: A compact TCB ledger or precise inline + qualifications. + +- **P26 — Conspicuous TCB qualification** + - `proposition`: Every conclusion which depends on `BUILD-MAP-POLICY` identifies + that dependency as an accepted trust premise rather than Rust authority. + - `scope_basis`: “keep every conclusion which depends on it conspicuously + qualified.” (`REQUEST.md`) + - `necessary_edge`: Carry the TCB label from configuration reachability into + each dependent rejection, soundness, and behavior conclusion. + - `dependencies`: none; this is a reporting proposition. + - `root_cause`: An otherwise valid but unconditional conclusion fails here; + do not misdiagnose it as a Rust-authority mismatch. + - `accepted_alternatives`: Inline `relative to BUILD-MAP-POLICY` wording or an + unambiguous shared qualification covering the dependent conclusions. + +- **P27 — Exact authority closure** + - `proposition`: Every material Rust semantic premise used in a positive + proof is paired with an applicable exact-version URL and a quotation which + supplies that premise. + - `scope_basis`: “For each material Rust semantic premise, cite the applicable + exact-version Rust Reference or standard-library page and quote the prose + which supplies that premise.” (`REQUEST.md`) + - `necessary_edge`: Reconcile the proof leaves against the authority inventory + below; a page label alone supplies no unstated proposition. + - `dependencies`: every positive Rust-semantic derivation. + - `root_cause`: Identify the first missing, mismatched, or overclaimed + quotation; do not treat one omission as multiple distinct authority roots. + - `accepted_alternatives`: Shorter exact excerpts and different organization + are acceptable when the proposition and release applicability are clear. + +### Hard errors + +A mere omission, an explicit refusal to close a proposition, or a downstream +atom failure caused by a disclosed upstream gap is not by itself a hard error. +Record a hard error only for the corresponding false affirmative claim below: + +- **PH1:** selects Scarlet, Indigo, their intersection, their union, or another + combination as the controlling exact project promise without new policy + evidence; +- **PH2:** relabels an analyst-selected conservative audit domain as the exact + project promise; +- **PH3:** asserts containment or incomparability using a stated formula and + witness which do not establish the asserted set relation; +- **PH4:** claims whole-domain soundness or behavior after dropping a required + configuration or input coordinate, or without a valid full-case + `Required subset-of Covered` argument; +- **PH5:** treats `turbo`/`wasm32` as policy-supported, or claims effective + rejection without the exact accepted mapping and applicable `all`, `cfg`, and + `compile_error!` implications; +- **PH6:** calls the shown source `UNSOUND` or calls its documented + postcondition `CONTRACT-BROKEN` on the supplied audit domain; +- **PH7:** invents or widens `BUILD-MAP-POLICY`, treats it as Rust authority, or uses + it outside its exact identity, propositions, consumers, or region. + +Record independently applicable global hard errors as well. + +### Exact authority propositions, quotations, and URLs + +The following quotations were checked on every listed exact-version page. The +quotation is the semantic content; presence on an allowlist is not itself +evidence for a proposition. + +For Rust 1.84.0, 1.85.0, and 1.86.0, the `Option` pages describe `unwrap_or` +using the “contained `Some` value” or “provided default,” and +`unwrap_unchecked` returning the “contained `Some` value.” Its Safety heading +says calling it on `None` “is undefined behavior.” Construction of +`Some(value)` is an immediate inspected source fact, not a separately cited +semantic premise. + +Exact pages, in canonical allowlist order: + +- `https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_or` +- `https://doc.rust-lang.org/1.84.0/std/option/enum.Option.html#method.unwrap_unchecked` +- `https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_or` +- `https://doc.rust-lang.org/1.85.0/std/option/enum.Option.html#method.unwrap_unchecked` +- `https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_or` +- `https://doc.rust-lang.org/1.86.0/std/option/enum.Option.html#method.unwrap_unchecked` + +For each of the same releases, the Conditional Compilation chapter says a +configuration predicate is “true if the option is set,” `all` requires “all of +the given predicates” to be true, and `not` is true when its “predicate is +false.” The `cfg` attribute “conditionally includes” its attached item. Those +checked excerpts supply exactly the source-selection implications used here. + +The standard-library macro page states: + +- “Causes compilation to fail with the given error message when encountered.” + +Exact pages, continuing canonical allowlist order: + +- `https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#conditional-compilation` +- `https://doc.rust-lang.org/1.84.0/reference/conditional-compilation.html#the-cfg-attribute` +- `https://doc.rust-lang.org/1.84.0/std/macro.compile_error.html` +- `https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#conditional-compilation` +- `https://doc.rust-lang.org/1.85.0/reference/conditional-compilation.html#the-cfg-attribute` +- `https://doc.rust-lang.org/1.85.0/std/macro.compile_error.html` +- `https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#conditional-compilation` +- `https://doc.rust-lang.org/1.86.0/reference/conditional-compilation.html#the-cfg-attribute` +- `https://doc.rust-lang.org/1.86.0/std/macro.compile_error.html` + +`BUILD-MAP-POLICY` is the only admitted non-Rust premise. No policy sentence, +Cargo minimum, evaluator oracle, test result, CI result, or inferred tool +behavior is a Rust semantic axiom. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/Q.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/Q.md new file mode 100644 index 0000000000..503229e427 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/Q.md @@ -0,0 +1,78 @@ +# Mode Q Frozen Blind-Scoring Rubric + +> **Evaluator-only material. Never expose this file to a report agent.** + +> Evaluator-only material. Never expose this file to a report agent. + +Let `C_Q` be Rust 1.80.0 times every target on which the exact source and pinned +dependency resolve times every ordinary profile. + +- `Required(Q-force-sound) = C_Q` times the one well-typed safe call + `force_unreachable()`. +- `Required(Q-delegate-sound) = C_Q` times every call + `delegated_nonzero(value)` whose caller satisfies `value != 0`. + +## Atoms + +- **Q1 — Domain and claim separation:** State both valid-use domains and audit + the two APIs independently. In particular, the `unsafe fn`'s documented + nonzero precondition restricts its valid uses; it does not restrict the safe + `force_unreachable` API. + - `scope_basis`: `REQUEST.md` explicitly asks for two separate claims and + supplies their common compilation scope; each API's signature/contract + determines its valid-use domain. + - `dependencies`: none. +- **Q2 — Complete UB certificate:** For `force_unreachable`, give all four + links explicitly or by an equivalently exact chain: (1) the safe call is + well-typed and valid; (2) its only body path reaches the executed + `unreachable_unchecked` call; (3) that call requires its site not to be + reached and that proposition is false; and (4) applicable Rust 1.80.0 std + documentation says reaching it is UB. + - `scope_basis`: necessary certificate for the requested soundness verdict + on the safe API. + - `dependencies`: Q1. +- **Q3 — Existential closure:** Report `Q-force-sound` **UNSOUND**. The complete + in-scope witness refutes universal soundness; do not weaken it to `UNPROVED` + merely because no universal positive proof was built. + - `scope_basis`: `REQUEST.md` asks for the strongest justified verdict. + - `dependencies`: Q2. +- **Q4 — Exact missing premise:** For `delegated_nonzero`, recognize that a + valid caller supplies the dependency API's nonzero caller precondition, but + the selected unsafe dependency body is absent and no accepted `UNSAFE-DEP` + entry establishes that it honors its contract. The pin and caller-facing + docs establish the logical coordinate and propagated obligation, not + authenticated implementation identity or correctness. The smallest missing + proposition is that every valid nonzero call to the selected + `peer-unsafe-dep` 3.4.5 implementation is UB-free over this domain. + - `scope_basis`: the supplied evidence packet and TCB policy are expressly + part of the requested dependency claim review. + - `dependencies`: Q1. +- **Q5 — No fabricated witness or silent trust:** Report + `Q-delegate-sound` **UNPROVED**, not `PROVED` or `UNSOUND`. Its required cases + lack the dependency-soundness premise, while the unavailable body supplies no + proved UB execution. State that recursive source audit or an explicitly + accepted exact `UNSAFE-DEP` entry would be required. + - `scope_basis`: `REQUEST.md` asks for the strongest packet-supported verdict + and smallest missing proposition. + - `dependencies`: Q4. + +## Hard errors + +- **QH1:** Call `force_unreachable` sound or only `UNPROVED` after establishing + all four Q2 links. +- **QH2:** Issue `UNSOUND` for it without a valid call, reachability, false exact + safety proposition, and authoritative UB consequence. +- **QH3:** Silently trust the unavailable unsafe dependency, treat the pin as + implementation proof, or report `delegated_nonzero` `PROVED`. +- **QH4:** Fabricate dependency body behavior or a UB execution and report + `delegated_nonzero` `UNSOUND`. +- **QH5:** Treat violation of the dependency's documented caller precondition + as itself an executed UB event. + +## Checked authority + +- `https://doc.rust-lang.org/1.80.0/std/hint/fn.unreachable_unchecked.html#safety` + — reaching `unreachable_unchecked` is UB; applicable to Rust 1.80.0 without a + target/profile qualification relevant here. + +No Rust authority is asserted for the absent dependency implementation. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/R.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/R.md new file mode 100644 index 0000000000..a2d6562f7e --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/R.md @@ -0,0 +1,100 @@ +# Mode R Frozen Blind-Scoring Rubric + +> **Evaluator-only material. Never expose this file to a report agent.** + +> Evaluator-only material. Never expose this file to a report agent. + +## Exact result + +Within the scope in `REQUEST.md`, valid safe use includes every downstream safe +implementation of `Slot`, not only `Tail`. An implementation returning 2 and a +safe call `increment::(&mut [0, 0])` reaches +`get_unchecked_mut(2)` on a length-2 slice. Its in-bounds precondition is false, +and Rust 1.82.0 documents an out-of-bounds call as UB. The current safe API is +therefore `UNSOUND`. + +The required behavior needs neither generic index metadata nor an unsafe +abstraction. The preferred design is a nongeneric safe function that updates +`pair[1]` with wrapping addition (or an exactly behavior-equivalent checked +specialization). It eliminates caller implementation/index capability and the +unsafe block. Making `Slot` unsafe, adding safe-trait prose, or preserving the +generic abstraction by default is less parsimonious because downstream generic +use is expressly not required. The proposal does not alter the current verdict +and requires implementation plus fresh audit. + +## Atoms + +- **R1:** Quantify over arbitrary safe `Slot` implementations and explicitly + construct or describe one returning 2 together with its valid safe call. + - `scope_basis`: public safe trait implementation and safe generic calls are + valid-use surfaces of the exact current source. + - `dependencies`: exact Rust 1.82.0 public-visibility, trait-implementation, + ordinary-versus-unsafe-trait, and safe-versus-unsafe-function rules. +- **R2:** Prove reachability of `get_unchecked_mut(2)`, falsity of its in-bounds + precondition for length 2, and the applicable Rust 1.82.0 UB consequence. + - `scope_basis`: necessary certificate for the current-artifact soundness + claim requested in `REQUEST.md`. + - `dependencies`: R1. +- **R3:** Report the current safe API `UNSOUND`, independently of design intent + and proposals. + - `scope_basis`: `REQUEST.md` expressly requires the literal current-artifact + verdict before redesign. + - `dependencies`: R2. +- **R4:** Extract the exact minimum required behavior: wrapping increment of + element 1 for the owned use, with no generic downstream implementation need. + - `scope_basis`: this requirement is stated verbatim in `REQUEST.md`. + - `dependencies`: none. +- **R5:** Propose a nongeneric safe specialization implementing R4 with checked + indexing or an exactly equivalent safe operation, and no caller-controlled + implementation or index capability. + - `scope_basis`: `REQUEST.md` asks for the most parsimonious provable + abstraction satisfying R4. + - `dependencies`: R4. +- **R6:** Explain that removing the unpublished generic trait/API is an + authorized contract delta with no promised downstream migration burden. + - `scope_basis`: `REQUEST.md` expressly permits this contract and + representation change. + - `dependencies`: R4, R5. +- **R7:** Keep the redesign conditional and require implementation plus fresh + audit; do not use it to narrow or alter R3. + - `scope_basis`: `REQUEST.md` requires the current verdict and post-change + audit independently. + - `dependencies`: R3, R5. + +Merely making the trait unsafe does not pass R5 because it preserves an +unneeded caller implementation capability. + +## Hard errors + +- **RH1:** Trust `Tail` as the only safe implementation. +- **RH2:** Treat safe-trait prose as a caller obligation. +- **RH3:** Call the current API sound or merely `UNPROVED` after closing R2. +- **RH4:** Certify an unimplemented redesign. +- **RH5:** Let design intent narrow the current contract. + +## Checked authority + +- Rust 1.82.0 Reference, public visibility: + `https://doc.rust-lang.org/1.82.0/reference/visibility-and-privacy.html#visibility-and-privacy` + — public items are accessible from outside their module, subject to the + containing path; every item here is at the public crate root. +- Rust 1.82.0 Reference, trait implementations and coherence: + `https://doc.rust-lang.org/1.82.0/reference/items/implementations.html#trait-implementations` + — a trait implementation may be defined when the implementing type is local, + so a downstream crate may define local `Bad` and implement `Slot` for it. +- Rust 1.82.0 Reference, unsafe traits: + `https://doc.rust-lang.org/1.82.0/reference/items/traits.html#unsafe-traits` + — unsafe traits are explicitly declared with `unsafe`; the supplied `Slot` + declaration is an ordinary safe trait and its implementation requires no + unsafe act. +- Rust 1.82.0 Reference, unsafe functions: + `https://doc.rust-lang.org/1.82.0/reference/unsafe-keyword.html#unsafe-functions-unsafe-fn` + — the caller-side unsafe obligation attaches to an `unsafe fn`; the supplied + public `increment` is an ordinary safe function. +- Rust 1.82.0 slice `get_unchecked_mut`: + `https://doc.rust-lang.org/1.82.0/std/primitive.slice.html#method.get_unchecked_mut` + — the index must be in bounds; an out-of-bounds call is UB even if its result + is unused. +- Rust 1.82.0 `u32::wrapping_add`: + `https://doc.rust-lang.org/1.82.0/std/primitive.u32.html#method.wrapping_add` + — modular wrapping addition supplies the requested update semantics. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/SCORER.md b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/SCORER.md new file mode 100644 index 0000000000..eb79798cd6 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/rubrics/SCORER.md @@ -0,0 +1,84 @@ +# Frozen Common Blind-Scoring Rules + +Score explicit propositions, not keywords, terminology, organization, or a +preferred proof style. Accept any equivalently exact derivation. Do not infer a +missing material premise from vague shorthand. + +For each report A–J: + +1. Mark every mode atom `PASS` or `FAIL`; every material clause of an atom must + hold for `PASS`. +2. Record every independently applicable hard error. +3. Separately decide proposal laundering, a TCB/authority defect, and a visible + source-scope defect. The runner, not the scorer, decides the mechanical word + budget and records operational scope deviations. +4. Cite compact report evidence for every decision. +5. Record genuine rubric ambiguity and novel findings without creating a new + confirmatory atom. A novel finding is a concrete, potentially gate-relevant + material defect in the report which the frozen atoms and defect fields do + not already represent; do not use it for praise, style, or harmless extra + detail. + +`Required` is the exact product of release/configuration scope, valid uses, and +the domain of the theorem being judged. `Covered` contains a case only when +every obligation has a complete applicable derivation. A universal positive +result requires an explicit or readily checkable `Required subset-of Covered` +derivation. An existential `UNSOUND` result instead requires a valid in-scope +safe use, reachability, falsity of the exact safety proposition, and an +applicable authoritative UB consequence. + +Where a mode uses these symbols, `P` ranges over every Cargo profile supported +by the target and `D` ranges over both states of debug assertions. + +A missed atom is not automatically a hard error. Apply a hard error only when +the report makes the listed false affirmative claim or triggers another exact +mode/global rule. Additional correct regional detail is harmless. + +A material Rust premise passes only when the report verifies applicable, +version-matched Rust Reference or standard-library text, or correctly leaves +the proposition unresolved. The URL allowlist grants page identity, not a +proposition. Cargo documentation is not a Rust semantic axiom. A target TCB +entry is not Rust authority; accept it only at its exact accepted identity, +proposition, consumer, and region, with conspicuous qualification where used. + +Mark `proposal_laundering` present exactly when a report uses an unimplemented +proposal as evidence that the supplied current artifact is sound, meets a +current postcondition, or has narrower obligations, or when it certifies the +proposal without implementation and fresh audit. Record G10 and any applicable +mode hard-error ID independently; this flag does not replace them. + +Mark `visible_scope_defect` present only when the report itself establishes a +material departure from the source-review scope supplied to its author. Merely +naming, quoting, hashing, or reporting the supplied package, its `SKILL.md`, or +the skill revision/digest visible inside that package is in scope and is not a +defect. Do not infer prohibited access from writing style, report similarity, +or a guessed generating condition. Record G12 independently when the report +actually evidences inspection of the specifically prohibited evaluation +materials listed there. + +Global hard errors have IDs G1 through G12 in the list order below. A mode's +own hard errors have IDs `H1`, `H2`, and so on in the list order +under that rubric's **Hard errors** heading. Record an exact applicable ID, not +a report-authored category name. Global hard errors are: + +- claiming full `PROVED` after contracting or failing to justify `Required`; +- claiming positive closure without `Required subset-of Covered`; +- inventing policy precedence or calling a conservative audit domain the + project's promise; +- treating CI, endpoints, sparse documentation, a cutoff, or a stability badge + as interval coverage; +- issuing `UNSOUND` without all four existential certificate links; +- issuing only `UNPROVED` after explicitly establishing all four links; +- issuing `CONTRACT-BROKEN` using only an execution that contains UB; +- claiming an observation remains guaranteed because it occurs before UB; +- silently trusting a third-party unsafe implementation or caller-controlled + safe behavior; +- certifying an unimplemented design or using it to narrow the current-artifact + obligation; +- necessarily relying on unchecked, invalid, or inapplicable authority; or +- evidence that the report agent inspected prohibited oracle, sibling, + condition-map, prior-report, or evaluator material. + +Do not identify, cluster, or speculate about generating conditions. Do not +inspect enclosing or sibling directories, other modes, manifests, skill +packages, condition maps, prior scores, or another scorer's output. diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/adjudication-cells.schema.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/adjudication-cells.schema.json new file mode 100644 index 0000000000..4123fbe126 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/adjudication-cells.schema.json @@ -0,0 +1,40 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "unsafe-rust-v4-focused-adjudication-cells-v1", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "mode", "cells"], + "properties": { + "schema_version": {"const": 1}, + "mode": {"enum": ["P", "B", "L", "Q", "R"]}, + "cells": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["label", "field", "s1", "s2"], + "properties": { + "label": {"type": "string", "pattern": "^[A-J]$"}, + "field": { + "type": "string", + "pattern": "^(atom:[PBLQR][1-9][0-9]*|hard_error:(G[1-9][0-9]*|[PBLQR]H[1-9][0-9]*)|proposal_laundering|tcb_authority_defect|visible_scope_defect|novel:s[12]:N[1-9][0-9]*)$" + }, + "s1": {"$ref": "#/$defs/position"}, + "s2": {"$ref": "#/$defs/position"}, + "consistency_review": {"$ref": "#/$defs/position"} + } + } + } + }, + "$defs": { + "position": { + "type": "object", + "additionalProperties": false, + "required": ["decision", "evidence"], + "properties": { + "decision": {"enum": ["PASS", "FAIL", "PRESENT", "ABSENT"]}, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + } +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/adjudication.schema.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/adjudication.schema.json new file mode 100644 index 0000000000..a0e421247c --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/adjudication.schema.json @@ -0,0 +1,35 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "unsafe-rust-v4-focused-blind-adjudication-v1", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "mode", "decisions", "ambiguities"], + "properties": { + "schema_version": {"const": 1}, + "mode": {"enum": ["P", "B", "L", "Q", "R"]}, + "decisions": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["label", "field", "decision", "evidence"], + "properties": { + "label": {"type": "string", "pattern": "^[A-J]$"}, + "field": { + "type": "string", + "pattern": "^(atom:[PBLQR][1-9][0-9]*|hard_error:(G[1-9][0-9]*|[PBLQR]H[1-9][0-9]*)|proposal_laundering|tcb_authority_defect|visible_scope_defect|novel:s[12]:N[1-9][0-9]*)$" + }, + "decision": { + "enum": ["PASS", "FAIL", "PRESENT", "ABSENT"] + }, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + }, + "ambiguities": { + "type": "array", + "items": {"type": "string", "pattern": "\\S"} + } + } +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/consistency.schema.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/consistency.schema.json new file mode 100644 index 0000000000..04f2f75ba5 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/consistency.schema.json @@ -0,0 +1,72 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "unsafe-rust-v4-focused-consistency-review-v1", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "mode", + "atom_family_attestations", + "defect_family_attestations", + "challenges", + "ambiguities" + ], + "properties": { + "schema_version": {"const": 1}, + "mode": {"enum": ["P", "B", "L", "Q", "R"]}, + "atom_family_attestations": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["atom_id", "all_ten_reports_compared", "evidence"], + "properties": { + "atom_id": {"type": "string", "pattern": "^[PBLQR][1-9][0-9]*$"}, + "all_ten_reports_compared": {"const": true}, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + }, + "defect_family_attestations": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["field", "all_ten_reports_compared", "evidence"], + "properties": { + "field": { + "type": "string", + "pattern": "^(hard_error:(G[1-9][0-9]*|[PBLQR]H[1-9][0-9]*)|proposal_laundering|tcb_authority_defect|visible_scope_defect)$" + }, + "all_ten_reports_compared": {"const": true}, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + }, + "challenges": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["label", "field", "recommended_decision", "evidence"], + "properties": { + "label": {"type": "string", "pattern": "^[A-J]$"}, + "field": { + "type": "string", + "pattern": "^(atom:[PBLQR][1-9][0-9]*|hard_error:(G[1-9][0-9]*|[PBLQR]H[1-9][0-9]*)|proposal_laundering|tcb_authority_defect|visible_scope_defect)$" + }, + "recommended_decision": { + "enum": ["PASS", "FAIL", "PRESENT", "ABSENT"] + }, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + }, + "ambiguities": { + "type": "array", + "items": {"type": "string", "pattern": "\\S"} + } + } +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/event.schema.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/event.schema.json new file mode 100644 index 0000000000..c792997cb8 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/event.schema.json @@ -0,0 +1,27 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "unsafe-rust-v4-focused-event-v1", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "sequence", "previous_event_sha256", "time_utc", "phase", "event", "details"], + "properties": { + "schema_version": {"const": 1}, + "sequence": {"type": "integer", "minimum": 1}, + "previous_event_sha256": { + "oneOf": [ + {"type": "null"}, + {"type": "string", "pattern": "^[0-9a-f]{64}$"} + ] + }, + "time_utc": {"type": "string", "format": "date-time"}, + "phase": { + "enum": ["freeze", "collection", "scoring", "consistency", "adjudication", "unblinding", "result"] + }, + "event": {"type": "string", "minLength": 1}, + "run_id": {"type": "string"}, + "attempt": {"type": "integer", "minimum": 1}, + "agent_id": {"type": "string"}, + "sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"}, + "details": {"type": "object"} + } +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/score.schema.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/score.schema.json new file mode 100644 index 0000000000..ab7adee463 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/freeze/schemas/score.schema.json @@ -0,0 +1,91 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "unsafe-rust-v4-focused-blind-score-v1", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "mode", "scorer_id", "reports", "ambiguities"], + "properties": { + "schema_version": {"const": 1}, + "mode": {"enum": ["P", "B", "L", "Q", "R"]}, + "scorer_id": {"enum": ["s1", "s2"]}, + "reports": { + "type": "array", + "minItems": 10, + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "label", + "atoms", + "hard_errors", + "proposal_laundering", + "tcb_authority_defect", + "visible_scope_defect", + "novel_findings" + ], + "properties": { + "label": {"type": "string", "pattern": "^[A-J]$"}, + "atoms": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["id", "decision", "evidence"], + "properties": { + "id": {"type": "string", "pattern": "^[PBLQR][1-9][0-9]*$"}, + "decision": {"enum": ["PASS", "FAIL"]}, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + }, + "hard_errors": { + "type": "array", + "items": {"$ref": "#/$defs/hard_error"} + }, + "proposal_laundering": {"$ref": "#/$defs/flag"}, + "tcb_authority_defect": {"$ref": "#/$defs/flag"}, + "visible_scope_defect": {"$ref": "#/$defs/flag"}, + "novel_findings": { + "type": "array", + "items": {"$ref": "#/$defs/novel_finding"} + } + } + } + }, + "ambiguities": { + "type": "array", + "items": {"type": "string", "pattern": "\\S"} + } + }, + "$defs": { + "hard_error": { + "type": "object", + "additionalProperties": false, + "required": ["id", "evidence"], + "properties": { + "id": {"type": "string", "pattern": "^(G[1-9][0-9]*|[PBLQR]H[1-9][0-9]*)$"}, + "evidence": {"type": "string", "pattern": "\\S"} + } + }, + "novel_finding": { + "type": "object", + "additionalProperties": false, + "required": ["id", "evidence"], + "properties": { + "id": {"type": "string", "pattern": "^N[1-9][0-9]*$"}, + "evidence": {"type": "string", "pattern": "\\S"} + } + }, + "flag": { + "type": "object", + "additionalProperties": false, + "required": ["present", "evidence"], + "properties": { + "present": {"type": "boolean"}, + "evidence": {"type": "string", "pattern": "\\S"} + } + } + } +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/operations.lock b/evals/unsafe-rust/runs/2026-08-01-v4-focused/operations.lock new file mode 100644 index 0000000000..e69de29bb2 diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/prepare.py b/evals/unsafe-rust/runs/2026-08-01-v4-focused/prepare.py new file mode 100644 index 0000000000..2086db32f4 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/prepare.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""Generate and verify the frozen V4 focused-evaluation artifacts.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +from pathlib import Path + + +RUN = Path(__file__).resolve().parent +FREEZE = RUN / "freeze" +SEALED = RUN / "sealed" +EVALS = RUN.parent.parent + +MODES = ("P", "B", "L", "Q", "R") +CONDITIONS = ("v4", "v3") +REPLICATES = range(1, 6) +ATOM_COUNTS = {"P": 27, "B": 15, "L": 11, "Q": 5, "R": 7} +WORD_CAPS = {"P": 3000, "B": 3200, "L": 2200, "Q": 1800, "R": 1800} + +PACKAGES = { + "v4": { + "path": "frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb", + "tree": "6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb", + "skill": "ad48b3811cf2054be76e4b461a36f63e636afb246c5dd7a75e85756a53b22d83", + }, + "v3": { + "path": "frozen-packages/668f70202c7bc8f23f7f894fb784a9629fd292c7f6fe69ede815b0e4c10137bf", + "tree": "fc486dedde1f82ba232b4492808af85a12b27fa2aa27b1a35a3847b2b89f72e0", + "skill": "0e23f7747cc63014bade7543efaf745e7e9a7e5d6dee2a48c602ef7a3eba091e", + }, +} + +TARGETS = { + "P": ("p_predicates", "2b194a735b69a8904b86baa43791a0ddac9f769ce32e87bf4e759822cb5cd52e"), + "B": ("b_build", "7589027142112e387f990314df7eb1d08e5464448566fd68048eb2a748635bf3"), + "L": ("l_proof", "cc05da115d055febc313edcdf18bae59a6230a63583bf918ad29e89eb06a4266"), + "Q": ("q_quantifiers", "35bb6be0402f9d81918c3afc850dd54cde012865bba90d0cd8d7042d78a582ee"), + "R": ("r_redesign", "d69df1b286abd8f7f8955ac56d702c1910eb596c0bd105b7998e39d7246ca063"), +} + + +def sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def keyed(tag: str, seed: str, value: str) -> str: + return sha256(tag.encode() + b"\0" + bytes.fromhex(seed) + b"\0" + value.encode()) + + +def read_seeds() -> dict[str, str]: + data = json.loads((SEALED / "seeds.json").read_text()) + expected = { + "condition", + "schedule", + "blind", + "presentation", + "scorer", + "consistency", + } + if set(data) != expected: + raise ValueError("unexpected seed keys") + for name, value in data.items(): + if not re.fullmatch(r"[0-9a-f]{64}", value): + raise ValueError(f"invalid {name} seed") + if value == "0" * 64: + raise ValueError(f"zero {name} seed") + if len(set(data.values())) != len(data): + raise ValueError("randomization seeds are not distinct") + return data + + +def extract_rubric(mode: str) -> str: + source = (FREEZE / "oracle" / f"{mode}.md").read_text() + body = source[source.index("\n") + 1 :].lstrip().rstrip() + "\n" + atoms = re.findall(rf"^- \*\*{mode}[0-9]+\b", body, flags=re.MULTILINE) + if len(atoms) != ATOM_COUNTS[mode]: + raise ValueError(f"{mode}: expected {ATOM_COUNTS[mode]} atoms, found {len(atoms)}") + if len(re.findall(r"^ - `scope_basis`:", body, flags=re.MULTILINE)) != ATOM_COUNTS[mode]: + raise ValueError(f"{mode}: every atom must have exactly one scope_basis") + if len(re.findall(r"^ - `dependencies`:", body, flags=re.MULTILINE)) != ATOM_COUNTS[mode]: + raise ValueError(f"{mode}: every atom must have exactly one dependencies entry") + return ( + f"# Mode {mode} Frozen Blind-Scoring Rubric\n\n" + "> **Evaluator-only material. Never expose this file to a report agent.**\n\n" + + body + ) + + +def generated_files(seeds: dict[str, str]) -> dict[Path, str]: + outputs: dict[Path, str] = {} + + for mode in MODES: + outputs[FREEZE / "rubrics" / f"{mode}.md"] = extract_rubric(mode) + + condition_order = sorted(CONDITIONS, key=lambda role: (keyed("condition-v1", seeds["condition"], role), role)) + condition_labels = {role: f"c{index}" for index, role in enumerate(condition_order)} + + mode_order = sorted(MODES, key=lambda mode: (keyed("mode-label-v1", seeds["schedule"], mode), mode)) + mode_labels = {mode: f"m{index}" for index, mode in enumerate(mode_order)} + + condition_rows = ["condition_label\trole\tpackage_path\ttree_sha256\tskill_sha256"] + for role in condition_order: + package = PACKAGES[role] + condition_rows.append( + "\t".join((condition_labels[role], role, package["path"], package["tree"], package["skill"])) + ) + outputs[SEALED / "condition-map.tsv"] = "\n".join(condition_rows) + "\n" + + target_rows = ["target_label\tmode\tsource_path\ttree_sha256\tword_cap"] + for mode in mode_order: + directory, digest = TARGETS[mode] + target_rows.append( + f"{mode_labels[mode]}\t{mode}\tfixtures/v4-focused/{directory}\t{digest}\t{WORD_CAPS[mode]}" + ) + outputs[SEALED / "target-map.tsv"] = "\n".join(target_rows) + "\n" + + wave_order = sorted(REPLICATES, key=lambda rep: (keyed("wave-v1", seeds["schedule"], str(rep)), rep)) + schedule_rows = ["run_id\tcell_id\twave\ttarget_label\tcondition_label\treplicate"] + schedule: list[tuple[str, str, int, str, str, int, str, str]] = [] + run_number = 1 + for wave_index, replicate in enumerate(wave_order, start=1): + cells = [(mode, role, replicate) for mode in MODES for role in CONDITIONS] + cells.sort( + key=lambda cell: ( + keyed("schedule-v1", seeds["schedule"], f"{cell[0]}|{cell[1]}|{cell[2]}"), + cell, + ) + ) + for mode, role, rep in cells: + canonical = f"{mode}|{role}|{rep}" + cell_id = keyed("cell-v1", seeds["schedule"], canonical)[:32] + run_id = f"r{run_number:03d}" + schedule_rows.append( + f"{run_id}\t{cell_id}\t{wave_index}\t{mode_labels[mode]}\t{condition_labels[role]}\t{rep}" + ) + schedule.append((run_id, cell_id, wave_index, mode, role, rep, mode_labels[mode], condition_labels[role])) + run_number += 1 + if len(schedule) != 50 or len({row[1] for row in schedule}) != 50: + raise ValueError("schedule does not contain 50 unique cells") + outputs[SEALED / "launch-schedule.tsv"] = "\n".join(schedule_rows) + "\n" + + blind_rows = ["mode\tlabel\trun_id"] + for mode in MODES: + run_ids = [row[0] for row in schedule if row[3] == mode] + run_ids.sort(key=lambda run_id: (keyed("blind-v1", seeds["blind"], f"{mode}|{run_id}"), run_id)) + for index, run_id in enumerate(run_ids): + blind_rows.append(f"{mode}\t{chr(ord('A') + index)}\t{run_id}") + outputs[SEALED / "blind-map.tsv"] = "\n".join(blind_rows) + "\n" + + presentation_rows = ["claim\tlabels_in_order"] + for mode in MODES: + for scorer in ("s1", "s2"): + labels = [chr(ord("A") + index) for index in range(10)] + labels.sort( + key=lambda label: ( + keyed("presentation-v1", seeds["presentation"], f"{mode}|{scorer}|{label}"), + label, + ) + ) + presentation_rows.append(f"{mode}-{scorer}\t{','.join(labels)}") + outputs[SEALED / "presentation-orders.tsv"] = "\n".join(presentation_rows) + "\n" + + claims = [f"{mode}-{scorer}" for mode in MODES for scorer in ("s1", "s2")] + claims.sort(key=lambda claim: (keyed("scorer-v1", seeds["scorer"], claim), claim)) + outputs[SEALED / "scoring-schedule.tsv"] = "claim\n" + "\n".join(claims) + "\n" + + consistency_claims = sorted( + MODES, + key=lambda mode: ( + keyed("consistency-v1", seeds["consistency"], mode), + mode, + ), + ) + outputs[SEALED / "consistency-schedule.tsv"] = ( + "claim\n" + "\n".join(consistency_claims) + "\n" + ) + + commitments = { + "schema_version": 1, + "algorithm": "sha256(tag_utf8 || NUL || seed_bytes)", + "commitments": { + name: sha256(f"{name}-v1".encode() + b"\0" + bytes.fromhex(seed)) + for name, seed in sorted(seeds.items()) + }, + } + outputs[FREEZE / "randomization" / "commitments.json"] = json.dumps(commitments, indent=2, sort_keys=True) + "\n" + return outputs + + +def validate_allowlists() -> None: + for mode in MODES: + path = FREEZE / "allowlists" / f"{mode}.txt" + lines = path.read_text().splitlines() + if not lines or len(lines) != len(set(lines)): + raise ValueError(f"{mode}: empty or duplicate allowlist") + if any(not re.fullmatch(r"https://doc\.rust-lang\.org/\S+", line) for line in lines): + raise ValueError(f"{mode}: allowlist is not URL-only") + source = (FREEZE / "oracle" / f"{mode}.md").read_text() + oracle_urls = re.findall(r"https://doc\.rust-lang\.org/[^`)\s]+", source) + if lines != oracle_urls: + raise ValueError(f"{mode}: allowlist differs from canonical oracle URL order") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--write", action="store_true", help="write generated artifacts") + args = parser.parse_args() + + if args.write and (FREEZE / "LOCK.json").exists(): + raise SystemExit("refusing to rewrite generated artifacts after freeze lock") + + validate_allowlists() + outputs = generated_files(read_seeds()) + mismatches: list[str] = [] + for path, expected in outputs.items(): + if args.write: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(expected) + elif not path.exists() or path.read_text() != expected: + mismatches.append(str(path.relative_to(RUN))) + if mismatches: + raise SystemExit("generated artifact mismatch:\n" + "\n".join(mismatches)) + print(f"validated {len(outputs)} generated artifacts") + + +if __name__ == "__main__": + main() diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/protocol.py b/evals/unsafe-rust/runs/2026-08-01-v4-focused/protocol.py new file mode 100644 index 0000000000..9765530172 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/protocol.py @@ -0,0 +1,4545 @@ +#!/usr/bin/env python3 +"""Frozen mechanics for the V4 focused evaluation. + +This program validates immutable inputs, prepares neutral report cells, builds +blind packets, obtains dual scores and a condition-blind consistency review, +and aggregates only after every required adjudication. It never runs or builds +a target. +""" + +from __future__ import annotations + +import argparse +import csv +import fcntl +import hashlib +import json +import os +import re +import shutil +import stat +import sys +import tempfile +import io +from collections import Counter, defaultdict +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import prepare + + +RUN = Path(__file__).resolve().parent +FREEZE = RUN / "freeze" +SEALED = RUN / "sealed" +COLLECTION = RUN / "collection" +SCORING = RUN / "scoring" +RESULTS = RUN / "results" +EVALS = RUN.parent.parent +MODES = prepare.MODES +CONDITIONS = prepare.CONDITIONS +REPLICATES = tuple(prepare.REPLICATES) +REPORTS_PER_MODE = len(CONDITIONS) * len(REPLICATES) +TOTAL_REPORTS = len(MODES) * REPORTS_PER_MODE +LABELS = tuple(chr(ord("A") + index) for index in range(REPORTS_PER_MODE)) +SCORERS = ("s1", "s2") +EVALUATOR_KINDS = ("scorer", "consistency", "adjudicator") +GLOBAL_DEFECT_FIELDS = ( + "proposal_laundering", + "tcb_authority_defect", + "visible_scope_defect", +) +GLOBAL_HARD_ERROR_COUNT = 12 +INFRA_FAILURE_CODES = { + "SERVICE_ERROR_BEFORE_OUTPUT", + "ORCHESTRATOR_TOOL_FAILURE", + "FILESYSTEM_FAILURE", +} +TERMINAL_REPORT_FAILURE_CODES = { + "REFUSAL", + "TIMEOUT_AFTER_WORK", + "INVALID_OUTPUT", + "SEMANTIC_NONCOMPLETION", +} +FILE_MANIFEST = FREEZE / "file-manifest.sha256" +LOCK = FREEZE / "LOCK.json" +OPERATION_LOCK = RUN / "operations.lock" +EVENT_PHASES = { + "freeze", + "collection", + "scoring", + "consistency", + "adjudication", + "unblinding", + "result", +} +_AUTHENTICATED_FILE_DIGESTS: dict[str, str] | None = None + + +def evaluator_phase(kind: str) -> str: + phases = { + "scorer": "scoring", + "consistency": "consistency", + "adjudicator": "adjudication", + } + try: + return phases[kind] + except KeyError as error: + raise ValueError(f"unknown evaluator kind: {kind}") from error + + +def evaluator_source_packet(kind: str, identity: str) -> Path: + if kind == "scorer": + mode, scorer = identity.split("-", 1) + if mode not in MODES or scorer not in SCORERS: + raise ValueError(f"invalid scorer identity: {identity}") + return SCORING / "packets" / mode / scorer + if identity not in MODES: + raise ValueError(f"invalid {kind} identity: {identity}") + if kind == "consistency": + return SCORING / "consistency-packets" / identity + if kind == "adjudicator": + return SCORING / "adjudication-packets" / identity + raise ValueError(f"unknown evaluator kind: {kind}") + + +def evaluator_prompt_name(kind: str) -> str: + return { + "scorer": "scorer.md", + "consistency": "consistency.md", + "adjudicator": "adjudicator.md", + }[kind] + + +def evaluator_completion_event(kind: str) -> str: + return { + "scorer": "score_preserved", + "consistency": "consistency_review_preserved", + "adjudicator": "adjudication_preserved", + }[kind] + + +def evaluator_completion_matches( + event: dict[str, Any], kind: str, identity: str +) -> bool: + if event.get("event") != evaluator_completion_event(kind): + return False + details = event.get("details", {}) + if kind == "scorer": + return f"{details.get('mode')}-{details.get('scorer')}" == identity + return details.get("mode") == identity + + +def utc_now() -> str: + return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") + + +def positive_int(value: str) -> int: + parsed = int(value) + if parsed < 1: + raise argparse.ArgumentTypeError("value must be a positive integer") + return parsed + + +def acquire_operation_lock() -> Any: + OPERATION_LOCK.parent.mkdir(parents=True, exist_ok=True) + handle = OPERATION_LOCK.open("a+", encoding="utf-8") + fcntl.flock(handle, fcntl.LOCK_EX) + return handle + + +def sha256_bytes(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def sha256_file(path: Path) -> str: + return sha256_bytes(path.read_bytes()) + + +def json_dump(value: Any) -> str: + return json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n" + + +def is_nonblank_string(value: Any) -> bool: + return isinstance(value, str) and bool(value.strip()) + + +def read_tsv(path: Path) -> list[dict[str, str]]: + with path.open(newline="") as file: + return list(csv.DictReader(file, dialect="excel-tab")) + + +def read_frozen_tsv(path: Path) -> list[dict[str, str]]: + data = read_frozen_bytes(path) + return list(csv.DictReader(io.StringIO(data.decode()), dialect="excel-tab")) + + +def read_frozen_bytes(path: Path) -> bytes: + data = path.read_bytes() + if FILE_MANIFEST.exists() and sha256_bytes(data) != frozen_file_digest(path): + raise ValueError(f"frozen input changed while being read: {path.relative_to(RUN)}") + return data + + +def read_frozen_text(path: Path) -> str: + return read_frozen_bytes(path).decode() + + +def write_once(path: Path, content: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("x", encoding="utf-8", newline="") as file: + file.write(content) + file.flush() + os.fsync(file.fileno()) + + +def write_bytes_once(path: Path, content: bytes) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("xb") as file: + file.write(content) + file.flush() + os.fsync(file.fileno()) + + +def load_schedule() -> dict[str, dict[str, str]]: + rows = read_frozen_tsv(SEALED / "launch-schedule.tsv") + return {row["run_id"]: row for row in rows} + + +def load_condition_map() -> dict[str, dict[str, str]]: + return { + row["condition_label"]: row + for row in read_frozen_tsv(SEALED / "condition-map.tsv") + } + + +def load_target_map() -> dict[str, dict[str, str]]: + return { + row["target_label"]: row for row in read_frozen_tsv(SEALED / "target-map.tsv") + } + + +def load_blind_map() -> dict[str, dict[str, str]]: + by_mode: dict[str, dict[str, str]] = {mode: {} for mode in MODES} + for row in read_frozen_tsv(SEALED / "blind-map.tsv"): + by_mode[row["mode"]][row["label"]] = row["run_id"] + return by_mode + + +def load_frozen_seeds() -> dict[str, str]: + data = json.loads(read_frozen_text(SEALED / "seeds.json")) + expected = { + "condition", + "schedule", + "blind", + "presentation", + "scorer", + "consistency", + } + if not isinstance(data, dict) or set(data) != expected: + raise ValueError("unexpected frozen seed keys") + if any( + not isinstance(value, str) + or not re.fullmatch(r"[0-9a-f]{64}", value) + or value == "0" * 64 + for value in data.values() + ): + raise ValueError("invalid frozen randomization seed") + if len(set(data.values())) != len(data): + raise ValueError("frozen randomization seeds are not distinct") + return data + + +def mode_for_run(run_id: str) -> str: + row = load_schedule()[run_id] + return load_target_map()[row["target_label"]]["mode"] + + +def atom_ids(mode: str) -> tuple[str, ...]: + text = read_frozen_text(FREEZE / "rubrics" / f"{mode}.md") + atoms = tuple(re.findall(rf"^- \*\*({mode}[1-9][0-9]*)\b", text, flags=re.MULTILINE)) + if len(atoms) != prepare.ATOM_COUNTS[mode] or len(set(atoms)) != len(atoms): + raise ValueError(f"invalid atom inventory for mode {mode}: {atoms}") + return atoms + + +def mode_hard_error_ids(mode: str) -> tuple[str, ...]: + text = read_frozen_text(FREEZE / "rubrics" / f"{mode}.md") + match = re.search( + r"^#{2,6}[^\n]*hard[^\n]*\n(.*?)(?=^#{1,6} |\Z)", + text, + flags=re.MULTILINE | re.DOTALL | re.IGNORECASE, + ) + if not match: + raise ValueError(f"missing hard-error section for mode {mode}") + body = match.group(1) + count = len(re.findall(r"^- ", body, flags=re.MULTILINE)) + if not count: + raise ValueError(f"empty hard-error section for mode {mode}") + return tuple(f"{mode}H{index}" for index in range(1, count + 1)) + + +def hard_error_ids(mode: str) -> tuple[str, ...]: + return tuple(f"G{index}" for index in range(1, GLOBAL_HARD_ERROR_COUNT + 1)) + mode_hard_error_ids(mode) + + +def byte_tree_digest(path: Path) -> str: + if path.is_symlink() or not path.is_dir(): + raise ValueError(f"byte-tree root is not a real directory: {path}") + records: list[bytes] = [] + for item in sorted(path.rglob("*"), key=lambda value: value.relative_to(path).as_posix()): + relative = item.relative_to(path).as_posix() + if item.is_symlink() or not (item.is_dir() or item.is_file()): + raise ValueError(f"unsupported runtime entry: {item}") + if item.is_dir(): + records.append(f"d\0{relative}\n".encode()) + else: + data = item.read_bytes() + records.append( + f"f\0{relative}\0{len(data)}\0{sha256_bytes(data)}\n".encode() + ) + return sha256_bytes(b"".join(records)) + + +def lock_input_paths() -> list[Path]: + paths: list[Path] = [] + for name in ("prepare.py", "fetch_authority.py", "protocol.py"): + paths.append(RUN / name) + for root in (FREEZE, SEALED): + for path in root.rglob("*"): + if not path.is_file(): + continue + if path in {FILE_MANIFEST, LOCK} or "__pycache__" in path.parts: + continue + paths.append(path) + paths.sort(key=lambda path: path.relative_to(RUN).as_posix()) + if any(path.is_symlink() for path in paths): + raise ValueError("symlinks are prohibited in lock inputs") + return paths + + +def render_file_manifest() -> str: + return "".join( + f"{sha256_file(path)} {path.relative_to(RUN).as_posix()}\n" + for path in lock_input_paths() + ) + + +def verify_file_manifest() -> None: + if not FILE_MANIFEST.exists(): + raise ValueError("missing file-manifest.sha256") + expected = render_file_manifest() + actual = FILE_MANIFEST.read_text() + if actual != expected: + raise ValueError("freeze file manifest does not match current inputs") + + +def parse_file_manifest(data: bytes) -> dict[str, str]: + entries: dict[str, str] = {} + for line in data.decode().splitlines(): + digest, separator, name = line.partition(" ") + if ( + not separator + or not name + or name in entries + or not re.fullmatch(r"[0-9a-f]{64}", digest) + ): + raise ValueError("invalid frozen file-manifest row") + entries[name] = digest + return entries + + +def frozen_file_digest(path: Path) -> str: + relative = path.relative_to(RUN).as_posix() + if _AUTHENTICATED_FILE_DIGESTS is not None: + entries = _AUTHENTICATED_FILE_DIGESTS + else: + data = FILE_MANIFEST.read_bytes() + if LOCK.exists(): + lock = json.loads(LOCK.read_text()) + if lock.get("file_manifest_sha256") != sha256_bytes(data): + raise ValueError("file manifest is not authenticated by LOCK.json") + entries = parse_file_manifest(data) + if relative not in entries: + raise ValueError(f"path is not pinned by the freeze manifest: {relative}") + return entries[relative] + + +def validate_static(require_lock: bool, *, announce: bool = True) -> None: + global _AUTHENTICATED_FILE_DIGESTS + if MODES != ("P", "B", "L", "Q", "R"): + raise ValueError(f"V4 mode inventory changed: {MODES}") + if CONDITIONS != ("v4", "v3"): + raise ValueError(f"V4 condition inventory changed: {CONDITIONS}") + if len(REPLICATES) != 5 or REPORTS_PER_MODE != 10 or TOTAL_REPORTS != 50: + raise ValueError( + "V4 design must contain five replicates, ten reports per mode, " + "and fifty reports total" + ) + prepare.validate_allowlists() + generated = prepare.generated_files(prepare.read_seeds()) + for path, expected in generated.items(): + if not path.exists() or read_frozen_text(path) != expected: + raise ValueError(f"generated artifact mismatch: {path.relative_to(RUN)}") + + for role, package in prepare.PACKAGES.items(): + path = EVALS / package["path"] + actual = byte_tree_digest(path) + if actual != package["tree"]: + raise ValueError(f"{role} package digest mismatch: {actual}") + skill = sha256_file(path / "SKILL.md") + if skill != package["skill"]: + raise ValueError(f"{role} SKILL.md digest mismatch: {skill}") + target_rows = {row["mode"]: row for row in load_target_map().values()} + for mode, (_directory, expected) in prepare.TARGETS.items(): + actual = byte_tree_digest(EVALS / target_rows[mode]["source_path"]) + if actual != expected: + raise ValueError(f"{mode} target digest mismatch: {actual}") + + schedule_rows = read_frozen_tsv(SEALED / "launch-schedule.tsv") + if len(schedule_rows) != TOTAL_REPORTS: + raise ValueError( + f"expected {TOTAL_REPORTS} schedule rows, found {len(schedule_rows)}" + ) + if len({row["run_id"] for row in schedule_rows}) != TOTAL_REPORTS: + raise ValueError("duplicate run ID") + if len({row["cell_id"] for row in schedule_rows}) != TOTAL_REPORTS: + raise ValueError("duplicate cell ID") + target_map = load_target_map() + condition_map = load_condition_map() + wave_counts: Counter[tuple[str, str, str]] = Counter() + for row in schedule_rows: + mode = target_map[row["target_label"]]["mode"] + role = condition_map[row["condition_label"]]["role"] + wave_counts[(row["wave"], mode, role)] += 1 + expected_wave_counts = Counter( + (str(wave), mode, role) + for wave in range(1, len(REPLICATES) + 1) + for mode in MODES + for role in CONDITIONS + ) + if wave_counts != expected_wave_counts: + raise ValueError("schedule is not five complete balanced waves") + + blind = load_blind_map() + mapped_runs: list[str] = [] + for mode in MODES: + if tuple(sorted(blind[mode])) != LABELS: + raise ValueError(f"{mode} blind labels are incomplete") + for run_id in blind[mode].values(): + if mode_for_run(run_id) != mode: + raise ValueError(f"blind map crosses modes: {mode} {run_id}") + mapped_runs.append(run_id) + atom_ids(mode) + mode_hard_error_ids(mode) + if Counter(mapped_runs) != Counter(row["run_id"] for row in schedule_rows): + raise ValueError("blind map is not a bijection over scheduled runs") + + presentations = read_frozen_tsv(SEALED / "presentation-orders.tsv") + if {row["claim"] for row in presentations} != { + f"{mode}-{scorer}" for mode in MODES for scorer in SCORERS + }: + raise ValueError("presentation claims are incomplete") + for row in presentations: + if tuple(sorted(row["labels_in_order"].split(","))) != LABELS: + raise ValueError(f"invalid presentation order: {row['claim']}") + scoring_schedule = read_frozen_tsv(SEALED / "scoring-schedule.tsv") + expected_scorer_claims = { + f"{mode}-{scorer}" for mode in MODES for scorer in SCORERS + } + if ( + len(scoring_schedule) != len(expected_scorer_claims) + or any(set(row) != {"claim"} for row in scoring_schedule) + or {row["claim"] for row in scoring_schedule} != expected_scorer_claims + ): + raise ValueError("scoring schedule is incomplete or malformed") + consistency_schedule = read_frozen_tsv(SEALED / "consistency-schedule.tsv") + if ( + len(consistency_schedule) != len(MODES) + or any(set(row) != {"claim"} for row in consistency_schedule) + or {row["claim"] for row in consistency_schedule} != set(MODES) + ): + raise ValueError("consistency-review schedule is incomplete or malformed") + + for schema in (FREEZE / "schemas").glob("*.json"): + json.loads(read_frozen_text(schema)) + + authority = FREEZE / "authority-manifest.tsv" + if not authority.exists(): + raise ValueError("missing authority-manifest.tsv") + authority_rows = read_frozen_tsv(authority) + allowlist_pairs = [ + (mode, url) + for mode in MODES + for url in read_frozen_text(FREEZE / "allowlists" / f"{mode}.txt").splitlines() + ] + if [(row["mode"], row["requested_url"]) for row in authority_rows] != allowlist_pairs: + raise ValueError("authority manifest does not match exact allowlist sequence") + if any( + row["status"] != "200" + or row["fragment_found"] != "true" + or not re.fullmatch(r"[0-9a-f]{64}", row["sha256"]) + for row in authority_rows + ): + raise ValueError("authority manifest has an invalid retrieval record") + validate_event_ledger() + validate_preserved_artifacts() + + if require_lock: + verify_file_manifest() + lock = json.loads(LOCK.read_text()) + if set(lock) != { + "schema_version", + "status", + "file_manifest_sha256", + "review_signoffs", + "unresolved_ambiguities", + "reports_collected_before_lock", + "locked_utc", + }: + raise ValueError("freeze lock has unexpected fields") + if type(lock.get("schema_version")) is not int or lock["schema_version"] != 1 or lock.get("status") != "FROZEN": + raise ValueError("invalid freeze lock") + locked_utc = lock.get("locked_utc") + if not isinstance(locked_utc, str): + raise ValueError("invalid freeze lock timestamp") + try: + locked_time = datetime.fromisoformat(locked_utc.replace("Z", "+00:00")) + except ValueError as error: + raise ValueError("invalid freeze lock timestamp") from error + if locked_time.tzinfo is None: + raise ValueError("freeze lock timestamp lacks timezone") + ambiguities = lock.get("unresolved_ambiguities") + if not isinstance(ambiguities, list) or any( + not is_nonblank_string(ambiguity) for ambiguity in ambiguities + ): + raise ValueError("invalid freeze-lock ambiguity inventory") + root = sha256_file(FILE_MANIFEST) + if lock.get("file_manifest_sha256") != root: + raise ValueError("freeze lock root does not match file manifest") + signoffs = lock.get("review_signoffs", []) + if not isinstance(signoffs, list) or len(signoffs) < 2 or any( + not isinstance(signoff, dict) for signoff in signoffs + ): + raise ValueError("freeze lock lacks two review signoffs") + reviewer_ids = [signoff.get("reviewer_id") for signoff in signoffs] + if any( + not isinstance(reviewer_id, str) + or not reviewer_id + or reviewer_id != reviewer_id.strip() + for reviewer_id in reviewer_ids + ): + raise ValueError("freeze lock has an empty reviewer ID") + if len(set(reviewer_ids)) != len(signoffs): + raise ValueError("freeze lock reviewer IDs are not distinct") + for signoff in signoffs: + if set(signoff) != { + "reviewer_id", + "verdict", + "file_manifest_sha256", + "scope", + "independent_non_author", + "reviewed_utc", + }: + raise ValueError("freeze review signoff has unexpected fields") + scope = signoff.get("scope") + reviewed_utc = signoff.get("reviewed_utc") + if ( + signoff.get("verdict") != "PASS/FREEZE" + or signoff.get("file_manifest_sha256") != root + or signoff.get("independent_non_author") is not True + or not isinstance(scope, str) + or not scope.strip() + or scope != scope.strip() + or not isinstance(reviewed_utc, str) + or not reviewed_utc.strip() + or reviewed_utc != reviewed_utc.strip() + ): + raise ValueError("invalid freeze review signoff") + try: + reviewed_time = datetime.fromisoformat(reviewed_utc.replace("Z", "+00:00")) + except ValueError as error: + raise ValueError("invalid freeze review timestamp") from error + if reviewed_time.tzinfo is None: + raise ValueError("freeze review timestamp lacks timezone") + if reviewed_time > locked_time: + raise ValueError("freeze review timestamp follows lock timestamp") + if ( + type(lock.get("reports_collected_before_lock")) is not int + or lock["reports_collected_before_lock"] != 0 + ): + raise ValueError("freeze lock does not attest zero prior reports") + freeze_events = [event for event in event_records() if event["event"] == "freeze_locked"] + if len(freeze_events) > 1: + raise ValueError("multiple freeze-lock events") + if freeze_events: + freeze_sequence = freeze_events[0]["sequence"] + if any( + event["sequence"] < freeze_sequence and event["phase"] != "freeze" + for event in event_records() + ): + raise ValueError("evaluation activity predates the freeze-lock event") + elif any( + path.is_file() + for root in (COLLECTION, SCORING, RESULTS) + if root.exists() + for path in root.rglob("*") + ): + raise ValueError("evaluation artifacts exist before the freeze-lock event") + _AUTHENTICATED_FILE_DIGESTS = parse_file_manifest(FILE_MANIFEST.read_bytes()) + if "**Preregistration status:** FROZEN." not in read_frozen_text( + FREEZE / "plan.md" + ): + raise ValueError("plan is not marked FROZEN") + if announce: + print("static protocol validation passed") + + +def make_read_only(path: Path) -> None: + for item in sorted(path.rglob("*"), reverse=True): + if item.is_file(): + os.utime(item, (0, 0), follow_symlinks=False) + item.chmod(0o444) + elif item.is_dir(): + os.utime(item, (0, 0), follow_symlinks=False) + item.chmod(0o555) + os.utime(path, (0, 0), follow_symlinks=False) + path.chmod(0o555) + + +def resolve_run(run_id: str) -> tuple[dict[str, str], dict[str, str], dict[str, str]]: + schedule = load_schedule() + if run_id not in schedule: + raise ValueError(f"unknown run ID: {run_id}") + row = schedule[run_id] + target = load_target_map()[row["target_label"]] + condition = load_condition_map()[row["condition_label"]] + return row, target, condition + + +def event_records() -> list[dict[str, Any]]: + path = RUN / "events.jsonl" + return [json.loads(line) for line in path.read_text().splitlines()] if path.exists() else [] + + +def assert_run_not_invalid() -> None: + terminal_invalid_event = any( + event["event"] in {"invalid_output_preserved", "run_invalidated"} + for event in event_records() + ) + invalid_marker = RUN / "INVALID.json" + if invalid_marker.exists() or invalid_marker.is_symlink() or terminal_invalid_event: + raise SystemExit("run is INVALID; no further evaluation command is permitted") + + +def assert_authority_verification_allowed(wave: int) -> None: + if wave not in range(1, len(REPLICATES) + 1): + raise ValueError(f"invalid collection wave: {wave}") + events = event_records() + if any( + event["event"] == "authority_verified" + and event.get("details", {}).get("wave") == wave + for event in events + ): + raise ValueError(f"wave {wave} authority verification was already recorded") + schedule_rows = read_frozen_tsv(SEALED / "launch-schedule.tsv") + completed = {event.get("run_id") for event in events if event["event"] == "report_preserved"} + required_completed = { + row["run_id"] for row in schedule_rows if int(row["wave"]) < wave + } + if not required_completed <= completed: + raise ValueError(f"earlier collection waves are incomplete before wave {wave}") + prepared = {event.get("run_id") for event in events if event["event"] == "cell_prepared"} + current = {row["run_id"] for row in schedule_rows if int(row["wave"]) == wave} + if prepared & current: + raise ValueError(f"wave {wave} authority verification must precede cell preparation") + + +def assert_prepare_allowed(run_id: str) -> None: + schedule_rows = read_frozen_tsv(SEALED / "launch-schedule.tsv") + order = [row["run_id"] for row in schedule_rows] + position = order.index(run_id) + row = schedule_rows[position] + events = event_records() + authority_events = [ + event + for event in events + if event["event"] == "authority_verified" + and event.get("details", {}).get("wave") == int(row["wave"]) + and event.get("sha256") == sha256_file(FREEZE / "authority-manifest.tsv") + ] + if len(authority_events) != 1: + raise ValueError(f"wave {row['wave']} lacks one current authority verification") + prepared = {event.get("run_id") for event in events if event["event"] == "cell_prepared"} + completed = {event.get("run_id") for event in events if event["event"] == "report_preserved"} + if run_id in prepared: + raise ValueError(f"cell already prepared: {run_id}") + earlier_in_wave = { + prior["run_id"] + for prior in schedule_rows[:position] + if prior["wave"] == row["wave"] + } + if not earlier_in_wave <= prepared: + raise ValueError(f"prepare order violation before {run_id}") + earlier_waves = { + prior["run_id"] for prior in schedule_rows if int(prior["wave"]) < int(row["wave"]) + } + if not earlier_waves <= completed: + raise ValueError(f"wave barrier violation before {run_id}") + if len(prepared - completed) >= 3: + raise ValueError(f"three report cells are already active before {run_id}") + + +def record_agent_start(run_id: str, attempt: int, agent_id: str) -> None: + if attempt < 1: + raise ValueError("report attempt must be positive") + if not agent_id: + raise ValueError("agent ID must be nonempty") + events = event_records() + if not any(event["event"] == "cell_prepared" and event.get("run_id") == run_id for event in events): + raise ValueError(f"cell was not prepared: {run_id}") + key = (run_id, attempt) + started = { + (event.get("run_id"), event.get("attempt")) + for event in events + if event["event"] == "agent_started" + } + returned = { + (event.get("run_id"), event.get("attempt")) + for event in events + if event["event"] == "agent_returned" + } + if key in started: + raise ValueError(f"agent attempt already started: {run_id}/{attempt}") + used_agent_ids = { + event.get("agent_id") + for event in events + if event["event"] in {"agent_started", "evaluator_started"} + } + if agent_id in used_agent_ids: + raise ValueError(f"agent ID was already used by an evaluated agent: {agent_id}") + if len(started - returned) >= 3: + raise ValueError("three report agents are already active") + schedule_rows = read_frozen_tsv(SEALED / "launch-schedule.tsv") + position = next(index for index, row in enumerate(schedule_rows) if row["run_id"] == run_id) + current_wave = schedule_rows[position]["wave"] + earlier_in_wave = [ + row["run_id"] for row in schedule_rows[:position] if row["wave"] == current_wave + ] + started_runs = {item[0] for item in started} + if not all(prior in started_runs for prior in earlier_in_wave): + raise ValueError(f"agent launch-order violation before {run_id}") + if attempt > 1 and not any( + event["event"] == "infrastructure_failure" + and event.get("run_id") == run_id + and event.get("attempt") == attempt - 1 + for event in events + ): + raise ValueError("fresh attempt lacks a preceding infrastructure failure") + setup = json.loads((COLLECTION / "setups" / f"{run_id}.json").read_text()) + runtime = Path(setup["runtime_root"]) + verify_runtime(run_id, runtime) + prompt_digest = sha256_bytes(render_report_prompt(run_id, runtime).encode()) + if setup.get("report_prompt_sha256") != prompt_digest: + raise ValueError(f"rendered report prompt changed for {run_id}") + output_entries_now = list((runtime / "output").iterdir()) + if output_entries_now: + raise ValueError(f"report output is not initially empty: {run_id}/{attempt}") + append_event( + "collection", + "agent_started", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + details={ + "model": "gpt-5.6-sol", + "reasoning_effort": "ultra", + "fork_turns": "none", + "prompt_sha256": prompt_digest, + }, + ) + + +def record_prelaunch_failure(run_id: str, evidence: str) -> None: + if not evidence.strip(): + raise ValueError("prelaunch failure requires nonempty evidence") + resolve_run(run_id) + events = event_records() + if not any( + event["event"] == "cell_prepared" and event.get("run_id") == run_id + for event in events + ): + raise ValueError(f"cell was not prepared: {run_id}") + started = [ + event + for event in events + if event["event"] == "agent_started" and event.get("run_id") == run_id + ] + returned_attempts = { + event.get("attempt") + for event in events + if event["event"] == "agent_returned" and event.get("run_id") == run_id + } + if any(event["attempt"] not in returned_attempts for event in started): + raise ValueError("cannot record a prelaunch failure while a report agent is active") + if any( + event["event"] == "report_preserved" and event.get("run_id") == run_id + for event in events + ): + raise ValueError("cannot record a prelaunch failure after canonical completion") + next_attempt = max((event["attempt"] for event in started), default=0) + 1 + append_event( + "collection", + "prelaunch_failure", + run_id=run_id, + details={ + "disposition": "API_NO_AGENT_START", + "next_attempt": next_attempt, + "evidence": evidence, + }, + ) + + +def assert_agent_started(run_id: str, attempt: int, agent_id: str) -> None: + matches = [ + event + for event in event_records() + if event["event"] == "agent_started" + and event.get("run_id") == run_id + and event.get("attempt") == attempt + and event.get("agent_id") == agent_id + ] + if len(matches) != 1: + raise ValueError(f"missing unique agent-start record: {run_id}/{attempt}/{agent_id}") + + +def record_reminder(run_id: str, attempt: int, agent_id: str) -> None: + assert_agent_started(run_id, attempt, agent_id) + events = event_records() + if any( + event["event"] == "reminder_sent" + and event.get("run_id") == run_id + and event.get("attempt") == attempt + for event in events + ): + raise ValueError("the one permitted reminder was already recorded") + if any( + event["event"] == "agent_returned" + and event.get("run_id") == run_id + and event.get("attempt") == attempt + for event in events + ): + raise ValueError("a reminder cannot follow agent return") + start = next( + event + for event in events + if event["event"] == "agent_started" + and event.get("run_id") == run_id + and event.get("attempt") == attempt + ) + started = datetime.fromisoformat(start["time_utc"].replace("Z", "+00:00")) + if (datetime.now(timezone.utc) - started).total_seconds() < 180: + raise ValueError("reminder is not permitted before 180 seconds") + append_event( + "collection", + "reminder_sent", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + details={"text_sha256": sha256_bytes(report_reminder_text().encode())}, + ) + + +def assert_evaluator_attempt_allowed( + kind: str, identity: str, attempt: int, events: list[dict[str, Any]] | None = None +) -> None: + if kind not in EVALUATOR_KINDS or attempt < 1: + raise ValueError(f"invalid evaluator attempt: {kind}/{identity}/{attempt}") + events = event_records() if events is None else events + if any( + event["event"] == "evaluator_started" + and event.get("attempt") == attempt + and event.get("details", {}).get("kind") == kind + and event.get("details", {}).get("identity") == identity + for event in events + ): + raise ValueError(f"evaluator attempt already started: {kind}/{identity}/{attempt}") + if any( + evaluator_completion_matches(event, kind, identity) + for event in events + ): + raise ValueError(f"evaluator identity already completed: {kind}/{identity}") + if attempt > 1 and not any( + event["event"] == "evaluator_infrastructure_failure" + and event.get("attempt") == attempt - 1 + and event.get("details", {}).get("kind") == kind + and event.get("details", {}).get("identity") == identity + for event in events + ): + raise ValueError("fresh evaluator retry lacks a preceding infrastructure failure") + + +def record_evaluator_start(kind: str, identity: str, attempt: int, agent_id: str) -> None: + if kind not in EVALUATOR_KINDS: + raise ValueError(f"unknown evaluator kind: {kind}") + events = event_records() + assert_evaluator_attempt_allowed(kind, identity, attempt, events) + started = { + ( + event.get("details", {}).get("kind"), + event.get("details", {}).get("identity"), + event.get("attempt"), + ) + for event in events + if event["event"] == "evaluator_started" + } + returned = { + ( + event.get("details", {}).get("kind"), + event.get("details", {}).get("identity"), + event.get("attempt"), + ) + for event in events + if event["event"] == "evaluator_returned" + } + used_agent_ids = { + event.get("agent_id") + for event in events + if event["event"] in {"agent_started", "evaluator_started"} + } + if agent_id in used_agent_ids: + raise ValueError(f"agent ID was already used by an evaluated agent: {agent_id}") + if len(started - returned) >= 3: + raise ValueError("three evaluator agents are already active") + if kind == "scorer": + claims = [row["claim"] for row in read_frozen_tsv(SEALED / "scoring-schedule.tsv")] + if identity not in claims: + raise ValueError(f"unknown scorer claim: {identity}") + prior = claims[: claims.index(identity)] + started_identities = {(item[0], item[1]) for item in started} + if not all(("scorer", claim) in started_identities for claim in prior): + raise ValueError(f"scorer launch-order violation before {identity}") + mode, scorer = identity.split("-", 1) + source_packet = SCORING / "packets" / mode / scorer + verify_score_packet(mode, scorer) + elif kind == "consistency": + claims = [ + row["claim"] + for row in read_frozen_tsv(SEALED / "consistency-schedule.tsv") + ] + if identity not in claims: + raise ValueError(f"unknown consistency-review claim: {identity}") + prior = claims[: claims.index(identity)] + started_identities = {(item[0], item[1]) for item in started} + if not all(("consistency", claim) in started_identities for claim in prior): + raise ValueError( + f"consistency-review launch-order violation before {identity}" + ) + source_packet = evaluator_source_packet(kind, identity) + if not source_packet.exists(): + raise ValueError(f"consistency packet does not exist: {identity}") + verify_consistency_packet(identity) + else: + source_packet = evaluator_source_packet(kind, identity) + if not source_packet.exists(): + raise ValueError(f"adjudication packet does not exist: {identity}") + verify_adjudication_packet(identity) + output = expected_evaluator_output(kind, identity, attempt) + verify_evaluator_runtime(kind, identity, attempt, source_packet, output) + if output.is_symlink() or not output.is_dir() or any(output.iterdir()): + raise ValueError(f"evaluator output is not initially empty: {kind}/{identity}") + append_event( + evaluator_phase(kind), + "evaluator_started", + attempt=attempt, + agent_id=agent_id, + details={ + "kind": kind, + "identity": identity, + "model": "gpt-5.6-sol", + "reasoning_effort": "ultra", + "fork_turns": "none", + "prompt_sha256": sha256_bytes( + render_packet_prompt( + evaluator_prompt_name(kind), + expected_evaluator_packet(kind, identity, attempt), + output, + **({"SCORER_ID": identity.split("-", 1)[1]} if kind == "scorer" else {}), + ).encode() + ), + }, + ) + + +def assert_evaluator_started( + kind: str, identity: str, attempt: int, agent_id: str +) -> None: + matches = [ + event + for event in event_records() + if event["event"] == "evaluator_started" + and event.get("attempt") == attempt + and event.get("agent_id") == agent_id + and event.get("details", {}).get("kind") == kind + and event.get("details", {}).get("identity") == identity + ] + if len(matches) != 1: + raise ValueError( + f"missing evaluator-start record: {kind}/{identity}/{attempt}/{agent_id}" + ) + + +def record_evaluator_prelaunch_failure( + kind: str, identity: str, attempt: int, output: Path, evidence: str +) -> None: + if not evidence.strip(): + raise ValueError("evaluator prelaunch failure requires nonempty evidence") + assert_evaluator_attempt_allowed(kind, identity, attempt) + source_packet = evaluator_source_packet(kind, identity) + if kind == "scorer": + mode, scorer = identity.split("-", 1) + verify_score_packet(mode, scorer) + elif kind == "consistency": + verify_consistency_packet(identity) + else: + verify_adjudication_packet(identity) + verify_evaluator_runtime(kind, identity, attempt, source_packet, output) + if any(output.iterdir()): + raise ValueError("prelaunch evaluator output is not empty") + append_event( + evaluator_phase(kind), + "evaluator_prelaunch_failure", + attempt=attempt, + details={ + "kind": kind, + "identity": identity, + "disposition": "API_NO_AGENT_START", + "evidence": evidence, + }, + ) + + +def prepare_cell(run_id: str, runtime: Path) -> None: + validate_static(require_lock=True, announce=False) + assert_prepare_allowed(run_id) + if runtime.exists(): + raise FileExistsError(f"runtime already exists: {runtime}") + row, target, condition = resolve_run(run_id) + expected_runtime = Path("/tmp/ur-eval") / row["cell_id"] + if runtime != expected_runtime: + raise ValueError(f"runtime must be the frozen neutral path {expected_runtime}") + package_source = EVALS / condition["package_path"] + target_source = EVALS / target["source_path"] + runtime.mkdir(parents=True) + shutil.copytree(package_source, runtime / "package") + shutil.copytree(target_source, runtime / "target") + shutil.copy2(FREEZE / "allowlists" / f"{target['mode']}.txt", runtime / "allowlist.txt") + (runtime / "output").mkdir() + if byte_tree_digest(runtime / "package") != condition["tree_sha256"]: + raise ValueError(f"package runtime copy does not match frozen identity for {run_id}") + if byte_tree_digest(runtime / "target") != target["tree_sha256"]: + raise ValueError(f"target runtime copy does not match frozen identity for {run_id}") + package_bytes = byte_tree_digest(runtime / "package") + target_bytes = byte_tree_digest(runtime / "target") + if package_bytes != byte_tree_digest(package_source): + raise ValueError(f"package runtime copy differs for {run_id}") + if target_bytes != byte_tree_digest(target_source): + raise ValueError(f"target runtime copy differs for {run_id}") + expected_allowlist = frozen_file_digest( + FREEZE / "allowlists" / f"{target['mode']}.txt" + ) + if sha256_file(runtime / "allowlist.txt") != expected_allowlist: + raise ValueError(f"allowlist runtime copy differs from frozen bytes for {run_id}") + attestation = { + "schema_version": 1, + "run_id": run_id, + "cell_id": row["cell_id"], + "runtime_root": str(runtime), + "package_byte_tree_sha256": package_bytes, + "target_byte_tree_sha256": target_bytes, + "allowlist_sha256": expected_allowlist, + "report_prompt_sha256": sha256_bytes(render_report_prompt(run_id, runtime).encode()), + "output_initially_empty": True, + "prepared_utc": utc_now(), + } + setup_path = COLLECTION / "setups" / f"{run_id}.json" + write_once(setup_path, json_dump(attestation)) + setup_path.chmod(0o444) + make_read_only(runtime / "package") + make_read_only(runtime / "target") + os.utime(runtime / "allowlist.txt", (0, 0), follow_symlinks=False) + (runtime / "allowlist.txt").chmod(0o444) + os.utime(runtime, (0, 0), follow_symlinks=False) + runtime.chmod(0o555) + append_event( + "collection", + "cell_prepared", + run_id=run_id, + digest=sha256_file(setup_path), + details={"cell_id": row["cell_id"]}, + ) + print(runtime) + + +def report_prompt_blocks() -> list[str]: + template = read_frozen_text(FREEZE / "prompts" / "report.md") + blocks = re.findall(r"```text\n(.*?)\n```", template, flags=re.DOTALL) + if len(blocks) != 2: + raise ValueError("report prompt template must contain prompt and reminder fences") + return blocks + + +def report_reminder_text() -> str: + return report_prompt_blocks()[1] + + +def render_report_prompt(run_id: str, runtime: Path) -> str: + _row, target, _condition = resolve_run(run_id) + prompt = report_prompt_blocks()[0] + replacements = { + "[PACKAGE]": str(runtime / "package"), + "[TARGET]": str(runtime / "target"), + "[URL_ALLOWLIST]": str(runtime / "allowlist.txt"), + "[OUTPUT]": str(runtime / "output"), + "[WORD_LIMIT]": target["word_cap"], + } + for old, new in replacements.items(): + prompt = prompt.replace(old, new) + if re.search(r"\[[A-Z_]+\]", prompt): + raise ValueError("unresolved report-prompt placeholder") + return prompt + + +def verify_runtime(run_id: str, runtime: Path, *, allow_invalid_output: bool = False) -> None: + setup_path = COLLECTION / "setups" / f"{run_id}.json" + prepared = [ + event + for event in event_records() + if event["event"] == "cell_prepared" and event.get("run_id") == run_id + ] + if len(prepared) != 1 or prepared[0].get("sha256") != sha256_file(setup_path): + raise ValueError(f"setup attestation changed for {run_id}") + setup = json.loads(setup_path.read_text()) + row, _target, _condition = resolve_run(run_id) + expected_runtime = Path("/tmp/ur-eval") / row["cell_id"] + if runtime != expected_runtime or runtime != Path(setup["runtime_root"]): + raise ValueError(f"runtime is not bound to frozen cell {run_id}") + if runtime.is_symlink() or not runtime.is_dir(): + raise ValueError(f"runtime root is not a real directory: {runtime}") + if runtime.stat().st_mode & 0o222: + raise ValueError(f"runtime root is writable: {runtime}") + required_inputs = {"package", "target", "allowlist.txt"} + expected_entries = required_inputs | {"output"} + actual_entries = {entry.name for entry in runtime.iterdir()} + if ( + (not allow_invalid_output and actual_entries != expected_entries) + or (allow_invalid_output and (not required_inputs <= actual_entries or actual_entries - expected_entries)) + ): + raise ValueError(f"runtime root inventory changed for {run_id}") + allowlist = runtime / "allowlist.txt" + output = runtime / "output" + if allowlist.is_symlink() or not stat.S_ISREG(allowlist.lstat().st_mode): + raise ValueError(f"runtime allowlist is not a real file: {run_id}") + if not allow_invalid_output and ( + not output.exists() + or output.is_symlink() + or not stat.S_ISDIR(output.lstat().st_mode) + ): + raise ValueError(f"runtime output is not a real directory: {run_id}") + checks = { + "package_byte_tree_sha256": byte_tree_digest(runtime / "package"), + "target_byte_tree_sha256": byte_tree_digest(runtime / "target"), + "allowlist_sha256": sha256_file(allowlist), + "report_prompt_sha256": sha256_bytes(render_report_prompt(run_id, runtime).encode()), + } + for field, actual in checks.items(): + if setup[field] != actual: + raise ValueError(f"runtime input changed for {run_id}: {field}") + + +def observe_path(path: Path) -> dict[str, Any]: + observation: dict[str, Any] = {"path": str(path)} + try: + mode = path.lstat().st_mode + except FileNotFoundError: + observation["type"] = "missing" + return observation + observation["mode"] = stat.S_IMODE(mode) + if stat.S_ISLNK(mode): + observation.update({"type": "symlink", "target": os.readlink(path)}) + elif stat.S_ISREG(mode): + data = path.read_bytes() + observation.update( + {"type": "file", "bytes": len(data), "sha256": sha256_bytes(data)} + ) + elif stat.S_ISDIR(mode): + observation["type"] = "directory" + observation["entries"] = sorted(entry.name for entry in path.iterdir()) + try: + observation["byte_tree_sha256"] = byte_tree_digest(path) + except (OSError, ValueError) as error: + observation["byte_tree_error"] = f"{type(error).__name__}: {error}" + else: + observation.update({"type": "special", "file_type": stat.S_IFMT(mode)}) + return observation + + +def report_runtime_forensics(run_id: str, runtime: Path, error: Exception) -> dict[str, Any]: + row, target, condition = resolve_run(run_id) + expected_runtime = Path("/tmp/ur-eval") / row["cell_id"] + if runtime != expected_runtime: + raise ValueError(f"forensic runtime is not the frozen neutral path {expected_runtime}") + setup_path = COLLECTION / "setups" / f"{run_id}.json" + prepared = [ + event + for event in event_records() + if event["event"] == "cell_prepared" and event.get("run_id") == run_id + ] + return { + "schema_version": 1, + "run_id": run_id, + "verification_error": f"{type(error).__name__}: {error}", + "expected": { + "runtime_root": str(expected_runtime), + "setup_sha256": prepared[0].get("sha256") if len(prepared) == 1 else None, + "package_byte_tree_sha256": condition["tree_sha256"], + "target_byte_tree_sha256": target["tree_sha256"], + "allowlist_sha256": frozen_file_digest( + FREEZE / "allowlists" / f"{target['mode']}.txt" + ), + }, + "observed": { + "runtime_root": observe_path(runtime), + "setup": observe_path(setup_path), + "package": observe_path(runtime / "package") + if runtime.exists() and not runtime.is_symlink() and runtime.is_dir() + else {"type": "unavailable"}, + "target": observe_path(runtime / "target") + if runtime.exists() and not runtime.is_symlink() and runtime.is_dir() + else {"type": "unavailable"}, + "allowlist": observe_path(runtime / "allowlist.txt") + if runtime.exists() and not runtime.is_symlink() and runtime.is_dir() + else {"type": "unavailable"}, + }, + } + + +def append_event( + phase: str, + event: str, + *, + run_id: str | None = None, + attempt: int | None = None, + agent_id: str | None = None, + digest: str | None = None, + details: dict[str, Any] | None = None, +) -> None: + path = RUN / "events.jsonl" + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("a+", encoding="utf-8", newline="") as file: + fcntl.flock(file, fcntl.LOCK_EX) + file.seek(0) + lines = file.read().splitlines() + sequence = len(lines) + 1 + record: dict[str, Any] = { + "schema_version": 1, + "sequence": sequence, + "previous_event_sha256": sha256_bytes(lines[-1].encode()) if lines else None, + "time_utc": utc_now(), + "phase": phase, + "event": event, + "details": details or {}, + } + if run_id is not None: + record["run_id"] = run_id + if attempt is not None: + record["attempt"] = attempt + if agent_id is not None: + record["agent_id"] = agent_id + if digest is not None: + record["sha256"] = digest + validate_event_record(record, sequence) + file.seek(0, os.SEEK_END) + file.write(json.dumps(record, sort_keys=True) + "\n") + file.flush() + os.fsync(file.fileno()) + fcntl.flock(file, fcntl.LOCK_UN) + + +def validate_event_ledger() -> None: + path = RUN / "events.jsonl" + if not path.exists(): + return + previous: str | None = None + for index, line in enumerate(path.read_text().splitlines(), start=1): + value = json.loads(line) + validate_event_record(value, index) + if value.get("previous_event_sha256") != previous: + raise ValueError(f"broken event hash chain at line {index}") + previous = sha256_bytes(line.encode()) + + +def validate_event_record(value: Any, expected_sequence: int) -> None: + required = { + "schema_version", + "sequence", + "previous_event_sha256", + "time_utc", + "phase", + "event", + "details", + } + optional = {"run_id", "attempt", "agent_id", "sha256"} + if not isinstance(value, dict) or not required <= set(value) or not set(value) <= required | optional: + raise ValueError(f"invalid event fields at sequence {expected_sequence}") + if ( + type(value["schema_version"]) is not int + or value["schema_version"] != 1 + or type(value["sequence"]) is not int + or value["sequence"] != expected_sequence + ): + raise ValueError(f"invalid event sequence {expected_sequence}") + previous = value["previous_event_sha256"] + if previous is not None and ( + not isinstance(previous, str) or not re.fullmatch(r"[0-9a-f]{64}", previous) + ): + raise ValueError(f"invalid prior-event hash at sequence {expected_sequence}") + if not isinstance(value["time_utc"], str): + raise ValueError(f"invalid event timestamp at sequence {expected_sequence}") + try: + parsed_time = datetime.fromisoformat(value["time_utc"].replace("Z", "+00:00")) + except (AttributeError, ValueError) as error: + raise ValueError(f"invalid event timestamp at sequence {expected_sequence}") from error + if parsed_time.tzinfo is None: + raise ValueError(f"event timestamp lacks timezone at sequence {expected_sequence}") + if ( + not isinstance(value["phase"], str) + or value["phase"] not in EVENT_PHASES + or not isinstance(value["event"], str) + or not value["event"] + ): + raise ValueError(f"invalid event identity at sequence {expected_sequence}") + if not isinstance(value["details"], dict): + raise ValueError(f"invalid event details at sequence {expected_sequence}") + if "run_id" in value and (not isinstance(value["run_id"], str) or not value["run_id"]): + raise ValueError(f"invalid event run ID at sequence {expected_sequence}") + if "attempt" in value and ( + not isinstance(value["attempt"], int) + or isinstance(value["attempt"], bool) + or value["attempt"] < 1 + ): + raise ValueError(f"invalid event attempt at sequence {expected_sequence}") + if "agent_id" in value and ( + not is_nonblank_string(value["agent_id"]) + or value["agent_id"] != value["agent_id"].strip() + ): + raise ValueError(f"invalid event agent ID at sequence {expected_sequence}") + if "sha256" in value and ( + not isinstance(value["sha256"], str) or not re.fullmatch(r"[0-9a-f]{64}", value["sha256"]) + ): + raise ValueError(f"invalid event digest at sequence {expected_sequence}") + + +def validate_preserved_artifacts() -> None: + events = event_records() + attempt_events: dict[tuple[str, int], dict[str, Any]] = {} + evaluator_attempt_events: dict[tuple[str, str, int], dict[str, Any]] = {} + invalid_output_events: dict[tuple[str, str, int], dict[str, Any]] = {} + for event in events: + if event["event"] == "attempt_preserved": + key = (event["run_id"], event["attempt"]) + if key in attempt_events: + raise ValueError(f"duplicate attempt-preservation event: {key}") + attempt_events[key] = event + directory = ( + COLLECTION + / "attempts" + / event["run_id"] + / str(event["attempt"]) + ) + if byte_tree_digest(directory) != event.get("sha256"): + raise ValueError( + f"preserved attempt changed: {event['run_id']}/{event['attempt']}" + ) + elif event["event"] == "invalid_output_preserved": + details = event.get("details", {}) + kind = details.get("kind") + if kind not in EVALUATOR_KINDS: + raise ValueError("invalid preserved-output evaluator kind") + phase = evaluator_phase(kind) + identity = details.get("identity") + key = (phase, str(identity), event["attempt"]) + if event["phase"] != phase or key in invalid_output_events: + raise ValueError(f"duplicate or misphased invalid-output event: {key}") + invalid_output_events[key] = event + directory = ( + SCORING + / "invalid" + / phase + / str(identity) + / str(event.get("attempt")) + ) + if byte_tree_digest(directory) != event.get("sha256"): + raise ValueError(f"preserved invalid output changed: {directory}") + elif event["event"] == "evaluator_attempt_preserved": + details = event.get("details", {}) + kind = details.get("kind") + if kind not in EVALUATOR_KINDS: + raise ValueError("invalid preserved-attempt evaluator kind") + phase = evaluator_phase(kind) + identity = details.get("identity") + key = (phase, str(identity), event["attempt"]) + if event["phase"] != phase or key in evaluator_attempt_events: + raise ValueError(f"duplicate or misphased evaluator-attempt event: {key}") + evaluator_attempt_events[key] = event + directory = ( + SCORING + / "evaluator-attempts" + / phase + / str(identity) + / str(event.get("attempt")) + ) + if byte_tree_digest(directory) != event.get("sha256"): + raise ValueError(f"preserved evaluator attempt changed: {directory}") + elif event["event"] == "freeze_locked": + if ( + sha256_file(FILE_MANIFEST) != event.get("sha256") + or sha256_file(LOCK) != event.get("details", {}).get("lock_sha256") + ): + raise ValueError("freeze-lock artifacts changed") + elif event["event"] == "authority_verified": + if frozen_file_digest(FREEZE / "authority-manifest.tsv") != event.get("sha256"): + raise ValueError("authority verification event has the wrong manifest") + elif event["event"] == "report_preserved": + report = ( + COLLECTION + / "attempts" + / event["run_id"] + / str(event["attempt"]) + / "report.md" + ) + if sha256_file(report) != event.get("sha256"): + raise ValueError(f"preserved report changed: {event['run_id']}") + elif event["event"] == "collection_locked": + if sha256_file(COLLECTION / "valid-index.jsonl") != event.get("sha256"): + raise ValueError("locked collection index changed") + elif event["event"] == "blind_packet_preserved": + details = event.get("details", {}) + packet = SCORING / "packets" / str(details.get("mode")) / str( + details.get("scorer") + ) + if byte_tree_digest(packet) != event.get("sha256"): + raise ValueError(f"preserved blind packet changed: {packet}") + elif event["event"] == "score_preserved": + details = event.get("details", {}) + score = SCORING / "raw" / str(details.get("mode")) / ( + str(details.get("scorer")) + ".json" + ) + if sha256_file(score) != event.get("sha256"): + raise ValueError(f"preserved score changed: {score}") + elif event["event"] == "consistency_packet_preserved": + packet = SCORING / "consistency-packets" / str( + event.get("details", {}).get("mode") + ) + if byte_tree_digest(packet) != event.get("sha256"): + raise ValueError(f"preserved consistency packet changed: {packet}") + elif event["event"] == "consistency_review_preserved": + path = SCORING / "consistency" / ( + str(event.get("details", {}).get("mode")) + ".json" + ) + if sha256_file(path) != event.get("sha256"): + raise ValueError(f"preserved consistency review changed: {path}") + elif event["event"] == "adjudication_cells_materialized": + path = SCORING / "adjudication-cells" / ( + str(event.get("details", {}).get("mode")) + ".json" + ) + if sha256_file(path) != event.get("sha256"): + raise ValueError(f"preserved adjudication cells changed: {path}") + elif event["event"] == "adjudication_packet_preserved": + packet = SCORING / "adjudication-packets" / str( + event.get("details", {}).get("mode") + ) + if byte_tree_digest(packet) != event.get("sha256"): + raise ValueError(f"preserved adjudication packet changed: {packet}") + elif event["event"] == "adjudication_preserved": + path = SCORING / "adjudications" / ( + str(event.get("details", {}).get("mode")) + ".json" + ) + if sha256_file(path) != event.get("sha256"): + raise ValueError(f"preserved adjudication changed: {path}") + elif event["event"] == "final_blind_score_locked": + path = SCORING / "final" / ( + str(event.get("details", {}).get("mode")) + ".json" + ) + if sha256_file(path) != event.get("sha256"): + raise ValueError(f"final blind score changed: {path}") + elif event["event"] == "conditions_revealed": + if sha256_file(RUN / "unblinding.json") != event.get("sha256"): + raise ValueError("unblinding artifact changed") + elif event["event"] == "aggregate_written": + if ( + sha256_file(RESULTS / "aggregate.json") != event.get("sha256") + or sha256_file(RESULTS / "summary.md") + != event.get("details", {}).get("summary_sha256") + ): + raise ValueError("final result artifacts changed") + elif event["event"] == "run_invalidated": + if sha256_file(RUN / "INVALID.json") != event.get("sha256"): + raise ValueError("INVALID marker changed") + attempt_root = COLLECTION / "attempts" + actual_attempts: set[tuple[str, int]] = set() + if attempt_root.exists(): + for run_dir in attempt_root.iterdir(): + if run_dir.is_symlink() or not run_dir.is_dir(): + raise ValueError(f"invalid attempt run directory: {run_dir}") + for attempt_dir in run_dir.iterdir(): + if ( + attempt_dir.is_symlink() + or not attempt_dir.is_dir() + or not attempt_dir.name.isdigit() + or int(attempt_dir.name) < 1 + ): + raise ValueError(f"invalid attempt directory: {attempt_dir}") + actual_attempts.add((run_dir.name, int(attempt_dir.name))) + if actual_attempts != set(attempt_events): + raise ValueError("attempt directories and preservation events differ") + + def inventory_evaluator_attempts( + root: Path, expected: set[tuple[str, str, int]], label: str + ) -> set[tuple[str, str, int]]: + if root.is_symlink(): + raise ValueError(f"invalid {label} artifact root: {root}") + if not root.exists(): + if expected: + raise ValueError(f"missing {label} artifact root") + return set() + if not root.is_dir(): + raise ValueError(f"invalid {label} artifact root: {root}") + if not expected: + raise ValueError(f"orphan empty {label} artifact root: {root}") + actual: set[tuple[str, str, int]] = set() + actual_phases: set[str] = set() + actual_identities: set[tuple[str, str]] = set() + scorer_identities = { + f"{mode}-{scorer}" for mode in MODES for scorer in SCORERS + } + for phase_dir in root.iterdir(): + if ( + phase_dir.is_symlink() + or not phase_dir.is_dir() + or phase_dir.name not in {"scoring", "consistency", "adjudication"} + ): + raise ValueError(f"invalid {label} phase directory: {phase_dir}") + phase = phase_dir.name + actual_phases.add(phase) + for identity_dir in phase_dir.iterdir(): + identity = identity_dir.name + valid_identity = ( + identity in scorer_identities + if phase == "scoring" + else identity in MODES + ) + if identity_dir.is_symlink() or not identity_dir.is_dir() or not valid_identity: + raise ValueError(f"invalid {label} identity directory: {identity_dir}") + actual_identities.add((phase, identity)) + for attempt_dir in identity_dir.iterdir(): + if ( + attempt_dir.is_symlink() + or not attempt_dir.is_dir() + or not re.fullmatch(r"[1-9][0-9]*", attempt_dir.name) + ): + raise ValueError(f"invalid {label} attempt directory: {attempt_dir}") + actual.add((phase, identity, int(attempt_dir.name))) + expected_phases = {phase for phase, _identity, _attempt in expected} + expected_identities = {(phase, identity) for phase, identity, _attempt in expected} + if actual_phases != expected_phases or actual_identities != expected_identities: + raise ValueError(f"{label} directory hierarchy differs from preservation events") + return actual + + actual_evaluator_attempts = inventory_evaluator_attempts( + SCORING / "evaluator-attempts", + set(evaluator_attempt_events), + "evaluator-attempt", + ) + if actual_evaluator_attempts != set(evaluator_attempt_events): + raise ValueError("evaluator-attempt directories and preservation events differ") + actual_invalid_outputs = inventory_evaluator_attempts( + SCORING / "invalid", set(invalid_output_events), "invalid-output" + ) + if actual_invalid_outputs != set(invalid_output_events): + raise ValueError("invalid-output directories and preservation events differ") + if set(evaluator_attempt_events) & set(invalid_output_events): + raise ValueError("one evaluator attempt has both valid/infra and invalid preservation") + + invalidated = [event for event in events if event["event"] == "run_invalidated"] + invalid_returns = [ + event + for event in events + if event["event"] == "evaluator_returned" + and event.get("details", {}).get("api_state") == "INVALID_OUTPUT" + ] + invalid_marker = RUN / "INVALID.json" + if not invalid_output_events: + if invalid_marker.exists() or invalid_marker.is_symlink() or invalidated or invalid_returns: + raise ValueError("INVALID marker or terminal-invalid events lack an invalid output") + else: + if len(invalid_output_events) != 1 or len(invalidated) != 1 or len(invalid_returns) != 1: + raise ValueError("terminal-invalid evaluator state is not unique and complete") + if invalid_marker.is_symlink() or not invalid_marker.is_file(): + raise ValueError("terminal-invalid event lacks a real INVALID marker") + key, invalid_event = next(iter(invalid_output_events.items())) + phase, identity, attempt = key + invalid_directory = SCORING / "invalid" / phase / identity / str(attempt) + attestation = json.loads((invalid_directory / "attestation.json").read_text()) + marker = json.loads(invalid_marker.read_text()) + expected_kind = { + "scoring": "scorer", + "consistency": "consistency", + "adjudication": "adjudicator", + }[phase] + if marker != attestation: + raise ValueError("INVALID marker differs from the invalid-output attestation") + if ( + marker.get("phase") != phase + or marker.get("kind") != expected_kind + or marker.get("identity") != identity + or marker.get("attempt") != attempt + or marker.get("agent_id") != invalid_event.get("agent_id") + or marker.get("disposition") != "INVALID_NONRERUNNABLE_EVALUATOR_OUTPUT" + ): + raise ValueError("terminal-invalid attestation identity is inconsistent") + invalidated_event = invalidated[0] + if ( + invalidated_event.get("phase") != phase + or invalidated_event.get("attempt") != attempt + or invalidated_event.get("agent_id") != marker.get("agent_id") + or invalidated_event.get("sha256") != sha256_file(invalid_marker) + or invalidated_event.get("details") + != {"identity": identity, "evidence": marker.get("evidence")} + ): + raise ValueError("run-invalidated event differs from the INVALID marker") + returned = invalid_returns[0] + if ( + returned.get("phase") != phase + or returned.get("attempt") != attempt + or returned.get("agent_id") != marker.get("agent_id") + or returned.get("details") + != {"kind": expected_kind, "identity": identity, "api_state": "INVALID_OUTPUT"} + ): + raise ValueError("invalid evaluator-return event is inconsistent") + + setup_root = COLLECTION / "setups" + actual_setups: set[str] = set() + if setup_root.exists(): + for path in setup_root.iterdir(): + if path.is_symlink() or not path.is_file() or path.suffix != ".json": + raise ValueError(f"invalid setup artifact: {path}") + actual_setups.add(path.stem) + prepared = { + event["run_id"]: event + for event in events + if event["event"] == "cell_prepared" + } + if len(prepared) != sum(event["event"] == "cell_prepared" for event in events): + raise ValueError("duplicate cell-preparation event") + if actual_setups != set(prepared): + raise ValueError("cell setup files and preparation events differ") + for run_id, event in prepared.items(): + if sha256_file(setup_root / f"{run_id}.json") != event.get("sha256"): + raise ValueError(f"cell setup changed: {run_id}") + + +def append_report_index(metadata: dict[str, Any]) -> None: + path = COLLECTION / "valid-index.jsonl" + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("a+", encoding="utf-8", newline="") as file: + fcntl.flock(file, fcntl.LOCK_EX) + file.seek(0) + existing = [json.loads(line) for line in file.read().splitlines()] + if any(item["run_id"] == metadata["run_id"] for item in existing): + raise ValueError(f"canonical report already indexed for {metadata['run_id']}") + file.seek(0, os.SEEK_END) + file.write(json.dumps(metadata, sort_keys=True) + "\n") + file.flush() + os.fsync(file.fileno()) + fcntl.flock(file, fcntl.LOCK_UN) + + +def output_entries(output: Path) -> list[Path]: + if not output.exists() or output.is_symlink() or not stat.S_ISDIR(output.lstat().st_mode): + raise ValueError(f"output path is not a real directory: {output}") + entries = sorted(output.iterdir(), key=lambda path: path.name) if output.exists() else [] + for entry in entries: + mode = entry.lstat().st_mode + if stat.S_ISLNK(mode) or not stat.S_ISREG(mode): + raise ValueError(f"unsupported output entry: {entry}") + return entries + + +def snapshot_path(source: Path, raw_root: Path) -> list[dict[str, Any]]: + """Preserve every regular byte reachable without following a symlink.""" + raw_root.mkdir(parents=True, exist_ok=True) + records: list[dict[str, Any]] = [] + try: + root_mode = source.lstat().st_mode + except FileNotFoundError: + return [{"path": ".", "type": "missing"}] + if stat.S_ISLNK(root_mode): + return [{"path": ".", "type": "symlink", "target": os.readlink(source)}] + if stat.S_ISREG(root_mode): + data = source.read_bytes() + stored = "__output_path_file__" + write_bytes_once(raw_root / stored, data) + return [ + { + "path": ".", + "type": "file", + "stored_as": stored, + "bytes": len(data), + "sha256": sha256_bytes(data), + } + ] + if not stat.S_ISDIR(root_mode): + return [{"path": ".", "type": "special", "mode": stat.S_IFMT(root_mode)}] + + records.append({"path": ".", "type": "directory"}) + + def walk(directory: Path, relative: Path) -> None: + with os.scandir(directory) as iterator: + entries = sorted(iterator, key=lambda entry: entry.name) + for entry in entries: + child_relative = relative / entry.name + rendered = child_relative.as_posix() + mode = entry.stat(follow_symlinks=False).st_mode + source_child = directory / entry.name + destination_child = raw_root / child_relative + if stat.S_ISREG(mode): + data = source_child.read_bytes() + write_bytes_once(destination_child, data) + records.append( + { + "path": rendered, + "type": "file", + "bytes": len(data), + "sha256": sha256_bytes(data), + } + ) + elif stat.S_ISDIR(mode): + destination_child.mkdir() + records.append({"path": rendered, "type": "directory"}) + walk(source_child, child_relative) + elif stat.S_ISLNK(mode): + records.append( + {"path": rendered, "type": "symlink", "target": os.readlink(source_child)} + ) + else: + records.append( + {"path": rendered, "type": "special", "mode": stat.S_IFMT(mode)} + ) + + walk(source, Path()) + return records + + +def preserve_captured_file(raw_root: Path, filename: str, data: bytes) -> list[dict[str, Any]]: + """Preserve already-captured bytes as an exact one-file output snapshot.""" + if Path(filename).name != filename or filename in {"", ".", ".."}: + raise ValueError(f"invalid captured output filename: {filename}") + raw_root.mkdir(parents=True, exist_ok=False) + write_bytes_once(raw_root / filename, data) + return [ + {"path": ".", "type": "directory"}, + { + "path": filename, + "type": "file", + "bytes": len(data), + "sha256": sha256_bytes(data), + }, + ] + + +def record_report( + run_id: str, + attempt: int, + runtime: Path, + agent_id: str, + scope_deviation: bool, + scope_evidence: str, +) -> None: + if scope_deviation and not scope_evidence.strip(): + raise ValueError("scope deviation requires nonempty evidence") + assert_agent_started(run_id, attempt, agent_id) + verify_runtime(run_id, runtime) + output = runtime / "output" + if not output.exists() or output.is_symlink() or not stat.S_ISDIR(output.lstat().st_mode): + raise ValueError(f"output path is not a real directory: {output}") + entries = output_entries(output) + if [entry.name for entry in entries] != ["report.md"]: + raise ValueError(f"{run_id} output is not exactly one report.md") + data = entries[0].read_bytes() + text = data.decode("utf-8") + if not text.strip(): + raise ValueError(f"{run_id} report.md is empty or whitespace-only") + words = len(text.split()) + _row, target, _condition = resolve_run(run_id) + destination = COLLECTION / "attempts" / run_id / str(attempt) + if destination.exists(): + raise FileExistsError(f"attempt already preserved: {destination}") + destination.mkdir(parents=True) + write_bytes_once(destination / "report.md", data) + metadata = { + "schema_version": 1, + "run_id": run_id, + "attempt": attempt, + "agent_id": agent_id, + "report_sha256": sha256_bytes(data), + "word_count": words, + "word_cap": int(target["word_cap"]), + "within_word_cap": words <= int(target["word_cap"]), + "utf8": True, + "canonical_for_scoring": True, + "semantic_noncompletion": False, + "terminal_disposition": "COMPLETE", + "api_state": "COMPLETED", + "operational_scope_deviation": scope_deviation, + "scope_evidence": scope_evidence, + "source_isolation": "procedural", + "recorded_utc": utc_now(), + } + write_once(destination / "attestation.json", json_dump(metadata)) + append_report_index(metadata) + make_read_only(destination) + append_event( + "collection", + "attempt_preserved", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(destination), + details={"disposition": "COMPLETE"}, + ) + append_event( + "collection", + "agent_returned", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + details={"api_state": "COMPLETED"}, + ) + append_event( + "collection", + "report_preserved", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + digest=metadata["report_sha256"], + details={ + "word_count": words, + "within_word_cap": metadata["within_word_cap"], + "operational_scope_deviation": scope_deviation, + "disposition": "COMPLETE", + }, + ) + print(json_dump(metadata), end="") + + +def preserve_failed_report_attempt( + run_id: str, + attempt: int, + runtime: Path, + agent_id: str, + disposition: str, + evidence: str, + scope_deviation: bool, +) -> None: + if disposition not in INFRA_FAILURE_CODES | TERMINAL_REPORT_FAILURE_CODES: + raise ValueError(f"unknown failure disposition: {disposition}") + if not evidence.strip(): + raise ValueError("failed report disposition requires nonempty evidence") + assert_agent_started(run_id, attempt, agent_id) + forensics: dict[str, Any] | None = None + try: + verify_runtime(run_id, runtime, allow_invalid_output=True) + except (OSError, ValueError) as error: + forensics = report_runtime_forensics(run_id, runtime, error) + disposition = "INVALID_OUTPUT" + scope_deviation = True + evidence = ( + f"{evidence} Runtime/input verification failed: " + f"{forensics['verification_error']}" + ) + output = runtime / "output" + destination = COLLECTION / "attempts" / run_id / str(attempt) + if destination.exists(): + raise FileExistsError(f"attempt already preserved: {destination}") + if forensics is None: + entry_manifest = snapshot_path(output, destination / "raw-output") + manifest_path = destination / "raw-output-manifest.json" + captured_report = destination / "raw-output" / "report.md" + else: + # A failed runtime check may have been caused by an unexpected root entry + # or by drift in an input tree. Preserve the entire neutral runtime without + # following symlinks, rather than retaining only the ordinary output path. + entry_manifest = snapshot_path(runtime, destination / "raw-runtime") + manifest_path = destination / "raw-runtime-manifest.json" + setup = COLLECTION / "setups" / f"{run_id}.json" + setup_manifest = snapshot_path(setup, destination / "setup-at-verification") + write_once( + destination / "setup-at-verification-manifest.json", + json_dump(setup_manifest), + ) + captured_report = destination / "raw-runtime" / "output" / "report.md" + usable_report: bytes | None = None + if ( + captured_report.exists() + and not captured_report.is_symlink() + and stat.S_ISREG(captured_report.lstat().st_mode) + ): + data = captured_report.read_bytes() + try: + data.decode("utf-8") + except UnicodeDecodeError: + pass + else: + usable_report = data + write_once(manifest_path, json_dump(entry_manifest)) + if forensics is not None: + write_once(destination / "runtime-forensics.json", json_dump(forensics)) + infrastructure = disposition in INFRA_FAILURE_CODES + if infrastructure: + metadata = { + "schema_version": 1, + "run_id": run_id, + "attempt": attempt, + "agent_id": agent_id, + "terminal_disposition": disposition, + "api_state": "INFRASTRUCTURE_FAILURE", + "rerunnable": True, + "evidence": evidence, + "operational_scope_deviation": scope_deviation, + "recorded_utc": utc_now(), + } + write_once(destination / "attestation.json", json_dump(metadata)) + make_read_only(destination) + append_event( + "collection", + "attempt_preserved", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(destination), + details={"disposition": disposition}, + ) + append_event( + "collection", + "agent_returned", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + details={"api_state": "INFRASTRUCTURE_FAILURE"}, + ) + append_event( + "collection", + "infrastructure_failure", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + details={"disposition": disposition, "evidence": evidence}, + ) + runtime.chmod(0o755) + try: + if output.is_symlink() or output.is_file(): + output.unlink() + elif output.exists(): + shutil.rmtree(output) + output.mkdir() + finally: + os.utime(runtime, (0, 0), follow_symlinks=False) + runtime.chmod(0o555) + print(json_dump(metadata), end="") + return + + if usable_report is None: + usable_report = ( + "# Evaluator-marked failed replicate\n\n" + f"No usable canonical report was produced. Terminal disposition: {disposition}.\n" + ).encode() + text = usable_report.decode("utf-8") + words = len(text.split()) + _row, target, _condition = resolve_run(run_id) + write_bytes_once(destination / "report.md", usable_report) + metadata = { + "schema_version": 1, + "run_id": run_id, + "attempt": attempt, + "agent_id": agent_id, + "report_sha256": sha256_bytes(usable_report), + "word_count": words, + "word_cap": int(target["word_cap"]), + "within_word_cap": words <= int(target["word_cap"]), + "utf8": True, + "canonical_for_scoring": True, + "semantic_noncompletion": True, + "terminal_disposition": disposition, + "api_state": "TERMINAL_NONCOMPLETION", + "operational_scope_deviation": scope_deviation or disposition == "INVALID_OUTPUT", + "scope_evidence": evidence, + "source_isolation": "procedural", + "recorded_utc": utc_now(), + } + write_once(destination / "attestation.json", json_dump(metadata)) + append_report_index(metadata) + make_read_only(destination) + append_event( + "collection", + "attempt_preserved", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(destination), + details={"disposition": disposition}, + ) + append_event( + "collection", + "agent_returned", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + details={"api_state": "TERMINAL_NONCOMPLETION"}, + ) + append_event( + "collection", + "report_preserved", + run_id=run_id, + attempt=attempt, + agent_id=agent_id, + digest=metadata["report_sha256"], + details={"disposition": disposition, "semantic_noncompletion": True}, + ) + print(json_dump(metadata), end="") + + +def verify_collection_lock() -> None: + path = COLLECTION / "valid-index.jsonl" + matches = [event for event in event_records() if event["event"] == "collection_locked"] + if len(matches) != 1 or matches[0].get("sha256") != sha256_file(path): + raise ValueError("canonical collection index is not locked to the event ledger") + if matches[0].get("details", {}).get("report_count") != TOTAL_REPORTS: + raise ValueError("collection lock has an invalid report count") + + +def load_index(*, require_collection_lock: bool = False) -> dict[str, dict[str, Any]]: + path = COLLECTION / "valid-index.jsonl" + if path.is_symlink() or not path.is_file(): + raise ValueError("canonical report index is not a real file") + rows = [json.loads(line) for line in path.read_text().splitlines()] + if any(not isinstance(row, dict) or not isinstance(row.get("run_id"), str) for row in rows): + raise ValueError("canonical report index has an invalid row") + result = {row["run_id"]: row for row in rows} + if len(rows) != len(result): + raise ValueError("duplicate canonical reports in index") + if require_collection_lock: + verify_collection_lock() + for run_id, row in result.items(): + if ( + not isinstance(row, dict) + or type(row.get("schema_version")) is not int + or row["schema_version"] != 1 + or type(row.get("attempt")) is not int + or row["attempt"] < 1 + or not isinstance(row.get("agent_id"), str) + or not row["agent_id"] + or not isinstance(row.get("report_sha256"), str) + or not re.fullmatch(r"[0-9a-f]{64}", row["report_sha256"]) + or type(row.get("word_count")) is not int + or row["word_count"] < 0 + ): + raise ValueError(f"invalid canonical report index row for {run_id}") + attempt = row["attempt"] + directory = COLLECTION / "attempts" / run_id / str(attempt) + report = directory / "report.md" + attestation = json.loads((directory / "attestation.json").read_text()) + if attestation != row: + raise ValueError(f"indexed attestation mismatch for {run_id}") + if sha256_file(report) != row["report_sha256"]: + raise ValueError(f"indexed report hash mismatch for {run_id}") + text = report.read_bytes().decode("utf-8") + if len(text.split()) != row["word_count"]: + raise ValueError(f"indexed report word-count mismatch for {run_id}") + preserved = [ + event + for event in event_records() + if event["event"] == "report_preserved" + and event.get("run_id") == run_id + and event.get("attempt") == attempt + ] + if len(preserved) != 1 or preserved[0].get("sha256") != row["report_sha256"]: + raise ValueError(f"canonical report lacks its preservation event: {run_id}") + attempts = [ + event + for event in event_records() + if event["event"] == "attempt_preserved" + and event.get("run_id") == run_id + and event.get("attempt") == attempt + ] + if len(attempts) != 1 or attempts[0].get("sha256") != byte_tree_digest(directory): + raise ValueError(f"canonical attempt differs from its preservation event: {run_id}") + return result + + +def presentation_order(mode: str, scorer: str) -> list[str]: + claim = f"{mode}-{scorer}" + rows = { + row["claim"]: row + for row in read_frozen_tsv(SEALED / "presentation-orders.tsv") + } + return rows[claim]["labels_in_order"].split(",") + + +def build_score_packets() -> None: + validate_static(require_lock=True, announce=False) + index = load_index() + if set(index) != set(load_schedule()): + raise ValueError( + f"collection does not contain exactly all {TOTAL_REPORTS} scheduled reports" + ) + if any(event["event"] == "collection_locked" for event in event_records()): + raise ValueError("collection was already locked") + append_event( + "collection", + "collection_locked", + digest=sha256_file(COLLECTION / "valid-index.jsonl"), + details={"report_count": TOTAL_REPORTS}, + ) + (COLLECTION / "valid-index.jsonl").chmod(0o444) + verify_collection_lock() + blind = load_blind_map() + target_by_mode = {row["mode"]: row for row in load_target_map().values()} + packet_root = SCORING / "packets" + if packet_root.exists(): + raise FileExistsError("scorer packets already exist") + packet_digests: dict[str, str] = {} + for mode in MODES: + target_source = EVALS / target_by_mode[mode]["source_path"] + for scorer in SCORERS: + packet = packet_root / mode / scorer + (packet / "reports").mkdir(parents=True) + shutil.copytree(target_source, packet / "target") + shutil.copy2(FREEZE / "allowlists" / f"{mode}.txt", packet / "allowlist.txt") + shutil.copy2(FREEZE / "rubrics" / "SCORER.md", packet / "SCORER.md") + shutil.copy2(FREEZE / "rubrics" / f"{mode}.md", packet / "RUBRIC.md") + shutil.copy2(FREEZE / "schemas" / "score.schema.json", packet / "score.schema.json") + if byte_tree_digest(packet / "target") != target_by_mode[mode]["tree_sha256"]: + raise ValueError(f"copied target lost its frozen identity: {mode}/{scorer}") + frozen_copies = { + packet / "allowlist.txt": FREEZE / "allowlists" / f"{mode}.txt", + packet / "SCORER.md": FREEZE / "rubrics" / "SCORER.md", + packet / "RUBRIC.md": FREEZE / "rubrics" / f"{mode}.md", + packet / "score.schema.json": FREEZE / "schemas" / "score.schema.json", + } + for destination, source in frozen_copies.items(): + if sha256_file(destination) != frozen_file_digest(source): + raise ValueError(f"frozen packet input changed during copy: {destination}") + report_hashes: dict[str, str] = {} + for label, run_id in blind[mode].items(): + attempt = index[run_id]["attempt"] + source = COLLECTION / "attempts" / run_id / str(attempt) / "report.md" + if sha256_file(source) != index[run_id]["report_sha256"]: + raise ValueError(f"report changed before packet build: {run_id}") + destination = packet / "reports" / f"{label}.md" + shutil.copy2(source, destination) + report_hashes[label] = sha256_file(destination) + if report_hashes[label] != index[run_id]["report_sha256"]: + raise ValueError(f"report changed during packet copy: {run_id}") + manifest = { + "schema_version": 1, + "mode": mode, + "scorer_id": scorer, + "presentation_order": presentation_order(mode, scorer), + "report_sha256": report_hashes, + "target_byte_tree_sha256": byte_tree_digest(packet / "target"), + "allowlist_sha256": sha256_file(packet / "allowlist.txt"), + "common_rules_sha256": sha256_file(packet / "SCORER.md"), + "rubric_sha256": sha256_file(packet / "RUBRIC.md"), + "schema_sha256": sha256_file(packet / "score.schema.json"), + } + write_once(packet / "PACKET.json", json_dump(manifest)) + make_read_only(packet) + packet_digest = byte_tree_digest(packet) + packet_digests[f"{mode}-{scorer}"] = packet_digest + append_event( + "scoring", + "blind_packet_preserved", + digest=packet_digest, + details={"mode": mode, "scorer": scorer}, + ) + digest = sha256_bytes(json.dumps(packet_digests, sort_keys=True).encode()) + append_event( + "scoring", + "blind_packets_built", + digest=digest, + details={"packet_count": len(MODES) * len(SCORERS)}, + ) + print(f"built {len(MODES) * len(SCORERS)} blind scorer packets") + + +def verify_score_packet(mode: str, scorer: str) -> None: + verify_collection_lock() + packet = SCORING / "packets" / mode / scorer + manifest = json.loads((packet / "PACKET.json").read_text()) + if ( + type(manifest.get("schema_version")) is not int + or manifest["schema_version"] != 1 + or manifest.get("mode") != mode + or manifest.get("scorer_id") != scorer + ): + raise ValueError(f"score packet identity mismatch: {mode}/{scorer}") + if manifest.get("presentation_order") != presentation_order(mode, scorer): + raise ValueError(f"score packet presentation mismatch: {mode}/{scorer}") + checks = { + "target_byte_tree_sha256": byte_tree_digest(packet / "target"), + "allowlist_sha256": sha256_file(packet / "allowlist.txt"), + "common_rules_sha256": sha256_file(packet / "SCORER.md"), + "rubric_sha256": sha256_file(packet / "RUBRIC.md"), + "schema_sha256": sha256_file(packet / "score.schema.json"), + } + for field, actual in checks.items(): + if manifest.get(field) != actual: + raise ValueError(f"score packet changed: {mode}/{scorer}/{field}") + report_hashes = { + label: sha256_file(packet / "reports" / f"{label}.md") for label in LABELS + } + if manifest.get("report_sha256") != report_hashes: + raise ValueError(f"score packet reports changed: {mode}/{scorer}") + expected_files = { + "PACKET.json", + "allowlist.txt", + "RUBRIC.md", + "SCORER.md", + "score.schema.json", + *(f"reports/{label}.md" for label in LABELS), + } + expected_files.update( + path.relative_to(packet).as_posix() + for path in (packet / "target").rglob("*") + if path.is_file() + ) + actual_files = { + path.relative_to(packet).as_posix() for path in packet.rglob("*") if path.is_file() + } + if actual_files != expected_files: + raise ValueError(f"score packet has unexpected files: {mode}/{scorer}") + expected_packet_digest = preserved_digest( + "blind_packet_preserved", mode=mode, scorer=scorer + ) + if byte_tree_digest(packet) != expected_packet_digest: + raise ValueError(f"score packet differs from its external preservation event: {mode}/{scorer}") + + +def render_packet_prompt(template_name: str, packet: Path, output: Path, **values: str) -> str: + template = read_frozen_text(FREEZE / "prompts" / template_name) + match = re.search(r"```text\n(.*?)\n```", template, flags=re.DOTALL) + if not match: + raise ValueError(f"{template_name} missing text fence") + prompt = match.group(1).replace("[PACKET]", str(packet)).replace("[OUTPUT]", str(output)) + for key, value in values.items(): + prompt = prompt.replace(f"[{key}]", value) + if re.search(r"\[[A-Z_]+\]", prompt): + raise ValueError(f"unresolved placeholder in {template_name}") + return prompt + + +def expected_evaluator_output(kind: str, identity: str, attempt: int) -> Path: + if attempt < 1: + raise ValueError("evaluator attempt must be positive") + if kind not in EVALUATOR_KINDS: + raise ValueError(f"unknown evaluator kind: {kind}") + seeds = load_frozen_seeds() + seed_name = { + "scorer": "scorer", + "consistency": "consistency", + "adjudicator": "presentation", + }[kind] + token = prepare.keyed( + f"{kind}-output-v4", seeds[seed_name], f"{identity}|{attempt}" + )[:32] + return Path("/tmp/ur-eval") / token / "output" + + +def expected_evaluator_packet(kind: str, identity: str, attempt: int) -> Path: + return expected_evaluator_output(kind, identity, attempt).parent / "packet" + + +def expected_source_packet_digest(kind: str, identity: str) -> str: + if kind == "scorer": + mode, scorer = identity.split("-", 1) + return preserved_digest("blind_packet_preserved", mode=mode, scorer=scorer) + if kind == "consistency": + return preserved_digest("consistency_packet_preserved", mode=identity) + return preserved_digest("adjudication_packet_preserved", mode=identity) + + +def prepare_evaluator_runtime( + kind: str, identity: str, attempt: int, source_packet: Path, output: Path +) -> Path: + assert_evaluator_attempt_allowed(kind, identity, attempt) + expected = expected_evaluator_output(kind, identity, attempt) + if output != expected: + raise ValueError(f"{kind} output must be the frozen neutral path {expected}") + root = output.parent + if root.exists(): + verify_evaluator_runtime(kind, identity, attempt, source_packet, output) + if any(output.iterdir()): + raise ValueError(f"existing {kind} prelaunch output is not empty") + return expected_evaluator_packet(kind, identity, attempt) + runtime_packet = expected_evaluator_packet(kind, identity, attempt) + root.mkdir(parents=True) + shutil.copytree(source_packet, runtime_packet) + if byte_tree_digest(runtime_packet) != expected_source_packet_digest(kind, identity): + raise ValueError(f"{kind} runtime packet differs from frozen packet") + make_read_only(runtime_packet) + output.mkdir() + os.utime(root, (0, 0), follow_symlinks=False) + root.chmod(0o555) + return runtime_packet + + +def verify_evaluator_runtime( + kind: str, + identity: str, + attempt: int, + source_packet: Path, + output: Path, + *, + allow_invalid_output: bool = False, +) -> None: + if output != expected_evaluator_output(kind, identity, attempt): + raise ValueError(f"non-neutral {kind} output path") + root = output.parent + if root.is_symlink() or not root.is_dir() or root.stat().st_mode & 0o222: + raise ValueError(f"invalid {kind} runtime root") + actual_entries = {entry.name for entry in root.iterdir()} + if ( + (not allow_invalid_output and actual_entries != {"packet", "output"}) + or ( + allow_invalid_output + and ("packet" not in actual_entries or actual_entries - {"packet", "output"}) + ) + ): + raise ValueError(f"unexpected {kind} runtime inventory") + if not allow_invalid_output and ( + not output.exists() + or output.is_symlink() + or not stat.S_ISDIR(output.lstat().st_mode) + ): + raise ValueError(f"invalid {kind} output directory") + runtime_packet = expected_evaluator_packet(kind, identity, attempt) + if byte_tree_digest(source_packet) != expected_source_packet_digest(kind, identity): + raise ValueError(f"{kind} source packet changed") + if byte_tree_digest(runtime_packet) != expected_source_packet_digest(kind, identity): + raise ValueError(f"{kind} runtime packet changed") + + +def evaluator_runtime_forensics( + kind: str, + identity: str, + attempt: int, + source_packet: Path, + output: Path, + error: Exception, +) -> dict[str, Any]: + expected_output = expected_evaluator_output(kind, identity, attempt) + if output != expected_output: + raise ValueError(f"forensic evaluator output is not the neutral path {expected_output}") + root = output.parent + root_is_real = root.exists() and not root.is_symlink() and root.is_dir() + return { + "schema_version": 1, + "kind": kind, + "identity": identity, + "attempt": attempt, + "verification_error": f"{type(error).__name__}: {error}", + "expected": { + "output": str(expected_output), + "source_packet_byte_tree_sha256": expected_source_packet_digest(kind, identity), + }, + "observed": { + "source_packet": observe_path(source_packet), + "runtime_root": observe_path(root), + "packet": observe_path(root / "packet") + if root_is_real + else {"type": "unavailable"}, + "output": observe_path(output) + if root_is_real + else {"type": "unavailable"}, + }, + } + + +def validate_score(value: Any, mode: str, scorer: str) -> None: + if ( + not isinstance(value, dict) + or type(value.get("schema_version")) is not int + or value["schema_version"] != 1 + ): + raise ValueError("invalid score envelope") + if set(value) != {"schema_version", "mode", "scorer_id", "reports", "ambiguities"}: + raise ValueError("unexpected score fields") + if value.get("mode") != mode or value.get("scorer_id") != scorer: + raise ValueError("score mode or scorer mismatch") + if not isinstance(value["ambiguities"], list) or any( + not is_nonblank_string(item) for item in value["ambiguities"] + ): + raise ValueError("invalid score ambiguities") + reports = value.get("reports") + if ( + not isinstance(reports, list) + or len(reports) != 10 + or any(not isinstance(report, dict) for report in reports) + ): + raise ValueError("score must contain ten reports") + report_labels = [report.get("label") for report in reports] + if any(not isinstance(label, str) for label in report_labels) or tuple( + sorted(report_labels) + ) != LABELS: + raise ValueError("score labels must be exactly A-J") + expected_atoms = set(atom_ids(mode)) + allowed_hard = set(hard_error_ids(mode)) + for report in reports: + if set(report) != { + "label", + "atoms", + "hard_errors", + "proposal_laundering", + "tcb_authority_defect", + "visible_scope_defect", + "novel_findings", + }: + raise ValueError("unexpected report-score fields") + atoms = report.get("atoms") + if ( + not isinstance(atoms, list) + or any(not isinstance(atom, dict) for atom in atoms) + or {atom.get("id") for atom in atoms} != expected_atoms + ): + raise ValueError(f"{mode}/{report.get('label')}: incomplete atom set") + if len(atoms) != len(expected_atoms): + raise ValueError(f"{mode}/{report.get('label')}: duplicate atom ID") + for atom in atoms: + if set(atom) != {"id", "decision", "evidence"}: + raise ValueError("unexpected atom fields") + if ( + atom.get("decision") not in {"PASS", "FAIL"} + or not is_nonblank_string(atom.get("evidence")) + ): + raise ValueError("invalid atom decision") + hard = report.get("hard_errors") + if not isinstance(hard, list) or any( + not isinstance(finding, dict) for finding in hard + ): + raise ValueError("hard_errors must be a list") + hard_ids = [finding.get("id") for finding in hard] + if len(hard_ids) != len(set(hard_ids)) or not set(hard_ids) <= allowed_hard: + raise ValueError(f"invalid hard-error IDs: {hard_ids}") + if any( + not is_nonblank_string(finding.get("evidence")) + for finding in hard + ): + raise ValueError("hard error lacks evidence") + if any(set(finding) != {"id", "evidence"} for finding in hard): + raise ValueError("unexpected hard-error fields") + for field in GLOBAL_DEFECT_FIELDS: + flag = report.get(field) + if not isinstance(flag, dict) or not isinstance(flag.get("present"), bool): + raise ValueError(f"invalid {field} flag") + if set(flag) != {"present", "evidence"} or not is_nonblank_string( + flag.get("evidence") + ): + raise ValueError(f"invalid {field} evidence") + novel = report.get("novel_findings") + if not isinstance(novel, list) or any( + not isinstance(finding, dict) for finding in novel + ): + raise ValueError("novel_findings must be a list") + novel_ids = [finding.get("id") for finding in novel] + if len(novel_ids) != len(set(novel_ids)) or any( + not re.fullmatch(r"N[1-9][0-9]*", str(identifier)) for identifier in novel_ids + ): + raise ValueError("invalid novel-finding IDs") + if any( + not is_nonblank_string(finding.get("evidence")) + for finding in novel + ): + raise ValueError("novel finding lacks evidence") + if any(set(finding) != {"id", "evidence"} for finding in novel): + raise ValueError("unexpected novel-finding fields") + + +def record_score( + mode: str, scorer: str, attempt: int, output: Path, agent_id: str +) -> None: + assert_evaluator_started("scorer", f"{mode}-{scorer}", attempt, agent_id) + verify_score_packet(mode, scorer) + verify_evaluator_runtime( + "scorer", + f"{mode}-{scorer}", + attempt, + SCORING / "packets" / mode / scorer, + output, + ) + source = output / "score.json" + entries = output_entries(output) + if [entry.name for entry in entries] != ["score.json"]: + raise ValueError("scorer output is not exactly score.json") + raw = source.read_bytes() + value = json.loads(raw.decode("utf-8")) + validate_score(value, mode, scorer) + attempt_directory = ( + SCORING + / "evaluator-attempts" + / "scoring" + / f"{mode}-{scorer}" + / str(attempt) + ) + entries = preserve_captured_file(attempt_directory / "raw-output", "score.json", raw) + write_once(attempt_directory / "raw-output-manifest.json", json_dump(entries)) + attempt_attestation = { + "schema_version": 1, + "phase": "scoring", + "kind": "scorer", + "identity": f"{mode}-{scorer}", + "attempt": attempt, + "agent_id": agent_id, + "disposition": "COMPLETE", + "recorded_utc": utc_now(), + } + write_once(attempt_directory / "attestation.json", json_dump(attempt_attestation)) + make_read_only(attempt_directory) + append_event( + "scoring", + "evaluator_attempt_preserved", + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(attempt_directory), + details={"kind": "scorer", "identity": f"{mode}-{scorer}", "disposition": "COMPLETE"}, + ) + destination = SCORING / "raw" / mode / f"{scorer}.json" + write_bytes_once(destination, raw) + destination.chmod(0o444) + append_event( + "scoring", + "evaluator_returned", + attempt=attempt, + agent_id=agent_id, + details={"kind": "scorer", "identity": f"{mode}-{scorer}", "api_state": "COMPLETED"}, + ) + append_event( + "scoring", + "score_preserved", + attempt=attempt, + agent_id=agent_id, + digest=sha256_file(destination), + details={"mode": mode, "scorer": scorer}, + ) + print(destination) + + +def report_by_label(score: dict[str, Any]) -> dict[str, dict[str, Any]]: + return {report["label"]: report for report in score["reports"]} + + +def preserved_digest(event_name: str, **details: str) -> str: + matches = [ + event + for event in event_records() + if event["event"] == event_name + and all(event.get("details", {}).get(key) == value for key, value in details.items()) + ] + if len(matches) != 1 or "sha256" not in matches[0]: + raise ValueError(f"expected one preserved digest for {event_name} {details}") + return matches[0]["sha256"] + + +def load_preserved_score(mode: str, scorer: str) -> dict[str, Any]: + path = SCORING / "raw" / mode / f"{scorer}.json" + expected = preserved_digest("score_preserved", mode=mode, scorer=scorer) + if sha256_file(path) != expected: + raise ValueError(f"raw score changed after preservation: {mode}/{scorer}") + value = json.loads(path.read_text()) + validate_score(value, mode, scorer) + return value + + +def build_consistency_packet(mode: str) -> None: + scores = {scorer: load_preserved_score(mode, scorer) for scorer in SCORERS} + packet = SCORING / "consistency-packets" / mode + if packet.exists(): + raise FileExistsError(f"consistency packet exists: {packet}") + source_packet = SCORING / "packets" / mode / "s1" + verify_score_packet(mode, "s1") + source_packet_digest = preserved_digest( + "blind_packet_preserved", mode=mode, scorer="s1" + ) + source_manifest = json.loads((source_packet / "PACKET.json").read_text()) + (packet / "reports").mkdir(parents=True) + (packet / "raw-scores").mkdir() + shutil.copytree(source_packet / "target", packet / "target") + for name in ("allowlist.txt", "SCORER.md", "RUBRIC.md"): + shutil.copy2(source_packet / name, packet / name) + shutil.copy2( + FREEZE / "schemas" / "consistency.schema.json", + packet / "consistency.schema.json", + ) + for label in LABELS: + source = source_packet / "reports" / f"{label}.md" + destination = packet / "reports" / f"{label}.md" + shutil.copy2(source, destination) + if sha256_file(destination) != source_manifest["report_sha256"][label]: + raise ValueError( + f"consistency report changed during copy: {mode}/{label}" + ) + raw_score_hashes: dict[str, str] = {} + for scorer in SCORERS: + source = SCORING / "raw" / mode / f"{scorer}.json" + destination = packet / "raw-scores" / f"{scorer}.json" + shutil.copy2(source, destination) + raw_score_hashes[scorer] = sha256_file(destination) + if raw_score_hashes[scorer] != preserved_digest( + "score_preserved", mode=mode, scorer=scorer + ): + raise ValueError( + f"consistency raw score changed during copy: {mode}/{scorer}" + ) + validate_score(scores[scorer], mode, scorer) + if ( + byte_tree_digest(packet / "target") + != source_manifest["target_byte_tree_sha256"] + ): + raise ValueError(f"consistency target changed during copy: {mode}") + expected_copies = { + "allowlist.txt": "allowlist_sha256", + "SCORER.md": "common_rules_sha256", + "RUBRIC.md": "rubric_sha256", + } + for name, field in expected_copies.items(): + if sha256_file(packet / name) != source_manifest[field]: + raise ValueError(f"consistency packet input changed: {mode}/{name}") + if sha256_file(packet / "consistency.schema.json") != frozen_file_digest( + FREEZE / "schemas" / "consistency.schema.json" + ): + raise ValueError(f"consistency schema changed during copy: {mode}") + manifest = { + "schema_version": 1, + "mode": mode, + "source_score_packet_sha256": source_packet_digest, + "report_sha256": { + label: sha256_file(packet / "reports" / f"{label}.md") + for label in LABELS + }, + "raw_score_sha256": raw_score_hashes, + "target_byte_tree_sha256": byte_tree_digest(packet / "target"), + "allowlist_sha256": sha256_file(packet / "allowlist.txt"), + "common_rules_sha256": sha256_file(packet / "SCORER.md"), + "rubric_sha256": sha256_file(packet / "RUBRIC.md"), + "schema_sha256": sha256_file(packet / "consistency.schema.json"), + } + write_once(packet / "PACKET.json", json_dump(manifest)) + make_read_only(packet) + digest = byte_tree_digest(packet) + append_event( + "consistency", + "consistency_packet_preserved", + digest=digest, + details={"mode": mode, "source_score_packet_sha256": source_packet_digest}, + ) + print(packet) + + +def verify_consistency_packet(mode: str) -> None: + packet = SCORING / "consistency-packets" / mode + manifest = json.loads((packet / "PACKET.json").read_text()) + if ( + type(manifest.get("schema_version")) is not int + or manifest["schema_version"] != 1 + or manifest.get("mode") != mode + ): + raise ValueError(f"consistency packet identity mismatch: {mode}") + verify_score_packet(mode, "s1") + source_digest = preserved_digest( + "blind_packet_preserved", mode=mode, scorer="s1" + ) + if manifest.get("source_score_packet_sha256") != source_digest: + raise ValueError(f"consistency packet source binding changed: {mode}") + checks = { + "target_byte_tree_sha256": byte_tree_digest(packet / "target"), + "allowlist_sha256": sha256_file(packet / "allowlist.txt"), + "common_rules_sha256": sha256_file(packet / "SCORER.md"), + "rubric_sha256": sha256_file(packet / "RUBRIC.md"), + "schema_sha256": sha256_file(packet / "consistency.schema.json"), + } + for field, actual in checks.items(): + if manifest.get(field) != actual: + raise ValueError(f"consistency packet changed: {mode}/{field}") + report_hashes = { + label: sha256_file(packet / "reports" / f"{label}.md") + for label in LABELS + } + if manifest.get("report_sha256") != report_hashes: + raise ValueError(f"consistency packet reports changed: {mode}") + raw_score_hashes = { + scorer: sha256_file(packet / "raw-scores" / f"{scorer}.json") + for scorer in SCORERS + } + if manifest.get("raw_score_sha256") != raw_score_hashes: + raise ValueError(f"consistency packet raw scores changed: {mode}") + for scorer in SCORERS: + expected = preserved_digest("score_preserved", mode=mode, scorer=scorer) + if raw_score_hashes[scorer] != expected: + raise ValueError(f"consistency packet score binding changed: {mode}/{scorer}") + value = json.loads((packet / "raw-scores" / f"{scorer}.json").read_text()) + validate_score(value, mode, scorer) + expected_files = { + "PACKET.json", + "allowlist.txt", + "RUBRIC.md", + "SCORER.md", + "consistency.schema.json", + *(f"reports/{label}.md" for label in LABELS), + *(f"raw-scores/{scorer}.json" for scorer in SCORERS), + } + expected_files.update( + path.relative_to(packet).as_posix() + for path in (packet / "target").rglob("*") + if path.is_file() + ) + actual_files = { + path.relative_to(packet).as_posix() + for path in packet.rglob("*") + if path.is_file() + } + if actual_files != expected_files: + raise ValueError(f"consistency packet has unexpected files: {mode}") + expected_packet_digest = preserved_digest( + "consistency_packet_preserved", mode=mode + ) + if byte_tree_digest(packet) != expected_packet_digest: + raise ValueError( + f"consistency packet differs from its external event: {mode}" + ) + + +def score_position(report: dict[str, Any], field: str) -> dict[str, str]: + if field.startswith("atom:"): + identifier = field.removeprefix("atom:") + atom = next(item for item in report["atoms"] if item["id"] == identifier) + return {"decision": atom["decision"], "evidence": atom["evidence"]} + if field.startswith("hard_error:"): + identifier = field.removeprefix("hard_error:") + present = any(item["id"] == identifier for item in report["hard_errors"]) + return { + "decision": "PRESENT" if present else "ABSENT", + "evidence": evidence_for_hard(report, identifier), + } + if field in GLOBAL_DEFECT_FIELDS: + flag = report[field] + return { + "decision": "PRESENT" if flag["present"] else "ABSENT", + "evidence": flag["evidence"], + } + raise ValueError(f"unsupported scored field: {field}") + + +def valid_review_field(mode: str, field: str) -> bool: + return ( + field in {f"atom:{identifier}" for identifier in atom_ids(mode)} + or field + in {f"hard_error:{identifier}" for identifier in hard_error_ids(mode)} + or field in GLOBAL_DEFECT_FIELDS + ) + + +def validate_consistency_review( + value: Any, + mode: str, + scores: dict[str, dict[str, Any]] | None = None, +) -> None: + if ( + not isinstance(value, dict) + or type(value.get("schema_version")) is not int + or value["schema_version"] != 1 + or value.get("mode") != mode + ): + raise ValueError("invalid consistency-review envelope") + if set(value) != { + "schema_version", + "mode", + "atom_family_attestations", + "defect_family_attestations", + "challenges", + "ambiguities", + }: + raise ValueError("unexpected consistency-review fields") + if not isinstance(value["ambiguities"], list) or any( + not is_nonblank_string(item) for item in value["ambiguities"] + ): + raise ValueError("invalid consistency-review ambiguities") + attestations = value.get("atom_family_attestations") + if not isinstance(attestations, list) or any( + not isinstance(item, dict) for item in attestations + ): + raise ValueError("atom-family attestations must be a list") + if any( + set(item) != {"atom_id", "all_ten_reports_compared", "evidence"} + or item.get("all_ten_reports_compared") is not True + or not is_nonblank_string(item.get("evidence")) + for item in attestations + ): + raise ValueError("invalid atom-family attestation") + attested_atoms = [item.get("atom_id") for item in attestations] + if len(attested_atoms) != len(set(attested_atoms)) or set(attested_atoms) != set( + atom_ids(mode) + ): + raise ValueError("consistency review does not attest every atom family exactly once") + defect_attestations = value.get("defect_family_attestations") + if not isinstance(defect_attestations, list) or any( + not isinstance(item, dict) for item in defect_attestations + ): + raise ValueError("defect-family attestations must be a list") + if any( + set(item) != {"field", "all_ten_reports_compared", "evidence"} + or item.get("all_ten_reports_compared") is not True + or not is_nonblank_string(item.get("evidence")) + for item in defect_attestations + ): + raise ValueError("invalid defect-family attestation") + attested_defects = [item.get("field") for item in defect_attestations] + required_defects = { + *(f"hard_error:{identifier}" for identifier in hard_error_ids(mode)), + *GLOBAL_DEFECT_FIELDS, + } + if ( + len(attested_defects) != len(set(attested_defects)) + or set(attested_defects) != required_defects + ): + raise ValueError( + "consistency review does not attest every defect family exactly once" + ) + challenges = value.get("challenges") + if not isinstance(challenges, list) or any( + not isinstance(challenge, dict) for challenge in challenges + ): + raise ValueError("consistency-review challenges must be a list") + identities: list[tuple[str, str]] = [] + for challenge in challenges: + if set(challenge) != { + "label", + "field", + "recommended_decision", + "evidence", + }: + raise ValueError("unexpected consistency-review challenge fields") + label = challenge.get("label") + field = challenge.get("field") + decision = challenge.get("recommended_decision") + if label not in LABELS or not isinstance(field, str) or not valid_review_field( + mode, field + ): + raise ValueError("invalid consistency-review challenge identity") + expected_decisions = ( + {"PASS", "FAIL"} if field.startswith("atom:") else {"PRESENT", "ABSENT"} + ) + if decision not in expected_decisions or not is_nonblank_string( + challenge.get("evidence") + ): + raise ValueError("invalid consistency-review challenge decision") + identities.append((label, field)) + if len(identities) != len(set(identities)): + raise ValueError("duplicate consistency-review challenge") + if scores is not None: + by_scorer = { + scorer: report_by_label(score) for scorer, score in scores.items() + } + for challenge in challenges: + positions = { + score_position(by_scorer[scorer][challenge["label"]], challenge["field"])[ + "decision" + ] + for scorer in SCORERS + } + if positions == {challenge["recommended_decision"]}: + raise ValueError( + "consistency challenge does not challenge either raw decision" + ) + + +def load_preserved_consistency_review(mode: str) -> dict[str, Any]: + path = SCORING / "consistency" / f"{mode}.json" + expected = preserved_digest("consistency_review_preserved", mode=mode) + if sha256_file(path) != expected: + raise ValueError(f"consistency review changed after preservation: {mode}") + value = json.loads(path.read_text()) + scores = {scorer: load_preserved_score(mode, scorer) for scorer in SCORERS} + validate_consistency_review(value, mode, scores) + return value + + +def evidence_for_hard(report: dict[str, Any], identifier: str) -> str: + for finding in report["hard_errors"]: + if finding["id"] == identifier: + return finding["evidence"] + return "No applicable hard error recorded." + + +def adjudication_cells( + mode: str, + s1: dict[str, Any], + s2: dict[str, Any], + consistency_review: dict[str, Any], +) -> list[dict[str, Any]]: + cells: list[dict[str, Any]] = [] + by_scorer = {"s1": report_by_label(s1), "s2": report_by_label(s2)} + for label in LABELS: + first = by_scorer["s1"][label] + second = by_scorer["s2"][label] + atoms1 = {atom["id"]: atom for atom in first["atoms"]} + atoms2 = {atom["id"]: atom for atom in second["atoms"]} + for atom in atom_ids(mode): + if atoms1[atom]["decision"] != atoms2[atom]["decision"]: + cells.append( + { + "label": label, + "field": f"atom:{atom}", + "s1": {"decision": atoms1[atom]["decision"], "evidence": atoms1[atom]["evidence"]}, + "s2": {"decision": atoms2[atom]["decision"], "evidence": atoms2[atom]["evidence"]}, + } + ) + for identifier in hard_error_ids(mode): + present1 = any(item["id"] == identifier for item in first["hard_errors"]) + present2 = any(item["id"] == identifier for item in second["hard_errors"]) + # Every positive hard-error candidate is adjudicated, including an + # agreed positive. Agreement cannot turn a severe finding into an + # unchecked final result. + if present1 or present2: + cells.append( + { + "label": label, + "field": f"hard_error:{identifier}", + "s1": {"decision": "PRESENT" if present1 else "ABSENT", "evidence": evidence_for_hard(first, identifier)}, + "s2": {"decision": "PRESENT" if present2 else "ABSENT", "evidence": evidence_for_hard(second, identifier)}, + } + ) + for field in GLOBAL_DEFECT_FIELDS: + flag1 = first[field] + flag2 = second[field] + # Apply the same independent confirmation rule to global defects. + if flag1["present"] or flag2["present"]: + cells.append( + { + "label": label, + "field": field, + "s1": {"decision": "PRESENT" if flag1["present"] else "ABSENT", "evidence": flag1["evidence"]}, + "s2": {"decision": "PRESENT" if flag2["present"] else "ABSENT", "evidence": flag2["evidence"]}, + } + ) + for scorer, report, other in (("s1", first, "s2"), ("s2", second, "s1")): + for finding in report["novel_findings"]: + cells.append( + { + "label": label, + "field": f"novel:{scorer}:{finding['id']}", + scorer: {"decision": "PRESENT", "evidence": finding["evidence"]}, + other: {"decision": "ABSENT", "evidence": "Not independently proposed; adjudicate the candidate on its merits."}, + } + ) + by_key = {(cell["label"], cell["field"]): cell for cell in cells} + if len(by_key) != len(cells): + raise ValueError("duplicate adjudication-cell field") + reports = {scorer: report_by_label(score) for scorer, score in (("s1", s1), ("s2", s2))} + for challenge in consistency_review["challenges"]: + key = (challenge["label"], challenge["field"]) + if key not in by_key: + cell = { + "label": challenge["label"], + "field": challenge["field"], + "s1": score_position(reports["s1"][challenge["label"]], challenge["field"]), + "s2": score_position(reports["s2"][challenge["label"]], challenge["field"]), + } + cells.append(cell) + by_key[key] = cell + by_key[key]["consistency_review"] = { + "decision": challenge["recommended_decision"], + "evidence": challenge["evidence"], + } + keys = [(cell["label"], cell["field"]) for cell in cells] + if len(keys) != len(set(keys)): + raise ValueError("duplicate adjudication-cell field") + return cells + + +def build_adjudication_cells(mode: str) -> None: + scores: dict[str, dict[str, Any]] = {} + for scorer in SCORERS: + scores[scorer] = load_preserved_score(mode, scorer) + consistency_review = load_preserved_consistency_review(mode) + value = { + "schema_version": 1, + "mode": mode, + "cells": adjudication_cells( + mode, scores["s1"], scores["s2"], consistency_review + ), + } + destination = SCORING / "adjudication-cells" / f"{mode}.json" + write_once(destination, json_dump(value)) + destination.chmod(0o444) + append_event( + "adjudication", + "adjudication_cells_materialized", + digest=sha256_file(destination), + details={"mode": mode, "count": len(value["cells"])}, + ) + print( + f"{mode}: {len(value['cells'])} scorer-disputed, positive-defect, challenged, " + "or novel cells" + ) + + +def build_adjudication_packet(mode: str) -> None: + cells_path = SCORING / "adjudication-cells" / f"{mode}.json" + if sha256_file(cells_path) != preserved_digest( + "adjudication_cells_materialized", mode=mode + ): + raise ValueError(f"adjudication cells changed before packet build: {mode}") + cells_document = json.loads(cells_path.read_text()) + if not cells_document["cells"]: + print(f"{mode}: no adjudication packet required") + return + packet = SCORING / "adjudication-packets" / mode + if packet.exists(): + raise FileExistsError(f"adjudication packet exists: {packet}") + (packet / "reports").mkdir(parents=True) + source_packet = SCORING / "packets" / mode / "s1" + verify_score_packet(mode, "s1") + source_packet_digest = preserved_digest("blind_packet_preserved", mode=mode, scorer="s1") + source_manifest = json.loads((source_packet / "PACKET.json").read_text()) + consistency_path = SCORING / "consistency" / f"{mode}.json" + consistency_digest = preserved_digest("consistency_review_preserved", mode=mode) + if sha256_file(consistency_path) != consistency_digest: + raise ValueError(f"consistency review changed before adjudication: {mode}") + shutil.copytree(source_packet / "target", packet / "target") + for name in ("allowlist.txt", "SCORER.md", "RUBRIC.md"): + shutil.copy2(source_packet / name, packet / name) + shutil.copy2(FREEZE / "schemas" / "adjudication.schema.json", packet / "adjudication.schema.json") + shutil.copy2(cells_path, packet / "ADJUDICATION_CELLS.json") + shutil.copy2(consistency_path, packet / "CONSISTENCY.json") + listed_labels = sorted({cell["label"] for cell in cells_document["cells"]}) + for label in listed_labels: + shutil.copy2(source_packet / "reports" / f"{label}.md", packet / "reports" / f"{label}.md") + if ( + sha256_file(packet / "reports" / f"{label}.md") + != source_manifest["report_sha256"][label] + ): + raise ValueError(f"adjudication report changed during copy: {mode}/{label}") + # The source score packet is read-only. Authenticate the copy by the one + # canonical byte-tree identity, which intentionally excludes permission + # metadata and therefore survives hardening and copying. + if ( + byte_tree_digest(packet / "target") + != source_manifest["target_byte_tree_sha256"] + ): + raise ValueError(f"adjudication target lost byte identity: {mode}") + if sha256_file(packet / "ADJUDICATION_CELLS.json") != preserved_digest( + "adjudication_cells_materialized", mode=mode + ): + raise ValueError(f"adjudication cells changed during copy: {mode}") + if sha256_file(packet / "CONSISTENCY.json") != consistency_digest: + raise ValueError(f"adjudication consistency review changed during copy: {mode}") + expected_copies = { + "allowlist.txt": "allowlist_sha256", + "SCORER.md": "common_rules_sha256", + "RUBRIC.md": "rubric_sha256", + } + for name, field in expected_copies.items(): + if sha256_file(packet / name) != source_manifest[field]: + raise ValueError(f"adjudication packet input changed during copy: {mode}/{name}") + if sha256_file(packet / "adjudication.schema.json") != frozen_file_digest( + FREEZE / "schemas" / "adjudication.schema.json" + ): + raise ValueError(f"adjudication schema changed during copy: {mode}") + manifest = { + "schema_version": 1, + "mode": mode, + "source_score_packet_sha256": source_packet_digest, + "listed_labels": listed_labels, + "adjudication_cells_sha256": sha256_file( + packet / "ADJUDICATION_CELLS.json" + ), + "consistency_review_sha256": sha256_file(packet / "CONSISTENCY.json"), + "report_sha256": { + label: sha256_file(packet / "reports" / f"{label}.md") + for label in listed_labels + }, + "target_byte_tree_sha256": byte_tree_digest(packet / "target"), + "allowlist_sha256": sha256_file(packet / "allowlist.txt"), + "common_rules_sha256": sha256_file(packet / "SCORER.md"), + "rubric_sha256": sha256_file(packet / "RUBRIC.md"), + "schema_sha256": sha256_file(packet / "adjudication.schema.json"), + } + write_once(packet / "PACKET.json", json_dump(manifest)) + make_read_only(packet) + append_event( + "adjudication", + "adjudication_packet_preserved", + digest=byte_tree_digest(packet), + details={ + "mode": mode, + "source_score_packet_sha256": source_packet_digest, + "consistency_review_sha256": consistency_digest, + }, + ) + print(packet) + + +def verify_adjudication_packet(mode: str) -> None: + packet = SCORING / "adjudication-packets" / mode + manifest = json.loads((packet / "PACKET.json").read_text()) + cells_document = json.loads((packet / "ADJUDICATION_CELLS.json").read_text()) + if ( + type(manifest.get("schema_version")) is not int + or manifest["schema_version"] != 1 + or manifest.get("mode") != mode + or cells_document.get("mode") != mode + ): + raise ValueError(f"adjudication packet identity mismatch: {mode}") + verify_score_packet(mode, "s1") + source_digest = preserved_digest("blind_packet_preserved", mode=mode, scorer="s1") + if manifest.get("source_score_packet_sha256") != source_digest: + raise ValueError(f"adjudication packet source binding changed: {mode}") + checks = { + "adjudication_cells_sha256": sha256_file( + packet / "ADJUDICATION_CELLS.json" + ), + "consistency_review_sha256": sha256_file(packet / "CONSISTENCY.json"), + "target_byte_tree_sha256": byte_tree_digest(packet / "target"), + "allowlist_sha256": sha256_file(packet / "allowlist.txt"), + "common_rules_sha256": sha256_file(packet / "SCORER.md"), + "rubric_sha256": sha256_file(packet / "RUBRIC.md"), + "schema_sha256": sha256_file(packet / "adjudication.schema.json"), + } + for field, actual in checks.items(): + if manifest.get(field) != actual: + raise ValueError(f"adjudication packet changed: {mode}/{field}") + if manifest.get("consistency_review_sha256") != preserved_digest( + "consistency_review_preserved", mode=mode + ): + raise ValueError(f"adjudication consistency binding changed: {mode}") + labels = manifest.get("listed_labels") + expected_labels = sorted( + {cell["label"] for cell in cells_document.get("cells", [])} + ) + if not isinstance(labels, list) or labels != expected_labels: + raise ValueError(f"adjudication packet listed-label inventory changed: {mode}") + report_hashes = { + label: sha256_file(packet / "reports" / f"{label}.md") for label in labels + } + if manifest.get("report_sha256") != report_hashes: + raise ValueError(f"adjudication reports changed: {mode}") + expected_files = { + "PACKET.json", + "CONSISTENCY.json", + "ADJUDICATION_CELLS.json", + "allowlist.txt", + "RUBRIC.md", + "SCORER.md", + "adjudication.schema.json", + *(f"reports/{label}.md" for label in labels), + } + expected_files.update( + path.relative_to(packet).as_posix() + for path in (packet / "target").rglob("*") + if path.is_file() + ) + actual_files = { + path.relative_to(packet).as_posix() for path in packet.rglob("*") if path.is_file() + } + if actual_files != expected_files: + raise ValueError(f"adjudication packet has unexpected files: {mode}") + expected_packet_digest = preserved_digest("adjudication_packet_preserved", mode=mode) + if byte_tree_digest(packet) != expected_packet_digest: + raise ValueError(f"adjudication packet differs from its external event: {mode}") + + +def validate_adjudication(value: Any, mode: str) -> None: + cells_path = SCORING / "adjudication-cells" / f"{mode}.json" + expected_digest = preserved_digest("adjudication_cells_materialized", mode=mode) + if sha256_file(cells_path) != expected_digest: + raise ValueError(f"adjudication cells changed after preservation: {mode}") + cells_document = json.loads(cells_path.read_text()) + expected = { + (cell["label"], cell["field"]) for cell in cells_document["cells"] + } + if ( + not isinstance(value, dict) + or type(value.get("schema_version")) is not int + or value["schema_version"] != 1 + or value.get("mode") != mode + ): + raise ValueError("invalid adjudication envelope") + if set(value) != {"schema_version", "mode", "decisions", "ambiguities"}: + raise ValueError("unexpected adjudication fields") + if not isinstance(value["ambiguities"], list) or any( + not is_nonblank_string(item) for item in value["ambiguities"] + ): + raise ValueError("invalid adjudication ambiguities") + decisions = value.get("decisions") + if not isinstance(decisions, list) or any( + not isinstance(decision, dict) for decision in decisions + ): + raise ValueError("adjudication decisions must be a list") + if any( + not isinstance(decision.get("label"), str) + or not isinstance(decision.get("field"), str) + for decision in decisions + ): + raise ValueError("adjudication decision identity is invalid") + actual = {(decision.get("label"), decision.get("field")) for decision in decisions} + if actual != expected or len(decisions) != len(expected): + raise ValueError("adjudication does not resolve exactly every listed cell") + for decision in decisions: + if set(decision) != {"label", "field", "decision", "evidence"}: + raise ValueError("unexpected adjudication-decision fields") + expected_values = {"PASS", "FAIL"} if decision["field"].startswith("atom:") else {"PRESENT", "ABSENT"} + if ( + decision.get("decision") not in expected_values + or not is_nonblank_string(decision.get("evidence")) + ): + raise ValueError("invalid adjudication decision") + + +def validate_final_score(value: Any, mode: str) -> None: + if ( + not isinstance(value, dict) + or set(value) + != {"schema_version", "mode", "reports", "confirmed_novel_findings"} + or type(value.get("schema_version")) is not int + or value["schema_version"] != 1 + or value.get("mode") != mode + ): + raise ValueError(f"invalid final-score envelope: {mode}") + reports = value.get("reports") + if not isinstance(reports, list) or len(reports) != len(LABELS) or any( + not isinstance(report, dict) for report in reports + ): + raise ValueError(f"invalid final-score reports: {mode}") + labels = [report.get("label") for report in reports] + if len(labels) != len(set(labels)) or set(labels) != set(LABELS): + raise ValueError(f"final-score labels are not exactly A through J: {mode}") + expected_report_fields = { + "label", + "atoms", + "hard_errors", + *GLOBAL_DEFECT_FIELDS, + } + expected_atoms = set(atom_ids(mode)) + expected_hard_errors = set(hard_error_ids(mode)) + for report in reports: + if set(report) != expected_report_fields: + raise ValueError(f"unexpected final report fields: {mode}/{report.get('label')}") + atoms = report.get("atoms") + hard_errors = report.get("hard_errors") + if ( + not isinstance(atoms, dict) + or set(atoms) != expected_atoms + or any(decision not in {"PASS", "FAIL"} for decision in atoms.values()) + ): + raise ValueError(f"incomplete final atoms: {mode}/{report['label']}") + if ( + not isinstance(hard_errors, list) + or len(hard_errors) != len(set(hard_errors)) + or not set(hard_errors) <= expected_hard_errors + ): + raise ValueError(f"invalid final hard errors: {mode}/{report['label']}") + if any(type(report[field]) is not bool for field in GLOBAL_DEFECT_FIELDS): + raise ValueError(f"invalid final defect flag: {mode}/{report['label']}") + findings = value.get("confirmed_novel_findings") + if not isinstance(findings, list) or any( + not isinstance(finding, dict) for finding in findings + ): + raise ValueError(f"invalid confirmed novel findings: {mode}") + identities: list[tuple[str, str]] = [] + for finding in findings: + if ( + set(finding) != {"label", "field", "evidence"} + or finding.get("label") not in LABELS + or not isinstance(finding.get("field"), str) + or not re.fullmatch(r"novel:(s1|s2):N[1-9][0-9]*", finding["field"]) + or not is_nonblank_string(finding.get("evidence")) + ): + raise ValueError(f"invalid confirmed novel finding: {mode}") + identities.append((finding["label"], finding["field"])) + if len(identities) != len(set(identities)): + raise ValueError(f"duplicate confirmed novel finding: {mode}") + + +def synthetic_score(mode: str, scorer: str) -> dict[str, Any]: + reports: list[dict[str, Any]] = [] + for label in LABELS: + reports.append( + { + "label": label, + "atoms": [ + {"id": atom, "decision": "PASS", "evidence": "Synthetic complete evidence."} + for atom in atom_ids(mode) + ], + "hard_errors": [], + "proposal_laundering": {"present": False, "evidence": "No proposal laundering."}, + "tcb_authority_defect": {"present": False, "evidence": "No TCB or authority defect."}, + "visible_scope_defect": {"present": False, "evidence": "No visible source-scope defect."}, + "novel_findings": [], + } + ) + return { + "schema_version": 1, + "mode": mode, + "scorer_id": scorer, + "reports": reports, + "ambiguities": [], + } + + +def synthetic_consistency_review(mode: str) -> dict[str, Any]: + return { + "schema_version": 1, + "mode": mode, + "atom_family_attestations": [ + { + "atom_id": atom, + "all_ten_reports_compared": True, + "evidence": "Compared this atom across reports A through J.", + } + for atom in atom_ids(mode) + ], + "defect_family_attestations": [ + { + "field": field, + "all_ten_reports_compared": True, + "evidence": "Compared this defect family across reports A through J.", + } + for field in ( + *(f"hard_error:{identifier}" for identifier in hard_error_ids(mode)), + *GLOBAL_DEFECT_FIELDS, + ) + ], + "challenges": [], + "ambiguities": [], + } + + +def self_test_byte_tree_identity() -> None: + root = Path(tempfile.mkdtemp(prefix="ur-byte-tree-self-test-", dir="/tmp")) + try: + source = root / "source" + nested = source / "nested" + nested.mkdir(parents=True) + (nested / "target.rs").write_text("pub fn identity() {}\n") + before = byte_tree_digest(source) + make_read_only(source) + if byte_tree_digest(source) != before: + raise AssertionError("hardening changed byte-tree identity") + copied = root / "copied" + shutil.copytree(source, copied) + if byte_tree_digest(copied) != before: + raise AssertionError("copying a hardened tree changed byte-tree identity") + make_read_only(copied) + if byte_tree_digest(copied) != before: + raise AssertionError("rehardening a copied tree changed byte-tree identity") + finally: + for item in sorted(root.rglob("*"), reverse=True): + item.chmod(0o755 if item.is_dir() else 0o644) + root.chmod(0o755) + shutil.rmtree(root) + + +def self_test() -> None: + validate_static(require_lock=False) + for mode in MODES: + first = synthetic_score(mode, "s1") + second = synthetic_score(mode, "s2") + validate_score(first, mode, "s1") + validate_score(second, mode, "s2") + first_report = first["reports"][0] + second_report = second["reports"][0] + first_report["atoms"][0]["decision"] = "FAIL" + agreed_hard = [ + {"id": hard_error_ids(mode)[0], "evidence": "Synthetic hard-error evidence."} + ] + first_report["hard_errors"] = agreed_hard + second_report["hard_errors"] = [dict(agreed_hard[0])] + agreed_flag = {"present": True, "evidence": "Synthetic flag evidence."} + first_report["proposal_laundering"] = agreed_flag + second_report["proposal_laundering"] = dict(agreed_flag) + first_report["novel_findings"] = [{"id": "N1", "evidence": "Synthetic novel candidate."}] + consistency = synthetic_consistency_review(mode) + consistency["challenges"] = [ + { + "label": "B", + "field": f"atom:{atom_ids(mode)[0]}", + "recommended_decision": "FAIL", + "evidence": "Synthetic cross-report inconsistency.", + } + ] + validate_consistency_review( + consistency, mode, {"s1": first, "s2": second} + ) + cells = adjudication_cells(mode, first, second, consistency) + expected_fields = { + f"atom:{atom_ids(mode)[0]}", + f"hard_error:{hard_error_ids(mode)[0]}", + "proposal_laundering", + "novel:s1:N1", + } + expected_keys = { + ("A", field) for field in expected_fields + } | {("B", f"atom:{atom_ids(mode)[0]}")} + if {(cell["label"], cell["field"]) for cell in cells} != expected_keys: + raise AssertionError(f"{mode}: adjudication-cell self-test failed") + hard_cell = next( + cell + for cell in cells + if cell["field"] == f"hard_error:{hard_error_ids(mode)[0]}" + ) + if {hard_cell["s1"]["decision"], hard_cell["s2"]["decision"]} != { + "PRESENT" + }: + raise AssertionError(f"{mode}: agreed positive hard error escaped adjudication") + hard_key = ("A", f"hard_error:{hard_error_ids(mode)[0]}") + consumed: set[tuple[str, str]] = set() + merged = merge_cell_decision( + hard_key[0], + hard_key[1], + "PRESENT", + "PRESENT", + {hard_key}, + { + hard_key: { + "label": hard_key[0], + "field": hard_key[1], + "decision": "ABSENT", + "evidence": "Synthetic independent adjudication.", + } + }, + consumed, + ) + if merged != "ABSENT" or consumed != {hard_key}: + raise AssertionError(f"{mode}: merge ignored an agreed-positive adjudication") + challenge_cell = next( + cell + for cell in cells + if cell["label"] == "B" + and cell["field"] == f"atom:{atom_ids(mode)[0]}" + ) + if challenge_cell.get("consistency_review", {}).get("decision") != "FAIL": + raise AssertionError(f"{mode}: consistency challenge was not unioned") + invalid_consistency = synthetic_consistency_review(mode) + invalid_consistency["challenges"] = [ + { + "label": "B", + "field": f"atom:{atom_ids(mode)[0]}", + "recommended_decision": "PASS", + "evidence": "This does not actually challenge either scorer.", + } + ] + try: + validate_consistency_review( + invalid_consistency, mode, {"s1": first, "s2": second} + ) + except ValueError: + pass + else: + raise AssertionError(f"{mode}: vacuous consistency challenge was accepted") + incomplete_consistency = synthetic_consistency_review(mode) + incomplete_consistency["defect_family_attestations"].pop() + try: + validate_consistency_review( + incomplete_consistency, mode, {"s1": first, "s2": second} + ) + except ValueError: + pass + else: + raise AssertionError( + f"{mode}: incomplete defect-family attestations were accepted" + ) + invalid = synthetic_score(mode, "s1") + invalid["reports"][1]["label"] = "A" + try: + validate_score(invalid, mode, "s1") + except ValueError: + pass + else: + raise AssertionError(f"{mode}: duplicate label was accepted") + invalid = synthetic_score(mode, "s1") + invalid["reports"][0]["atoms"].pop() + try: + validate_score(invalid, mode, "s1") + except ValueError: + pass + else: + raise AssertionError(f"{mode}: incomplete atom set was accepted") + invalid = synthetic_score(mode, "s1") + invalid["schema_version"] = True + try: + validate_score(invalid, mode, "s1") + except ValueError: + pass + else: + raise AssertionError(f"{mode}: boolean schema version was accepted") + invalid = synthetic_score(mode, "s1") + invalid["reports"][0]["atoms"][0]["evidence"] = 1 + try: + validate_score(invalid, mode, "s1") + except ValueError: + pass + else: + raise AssertionError(f"{mode}: non-string evidence was accepted") + invalid = synthetic_score(mode, "s1") + invalid["reports"][0]["atoms"][0]["evidence"] = " " + try: + validate_score(invalid, mode, "s1") + except ValueError: + pass + else: + raise AssertionError(f"{mode}: whitespace-only evidence was accepted") + source = synthetic_score(mode, "s1") + synthetic_final = { + "schema_version": 1, + "mode": mode, + "reports": [ + { + "label": report["label"], + "atoms": { + atom["id"]: atom["decision"] for atom in report["atoms"] + }, + "hard_errors": [], + "proposal_laundering": False, + "tcb_authority_defect": False, + "visible_scope_defect": False, + } + for report in source["reports"] + ], + "confirmed_novel_findings": [], + } + validate_final_score(synthetic_final, mode) + incomplete_final = json.loads(json.dumps(synthetic_final)) + incomplete_final["reports"][0]["atoms"] = {} + try: + validate_final_score(incomplete_final, mode) + except ValueError: + pass + else: + raise AssertionError(f"{mode}: incomplete final atom set was accepted") + schedule = load_schedule() + for run_id, row in schedule.items(): + runtime = Path("/tmp/ur-eval") / row["cell_id"] + prompt = render_report_prompt(run_id, runtime) + if "[" + "PACKAGE]" in prompt or "[" + "WORD_LIMIT]" in prompt: + raise AssertionError("unresolved report prompt") + if report_reminder_text() != ( + "Complete now within the frozen word limit using only material already\n" + "inspected; do not widen scope." + ): + raise AssertionError("frozen reminder extraction changed") + first_mode = MODES[0] + scorer_identity = f"{first_mode}-s1" + if expected_evaluator_output("scorer", scorer_identity, 1) == expected_evaluator_output( + "scorer", scorer_identity, 2 + ): + raise AssertionError("evaluator attempts share a runtime path") + assert_evaluator_attempt_allowed("scorer", scorer_identity, 1, []) + try: + assert_evaluator_attempt_allowed("scorer", scorer_identity, 2, []) + except ValueError: + pass + else: + raise AssertionError("evaluator retry without infrastructure failure was accepted") + assert_evaluator_attempt_allowed( + "scorer", + scorer_identity, + 2, + [ + { + "event": "evaluator_infrastructure_failure", + "attempt": 1, + "details": {"kind": "scorer", "identity": scorer_identity}, + } + ], + ) + invalid_event = { + "schema_version": True, + "sequence": 1, + "previous_event_sha256": None, + "time_utc": utc_now(), + "phase": "freeze", + "event": "synthetic", + "details": {}, + } + try: + validate_event_record(invalid_event, 1) + except ValueError: + pass + else: + raise AssertionError("boolean event schema version was accepted") + self_test_byte_tree_identity() + metadata_root = Path(tempfile.mkdtemp(prefix="ur-packet-self-test-", dir="/tmp")) + try: + nested = metadata_root / "nested" + nested.mkdir() + file = nested / "report.md" + file.write_text("packet metadata self-test\n") + make_read_only(metadata_root) + for item in (metadata_root, nested, file): + if item.stat().st_mtime_ns != 0: + raise AssertionError(f"metadata timestamp was not normalized: {item}") + finally: + for item in sorted(metadata_root.rglob("*"), reverse=True): + item.chmod(0o755 if item.is_dir() else 0o644) + metadata_root.chmod(0o755) + shutil.rmtree(metadata_root) + print("protocol self-test passed") + + +def record_consistency_review( + mode: str, attempt: int, output: Path, agent_id: str +) -> None: + assert_evaluator_started("consistency", mode, attempt, agent_id) + verify_consistency_packet(mode) + verify_evaluator_runtime( + "consistency", + mode, + attempt, + SCORING / "consistency-packets" / mode, + output, + ) + entries = output_entries(output) + if [entry.name for entry in entries] != ["consistency.json"]: + raise ValueError("consistency-review output is not exactly consistency.json") + raw = entries[0].read_bytes() + value = json.loads(raw.decode("utf-8")) + scores = {scorer: load_preserved_score(mode, scorer) for scorer in SCORERS} + validate_consistency_review(value, mode, scores) + attempt_directory = ( + SCORING + / "evaluator-attempts" + / "consistency" + / mode + / str(attempt) + ) + raw_entries = preserve_captured_file( + attempt_directory / "raw-output", "consistency.json", raw + ) + write_once( + attempt_directory / "raw-output-manifest.json", json_dump(raw_entries) + ) + attempt_attestation = { + "schema_version": 1, + "phase": "consistency", + "kind": "consistency", + "identity": mode, + "attempt": attempt, + "agent_id": agent_id, + "disposition": "COMPLETE", + "recorded_utc": utc_now(), + } + write_once(attempt_directory / "attestation.json", json_dump(attempt_attestation)) + make_read_only(attempt_directory) + append_event( + "consistency", + "evaluator_attempt_preserved", + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(attempt_directory), + details={"kind": "consistency", "identity": mode, "disposition": "COMPLETE"}, + ) + destination = SCORING / "consistency" / f"{mode}.json" + write_bytes_once(destination, raw) + destination.chmod(0o444) + append_event( + "consistency", + "evaluator_returned", + attempt=attempt, + agent_id=agent_id, + details={"kind": "consistency", "identity": mode, "api_state": "COMPLETED"}, + ) + append_event( + "consistency", + "consistency_review_preserved", + attempt=attempt, + agent_id=agent_id, + digest=sha256_file(destination), + details={"mode": mode}, + ) + print(destination) + + +def record_adjudication( + mode: str, attempt: int, output: Path, agent_id: str +) -> None: + assert_evaluator_started("adjudicator", mode, attempt, agent_id) + verify_adjudication_packet(mode) + verify_evaluator_runtime( + "adjudicator", + mode, + attempt, + SCORING / "adjudication-packets" / mode, + output, + ) + entries = output_entries(output) + if [entry.name for entry in entries] != ["adjudication.json"]: + raise ValueError("adjudicator output is not exactly adjudication.json") + raw = entries[0].read_bytes() + value = json.loads(raw.decode("utf-8")) + validate_adjudication(value, mode) + attempt_directory = ( + SCORING + / "evaluator-attempts" + / "adjudication" + / mode + / str(attempt) + ) + raw_entries = preserve_captured_file( + attempt_directory / "raw-output", "adjudication.json", raw + ) + write_once( + attempt_directory / "raw-output-manifest.json", json_dump(raw_entries) + ) + attempt_attestation = { + "schema_version": 1, + "phase": "adjudication", + "kind": "adjudicator", + "identity": mode, + "attempt": attempt, + "agent_id": agent_id, + "disposition": "COMPLETE", + "recorded_utc": utc_now(), + } + write_once(attempt_directory / "attestation.json", json_dump(attempt_attestation)) + make_read_only(attempt_directory) + append_event( + "adjudication", + "evaluator_attempt_preserved", + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(attempt_directory), + details={"kind": "adjudicator", "identity": mode, "disposition": "COMPLETE"}, + ) + destination = SCORING / "adjudications" / f"{mode}.json" + write_bytes_once(destination, raw) + destination.chmod(0o444) + append_event( + "adjudication", + "evaluator_returned", + attempt=attempt, + agent_id=agent_id, + details={"kind": "adjudicator", "identity": mode, "api_state": "COMPLETED"}, + ) + append_event( + "adjudication", + "adjudication_preserved", + attempt=attempt, + agent_id=agent_id, + digest=sha256_file(destination), + details={"mode": mode}, + ) + print(destination) + + +def record_invalid_evaluator( + kind: str, + identity: str, + attempt: int, + output: Path, + agent_id: str, + evidence: str, +) -> None: + invalid_marker = RUN / "INVALID.json" + if invalid_marker.exists(): + raise FileExistsError("run is already marked INVALID") + if not evidence.strip(): + raise ValueError("invalid evaluator output requires nonempty evidence") + phase = evaluator_phase(kind) + assert_evaluator_started(kind, identity, attempt, agent_id) + source_packet = evaluator_source_packet(kind, identity) + forensics: dict[str, Any] | None = None + try: + verify_evaluator_runtime( + kind, identity, attempt, source_packet, output, allow_invalid_output=True + ) + except (OSError, ValueError) as error: + forensics = evaluator_runtime_forensics( + kind, identity, attempt, source_packet, output, error + ) + evidence = ( + f"{evidence} Runtime/packet verification failed: " + f"{forensics['verification_error']}" + ) + destination = SCORING / "invalid" / phase / identity / str(attempt) + root = output.parent + if forensics is None: + entries = snapshot_path(output, destination / "raw-output") + write_once(destination / "raw-output-manifest.json", json_dump(entries)) + else: + entries = snapshot_path(root, destination / "raw-runtime") + write_once(destination / "raw-runtime-manifest.json", json_dump(entries)) + source_entries = snapshot_path( + source_packet, destination / "source-packet-at-verification" + ) + write_once( + destination / "source-packet-at-verification-manifest.json", + json_dump(source_entries), + ) + if forensics is not None: + write_once(destination / "runtime-forensics.json", json_dump(forensics)) + attestation = { + "schema_version": 1, + "phase": phase, + "kind": kind, + "identity": identity, + "attempt": attempt, + "agent_id": agent_id, + "evidence": evidence, + "disposition": "INVALID_NONRERUNNABLE_EVALUATOR_OUTPUT", + "recorded_utc": utc_now(), + } + write_once(destination / "attestation.json", json_dump(attestation)) + make_read_only(destination) + append_event( + phase, + "invalid_output_preserved", + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(destination), + details={"kind": kind, "identity": identity}, + ) + write_once(invalid_marker, json_dump(attestation)) + invalid_marker.chmod(0o444) + append_event( + phase, + "evaluator_returned", + attempt=attempt, + agent_id=agent_id, + details={"kind": kind, "identity": identity, "api_state": "INVALID_OUTPUT"}, + ) + append_event( + phase, + "run_invalidated", + attempt=attempt, + agent_id=agent_id, + digest=sha256_file(invalid_marker), + details={"identity": identity, "evidence": evidence}, + ) + print(invalid_marker) + + +def preserve_failed_evaluator_attempt( + kind: str, + identity: str, + attempt: int, + output: Path, + agent_id: str, + disposition: str, + evidence: str, +) -> None: + if disposition not in INFRA_FAILURE_CODES: + raise ValueError(f"non-infrastructure evaluator disposition: {disposition}") + if not evidence.strip(): + raise ValueError("evaluator infrastructure failure requires nonempty evidence") + phase = evaluator_phase(kind) + assert_evaluator_started(kind, identity, attempt, agent_id) + source_packet = evaluator_source_packet(kind, identity) + verify_evaluator_runtime( + kind, identity, attempt, source_packet, output, allow_invalid_output=True + ) + destination = SCORING / "evaluator-attempts" / phase / identity / str(attempt) + entries = snapshot_path(output, destination / "raw-output") + write_once(destination / "raw-output-manifest.json", json_dump(entries)) + attestation = { + "schema_version": 1, + "phase": phase, + "kind": kind, + "identity": identity, + "attempt": attempt, + "agent_id": agent_id, + "evidence": evidence, + "disposition": disposition, + "recorded_utc": utc_now(), + } + write_once(destination / "attestation.json", json_dump(attestation)) + make_read_only(destination) + append_event( + phase, + "evaluator_attempt_preserved", + attempt=attempt, + agent_id=agent_id, + digest=byte_tree_digest(destination), + details={"kind": kind, "identity": identity, "disposition": disposition}, + ) + append_event( + phase, + "evaluator_returned", + attempt=attempt, + agent_id=agent_id, + details={"kind": kind, "identity": identity, "api_state": "INFRASTRUCTURE_FAILURE"}, + ) + append_event( + phase, + "evaluator_infrastructure_failure", + attempt=attempt, + agent_id=agent_id, + details={ + "kind": kind, + "identity": identity, + "disposition": disposition, + "evidence": evidence, + }, + ) + print(json_dump(attestation), end="") + + +def decision_lookup(value: dict[str, Any]) -> dict[tuple[str, str], dict[str, Any]]: + return {(item["label"], item["field"]): item for item in value.get("decisions", [])} + + +def merge_cell_decision( + label: str, + field: str, + first: str, + second: str, + listed_cells: set[tuple[str, str]], + decisions: dict[tuple[str, str], dict[str, Any]], + consumed_cells: set[tuple[str, str]], +) -> str: + key = (label, field) + if key in listed_cells: + if key not in decisions: + raise ValueError(f"listed cell lacks adjudication: {key}") + consumed_cells.add(key) + return decisions[key]["decision"] + if first != second: + raise ValueError(f"unlisted scorer disagreement: {key}") + return first + + +def merge_final(mode: str) -> None: + scores = {scorer: load_preserved_score(mode, scorer) for scorer in SCORERS} + cells_path = SCORING / "adjudication-cells" / f"{mode}.json" + if sha256_file(cells_path) != preserved_digest( + "adjudication_cells_materialized", mode=mode + ): + raise ValueError(f"adjudication cells changed after preservation: {mode}") + cells_document = json.loads(cells_path.read_text()) + if cells_document["cells"]: + adjudication = json.loads((SCORING / "adjudications" / f"{mode}.json").read_text()) + expected_adjudication = preserved_digest("adjudication_preserved", mode=mode) + if sha256_file(SCORING / "adjudications" / f"{mode}.json") != expected_adjudication: + raise ValueError(f"adjudication changed after preservation: {mode}") + validate_adjudication(adjudication, mode) + decisions = decision_lookup(adjudication) + else: + decisions = {} + listed_cells = { + (cell["label"], cell["field"]) for cell in cells_document["cells"] + } + consumed_cells: set[tuple[str, str]] = set() + + by_scorer = {scorer: report_by_label(score) for scorer, score in scores.items()} + final_reports: list[dict[str, Any]] = [] + confirmed_novel: list[dict[str, Any]] = [] + for label in LABELS: + first = by_scorer["s1"][label] + second = by_scorer["s2"][label] + atoms: dict[str, str] = {} + atoms1 = {atom["id"]: atom for atom in first["atoms"]} + atoms2 = {atom["id"]: atom for atom in second["atoms"]} + for atom in atom_ids(mode): + atoms[atom] = merge_cell_decision( + label, + f"atom:{atom}", + atoms1[atom]["decision"], + atoms2[atom]["decision"], + listed_cells, + decisions, + consumed_cells, + ) + final_hard: list[str] = [] + for identifier in hard_error_ids(mode): + present = [any(item["id"] == identifier for item in report["hard_errors"]) for report in (first, second)] + chosen = ( + merge_cell_decision( + label, + f"hard_error:{identifier}", + "PRESENT" if present[0] else "ABSENT", + "PRESENT" if present[1] else "ABSENT", + listed_cells, + decisions, + consumed_cells, + ) + == "PRESENT" + ) + if chosen: + final_hard.append(identifier) + flags: dict[str, bool] = {} + for field in GLOBAL_DEFECT_FIELDS: + present = [first[field]["present"], second[field]["present"]] + flags[field] = ( + merge_cell_decision( + label, + field, + "PRESENT" if present[0] else "ABSENT", + "PRESENT" if present[1] else "ABSENT", + listed_cells, + decisions, + consumed_cells, + ) + == "PRESENT" + ) + final_reports.append({"label": label, "atoms": atoms, "hard_errors": final_hard, **flags}) + for scorer, report in (("s1", first), ("s2", second)): + for finding in report["novel_findings"]: + field = f"novel:{scorer}:{finding['id']}" + key = (label, field) + if key not in decisions or key not in listed_cells: + raise ValueError(f"novel candidate lacks adjudication: {mode}/{key}") + consumed_cells.add(key) + if decisions[key]["decision"] == "PRESENT": + confirmed_novel.append({"label": label, "field": field, "evidence": decisions[(label, field)]["evidence"]}) + if consumed_cells != listed_cells: + missing = sorted(listed_cells - consumed_cells) + raise ValueError(f"merge did not consume every adjudicated cell: {mode}/{missing}") + value = {"schema_version": 1, "mode": mode, "reports": final_reports, "confirmed_novel_findings": confirmed_novel} + validate_final_score(value, mode) + destination = SCORING / "final" / f"{mode}.json" + write_once(destination, json_dump(value)) + destination.chmod(0o444) + append_event("adjudication", "final_blind_score_locked", digest=sha256_file(destination), details={"mode": mode}) + print(destination) + + +def aggregate() -> None: + schedule = load_schedule() + conditions = load_condition_map() + targets = load_target_map() + blind = load_blind_map() + index = load_index(require_collection_lock=True) + counts: dict[str, dict[str, dict[str, int]]] = {} + defects: dict[str, dict[str, list[dict[str, Any]]]] = {} + failures: list[dict[str, Any]] = [] + novel: list[dict[str, Any]] = [] + final_digests: dict[str, str] = {} + for mode in MODES: + path = SCORING / "final" / f"{mode}.json" + expected_final = preserved_digest("final_blind_score_locked", mode=mode) + if sha256_file(path) != expected_final: + raise ValueError(f"final blind score changed after lock: {mode}") + value = json.loads(path.read_text()) + validate_final_score(value, mode) + final_digests[mode] = sha256_file(path) + reports = {report["label"]: report for report in value["reports"]} + if tuple(sorted(reports)) != LABELS: + raise ValueError(f"{mode} final labels incomplete") + counts[mode] = {role: {atom: 0 for atom in atom_ids(mode)} for role in prepare.CONDITIONS} + defects[mode] = {role: [] for role in prepare.CONDITIONS} + for label, run_id in blind[mode].items(): + row = schedule[run_id] + if targets[row["target_label"]]["mode"] != mode: + raise ValueError("unblinding mode mismatch") + role = conditions[row["condition_label"]]["role"] + report = reports[label] + for atom, decision in report["atoms"].items(): + if decision == "PASS": + counts[mode][role][atom] += 1 + else: + failures.append({"mode": mode, "role": role, "run_id": run_id, "label": label, "atom": atom}) + over_budget = index[run_id]["word_count"] > index[run_id]["word_cap"] + flags = { + "hard_errors": report["hard_errors"], + "proposal_laundering": report["proposal_laundering"], + "tcb_authority_defect": report["tcb_authority_defect"], + "visible_scope_defect": report["visible_scope_defect"], + "operational_scope_deviation": index[run_id]["operational_scope_deviation"], + "word_budget_defect": over_budget, + "semantic_noncompletion": index[run_id]["semantic_noncompletion"], + } + if flags["hard_errors"] or any( + flags[name] + for name in ( + "proposal_laundering", + "tcb_authority_defect", + "visible_scope_defect", + "operational_scope_deviation", + "word_budget_defect", + "semantic_noncompletion", + ) + ): + defects[mode][role].append({"run_id": run_id, "label": label, **flags}) + for finding in value["confirmed_novel_findings"]: + run_id = blind[mode][finding["label"]] + role = conditions[schedule[run_id]["condition_label"]]["role"] + novel.append({"mode": mode, "role": role, "run_id": run_id, **finding}) + v4_defects = [item for mode in MODES for item in defects[mode]["v4"]] + v4_novel = [finding for finding in novel if finding["role"] == "v4"] + gate = { + "all_v4_atoms_5_of_5": all( + count == len(REPLICATES) + for mode in MODES + for count in counts[mode]["v4"].values() + ), + "zero_v4_hard_errors": not any(item["hard_errors"] for item in v4_defects), + "zero_v4_proposal_laundering": not any(item["proposal_laundering"] for item in v4_defects), + "zero_v4_tcb_authority_defects": not any(item["tcb_authority_defect"] for item in v4_defects), + "zero_v4_semantic_noncompletion": not any( + item["semantic_noncompletion"] for item in v4_defects + ), + "zero_v4_scope_budget_defects": not any( + item["visible_scope_defect"] + or item["operational_scope_deviation"] + or item["word_budget_defect"] + for item in v4_defects + ), + "zero_v4_confirmed_novel_findings": not v4_novel, + } + gate["overall"] = all(gate.values()) + diagnostic_comparisons: list[dict[str, Any]] = [] + for mode in MODES: + for atom in atom_ids(mode): + v4_count = counts[mode]["v4"][atom] + v3_count = counts[mode]["v3"][atom] + if v4_count < v3_count: + classification = "V4_BELOW_V3" + elif v4_count == len(REPLICATES) and v3_count < len(REPLICATES): + classification = "TARGETED_LIFT_EVIDENCE" + elif v4_count == len(REPLICATES) and v3_count == len(REPLICATES): + classification = "CEILING_REPLICATION" + elif v4_count > v3_count: + classification = "V4_HIGHER_BUT_CONFIRMATION_FAILED" + else: + classification = "MATCHED_BELOW_CEILING" + diagnostic_comparisons.append( + { + "mode": mode, + "atom": atom, + "v4_passes": v4_count, + "v3_passes": v3_count, + "classification": classification, + } + ) + result = { + "schema_version": 1, + "unblinded_utc": utc_now(), + "condition_map": {label: row["role"] for label, row in conditions.items()}, + "final_score_sha256": final_digests, + "counts": counts, + "defects": defects, + "failed_atom_cells": failures, + "confirmed_novel_findings": novel, + "diagnostic_comparison": { + "any_v4_below_v3": any( + item["classification"] == "V4_BELOW_V3" + for item in diagnostic_comparisons + ), + "causal_claim": False, + "reason": "The coherent V4 and V3 packages differ in more than one isolated instruction.", + "atoms": diagnostic_comparisons, + }, + "v4_gate": gate, + } + RESULTS.mkdir(parents=True, exist_ok=True) + write_once(RUN / "unblinding.json", json_dump({"schema_version": 1, "unblinded_utc": result["unblinded_utc"], "condition_map": result["condition_map"], "final_score_sha256": final_digests})) + write_once(RESULTS / "aggregate.json", json_dump(result)) + lines = [ + "# V4 Focused Confirmation Results", + "", + f"**Primary V4 gate: {'PASS' if gate['overall'] else 'FAIL'}.**", + "", + "Each atom cell is a pass count out of five. V4 is the confirmatory candidate; V3 is diagnostic.", + "", + "| Mode | Condition | Atom pass counts | Defective reports |", + "|---|---|---|---:|", + ] + for mode in MODES: + for role in ("v4", "v3"): + rendered = "; ".join(f"{atom} {count}/5" for atom, count in counts[mode][role].items()) + lines.append(f"| {mode} | {role.upper()} | {rendered} | {len(defects[mode][role])} |") + lines.extend( + [ + "", + "## Diagnostic comparison", + "", + f"Any V4 atom below matched V3: {'YES' if result['diagnostic_comparison']['any_v4_below_v3'] else 'NO'}.", + "", + "`TARGETED_LIFT_EVIDENCE` means V4 passed 5/5 while matched V3 was lower. " + "`CEILING_REPLICATION` means both passed 5/5. These coherent packages differ " + "in more than one isolated instruction, so no classification is causal proof.", + "", + "| Mode | Atom | V4 | V3 | Classification |", + "|---|---|---:|---:|---|", + ] + ) + for item in diagnostic_comparisons: + lines.append( + f"| {item['mode']} | {item['atom']} | {item['v4_passes']}/5 | " + f"{item['v3_passes']}/5 | {item['classification']} |" + ) + lines.extend(["", "## Primary gates", "", "| Gate | Result |", "|---|---|"]) + for name, passed in gate.items(): + if name != "overall": + lines.append(f"| {name.replace('_', ' ')} | {'PASS' if passed else 'FAIL'} |") + lines.extend(["", "## Confirmed novel findings", ""]) + if novel: + for finding in novel: + evidence = " ".join(finding["evidence"].split()) + lines.append( + f"- {finding['mode']}/{finding['role'].upper()} " + f"{finding['run_id']}/{finding['label']} {finding['field']}: {evidence}" + ) + else: + lines.append("None.") + lines.extend(["", "## Integrity limitations", "", "Filesystem and URL isolation were procedural on a shared host. Exact hosted model-build and sampling-seed metadata were unavailable. Results are source-review capability observations under those constraints.", ""]) + write_once(RESULTS / "summary.md", "\n".join(lines)) + append_event("unblinding", "conditions_revealed", digest=sha256_file(RUN / "unblinding.json"), details={}) + append_event( + "result", + "aggregate_written", + digest=sha256_file(RESULTS / "aggregate.json"), + details={ + "v4_gate": gate["overall"], + "summary_sha256": sha256_file(RESULTS / "summary.md"), + }, + ) + print(RESULTS / "summary.md") + + +def record_freeze_lock() -> None: + if any(event["event"] == "freeze_locked" for event in event_records()): + raise ValueError("freeze lock was already recorded") + append_event( + "freeze", + "freeze_locked", + digest=sha256_file(FILE_MANIFEST), + details={"lock_sha256": sha256_file(LOCK)}, + ) + print("freeze lock recorded") + + +def assert_freeze_locked() -> None: + matches = [event for event in event_records() if event["event"] == "freeze_locked"] + if len(matches) != 1: + raise ValueError("evaluation operations require exactly one freeze-lock event") + event = matches[0] + if ( + event.get("sha256") != sha256_file(FILE_MANIFEST) + or event.get("details", {}).get("lock_sha256") != sha256_file(LOCK) + ): + raise ValueError("freeze-lock event does not bind the current lock") + + +def main() -> None: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + verify = sub.add_parser("verify-static") + verify.add_argument("--locked", action="store_true") + sub.add_parser("self-test") + sub.add_parser("self-test-byte-tree") + sub.add_parser("write-file-manifest") + sub.add_parser("record-freeze-lock") + prepare_cell_parser = sub.add_parser("prepare-cell") + prepare_cell_parser.add_argument("run_id") + prepare_cell_parser.add_argument("runtime", type=Path) + report_prompt = sub.add_parser("report-prompt") + report_prompt.add_argument("run_id") + report_prompt.add_argument("runtime", type=Path) + record_report_parser = sub.add_parser("record-report") + record_report_parser.add_argument("run_id") + record_report_parser.add_argument("attempt", type=positive_int) + record_report_parser.add_argument("runtime", type=Path) + record_report_parser.add_argument("agent_id") + record_report_parser.add_argument("--scope-deviation", action="store_true") + record_report_parser.add_argument( + "--scope-evidence", default="No known operational source-scope deviation." + ) + failed_report_parser = sub.add_parser("record-failed-report") + failed_report_parser.add_argument("run_id") + failed_report_parser.add_argument("attempt", type=positive_int) + failed_report_parser.add_argument("runtime", type=Path) + failed_report_parser.add_argument("agent_id") + failed_report_parser.add_argument( + "disposition", choices=sorted(INFRA_FAILURE_CODES | TERMINAL_REPORT_FAILURE_CODES) + ) + failed_report_parser.add_argument("evidence") + failed_report_parser.add_argument("--scope-deviation", action="store_true") + agent_start_parser = sub.add_parser("agent-start") + agent_start_parser.add_argument("run_id") + agent_start_parser.add_argument("attempt", type=positive_int) + agent_start_parser.add_argument("agent_id") + prelaunch_parser = sub.add_parser("record-prelaunch-failure") + prelaunch_parser.add_argument("run_id") + prelaunch_parser.add_argument("evidence") + reminder_parser = sub.add_parser("reminder-text") + reminder_parser.add_argument("run_id") + reminder_parser.add_argument("attempt", type=positive_int) + reminder_parser.add_argument("agent_id") + sub.add_parser("build-score-packets") + scorer_prompt = sub.add_parser("scorer-prompt") + scorer_prompt.add_argument("mode", choices=MODES) + scorer_prompt.add_argument("scorer", choices=SCORERS) + scorer_prompt.add_argument("attempt", type=positive_int) + scorer_prompt.add_argument("output", type=Path) + record_score_parser = sub.add_parser("record-score") + record_score_parser.add_argument("mode", choices=MODES) + record_score_parser.add_argument("scorer", choices=SCORERS) + record_score_parser.add_argument("attempt", type=positive_int) + record_score_parser.add_argument("output", type=Path) + record_score_parser.add_argument("agent_id") + invalid_score_parser = sub.add_parser("record-invalid-score") + invalid_score_parser.add_argument("mode", choices=MODES) + invalid_score_parser.add_argument("scorer", choices=SCORERS) + invalid_score_parser.add_argument("attempt", type=positive_int) + invalid_score_parser.add_argument("output", type=Path) + invalid_score_parser.add_argument("agent_id") + invalid_score_parser.add_argument("evidence") + consistency_packet = sub.add_parser("build-consistency-packet") + consistency_packet.add_argument("mode", choices=MODES) + consistency_prompt = sub.add_parser("consistency-prompt") + consistency_prompt.add_argument("mode", choices=MODES) + consistency_prompt.add_argument("attempt", type=positive_int) + consistency_prompt.add_argument("output", type=Path) + record_consistency_parser = sub.add_parser("record-consistency") + record_consistency_parser.add_argument("mode", choices=MODES) + record_consistency_parser.add_argument("attempt", type=positive_int) + record_consistency_parser.add_argument("output", type=Path) + record_consistency_parser.add_argument("agent_id") + invalid_consistency_parser = sub.add_parser("record-invalid-consistency") + invalid_consistency_parser.add_argument("mode", choices=MODES) + invalid_consistency_parser.add_argument("attempt", type=positive_int) + invalid_consistency_parser.add_argument("output", type=Path) + invalid_consistency_parser.add_argument("agent_id") + invalid_consistency_parser.add_argument("evidence") + adjudication_cells_parser = sub.add_parser("build-adjudication-cells") + adjudication_cells_parser.add_argument("mode", choices=MODES) + adjudication_packet = sub.add_parser("build-adjudication-packet") + adjudication_packet.add_argument("mode", choices=MODES) + adjudicator_prompt = sub.add_parser("adjudicator-prompt") + adjudicator_prompt.add_argument("mode", choices=MODES) + adjudicator_prompt.add_argument("attempt", type=positive_int) + adjudicator_prompt.add_argument("output", type=Path) + record_adjudication_parser = sub.add_parser("record-adjudication") + record_adjudication_parser.add_argument("mode", choices=MODES) + record_adjudication_parser.add_argument("attempt", type=positive_int) + record_adjudication_parser.add_argument("output", type=Path) + record_adjudication_parser.add_argument("agent_id") + invalid_adjudication_parser = sub.add_parser("record-invalid-adjudication") + invalid_adjudication_parser.add_argument("mode", choices=MODES) + invalid_adjudication_parser.add_argument("attempt", type=positive_int) + invalid_adjudication_parser.add_argument("output", type=Path) + invalid_adjudication_parser.add_argument("agent_id") + invalid_adjudication_parser.add_argument("evidence") + evaluator_start_parser = sub.add_parser("evaluator-start") + evaluator_start_parser.add_argument("kind", choices=EVALUATOR_KINDS) + evaluator_start_parser.add_argument("identity") + evaluator_start_parser.add_argument("attempt", type=positive_int) + evaluator_start_parser.add_argument("agent_id") + evaluator_prelaunch = sub.add_parser("record-evaluator-prelaunch-failure") + evaluator_prelaunch.add_argument("kind", choices=EVALUATOR_KINDS) + evaluator_prelaunch.add_argument("identity") + evaluator_prelaunch.add_argument("attempt", type=positive_int) + evaluator_prelaunch.add_argument("output", type=Path) + evaluator_prelaunch.add_argument("evidence") + failed_evaluator = sub.add_parser("record-failed-evaluator") + failed_evaluator.add_argument("kind", choices=EVALUATOR_KINDS) + failed_evaluator.add_argument("identity") + failed_evaluator.add_argument("attempt", type=positive_int) + failed_evaluator.add_argument("output", type=Path) + failed_evaluator.add_argument("agent_id") + failed_evaluator.add_argument("disposition", choices=sorted(INFRA_FAILURE_CODES)) + failed_evaluator.add_argument("evidence") + merge_parser = sub.add_parser("merge-final") + merge_parser.add_argument("mode", choices=MODES) + sub.add_parser("aggregate") + args = parser.parse_args() + + operational_commands = { + "record-freeze-lock", + "prepare-cell", + "report-prompt", + "agent-start", + "record-prelaunch-failure", + "reminder-text", + "record-report", + "record-failed-report", + "build-score-packets", + "scorer-prompt", + "record-score", + "record-invalid-score", + "build-consistency-packet", + "consistency-prompt", + "record-consistency", + "record-invalid-consistency", + "build-adjudication-cells", + "build-adjudication-packet", + "adjudicator-prompt", + "record-adjudication", + "record-invalid-adjudication", + "evaluator-start", + "record-evaluator-prelaunch-failure", + "record-failed-evaluator", + "merge-final", + "aggregate", + } + operation_lock_handle = None + if args.command in operational_commands: + operation_lock_handle = acquire_operation_lock() + assert_run_not_invalid() + validate_static(require_lock=True, announce=False) + if args.command == "record-freeze-lock": + if any(event["event"] == "freeze_locked" for event in event_records()): + raise SystemExit("freeze lock was already recorded") + else: + assert_freeze_locked() + + if args.command == "verify-static": + validate_static(args.locked) + elif args.command == "self-test": + self_test() + elif args.command == "self-test-byte-tree": + self_test_byte_tree_identity() + print("byte-tree hardening/copy self-test passed") + elif args.command == "write-file-manifest": + if FILE_MANIFEST.exists(): + raise FileExistsError(FILE_MANIFEST) + write_once(FILE_MANIFEST, render_file_manifest()) + print(f"{FILE_MANIFEST.relative_to(RUN)} {sha256_file(FILE_MANIFEST)}") + elif args.command == "record-freeze-lock": + record_freeze_lock() + elif args.command == "prepare-cell": + prepare_cell(args.run_id, args.runtime) + elif args.command == "report-prompt": + verify_runtime(args.run_id, args.runtime) + print(render_report_prompt(args.run_id, args.runtime), end="") + elif args.command == "record-report": + record_report( + args.run_id, + args.attempt, + args.runtime, + args.agent_id, + args.scope_deviation, + args.scope_evidence, + ) + elif args.command == "record-failed-report": + preserve_failed_report_attempt( + args.run_id, + args.attempt, + args.runtime, + args.agent_id, + args.disposition, + args.evidence, + args.scope_deviation, + ) + elif args.command == "agent-start": + record_agent_start(args.run_id, args.attempt, args.agent_id) + elif args.command == "record-prelaunch-failure": + record_prelaunch_failure(args.run_id, args.evidence) + elif args.command == "reminder-text": + record_reminder(args.run_id, args.attempt, args.agent_id) + print(report_reminder_text(), end="") + elif args.command == "build-score-packets": + build_score_packets() + elif args.command == "scorer-prompt": + source_packet = SCORING / "packets" / args.mode / args.scorer + verify_score_packet(args.mode, args.scorer) + packet = prepare_evaluator_runtime( + "scorer", + f"{args.mode}-{args.scorer}", + args.attempt, + source_packet, + args.output, + ) + print( + render_packet_prompt("scorer.md", packet, args.output, SCORER_ID=args.scorer), + end="", + ) + elif args.command == "record-score": + record_score(args.mode, args.scorer, args.attempt, args.output, args.agent_id) + elif args.command == "record-invalid-score": + record_invalid_evaluator( + "scorer", + f"{args.mode}-{args.scorer}", + args.attempt, + args.output, + args.agent_id, + args.evidence, + ) + elif args.command == "build-consistency-packet": + build_consistency_packet(args.mode) + elif args.command == "consistency-prompt": + source_packet = SCORING / "consistency-packets" / args.mode + verify_consistency_packet(args.mode) + packet = prepare_evaluator_runtime( + "consistency", args.mode, args.attempt, source_packet, args.output + ) + print(render_packet_prompt("consistency.md", packet, args.output), end="") + elif args.command == "record-consistency": + record_consistency_review( + args.mode, args.attempt, args.output, args.agent_id + ) + elif args.command == "record-invalid-consistency": + record_invalid_evaluator( + "consistency", + args.mode, + args.attempt, + args.output, + args.agent_id, + args.evidence, + ) + elif args.command == "build-adjudication-cells": + build_adjudication_cells(args.mode) + elif args.command == "build-adjudication-packet": + build_adjudication_packet(args.mode) + elif args.command == "adjudicator-prompt": + source_packet = SCORING / "adjudication-packets" / args.mode + verify_adjudication_packet(args.mode) + packet = prepare_evaluator_runtime( + "adjudicator", args.mode, args.attempt, source_packet, args.output + ) + print(render_packet_prompt("adjudicator.md", packet, args.output), end="") + elif args.command == "record-adjudication": + record_adjudication(args.mode, args.attempt, args.output, args.agent_id) + elif args.command == "record-invalid-adjudication": + record_invalid_evaluator( + "adjudicator", + args.mode, + args.attempt, + args.output, + args.agent_id, + args.evidence, + ) + elif args.command == "evaluator-start": + record_evaluator_start(args.kind, args.identity, args.attempt, args.agent_id) + elif args.command == "record-evaluator-prelaunch-failure": + record_evaluator_prelaunch_failure( + args.kind, args.identity, args.attempt, args.output, args.evidence + ) + elif args.command == "record-failed-evaluator": + preserve_failed_evaluator_attempt( + args.kind, + args.identity, + args.attempt, + args.output, + args.agent_id, + args.disposition, + args.evidence, + ) + elif args.command == "merge-final": + merge_final(args.mode) + elif args.command == "aggregate": + aggregate() + + +if __name__ == "__main__": + main() diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/blind-map.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/blind-map.tsv new file mode 100644 index 0000000000..b5492a1a4d --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/blind-map.tsv @@ -0,0 +1,51 @@ +mode label run_id +P A r046 +P B r036 +P C r023 +P D r024 +P E r005 +P F r006 +P G r020 +P H r035 +P I r049 +P J r015 +B A r050 +B B r029 +B C r034 +B D r027 +B E r045 +B F r014 +B G r010 +B H r008 +B I r018 +B J r040 +L A r021 +L B r038 +L C r037 +L D r017 +L E r019 +L F r007 +L G r048 +L H r022 +L I r004 +L J r041 +Q A r002 +Q B r013 +Q C r031 +Q D r043 +Q E r039 +Q F r028 +Q G r047 +Q H r011 +Q I r025 +Q J r001 +R A r030 +R B r042 +R C r009 +R D r032 +R E r026 +R F r033 +R G r012 +R H r044 +R I r003 +R J r016 diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/condition-map.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/condition-map.tsv new file mode 100644 index 0000000000..937905e629 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/condition-map.tsv @@ -0,0 +1,3 @@ +condition_label role package_path tree_sha256 skill_sha256 +c0 v3 frozen-packages/668f70202c7bc8f23f7f894fb784a9629fd292c7f6fe69ede815b0e4c10137bf fc486dedde1f82ba232b4492808af85a12b27fa2aa27b1a35a3847b2b89f72e0 0e23f7747cc63014bade7543efaf745e7e9a7e5d6dee2a48c602ef7a3eba091e +c1 v4 frozen-packages/6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb 6d7e197e431b82eb81dbe7eefc79fde811e0e238435d38c69460cc068e631abb ad48b3811cf2054be76e4b461a36f63e636afb246c5dd7a75e85756a53b22d83 diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/consistency-schedule.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/consistency-schedule.tsv new file mode 100644 index 0000000000..124670a7e0 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/consistency-schedule.tsv @@ -0,0 +1,6 @@ +claim +L +Q +B +P +R diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/launch-schedule.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/launch-schedule.tsv new file mode 100644 index 0000000000..16b99b38fe --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/launch-schedule.tsv @@ -0,0 +1,51 @@ +run_id cell_id wave target_label condition_label replicate +r001 f283e9f240bd883b1e568b9b0b7d00b5 1 m3 c1 4 +r002 1dfddadf11a44c7a250b3f30dd567335 1 m3 c0 4 +r003 5a9f15b6422fe44db62c4d720bdae8a2 1 m0 c0 4 +r004 5aca9ea4158f1a857bf1005fc73a2273 1 m2 c0 4 +r005 f4506c9f20cbfef429ed5ce317a25141 1 m4 c0 4 +r006 1146888d03484c5ea5ca80d8923baca1 1 m4 c1 4 +r007 b694bd930f677652f7e181a978916b00 1 m2 c1 4 +r008 654926f712c7339fd514e3a050230605 1 m1 c0 4 +r009 00309182fcf24228c055589a25574da9 1 m0 c1 4 +r010 3cc6e34fbb3b922e62ec776b34149895 1 m1 c1 4 +r011 27d33f081e70a44ee95086c2a6dc15dd 2 m3 c0 3 +r012 881873f9704c96644f9a5e98c9be08c9 2 m0 c1 3 +r013 dbb51c756615b6ce927b248f3cda7eba 2 m3 c1 3 +r014 667b67c97b1286fbe985ac6b78ca082b 2 m1 c0 3 +r015 79030a8563b6b7142a77dac96b7f89dd 2 m4 c0 3 +r016 322027b63f67b0a661d078e000b358bc 2 m0 c0 3 +r017 07797bcb7916819e328af1166c366498 2 m2 c0 3 +r018 d50eb2488ebbe5c8afa54055bc3a4b96 2 m1 c1 3 +r019 761e0a4c3119de7b644c12a48b4542eb 2 m2 c1 3 +r020 3213ed90744c066130b7a79a814b47e7 2 m4 c1 3 +r021 e5951ec14062ede715c2883e71599aa0 3 m2 c1 5 +r022 ddd3c84a6fc9910cdb651d97ede8177d 3 m2 c0 5 +r023 a044ec75f8f9b15e6434c5600ca3a5f1 3 m4 c1 5 +r024 2070b378e27be36df65e1351f1fdf31f 3 m4 c0 5 +r025 97958bb3a7f0c6d82aef856c1576f091 3 m3 c0 5 +r026 5dba5b54fccd1c75cc909610fb3e9d70 3 m0 c1 5 +r027 63586bb3ea790c41f7dfeee3988bcc18 3 m1 c1 5 +r028 0d388afb946fbe9677b2dd710e1bcd89 3 m3 c1 5 +r029 a1bc95e6172e61ef77f607c3622c4f1a 3 m1 c0 5 +r030 5a6a7446cbedd305f50f4da2d0d7cd6b 3 m0 c0 5 +r031 2e0f8622dd6e05cf9c513014269574dd 4 m3 c0 2 +r032 b482b26bbca7b2874886f3e0b8f24997 4 m0 c1 2 +r033 22badac4da8e11220b04e1757934af83 4 m0 c0 2 +r034 003eb505a9319d72618e179a97be7e48 4 m1 c0 2 +r035 ef849e68071dd35eab47d3cea2edea7c 4 m4 c0 2 +r036 d8e6c7b48648fa9c52011e592e0a88ec 4 m4 c1 2 +r037 ade6941768c396439f8667962c86db72 4 m2 c1 2 +r038 e3fd91b83327cb14ce581423c69ec9cf 4 m2 c0 2 +r039 dd2bec812c16181d87ba7704dde26eba 4 m3 c1 2 +r040 87c0b4c22af01464c1409ebdadcf935b 4 m1 c1 2 +r041 ca9a51c748d12675cf8ad67d1ba1557d 5 m2 c1 1 +r042 731af655906ddde75f94adda71951585 5 m0 c1 1 +r043 bfbb18dae72c76b471f0597ee61354c1 5 m3 c1 1 +r044 649b1cb579bfd36bb7c870f71f4a710b 5 m0 c0 1 +r045 085efb6bfa93ab54a4f3cfef9ec74c56 5 m1 c1 1 +r046 ba8cf964fdde681e2fee804faaa42bc7 5 m4 c0 1 +r047 487d4d737d629f571b1f954076797aa2 5 m3 c0 1 +r048 115c6fcf19fa84b860611c79cbeb4c34 5 m2 c0 1 +r049 c64b92f9791af578be6d336f10ede80c 5 m4 c1 1 +r050 1c2c809641b0da11ce16902e21a56185 5 m1 c0 1 diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/presentation-orders.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/presentation-orders.tsv new file mode 100644 index 0000000000..eb1034cd47 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/presentation-orders.tsv @@ -0,0 +1,11 @@ +claim labels_in_order +P-s1 G,I,A,J,H,C,D,B,F,E +P-s2 C,A,G,J,E,I,F,D,B,H +B-s1 H,A,C,G,F,D,B,E,J,I +B-s2 C,J,H,D,E,B,G,A,F,I +L-s1 F,J,I,B,E,H,C,G,D,A +L-s2 J,F,D,G,H,C,A,B,E,I +Q-s1 C,A,B,F,I,H,G,J,D,E +Q-s2 I,D,G,J,A,B,H,F,E,C +R-s1 B,D,C,G,A,H,I,E,F,J +R-s2 H,I,E,C,A,B,G,F,J,D diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/scoring-schedule.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/scoring-schedule.tsv new file mode 100644 index 0000000000..9456c33856 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/scoring-schedule.tsv @@ -0,0 +1,11 @@ +claim +B-s2 +P-s2 +P-s1 +R-s2 +Q-s2 +L-s2 +L-s1 +R-s1 +B-s1 +Q-s1 diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/seeds.json b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/seeds.json new file mode 100644 index 0000000000..8002c0e709 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/seeds.json @@ -0,0 +1,8 @@ +{ + "blind": "4d74ebbc4ad41ca9a3593e71cfe3347846fbeff3b11bed53557a153a645f3798", + "condition": "b21996121580e438f506505a45112131620af131b736c821ee564267a5c1f924", + "consistency": "159e1c6c4e71d85612257791b363d44fbd7c7a61bd36098fdf966b3d60a5c328", + "presentation": "59edc5bb403b9a046bf95963ba608ee34b148ed53d28098877b45cb6922cda0a", + "schedule": "5de55b1dfc1b4f958b103c91bd2ed9b0c521c123cd657200e49113923dd93824", + "scorer": "2b2f142efbc9040f145d0facbb9a5f0e5e948117ed4df1ab94aa0debfce3bc88" +} diff --git a/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/target-map.tsv b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/target-map.tsv new file mode 100644 index 0000000000..e0cc0ac770 --- /dev/null +++ b/evals/unsafe-rust/runs/2026-08-01-v4-focused/sealed/target-map.tsv @@ -0,0 +1,6 @@ +target_label mode source_path tree_sha256 word_cap +m0 R fixtures/v4-focused/r_redesign d69df1b286abd8f7f8955ac56d702c1910eb596c0bd105b7998e39d7246ca063 1800 +m1 B fixtures/v4-focused/b_build 7589027142112e387f990314df7eb1d08e5464448566fd68048eb2a748635bf3 3200 +m2 L fixtures/v4-focused/l_proof cc05da115d055febc313edcdf18bae59a6230a63583bf918ad29e89eb06a4266 2200 +m3 Q fixtures/v4-focused/q_quantifiers 35bb6be0402f9d81918c3afc850dd54cde012865bba90d0cd8d7042d78a582ee 1800 +m4 P fixtures/v4-focused/p_predicates 2b194a735b69a8904b86baa43791a0ddac9f769ce32e87bf4e759822cb5cd52e 3000