feat(knowledge): judge calibration harness (agreement_rate, meets_bar)#200
Merged
Conversation
…match Long-loop review of plan.md's own draft gate: verified design.md's four Open Artifact Decisions are already resolved in the required format, and the step count (6, Step 0-5) is within the 3-7 range. Caught two defects against the actual worktree before build: - Step 0's HumanVerdict/Agreement derive lists omitted Serialize/Deserialize, which ExampleAgreement (embedding both, itself deriving Serialize/ Deserialize per report.rs's AssertionReport convention) requires to compile. - The committed feature test (d08a77c) called compute_calibration and read the return value's fields directly, with no .expect()/?, i.e. it was authored against the design's original bare-CalibrationReport sketch before decision 4 changed the signature to Result<CalibrationReport, CalibrationError>. Kept the Result-returning signature (decision 4's typed-error rationale stands on its own merits, matching EngineError/ScriptError's library-boundary convention) and fixed the two call sites in the feature test with .expect(...) instead -- a syntax-only fix; the fixture is fully well-formed and only ever hits Ok. Removed the Draft marker; plan.md's Resolved-by-the-long-loop-reviewer block records both fixes and the verification pass.
Add HumanVerdict, Agreement, ExampleAgreement, JUDGE_CALIBRATION_BAR, CalibrationReport, CalibrationError, and the compute_calibration signature (returning Result<CalibrationReport, CalibrationError> per design decision 4). No folding logic yet -- body is todo!(). Registers pub mod calibration in knowledge/mod.rs. The feature test now fails with a todo!() panic (compiles cleanly) instead of the prior unresolved-import error.
compute_calibration now validates, in case order, that every case matches exactly one conversation (else NoMatch/MultipleMatches) and that match carries exactly one assertion (else MultipleAssertions), short-circuiting before any label lookup or folding. Adds three unit tests covering each rejected shape. compute_calibration still ends in todo!() for the well-formed path (Step 2+).
…plan Step 2) compute_calibration now checks, after Step 1's shape validation, that each case's name has an entry in the supplied labels map, returning CalibrationError::MissingLabel before touching any counters otherwise. Adds a unit test proving a case absent from labels is rejected by name. Still ends in todo!() for the fully-covered happy path (Step 3+).
…ionReport (plan Step 3) compute_calibration now assembles the full result: cases clearing Steps 1-2's validation are folded into ValidatedCase, "errored"/"deferred" outcomes are marked Agreement::Excluded and counted in `excluded` (not `agreements`), and `considered = total_examples - excluded`. The per-outcome agree/disagree comparison (Step 4) and the agreement_rate/ meets_bar formula (Step 5) are still placeholders (every non-excluded outcome defaults to Agree; rate/bar are 0.0/false), marked with TODOs. Adds a unit test for the exclusion accounting. The feature test now progresses past total_examples/excluded/considered and fails at the agreements == 9 assertion (currently 10, since the real pass/fail/malformed comparison lands in Step 4), which is the expected next-step boundary.
…e (plan Step 4)
Replace the Step 3 placeholder ("every non-excluded outcome agrees") with
the real outcome-to-agreement mapping: "pass" agrees with HumanVerdict::Pass,
"fail" agrees with HumanVerdict::Fail, "malformed" (and any unrecognized
outcome) always disagrees regardless of label -- a hedge is never treated
as agreement. Adds unit tests for pass-vs-both-labels and malformed-vs-both-
labels.
The feature test now progresses past agreements == 9 / example-07's
Disagree / agree_count == 9 and fails at agreement_rate (currently the
Step 3 placeholder 0.0), which is Step 5's job.
Replace the Step 3 placeholder rate/bar (0.0/false) with the real formula: agreement_rate = agreements / considered (0.0 when considered == 0), meets_bar = considered > 0 && agreement_rate >= JUDGE_CALIBRATION_BAR (inclusive, per design.md's Bar semantics). Adds unit tests for the 8/9 under-the-bar boundary and the 9/10 exactly-on-the-bar boundary. The feature test (tests/judge_calibration.rs) now passes end to end: total_examples, excluded, considered, agreements, agreement_rate, meets_bar, and the per-example example-07 disagreement are all correct. Full crate suite: 255/255 passing. cargo clippy --all-features --all-targets clean (added one #[expect(clippy::cast_precision_loss, ...)] on the usize->f64 ratio, matching main.rs's #[expect] convention over #[allow]). cargo fmt --check clean.
…g-conversation placeholder compute_calibration's NoMatch branch only fired on case.matches.len() == 0, but evaluate() never actually leaves matches empty for a named case with zero real conversation matches -- it synthesizes exactly one MatchReport with an empty conversation string and a single "missing_conversation"-class AssertionReport (outcome "malformed"). That made case.matches.len() >= 1 for every realistic name:-based calibration case, so a typo'd/missing conversation file silently became a counted Disagree instead of surfacing CalibrationError::NoMatch -- exactly the false-confidence failure mode this type exists to prevent. Detect that synthesized placeholder by its reserved MISSING_CONVERSATION_CLASS assertion class (now pub(crate) in eval.rs) and return Err(CalibrationError::NoMatch) for it before folding it in as a graded outcome. Adds a unit test driving compute_calibration directly and an integration regression test in tests/judge_calibration.rs that drives the real evaluate() orchestrator with a named case whose conversation file does not exist, asserting the NoMatch error surfaces end to end. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…Assertions
Assertion-count validation collapsed both zero and more-than-one assertions
into the single MultipleAssertions variant, so a case whose match carried
assertions: [] produced the internally contradictory
MultipleAssertions { count: 0 } -- a "multiple" error reporting a count of
zero. Split it the same way match-count validation already splits
NoMatch (zero) from MultipleMatches (more than one): add a distinct
NoAssertions { case } variant for the zero case, keeping MultipleAssertions
strictly for count > 1. Adds a regression test for assertions: [] asserting
the new NoAssertions variant, alongside the existing >1-assertions test
confirming MultipleAssertions still fires correctly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DavidSouther
force-pushed
the
feature/e-judge-calibration
branch
from
1bddfda to
8b27529
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of the
2026-07-06-A-ailly-evalsproject, Feature-step E (core). No prior mechanism checked thejudgeassertion's ternary P/F/I verdicts against a human labeler's ground truth. This addscalibration.rs: look up each case's human label (hard error on omission), map judge-outcome x human-verdict into Agree/Disagree, exclude errored/deferred outcomes, and computeagreement_rate/meets_barinto an assembledCalibrationReport.Status
10 commits, feature test green. Adversarial review found two real bugs, both fixed in the last two commits:
CalibrationError::NoMatchwas unreachable becauseevaluate()always synthesized a placeholder match for a named case (a typo'd conversation filename silently counted as "Disagree" instead of erroring), and a zero-assertion case was wrongly folded into aMultipleAssertions{count: 0}variant — split intoNoAssertions/MultipleAssertions.The actual calibration run against this harness (19 human-graded cells, see follow-on PRs) came back at 42% agreement against a 90% bar — real miscalibration found in
newtypeandemitting-logsjudge rubrics, addressed in a separate PR (fix/judge-rubric-calibration).Test plan
cargo test