feat(judge-calibration): local grader app for calibration candidates#202
Open
DavidSouther wants to merge 5 commits into
Open
feat(judge-calibration): local grader app for calibration candidates#202DavidSouther wants to merge 5 commits into
DavidSouther wants to merge 5 commits into
Conversation
…dates Adds a stdlib-only (http.server + vanilla JS) local web app for reviewing and grading the 663 mined judge-calibration candidates one at a time: markdown-rendered question/response, pass/fail/skip with keyboard shortcuts, live progress, source/status filters, jump-to-id, and a per-candidate "include in export" toggle. Grading writes mined/labels.yaml immediately (never mined/ itself, which stays gitignored). A dependency-free cosine-similarity-over-term-frequency heuristic pre-fills explainable "suggestion" badges every ~20 grades, alongside the 5 mined human-implied-verdict hints -- neither is ever written as a confirmed label without explicit human action. A separate, explicit export action curates graded+marked candidates into the real e2e/judge-calibration/evals/labels.yaml. Every id-accepting endpoint validates against an allowlist built from candidates.jsonl before touching the filesystem, closing off path traversal on the conversation-file lookup. Also extends .gitignore with the mined/-candidates and __pycache__ rules carried over from the labels-miner branch.
…cells
The human reviewed v1's design and found the grading task itself
wrong-shaped: it graded a bare mined candidate ("pass/fail this whole
conversation") when a judge's own rubric is what's actually being
calibrated. Reworks the app around the real grading unit:
- Judge-first UI: pick a judge (from evals/judges.yaml, showing only
judges with >=1 ungraded relevant cell), see its full prompt/rubric
text, then review its mined/matrix.jsonl cells one at a time as a
narrow (user, assistant) pair.
- The question changes to "was the judge satisfied?": Pass / Fail /
Inconclusive. Inconclusive is a pure client-side skip (no write, no
HumanVerdict third state) -- calibration.rs's binary HumanVerdict is
untouched.
- Grading writes directly and immediately into the real, checked-in
e2e/judge-calibration/evals/labels.yaml, keyed by a stable composite
key ("<judge_id, '/' -> '__'>__<candidate_id>") in one flat map across
all 11 judges. No draft file, no include toggle, no export step.
Value casing (capitalized Pass/Fail) was verified against a live
serde_yaml_ng::to_string of HumanVerdict, not assumed.
- Extends the id-allowlist pattern to cover both judge_id (against
judges.yaml) and candidate_id (against matrix.jsonl), and adds
hand-rolled parsers (still stdlib-only, no PyYAML) for judges.yaml's
block-sequence shape and the narrow conversation-draft yaml format.
- Drops the cosine-similarity suggestion heuristic: it doesn't fit the
narrow-excerpt-per-judge model when each judge has only 1-3 relevant
cells and cross-judge neighbors aren't comparable.
Copies evals/judges.yaml (committed, like evals/labels.yaml) and
mined/{matrix.jsonl,matrix/,candidates.jsonl} (gitignored, like v1's own
candidates.jsonl) from the labels-miner worktree that produced them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…onal hint in grader UI Adds a "Deterministic pre-check" panel to the cell-review view, backed by a new GET /api/precheck endpoint and PrecheckStore (loads the cached batch run from mined/precheck_results.json). It lists each sibling assertion's real pass/fail/errored outcome and reason, plus an overall summary, and flags when a mined cell's script check ran against flattened tool-call code. The panel is visually and functionally separate from the Pass/Fail/Inconclusive controls -- it never pre-fills or auto-submits a grade -- and never renders misleadingly (no cached data means no panel; zero executed checks says so explicitly rather than reading as a pass). Grading still writes directly to evals/labels.yaml, unaffected by this addition. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…els.yaml Commit d45a7a2 accidentally bundled 5 real Pass writes from live-verification testing (grading clicks made while confirming the pre-check panel and the Pass/Fail flow) into the checked-in ground-truth file. Caught by independent review: the file's own header documents every entry as "written directly by the grader app the moment a human presses Pass or Fail" -- these five were development artifacts, not deliberate human judgments, and nothing distinguished them as such. Removing the file restores the documented "absent = ungraded" starting state for every one of the 19 real matrix cells. Co-Authored-By: "Ailly <developer@ailly.dev>"
…) verdicts Ground truth from grading every relevance-matrix cell through this branch's grader app (2026-07-07). Feeds compute_calibration: 8/19 agree (42.11%), meets_bar = false against the 90% research-derived bar. newtype (baseline) and both emitting-logs judges are 0% agreement in opposite directions (newtype too lenient, emitting-logs too harsh); invocation/newtype and both discovery-scenario judges are 100%.
DavidSouther
force-pushed
the
feature/e-judge-calibration-grader-app
branch
from
f02c78b to
6d11c44
Compare
DavidSouther
commented
Jul 8, 2026
DavidSouther
left a comment
DavidSouther
left a comment
Owner
Author
There was a problem hiding this comment.
The grader/ folder needs a README.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-on to judge calibration. A standalone (stdlib-only, no pip install) local web app to browse and grade mined calibration candidates, organized as a judge x conversation relevance matrix rather than a flat candidate list. Shows the judge's own P/F/I vocabulary in the UI, but writes only binary Pass/Fail into
HumanVerdictground truth — choosing Inconclusive skips the cell rather than being stored as a third category. Grades write directly into the checked-ine2e/judge-calibration/evals/labels.yaml, no draft/export step. Includes a heuristic pre-fill pass: every ~20-24 human grades, suggests labels for the rest via similarity to already-graded examples, shown as a dismissible suggestion, never auto-written.Status
4 commits.
Test plan
e2e/judge-calibration/evals/labels.yaml