feat(judge-calibration): mine calibration candidates from past sessions#201
Merged
Merged
Conversation
…agent sessions Resolves feature-e-judge-calibration/design.md's Open Artifact Decision #5: rather than hand-authoring 20-50 labeled examples, mine (question, candidate response) pairs out of real Claude Code and Codex session transcripts, wherever those sessions actually invoked Ailly. - Detects invocation via structural signals (attributionSkill/attributionPlugin and <command-name> tags for Claude Code; clean event_msg user-message text against Ailly-specific patterns for Codex) rather than naive text search, which would false-positive on every session's skill-catalog boilerplate. - Walks both tools' dedicated sub-agent transcripts (Codex's per-thread rollout files, Claude Code's <session>/subagents/agent-*.jsonl) in addition to top-level sessions. robustly. - Captures the model that produced each response when the transcript records one, null otherwise; never assigns a pass/fail verdict itself (labels.yaml entries are always TODO), and only records an explicit "human-implied" candidate hint, kept structurally separate from a confirmed label. - Skips malformed lines/files and missing source roots without crashing. Output goes to e2e/judge-calibration/mined/, now gitignored: transcripts span personal and client codebases alike and must stay local until a human curates a subset into the real evals/labels.yaml and runs/ tree. Real run against this machine's own history: 663 candidates (602 Claude Code, 61 Codex) across 1102 scanned files, 100% model-metadata coverage, 0 files/lines skipped.
Adds the judge-to-candidate relevance matrix the calibration app needs: given a real `judge` assertion, which mined conversation excerpts are even about the skill it checks, narrowed to the message pair nearest the match. - discover_judges.py (step 1): scans every e2e/*/evals/*.yaml for `type: judge` assertions and derives each one's topic keywords, preferring exact sibling text_contains/text_not_contains tokens, then the judge's own prompt text, then a deterministic assembly cross-reference (which skill's SKILL.md the case's matrix binding actually loaded) plus same-suite convention. Writes the checked-in e2e/judge-calibration/evals/judges.yaml. 2 of 11 judges (delegate-52's cross-provider corruption check, insurance-claim's $10k-ceiling regression check) are domain-specific with no skill signal and are flagged needs_human_review rather than forced. - skill_signals.py: shared regex-based skill/reference-tag extraction and matching, used identically by judge discovery, candidate tagging, and matrix building so "does this judge's keyword match this candidate's tag" means the same thing everywhere. - mine_calibration_candidates.py (step 2, extended): every mined candidate is now also tagged with skill_tags — Skill-tool invocations, <command-name> tags, plugin:name tokens, and references/SKILL.md path mentions found anywhere in the turn's raw transcript objects, not just the flattened question/response text. All 657 previously-mined candidates are kept; none are discarded by this pass. - build_relevance_matrix.py (steps 3-5): for each judge with resolved keywords, finds every candidate whose tags overlap, re-opens that candidate's raw transcript to locate the narrow (AskUserQuestion-or-user- prompt, assistant-message) pair nearest the match, and writes one draft conversation-schema YAML per matrix cell plus a matrix.jsonl index. Current run: 9/11 judges matched >=1 candidate, 17/657 candidates matched >=1 judge, 51 total matrix cells. mined/ (including the new matrix/ subpath) stays gitignored — only the derived judges.yaml and the scripts that produce it are committed.
…iffs Mining and the relevance-matrix narrow excerpt previously flattened an agentic turn to the assistant's prose text only, dropping the actual code change when it lived in a tool_use block's input instead of the final narration (e.g. "I've applied the five-layer pipeline" with no code in the text at all). mine_calibration_candidates.py now walks each turn's assistant objects for Edit/Write/MultiEdit/NotebookEdit tool_use blocks and stores their real file_path + diff/content verbatim on a new Candidate.tool_calls field, alongside (not merged into) the existing response text. build_relevance_matrix.py's narrow-excerpt writer renders that as a proper ContentBlock array (DESIGN.md's text/tool_use shape) when present, so a curled matrix draft shows the human reviewer both the narration and the exact diff a coding-pattern judge needs to grade. Scoped to Claude Code transcripts only, per the confirmed root cause; Codex and the synthetic-live conversations are unaffected. Re-ran both scripts end to end: 349/662 mined candidates now carry >=1 captured tool call, and 14/18 matrix-matched candidates (36/60 cells) render as ContentBlock arrays instead of plain text. mined/ output stays gitignored per existing policy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DavidSouther
force-pushed
the
feature/e-judge-calibration-labels-miner
branch
from
366f9d3 to
e128286
Compare
DavidSouther
commented
Jul 8, 2026
DavidSouther
left a comment
DavidSouther
left a comment
Owner
Author
There was a problem hiding this comment.
e2e/judge-calibration needs a README describing its purpose & usage.
PR #201 review: e2e/judge-calibration needs a README describing its purpose and usage. Covers the four-script pipeline (discover_judges, mine_calibration_candidates, build_relevance_matrix, skill_signals), the mined/ confidentiality rule, and how ground truth feeds compute_calibration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-on to judge calibration. A durable, reusable script that mines candidate labeling examples from the user's own past Claude Code / Codex session transcripts wherever they invoked Ailly, tags each by which skill/pattern/reference it actually used, and captures real
Edit/Write/MultiEdittool-call diffs alongside text (the initial pass flattened everything to final assistant text, losing the actual code changes for agentic sessions).Mined output (
e2e/judge-calibration/mined/) stays gitignored — it contains excerpts from other clients'/employers' private codebases mixed into the same transcript directories. Only the script itself is committed.Status
3 commits. Verdicts are never auto-assigned; a human confirms labels before they count as calibration ground truth.
Test plan
e2e/judge-calibration/mined/stays gitignored