[KatalKavya96]

<!-- SPDX-License-Identifier: Apache-2.0 https://www.apache.org/licenses/LICENSE-2.0 -->

Summary

• Fixes skill-evals: Haiku judge rejects rationales its own reason calls a match; three issue-reproducer fixtures assert unsupported facts #1688: The Haiku prose grader sometimes returns match: false even when its own explanation acknowledges that the candidate reaches the same conclusion, particularly when the candidate includes an additional recommended action.

• Strengthens both single-field and batch grading rubrics with a concrete example and explicit verdict/reason consistency guidance, without changing verdict handling or retry semantics.

• Corrects three issue-reproducer fixtures containing incomplete or unsupported expected reasoning, updates the evaluation documentation, and adds deterministic regression tests.

Type of change

• Skill change (.claude/skills/<name>/) — eval fixtures updated below

• Tool / bridge contract (tools/<system>/*.md)

• Python package (tools/*/ with pyproject.toml)

• Groovy reference impl

• Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)

• Documentation (docs/, README.md, CONTRIBUTING.md)

• Project template (projects/_template/)

• CI / dev loop (prek, workflows, validators)

• Other: Evaluation rubric improvements, fixture corrections, and deterministic regression tests.

Test plan

• prek run --all-files passes — all repository hooks green.

• For Python packages touched: uv run pytest / ruff check / mypy passes.

  • Full tools/skill-evals/tests/ test suite passed.

  • Focused test_runner.py test suite passed.

  • Ruff checks passed.

  • Workspace mypy hook passed.

• For Groovy bridges touched: command-line invocation tested end-to-end — not applicable.

• For skill changes: eval suite passes for the affected skill — live evaluation was only partially completed.

• For skill behaviour changes: a new or updated eval fixture is included in this PR.

  • Corrected three existing issue-reproducer fixtures.

  • Added four deterministic regression tests covering rubric guidance, grader prompt construction, and preservation of explicit negative verdicts.

• Other:

  • python3 tools/dev/check-doc-sync.py — passed.

  • git diff --check — passed.

Live Claude evaluation (post-fix, partial run)

Executed:

PYTHONPATH=tools/skill-evals/src \
uv run --project tools/skill-evals \
python -m skill_evals.runner \
  tools/skill-evals/evals/issue-triage/ \
  --cli "claude -p" \
  --timeout 180

Results: 22 cases attempted — 13 passed, 0 grading failures, 0 manual, 9 execution errors.

Evaluation step | Passed | Errors -- | -- | -- Step 1 — Resolve selector | 4/4 | 0 Step 3 — Classify | 7/7 | 0 Step 4 — Compose comment | 2/4 | 2 Step 5 — Confirm | 0/4 | 4 Step 7 — Recap | 0/3 | 3 Total | 13/22 | 9

The post-fix run provides initial positive evidence for the affected issue-triage classification cases. However, it is not directly comparable to the historical full-suite pass rates.

Repeated live evaluations are still needed to establish whether the new rubric reduces intermittent Haiku grading failures.

Deterministic regression coverage

Added four tests in tools/skill-evals/tests/test_runner.py covering:

1. Both grading rubrics include compatible-extra-action guidance and verdict/reason consistency instructions.

2. The single-field grader receives the updated guidance.

3. The batch grader receives the updated guidance.

4. An explicit match: false verdict remains negative even when the judge's explanation says the candidate matches, preserving the original verdict and preventing retry-based pass inflation.

All focused tests and the complete skill-evals Python test suite passed.

RFC-AI-0004 compliance

• HITL — any new mutation is gated on explicit user confirmation. Not applicable; no new mutation is introduced.

• Sandbox — no new unrestricted host access or network capability is introduced.

• Vendor neutrality — grading guidance remains model-independent and does not introduce provider-specific logic. check-placeholders passed.

• Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable. Not applicable; no adopter-facing override is introduced.

• Write-access discipline — no autonomous outbound messages, tracker changes, or additional write operations are introduced.

• Privacy LLM — existing privacy and model-routing boundaries remain unchanged; no new private-data exposure path is introduced.

Linked issues

Fixes #1688.

Notes for reviewers (optional)

1. Grader rubric improvements

Updated GRADER_RUBRIC and BATCH_GRADER_RUBRIC to explicitly clarify that an additional compatible recommended action should not be treated as a contradiction merely because the reference answer omits it.

For example:

• Expected: The reports describe the same root cause.

• Candidate: The reports describe the same root cause; close the newer report as a duplicate.

• Expected grading: Match, because the additional action is consistent with the conclusion.

The rubrics also instruct the judge to ensure that its boolean verdict agrees with its explanation.

2. Fixture corrections

step-1-inventory/case-3-no-code-blocks

The expected reporter_env.notes previously contained only PostgreSQL 15, while the report also described the scheduler failing after approximately 48 hours without restart and no logged error.

Updated the expected notes to include those source-supported details.

step-3-classify-shape/case-2-shape-b-missing-imports

The expected rationale incorrectly claimed that the snippet lacked a main entry point, even though task.execute(context={}) was present.

Removed that claim while preserving the correct Shape B classification and the genuinely missing import and DAG definition.

step-7-verify/case-4-timeout

The expected rationale claimed that the scheduler was operating normally up to the cutoff, which was not established by the execution output.

It also contained an inconsistent compressed-time simulation note.

Updated the fixture to reflect only the supported observations: the bounded execution timed out with exit code 124 after 60 seconds, and the output does not establish whether the reported 48-hour failure reproduces.

The structured classification and nature values remain unchanged.

3. Design decisions and limitations

This PR intentionally preserves existing verdict handling:

• No automatic conversion of match: false to true based on the judge's explanation.

• No retry of valid negative verdicts.

• No grading threshold changes.

• No changes to grading schemas or structured decision comparison.

These decisions prevent the reliability fix from masking genuine model errors or making evaluation results artificially optimistic.

Review focus: Whether the new rubric guidance sufficiently clarifies compatible extra actions without encouraging overly permissive matching.

The deterministic implementation is validated, but repeated post-fix live evaluations remain necessary to quantify the improvement in Haiku grading consistency.

[KatalKavya96]

@potiuk Implementation and validation are complete.

Changes:

• Improved both single-field and batch grader rubrics to clarify that compatible additional actions should not cause false-negative matches.
• Corrected three issue-reproducer fixtures containing incomplete or unsupported expectations.
• Added four deterministic regression tests while preserving existing negative-verdict and retry behavior.

Validation:

• prek run --all-files — passed.
• Full skill-evals pytest suite — passed.
• Ruff and documentation synchronization — passed.
• Live Claude issue-triage evaluation — 13 passed, 0 grading failures, 9 CLI execution errors out of 22 cases.

All seven classification cases passed, including the previously intermittent cases. The remaining nine cases could not be graded because the Claude CLI exited with code 1 without usable JSON.

Claude usage was exhausted before completing further evaluations. Repeated live runs are still needed to quantify the reliability improvement.

The changes are intentionally limited to rubric guidance, fixture accuracy, and regression coverage.

Thanks!