test(evals): harden plan-validator discrepancy stimulus against prompt echo - #2446
test(evals): harden plan-validator discrepancy stimulus against prompt echo#2446vdstrizhkova wants to merge 7 commits into
Conversation
ensure that the evaluation fixtures are correctly validated. This includes updating the eval.yaml configuration and refining the stimuli in plan-validator.yml to cover edge cases and enhance test coverage.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2446 +/- ##
=======================================
Coverage ? 82.66%
=======================================
Files ? 142
Lines ? 20688
Branches ? 0
=======================================
Hits ? 17101
Misses ? 3587
Partials ? 0
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Add a cspell:ignore directive for the regex stems (discrepanc, deviat, authenticat, paginat) used in the plan-validator-discrepancy-log grader patterns, fixing the Spell Check CI failure on PR microsoft#2446.
|
@vdstrizhkova - I think you found a bug in the eval runner .. looks like when the eval files change, the evals are not being re-run. Can you give copilot a quick kick to see if it can fix this such that when eval files change, the matrix runner sees that eval change and runs those evals? |
regenerate eval file
The eval-execute job is skipped on fork PRs (repository secrets are not exposed to forks), which let a changed stimulus pass green without any eval running. Add an eval-execute-guard job that runs regardless of fork status and fails with an actionable message when eval-relevant files changed but eval execution did not run, so the change is never a silent no-op.
|
@WilliamBerryiii the issue was that if the stimuli was change eval is not being regenerated and the tests would be skipped. |
Fixes #2444.
Problem
The
plan-validator-discrepancy-logstimulus inevals/agent-behavior/stimuli/plan-validator.ymldid not meaningfully test discrepancy detection:.copilot-tracking/plans/example.mdand.copilot-tracking/research/example.md, which do not exist in the repo, so there was nothing real to validate..copilot-tracking/plans), so a reply could pass by echoing the prompt — even a "files not found" response with zero findings.Fix
Rewrote the stimulus to give an inline plan + research pair with two genuine contradictions (missing OAuth authentication; a single-response endpoint vs. a 500-row pagination cap), and removed all output-vocabulary hints from the prompt. Tightened the graders:
discrepancy-detected— matches analytical vocabulary (discrepanc|deviat|not addressed|missing|gap|DR-\d|DD-\d) that no longer appears in the prompt, so a pass requires the agent to actually surface the conflict.substantive-gap-named— confirms engagement with the real gaps (authenticat|oauth|paginat|500).validation-status-reported— confirms a reported validation status.Regenerated
evals/agent-behavior/eval.yamlfrom the partial.Validation
Build-AgentBehaviorSpec.ps1 -Force— spec regeneratedeval:lint:schema— passBuild-AgentBehaviorSpec.ps1 -WhatIf— no driftScope
Scoped to the primary
plan-validator-discrepancy-logstimulus named in the issue. The relatedplan-validator-coverage-matrixandrpi-validator.ymlshare the same pattern and can be addressed in a follow-up.