Skip to content

test(evals): harden plan-validator discrepancy stimulus against prompt echo - #2446

Open
vdstrizhkova wants to merge 7 commits into
microsoft:mainfrom
vdstrizhkova:fix/2444-plan-validator-eval-fixtures
Open

test(evals): harden plan-validator discrepancy stimulus against prompt echo#2446
vdstrizhkova wants to merge 7 commits into
microsoft:mainfrom
vdstrizhkova:fix/2444-plan-validator-eval-fixtures

Conversation

@vdstrizhkova

Copy link
Copy Markdown
Contributor

Fixes #2444.

Problem

The plan-validator-discrepancy-log stimulus in evals/agent-behavior/stimuli/plan-validator.yml did not meaningfully test discrepancy detection:

  1. Dangling fixtures — it pointed the agent at .copilot-tracking/plans/example.md and .copilot-tracking/research/example.md, which do not exist in the repo, so there was nothing real to validate.
  2. Echo-passable graders — the prompt already contained the exact tokens the graders matched ("Discrepancy Log", "DR-", "DD-", .copilot-tracking/plans), so a reply could pass by echoing the prompt — even a "files not found" response with zero findings.

Fix

Rewrote the stimulus to give an inline plan + research pair with two genuine contradictions (missing OAuth authentication; a single-response endpoint vs. a 500-row pagination cap), and removed all output-vocabulary hints from the prompt. Tightened the graders:

  • discrepancy-detected — matches analytical vocabulary (discrepanc|deviat|not addressed|missing|gap|DR-\d|DD-\d) that no longer appears in the prompt, so a pass requires the agent to actually surface the conflict.
  • substantive-gap-named — confirms engagement with the real gaps (authenticat|oauth|paginat|500).
  • validation-status-reported — confirms a reported validation status.

Regenerated evals/agent-behavior/eval.yaml from the partial.

Validation

  • Build-AgentBehaviorSpec.ps1 -Force — spec regenerated
  • eval:lint:schema — pass
  • Build-AgentBehaviorSpec.ps1 -WhatIf — no drift

Scope

Scoped to the primary plan-validator-discrepancy-log stimulus named in the issue. The related plan-validator-coverage-matrix and rpi-validator.yml share the same pattern and can be addressed in a follow-up.

ensure that the evaluation fixtures are correctly
validated. This includes updating the eval.yaml
configuration and refining the stimuli in
plan-validator.yml to cover edge cases and enhance
test coverage.
@vdstrizhkova
vdstrizhkova requested a review from a team as a code owner July 10, 2026 12:47
@codecov-commenter

codecov-commenter commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@5c15a03). Learn more about missing BASE report.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2446   +/-   ##
=======================================
  Coverage        ?   82.66%           
=======================================
  Files           ?      142           
  Lines           ?    20688           
  Branches        ?        0           
=======================================
  Hits            ?    17101           
  Misses          ?     3587           
  Partials        ?        0           
Flag Coverage Δ
pester 86.20% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add a cspell:ignore directive for the regex stems (discrepanc, deviat, authenticat, paginat) used in the plan-validator-discrepancy-log grader patterns, fixing the Spell Check CI failure on PR microsoft#2446.
@WilliamBerryiii

Copy link
Copy Markdown
Member

@vdstrizhkova - I think you found a bug in the eval runner .. looks like when the eval files change, the evals are not being re-run. Can you give copilot a quick kick to see if it can fix this such that when eval files change, the matrix runner sees that eval change and runs those evals?

Comment thread evals/agent-behavior/stimuli/plan-validator.yml Outdated
vdstrizhkova and others added 5 commits July 16, 2026 10:52
The eval-execute job is skipped on fork PRs (repository secrets are not
exposed to forks), which let a changed stimulus pass green without any
eval running. Add an eval-execute-guard job that runs regardless of fork
status and fails with an actionable message when eval-relevant files
changed but eval execution did not run, so the change is never a silent
no-op.
@vdstrizhkova

Copy link
Copy Markdown
Contributor Author

@WilliamBerryiii the issue was that if the stimuli was change eval is not being regenerated and the tests would be skipped.
so a few changes added:
1 - drift will be detected and i have added a command to the message so user can see what needs to be rerun
2 - once the eval is regenerated, the reviewer needs to trigger the PR validation. There is also a message in the failure but it requires the maintainer to look at it. Not sure what the exact process is for reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: plan-validator eval stimuli reference nonexistent fixtures and pass on prompt echo

4 participants