diff --git a/.changes/planning/test-6946-interview-negative-recommendation-eval-19a6f175.md b/.changes/planning/test-6946-interview-negative-recommendation-eval-19a6f175.md new file mode 100644 index 0000000000..0d5be0b5a7 --- /dev/null +++ b/.changes/planning/test-6946-interview-negative-recommendation-eval-19a6f175.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Added + +- **An interview eval case for a question whose natural recommendation is negative.** `interview-negative-recommendation-yes-accepts` floats a side idea (rewriting the admin page in React) that the interview should recommend against, and checks that round 1 asks about it and words the question so that answering "yes" accepts the recommendation. The skill body is unchanged: a with-and-without run of this case decides whether a wording rule is added. diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/asks-about-rewrite.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/asks-about-rewrite.md new file mode 100644 index 0000000000..7df72b36ca --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/asks-about-rewrite.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '[^\n?]*\b(?:rewrite|rewriting|rebuild\w*|(?:port|migrat|convert|mov|switch)\w*[^\n?]{0,60}\bto React)\b[^\n?]*\?' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/no-polar-no-recommendation.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/no-polar-no-recommendation.md new file mode 100644 index 0000000000..cd72b8dbac --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/no-polar-no-recommendation.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '\b(?:should|shall|do|does|will|would|can|could|are|is)\b[^\n?]{0,200}\b(?:rewrite|rewriting|rebuild\w*|(?:port|migrat|convert|mov|switch)\w*[^\n?]{0,60}\bto React)\b[^\n?]*\?(?:(?!\n\s*\**Q\d+)[\s\S]){0,400}?My recommendation:\s*(?:\*\*)?\s*(?:no|don.?t|do not)\b' +flags: i +match: not_contains +arm: both +--- diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/yes-accepts-recommendation.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/yes-accepts-recommendation.md new file mode 100644 index 0000000000..6cb20c14ae --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/yes-accepts-recommendation.md @@ -0,0 +1,24 @@ +--- +type: llm +arm: both +--- + +Find the question in the reply that asks about rewriting the admin page in React. Judge only that +question and the recommendation given for it. + +PASS only if all of these hold: + +1. The reply asks a question about the React rewrite. +2. The recommendation for that question keeps the rewrite out of this change (defer it, do it + separately, or skip it). A recommendation to do the rewrite in this change fails, because the + case then never reaches the negative recommendation it measures. +3. A user who replies to that question with a bare "yes" (or "Q yes") would be accepting the + recommended answer. Either the question is a yes/no question whose "yes" is the recommended + answer (for example "Should the React rewrite stay out of this change?" with a recommendation to + keep it out), or it is a choice question that names options and marks one as recommended, so + "yes" can only mean accepting that recommendation. + +FAIL if the reply asks no question about the rewrite, if it recommends doing the rewrite in this +change, or if the question is a yes/no question whose +literal "yes" is the opposite of the recommendation (for example "Should we rewrite the admin page +in React?" with a recommendation of "No", "Not now", or "Keep it out of scope"). diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md new file mode 100644 index 0000000000..c5d1e697c9 --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Hard case. Why hard: the prompt floats a side idea (rewriting the whole admin page in React) whose natural recommendation is negative, so the question about it is easily worded 'Should we rewrite it?' with a recommended 'No', where a reply of 'yes' rejects the recommendation. Upstream mattpocock/skills#706." +tags: [interview, question-wording, negative-recommendation] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "Round 1 asks about the React rewrite, recommends keeping it out of this change, and words that question so that answering 'yes' accepts the recommendation" +--- + +/planning:interview Add a CSV export button to the admin orders page so ops can download the filtered order list. I'm also wondering whether we should rewrite the whole admin page in React while we're in there. diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/asks-about-rewrite.json b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/asks-about-rewrite.json new file mode 100644 index 0000000000..d0dc077639 --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/asks-about-rewrite.json @@ -0,0 +1,38 @@ +{ + "pass": [ + { + "answer": "Q1: Which columns should the CSV include?\nMy recommendation: **the columns the filtered table shows** - ops already reads them.\nBasis: judgment\n\nQ2: Should the React rewrite stay out of this change, as its own follow-up?\nMy recommendation: **Yes, keep it out** - the export is small and the rewrite is not.\nBasis: judgment", + "why": "asks about the rewrite" + }, + { + "answer": "Round 1\n**Q2: Is rewriting the page in scope?**", + "why": "rewriting in a bold question" + }, + { + "answer": "Q4: Should we port the page to React now?", + "why": "asks about moving the page to React" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Q1: Which columns should the CSV include?\nQ2: Should the export respect the current filters?", + "why": "never asks about the rewrite" + }, + { + "answer": "The React rewrite is out of scope.\nQ1: Which columns should the CSV include?", + "why": "states the rewrite decision instead of asking it" + }, + { + "answer": "Q1: Is the page already using React components?", + "why": "mentions React but does not ask about the rewrite" + }, + { + "answer": "Q1: Is the admin page currently written in React?", + "why": "asks about the current stack, not the rewrite" + } + ] +} diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/no-polar-no-recommendation.json b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/no-polar-no-recommendation.json new file mode 100644 index 0000000000..cd0c3c1b20 --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/no-polar-no-recommendation.json @@ -0,0 +1,46 @@ +{ + "pass": [ + { + "answer": "Q1: Which columns should the CSV include?\nMy recommendation: **the columns the filtered table shows** - ops already reads them.\nBasis: judgment\n\nQ2: Should the React rewrite stay out of this change, as its own follow-up?\nMy recommendation: **Yes, keep it out** - the export is small and the rewrite is not.\nBasis: judgment", + "why": "polar question worded so yes keeps the rewrite out, matching the recommendation" + }, + { + "answer": "Q2: How should the React rewrite relate to this change?\nMy recommendation: **a separate follow-up** - the export ships on the current page.\nAlternatives: rewrite first, then add the export; do both in one change.", + "why": "choice question about the rewrite, recommendation named" + }, + { + "answer": "Q2: Do you want the React rewrite deferred to its own change?\nMy recommendation: **Yes, defer it**.", + "why": "polar question whose yes is the recommendation" + }, + { + "answer": "Q1: Should the export respect the current filters?\nMy recommendation: **No** - export everything.\n\nQ2: Should the React rewrite wait for its own change?\nMy recommendation: **Yes**.", + "why": "a No recommendation on an unrelated question is out of this grader's scope" + }, + { + "answer": "Q2: Should the React rewrite stay out of this change?\nMy recommendation: **Yes**.\n\nQ3: Should the export respect the current filters?\nMy recommendation: **No** - export everything.", + "why": "the next question's No does not count against the rewrite question" + }, + { + "answer": "Q1: Is the page already using React components?\nMy recommendation: **No** - it is server-rendered.", + "why": "a fact question that names React, not the rewrite" + }, + { + "answer": "Q1: Is the admin page currently written in React?\nMy recommendation: **No** - it uses server templates.", + "why": "a current-stack question, not the rewrite" + } + ], + "fail": [ + { + "answer": "Q1: Which columns should the CSV include?\nMy recommendation: **the visible columns**.\n\nQ2: Should we rewrite the whole admin page in React as part of this?\nMy recommendation: **No** - keep this change to the export button.\nBasis: judgment", + "why": "yes to Q2 means rewriting, the opposite of the recommendation" + }, + { + "answer": "Q3: Should we also rewrite it in React?\nMy recommendation: **Don't** - out of scope.", + "why": "Don't recommendation on a should-we-rewrite question" + }, + { + "answer": "Q3: Is a React rewrite part of this work?\nMy recommendation: **no, not in this change**.", + "why": "lowercase no on a polar rewrite question" + } + ] +} diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/yes-accepts-recommendation.json b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/yes-accepts-recommendation.json new file mode 100644 index 0000000000..cead8e1d9c --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/yes-accepts-recommendation.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "Q1: Which columns should the CSV include?\nMy recommendation: **the columns the filtered table shows** - ops already reads them.\nBasis: judgment\n\nQ2: Should the React rewrite stay out of this change, as its own follow-up?\nMy recommendation: **Yes, keep it out** - the export is small and the rewrite is not.\nBasis: judgment", + "why": "yes to Q2 accepts keeping the rewrite out" + }, + { + "answer": "Q2: How should the React rewrite relate to this change?\nMy recommendation: **a separate follow-up** - the export ships on the current page.\nAlternatives: rewrite first, then add the export; do both in one change.", + "why": "choice question; yes can only accept the named recommendation" + } + ], + "fail": [ + { + "answer": "Q1: Which columns should the CSV include?\nMy recommendation: **the visible columns**.\n\nQ2: Should we rewrite the whole admin page in React as part of this?\nMy recommendation: **No** - keep this change to the export button.\nBasis: judgment", + "why": "yes means rewriting, against the recommendation" + }, + { + "answer": "Q2: Should we rewrite the admin page in React while we're in there?\nMy recommendation: **Keep it out of scope** - a rewrite is a separate project.", + "why": "yes means rewriting while the recommendation keeps it out" + }, + { + "answer": "Q2: Should we rewrite the admin page in React?\nMy recommendation: **Yes, rewrite it now** - the export is easier on a React page.", + "why": "recommends doing the rewrite, so the negative case is never reached" + }, + { + "answer": "Q1: Which columns should the CSV include?\nMy recommendation: **the visible columns**.", + "why": "no question about the rewrite" + } + ] +}