From d57b1c690632b06c3b5da3183c1658f982991b00 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 17:32:02 -0400 Subject: [PATCH 1/4] feat(planning): route person-held interview questions to questionnaire and add eval-first cases Step 5 of /planning:interview now recommends /planning:questionnaire beside the next step when the Brief defers a question only a named person other than the user can answer (an assumption-sweep person item). This is a composition contract under ADR 0025: questionnaire's own description already expects interview branches that defer to a person. The interview-person-deferred-questionnaire eval case grades it. Seven eval-first cases cover interview behaviors the skill does not yet state: per-answer read-back of a dictated answer, an active final confirmation, a changes-only final restatement, a checkable basis and flip condition per recommendation, the "something else" closing probe, a MoSCoW scope cut before /planning:wayfind, and one crux question on a rejected recommendation. No rule for them ships here; each lands only if its case fails without it, which is the stumble evidence the instruction-economy rule asks for. Co-Authored-By: Claude Opus 5.5 --- .../feat-interview-techniques-323edf76.md | 8 +++++ .../graders/no-anything-else.md | 7 +++++ .../graders/says-something-else.md | 6 ++++ .../prompt.md | 11 +++++++ .../samples/no-anything-else.json | 22 ++++++++++++++ .../samples/says-something-else.json | 22 ++++++++++++++ .../graders/asks-about-kafka.md | 6 ++++ .../graders/no-reargue-name.md | 7 +++++ .../graders/one-crux-and-defers.md | 16 ++++++++++ .../interview-crux-on-rejection/prompt.md | 19 ++++++++++++ .../samples/asks-about-kafka.json | 22 ++++++++++++++ .../samples/no-reargue-name.json | 22 ++++++++++++++ .../samples/one-crux-and-defers.json | 18 +++++++++++ .../graders/changes-only-restatement.md | 16 ++++++++++ .../graders/omits-unchanged-items.md | 7 +++++ .../graders/shows-changed-item.md | 6 ++++ .../graders/shows-new-assumptions.md | 6 ++++ .../prompt.md | 13 ++++++++ .../samples/changes-only-restatement.json | 18 +++++++++++ .../samples/omits-unchanged-items.json | 18 +++++++++++ .../samples/shows-changed-item.json | 18 +++++++++++ .../samples/shows-new-assumptions.json | 22 ++++++++++++++ .../graders/active-reply-requested.md | 12 ++++++++ .../graders/asks-confirm-or-numbers.md | 6 ++++ .../graders/no-vague-confirm.md | 7 +++++ .../prompt.md | 13 ++++++++ .../samples/active-reply-requested.json | 18 +++++++++++ .../samples/asks-confirm-or-numbers.json | 30 +++++++++++++++++++ .../samples/no-vague-confirm.json | 26 ++++++++++++++++ .../graders/names-questionnaire.md | 6 ++++ .../graders/questionnaire-beside-next-step.md | 15 ++++++++++ .../prompt.md | 11 +++++++ .../samples/names-questionnaire.json | 26 ++++++++++++++++ .../questionnaire-beside-next-step.json | 18 +++++++++++ .../graders/names-feature-flag.md | 6 ++++ .../graders/not-verbatim-echo.md | 7 +++++ .../graders/reads-back-and-confirms.md | 17 +++++++++++ .../prompt.md | 19 ++++++++++++ .../samples/names-feature-flag.json | 22 ++++++++++++++ .../samples/not-verbatim-echo.json | 22 ++++++++++++++ .../samples/reads-back-and-confirms.json | 18 +++++++++++ .../graders/checkable-basis-and-flip.md | 18 +++++++++++ .../graders/states-flip-condition.md | 6 ++++ .../prompt.md | 11 +++++++ .../samples/checkable-basis-and-flip.json | 18 +++++++++++ .../samples/states-flip-condition.json | 26 ++++++++++++++++ .../graders/applies-must-test.md | 6 ++++ .../graders/scope-cut-before-wayfind.md | 18 +++++++++++ .../interview-scope-cut-moscow/prompt.md | 11 +++++++ .../samples/applies-must-test.json | 26 ++++++++++++++++ .../samples/scope-cut-before-wayfind.json | 18 +++++++++++ plugins/planning/skills/interview/SKILL.md | 2 +- 52 files changed, 768 insertions(+), 1 deletion(-) create mode 100644 .changes/planning/feat-interview-techniques-323edf76.md create mode 100644 plugins/planning/evals/interview-closing-probe-something-else/graders/no-anything-else.md create mode 100644 plugins/planning/evals/interview-closing-probe-something-else/graders/says-something-else.md create mode 100644 plugins/planning/evals/interview-closing-probe-something-else/prompt.md create mode 100644 plugins/planning/evals/interview-closing-probe-something-else/samples/no-anything-else.json create mode 100644 plugins/planning/evals/interview-closing-probe-something-else/samples/says-something-else.json create mode 100644 plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md create mode 100644 plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md create mode 100644 plugins/planning/evals/interview-crux-on-rejection/graders/one-crux-and-defers.md create mode 100644 plugins/planning/evals/interview-crux-on-rejection/prompt.md create mode 100644 plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json create mode 100644 plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json create mode 100644 plugins/planning/evals/interview-crux-on-rejection/samples/one-crux-and-defers.json create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/graders/changes-only-restatement.md create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/graders/omits-unchanged-items.md create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/graders/shows-changed-item.md create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/graders/shows-new-assumptions.md create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/prompt.md create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/samples/changes-only-restatement.json create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/samples/shows-changed-item.json create mode 100644 plugins/planning/evals/interview-final-brief-changes-only/samples/shows-new-assumptions.json create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/graders/active-reply-requested.md create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/graders/no-vague-confirm.md create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/prompt.md create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/samples/active-reply-requested.json create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json create mode 100644 plugins/planning/evals/interview-final-confirm-active-reply/samples/no-vague-confirm.json create mode 100644 plugins/planning/evals/interview-person-deferred-questionnaire/graders/names-questionnaire.md create mode 100644 plugins/planning/evals/interview-person-deferred-questionnaire/graders/questionnaire-beside-next-step.md create mode 100644 plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md create mode 100644 plugins/planning/evals/interview-person-deferred-questionnaire/samples/names-questionnaire.json create mode 100644 plugins/planning/evals/interview-person-deferred-questionnaire/samples/questionnaire-beside-next-step.json create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/graders/names-feature-flag.md create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/graders/not-verbatim-echo.md create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/graders/reads-back-and-confirms.md create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/prompt.md create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/samples/names-feature-flag.json create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/samples/not-verbatim-echo.json create mode 100644 plugins/planning/evals/interview-readback-dictated-answer/samples/reads-back-and-confirms.json create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/samples/checkable-basis-and-flip.json create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/graders/applies-must-test.md create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/graders/scope-cut-before-wayfind.md create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/prompt.md create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/samples/applies-must-test.json create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/samples/scope-cut-before-wayfind.json diff --git a/.changes/planning/feat-interview-techniques-323edf76.md b/.changes/planning/feat-interview-techniques-323edf76.md new file mode 100644 index 0000000000..4627d23a99 --- /dev/null +++ b/.changes/planning/feat-interview-techniques-323edf76.md @@ -0,0 +1,8 @@ +--- +bump: minor +--- + +### Added + +- `/planning:interview` recommends `/planning:questionnaire` at handoff for a deferred question only a named person other than the user can answer (an assumption-sweep `person` item), beside the usual next step. The new `interview-person-deferred-questionnaire` eval case grades it. +- Seven eval cases for interview behaviors not yet in the skill (per-answer read-back of a dictated answer, an active final confirmation, a changes-only final restatement, a checkable basis and flip condition per recommendation, the "something else" closing probe, a MoSCoW scope cut before `/planning:wayfind`, and one crux question on a rejected recommendation). Each rule lands only if its case fails without it. diff --git a/plugins/planning/evals/interview-closing-probe-something-else/graders/no-anything-else.md b/plugins/planning/evals/interview-closing-probe-something-else/graders/no-anything-else.md new file mode 100644 index 0000000000..bd44b9f47c --- /dev/null +++ b/plugins/planning/evals/interview-closing-probe-something-else/graders/no-anything-else.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '\banything else\b' +flags: i +match: not_contains +arm: both +--- diff --git a/plugins/planning/evals/interview-closing-probe-something-else/graders/says-something-else.md b/plugins/planning/evals/interview-closing-probe-something-else/graders/says-something-else.md new file mode 100644 index 0000000000..ad970bc912 --- /dev/null +++ b/plugins/planning/evals/interview-closing-probe-something-else/graders/says-something-else.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '\bsomething else\b[^\n]*\?' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-closing-probe-something-else/prompt.md b/plugins/planning/evals/interview-closing-probe-something-else/prompt.md new file mode 100644 index 0000000000..e633376d02 --- /dev/null +++ b/plugins/planning/evals/interview-closing-probe-something-else/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Eval-first case for the closing probe wording. Why hard: 'Anything else?' is the habitual close, and it invites a 'no'; 'Is there something else...?' left fewer concerns unmet in a primary-care trial (Heritage et al. 2007)." +tags: [interview, closing-probe, wording] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "Round 1 closes with a probe worded 'something else' (for example 'Is there something else that would change these recommendations?') and does not use 'anything else'" +--- + +/planning:interview Add a nightly job that deletes expired password-reset tokens from the database. diff --git a/plugins/planning/evals/interview-closing-probe-something-else/samples/no-anything-else.json b/plugins/planning/evals/interview-closing-probe-something-else/samples/no-anything-else.json new file mode 100644 index 0000000000..d5b64fdacb --- /dev/null +++ b/plugins/planning/evals/interview-closing-probe-something-else/samples/no-anything-else.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Is there something else about this job that would change these recommendations?", + "why": "something-else wording" + }, + { + "answer": "", + "why": "an empty reply holds no anything-else; the other grader fails it" + } + ], + "fail": [ + { + "answer": "Anything else I should know?", + "why": "anything else" + }, + { + "answer": "Is there anything else that would change these?", + "why": "anything else inside a longer probe" + } + ] +} diff --git a/plugins/planning/evals/interview-closing-probe-something-else/samples/says-something-else.json b/plugins/planning/evals/interview-closing-probe-something-else/samples/says-something-else.json new file mode 100644 index 0000000000..6d316c6814 --- /dev/null +++ b/plugins/planning/evals/interview-closing-probe-something-else/samples/says-something-else.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Q1: How long should an expired token be kept before deletion?\nMy recommendation: delete at once.\n\nIs there something else about this job that would change these recommendations?", + "why": "something-else probe as a question" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Q1: How long should an expired token be kept?\n\nAnything else I should know?", + "why": "anything-else probe" + }, + { + "answer": "If something else comes up, tell me.", + "why": "not asked as a question" + } + ] +} diff --git a/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md b/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md new file mode 100644 index 0000000000..e90ebe312d --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '[^\n]*\bkafka\b[^\n]*\?' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md b/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md new file mode 100644 index 0000000000..aa234889da --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '\bdocs-publish\b[^\n]{0,160}\b(?:instead|consistent|consistency|match\w*|reconsider|sure)\b|\b(?:reconsider|sure)\b[^\n]{0,80}\bship-docs\b' +flags: i +match: not_contains +arm: both +--- diff --git a/plugins/planning/evals/interview-crux-on-rejection/graders/one-crux-and-defers.md b/plugins/planning/evals/interview-crux-on-rejection/graders/one-crux-and-defers.md new file mode 100644 index 0000000000..b5313d7f9c --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/graders/one-crux-and-defers.md @@ -0,0 +1,16 @@ +--- +type: llm +arm: both +--- + +PASS only if all of these hold: + +1. For Q2 the reply asks one question aimed at what would have to be true for Kafka, or for the + Postgres queue, to be the right choice (for example a need Kafka meets that the user knows of, + such as other consumers or replay), or it proposes one concrete check that would settle it. It + does not argue for Postgres more than once, and it accepts Kafka if the user holds to it. +2. For Q3 the reply accepts `ship-docs` as the user's choice without arguing for `docs-publish`. +3. Q1 is accepted. + +FAIL if the reply re-argues the Postgres recommendation at length or more than once, challenges +the `ship-docs` name, or records Kafka with no question or check at all. diff --git a/plugins/planning/evals/interview-crux-on-rejection/prompt.md b/plugins/planning/evals/interview-crux-on-rejection/prompt.md new file mode 100644 index 0000000000..d9dc5ef62e --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/prompt.md @@ -0,0 +1,19 @@ +--- +description: "Eval-first case for the disagreement move. Why hard: the user rejects two recommendations, one technical (Postgres queue versus Kafka) and one personal (the command's name). Re-arguing both, or silently giving up on both, are the easy replies; the target is one crux question on the technical one and plain deference on the name." +tags: [interview, disagreement, crux] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The reply asks one crux question about Q2 (what would have to be true for Kafka, or for the Postgres queue, to be right), or proposes one check that would settle it, and accepts `ship-docs` for Q3 without arguing" +--- + +/planning:interview Add a command-line tool that publishes our docs site. + +You already asked round 1: + +Q1: Where should the build run? My recommendation: in the existing CI workflow, since the repo already builds there. +Q2: How should publish jobs be queued? My recommendation: a Postgres table used as a queue, since Postgres is already deployed and the volume is a few jobs a day. +Q3: What should the command be called? My recommendation: `docs-publish`, matching our other `docs-*` commands. + +My answers: Q1 yes. Q2 no, use Kafka. Q3 no, call it `ship-docs`, I just like it better. diff --git a/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json b/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json new file mode 100644 index 0000000000..ecfc1523b1 --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Q2: What would have to be true for Kafka to be the better queue here, for example other services consuming these publish events?", + "why": "crux question naming Kafka" + }, + { + "answer": "Q2: Is there a consumer besides this tool that needs the Kafka topic?", + "why": "question about the Kafka choice" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Q2 recorded: Kafka.", + "why": "records Kafka with no question" + } + ] +} diff --git a/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json b/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json new file mode 100644 index 0000000000..e56cfedc9e --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Q3: `ship-docs` it is.", + "why": "accepts the name" + }, + { + "answer": "", + "why": "an empty reply re-argues nothing; the other graders fail it" + } + ], + "fail": [ + { + "answer": "Q3: I'd still suggest docs-publish instead, for consistency with the other docs-* commands.", + "why": "re-argues the name" + }, + { + "answer": "Q3: Are you sure about ship-docs? It breaks the docs-* pattern.", + "why": "challenges the personal choice" + } + ] +} diff --git a/plugins/planning/evals/interview-crux-on-rejection/samples/one-crux-and-defers.json b/plugins/planning/evals/interview-crux-on-rejection/samples/one-crux-and-defers.json new file mode 100644 index 0000000000..8b069e5a91 --- /dev/null +++ b/plugins/planning/evals/interview-crux-on-rejection/samples/one-crux-and-defers.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Q1: accepted, the build runs in the existing CI workflow.\nQ2: One question before I record Kafka: what would have to be true for Kafka to be right here, such as other services consuming these publish events or a need to replay them? If there is one, Kafka it is.\nQ3: `ship-docs` it is.", + "why": "one crux question on Q2, defers on Q3" + } + ], + "fail": [ + { + "answer": "Q2: Kafka is overkill for a few jobs a day; Postgres is already deployed and simpler to run. I strongly recommend keeping Postgres. Kafka also adds an operational burden you do not need.\nQ3: I'd still go with docs-publish for consistency.", + "why": "re-argues Q2 and challenges Q3" + }, + { + "answer": "Recorded: Q1 CI, Q2 Kafka, Q3 ship-docs. Round 2 next.", + "why": "no crux question on the technical rejection" + } + ] +} diff --git a/plugins/planning/evals/interview-final-brief-changes-only/graders/changes-only-restatement.md b/plugins/planning/evals/interview-final-brief-changes-only/graders/changes-only-restatement.md new file mode 100644 index 0000000000..824d2ee50f --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/graders/changes-only-restatement.md @@ -0,0 +1,16 @@ +--- +type: llm +arm: both +--- + +PASS only if all of these hold: + +1. The reply restates the intent of the work (CSV export of the admin orders page) in a sentence + or two. +2. It shows the Q5 change: the file name is now orders-export-.csv. +3. It shows both assumptions the sweep added: exports run synchronously in the request, and the + export respects the user's row-level permissions. +4. It does not re-list the answers that were accepted and read back unchanged (Q1, Q2, Q3, Q4, + Q6, Q7, Q8). A count of them, or one line saying they stand as read back, is fine. + +FAIL if any of the four does not hold. diff --git a/plugins/planning/evals/interview-final-brief-changes-only/graders/omits-unchanged-items.md b/plugins/planning/evals/interview-final-brief-changes-only/graders/omits-unchanged-items.md new file mode 100644 index 0000000000..bf0204f2e1 --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/graders/omits-unchanged-items.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '\bgzip\b|\b50,?000\b|\btoolbar\b' +flags: i +match: not_contains +arm: both +--- diff --git a/plugins/planning/evals/interview-final-brief-changes-only/graders/shows-changed-item.md b/plugins/planning/evals/interview-final-brief-changes-only/graders/shows-changed-item.md new file mode 100644 index 0000000000..d08d1b3dd7 --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/graders/shows-changed-item.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: 'orders-export' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-final-brief-changes-only/graders/shows-new-assumptions.md b/plugins/planning/evals/interview-final-brief-changes-only/graders/shows-new-assumptions.md new file mode 100644 index 0000000000..bc79168d6d --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/graders/shows-new-assumptions.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?=[\s\S]*\bsynchronous)(?=[\s\S]*\bpermission)' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-final-brief-changes-only/prompt.md b/plugins/planning/evals/interview-final-brief-changes-only/prompt.md new file mode 100644 index 0000000000..559c522a82 --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/prompt.md @@ -0,0 +1,13 @@ +--- +description: "Eval-first case for the final restatement. Why hard: eight answers are on record and listing all of them looks thorough, but six were already read back and accepted, so the list buries the one changed answer and the two assumptions the user has never seen." +tags: [interview, confirmation, restatement] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The restatement gives the intent, the Q5 file name change, and the sweep's two new assumptions (synchronous export, row-level permissions), and does not re-list the six unchanged accepted answers" +--- + +/planning:interview Add CSV export to the admin orders page. + +Where we are: every question is answered and each answer was read back when you received it. Q1 export the filtered rows (accepted). Q2 columns as shown in the table (accepted). Q3 timestamps in UTC (accepted). Q4 a cap of 50,000 rows (accepted). Q5 file name: you recommended orders-YYYY-MM-DD.csv and read it back; I later changed it to orders-export-.csv. Q6 the button sits in the table toolbar (accepted). Q7 gzip compression off (accepted). Q8 one audit log entry per export (accepted). The assumption sweep then added two assumptions I have not seen: exports run synchronously in the request, and the export respects the user's row-level permissions. Restate the contract for my final confirmation. diff --git a/plugins/planning/evals/interview-final-brief-changes-only/samples/changes-only-restatement.json b/plugins/planning/evals/interview-final-brief-changes-only/samples/changes-only-restatement.json new file mode 100644 index 0000000000..420f45555f --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/samples/changes-only-restatement.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Intent: ops can download the filtered order list from the admin orders page as CSV.\nChanged since read-back: Q5, the file is now orders-export-.csv.\nNew assumptions from the sweep: exports run synchronously in the request; the export respects the user's row-level permissions.\nQ1-Q4 and Q6-Q8 stand as read back.\n\nConfirm, or give me the numbers to change.", + "why": "intent, the change, the two new assumptions, nothing re-listed" + } + ], + "fail": [ + { + "answer": "Contract:\nQ1 filtered rows. Q2 columns as shown. Q3 UTC. Q4 50,000-row cap. Q5 orders-export-.csv. Q6 toolbar button. Q7 gzip off. Q8 audit log entry.\nAssumptions: synchronous export; row-level permissions.", + "why": "re-lists every unchanged answer" + }, + { + "answer": "Intent: CSV export of the admin orders page. Changed: Q5, orders-export-.csv. Confirm?", + "why": "omits the two new assumptions" + } + ] +} diff --git a/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json b/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json new file mode 100644 index 0000000000..7a23e59391 --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Intent: ops can download the filtered order list as CSV.\nChanged: Q5, file name orders-export-.csv.\nNew assumptions: synchronous export; row-level permissions apply.\nThe other six answers stand as read back.\n\nConfirm, or give me the numbers to change.", + "why": "no unchanged item re-listed" + } + ], + "fail": [ + { + "answer": "Q1 filtered rows. Q2 columns as shown. Q3 UTC. Q4 50,000-row cap. Q5 orders-export-.csv. Q6 toolbar button. Q7 gzip off. Q8 audit log entry.", + "why": "re-lists every accepted answer" + }, + { + "answer": "Changed: Q5. Also note the cap stays at 50000 rows.", + "why": "re-shows the unchanged Q4 cap" + } + ] +} diff --git a/plugins/planning/evals/interview-final-brief-changes-only/samples/shows-changed-item.json b/plugins/planning/evals/interview-final-brief-changes-only/samples/shows-changed-item.json new file mode 100644 index 0000000000..fec97b54c2 --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/samples/shows-changed-item.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Changed since read-back: Q5, the file is now orders-export-.csv.", + "why": "shows the changed file name" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "File name: orders-YYYY-MM-DD.csv.", + "why": "shows the superseded recommendation, not the change" + } + ] +} diff --git a/plugins/planning/evals/interview-final-brief-changes-only/samples/shows-new-assumptions.json b/plugins/planning/evals/interview-final-brief-changes-only/samples/shows-new-assumptions.json new file mode 100644 index 0000000000..97cc7b6956 --- /dev/null +++ b/plugins/planning/evals/interview-final-brief-changes-only/samples/shows-new-assumptions.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "New assumptions from the sweep:\n- Exports run synchronously in the request.\n- The export respects the user's row-level permissions.", + "why": "both assumptions" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "New assumption: exports run synchronously in the request.", + "why": "missing the permissions assumption" + }, + { + "answer": "New assumption: the export respects row-level permissions.", + "why": "missing the synchronous assumption" + } + ] +} diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/graders/active-reply-requested.md b/plugins/planning/evals/interview-final-confirm-active-reply/graders/active-reply-requested.md new file mode 100644 index 0000000000..0ca3c6d689 --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/graders/active-reply-requested.md @@ -0,0 +1,12 @@ +--- +type: llm +arm: both +--- + +Judge only the request that closes the restatement. + +PASS only if the reply asks the user to give an active answer: to confirm the contract, or to name +the numbers (or ids) of the items they want changed. + +FAIL if the closing request is a yes/no check such as "Sound good?", "Does this look right?", +"Any thoughts?" or "OK?", or if the reply asks for no confirmation at all. diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md b/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md new file mode 100644 index 0000000000..9f08a7bc9b --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '\bconfirm\b[^\n]{0,200}\b(?:numbers?|Q\d+|which (?:items?|questions?|rows?))\b|\b(?:numbers?|Q\d+)\b[^\n]{0,200}\b(?:to )?change\b' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/graders/no-vague-confirm.md b/plugins/planning/evals/interview-final-confirm-active-reply/graders/no-vague-confirm.md new file mode 100644 index 0000000000..0d097c8313 --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/graders/no-vague-confirm.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '\b(?:sounds?|looks?|seems?) (?:good|right|ok(?:ay)?|fine)\?|\b(?:does|do) (?:this|that|everything|it) (?:look|sound|seem) (?:good|right|ok(?:ay)?|fine)\?|\b(?:any|other) (?:thoughts|concerns|feedback)\?|\b(?:ok(?:ay)?|good to go|all good)\?' +flags: i +match: not_contains +arm: both +--- diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md b/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md new file mode 100644 index 0000000000..f4f539ee95 --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md @@ -0,0 +1,13 @@ +--- +description: "Eval-first case for the final confirmation wording. Why hard: every answer is in and the sweep is clean, so closing with 'Sound good?' reads as polite and natural, but a bare 'yes' to it confirms nothing specific." +tags: [interview, confirmation, wording] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The restatement ends by asking the user to confirm, or to name the numbers of the items to change, and never asks 'sound good?' or a similar yes/no" +--- + +/planning:interview Add CSV export to the admin orders page. + +Where we are: rounds 1 and 2 are answered. Q1 export the filtered rows. Q2 columns as shown in the table. Q3 timestamps in UTC. Q4 a cap of 50,000 rows. Q5 file name orders-YYYY-MM-DD.csv. The frontier is empty and the assumption sweep added no open row. Restate the contract and ask me to confirm it. diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/samples/active-reply-requested.json b/plugins/planning/evals/interview-final-confirm-active-reply/samples/active-reply-requested.json new file mode 100644 index 0000000000..2f34fd7843 --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/samples/active-reply-requested.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Contract: export the filtered orders as CSV, columns as shown, UTC timestamps, 50,000-row cap, file orders-YYYY-MM-DD.csv.\n\nConfirm, or give me the numbers of the items to change.", + "why": "active confirm-or-numbers request" + } + ], + "fail": [ + { + "answer": "Contract: export the filtered orders as CSV with UTC timestamps and a 50,000-row cap. Sound good?", + "why": "yes/no check" + }, + { + "answer": "Contract: export the filtered orders as CSV with UTC timestamps. Writing the Brief now.", + "why": "asks for no confirmation" + } + ] +} diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json b/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json new file mode 100644 index 0000000000..661d68851e --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "Confirm, or give me the numbers of the items to change.", + "why": "confirm or numbers" + }, + { + "answer": "Reply \"confirm\" to lock it, or list the Q numbers you want to change.", + "why": "Q numbers to change" + }, + { + "answer": "Please confirm the contract, or tell me which items to revise.", + "why": "confirm or which items" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Contract restated above. Sound good?", + "why": "no confirm-or-numbers request" + }, + { + "answer": "Shall I lock this?", + "why": "a yes/no question naming no items" + } + ] +} diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/samples/no-vague-confirm.json b/plugins/planning/evals/interview-final-confirm-active-reply/samples/no-vague-confirm.json new file mode 100644 index 0000000000..c424a46ed0 --- /dev/null +++ b/plugins/planning/evals/interview-final-confirm-active-reply/samples/no-vague-confirm.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Contract: export the filtered orders as CSV, columns as shown, UTC timestamps, 50,000-row cap, file orders-YYYY-MM-DD.csv.\n\nConfirm, or give me the numbers of the items to change.", + "why": "active request, no yes/no check" + }, + { + "answer": "", + "why": "an empty reply holds no vague check; the other graders fail it" + } + ], + "fail": [ + { + "answer": "Contract: export the filtered orders as CSV with UTC timestamps. Sound good?", + "why": "sound good" + }, + { + "answer": "Here is the restated contract. Does this look right?", + "why": "does this look right" + }, + { + "answer": "That's the contract. Any thoughts?", + "why": "any thoughts" + } + ] +} diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/graders/names-questionnaire.md b/plugins/planning/evals/interview-person-deferred-questionnaire/graders/names-questionnaire.md new file mode 100644 index 0000000000..75f0b90e8c --- /dev/null +++ b/plugins/planning/evals/interview-person-deferred-questionnaire/graders/names-questionnaire.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '/?planning:questionnaire\b' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/graders/questionnaire-beside-next-step.md b/plugins/planning/evals/interview-person-deferred-questionnaire/graders/questionnaire-beside-next-step.md new file mode 100644 index 0000000000..9b0e723238 --- /dev/null +++ b/plugins/planning/evals/interview-person-deferred-questionnaire/graders/questionnaire-beside-next-step.md @@ -0,0 +1,15 @@ +--- +type: llm +arm: both +--- + +PASS only if all of these hold: + +1. The reply recommends turning the deferred SSH key type question into a questionnaire or + document for the partner's ops lead to answer, naming `/planning:questionnaire`. +2. The reply also recommends a next step for the work itself (planning, design, exploration, + research, or implementation), so the questionnaire runs beside the pipeline, not in place of it. +3. The reply does not choose an SSH key type or record one as an assumption. + +FAIL if the reply leaves the deferred question with no one asked to answer it, recommends only +the questionnaire with no next step for the work, or picks a key type. diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md b/plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md new file mode 100644 index 0000000000..e2ff056cf9 --- /dev/null +++ b/plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Contract case for the questionnaire hand-off. Why hard: the Brief is locked and the natural handoff is /planning:plan alone, so the one question only the partner's ops lead can answer stays parked in the Brief with nobody asked to answer it." +tags: [interview, handoff, questionnaire] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The handoff recommends /planning:questionnaire for the deferred SSH key question the partner's ops lead holds, alongside a next pipeline step, and does not pick a key type" +--- + +/planning:interview We just finished the interview for moving our nightly invoice export from FTP to a partner's new SFTP endpoint, and the Brief is locked. One question stayed deferred: which SSH key type the partner's endpoint accepts. I can't answer it; only the partner's ops lead knows. Everything else is settled and we know this codebase well. Hand off: what should I run next? diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/samples/names-questionnaire.json b/plugins/planning/evals/interview-person-deferred-questionnaire/samples/names-questionnaire.json new file mode 100644 index 0000000000..21ded3e1fe --- /dev/null +++ b/plugins/planning/evals/interview-person-deferred-questionnaire/samples/names-questionnaire.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Next: /planning:plan for the export change. In parallel, run /planning:questionnaire to send the SSH key question to the partner's ops lead.", + "why": "names the questionnaire skill with its slash" + }, + { + "answer": "Run planning:questionnaire for the deferred key-type question.", + "why": "names the skill without the slash" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Next: /planning:plan. The SSH key type stays in Deferred questions until the partner replies.", + "why": "no questionnaire hand-off" + }, + { + "answer": "Email the partner's ops lead and ask which key type they accept, then run /planning:plan.", + "why": "advises asking but never names the skill" + } + ] +} diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/samples/questionnaire-beside-next-step.json b/plugins/planning/evals/interview-person-deferred-questionnaire/samples/questionnaire-beside-next-step.json new file mode 100644 index 0000000000..f7ca47ce9e --- /dev/null +++ b/plugins/planning/evals/interview-person-deferred-questionnaire/samples/questionnaire-beside-next-step.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Next: /planning:plan for the export change. Alongside it, /planning:questionnaire turns the deferred SSH key type question into a document the partner's ops lead fills in.", + "why": "questionnaire beside the next pipeline step, no key type chosen" + } + ], + "fail": [ + { + "answer": "Run /planning:questionnaire for the partner's ops lead and wait for the answer before doing anything else.", + "why": "questionnaire replaces the next step" + }, + { + "answer": "Next: /planning:plan. Assume ed25519 keys; most SFTP servers accept them.", + "why": "no questionnaire, and picks a key type" + } + ] +} diff --git a/plugins/planning/evals/interview-readback-dictated-answer/graders/names-feature-flag.md b/plugins/planning/evals/interview-readback-dictated-answer/graders/names-feature-flag.md new file mode 100644 index 0000000000..c91c0632a3 --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/graders/names-feature-flag.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '\bfeature[- ]flag' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-readback-dictated-answer/graders/not-verbatim-echo.md b/plugins/planning/evals/interview-readback-dictated-answer/graders/not-verbatim-echo.md new file mode 100644 index 0000000000..a94b8c2992 --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/graders/not-verbatim-echo.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: 'behind the fee chair flag but turn it on for staff' +flags: i +match: not_contains +arm: both +--- diff --git a/plugins/planning/evals/interview-readback-dictated-answer/graders/reads-back-and-confirms.md b/plugins/planning/evals/interview-readback-dictated-answer/graders/reads-back-and-confirms.md new file mode 100644 index 0000000000..5ed4306b79 --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/graders/reads-back-and-confirms.md @@ -0,0 +1,17 @@ +--- +type: llm +arm: both +--- + +PASS only if all of these hold: + +1. The reply restates the Q2 answer in its own words (not a quotation of the user's reply) and + says it understood "fee chair flag" as a feature flag, with the flag on for staff first. +2. The reply restates the Q3 answer (local storage instead of the user profile) in its own words. +3. The reply asks the user to confirm Q2 and Q3, or to correct them, before treating them as + settled. + +Q1 may be accepted without a confirmation question. + +FAIL if the reply records "fee chair" as the answer without saying how it interpreted it, quotes +the user's words back instead of restating them, or treats Q2 and Q3 as settled without asking. diff --git a/plugins/planning/evals/interview-readback-dictated-answer/prompt.md b/plugins/planning/evals/interview-readback-dictated-answer/prompt.md new file mode 100644 index 0000000000..4d9d7d877e --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/prompt.md @@ -0,0 +1,19 @@ +--- +description: "Eval-first case for the per-answer read-back. Why hard: the dictated reply reads 'fee chair flag' for 'feature flag' and departs from two recommendations, so recording the answers as typed, or echoing them verbatim, both look like progress." +tags: [interview, read-back, dictation] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The reply restates Q2 and Q3 in its own words, says it took 'fee chair flag' to mean a feature flag, and asks the user to confirm Q2 and Q3 before treating them as settled; Q1 needs no confirmation question" +--- + +/planning:interview Add a dark mode toggle to the settings page. + +You already asked round 1: + +Q1: Should dark mode follow the operating system setting until the user picks one? My recommendation: yes. +Q2: How should the toggle roll out? My recommendation: behind a flag, off by default, then on for everyone. +Q3: Where should the user's choice be stored? My recommendation: on the user profile, so it follows them across devices. + +My answers, dictated: Q1 yes. Q2 put it behind the fee chair flag but turn it on for staff first. Q3 local storage is fine for now. diff --git a/plugins/planning/evals/interview-readback-dictated-answer/samples/names-feature-flag.json b/plugins/planning/evals/interview-readback-dictated-answer/samples/names-feature-flag.json new file mode 100644 index 0000000000..807b00679f --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/samples/names-feature-flag.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Q2: I read \"fee chair flag\" as a feature flag, turned on for staff before everyone else. Is that right?", + "why": "names the interpreted feature flag" + }, + { + "answer": "Q2 understood as: a feature-flag rollout, staff first.", + "why": "hyphenated form" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Q2 recorded: behind the fee chair flag, staff first.", + "why": "keeps the dictation error and never names the feature flag" + } + ] +} diff --git a/plugins/planning/evals/interview-readback-dictated-answer/samples/not-verbatim-echo.json b/plugins/planning/evals/interview-readback-dictated-answer/samples/not-verbatim-echo.json new file mode 100644 index 0000000000..7276e105e8 --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/samples/not-verbatim-echo.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Q2: I read \"fee chair flag\" as a feature flag, turned on for staff before everyone else. Is that right?", + "why": "restated in the agent's words" + }, + { + "answer": "", + "why": "an empty reply holds no echo; the other graders fail it" + } + ], + "fail": [ + { + "answer": "Got it. Q2: put it behind the fee chair flag but turn it on for staff first.", + "why": "verbatim echo of the dictated answer" + }, + { + "answer": "You said: \"Put it behind the fee chair flag but turn it on for staff first.\" Confirm?", + "why": "a quotation is not a read-back in the agent's words" + } + ] +} diff --git a/plugins/planning/evals/interview-readback-dictated-answer/samples/reads-back-and-confirms.json b/plugins/planning/evals/interview-readback-dictated-answer/samples/reads-back-and-confirms.json new file mode 100644 index 0000000000..4c5fc74ad7 --- /dev/null +++ b/plugins/planning/evals/interview-readback-dictated-answer/samples/reads-back-and-confirms.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Q1: following the OS setting, accepted.\nQ2: I took \"fee chair flag\" to mean a feature flag: off by default, on for staff first, then everyone. Right?\nQ3: you want the choice kept in the browser's local storage for now rather than on the profile, so it will not follow users across devices. Confirm, or tell me what to change.", + "why": "own-words read-back of Q2 and Q3 with an explicit confirm request" + } + ], + "fail": [ + { + "answer": "Recorded: Q1 yes, Q2 fee chair flag with staff first, Q3 local storage. Round 2 next.", + "why": "records the dictation error and asks nothing" + }, + { + "answer": "Q2: feature flag, staff first. Q3: local storage. Moving on to round 2:\nQ4: Which pages get dark mode first?", + "why": "restates but treats Q2 and Q3 as settled without asking" + } + ] +} diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md new file mode 100644 index 0000000000..f924384cbb --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md @@ -0,0 +1,18 @@ +--- +type: llm +arm: both +--- + +Look at every question in the reply that carries a recommended answer. + +PASS only if every such recommendation has both: + +1. A checkable basis: a file path with a line, a URL, or a pointer to a fact the user stated in + the prompt (for example "you said the partner terms promise 1,000 requests per minute per key" + or "Redis 7 is already deployed, per your message"). A bare "judgment" or "best practice" with + nothing the user could check does not count. +2. The condition that would change it: what the user could tell the agent, or what a check could + find, that would make a different answer right. + +FAIL if any recommendation lacks a checkable basis or a condition that would change it, or if the +reply asks no question with a recommendation. diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md new file mode 100644 index 0000000000..b505200490 --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '\b(?:would|could|should) (?:flip|change|switch|reverse)\b|\bflips?\b|\bchanges? (?:if|when)\b|\bunless\b|\bif instead\b' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md new file mode 100644 index 0000000000..44697e0907 --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Eval-first case for checkable recommendations. Why hard: the prompt states enough facts that every recommendation can rest on something the user said, but 'Basis: judgment' with no reference and no condition that would change it is the easy default." +tags: [interview, recommendation, basis] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "Every recommendation in round 1 names a checkable basis (a file and line, a URL, or a fact the user stated) and the condition that would change it" +--- + +/planning:interview We need rate limiting on our public REST API. Facts: we run on Postgres 16 with a single Redis 7 instance already used for sessions; traffic peaks at about 2,000 requests per second; our published partner terms promise 1,000 requests per minute per API key. Ask me your first round. diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/checkable-basis-and-flip.json b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/checkable-basis-and-flip.json new file mode 100644 index 0000000000..be696d22e1 --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/checkable-basis-and-flip.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Q1: Where should the counters live?\nMy recommendation: the existing Redis 7 instance.\nBasis: you said Redis 7 is already deployed for sessions.\nWould change if: that instance cannot absorb about 2,000 extra writes a second at peak.\n\nQ2: What is the limit unit?\nMy recommendation: per API key, 1,000 requests per minute.\nBasis: your published partner terms, as you quoted them.\nWould change if: the terms also bind anonymous callers.", + "why": "each recommendation has a checkable basis and a flip condition" + } + ], + "fail": [ + { + "answer": "Q1: Where should the counters live?\nMy recommendation: Redis.\nBasis: judgment\n\nQ2: What is the limit unit?\nMy recommendation: per API key.\nBasis: judgment", + "why": "no checkable basis, no flip condition" + }, + { + "answer": "Q1: Where should the counters live?\nMy recommendation: Redis, since you said it is deployed.\n\nQ2: Which algorithm?\nMy recommendation: token bucket. Would change if you need strict per-minute windows.", + "why": "Q1 has no flip condition and Q2 has no checkable basis" + } + ] +} diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json new file mode 100644 index 0000000000..ee4fcfdbf6 --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Q1: Where should the counters live?\nMy recommendation: Redis 7, already deployed for sessions.\nBasis: your message (Redis 7 in place).\nWould change if: the session Redis cannot take about 2,000 extra writes a second.", + "why": "would change if" + }, + { + "answer": "Q2: Per key or per IP?\nMy recommendation: per API key, as your partner terms state.\nFlips to per IP if anonymous traffic matters more than partners.", + "why": "flips" + }, + { + "answer": "Recommendation: a sliding window, unless you need burst allowance.", + "why": "unless" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Q1: Where should the counters live?\nMy recommendation: Redis.\nBasis: judgment", + "why": "no condition that would change it" + } + ] +} diff --git a/plugins/planning/evals/interview-scope-cut-moscow/graders/applies-must-test.md b/plugins/planning/evals/interview-scope-cut-moscow/graders/applies-must-test.md new file mode 100644 index 0000000000..97e2fbaab4 --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/graders/applies-must-test.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?=[\s\S]*\bmust\b)(?=[\s\S]*\b(?:cancel\w*|work(?:ed|ing)? ?around)\b)' +flags: i +arm: both +--- diff --git a/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-cut-before-wayfind.md b/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-cut-before-wayfind.md new file mode 100644 index 0000000000..0079404519 --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-cut-before-wayfind.md @@ -0,0 +1,18 @@ +--- +type: llm +arm: both +--- + +PASS only if all of these hold: + +1. The reply sorts the eight listed capabilities into priority buckets (Must, Should, Could, + Won't this time, or equivalent names), with a recommended bucket for each item. +2. The Must bucket is decided by asking, or applying, whether the effort would be cancelled or + could work around the item if it were not delivered. +3. The Must bucket is kept well short of the whole effort (about 60% of it or less), not all eight + items. +4. Items outside Must are routed to later work or to `/planning:wayfind`. If the reply mentions + wayfind, the scope cut comes before or together with that recommendation, not instead of it. + +FAIL if the reply recommends `/planning:wayfind` with no per-item bucket, treats all eight items as +must-haves, or asks no scope question at all. diff --git a/plugins/planning/evals/interview-scope-cut-moscow/prompt.md b/plugins/planning/evals/interview-scope-cut-moscow/prompt.md new file mode 100644 index 0000000000..96302ed22a --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Eval-first case for the scope-cut round. Why hard: eight capabilities for two engineers in one quarter is plainly too big, so recommending /planning:wayfind at once looks right, but nobody has yet asked which items the effort fails without." +tags: [interview, scope, wayfind] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "Before recommending /planning:wayfind, the reply runs a scope cut: for each capability it asks or applies the Must test (if this is not delivered, is the effort cancelled or worked around?), recommends a bucket per item, keeps Must to about 60% of the effort or less, and routes the rest to /planning:wayfind or later" +--- + +/planning:interview Build our internal developer portal this quarter with two engineers: a service catalog, team scorecards, onboarding guides, cloud cost dashboards, on-call schedules, an incident timeline, an API explorer, and a plugin system so other teams can extend it. diff --git a/plugins/planning/evals/interview-scope-cut-moscow/samples/applies-must-test.json b/plugins/planning/evals/interview-scope-cut-moscow/samples/applies-must-test.json new file mode 100644 index 0000000000..3e9793616c --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/samples/applies-must-test.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "For each item: if it is not delivered this quarter, is the portal cancelled, or do you work around it? Only the cancel answers are Must.", + "why": "Must with the cancel or work-around test" + }, + { + "answer": "Must test (would you cancel the portal without it?): service catalog yes.", + "why": "Must and cancel" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "This is too big and still foggy. I recommend /planning:wayfind to chart it.", + "why": "no Must test" + }, + { + "answer": "Which of these must ship this quarter?", + "why": "must without the cancel or work-around test" + } + ] +} diff --git a/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-cut-before-wayfind.json b/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-cut-before-wayfind.json new file mode 100644 index 0000000000..8d4f32f739 --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-cut-before-wayfind.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Q1: Scope cut. For each item, if it is not delivered this quarter, is the portal cancelled or worked around? My recommended buckets:\n- Must: service catalog, on-call schedules, onboarding guides\n- Should: team scorecards, incident timeline\n- Could: API explorer, cloud cost dashboards\n- Won't this quarter: plugin system\nMust is about half the effort. Everything outside Must goes to /planning:wayfind to chart once this cut is agreed.", + "why": "per-item buckets from the Must test, Must under 60%, the rest routed to wayfind" + } + ], + "fail": [ + { + "answer": "This is too big and still foggy for one interview. I recommend /planning:wayfind to chart it as a decision map.", + "why": "wayfind with no scope cut" + }, + { + "answer": "All eight are must-haves for the portal to be useful. Q1: Which do you want first?", + "why": "every item treated as Must" + } + ] +} diff --git a/plugins/planning/skills/interview/SKILL.md b/plugins/planning/skills/interview/SKILL.md index 0448ef340e..f8057bfe43 100644 --- a/plugins/planning/skills/interview/SKILL.md +++ b/plugins/planning/skills/interview/SKILL.md @@ -429,7 +429,7 @@ Section schema: write the literal `## Brief` template (TLDR / Goal / Constraints ### Step 5. Hand off -Route the handoff by what the session produced. **A general (non-engineering) session is terminal**. It produced a shared-understanding summary, not a Brief; deliver that summary and stop, offering no pipeline handoff (nothing downstream consumes it). **An engineering session** wrote a PLAN.md Brief. Recommend the next step per task shape: +Route the handoff by what the session produced. **A general (non-engineering) session is terminal**. It produced a shared-understanding summary, not a Brief; deliver that summary and stop, offering no pipeline handoff (nothing downstream consumes it). **An engineering session** wrote a PLAN.md Brief. When the Brief defers a question only a named person other than the user can answer (an assumption-sweep `person` item the user could not answer), also recommend `/planning:questionnaire` (if enabled), which turns it into a document that person fills in; it runs beside the next step, not in place of it. Recommend the next step per task shape: - **Code change with unknowns about the codebase** → clear context, then codebase exploration (`/discovery:explore` if enabled. It reads the Brief as scope) - **Code change relying on external libs/APIs/best-practices** → external research (`/discovery:research` if enabled) From 51129ac98dd9874c45a44ef5905f458c3a9166d8 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 19:47:24 -0400 Subject: [PATCH 2/4] fix(planning): tighten interview eval graders and fix unchanged-answer count Graders no longer fail correct replies (plain "Sure, ship-docs" deference) or pass incorrect ones (a stray question mark, an unrelated "unless", a restatement line naming a change). The changes-only case counts seven unchanged answers, matching its prompt. Each tightened grader gains the samples that showed the gap. Co-Authored-By: Claude Opus 5.5 --- .../graders/asks-about-kafka.md | 2 +- .../graders/no-reargue-name.md | 2 +- .../samples/asks-about-kafka.json | 8 ++++++++ .../samples/no-reargue-name.json | 8 ++++++++ .../evals/interview-final-brief-changes-only/prompt.md | 4 ++-- .../samples/omits-unchanged-items.json | 2 +- .../graders/asks-confirm-or-numbers.md | 2 +- .../samples/asks-confirm-or-numbers.json | 4 ++++ .../graders/states-flip-condition.md | 2 +- .../samples/states-flip-condition.json | 4 ++++ 10 files changed, 31 insertions(+), 7 deletions(-) diff --git a/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md b/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md index e90ebe312d..872e9bc40c 100644 --- a/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md +++ b/plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '[^\n]*\bkafka\b[^\n]*\?' +pattern: '\bkafka\b[^\n?.;]*\?' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md b/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md index aa234889da..50e4163b71 100644 --- a/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md +++ b/plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '\bdocs-publish\b[^\n]{0,160}\b(?:instead|consistent|consistency|match\w*|reconsider|sure)\b|\b(?:reconsider|sure)\b[^\n]{0,80}\bship-docs\b' +pattern: '\bdocs-publish\b[^\n]{0,160}\b(?:instead|consistent|consistency|reconsider)\b|\b(?:reconsider|are you sure)\b[^\n]{0,80}\bship-docs\b' flags: i match: not_contains arm: both diff --git a/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json b/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json index ecfc1523b1..32da8dbc95 100644 --- a/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json +++ b/plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json @@ -17,6 +17,14 @@ { "answer": "Q2 recorded: Kafka.", "why": "records Kafka with no question" + }, + { + "answer": "Recorded: Q1 CI, Q2 Kafka, Q3 ship-docs. Ready for round 2?", + "why": "the question mark belongs to another sentence" + }, + { + "answer": "Q2: Kafka is overkill... Are you sure?", + "why": "re-argues instead of asking the crux" } ] } diff --git a/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json b/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json index e56cfedc9e..ad82c381bc 100644 --- a/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json +++ b/plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json @@ -4,6 +4,14 @@ "answer": "Q3: `ship-docs` it is.", "why": "accepts the name" }, + { + "answer": "Q3: Sure, `ship-docs` it is.", + "why": "plain deference that says sure" + }, + { + "answer": "Q3: renamed docs-publish to ship-docs, matching your preference.", + "why": "accepts the rename" + }, { "answer": "", "why": "an empty reply re-argues nothing; the other graders fail it" diff --git a/plugins/planning/evals/interview-final-brief-changes-only/prompt.md b/plugins/planning/evals/interview-final-brief-changes-only/prompt.md index 559c522a82..d492a45dbe 100644 --- a/plugins/planning/evals/interview-final-brief-changes-only/prompt.md +++ b/plugins/planning/evals/interview-final-brief-changes-only/prompt.md @@ -1,11 +1,11 @@ --- -description: "Eval-first case for the final restatement. Why hard: eight answers are on record and listing all of them looks thorough, but six were already read back and accepted, so the list buries the one changed answer and the two assumptions the user has never seen." +description: "Eval-first case for the final restatement. Why hard: eight answers are on record and listing all of them looks thorough, but seven were already read back and accepted, so the list buries the one changed answer and the two assumptions the user has never seen." tags: [interview, confirmation, restatement] runs: 3 max_turns: 12 timeout_seconds: 600 allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The restatement gives the intent, the Q5 file name change, and the sweep's two new assumptions (synchronous export, row-level permissions), and does not re-list the six unchanged accepted answers" +expected_outcome: "The restatement gives the intent, the Q5 file name change, and the sweep's two new assumptions (synchronous export, row-level permissions), and does not re-list the seven unchanged accepted answers" --- /planning:interview Add CSV export to the admin orders page. diff --git a/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json b/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json index 7a23e59391..6c5645e2ca 100644 --- a/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json +++ b/plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json @@ -1,7 +1,7 @@ { "pass": [ { - "answer": "Intent: ops can download the filtered order list as CSV.\nChanged: Q5, file name orders-export-.csv.\nNew assumptions: synchronous export; row-level permissions apply.\nThe other six answers stand as read back.\n\nConfirm, or give me the numbers to change.", + "answer": "Intent: ops can download the filtered order list as CSV.\nChanged: Q5, file name orders-export-.csv.\nNew assumptions: synchronous export; row-level permissions apply.\nThe other seven answers stand as read back.\n\nConfirm, or give me the numbers to change.", "why": "no unchanged item re-listed" } ], diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md b/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md index 9f08a7bc9b..41940435ac 100644 --- a/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md +++ b/plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '\bconfirm\b[^\n]{0,200}\b(?:numbers?|Q\d+|which (?:items?|questions?|rows?))\b|\b(?:numbers?|Q\d+)\b[^\n]{0,200}\b(?:to )?change\b' +pattern: '\bconfirm\b[^\n]{0,200}\b(?:numbers?|Q\d+|which (?:items?|questions?|rows?))\b|\b(?:give|tell|list|name|send)\b[^\n]{0,60}\b(?:numbers?|Q\d+)\b[^\n]{0,80}\bchange\b' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json b/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json index 661d68851e..71cbe607f8 100644 --- a/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json +++ b/plugins/planning/evals/interview-final-confirm-active-reply/samples/asks-confirm-or-numbers.json @@ -25,6 +25,10 @@ { "answer": "Shall I lock this?", "why": "a yes/no question naming no items" + }, + { + "answer": "Q5 file name orders-export-.csv (no change since round 2).\nIs that correct?", + "why": "a restatement line, then a yes/no close" } ] } diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md index b505200490..fa846a2b5e 100644 --- a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/states-flip-condition.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '\b(?:would|could|should) (?:flip|change|switch|reverse)\b|\bflips?\b|\bchanges? (?:if|when)\b|\bunless\b|\bif instead\b' +pattern: '\b(?:would|could|should) (?:flip|change|switch|reverse)\b|\bflips?\b|\bchanges? (?:if|when)\b|\brecommend\w*\b[^\n]*\bunless\b|\bif instead\b' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json index ee4fcfdbf6..3acf378e05 100644 --- a/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/states-flip-condition.json @@ -21,6 +21,10 @@ { "answer": "Q1: Where should the counters live?\nMy recommendation: Redis.\nBasis: judgment", "why": "no condition that would change it" + }, + { + "answer": "Q1: Where should the counters live?\nMy recommendation: Redis.\nBasis: judgment\nI won't proceed unless you answer.", + "why": "an unless that is not a flip condition" } ] } From cd8044e7894c6c70095f7d7fa6ace08008a29397 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 19:55:45 -0400 Subject: [PATCH 3/4] fix(planning): route owner-held tenant items to the questionnaire too The assumption sweep classes a target-environment setting owned by someone else as `tenant`, not `person`, and defers it when the user is not the owner. Step 5 now recommends /planning:questionnaire for that case as well, which is the scenario the questionnaire eval case exercises. Co-Authored-By: Claude Opus 5.5 --- .changes/planning/feat-interview-techniques-323edf76.md | 2 +- plugins/planning/skills/interview/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.changes/planning/feat-interview-techniques-323edf76.md b/.changes/planning/feat-interview-techniques-323edf76.md index 4627d23a99..6d3f054551 100644 --- a/.changes/planning/feat-interview-techniques-323edf76.md +++ b/.changes/planning/feat-interview-techniques-323edf76.md @@ -4,5 +4,5 @@ bump: minor ### Added -- `/planning:interview` recommends `/planning:questionnaire` at handoff for a deferred question only a named person other than the user can answer (an assumption-sweep `person` item), beside the usual next step. The new `interview-person-deferred-questionnaire` eval case grades it. +- `/planning:interview` recommends `/planning:questionnaire` at handoff for a deferred question only a named person other than the user can answer (an assumption-sweep `person` item, or a `tenant` item whose owner is not the user), beside the usual next step. The new `interview-person-deferred-questionnaire` eval case grades it. - Seven eval cases for interview behaviors not yet in the skill (per-answer read-back of a dictated answer, an active final confirmation, a changes-only final restatement, a checkable basis and flip condition per recommendation, the "something else" closing probe, a MoSCoW scope cut before `/planning:wayfind`, and one crux question on a rejected recommendation). Each rule lands only if its case fails without it. diff --git a/plugins/planning/skills/interview/SKILL.md b/plugins/planning/skills/interview/SKILL.md index f8057bfe43..c02ed2fece 100644 --- a/plugins/planning/skills/interview/SKILL.md +++ b/plugins/planning/skills/interview/SKILL.md @@ -429,7 +429,7 @@ Section schema: write the literal `## Brief` template (TLDR / Goal / Constraints ### Step 5. Hand off -Route the handoff by what the session produced. **A general (non-engineering) session is terminal**. It produced a shared-understanding summary, not a Brief; deliver that summary and stop, offering no pipeline handoff (nothing downstream consumes it). **An engineering session** wrote a PLAN.md Brief. When the Brief defers a question only a named person other than the user can answer (an assumption-sweep `person` item the user could not answer), also recommend `/planning:questionnaire` (if enabled), which turns it into a document that person fills in; it runs beside the next step, not in place of it. Recommend the next step per task shape: +Route the handoff by what the session produced. **A general (non-engineering) session is terminal**. It produced a shared-understanding summary, not a Brief; deliver that summary and stop, offering no pipeline handoff (nothing downstream consumes it). **An engineering session** wrote a PLAN.md Brief. When the Brief defers a question only a named person other than the user can answer (an assumption-sweep `person` item the user could not answer, or a `tenant` item whose owner is not the user), also recommend `/planning:questionnaire` (if enabled), which turns it into a document that person fills in; it runs beside the next step, not in place of it. Recommend the next step per task shape: - **Code change with unknowns about the codebase** → clear context, then codebase exploration (`/discovery:explore` if enabled. It reads the Brief as scope) - **Code change relying on external libs/APIs/best-practices** → external research (`/discovery:research` if enabled) From 994394294542676047a03f5a2ba3acc10086a141 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 20:05:13 -0400 Subject: [PATCH 4/4] fix(planning): name the questionnaire eval case for the tenant item it tests The SSH key type a partner endpoint accepts is a `tenant` item in the assumption-sweep taxonomy, not a `person` item. Rename the case to interview-owner-deferred-questionnaire and state the class in its description. Co-Authored-By: Claude Opus 5.5 --- .changes/planning/feat-interview-techniques-323edf76.md | 2 +- .../graders/names-questionnaire.md | 0 .../graders/questionnaire-beside-next-step.md | 0 .../prompt.md | 2 +- .../samples/names-questionnaire.json | 0 .../samples/questionnaire-beside-next-step.json | 0 6 files changed, 2 insertions(+), 2 deletions(-) rename plugins/planning/evals/{interview-person-deferred-questionnaire => interview-owner-deferred-questionnaire}/graders/names-questionnaire.md (100%) rename plugins/planning/evals/{interview-person-deferred-questionnaire => interview-owner-deferred-questionnaire}/graders/questionnaire-beside-next-step.md (100%) rename plugins/planning/evals/{interview-person-deferred-questionnaire => interview-owner-deferred-questionnaire}/prompt.md (66%) rename plugins/planning/evals/{interview-person-deferred-questionnaire => interview-owner-deferred-questionnaire}/samples/names-questionnaire.json (100%) rename plugins/planning/evals/{interview-person-deferred-questionnaire => interview-owner-deferred-questionnaire}/samples/questionnaire-beside-next-step.json (100%) diff --git a/.changes/planning/feat-interview-techniques-323edf76.md b/.changes/planning/feat-interview-techniques-323edf76.md index 6d3f054551..15232e0257 100644 --- a/.changes/planning/feat-interview-techniques-323edf76.md +++ b/.changes/planning/feat-interview-techniques-323edf76.md @@ -4,5 +4,5 @@ bump: minor ### Added -- `/planning:interview` recommends `/planning:questionnaire` at handoff for a deferred question only a named person other than the user can answer (an assumption-sweep `person` item, or a `tenant` item whose owner is not the user), beside the usual next step. The new `interview-person-deferred-questionnaire` eval case grades it. +- `/planning:interview` recommends `/planning:questionnaire` at handoff for a deferred question only a named person other than the user can answer (an assumption-sweep `person` item, or a `tenant` item whose owner is not the user), beside the usual next step. The new `interview-owner-deferred-questionnaire` eval case grades it on a `tenant` item. - Seven eval cases for interview behaviors not yet in the skill (per-answer read-back of a dictated answer, an active final confirmation, a changes-only final restatement, a checkable basis and flip condition per recommendation, the "something else" closing probe, a MoSCoW scope cut before `/planning:wayfind`, and one crux question on a rejected recommendation). Each rule lands only if its case fails without it. diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/graders/names-questionnaire.md b/plugins/planning/evals/interview-owner-deferred-questionnaire/graders/names-questionnaire.md similarity index 100% rename from plugins/planning/evals/interview-person-deferred-questionnaire/graders/names-questionnaire.md rename to plugins/planning/evals/interview-owner-deferred-questionnaire/graders/names-questionnaire.md diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/graders/questionnaire-beside-next-step.md b/plugins/planning/evals/interview-owner-deferred-questionnaire/graders/questionnaire-beside-next-step.md similarity index 100% rename from plugins/planning/evals/interview-person-deferred-questionnaire/graders/questionnaire-beside-next-step.md rename to plugins/planning/evals/interview-owner-deferred-questionnaire/graders/questionnaire-beside-next-step.md diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md b/plugins/planning/evals/interview-owner-deferred-questionnaire/prompt.md similarity index 66% rename from plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md rename to plugins/planning/evals/interview-owner-deferred-questionnaire/prompt.md index e2ff056cf9..60c3b847e1 100644 --- a/plugins/planning/evals/interview-person-deferred-questionnaire/prompt.md +++ b/plugins/planning/evals/interview-owner-deferred-questionnaire/prompt.md @@ -1,5 +1,5 @@ --- -description: "Contract case for the questionnaire hand-off. Why hard: the Brief is locked and the natural handoff is /planning:plan alone, so the one question only the partner's ops lead can answer stays parked in the Brief with nobody asked to answer it." +description: "Contract case for the questionnaire hand-off on a `tenant` item whose owner is not the user (the partner endpoint's accepted SSH key type). Why hard: the Brief is locked and the natural handoff is /planning:plan alone, so the one question only the partner's ops lead can answer stays parked in the Brief with nobody asked to answer it." tags: [interview, handoff, questionnaire] runs: 3 max_turns: 12 diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/samples/names-questionnaire.json b/plugins/planning/evals/interview-owner-deferred-questionnaire/samples/names-questionnaire.json similarity index 100% rename from plugins/planning/evals/interview-person-deferred-questionnaire/samples/names-questionnaire.json rename to plugins/planning/evals/interview-owner-deferred-questionnaire/samples/names-questionnaire.json diff --git a/plugins/planning/evals/interview-person-deferred-questionnaire/samples/questionnaire-beside-next-step.json b/plugins/planning/evals/interview-owner-deferred-questionnaire/samples/questionnaire-beside-next-step.json similarity index 100% rename from plugins/planning/evals/interview-person-deferred-questionnaire/samples/questionnaire-beside-next-step.json rename to plugins/planning/evals/interview-owner-deferred-questionnaire/samples/questionnaire-beside-next-step.json