feat(planning): route person-held interview questions to questionnaire and add eval-first cases

.changes/planning/feat-interview-techniques-323edf76.md

@@ -0,0 +1,8 @@
+---
+bump: minor
+---
+
+### Added
+
+- `/planning:interview` recommends `/planning:questionnaire` at handoff for a deferred question only a named person other than the user can answer (an assumption-sweep `person` item, or a `tenant` item whose owner is not the user), beside the usual next step. The new `interview-owner-deferred-questionnaire` eval case grades it on a `tenant` item.
+- Seven eval cases for interview behaviors not yet in the skill (per-answer read-back of a dictated answer, an active final confirmation, a changes-only final restatement, a checkable basis and flip condition per recommendation, the "something else" closing probe, a MoSCoW scope cut before `/planning:wayfind`, and one crux question on a rejected recommendation). Each rule lands only if its case fails without it.

plugins/planning/evals/interview-closing-probe-something-else/graders/no-anything-else.md

@@ -0,0 +1,7 @@
+---
+type: regex
+pattern: '\banything else\b'
+flags: i
+match: not_contains
+arm: both
+---

plugins/planning/evals/interview-closing-probe-something-else/graders/says-something-else.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '\bsomething else\b[^\n]*\?'
+flags: i
+arm: both
+---

plugins/planning/evals/interview-closing-probe-something-else/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "Eval-first case for the closing probe wording. Why hard: 'Anything else?' is the habitual close, and it invites a 'no'; 'Is there something else...?' left fewer concerns unmet in a primary-care trial (Heritage et al. 2007)."
+tags: [interview, closing-probe, wording]
+runs: 3
+max_turns: 12
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill]
+expected_outcome: "Round 1 closes with a probe worded 'something else' (for example 'Is there something else that would change these recommendations?') and does not use 'anything else'"
+---
+
+/planning:interview Add a nightly job that deletes expired password-reset tokens from the database.

plugins/planning/evals/interview-closing-probe-something-else/samples/no-anything-else.json

@@ -0,0 +1,22 @@
+{
+  "pass": [
+    {
+      "answer": "Is there something else about this job that would change these recommendations?",
+      "why": "something-else wording"
+    },
+    {
+      "answer": "",
+      "why": "an empty reply holds no anything-else; the other grader fails it"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Anything else I should know?",
+      "why": "anything else"
+    },
+    {
+      "answer": "Is there anything else that would change these?",
+      "why": "anything else inside a longer probe"
+    }
+  ]
+}

plugins/planning/evals/interview-closing-probe-something-else/samples/says-something-else.json

@@ -0,0 +1,22 @@
+{
+  "pass": [
+    {
+      "answer": "Q1: How long should an expired token be kept before deletion?\nMy recommendation: delete at once.\n\nIs there something else about this job that would change these recommendations?",
+      "why": "something-else probe as a question"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty reply"
+    },
+    {
+      "answer": "Q1: How long should an expired token be kept?\n\nAnything else I should know?",
+      "why": "anything-else probe"
+    },
+    {
+      "answer": "If something else comes up, tell me.",
+      "why": "not asked as a question"
+    }
+  ]
+}

plugins/planning/evals/interview-crux-on-rejection/graders/asks-about-kafka.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '\bkafka\b[^\n?.;]*\?'
+flags: i
+arm: both
+---

plugins/planning/evals/interview-crux-on-rejection/graders/no-reargue-name.md

@@ -0,0 +1,7 @@
+---
+type: regex
+pattern: '\bdocs-publish\b[^\n]{0,160}\b(?:instead|consistent|consistency|reconsider)\b|\b(?:reconsider|are you sure)\b[^\n]{0,80}\bship-docs\b'
+flags: i
+match: not_contains
+arm: both
+---

plugins/planning/evals/interview-crux-on-rejection/graders/one-crux-and-defers.md

@@ -0,0 +1,16 @@
+---
+type: llm
+arm: both
+---
+
+PASS only if all of these hold:
+
+1. For Q2 the reply asks one question aimed at what would have to be true for Kafka, or for the
+   Postgres queue, to be the right choice (for example a need Kafka meets that the user knows of,
+   such as other consumers or replay), or it proposes one concrete check that would settle it. It
+   does not argue for Postgres more than once, and it accepts Kafka if the user holds to it.
+2. For Q3 the reply accepts `ship-docs` as the user's choice without arguing for `docs-publish`.
+3. Q1 is accepted.
+
+FAIL if the reply re-argues the Postgres recommendation at length or more than once, challenges
+the `ship-docs` name, or records Kafka with no question or check at all.

plugins/planning/evals/interview-crux-on-rejection/prompt.md

@@ -0,0 +1,19 @@
+---
+description: "Eval-first case for the disagreement move. Why hard: the user rejects two recommendations, one technical (Postgres queue versus Kafka) and one personal (the command's name). Re-arguing both, or silently giving up on both, are the easy replies; the target is one crux question on the technical one and plain deference on the name."
+tags: [interview, disagreement, crux]
+runs: 3
+max_turns: 12
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill]
+expected_outcome: "The reply asks one crux question about Q2 (what would have to be true for Kafka, or for the Postgres queue, to be right), or proposes one check that would settle it, and accepts `ship-docs` for Q3 without arguing"
+---
+
+/planning:interview Add a command-line tool that publishes our docs site.
+
+You already asked round 1:
+
+Q1: Where should the build run? My recommendation: in the existing CI workflow, since the repo already builds there.
+Q2: How should publish jobs be queued? My recommendation: a Postgres table used as a queue, since Postgres is already deployed and the volume is a few jobs a day.
+Q3: What should the command be called? My recommendation: `docs-publish`, matching our other `docs-*` commands.
+
+My answers: Q1 yes. Q2 no, use Kafka. Q3 no, call it `ship-docs`, I just like it better.

plugins/planning/evals/interview-crux-on-rejection/samples/asks-about-kafka.json

@@ -0,0 +1,30 @@
+{
+  "pass": [
+    {
+      "answer": "Q2: What would have to be true for Kafka to be the better queue here, for example other services consuming these publish events?",
+      "why": "crux question naming Kafka"
+    },
+    {
+      "answer": "Q2: Is there a consumer besides this tool that needs the Kafka topic?",
+      "why": "question about the Kafka choice"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty reply"
+    },
+    {
+      "answer": "Q2 recorded: Kafka.",
+      "why": "records Kafka with no question"
+    },
+    {
+      "answer": "Recorded: Q1 CI, Q2 Kafka, Q3 ship-docs. Ready for round 2?",
+      "why": "the question mark belongs to another sentence"
+    },
+    {
+      "answer": "Q2: Kafka is overkill... Are you sure?",
+      "why": "re-argues instead of asking the crux"
+    }
+  ]
+}

plugins/planning/evals/interview-crux-on-rejection/samples/no-reargue-name.json

@@ -0,0 +1,30 @@
+{
+  "pass": [
+    {
+      "answer": "Q3: `ship-docs` it is.",
+      "why": "accepts the name"
+    },
+    {
+      "answer": "Q3: Sure, `ship-docs` it is.",
+      "why": "plain deference that says sure"
+    },
+    {
+      "answer": "Q3: renamed docs-publish to ship-docs, matching your preference.",
+      "why": "accepts the rename"
+    },
+    {
+      "answer": "",
+      "why": "an empty reply re-argues nothing; the other graders fail it"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Q3: I'd still suggest docs-publish instead, for consistency with the other docs-* commands.",
+      "why": "re-argues the name"
+    },
+    {
+      "answer": "Q3: Are you sure about ship-docs? It breaks the docs-* pattern.",
+      "why": "challenges the personal choice"
+    }
+  ]
+}

plugins/planning/evals/interview-crux-on-rejection/samples/one-crux-and-defers.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Q1: accepted, the build runs in the existing CI workflow.\nQ2: One question before I record Kafka: what would have to be true for Kafka to be right here, such as other services consuming these publish events or a need to replay them? If there is one, Kafka it is.\nQ3: `ship-docs` it is.",
+      "why": "one crux question on Q2, defers on Q3"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Q2: Kafka is overkill for a few jobs a day; Postgres is already deployed and simpler to run. I strongly recommend keeping Postgres. Kafka also adds an operational burden you do not need.\nQ3: I'd still go with docs-publish for consistency.",
+      "why": "re-argues Q2 and challenges Q3"
+    },
+    {
+      "answer": "Recorded: Q1 CI, Q2 Kafka, Q3 ship-docs. Round 2 next.",
+      "why": "no crux question on the technical rejection"
+    }
+  ]
+}

plugins/planning/evals/interview-final-brief-changes-only/graders/changes-only-restatement.md

@@ -0,0 +1,16 @@
+---
+type: llm
+arm: both
+---
+
+PASS only if all of these hold:
+
+1. The reply restates the intent of the work (CSV export of the admin orders page) in a sentence
+   or two.
+2. It shows the Q5 change: the file name is now orders-export-<timestamp>.csv.
+3. It shows both assumptions the sweep added: exports run synchronously in the request, and the
+   export respects the user's row-level permissions.
+4. It does not re-list the answers that were accepted and read back unchanged (Q1, Q2, Q3, Q4,
+   Q6, Q7, Q8). A count of them, or one line saying they stand as read back, is fine.
+
+FAIL if any of the four does not hold.

plugins/planning/evals/interview-final-brief-changes-only/graders/omits-unchanged-items.md

@@ -0,0 +1,7 @@
+---
+type: regex
+pattern: '\bgzip\b|\b50,?000\b|\btoolbar\b'
+flags: i
+match: not_contains
+arm: both
+---

plugins/planning/evals/interview-final-brief-changes-only/graders/shows-changed-item.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: 'orders-export'
+flags: i
+arm: both
+---

plugins/planning/evals/interview-final-brief-changes-only/graders/shows-new-assumptions.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '^(?=[\s\S]*\bsynchronous)(?=[\s\S]*\bpermission)'
+flags: i
+arm: both
+---

plugins/planning/evals/interview-final-brief-changes-only/prompt.md

@@ -0,0 +1,13 @@
+---
+description: "Eval-first case for the final restatement. Why hard: eight answers are on record and listing all of them looks thorough, but seven were already read back and accepted, so the list buries the one changed answer and the two assumptions the user has never seen."
+tags: [interview, confirmation, restatement]
+runs: 3
+max_turns: 12
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill]
+expected_outcome: "The restatement gives the intent, the Q5 file name change, and the sweep's two new assumptions (synchronous export, row-level permissions), and does not re-list the seven unchanged accepted answers"
+---
+
+/planning:interview Add CSV export to the admin orders page.
+
+Where we are: every question is answered and each answer was read back when you received it. Q1 export the filtered rows (accepted). Q2 columns as shown in the table (accepted). Q3 timestamps in UTC (accepted). Q4 a cap of 50,000 rows (accepted). Q5 file name: you recommended orders-YYYY-MM-DD.csv and read it back; I later changed it to orders-export-<timestamp>.csv. Q6 the button sits in the table toolbar (accepted). Q7 gzip compression off (accepted). Q8 one audit log entry per export (accepted). The assumption sweep then added two assumptions I have not seen: exports run synchronously in the request, and the export respects the user's row-level permissions. Restate the contract for my final confirmation.

plugins/planning/evals/interview-final-brief-changes-only/samples/changes-only-restatement.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Intent: ops can download the filtered order list from the admin orders page as CSV.\nChanged since read-back: Q5, the file is now orders-export-<timestamp>.csv.\nNew assumptions from the sweep: exports run synchronously in the request; the export respects the user's row-level permissions.\nQ1-Q4 and Q6-Q8 stand as read back.\n\nConfirm, or give me the numbers to change.",
+      "why": "intent, the change, the two new assumptions, nothing re-listed"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Contract:\nQ1 filtered rows. Q2 columns as shown. Q3 UTC. Q4 50,000-row cap. Q5 orders-export-<timestamp>.csv. Q6 toolbar button. Q7 gzip off. Q8 audit log entry.\nAssumptions: synchronous export; row-level permissions.",
+      "why": "re-lists every unchanged answer"
+    },
+    {
+      "answer": "Intent: CSV export of the admin orders page. Changed: Q5, orders-export-<timestamp>.csv. Confirm?",
+      "why": "omits the two new assumptions"
+    }
+  ]
+}

plugins/planning/evals/interview-final-brief-changes-only/samples/omits-unchanged-items.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Intent: ops can download the filtered order list as CSV.\nChanged: Q5, file name orders-export-<timestamp>.csv.\nNew assumptions: synchronous export; row-level permissions apply.\nThe other seven answers stand as read back.\n\nConfirm, or give me the numbers to change.",
+      "why": "no unchanged item re-listed"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Q1 filtered rows. Q2 columns as shown. Q3 UTC. Q4 50,000-row cap. Q5 orders-export-<timestamp>.csv. Q6 toolbar button. Q7 gzip off. Q8 audit log entry.",
+      "why": "re-lists every accepted answer"
+    },
+    {
+      "answer": "Changed: Q5. Also note the cap stays at 50000 rows.",
+      "why": "re-shows the unchanged Q4 cap"
+    }
+  ]
+}

plugins/planning/evals/interview-final-brief-changes-only/samples/shows-changed-item.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Changed since read-back: Q5, the file is now orders-export-<timestamp>.csv.",
+      "why": "shows the changed file name"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty reply"
+    },
+    {
+      "answer": "File name: orders-YYYY-MM-DD.csv.",
+      "why": "shows the superseded recommendation, not the change"
+    }
+  ]
+}

plugins/planning/evals/interview-final-brief-changes-only/samples/shows-new-assumptions.json

@@ -0,0 +1,22 @@
+{
+  "pass": [
+    {
+      "answer": "New assumptions from the sweep:\n- Exports run synchronously in the request.\n- The export respects the user's row-level permissions.",
+      "why": "both assumptions"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty reply"
+    },
+    {
+      "answer": "New assumption: exports run synchronously in the request.",
+      "why": "missing the permissions assumption"
+    },
+    {
+      "answer": "New assumption: the export respects row-level permissions.",
+      "why": "missing the synchronous assumption"
+    }
+  ]
+}

plugins/planning/evals/interview-final-confirm-active-reply/graders/active-reply-requested.md

@@ -0,0 +1,12 @@
+---
+type: llm
+arm: both
+---
+
+Judge only the request that closes the restatement.
+
+PASS only if the reply asks the user to give an active answer: to confirm the contract, or to name
+the numbers (or ids) of the items they want changed.
+
+FAIL if the closing request is a yes/no check such as "Sound good?", "Does this look right?",
+"Any thoughts?" or "OK?", or if the reply asks for no confirmation at all.

plugins/planning/evals/interview-final-confirm-active-reply/graders/asks-confirm-or-numbers.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '\bconfirm\b[^\n]{0,200}\b(?:numbers?|Q\d+|which (?:items?|questions?|rows?))\b|\b(?:give|tell|list|name|send)\b[^\n]{0,60}\b(?:numbers?|Q\d+)\b[^\n]{0,80}\bchange\b'
+flags: i
+arm: both
+---

plugins/planning/evals/interview-final-confirm-active-reply/graders/no-vague-confirm.md

@@ -0,0 +1,7 @@
+---
+type: regex
+pattern: '\b(?:sounds?|looks?|seems?) (?:good|right|ok(?:ay)?|fine)\?|\b(?:does|do) (?:this|that|everything|it) (?:look|sound|seem) (?:good|right|ok(?:ay)?|fine)\?|\b(?:any|other) (?:thoughts|concerns|feedback)\?|\b(?:ok(?:ay)?|good to go|all good)\?'
+flags: i
+match: not_contains
+arm: both
+---

plugins/planning/evals/interview-final-confirm-active-reply/prompt.md

@@ -0,0 +1,13 @@
+---
+description: "Eval-first case for the final confirmation wording. Why hard: every answer is in and the sweep is clean, so closing with 'Sound good?' reads as polite and natural, but a bare 'yes' to it confirms nothing specific."
+tags: [interview, confirmation, wording]
+runs: 3
+max_turns: 12
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill]
+expected_outcome: "The restatement ends by asking the user to confirm, or to name the numbers of the items to change, and never asks 'sound good?' or a similar yes/no"
+---
+
+/planning:interview Add CSV export to the admin orders page.
+
+Where we are: rounds 1 and 2 are answered. Q1 export the filtered rows. Q2 columns as shown in the table. Q3 timestamps in UTC. Q4 a cap of 50,000 rows. Q5 file name orders-YYYY-MM-DD.csv. The frontier is empty and the assumption sweep added no open row. Restate the contract and ask me to confirm it.