Repository navigation
test(planning): run the seven interview eval-first cases and land the rules whose case fails #7269
Description
Summary
PR #7247 adds seven eval-first cases for /planning:interview behaviors, a to g: read-back of a dictated answer, active-reply final confirmation, changes-only final brief, a checkable basis and flip condition on each recommendation, the "something else" closing probe, a MoSCoW Must-test scope cut before /planning:wayfind, and one crux question on a rejected recommendation. Under Instruction economy, the rule for each case lands only if its case fails on main.
State
- All seven llm judges were calibrated against an opus judge at 100% agreement.
- The first full pass stopped at its cost ceiling (
partial: true). Five cases finished,interview-recommendation-basis-and-flipfinished partly, andinterview-scope-cut-moscowdid not run. Under the plugin-eval reading rules, a partial run's scores are not comparable, so they are not used. - Per-run cost came out well above the fresh-suite headroom estimate. Size the next pass from that pass's own
costUsd(see/evals:plugin-eval, Cost).
Fix
- Run the cases one or two at a time (
--case <name>,--runs 3,--judge-model opus,--keep-temp) on a copy ofplugins/planningoutside git, because the repository has too many linked worktrees for the CLI to screen. - Read each result with
run-validity.pyandnoise-report.py. Forfinal-confirm-active-replyandfinal-brief-changes-only, read the kept traces: those cases allow no Bash, so the Step 3 register-gate script cannot run in them. - For each case that fails on main, add its rule to the interview skill's context spokes in a PR that cites the result.
Activity
github-actions commentedon Oct 11, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Large
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I understood this as the umbrella for running the seven
/planning:intervieweval-first cases from #7247. Each rule lands only if its case fails on main. The first pass hit its cost ceiling, so the next pass needs sizing from its owncostUsd.A comment says the work was split into sub-issues: #9793 (assumption-sweep line, eval case 30, digest re-pins) and #9794 (run the seven cases and file a rule slice for each failing case). Remaining work should be tracked there. This issue is too large for one change and is better kept as the parent. No likely duplicates beyond those slices.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 11, 2026
Also owed (attended-only, because
interview-defenses.test.shpins both files):skills/interview/context/assumption-sweep.md: after "### Deferred questionswith its arbiter tag." add "A deferredpersonitem is also named in the Step 5 recommendation of/planning:questionnaire, which sends it to that person." The new file digest isd4fc0b53fbcd9dde548e3a1d65e075972bb8a8c1cd78a09ca0dd5b82a18b6f07.skills/interview/evals/evals.json: append rubric case 30person-deferred-routes-to-questionnaire(a DBA owns the retention window; the question is deferred to her and routed to questionnaire, with no row resolved on her behalf). The roster digest with that case isb7197f9f75d8966eee2a7e35ad6cdcdec5748d8e5e0dd02d922ca3c49774849f.#7247 ships the hand-off through the Step 5 sentence and the plugin eval case
interview-person-deferred-questionnaire. On its one comparable run that case scored 0.83 with the plugin against 0 without, above the 0.8 threshold; one case is too few for the noise report to call that a gain.kyle-sexton commentedon Oct 19, 2026
This was generated by AI during triage.
Hold: running the seven cases spends eval budget, and the last pass stopped at its cost ceiling, so the owner has to approve the spend for the next pass. The
interview-defenses.test.shpin updates in the 2026-10-11 comment also have to be done in an attended session.kyle-sexton commentedon Oct 24, 2026
Released by the owner's HITL session (2026-10-24, C12): it waited only on removed gates (protected paths, attended-only pins, lane limits, sync allowlists). Split into sub-issues; intake triage re-runs on each slice to set its fields, then admission takes it.
Slices:
github-actions commentedon Oct 25, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
melodic-automation-lanes commentedon Oct 25, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited; Size large: the split into sub-issues at intake is pending (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 25, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: needs-decision; its comment says why.melodic-automation-lanes commentedon Oct 25, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Close this issue as a parent: #9794 carries the eval run and per-case rule slices, and #9793 carries the digest re-pins and eval case 30.
Related work
Open questions
RECOMMENDED: Yes, close it and track the work in #9794 (needs-human paid eval run) and #9793 (lane slice for the assumption-sweep line, case 30 and re-pins). Basis: The owner's 2026-10-24 comment splits this issue into #9793 and #9794; candidate #9794 is open and states the same outcome.
Sources read
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.