Skip to content

test(planning): run the seven interview eval-first cases and file a rule slice for each case that fails #9794

Open

Description

@kyle-sexton

needs-human: a paid claude plugin eval run (pre-approved within prepaid credit, Q46); no unattended lane runs plugin evals.

Parent: #7269.

Run

Run each case on a copy of plugins/planning outside git (the repository has too many linked worktrees for the CLI to screen), one or two at a time:

claude plugin eval <copy-of-plugins/planning> --case <name> --runs 3 --judge-model opus --keep-temp

Cases: the seven from #7247 (read-back of a dictated answer, active-reply final confirmation, changes-only final brief, recommendation basis and flip, "something else" closing probe, MoSCoW scope cut before wayfind, crux question on a rejected recommendation). Size the cost ceiling from the last partial pass's costUsd. Read each result with run-validity.py and noise-report.py; for final-confirm-active-reply and final-brief-changes-only read the kept traces.

Done when

Activity

  1. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 24, 2026
  2. github-actions commented on Oct 24, 2026

    @github-actions
    Contributor

    Proposed work class: none

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none

    Understood: run the seven interview eval-first cases from #7247 against a copy of plugins/planning, post a VALID with-plugin result for each on #7269, and file a rule slice under #7269 for each case that fails on main. The paid claude plugin eval runs need an attended session, so an unattended lane can't do them. The slice-filing item depends on those results. No likely duplicates found among the open issues.

    Automated triage. A maintainer reviews it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.priority: mediumReal value, no hard deadline; normal backlog flow.status: needs-decisionAwaiting a human or maintainer judgment call.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions