Repository navigation
test(intake): eval cases for intake triage's re-entry handling #9040
Description
Problem
#8491 asks for four eval cases covering intake triage's re-entry handling:
- A partial answer.
- A case that needs a human.
- An answer that settles everything.
- A re-park.
Neither repository has an eval convention for a CI lane's prompt. ci-workflows has only the model-free intake-triage-*.test.cjs files, and claude-code-plugins evals exist only under plugins/*/evals, for skills. The prompt change itself is in #9038 and melodic-software/ci-workflows#854.
Proposal
Decide where prompt evals for CI lanes live and how they run without a live issue. For example, a claude plugin eval-style suite fed recorded, de-identified issue fixtures, run on demand rather than on every PR. Then add the four cases above, each holding an issue body, its comments with one **Decision recorded.** comment, and the expected label and comment shape.
Acceptance
- The four cases exist and run with a documented command.
- Each case fails against the prompt before feat(pr-pipeline): carry a recorded decision through intake triage on re-entry #9038's instructions and passes after them.
- No case holds a raw session or issue transcript (
.claude/rules/eval-case-transcripts.md).
Activity
github-actions commentedon Oct 22, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: intent
Blocked by: none
#9040 (comment)
I understood this as a request for four re-entry eval cases plus a home for CI-lane prompt evals. The decision parks both. The later release comment only re-ran triage, so it doesn't say whether the park still holds.
Related: #8491, #9038.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 22, 2026
Decision recorded.
Question: Where do evals for CI-lane prompts live?
Answer: Owner, 2026-10-22, Issue Triage session: "Go with recommendations ... My focus right now is getting the intake, admit, build, refine, review, fix, verify, merge, and post-merge lanes ... working flawlessly as just a continuous conveyor belt." The owner delegated the open calls to the lane within that focus.
Decision: parked. No new eval convention now; the intake re-entry cases wait until the conveyor belt is stable. Revisit with the next intake-triage prompt change.
kyle-sexton commentedon Oct 25, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.melodic-automation-lanes commentedon Oct 25, 2026
The triage verdict left this issue at
status: needs-decision:melodic-automation-lanes commentedon Oct 25, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Four intake re-entry eval cases (partial answer, needs human, settling answer, re-park) with a documented run command, once the owner's park lifts at the next intake-triage prompt change.
Dependencies
Related work
Open questions
RECOMMENDED: Stay parked. The owner's Decision recorded comment of 2026-10-22 parks the eval convention and the re-entry cases until the lanes are stable, and the 2026-10-25 release only re-ran triage without reversing it. Basis: Decision recorded comment on #9040 (2026-10-22)
Sources read
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Oct 26, 2026
Closing as a duplicate of #8491, whose item 4 covers the same four intake re-entry eval cases. #8491's 2026-10-25 decision (#8491 (comment)) supersedes this issue's 2026-10-22 park.