verification/confirm: probe whether confirm reads the PLAN.md Brief #7594

Activity

1. kyle-sexton

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 16, 2026
2. kyle-sexton

   added the

   Task

   issue type on Oct 16, 2026
3. kyle-sexton

   added a parent issue on Oct 16, 2026

   • verification: intent chain, pstack ready rules and dispatch brief #7587
4. kyle-sexton

   set

   Work status

   to

   needs-spec

   on Oct 16, 2026
5. kyle-sexton

   marked this as blocking 5 issues on Oct 16, 2026

   • verification/confirm: intent-lookup ladder starting from the Brief #7602
   • implementation: pass the PLAN.md pointer at each handoff #7603
   • source-control/pull-request: carry the Brief's acceptance criteria into the PR body #7604
   • work-items/decompose: carry the Brief's goal and constraints into each item #7605
   • work-items: decompose and track add stamp Issue Type only #7606
6. github-actions commented on Oct 16, 2026

   

   github-actionsBot

   on Oct 16, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for the report. Automated triage handed this issue to a maintainer.

   _{Automated triage. A maintainer reviews it.}
7. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 16, 2026
8. kyle-sexton

   added

   needs-specNeeds a Brief or plan before anyone can build it.Needs a Brief or plan before anyone can build it.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 16, 2026
9. kyle-sexton commented on Oct 16, 2026

   

   kyle-sexton

   on Oct 16, 2026

   MemberAuthor

   More actions

   Probe result: yes in this setup, but by the session's own file search, not by anything the skill says. In a fresh session with no intent in the prompt, /verification:confirm found .work/slugify/PLAN.md, read it, and judged the change against the Brief's acceptance criteria (AC1 met, AC2 not met, AC3 incomplete). It reached the file through ad-hoc ls -la and find calls in a five-file repo. The skill's intent step (plugins/verification/skills/confirm/SKILL.md:176, "Scan the conversation for the original request, the approved plan...") and context/outcome.md:15 ("Find the approved plan in the conversation") never point at PLAN.md. In a larger repo nothing makes that read happen, so R1 still has work to do.

   Setup

   • Claude Code 2.1.301, verification plugin 0.9.0 (its intent step is the same as main's), run on 2026-10-16.
   • A throwaway git repo with one base commit (README, .gitignore covering .work/). The change was textutil.py with slugify(text) as "-".join(text.lower().split()) and test_textutil.py with one test ("Hello World" → "hello-world"), both intent-to-add so they appear in git diff HEAD.
   • An approved PLAN.md at .work/slugify/PLAN.md, the <memory_dir>/<topic-slug>/PLAN.md default the plan skill writes to. It follows the plan-md-anatomy template and carries an Approval: line. Its Brief has three acceptance criteria:
     • AC1: lowercase and hyphenate (met by the change).
     • AC2: strip punctuation, slugify("Hello, World!") → "hello-world" (missed).
     • AC3: a unit test per criterion (missed for AC2).
   • Each run was a fresh headless session started in the repo: claude -p "<prompt>" --output-format stream-json --verbose --permission-mode auto.
   • The throwaway repo sat inside a checkout of this repository, so that checkout's CLAUDE.md/AGENTS.md probably loaded too. Nothing in them states this task's intent.

   Probe run (prompt: /verification:confirm, no intent)

   • Tool sequence: the four git context calls, then Read textutil.py and test_textutil.py, then ls -la and find ... -type f over the repo, which listed .work/slugify/PLAN.md, then Read README and Read PLAN.md. Stage 1 (/toolchain:check python, /toolchain:lint cross-cutting) came after that. It then dispatched a fresh-context verifier, which also read PLAN.md.
   • It cited the Brief. Quotes from the report:
     • "Mode: outcome, because an approved plan exists."
     • "slugify("Hello, World!") returns "hello,-world!" instead of the planned "hello-world"."
     • "| AC1: lowercase, words joined with hyphens | Met |"
     • "| AC2: strip punctuation | Not met |"
     • "| AC3: a unit test for each criterion | Incomplete | There is a test for AC1 only, none for AC2. |"
   • Verdict: NEEDS WORK, which is correct.
   • The run's permission classifier denied one Bash call, a direct python3 -I -c call of slugify. The run said the AC2 result was "worked out by reading the code, not observed".

   Control run (intent stated in the prompt, same repo and PLAN.md)

   • Prompt: /verification:confirm The intent: slugify(text) must lowercase the input and join words with single hyphens, strip punctuation (...), and have a unit test for each behavior.
   • It never opened or listed PLAN.md: no tool call touched .work/, and "PLAN" appears nowhere in the stream. It judged against the prompt's intent instead:
     • "| Strip punctuation: "Hello, World!" → "hello-world" | None | None | Missing. |"
   • Verdict: NEEDS WORK, which is also correct. No permission denials.

   Reading

   • The Brief reached the verdict only because the session went looking through a tiny tree. When intent is in the prompt, the skill does not look for PLAN.md at all.
   • The skill text gives no step that loads PLAN.md, so whether a larger repo or another run would find it is untested. Both runs are n=1.
10. kyle-sexton

    closed this as completedon Oct 16, 2026