test(work-items): run the new work-loop and attend-queue evals and a live C3 check

Follow-up to #7159, #7204 and #7208.

Those PRs added or changed eval cases that have only passed the static checks (check-changed-skills, the evals.json schema). None has been run against a model:

plugins/work-items/skills/work-loop/evals/evals.json: case 6 (changed); cases 16, 17 and 18 (truncated snapshot: escalate at once, then stop a headless lane); the first-drain-ratification cases from fix(work-items): end the first-drain period only on a person-applied label #7204.
plugins/work-items/skills/attend-queue/evals/evals.json: case 9 (a ratify-first-drain proposal on a telemetry issue lists as a [ratify] row, and other telemetry markers stay hidden).

The manual C3 ratification check in plugins/work-items/skills/work-loop/AGENTS.md was also done on paper by a fresh-context agent, not against real tracker items, because a live run posts to public issues.

Do

Run these suites with /evals:plugin-eval against a no-plugin baseline, and record pass rates.
Run the AGENTS.md C3 check live against a throwaway test issue, or a local-markdown binding, so nothing posts to real public issues.
Fix any case or skill text that fails.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #9668

I understood this as a measurement task. It covers three things:

Run the new work-loop and attend-queue eval cases with /evals:plugin-eval against a no-plugin baseline.
Run the C3 ratification check once on a local-markdown binding.
Open a follow-up issue for each failing case.

The comment that released the hold already settles the open questions about who runs the evals and what they cost. The ratify cases and the C3 check wait on #9668, which is still open. The work-loop cases 6, 16, 17 and 18 can be run now. No duplicates found.

_{Automated triage, applied by the lanes App.}

The triage sweep dispatched intake triage for this issue (attempt 2 of 2).

Not admitted to the issue lane yet: blocker #9668 is not closed by a listed human or a merge.

The latest triage verdict holds nothing: the issue is at status: ready.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: Record model-run pass rates for the named work-loop and attend-queue eval cases against a no-plugin baseline, run the AGENTS.md C3 check on a local-markdown binding, and fix any failing case or skill text.

Related work

fix(work-items): stop work-loop trusting failed, truncated or agent-editable reads #7159 (related): merged PR that added the truncated-snapshot cases; this issue follows it up
fix(work-items): filter list reads in the provider and order work by created_at #7208 (related): merged PR named as a source of the changed eval cases
fix(work-items): make every lane escalation reach attend-queue #9558 (related): merged PR on lane escalations reaching attend-queue; may have changed the same skill text and cases

Open questions

Can an unattended lane run /evals:plugin-eval against a model with a no-plugin baseline, or must a person run it?
RECOMMENDED: Treat the eval runs and the live C3 check as person-run checks, and split any case or skill fixes into separate issues. Basis: judgment
Where are the pass rates recorded, and what counts as passing?
RECOMMENDED: Post per-case pass rates as an issue comment; a case passes when it passes every sampled run with the plugin and does worse in the baseline. Basis: judgment
Which cases are the first-drain-ratification cases from #7204? That PR is not among the candidates, so I could not identify them.
RECOMMENDED: List the case numbers in plugins/work-items/skills/work-loop/evals/evals.json in the issue body. Basis: judgment
Should the live C3 check use a throwaway GitHub repo or the local-markdown binding?
RECOMMENDED: Use the local-markdown binding so nothing posts to public issues. Basis: plugins/work-items/skills/work-loop/AGENTS.md:5
Should failing cases or skill text be fixed in this issue or in follow-up issues?
RECOMMENDED: Open one follow-up issue per failure so this issue stays a measurement with checkable done-when items. Basis: judgment

Sources read

plugins/work-items/skills/work-loop/AGENTS.md:3: manual C3 ratification check has no automated test surface; needs an item with the ratification phrase and a prior Superseded comment, and ends with one kind=ratify-c3 comment and a [ratify] row in attend-queue
plugins/evals/skills/plugin-eval/SKILL.md: the plugin-eval skill the issue asks to run exists

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

This was generated by AI.

Releasing the hold: no owner question is open. The refinement asked about eval spend or who runs the eval. Owner rules settle both. Evals run on the subscription login or not at all, never with an API key, and none start while the rate limit is near (Q110). Subscription and prepaid spend are not induced cost (Q46, Q91). Under the owner's 2026-10-26 stop rule (C14), an eval or host run is no longer an owner step, so an agent session runs it. The label comes off and intake triage runs again. Criteria:

Per-case pass rates with and without the plugin are posted here for work-loop cases 6, 16, 17 and 18, from /evals:plugin-eval runs on the subscription login.
After feat(pr-pipeline): stop gating any step on work class (owner Q85) #9668 merges, the ratify cases (attend-queue case 9 and the first-drain cases) are each run with the pass rate posted, or removed if feat(pr-pipeline): stop gating any step on work class (owner Q85) #9668 removed their gate.
After feat(pr-pipeline): stop gating any step on work class (owner Q85) #9668 merges, either the C3 check is gone from work-loop/AGENTS.md, or it was run once on a local-markdown binding and the result is posted.
Each failing case has its own follow-up issue linked from here.

The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
It added host-run and removed agent-ready: an agent run on a host builds it, and it waits on no decision from a person.

In the run's words:

host-only: the issue's criteria need model runs of claude plugin eval (the /evals:plugin-eval CLI step, plugins/evals/skills/plugin-eval/SKILL.md "Run" step 3) on the subscription login, with per-case pass rates posted for work-loop cases 6, 16, 17 and 18. This lane loads only work-items, cannot run the claude CLI, and can post no issue comment. Two criteria also wait on ＃9668, which has no commit on this checkout's history: the ratify cases (attend-queue case 9 and the first-drain cases) and the AGENTS.md C3 check, which either gets removed or is run once live on a local-markdown binding. A host session on the subscription login has to run the suites, post the rates, and file one follow-up issue per failing case.

What the run says it did not do:

Eval runs for work-loop cases 6, 16, 17, 18 with and without the plugin: the claude plugin eval CLI cannot run in this lane
Ratify cases (attend-queue case 9, first-drain cases) and the work-loop/AGENTS.md C3 check: they wait on ＃9668, which has not merged into this checkout
Follow-up issues for failing cases: no case was run, so none could be filed
tidy, review, verify, checks steps: nothing changed, so none ran

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40896934130

To hand it back to the lane, remove host-run and add agent-ready again.

Navigation Menu

Repository navigation

test(work-items): run the new work-loop and attend-queue evals and a live C3 check #7226

Do

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

test(work-items): run the new work-loop and attend-queue evals and a live C3 check #7226

Description

Do

Activity

github-actions commented on Oct 11, 2026

github-actions commented on Oct 25, 2026

melodic-automation-lanes commented on Oct 25, 2026

melodic-automation-lanes commented on Oct 25, 2026

melodic-automation-lanes commented on Oct 25, 2026

kyle-sexton commented on Oct 26, 2026

melodic-automation-lanes commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions