ci: gate changed test files on the can't-fail scan in ci-status and run mutation testing post-merge only #9184

Refs: #7581
Refs: #7555
Blocked by #9194

Summary

The can't-fail scanner (/testing:audit's deterministic scan,
plugins/testing/skills/audit/scripts/cant-fail-scan.sh) runs only on demand today: no step in
.github/workflows/pr-require-checks.yml calls it (a search of .github/workflows/ for
cant-fail finds nothing; its only CI mention is the resolve-issue verify-script allow-list,
.github/actions/resolve-config/verify-scripts.test.mjs:55, :63). PLAN Phase 13 asks for the
scan, scoped, inside the required check with the test guards on, and for mutation testing to run
post-merge only, never per PR (design D8/Q18, D71/Q79; #7581 Decisions "Tests that can't fail").

Attended build (not lane). It edits .github/workflows/pr-require-checks.yml, the workflow
behind ci-status, the one required check (AGENTS.md "Pull request states"). A wrong gate there
blocks every PR, so the change needs a person and a Security and Cost review: the new step runs
repository code on every PR, and Part B adds a post-merge model lane that spends plan usage.

Predicted pr-merge risk: HIGH. It touches .github/**, a HIGH-floor path
(plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py:209).

Current behavior (origin/main ba598f0)

• cant-fail-scan.sh --check exits 1 when a gating rule fires (zero-assertion,
  recomputed-expectation), 2 when the scan cannot run, cannot read inputs or examines 0 test files,
  0 only for a clean read of at least one test file (cant-fail-scan.sh:113-119, usage
  :163-169). --file <path> scans one file with the same exit codes (:175).
• The whole tree fails the gate today. A --check run over an export of origin/main
  ba598f08 exited 1 with 11 gating findings, in 82 s wall on a WSL workstation, over 1,279 test
  files: 10 zero-assertion findings in plugins/planning/shared/tests/test_people_proposals.py
  (lines 126 to 240) and 1 in plugins/planning/shared/tests/test_session.py:3111. app-control: new plugin plus testing and debugging changes #7581's
  done-when line "cant-fail-scan.sh --check exits 0" is therefore false on main.
  • The 10 in test_people_proposals.py assert through a helper: they call self.refused_ops(...)
    (test_people_proposals.py:127-128), which runs assertRaises and assertEqual
    (test_session.py:182-187). These are scanner false positives. The file last changed in
    feat(planning): Claude proposes merges and admits that the host approves with one tap #8961 and test: declare test scopes for the selection audit gaps #9062 (both 2026-10-22).
  • test_the_owner_can_reopen_an_ended_session (test_session.py:3110-3113) calls self.act
    three times and asserts nothing; it fails only if an action raises. That is a real finding.
    Blame puts it in f30e4f09c3, feat(planning): shared interview sessions #7670 (2026-10-18).
• --file measured on the same export: a non-test path (README.md) exits 2; a flagged file
  exits 1 in about 160 ms; a clean test file exits 0.
• Scheduled and dispatch runs take "every diff-scoped gate at its widest"
  (pr-require-checks.yml:17-22, cron :26-27), so a diff-scoped gate also runs whole-tree twice
  a day.
• Timeout rule: every job's timeout is ceil(max(1.5 x p95, 1.1 x max) / 60) minutes, at least 3,
  never above 16, and "a job that outgrows its timeout is made faster or split, not given a longer
  one" (pr-require-checks.yml:128-146; the sentence is at :145-146). A separate job pays off
  only when it shortens the critical path by more than about 15 s of fixed setup
  (:119-120). check-plugins measures 110 / 120 s p95 / max with a 3-minute cap (:136-137,
  job at :1549-1554); check-skills is the longest at 585 / 645 s, 15 minutes (:137-138).
• Test guards: test_guards_enabled is a testing-plugin userConfig option, default false,
  that turns on the test-weaken PreToolUse hook (plugins/testing/README.md:73-77, :178).
  A pluginConfigs value "always lands in user settings" whatever the install scope
  (docs/conventions/plugin-reconfiguration/README.md:44-47), and the team layer
  docs/conventions/testing.yaml sets only e2e_driver and reuse_running_instance
  (testing.yaml:4-5). CI already runs scripts/check-discriminating-test-skips.sh
  (pr-require-checks.yml:1345).
• Mutation testing: /mutation-testing:audit is a model-driven skill, diff-scoped by default
  (plugins/mutation-testing/README.md:11), with no score gate and no whole-repo nightly run by
  design (README.md:33-39). This repository has no .claude/mutation-testing.md; the
  only copies are eval fixtures. No workflow runs mutation testing.

Fix

Part A: scoped scan in the required check.

1. Precondition: bash plugins/testing/skills/audit/scripts/cant-fail-scan.sh --check exits 0 on
   main, or the scheduled run goes red. See Q1.
2. Add a gate step cant_fail_scan to check-plugins through scripts/run-hygiene-gate.sh, as
   the other gate steps do (pr-require-checks.yml:1601-1620). On a pull request or merge group,
   list the changed test files against needs.select-tests.outputs.lane_base (:287) and run
   cant-fail-scan.sh --check --file <path> on each. No changed test file reports the gate not
   applicable, never a silent pass. The loop must tell "not a test file" apart from a real exit 2
   (unreadable input), for example by filtering with the scanner's own adapter claim instead of
   treating every exit 2 as skip. On a schedule or dispatch run, run the whole-tree --check.
3. Size the host job by the formula from measured runs. Whole-tree on a schedule would add about
   82 s to check-plugins: about 192 / 202 s, giving ceil(max(288, 222) / 60) = 5 minutes
   against today's 3. Where it runs is Q2.
4. Pin the new step in the workflow's existing contract tests and add a stay-quiet test: a PR with
   no changed test file and a PR with a clean changed test file both pass the step.

Part B: mutation testing post-merge only. Prerequisites: /mutation-testing:setup apply
writes .claude/mutation-testing.md for this repository, and a lane is chosen (Q4). Then a
post-merge, diff-scoped /mutation-testing:audit run per merge, never a PR check, reports
survivors and blocks nothing.

Open questions (decide before building)

• Q1. How does main reach a clean --check? The 10 helper findings and the 1 real finding
  need different fixes.
  • For the 10: (a) add refused_ops to the Python adapter's assertion calls in
    docs/conventions/testing.yaml. The schema's extend key appends to an adapter's list fields,
    assertion.calls among them (plugins/testing/schemas/testing.schema.json, extend
    description), and the scan reads that layer (cant-fail-scan.sh:145-150). Whether a bare
    method name matches self.refused_ops( is UNVERIFIED until a --file run shows it. (b) Add a
    cant-fail-ok: <reason> annotation to each of the 10 tests.
  • For the 1: add a real assertion on the reopened session's state, never an annotation.
  • Either way, (c) the fixes can land as a sibling issue like testing: fix the two tests the can't-fail scan flags #7597, before this slice.
    Recommendation: (a) plus a real assertion, landed as (c). Basis: judgment. The helper does
    assert, so annotating would exempt tests that can fail.
• Q2. Which job holds the step? Each option conflicts with one of the workflow's rules.
  • (a) check-plugins, timeout re-derived to 5 minutes. This conflicts with "made faster or
    split, not given a longer one" (pr-require-checks.yml:145-146).
  • (b) A new check-tests job, bare bash, which needs 3 minutes by the formula. This conflicts
    with :119-120: at about 192 s, check-plugins stays far below check-skills' 585 s, so a
    separate job would not shorten the critical path.
  • (c) Whole-tree on a schedule only in a new job, and the per-file PR loop (seconds) in
    check-plugins under its current 3 minutes. This adds a job for the schedule alone.
    Recommendation: none; this is the owner's call between two written rules. Basis: the lines
    cited above. The 82 s figure was measured on WSL, not on a hosted runner, so a runner
    measurement should come first.
• Q3. What does "test guards on" mean here? The session hook switch is user-scope only
  (plugin-reconfiguration/README.md:44-47), so a workflow change cannot turn it on. Options:
  (a) drop it from this slice and record it as a user-setup step in the plugin README; (b) add a
  repo-scoped switch to the testing plugin's config cascade (a separate plugin change); (c) read
  "test guards" as the CI checks only (this scan plus check-discriminating-test-skips.sh).
  Recommendation: (c) for this slice, (b) as a follow-up. Basis: judgment, plus the user-scope
  evidence above.
• Q4. Mutation lane and scope. /mutation-testing:audit needs a Claude session, so a
  post-merge run is a model lane under ADR 0049's hardening, with plan-usage cost. Options:
  (a) split Part B into its own slice; (b) keep it here. Recommendation: (a). Basis: judgment;
  Part A is a deterministic gate, Part B is a new lane with its own cost and security review.

Verification

• bash plugins/testing/skills/audit/scripts/cant-fail-scan.sh --check exits 0 on the PR's
  head (exits 1 with 11 gating findings on ba598f0).
• git grep -n 'cant-fail-scan.sh' -- .github/workflows/pr-require-checks.yml returns at least
  one line (none on main).
• The PR's own ci-status run shows the cant_fail_scan gate. A throwaway commit adding a
  def test_x(self): pass test file turns it red, and its revert turns it green; both run links
  are in the PR.
• A PR run with no changed test file reports the gate not applicable, shown in the PR.
• The host job's timeout-minutes equals the formula's result from the measured wall, with the
  numbers in the workflow comment.
• Part B, if kept: .claude/mutation-testing.md exists, and no pull_request or merge_group
  trigger runs /mutation-testing:audit (git grep -n 'mutation-testing:audit' -- .github/workflows
  matches only a post-merge workflow).

Related

• Parent thread: app-control: new plugin plus testing and debugging changes #7581 (app-control, testing and debugging), epic Epic: absorb Lauren Tan's and Matt Pocock's agent workflows (poteto-review) #7555. Source: PLAN Phase 13,
  "Testing" item ("scoped can't-fail scan in the required check with test guards on; mutation
  testing post-merge only") and the Sanity Check "cant-fail-scan.sh --check exits 0".
• testing: fix the two tests the can't-fail scan flags #7597 fixed the two earlier scan findings. feat(testing): flag absence-only and never-called tests in /testing:test-value #8826 added absence-only and never-called checks to
  /testing:test-value.