From 13f8f3f9f514f5cf0bd3f6752f82f84d0df05d30 Mon Sep 17 00:00:00 2001 From: leynos Date: Mon, 20 Jul 2026 16:33:57 +0200 Subject: [PATCH 1/4] Add mutation-testing-rollout skill Distil the estate-wide mutation-testing programme into a reusable skill: per-repository adoption of the shared `mutation-cargo`/ `mutation-mutmut` callers, baseline verification under the mutation runner's conditions (nextest isolation, doctest blind spots, the mutmut sandbox), contract-test conventions (shape-only SHA pinning), survivor triage discipline, run sweeps, and the hard-won estate lessons (unpinned-linter drift, stale checkouts, false survivors from companion crates). `install-skills` picks the new directory up automatically; no registration is required. --- skills/mutation-testing-rollout/SKILL.md | 163 +++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 skills/mutation-testing-rollout/SKILL.md diff --git a/skills/mutation-testing-rollout/SKILL.md b/skills/mutation-testing-rollout/SKILL.md new file mode 100644 index 0000000..250a049 --- /dev/null +++ b/skills/mutation-testing-rollout/SKILL.md @@ -0,0 +1,163 @@ +--- +description: Roll out scheduled, informational mutation testing across an estate of repositories, triage the results into issues and killing tests, and keep the callers documented and drift-free. +--- + +# Mutation-testing estate rollout and triage + +This skill captures the process used to roll mutation testing out across +github.com/leynos/\* — approximately forty Rust and Python repositories — +and to operate it afterwards: harvesting survivors into issues, fixing +broken baselines, writing killing tests, and documenting the machinery. +Apply it when adopting mutation testing in a new repository, when triaging +the output of scheduled runs, or when running an estate-wide sweep. + +## Architecture + +One pair of reusable workflows lives in `leynos/shared-actions`: + +- `mutation-cargo.yml` (Rust, [cargo-mutants](https://mutants.rs/)) and +- `mutation-mutmut.yml` (Python, [mutmut](https://mutmut.readthedocs.io/)). + +Each consumer repository carries only a thin caller, +`.github/workflows/mutation-testing.yml`, pinned to a shared-actions commit +SHA that Dependabot owns. The runs are **informational only**: they never +gate pull requests. A daily cron fires a change-scoped run (mutating only +files changed within a detection window, so quiet days are cheap no-ops); +manual dispatch runs the full mutation set, fanned out across shards. +Survivors surface in the job summary and in downloadable +`mutation-report-*` artefacts. + +The mutation run is a measurement of the test suite, not of the code. A +surviving mutant is a test gap; the goal is a falling survivor count, not +zero. + +## Per-repository adoption recipe + +1. **Verify eligibility.** The repository needs a healthy test suite that + passes in CI. No tests means every mutant survives as noise — defer + until tests exist. Python repositories must be uv-managed (a hard + requirement of the mutmut workflow). +2. **Verify the baseline under the mutation runner's conditions, not + CI's.** This is the single most common failure. Hazards found in + practice: + - Tests that pass under nextest's process isolation but fail under + plain `cargo test` (shared global state, log capture, a claimed + global tracing dispatcher). Either fix the test or pass + `--test-tool=nextest`. + - CI running `--all-targets` never runs doctests, but cargo-mutants + does. Broken doctests (including in vendored code) break the + baseline. Fix them, mark them `ignore`, or set `doctest = false`. + - mutmut copies sources into a `mutants/` sandbox that omits + `.github/`; any test reading workflow files needs a + `pytest.mark.skipif(not WORKFLOW_PATH.exists(), ...)` guard (or put + `.github/` in `also_copy`, or exclude the test from selection). + - Hygiene tests that walk the file system see the workflow-source + checkout; enumerate `git ls-files` instead. + - Stateful test dependencies (embedded PostgreSQL) break on re-run; + pin credentials/versions via the workflow's `setup-commands` input. +3. **Write the caller.** Copy a proven caller (wireframe for Rust, + cmd-mox for Python) and adapt: + - `paths`: change-detection globs matching the repository's source + layout. + - `exclude-globs`: example code, test scaffolding, fixture crates, and + `cfg(test)`-only companion files — anything whose survivors are + noise. Note that a module-root glob (`src/foo.rs`) does not match the + directory's submodules; add `src/foo/**` too. + - `extra-args`: match the CI baseline exactly (`--all-features`, + `--test-workspace=true` for workspaces). A mismatch reports + feature-gated code as untested. For workspaces where bare feature + names fail to resolve against the shard-scoped baseline, ensure every + mutated member defines the baseline feature set (forwarding features + if necessary). + - `setup-commands`: install anything `.cargo/config.toml` assumes + (clang/lld/mold), pin environmental state. + - `shard-count`: size from the first full run; a shard that brushes the + 360-minute job ceiling needs more shards. + - For mutmut: configure `[tool.mutmut]` in `pyproject.toml` + (`source_paths`, test selection, `do_not_mutate` for + subprocess-executed or generated code, `also_copy` for fixtures the + tests need). mutmut hard-codes importable layouts (`./`, `src/`, + `source/`); a `pkg/` layout needs a committed `src -> pkg` symlink + and `module-prefix-strip`. +4. **Pin and stagger.** Pin the `uses:` line to a full 40-character + shared-actions commit SHA. Give each repository a distinct cron slot; + mutation runs are heavy and synchronized schedules concentrate load. +5. **Add a contract test.** A PyYAML-based test + (`tests/workflow_contracts/` or `tests/test_workflow_contract.py`) + pins the caller's shape so drift fails the pull request: the `uses:` + path and that its ref is a 40-hex commit SHA (match the *shape* with a + regex — never hard-code the SHA value, or every Dependabot bump becomes + a manual chore), least-privilege permissions (`contents: read`, + `id-token: write`; empty workflow default), concurrency that queues per + ref without cancelling, the schedule-plus-plain-dispatch triggers, and + the `with:` block contents. +6. **Dispatch a full run** and calibrate: verify the summary populates, + record survivor counts and runtime, and adjust shards/timeouts. +7. **Document.** Add a "Mutation-testing workflow contract tests" section + to `docs/developers-guide.md`: why the workflow exists (informational + only), the two run modes, the rationale for each `with:` input, the + SHA-pin policy, the exact aspects the contract test validates, and the + local run command (a `make test-workflow-contracts` target where the + repository has a Makefile). Create the guide (and file an issue for a + fuller one) where it is missing. + +## Triage discipline + +Harvest survivors from `mutants.out/outcomes.json` (cargo; +`scenario.Mutant.file`, `span.start.line`, `name`) or the mutmut summary. +File **one issue per coherent area** (module or concern), not per mutant, +with counts, representative survivors (`file:line` plus the mutation), and +a proposed next step. Then work the issues as draft PRs: + +- **Real test gap** — write a killing test. Prefer exact assertions; + "equivalent" verdicts frequently fall to a sharper test. +- **Equivalent mutant** — suppress with justification: + `#[cfg_attr(test, mutants::skip)]` (Rust, via the `mutants` crate) or + `# pragma: no mutate` (Python). Keep skips rare and justified; prefer a + killing test wherever the mutation is observable. Beware: mutmut honours + the pragma only on single logical lines — trailing pragmas on + continuation lines of multi-line statements are ignored, so restructure + to a single line instead. +- **Dead code** — delete it; this is one of mutation testing's best + yields. +- **Untestable boundary** — document and move on. + +Link every kill site to its tracking issue (issue refs in PR titles and +doc comments at the test), so the provenance survives the merge. + +Red-green verification traps: stale `.pyc` bytecode can fake a red-green +cycle (`PYTHONDONTWRITEBYTECODE=1`); an ambient environment variable can +turn a gate into a no-op — check what the gate actually ran. + +## Operating the estate: run sweeps + +Periodically sweep all repositories' runs (`gh api +repos///actions/workflows`, then the runs endpoint). Classify +by duration: sub-minute runs are usually the skip path or an early +failure; long runs are real. Remember success does not mean clean — read +the job summary for survivor counts. For failures, read +`gh run view --log-failed`, identify the root cause, and file one +issue per distinct cause (not per run); recurring identical quick failures +are one issue listing occurrences. Deduplicate against existing issues +first — comment on a persisting area rather than re-filing it. If the +cause lies in the reusable workflow, file it on the shared-actions +repository. + +## Estate lessons (hard-won) + +- **Unpinned linters break baselines estate-wide.** `uv tool install ty`, + unpinned ruff preview rules, and rolling-release dylint each turned CI + red across multiple repositories at once. Pin linter versions. +- **Stale local checkouts lie.** Before concluding a file is absent, + verify against the remote default branch (`gh api + repos///contents/...`), and branch from a freshly fetched + `origin/`. +- **Companion and fixture crates produce false survivors.** Where a + crate's coverage lives in a sibling, scope the run or record that its + survivor table is advisory. +- **Keep a living rollout ledger** (per-repository status, hazards, + issue/PR numbers). The tracker outlives any one session and is the only + reliable memory across an estate this size. +- Spelling gates (typos, en-GB Oxford `-ize`) apply to generated prose + too; expect `summarising`/`serialises` to be rejected in favour of + `summarizing`/`serializes`. From 5123deb4fe65a4cead9f511730bab54f3787141b Mon Sep 17 00:00:00 2001 From: leynos Date: Mon, 20 Jul 2026 16:41:26 +0200 Subject: [PATCH 2/4] Fold the 14-day sweep's lessons into the sweep guidance The first estate sweep disproved wall-clock duration as a triage signal (runner-queue wait inflates 15-second no-ops to an hour) and surfaced two silent-suppression traps and the change-detection gate's coverage suppression. Key sweeps on `mutation_detect_has_changes` and the mutants job conclusion instead, verify suppressions by scoped re-run, and dispatch a full run after merging a baseline fix rather than waiting for the schedule. --- skills/mutation-testing-rollout/SKILL.md | 40 +++++++++++++++++------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/skills/mutation-testing-rollout/SKILL.md b/skills/mutation-testing-rollout/SKILL.md index 250a049..db19467 100644 --- a/skills/mutation-testing-rollout/SKILL.md +++ b/skills/mutation-testing-rollout/SKILL.md @@ -114,10 +114,15 @@ a proposed next step. Then work the issues as draft PRs: - **Equivalent mutant** — suppress with justification: `#[cfg_attr(test, mutants::skip)]` (Rust, via the `mutants` crate) or `# pragma: no mutate` (Python). Keep skips rare and justified; prefer a - killing test wherever the mutation is observable. Beware: mutmut honours - the pragma only on single logical lines — trailing pragmas on - continuation lines of multi-line statements are ignored, so restructure - to a single line instead. + killing test wherever the mutation is observable. Beware two silent + no-op traps: mutmut honours the pragma only on single logical lines + (trailing pragmas on continuation lines of multi-line statements are + ignored — restructure to a single line), and cargo-mutants' skip + attribute does not cover plain `const` items or binary-expression + initializers (only functions, impls, and certain expression kinds) — a + skip placed there changes nothing and the mutant resurfaces next run. + Verify a suppression actually suppressed by re-running scoped + (`cargo mutants --file ` / `mutmut run `). - **Dead code** — delete it; this is one of mutation testing's best yields. - **Untestable boundary** — document and move on. @@ -132,16 +137,29 @@ turn a gate into a no-op — check what the gate actually ran. ## Operating the estate: run sweeps Periodically sweep all repositories' runs (`gh api -repos///actions/workflows`, then the runs endpoint). Classify -by duration: sub-minute runs are usually the skip path or an early -failure; long runs are real. Remember success does not mean clean — read -the job summary for survivor counts. For failures, read +repos///actions/workflows`, then the runs endpoint). Do NOT +classify by wall-clock duration — runner-queue wait routinely inflates a +15-second no-op to an hour. The reliable signals are: +`mutation_detect_has_changes` in the detect step's log (the mutmut +workflow runs its gate *inside* the single mutants job, so a no-op still +reports a green job), and the `mutants` job's conclusion (the cargo +workflow skips it visibly). Remember success does not mean clean — a real +run's survivors live in the job summary — and conversely a "failure" can +mask a fully completed run (a teardown-step regression once failed +otherwise-green runs estate-wide). For failures, read `gh run view --log-failed`, identify the root cause, and file one issue per distinct cause (not per run); recurring identical quick failures are one issue listing occurrences. Deduplicate against existing issues -first — comment on a persisting area rather than re-filing it. If the -cause lies in the reusable workflow, file it on the shared-actions -repository. +first — comment on a persisting area rather than re-filing it (and when +concurrent sweep agents race, reconcile their duplicate filings onto one +canonical issue). If the cause lies in the reusable workflow, file it on +the shared-actions repository. + +Be aware the change-detection gate suppresses scheduled coverage on quiet +repositories: a repo can report weeks of green runs while executing zero +mutants, leaving a broken baseline undetected. After merging a +baseline fix, trigger a full `workflow_dispatch` to prove it end-to-end +and produce a survivor dataset — do not wait for the schedule. ## Estate lessons (hard-won) From 9e7076fe49836f8126f7abf9523f2a447291f891 Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 10 Sep 2026 21:53:14 +0200 Subject: [PATCH 3/4] Address the mutation-testing-rollout review round MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Act on four review findings and on two problems the rebase onto `main` exposed. The caller recipe prescribed the wireframe Rust caller without a manifest prerequisite. `mutation-cargo.yml` exposes no `manifest-path`/ `working-directory` input, and a full `workflow_dispatch` always fans out a repository-root cargo-mutants target, so a workspace manifest in a subdirectory — cuprum's `rust/Cargo.toml`, for example — yielded shards that fail on every manual run. Record the root-manifest requirement in both the eligibility step and the caller recipe. `PYTHONDONTWRITEBYTECODE=1` only stops new bytecode being written; CPython still imports caches already on disk. Direct the reader to delete existing `__pycache__` directories and `.pyc` files before setting it. The sweep read each repository's workflows and runs with an unpaginated `gh api` call, so a truncated sweep silently misclassified. Paginate both, and record the verified `gh` behaviour: both endpoints answer with an envelope, `--slurp` yields one outer array of those envelopes while `--paginate --jq` alone prints one document per page, and `--slurp` rejects `--jq` and `--template`. Correct the subject-verb agreement in the eligibility bullet. `main` landed mandatory manifest validation after this branch opened, so this was the only skill without a `name` field and `make lint` would have failed. Add it, and document the skill in the users' guide, which the user-facing documentation check requires. `make spelling` rejected the two illustrative `-ise` tokens in the estate-lessons bullet. The original "typos clean" note was a false green: the spelling gate only sees git-tracked files, and the manifest was still untracked when it was first checked. Reword the examples rather than allowlist them. Co-Authored-By: Claude Code --- docs/users-guide.md | 14 ++++++++ skills/mutation-testing-rollout/SKILL.md | 43 ++++++++++++++++++------ 2 files changed, 46 insertions(+), 11 deletions(-) diff --git a/docs/users-guide.md b/docs/users-guide.md index ad2fc4e..0e5bccf 100644 --- a/docs/users-guide.md +++ b/docs/users-guide.md @@ -80,6 +80,20 @@ and [`cs rules-config`](../skills/codescene-cli/references/command-reference.md# Use [`codescene-health-rules`](../skills/codescene-health-rules/SKILL.md) to configure CodeScene rule weights, thresholds, and source directives. +## Mutation-testing rollout + +Use [`mutation-testing-rollout`](../skills/mutation-testing-rollout/SKILL.md) +to roll out scheduled, informational mutation testing across an estate of +repositories and to operate it afterwards; the runs never gate pull requests. +Invoke it to adopt mutation testing in a new repository, to triage the output +of scheduled runs, or to run an estate-wide sweep. + +The skill documents the shared `mutation-cargo.yml` and `mutation-mutmut.yml` +reusable workflows behind thin, SHA-pinned callers, the adoption recipe and +its baseline hazards, the caller contract test, survivor triage, and run +sweeps. Run `install-skills` to copy it into `${HOME}/.codex/skills` and +`${HOME}/.claude/skills`; the `rust-entrypoint` home phase runs it too. + ## Shared spelling tools Run `make spelling` in this checkout to generate and validate the estate-wide diff --git a/skills/mutation-testing-rollout/SKILL.md b/skills/mutation-testing-rollout/SKILL.md index db19467..1acb269 100644 --- a/skills/mutation-testing-rollout/SKILL.md +++ b/skills/mutation-testing-rollout/SKILL.md @@ -1,4 +1,5 @@ --- +name: mutation-testing-rollout description: Roll out scheduled, informational mutation testing across an estate of repositories, triage the results into issues and killing tests, and keep the callers documented and drift-free. --- @@ -34,9 +35,10 @@ zero. ## Per-repository adoption recipe 1. **Verify eligibility.** The repository needs a healthy test suite that - passes in CI. No tests means every mutant survives as noise — defer + passes in CI. No tests mean every mutant survives as noise — defer until tests exist. Python repositories must be uv-managed (a hard - requirement of the mutmut workflow). + requirement of the mutmut workflow), and Rust repositories must carry + their workspace manifest at the repository root (see step 3). 2. **Verify the baseline under the mutation runner's conditions, not CI's.** This is the single most common failure. Hazards found in practice: @@ -56,7 +58,15 @@ zero. - Stateful test dependencies (embedded PostgreSQL) break on re-run; pin credentials/versions via the workflow's `setup-commands` input. 3. **Write the caller.** Copy a proven caller (wireframe for Rust, - cmd-mox for Python) and adapt: + cmd-mox for Python) and adapt. The Rust caller assumes the workspace + manifest sits at the repository root, and the shared workflow cannot + be told otherwise: `mutation-cargo.yml` exposes no + `manifest-path`/`working-directory` input, and a full + `workflow_dispatch` always fans out a repository-root cargo-mutants + target. A Rust workspace whose manifest lives in a subdirectory (for + example `rust/Cargo.toml`) therefore has no working caller — enrol + only the root-manifest side and leave the subdirectory workspace out + until the shared workflow can target it: - `paths`: change-detection globs matching the repository's source layout. - `exclude-globs`: example code, test scaffolding, fixture crates, and @@ -131,15 +141,26 @@ Link every kill site to its tracking issue (issue refs in PR titles and doc comments at the test), so the provenance survives the merge. Red-green verification traps: stale `.pyc` bytecode can fake a red-green -cycle (`PYTHONDONTWRITEBYTECODE=1`); an ambient environment variable can -turn a gate into a no-op — check what the gate actually ran. +cycle, and `PYTHONDONTWRITEBYTECODE=1` only stops new bytecode being +written — CPython still imports caches already on disk. Before trusting a +cycle, delete existing bytecode (`find . -type d -name __pycache__ +-exec rm -rf {} +`, plus any stray `*.pyc`), set +`PYTHONDONTWRITEBYTECODE=1`, and re-run. An ambient environment variable +can also turn a gate into a no-op — check what the gate actually ran. ## Operating the estate: run sweeps -Periodically sweep all repositories' runs (`gh api -repos///actions/workflows`, then the runs endpoint). Do NOT -classify by wall-clock duration — runner-queue wait routinely inflates a -15-second no-op to an hour. The reliable signals are: +Periodically sweep all repositories' runs (`gh api --paginate +repos///actions/workflows`, then the +`repos///actions/runs` endpoint). Page both to exhaustion — +an unpaginated read returns only the first 30 items, so a truncated sweep +silently misclassifies. Both endpoints answer with an envelope +(`.workflows`/`.workflow_runs`), so aggregate every page before +classifying: `--slurp` yields one outer array of those envelopes, while +`--paginate --jq` alone prints one document per page rather than one +array. `--slurp` rejects `--jq` and `--template`, so fold the pages in a +second pass. Do NOT classify by wall-clock duration — runner-queue wait +routinely inflates a 15-second no-op to an hour. The reliable signals are: `mutation_detect_has_changes` in the detect step's log (the mutmut workflow runs its gate *inside* the single mutants job, so a no-op still reports a green job), and the `mutants` job's conclusion (the cargo @@ -177,5 +198,5 @@ and produce a survivor dataset — do not wait for the schedule. issue/PR numbers). The tracker outlives any one session and is the only reliable memory across an estate this size. - Spelling gates (typos, en-GB Oxford `-ize`) apply to generated prose - too; expect `summarising`/`serialises` to be rejected in favour of - `summarizing`/`serializes`. + too; expect `-ise` endings to be rejected, so write `summarize` and + `serialize`. From 396332e7e1ac9d1f6548b34dfe296acfa236a92e Mon Sep 17 00:00:00 2001 From: leynos Date: Fri, 11 Sep 2026 00:18:58 +0200 Subject: [PATCH 4/4] Scope the mutmut re-run example by module glob MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mutmut run` takes a mutant-name filter — an exact key or a Unix `fnmatch` glob matched against keys built from the import path — not a filesystem path. A path matches no key and aborts the run with `AssertionError: Filtered for specific mutants, but nothing matches` (mutmut 3.6.0, `__main__.py:885`), so the scoped re-run the triage guidance prescribes could never have verified a suppression. Use the `.*` form that mutmut's own TUI builds for its retest module action, quoted so the shell cannot expand the glob first, and note that the glob is over the import path rather than a file path. `cargo mutants --file ` is genuinely file-based and is unchanged. Co-Authored-By: Claude Code --- skills/mutation-testing-rollout/SKILL.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/skills/mutation-testing-rollout/SKILL.md b/skills/mutation-testing-rollout/SKILL.md index 1acb269..c8f8e20 100644 --- a/skills/mutation-testing-rollout/SKILL.md +++ b/skills/mutation-testing-rollout/SKILL.md @@ -132,7 +132,8 @@ a proposed next step. Then work the issues as draft PRs: initializers (only functions, impls, and certain expression kinds) — a skip placed there changes nothing and the mutant resurfaces next run. Verify a suppression actually suppressed by re-running scoped - (`cargo mutants --file ` / `mutmut run `). + (`cargo mutants --file ` / `mutmut run '.*'`; mutmut's + argument is a mutant-name glob over the import path, not a file path). - **Dead code** — delete it; this is one of mutation testing's best yields. - **Untestable boundary** — document and move on.