Skip to content

Add mutation testing via Infection - #899

Merged
Firehed merged 10 commits into
mainfrom
mutation-testing
Nov 3, 2026
Merged

Firehed merged 10 commits into
mainfrom
mutation-testing

Conversation

@Firehed

@Firehed Firehed commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Problem

Review keeps asking one question by hand: does any test notice if a changed line is wrong?
Each reviewer tries the changes they think of, so each round finds a few more, and a review never comes back clean.
Coverage does not answer it either: a covered line can still be wrong without any test failing.

Change

Adds Infection, a mutation-testing tool, as a check on pull requests.
It makes small changes (flipping a comparison, removing a call, swapping a value) to each line of src/ that a branch changed, reruns the unit tests that cover the line, and reports every change that nothing notices.
The list is complete for its kinds of change and the same on every run.

  • infection/infection as a dev dependency, pinned to an exact version so local and CI runs match (composer.lock is gitignored). Its Composer plugin, which only installs Infection extensions, is not allowed.
  • infection.json5: mutates src/, runs the unit suite only, counts a change PHPStan rejects as caught (PHPStan already gates every change), and fails unless every change is caught. A run with nothing to mutate passes.
  • composer mutation: mutates the lines changed since origin/main. It fails locally exactly as it does in CI.
  • A Mutation Testing job on pull requests runs composer mutation with the same setup as the unit job, assertions enabled. Without assertions, changes the tests catch through an assert() would survive only in CI. Surviving changes appear as annotations on the diff, and the job fails while any survive; an annotation on a passing check was easy to miss.- CONTRIBUTING.md: how to run it, and what to do with a surviving change. A change no input could detect is resolved by rewriting the code so the change cannot be made, or else by an approved Class::method ignore entry. Tests name the behavior they check, not the change they catch.

Known limits

  • It does not detect two separate, well-tested places that decide the same fact. Review still asks that question.
  • A pull request that edits a weakly tested line will see that line's existing gaps.
  • Code that runs inside a data provider is not counted as covered.
  • A local run compares the working tree, uncommitted edits included, with origin/main; CI tests the pull request merged with main.

🤖 Generated with Claude Code

Firehed and others added 5 commits November 3, 2026 13:20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.71%. Comparing base (44f6a1f) to head (0effc6d).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #899   +/-   ##
=========================================
  Coverage     99.71%   99.71%           
  Complexity     1970     1970           
=========================================
  Files           147      147           
  Lines          4488     4488           
=========================================
  Hits           4475     4475           
  Misses           13       13           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Firehed and others added 5 commits November 3, 2026 13:46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Firehed Firehed changed the title Report surviving mutations on changed lines Add mutation testing via Infection Nov 3, 2026
@Firehed
Firehed marked this pull request as ready for review November 3, 2026 22:08
@Firehed
Firehed merged commit a555696 into main Nov 3, 2026
9 checks passed
@Firehed
Firehed deleted the mutation-testing branch November 3, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant