test: add mutation testing over the pure logic - #58
Merged
Merged
Conversation
Coverage says a line ran, not that anything checked what it did. This changes the code in small ways and reruns the tests: a mutant that survives is a change nobody noticed. Scoped to lib/utils and lib/composables, where it takes three minutes and where a survivor usually means a real gap rather than a change that cannot be observed. Not wired into CI and no threshold: chasing a score produces padded assertions, and the report is for reading. Run once, it put useRotation at 69.61% with twenty-nine survivors, which is code I had just written twenty tests for. Three of those survivors were worth having: The direction was never checked. The stub for rotateOrientation ignored which way it was asked to turn, so changing 'left' to 'right' in the composable passed. For a button whose entire job is to rotate left. `saving` is returned by the composable and no test read it, so three mutants lived there. Observing it needs the request held open, because advancing the timers also drains the promises. An etag arrives quoted either literally or as the escaped entity, and only the literal was exercised. Four tests later it is 78.43% with twenty left, most of which cannot be killed while axios is mocked. plyrTranslations is excluded: it is a table of strings and scoring it measures nothing. Assisted-by: ClaudeCode:claude-opus-5 Signed-off-by: skjnldsv <skjnldsv@protonmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #58 +/- ##
=======================================
Coverage 88.62% 88.62%
=======================================
Files 39 39
Lines 2743 2743
Branches 561 562 +1
=======================================
Hits 2431 2431
Misses 305 305
Partials 7 7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage tells you a line ran. It does not tell you anything checked what the line did. My SVG sanitiser test in #55 was the clean example: it passed with the sanitiser deleted, because a browser never runs script in an
<img>-loaded SVG. I only caught that because I happened to try.Stryker does that systematically. It changes the code in small ways, flips a comparison, deletes a call, swaps a string, and reruns the tests. A mutant that survives is a change to the library that no test noticed.
Scoped to
lib/utilsandlib/composables, which takes about three minutes and is where a survivor usually means a real gap rather than a change nothing could observe.npm run test:mutation.Not wired into CI and no threshold set. Chasing a percentage produces padded assertions; the report is for reading.
What the first run found
useRotation.tscame out at 69.61%, twenty-nine survivors. That is code I wrote two days ago and gave twenty tests to, and was pleased with.Three of those survivors were worth fixing:
The direction was never checked. My stub for
rotateOrientationignored which way it was asked to turn, so changing'left'to'right'in the composable passed the whole suite. For a button whose entire job is to rotate left. The end-to-end test would have caught it, but nothing at unit level did.savingis returned by the composable and no test ever read it, so three mutants lived in it. Observing it needs the request held open, because advancing fake timers also drains the promise chain and the write is over before there is anything to see.An etag comes back quoted either literally or as
", and only the literal form was exercised.Four tests later: 78.43%, twenty survivors. Most of the rest cannot be killed while axios is mocked, for instance the
responseType, and I have left them rather than writing assertions that pretend otherwise.plyrTranslations.tsis excluded. It is a table of strings and scored 1.43%, which measures nothing useful.Where the rest of the code stands
previewUtils.tslivePhotoUtils.tscanDownload.tshistory.tsuseViewerActions.tsusePlyrPlayer.tshistory.tsandusePlyrPlayer.tsare the obvious next places to look. I have not touched them here; this PR is the tool plus the one file I used to find out whether it was worth having.It runs with
inPlace, because the sandbox copy breaks vite's filesystem allowlist on the?rawicon imports. Stryker backs the files up and restores them, andgit statusis clean after a run.382 unit tests.
👾 This pull request was assisted by Claude Code, commits carry an
Assisted-bytrailer.