Skip to content

test: add mutation testing over the pure logic - #58

Merged
skjnldsv merged 1 commit into
mainfrom
test/mutation-testing
Sep 21, 2026
Merged

skjnldsv merged 1 commit into
mainfrom
test/mutation-testing

Conversation

@skjnldsv

Copy link
Copy Markdown
Contributor

Coverage tells you a line ran. It does not tell you anything checked what the line did. My SVG sanitiser test in #55 was the clean example: it passed with the sanitiser deleted, because a browser never runs script in an <img>-loaded SVG. I only caught that because I happened to try.

Stryker does that systematically. It changes the code in small ways, flips a comparison, deletes a call, swaps a string, and reruns the tests. A mutant that survives is a change to the library that no test noticed.

Scoped to lib/utils and lib/composables, which takes about three minutes and is where a survivor usually means a real gap rather than a change nothing could observe. npm run test:mutation.

Not wired into CI and no threshold set. Chasing a percentage produces padded assertions; the report is for reading.

What the first run found

useRotation.ts came out at 69.61%, twenty-nine survivors. That is code I wrote two days ago and gave twenty tests to, and was pleased with.

Three of those survivors were worth fixing:

The direction was never checked. My stub for rotateOrientation ignored which way it was asked to turn, so changing 'left' to 'right' in the composable passed the whole suite. For a button whose entire job is to rotate left. The end-to-end test would have caught it, but nothing at unit level did.

saving is returned by the composable and no test ever read it, so three mutants lived in it. Observing it needs the request held open, because advancing fake timers also drains the promise chain and the write is over before there is anything to see.

An etag comes back quoted either literally or as &quot;, and only the literal form was exercised.

Four tests later: 78.43%, twenty survivors. Most of the rest cannot be killed while axios is mocked, for instance the responseType, and I have left them rather than writing assertions that pretend otherwise.

plyrTranslations.ts is excluded. It is a table of strings and scored 1.43%, which measures nothing useful.

Where the rest of the code stands

file score survivors
previewUtils.ts 97.5% 1
livePhotoUtils.ts 90.6% 2
canDownload.ts 86.7% 6
history.ts 73.3% 31
useViewerActions.ts 74.3% 15
usePlyrPlayer.ts 43.0% 44

history.ts and usePlyrPlayer.ts are the obvious next places to look. I have not touched them here; this PR is the tool plus the one file I used to find out whether it was worth having.

It runs with inPlace, because the sandbox copy breaks vite's filesystem allowlist on the ?raw icon imports. Stryker backs the files up and restores them, and git status is clean after a run.

382 unit tests.

👾 This pull request was assisted by Claude Code, commits carry an Assisted-by trailer.

Coverage says a line ran, not that anything checked what it did. This
changes the code in small ways and reruns the tests: a mutant that
survives is a change nobody noticed.

Scoped to lib/utils and lib/composables, where it takes three minutes and
where a survivor usually means a real gap rather than a change that
cannot be observed. Not wired into CI and no threshold: chasing a score
produces padded assertions, and the report is for reading.

Run once, it put useRotation at 69.61% with twenty-nine survivors, which
is code I had just written twenty tests for. Three of those survivors
were worth having:

The direction was never checked. The stub for rotateOrientation ignored
which way it was asked to turn, so changing 'left' to 'right' in the
composable passed. For a button whose entire job is to rotate left.

`saving` is returned by the composable and no test read it, so three
mutants lived there. Observing it needs the request held open, because
advancing the timers also drains the promises.

An etag arrives quoted either literally or as the escaped entity, and
only the literal was exercised.

Four tests later it is 78.43% with twenty left, most of which cannot be
killed while axios is mocked. plyrTranslations is excluded: it is a table
of strings and scoring it measures nothing.

Assisted-by: ClaudeCode:claude-opus-5
Signed-off-by: skjnldsv <skjnldsv@protonmail.com>
@skjnldsv skjnldsv added status: review Waiting for reviews type: tests 🧪 Tests related issue or changes AI assisted labels Sep 21, 2026
@codecov

codecov Bot commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.62%. Comparing base (e2333e0) to head (70c298c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #58   +/-   ##
=======================================
  Coverage   88.62%   88.62%           
=======================================
  Files          39       39           
  Lines        2743     2743           
  Branches      561      562    +1     
=======================================
  Hits         2431     2431           
  Misses        305      305           
  Partials        7        7           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@skjnldsv
skjnldsv merged commit 894058f into main Sep 21, 2026
18 checks passed
@skjnldsv
skjnldsv deleted the test/mutation-testing branch September 21, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI assisted status: review Waiting for reviews type: tests 🧪 Tests related issue or changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant