Add mutation testing as part of ci pipeline.#39
Open
bernata wants to merge 4 commits into
Open
Conversation
Or run manually: npx stryker run
Contributor
|
I like the idea, just wondering how to operationalize it. If it runs silently and I have to manually look for the report, it'll eventually be ignored. Thoughts? |
wolpert
added a commit
to bernata/pk-auth
that referenced
this pull request
Jun 8, 2026
StrykerJS mutation testing — added by Andrew Bernat (@bernata) in PR codeheadsystems#39 (codeheadsystems#39) — reported a 65.1% mutation score for the browser SDK and pinpointed real test gaps that plain line coverage hid: code that ran but had no assertion pinning its behaviour, so a mutated version still passed. This adds the missing assertions. Why this matters: these guards, header rules, and error-mapping branches are the SDK's public contract with the server. A test that exercises them without asserting on them gives false confidence — exactly what mutation testing flags. Credit to Andrew for wiring up the tooling that surfaced these. Gaps closed (each new case kills a previously-surviving mutant): - results.ts: had ZERO coverage (0% score). New results.test.ts pins both branches of isCeremonySuccess / isCeremonyFailure and the success narrowing. - http.ts: assert the `accept` header is always sent; content-type/body are omitted on body-less requests; JSON `null`/primitive bodies leave PkAuthHttpError.data undefined (the && vs || object check); and an absent getToken yields the friendly validation error, not a raw TypeError. - refresh.ts: assert the negative branch of both type guards, the default /auth/refresh path, and that a non-JSON 401 body maps to "unknown" instead of throwing (the e.data?.detail optional-chaining branch). - admin.ts: cover the two previously-untested methods (completeEmailVerification, startPhoneVerification) and add a matrix that asserts every admin call carries the Bearer token, so the `authenticated` flag can no longer be mutated to false undetected. Note: a handful of remaining base64url survivors are equivalent mutants (out-of-bounds typed-array writes are ignored, atob is lenient, the padding regex anchor is moot for btoa output) and are intentionally left as-is. All 67 browser-SDK tests pass; tsc --noEmit clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wolpert
added a commit
that referenced
this pull request
Jun 8, 2026
The updated StrykerJS report (PR #39, @bernata) lifted the SDK from 65.1% to 79.3% after the first hardening pass, leaving ceremonies.ts at 43.2% as the dominant gap: 36 of its mutants were no-coverage. The existing suite drove only the register() happy path, so authenticate(), the credential create/get helpers, the cancellation branches, and the navigator.credentials guard were never executed. These flows ARE unit-testable — CeremonyOptions.credentials injects a fake CredentialsContainer — so this drives them end to end: - authenticate(): start -> get -> finish, token returned, POST verbs, and the start-body username ?? null default; - conditional mediation set only when conditional=true (and absent otherwise); - register() start/finish body contract: displayName ?? username, label ?? null, challenge null, and both steps POSTed (kills the surviving "POST"/default and logical-operator mutants); - create/get cancellation: a null credential rejects with the cancelled error; - the navigator.credentials guard throws its clear message (jsdom has no navigator.credentials), killing the guard's conditional/string mutants. Also adds a base64url DataView/subarray case that kills the toUint8Array ArrayBufferView dispatch mutant. The few remaining base64url survivors are equivalent mutants (lenient atob, ignored out-of-bounds writes, the padding regex anchor) and are left as-is, as are the navigator typeof-flips that only differ in a non-browser environment. All 77 browser-SDK tests pass; tsc --noEmit clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
Yes. This is a problem [i.e. making it operationally useful]. Here is what I'd like to do:
I think the same problem exists for code coverage [i.e. jacoco report is some artifact uploaded on the workflow run]. I'd like to do the same thing with SonarQube free for open source to create saved coverage reports tracked over time and a dash of sast. After that, I think i can write some test code. Let me know what you think of this plan and we can sort out next steps. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Or run manually: npx stryker run
Sample report:
An example of a specific mutant of product code that survived all tests of ceremonies [red is original code, green is what got replaced; i.e. mutant, but all the tests passed]
