Repository navigation
test(semantic): close the mutation-testing gaps in the decision core - #490
Merged
Merged
Conversation
Whole-suite mutation is infeasible at ~7100 tests, so this scopes it to the two
modules where a surviving mutant is most alarming: semantic/policy.py, the gate
that decides whether a repair is applied, and guard.py, which enforces
protected-column byte-identity. 19 targeted mutants covering the classes the
brief names -- comparison flips, boolean-operator swaps, constant flips.
before 19 mutants, 12 killed, 7 survived 63.2%
after 19 mutants, 18 killed, 1 survived 94.7%
All seven original survivors were re-run against the entire 7120-test suite and
survived it too, so these were genuine gaps rather than artefacts of a targeted
subset. 94.96% line coverage was hiding them.
Four of the five policy survivors sat in the safety core:
policy bool#4 `bool(raw_payload) and raw_payload == payload(proposed)`
flipped to `or`: any non-empty payload passes, so a proposal
that CHANGES an identifier's content slips through the
carve-out -- exactly what the docstring says it prevents.
policy bool#8 target-column protection `or` -> `and`: protection lost
unless both the declared name and the inferred role match.
policy bool#12 preserve_columns protection `or` -> `and`: same.
policy bool#1
const_bool#2 the non-string early return in _payload_preserving, so a
non-string pair reaches the payload comparison.
guard const_bool#4
protected snapshot copy(deep=True) -> deep=False: the
snapshot verify_protected compares against would share the
frame's buffer, so an in-place edit would change the snapshot
too and the violation would go unnoticed.
None of these were bugs in shipped behaviour -- the code is correct. Nothing was
checking that it stays correct. The 18 tests here do.
The remaining survivor, guard bool#5 (`if left_na or right_na` -> `and`), is an
equivalent mutant and is documented as such rather than chased. It changes only
the one-sided-missing branch of _cell_equal, where the fall-through comparison
returns False anyway -- the same answer the early return gives. Both versions
were run over 289 input pairs (None, nan, NaT, pd.NA, ints, strings, bools,
lists, dicts, ndarray, inf, bytes): zero behavioural differences. A test written
to chase it would assert nothing.
Scope stated plainly: two modules, not the codebase. scoring.py, experts.py,
steps/dtypes.py and fieldcheck.py remain unmeasured and branch coverage is still
false repo-wide. This establishes that the decision gate and the protection
guard are genuinely tested, not that the other 7100 tests are.
Full suite 7138 passed / 0 failed, coverage 94.96%; ruff clean repo-wide.
No library code changed.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kevincostner17
added a commit
that referenced
this pull request
Sep 20, 2026
) (#494) PR #490 reported "19 mutants, 94.7%" for policy.py + guard.py. That was measured over 19 of the module pair's 62 mutation sites, because of an indexing defect in the mutation harness, so it described roughly a third of the module rather than the module. The true figure was 69.4% with 19 survivors. This adds 30 tests and takes it to 62/62 with no survivors. The survivors included the most safety-critical mutants found anywhere in this work: is_id = column in ctx.id_columns or (info is not None and info.identifier_like) flipped to `and` narrows identifier protection so a column detected as identifier-like, but not explicitly listed, silently loses it. The gate's own review-threshold comparison was movable. `format_alignment and _payload_preserving` flipped to `or` would let a non-payload-preserving repair through. In guard.py the row-count check was movable, and the NA comparison could be flipped so the byte-identity guard misses a violation. It also corrects a false claim this file shipped in #490. The docstring asserted that flipping `if left_na or right_na` to `and` was an equivalent mutant, "verified across 289 input pairs ... zero behavioural differences. It cannot be killed." It can. The pool held only ordinary scalars and no cell whose __eq__ returns True against anything; for such a cell the fall-through comparison returns True, so the mutant reports a missing value and a present one as equal and the final identifier-protection layer misses the violation. A 1296-pair differential finds 18 such disagreements. The mutant is now killed and the docstring says what went wrong. That note also cited mutant ids, which are positional and did not survive the harness fix: its `bool#5` is today's `bool#2`. The corrected docstring quotes the source line instead. Tests only; no src change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This is the measurement the programme was missing: 7120 green tests,
branch = false, and no mutant had ever been killed. Line coverage of 94.96% says nothing about whether the assertions would catch a defect.Whole-suite mutation is infeasible at this size, so this scopes it to the two modules where a surviving mutant is most alarming —
semantic/policy.py(the gate deciding whether a repair is applied) andguard.py(protected-column byte-identity). 19 targeted mutants covering comparison flips, boolean-operator swaps and constant flips.Result
All seven original survivors were re-run against the entire 7120-test suite and survived that too — so these were genuine gaps, not artefacts of a targeted subset.
What was untested
Four of the five policy survivors sat in the safety core:
policy bool#4bool(raw_payload) and raw_payload == payload(proposed)→or. Any non-empty payload passes, so a proposal that changes an identifier's content slips through the carve-out — exactly what that docstring says it preventspolicy bool#8or→and: lost unless both declared name and inferred role matchpolicy bool#12preserve_columnsprotectionor→and: samepolicy bool#1,const_bool#2_payload_preservingguard const_bool#4copy(deep=True)→deep=False. The snapshotverify_protectedcompares against would share the frame's buffer, so an in-place edit would change the snapshot too and the violation would go unnoticedNone of these were bugs in shipped behaviour — the code is correct. Nothing was checking that it stays correct. The 18 tests here do.
The remaining survivor is equivalent, and that is proven
guard bool#5(if left_na or right_na→and) changes only the one-sided-missing branch of_cell_equal, where the fall-through comparison returnsFalseanyway — the same answer the early return gives.Both versions were run over 289 input pairs (None, nan, NaT,
pd.NA, ints, strings, bools, lists, dicts, ndarray, inf, bytes): zero behavioural differences. It cannot be killed because it does not change behaviour, so a test chasing it would assert nothing. Documented in the test file so nobody re-derives that.Effective kill rate on non-equivalent mutants: 18/18.
Scope, stated plainly
Two modules, not the codebase.
scoring.py,experts.py,steps/dtypes.pyandfieldcheck.pyremain unmeasured, and branch coverage is stillfalserepo-wide. This establishes that the decision gate and the protection guard are genuinely tested — not that the other 7100 tests are.Verification
ruff check .clean repo-wide.