feat(ci): catch vacuous regression tests via mutation (#1799)

.gitapex/ssot.json

@@ -2184,6 +2184,37 @@
         {"kind": "file-glob", "ref": "docs/gitapex/plans/*.md"},
         {"kind": "workflow-event", "ref": "plans-traceability-gate.yml:pull_request"}
       ]
+    },
+    {
+      "id": "defeat-test-mutation-coverage",
+      "kind": "script",
+      "script": [
+        ".github/scripts/gitapex_gate_defeat_test_mutation_coverage.py",
+        ".github/scripts/gitapex_run_base_diff.py",
+        ".github/scripts/_gitapex_base_ref.py",
+        ".github/workflows/defeat-test-mutation-coverage-gate.yml"
+      ],
+      "rule": "A regex alternation branch (a top-level '|'-joined alternative, at paren-depth 0 and outside a [...] character class, inside a plain string-literal pattern argument to re.compile()/re.match()/re.search()/re.fullmatch(), the .compile receiver resolved through the file's own 'import re' statements and .match/.search/.fullmatch receiver-agnostic on any receiver), a key of an ast.Dict assigned to a module-level ast.Name target (CONST = {...}), or a str/int/float/bool literal that is a direct List/Tuple/Set element or a non-module-level-dict Dict entry's value, built unconditionally (no enclosing If/Try/TryStar/For/AsyncFor/While) inside a function body in skills/*/scripts/*.py, .github/scripts/*.py or hooks/*.py (test_*.py and conftest.py excluded), newly added or changed by a PR diff, must be actually caught -- not merely mentioned -- by this same diff's own paired tests/test_<source-stem>.py or tests/test_<source-stem>_properties.py: the element is spliced out of a temporary, byte-exact mutated copy of the source file, a real pytest subprocess is run against exactly the paired test file(s) this diff itself touches, and the original bytes are restored regardless of outcome. A paired suite that still exits 0 against the mutated source is a defeat-test-mutation-gap finding; a source element with no diff-touched paired test file at all is not graded. Scope is the diff, not the repository. An inline '# defeat-test-mutation-coverage: WAIVED: <reason>' comment waives findings on that exact line. Exits 2 rather than pass silently on a malformed diff, an in-scope or paired-test file that cannot be read or parsed, or a pytest subprocess invocation that itself errors for a reason other than 'tests ran and passed/failed' (e.g. a collection error). See the script's own module docstring for the three element categories' disclosed known misses and non-goals.",
+      "planes": ["ci", "local"],
+      "local_invocation": ["uv", "run", "--frozen", "python3", ".github/scripts/gitapex_gate_defeat_test_mutation_coverage.py"],
+      "local_stdin": ["uv", "run", "--frozen", "python3", ".github/scripts/gitapex_run_base_diff.py", "--", "*.py"],
+      "trigger": ".github/workflows/defeat-test-mutation-coverage-gate.yml on pull_request, plus tests/test_gitapex_gate_defeat_test_mutation_coverage.py and tests/test_gitapex_gate_defeat_test_mutation_coverage_properties.py inside the pytest step of .github/workflows/test.yml",
+      "policy_refs": [],
+      "cluster": "test-integrity",
+      "tracking_issue": 1799,
+      "status": "active",
+      "supersedes": null,
+      "bypass_review_status": "not-yet-reviewed",
+      "target": [
+        {"kind": "file-glob", "ref": "skills/*/scripts/*.py"},
+        {"kind": "file-glob", "ref": ".github/scripts/*.py"},
+        {"kind": "file-glob", "ref": "hooks/*.py"},
+        {"kind": "file-glob", "ref": "tests/test_*.py"},
+        {"kind": "cross-registry-consistency", "ref": "new/changed regex alternation branches, module-level dict entries, and unconditionally-emitted literals in skills/*/scripts/*.py, .github/scripts/*.py, hooks/*.py <-> a real pytest-subprocess mutation-kill in the co-located tests/test_<stem>.py or tests/test_<stem>_properties.py this same diff also touches"},
+        {"kind": "workflow-event", "ref": "defeat-test-mutation-coverage-gate.yml:pull_request"},
+        {"kind": "workflow-event", "ref": "test.yml:pull_request"},
+        {"kind": "workflow-event", "ref": "test.yml:push"}
+      ]
     }
   ],
   "clusters": {

.github/scripts/gitapex_gate_local_preflight.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """One consolidated local pre-push / pre-PR-open gate runner (issue #876).
 
-This repository enforces 87 registered deterministic gates. Before this
+This repository enforces 88 registered deterministic gates. Before this
 script existed, roughly half of them had a perfectly good working-tree-only
 invocation and yet ran *only* as separate CI jobs, so an agent preparing a
 PR discovered gaps one CI job at a time on an already-open PR -- push, wait,
@@ -107,7 +107,7 @@
   (issue #890), which closes the "configured here but never actually
   installed" half; nothing closes the ``--no-verify`` half. CI remains the
   authoritative merge gate for every gate carrying a ``ci`` plane -- true
-  for 48 of the 50 wired gates. ``behind-base`` (issue #985) and
+  for 49 of the 51 wired gates. ``behind-base`` (issue #985) and
   ``real-checkout-git-write`` (issue #991) are the two exceptions: each
   carries only ``local``, so for those two gates specifically this
   pre-push hook -- bypassable the same way as any other -- is the *only*
@@ -119,7 +119,7 @@
   ``jsonschema`` -- a real, non-stdlib dependency, contrary to an earlier
   revision of this paragraph's own "the runner itself needs no
   dependencies" claim. A bare system ``python3`` with no ``jsonschema``
-  installed crashed the whole runner on import before any of the 50 wired
+  installed crashed the whole runner on import before any of the 51 wired
   gates got a chance to run individually, so CONTRIBUTING.md's standalone
   example and the pre-push hook's own ``entry`` both now invoke it as ``uv
   run --frozen python3`` too, the same pin every wired gate's own argv
@@ -207,10 +207,19 @@
 # own _GROUP_TIMEOUT_SECONDS = 600 -- so that one gate's own theoretical
 # worst case is ~4200 s, not 600 s. A ceiling matching that would be useless
 # as a hang guard (80 minutes of a silent pre-push), so this is a judgment
-# call in the other direction. For scale: a warm run of all 50 wired gates
-# combined measures roughly 24 s end to end (issue #1512's network-
-# exception-set-drift gate; different hardware than the figures
-# below, see this paragraph's own closing parenthetical -- the
+# call in the other direction. For scale: a warm run of all 51 wired gates
+# combined measures roughly 47 s end to end (issue #1965's own
+# skill-contract-drift gate and issue #1799's own defeat-test-mutation-
+# coverage gate landed on parallel branches and both bumped the 49-gate
+# baseline to their own independent 50-gate counts before this merge
+# combined them into 51; different hardware than the figures below in any
+# case, see this paragraph's own closing parenthetical -- the prior
+# 50-gate set (defeat-test-mutation-coverage) measured roughly 49 s end to
+# end, unlike every other wired gate, which is pure AST inspection, it
+# spawns a real pytest subprocess per graded element and self-grades its
+# own regex/dict/literal elements against its own paired tests; the prior
+# 49-gate set (network-exception-set-drift, issue #1512) measured roughly
+# 24 s, the
 # prior 48-gate set measured roughly 24 s, the
 # prior 47-gate set measured roughly 22 s, the
 # prior 45-gate set measured roughly 14 s, the

.github/workflows/defeat-test-mutation-coverage-gate.yml

@@ -0,0 +1,99 @@
+# Issue #1799 (six consolidated rows across #1733, #1734, #1735, #1736,
+# #1990, #1991). A new or materially changed regex alternation branch,
+# module-level dict entry, or unconditionally-emitted literal added by a PR
+# diff to one of this repository's own checker scripts must actually be
+# caught -- not merely mentioned -- by this same diff's own paired test: see
+# .github/scripts/gitapex_gate_defeat_test_mutation_coverage.py's own module
+# docstring for the three graded element categories, the mutation mechanism
+# (a real `pytest` subprocess run against a temporarily mutated copy of the
+# source file), the motivating gaps, and the disclosed scope limits and known
+# misses/non-goals. This job runs that scan against exactly the in-scope
+# files a diff actually adds trigger-shaped lines to.
+#
+# Deliberately no `paths:` filter -- drift gate:
+# tests/test_gitapex_gate_defeat_test_mutation_coverage.py's own
+# test_the_workflow_has_no_paths_filter. See that test's docstring for why
+# (GitHub's own skipped-vs-never-ran distinction for a required check).
+name: Defeat-test mutation-coverage gate
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  defeat-test-mutation-coverage:
+    runs-on: ubuntu-latest
+    # Generous relative to the other AST-only gates' own 5-minute ceiling
+    # (they spawn no subprocess at all): this gate spawns one real `pytest`
+    # subprocess per graded element -- see the gate script's own module
+    # docstring, "Invocation shape, runtime/CI cost disclosed rather than
+    # hidden" -- so its own job needs real headroom rather than a ceiling
+    # copied from a gate with a fundamentally different cost shape.
+    timeout-minutes: 20
+    permissions:
+      contents: read
+    steps:
+      # `ref:`/`fetch-depth: '0'` below is this gate's own diff-post-image
+      # checkout pin -- drift gate:
+      # tests/test_gitapex_gate_defeat_test_mutation_coverage.py's own
+      # test_the_workflow_checks_out_the_head_sha_with_full_history. See
+      # that test's docstring for why (this gate correlates diff-derived
+      # line numbers with tree content, the same requirement both sibling
+      # gates' own workflows state for themselves and for the identical
+      # reason -- this gate additionally reads and temporarily overwrites the
+      # graded source file's own on-disk content, and runs the corresponding
+      # tests/*.py files' own tree content at the same head, not only reads
+      # it).
+      #
+      # The accepted cost, stated rather than discovered later: this runs
+      # the PR branch's own copy of the gate, so a branch that forked before
+      # a hardening of the gate landed on main is graded by the older rules.
+      # Branch protection's "Require branches to be up to date" closes that,
+      # and it is a GitHub-side setting no in-repo file can record.
+      - name: Harden runner + checkout
+        uses: tvna/gitapex/.github/actions/harden-checkout@2f62b5648552a0f800b1b85e75ec108a7016dd02
+        with:
+          fetch-depth: '0'
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9  # v9.0.0
+        with:
+          enable-cache: true
+
+      # git merge-base below -- drift gate:
+      # tests/test_gitapex_gate_defeat_test_mutation_coverage.py's own
+      # test_the_workflow_uses_merge_base_not_base_sha.
+      #
+      # The two `git diff` flags below (--no-renames, -c
+      # core.quotePath=false) -- drift gate:
+      # tests/test_gitapex_gate_defeat_test_mutation_coverage.py's own
+      # test_the_workflow_passes_the_two_flags_the_gate_depends_on. See that
+      # test's docstring for why each matters.
+      #
+      # Issue #1035: every `.github/scripts/*.py` invocation goes through
+      # `uv run` regardless of whether the script needs a third-party
+      # dependency. This gate carries a real `pydantic` import (its own
+      # --root validation) and spawns its own `pytest` subprocesses via the
+      # same interpreter `uv run` resolves, so `uv run` is load-bearing here
+      # too -- both for the gate's own dependencies and for the venv its own
+      # mutation subprocesses run pytest under.
+      - name: Mutation-test the elements this diff adds
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+          if ! merge_base=$(git merge-base "$BASE_SHA" "$HEAD_SHA"); then
+            echo "::error::git merge-base failed between $BASE_SHA and $HEAD_SHA" >&2
+            exit 1
+          fi
+          git -c core.quotePath=false diff -U0 --no-renames "$merge_base" "$HEAD_SHA" -- '*.py' \
+            | uv run --frozen python3 .github/scripts/gitapex_gate_defeat_test_mutation_coverage.py

.pre-commit-config.yaml

@@ -157,9 +157,13 @@ repos:
       # already-open PR. This runs all of them in one pass at pre-push --
       # the last local moment before a gap becomes a CI round-trip, and
       # deliberately not pre-commit: it grades committed state (HEAD vs
-      # origin/main), not a staged index, and at ~24 s
-      # warm end to end for all 50 wired gates it is too slow to sit on every
-      # single commit.
+      # origin/main), not a staged index, and at ~47 s
+      # warm end to end for all 51 wired gates (issue #1799's own
+      # defeat-test-mutation-coverage gate roughly doubled the prior ~24 s
+      # baseline on its own, being the first wired gate that spawns a real
+      # pytest subprocess per graded element rather than pure AST
+      # inspection; issue #1965's own skill-contract-drift gate landed on a
+      # parallel branch) it is too slow to sit on every single commit.
       #
       # The wired set is not listed here: the runner discovers it from
       # .gitapex/ssot.json's `local` plane, so a newly wired gate needs no
@@ -174,6 +178,26 @@ repos:
       # enforcement for the ordinary path, not a command a contributor has
       # to remember to type.
       #
+      # Issue #1799's own defeat-test-mutation-coverage gate is also the
+      # first wired gate whose own detection mechanism writes to disk --
+      # every other wired gate only reads and reasons about the AST. It
+      # temporarily overwrites an in-scope source file's own real bytes on
+      # this machine (not a copy) for the duration of one pytest subprocess
+      # per graded element, restoring the original bytes in a `finally`
+      # block regardless of outcome; found by an independent adversarial
+      # review (issue #1799, PR #2000). Gates run sequentially here (see
+      # gitapex_gate_local_preflight.py's own docstring), so no other wired
+      # gate ever reads a file mid-mutation. If the restore write itself
+      # fails (disk full, permissions), that gate's own raised ScanError
+      # names `git checkout -- <path>` as the recovery command -- which
+      # discards whatever uncommitted edit was sitting in that file at the
+      # time, including the very change you were about to push. If this
+      # process is instead killed before the restore write ever runs
+      # (SIGKILL, OOM), Python never executes the `finally` block that
+      # would raise that ScanError -- there is no error, no message, and
+      # no automatic notice at all; the file is silently left holding
+      # mutated bytes on disk until a `git status`/`git diff` catches it.
+      #
       # `uv run --frozen python3`, not a bare `python3`: the runner imports
       # _gitapex_schema_validation.py, which needs jsonschema -- not
       # guaranteed present on a bare system python3 (issue #1485). Each

CONTRIBUTING.md

@@ -125,10 +125,18 @@ used to be discovered one red check at a time on an already-open PR.
 
 The same `uv run prek install -t pre-commit -t pre-push -t commit-msg` above also installs
 a **pre-push** hook that runs every gate with a working-tree-only form in
-one pass, before the push leaves your machine. A warm run of all 50 wired
-gates measures roughly 24 seconds end to end (issue #1512's network-
-exception-set-drift gate; different hardware than the figures below --
-the prior 48-gate set measured roughly 24 seconds, the
+one pass, before the push leaves your machine. A warm run of all 51 wired
+gates measures roughly 47 seconds end to end (issue #1799's own
+defeat-test-mutation-coverage gate: unlike every other wired gate, which is
+pure AST inspection, it spawns a real `pytest` subprocess per graded
+element and self-grades its own regex/dict/literal elements against its own
+paired tests, roughly doubling the prior warm-run baseline on its own --
+different hardware than the figures below in any case; issue #1965's own
+skill-contract-drift gate landed on a parallel branch --
+the prior 50-gate set (defeat-test-mutation-coverage) measured roughly 49
+seconds, the prior 49-gate set (network-exception-set-drift, issue #1512)
+measured roughly 24 seconds, the
+prior 48-gate set measured roughly 24 seconds, the
 prior 47-gate set measured roughly 22 seconds, the
 prior 45-gate set measured roughly 14 seconds, the
 prior 44-gate set measured roughly 15 seconds, the 43-gate set before that
@@ -168,7 +176,7 @@ it up, then confirm both shims with the check in the previous section.
 The runner itself also resolves through `uv` (issue #1485: it imports
 `_gitapex_schema_validation.py`, which needs `jsonschema` -- a real,
 non-stdlib dependency a bare system `python3` is not guaranteed to have),
-and so do all 50 wired gates (the same `uv run` pins CI uses). Without `uv`
+and so do all 51 wired gates (the same `uv run` pins CI uses). Without `uv`
 on PATH every one of them reports `FAIL ... failed to run` -- that is one
 missing tool, not a whole broken wired set.