Skip to content

feat(gates): add consolidated local PR-body preflight check - #1741

Merged
tvna merged 9 commits into
mainfrom
claude/pr-1725-prep-74n4o8
Sep 4, 2026
Merged

tvna merged 9 commits into
mainfrom
claude/pr-1725-prep-74n4o8

Conversation

@tvna

@tvna tvna commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a single consolidated local PR-body preflight check (CLI + PreToolUse hook) so an editing session no longer has to remember which individual PR-body-affecting gate scripts apply to a given body edit before calling create_pull_request/update_pull_request.

Facts

Assumptions

  • Implementation shape (hook + CLI, mirroring the existing skill-audit-disclosure pattern) was confirmed with the repository owner via AskUserQuestion in this session, not re-derived from the issue text alone.

Risk / blast radius

  • New PreToolUse hook can block create_pull_request/update_pull_request calls -- scoped to this repository's own agent-mediated PR-write path, same blast radius class as the existing skill-audit-disclosure hook it mirrors.
  • No existing gate's pass/fail behavior is narrowed or loosened; this PR only adds local-plane coverage for gates that already exist and already run in CI.

Rollback

  • Revert this PR's merge commit; the new hook/script/registry entry are all additive, so a revert cleanly restores current behavior (skill-audit-disclosure hook only, provenance-disclosure/ASCII checks CI-only).

Verification

Acceptance Criteria Map (from issue #1725):

Criterion Interpretation Planned ops Proof method Residual risk
[from #1707] skill-audit-disclosure CI gate rejected a hand-edited PR body When this session rewrote the PR body's own "## Skill audit evidence" section to record independent-review findings, the deterministic-gate-quality disclosure line was phrased "RAN, iteratively, ..." -- a comma immediately after RAN broke that gate's own regex (which requires either bare RAN or RAN followed by whitespace then free text), failing skill-audit-disclosure in CI twice before the wording was corrected locally-verifiable-first. Before calling mcp__github__update_pull_request to change a PR body, run the repository's own relevant PR-body-shape gates locally against the exact draft text first (.github/scripts/gitapex_gate_skill_audit_disclosure.py --body and .github/scripts/gitapex_gate_independent_review_pending.py --body --head-sha at minimum), and require both to exit 0 before submitting the update -- add this as an explicit step in drafting-a-pr-to-merge's own Step 8/9 PR-body-editing guidance. implementing PR adds the check plus a regression test; confirm it fails against a reintroduced instance of the original defect, then passes none identified
[from #1711] provenance-disclosure CI false positive from incomplete local pre-push gate coverage This session's own newly-added PR-body prose (an Execution log bullet reporting that skill-audit-disclosure found the body missing its required section) coincidentally combined provenance-disclosure's own limitation-cue and tool-fingerprint-cue vocabulary in one paragraph, false-triggering the gate. The session had already been running some local gate scripts before pushing PR-body edits (skill-audit-disclosure, ASCII/provenance-marker scans) but had not included gitapex_gate_provenance_disclosure.py in that same pre-push pass, so this specific false-positive-prone pattern was not caught before the push that triggered it. Add a single consolidated local pre-push/pre-update-pull-request command (or hook) that runs the full set of PR-body-affecting local gate scripts together -- skill-audit-disclosure, provenance-disclosure, and the ASCII/provenance-marker scan at minimum -- rather than leaving it to the operating agent's own memory of which individual scripts apply to a given body edit. implementing PR adds the check plus a regression test; confirm it fails against a reintroduced instance of the original defect, then passes The underlying provenance-disclosure heuristic itself is unchanged and already discloses its own known false-positive/false-negative limitations in its module docstring; this gate proposal is about consolidating pre-push coverage, not about narrowing or loosening that heuristic.

Result, per criterion:

  • [from gate-proposal: retro #1702 repair 6: skill-audit-disclosure CI gate rejected a hand-edited PR body #1707]: Implemented -- .github/scripts/gitapex_gate_pr_body_preflight.py's check_skill_audit_disclosure wraps this gate; skills/drafting-a-pr-to-merge/SKILL.md Step 8 names the consolidated command explicitly. tests/test_gitapex_gate_pr_body_preflight.py::test_check_skill_audit_disclosure_reconstructs_1707_regex_break reconstructs the stray-comma defect (fails against it, passes once corrected). This PR body itself was preflighted with the new command before every push, per the new Step 8 guidance.
  • [from gate-proposal: retro #1708 repair 3: provenance-disclosure CI false positive from incomplete local pre-push gate coverage #1711]: Implemented -- .github/scripts/gitapex_gate_pr_body_preflight.py runs all four checks in one command; hooks/check-pr-body-preflight.sh wires it as a PreToolUse hook on create_pull_request/update_pull_request. tests/test_gitapex_gate_pr_body_preflight.py::test_check_provenance_disclosure_reconstructs_1711_false_positive and test_main_exits_nonzero_reconstructing_1711 reconstruct the exact false-positive paragraph shape (fail against it), plus a clean-body pass.

Local verification commands run (all green on the final head commit):

  • uv run --frozen python3 -m pytest tests/test_gitapex_gate_pr_body_preflight.py tests/test_gitapex_gate_pr_body_preflight_properties.py hooks/test_gitapex_check_pr_body_preflight_shell.py -q -- 95 passed (includes 100% line coverage on the new Python module)
  • uv run --frozen python3 -m pytest -q (full suite) -- 8324 passed
  • uv run --frozen python3 .github/scripts/gitapex_gate_local_preflight.py -- all 46 wired gates PASS

Checklist

  • Tests pass locally
  • Docs updated if behavior changed
  • Issue number cited in every commit
  • If this PR adds/modifies a skills/*/SKILL.md, a docs/superpowers/specs/*.md design doc, a security-relevant skill, or a deterministic checker script (skills/*/scripts/*.py, evals/scripts/*.py, .github/scripts/*.py), a ## Skill audit evidence section discloses the required verdicts/waivers (see .github/scripts/gitapex_gate_skill_audit_disclosure.py)
  • If this PR adds a new Kept-edit-log entry to any evals/*/split.md, that entry discloses a Transfer check line (see .github/scripts/gitapex_gate_transfer_check_disclosure.py) -- not applicable, no evals/*/split.md touched
  • If this PR adds or increases a skills/*/SKILL.md's Stop-boundary bullets or named dispatch branches, evals/<skill>/tasks/*.yaml gained at least as many new fixtures (see .github/scripts/gitapex_gate_skill_branch_fixture_coverage.py) -- not applicable, this PR adds a paragraph of prose to an existing step, not a new Stop-boundary bullet or dispatch branch

Skill audit evidence

  • battle-testing-a-skill: WAIVED: skills/drafting-a-pr-to-merge/SKILL.md's frontmatter description: line is unchanged; the diff adds one paragraph of prose inside Step 8 naming a new consolidated local command, with no new Stop-boundary bullet, dispatch branch, or behavior-affecting decision logic for this audit to exercise.
  • evaluating-skill-quality: WAIVED: same reason -- a one-paragraph addition to existing Step 8 prose, not a shape or maturity change to the skill itself.
  • checker-script-adversarial-review: RAN -- .github/scripts/gitapex_gate_pr_body_preflight.py (new checker script) and hooks/check-pr-body-preflight.sh (new hook) were independently reviewed twice by fresh review-persona dispatches against correctness, blast-radius, reuse, convention, and security; see Facts above and the ## Independent review verdict section for the full findings and fixes.
  • deterministic-gate-quality: RAN -- the new pr-body-preflight gate (.gitapex/ssot.json, hooks/check-pr-body-preflight.sh, .github/scripts/gitapex_gate_pr_body_preflight.py) was read against skills/evaluating-deterministic-gate-quality/references/dimensions.md, in particular dimension 15 (fail-closed default on incomplete/malformed input -- the hook fails open only on inconclusive local git state that keeps the preflight script from ever running, and fails closed once the script actually runs but exits unrecognizably, matching hooks/check-pr-skill-audit-disclosure.sh's own documented posture after a review-round fix corrected an initial fail-open inversion) and shape check 6 (explicit, proportionate timeout: SUBPROCESS_TIMEOUT_SECONDS = 5 per sub-check, sized against and empirically verified under the hook's own 30s harness-level timeout in hooks.json, with a separate longer dedicated timeout for the one call site never part of that sequential budget).
  • defeat-test-disclosure: RAN -- the ASCII-only sub-check's own detection regex was deliberately narrowed (dropping the tab exemption) and the property test test_ascii_clean_text_always_passes was confirmed to fail against that narrowed regex before being restored; the two reconstruction tests above (test_check_skill_audit_disclosure_reconstructs_1707_regex_break, test_check_provenance_disclosure_reconstructs_1711_false_positive) are themselves defeat tests built from the two original production defects; the review round added further defeat tests (a reconstructed _isolated-bypass crash, a flag-shaped required-package name, a live-confirmed monkeypatch-staleness reproduction for _run's own timeout parameter), none merely happy-path exercises.

Independent review verdict

  • Verdict: CLEAN
  • Verified commit: 8d6b1fd
  • Outer layer: GitHub Copilot review was requested (request_copilot_review) twice (once against an earlier head commit, once against this one) and posted no response within 30 minutes of either request -- treated as unreachable for this step, per drafting-a-pr-to-merge's own Step 8 disclosure requirement. CodeRabbit auto-skips review on this repository ("fewer than 10 stars"). Neither outer-layer mechanism produced a usable signal for this PR.
  • Inner layer: reviewing-an-artifact's own 5-axis fan-out (correctness, blast-radius, reuse-and-simplification, convention, security) ran twice against this PR's own diff, at low effort -- once against the initial implementation, once against the fix round that followed. Every axis's own candidate findings were independently re-verified (FABRICATED pre-check, direct code read, counterfactual check) before being treated as confirmed; several were personally reproduced (a live monkeypatch demonstration for the _run default-argument staleness bug, a direct timing measurement of git diff -U1000000 against this repository's own largest tracked Markdown files for the timeout-margin question, a direct python3 -c reproduction of the uncaught FileNotFoundError path). All confirmed findings across both rounds were fixed and are covered by new or changed tests; the audit trail below lists every candidate raised, confirmed or not.

Round 1 (initial implementation, 3 confirmed findings, all fixed): no dependency-precondition probe before invoking skill-audit-disclosure's own gate script (risked a misleading FAIL on a missing pydantic); the new hook duplicated skill-audit-disclosure's own verdict already covered by a separate existing hook; the new registry entry did not disclose its own transitive pydantic dependency.

Round 2 (fix round, 5 axes, confirmed findings all fixed):

  • Correctness: run_all_checks aborted the whole aggregate report when one sub-check's own setup failed, contradicting this module's own "always reports the whole set" docstring -- fixed via a new _isolated per-check wrapper.
  • Blast-radius: the hook fell back to the default branch on a stacked PR with no explicit base, dragging the parent branch's own changes into --check-diff's scope (a false-deny risk the sibling hook already guards against) -- fixed via the same base_is_explicit pattern; the hook's own regression fixture for this was itself corrected, since its first version published the parent change to origin/main before forking the stacked branch, which made the wrong- and right-ancestor computations produce the same diff scope regardless of the fix under test.
  • Security: the hook failed OPEN when the preflight script itself crashed without a recognized FAIL line, inverting the sibling hook's own explicit fail-closed policy for the identical situation (PR fix(hooks): fail closed on missing/malformed jq in four PreToolUse gates #1213) -- fixed to deny, matching the sibling.
  • Correctness/blast-radius/security (raised independently by three reviewers): a required-package precondition check omitted the -- argv separator the sibling hook's own identical call already carries (issue gate-proposal-umbrella: environment/repo-state preconditions #1566's own fix) -- fixed, with a defeat test for a flag-shaped package name mirroring the sibling's own.
  • Blast-radius/security/convention: the per-subprocess timeout (120s, then an under-corrected 8s) could still exceed the hook's own 30s harness timeout across up to four sequential calls -- lowered to 5s (20s worst case), empirically verified against this repository's own largest tracked Markdown files to carry roughly 90x headroom over real git diff -U1000000 runtimes; the one call site never part of that sequential chain (the precondition probe) got its own longer, dedicated timeout instead.
  • Reuse: _isolated's own exception scope was narrower than the run_check helper its docstring compared itself to, and a duplicate exception-to-result conversion existed alongside it -- widened to match, and deduplicated into one shared helper; found in the course of that fix, a timeout parameter added to _run using a plain default-argument value silently froze the module constant's value at function-definition time, breaking every test that monkeypatches it afterward -- fixed with a call-time-resolved sentinel, live-confirmed via _run.__defaults__ before and after.
  • Convention: four places (this module's own docstring, its --check-diff help text, the hook's own header comment and runtime notice, and CONTRIBUTING.md) inaccurately claimed omitting --check-diff skips two sub-checks; only skill-audit-disclosure is actually skipped, provenance-disclosure always runs in a body-only mode -- corrected in all four.
  • Blast-radius: the hook's own --check-diff array was expanded unconditionally even when empty, which raises "unbound variable" under set -u on bash older than 4.4 (e.g. macOS's shipped bash 3.2) -- fixed via the same length-guarded build the sibling hook already uses for its own optional array.
  • Also fixed: the .gitapex/ssot.json registry entry's own script array under-listed its cooperating files relative to the sibling entry's own precedent.

Deliberately not fixed (disclosed, not silently dropped): the _run subprocess helper's own duplication with gitapex_gate_local_preflight.py's identical helper, and the hook's own bash-level duplication with hooks/check-pr-skill-audit-disclosure.sh (jq guards, deny(), base-branch resolution, the fail-closed backstop) -- both would require extracting a shared module/library and modifying that second, stable, already-tested file, judged out of this fix round's own narrow scope rather than bundled in.

Unconfirmed concerns (security-tier, reported per policy though not independently reproduced as reachable): a theoretical dash-leading-ref argument-injection shape in build_diff_added_corpus's own git diff invocation, confirmed unreachable in the current call graph (the hook always supplies a computed SHA, never attacker-controlled text) but flagged for a hypothetical future caller of the standalone --check-diff CLI flag; whether this harness's own PreToolUse hook timeout truly discards output on expiry (fail-open) was not independently verified in this environment, cited from this repository's own threat-model-and-authorization.md.

Related Issue

Closes #1725
Refs #1707, #1711

Execution log

  • PlanApproved{run_id: 1725-2026-09-03}
  • TaskStarted{run_id: 1725-2026-09-03, task_id: task-1}
  • TaskCompleted{run_id: 1725-2026-09-03, task_id: task-1, commit_sha: e734ae1}

@tvna
tvna deployed to ruleset-verify September 3, 2026 22:31 — with GitHub Actions Active
@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: 85210779-e0c0-4ca0-8ff7-e1f448e323ac


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.68%. Comparing base (94ca689) to head (8d6b1fd).

Additional details and impacted files
@@           Coverage Diff            @@
##             main    #1741    +/-   ##
========================================
  Coverage   99.67%   99.68%            
========================================
  Files         156      158     +2     
  Lines       25572    25939   +367     
  Branches     3118     3150    +32     
========================================
+ Hits        25490    25857   +367     
  Misses         82       82            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Adds a single command and PreToolUse hook that run every PR-body-
affecting local gate together (skill-audit-disclosure, provenance-
disclosure, ASCII-only, provenance-marker scan) before a
create_pull_request/update_pull_request call, so an editing session no
longer has to remember which individual script applies to a given body
edit.

Closes #1725
Refs #1707, #1711
Adds hypothesis property coverage for check_ascii_only, direct test
coverage for CheckResult.status/_run/run_all_checks, and a pipefail
disclosure on the module docstring's shell-pipe usage example -- closes
the local preflight gaps gitapex_gate_local_preflight.py's own
pre-push run surfaced against the prior commit.

Refs #1725
@tvna
tvna deployed to ruleset-verify September 3, 2026 22:55 — with GitHub Actions Active
CI coverage-combine flagged gitapex_gate_pr_body_preflight.py below
the 90 percent floor gitapex_gate_evals_scripts_coverage.py enforces.
Adds direct tests for the missing-sibling-script error paths, the
check-diff end-to-end entry point, invalid UTF-8 body handling, and a
sub-check timeout.

Refs #1725
Independent review found three issues: the skill-audit-disclosure
sub-check had no dependency precondition probe, matching the fix
hooks/check-pr-skill-audit-disclosure.sh already carries for the
identical pydantic-import risk; the new hook duplicated that same
sub-check verdict alongside the already-wired skill-audit-disclosure
hook on every PR write call; and the new registry entry did not
disclose the transitive pydantic dependency.

Adds a precondition probe, a skip flag so a caller can avoid a
redundant sub-check, wires the hook to skip skill-audit-disclosure
since a dedicated hook already covers it, and declares the
precondition in the registry entry.

Refs #1725
@tvna
tvna deployed to ruleset-verify September 3, 2026 23:13 — with GitHub Actions Active
@tvna tvna removed the branch-plan-executing label Sep 3, 2026 — with Claude
@tvna
tvna marked this pull request as ready for review September 3, 2026 23:30
Fan-out review across correctness/blast-radius/reuse/convention/security
axes against PR #1741's diff surfaced real defects, all fixed here:

- run_all_checks aborted the whole aggregate report when one sub-check's
  own setup failed (e.g. an unresolvable --check-diff ref), contradicting
  this module's own "always reports the whole set" docstring. Every
  sub-check dispatch now goes through a new _isolated helper, matching
  gitapex_gate_local_preflight.py's own per-check isolation.
- check-pr-body-preflight.sh fell back to the default branch on a stacked
  PR with no explicit base, dragging the parent branch's own changes into
  --check-diff's scope -- the same false-deny class hooks/check-pr-skill-
  audit-disclosure.sh's own base_is_explicit gate already exists to
  prevent. Mirrors that fix: narrows to the two body-only sub-checks
  instead of denying against the wrong ancestor.
- The hook failed OPEN when the preflight script itself crashed without a
  recognized FAIL line, inverting the sibling hook's own explicit
  fail-closed policy for the identical situation (PR #1213). Now denies,
  same as the sibling.
- _missing_packages_report called gitapex_check_python_precondition.py
  without the "--" argv separator the sibling hook's own tier-1 call
  already adds for this exact reason (issue #1566) -- a future hyphen-
  leading required-package name would otherwise be read as an option.
- SUBPROCESS_TIMEOUT_SECONDS (120s) exceeded the hook's own wired 30s
  PreToolUse timeout, starving the internal graceful-timeout path of any
  chance to fire before the harness kills the process. Lowered to 8s.

Also: corrected an inaccurate comment claiming every sub-check this hook
runs flags an empty body (verified false for all three); extracted a
small _temp_text_file helper, closing a duplicated mkstemp/write/cleanup
shape between main() and check_provenance_disclosure; completed the
pr-body-preflight registry entry's own script array to list every
cooperating file, matching the skill-audit-disclosure entry's precedent;
added hooks/test_gitapex_check_pr_body_preflight_shell.py (previously no
shell-level coverage at all for this hook's own bash logic).

Deliberately not fixed: the _run() helper's own duplication with
gitapex_gate_local_preflight.py's identical helper. Closing it would mean
extracting a shared module and importing it into that second, stable,
already-tested gate script -- out of this fix round's own narrow scope.

Issue #1725.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Rq64ZNBDoyLYzDdDaE2DG
@tvna
tvna deployed to ruleset-verify September 4, 2026 00:01 — with GitHub Actions Active
A fresh 5-axis review against this fix round's own diff surfaced more
issues, fixed here:

- _isolated's own exception scope (PrBodyPreflightError, TimeoutExpired
  only) was narrower than gitapex_gate_local_preflight.py's own run_check,
  the comparison its docstring drew -- an uncaught OSError (e.g. git
  itself missing from PATH inside build_diff_added_corpus) still crashed
  the whole aggregate run. Widened to match run_check's scope (OSError,
  ValueError, subprocess.SubprocessError too), and deduplicated the
  exception-to-CheckResult conversion (previously written out twice,
  once in _isolated and once in run_all_checks's own diff_added_corpus
  handling) into one shared _error_result helper.
- _run's own new timeout parameter used a plain default-argument value
  (`timeout: float = SUBPROCESS_TIMEOUT_SECONDS`), which freezes that
  module global's value at function-definition time -- silently breaking
  every test (and every real call) that relies on monkeypatching
  SUBPROCESS_TIMEOUT_SECONDS afterward. Live-confirmed via
  _run.__defaults__ staying stale post-monkeypatch. Switched to a
  None-sentinel resolved at call time.
- The prior round's SUBPROCESS_TIMEOUT_SECONDS=8 could still sum to 32s
  across the hook's own up-to-four sequential _run calls, exceeding its
  30s harness timeout. Lowered to 5s (20s worst case); empirically
  verified against this repo's own largest tracked Markdown files that
  5s carries roughly 90x headroom over real git diff -U1000000 runtimes.
  _missing_packages_report's own call (never part of that sequential
  chain -- always --skip'd by the hook) gets its own longer, dedicated
  timeout instead, so it isn't cut short before
  gitapex_check_python_precondition.py's own slower per-package probe.
- check-pr-body-preflight.sh's own --check-diff array was expanded
  unconditionally even when empty, which raises "unbound variable" under
  `set -u` on bash older than 4.4 (e.g. macOS's shipped bash 3.2) --
  hooks/check-pr-skill-audit-disclosure.sh already guards its own
  optional array this way; mirrored here via a length-guarded build.
- Corrected an inaccurate claim, in four places (this module's own
  docstring and --check-diff help text, the hook's own header comment and
  runtime Notice message, CONTRIBUTING.md), that omitting --check-diff
  skips two sub-checks. Only skill-audit-disclosure is actually skipped;
  provenance-disclosure always runs, in a body-only mode. The hook's own
  stacked-PR test fixture was also fixed: it previously published the
  parent branch's change to origin/main before forking the stacked
  branch, which meant the "wrong" and "right" ancestor computations
  produced the same diff scope regardless of the fix under test -- a
  false reassurance. The fixture now proves the risk is real (the
  wrong-ancestor diff genuinely includes the parent's own change) before
  checking the fix.

Also added: a regression test for the "--" argv separator fix (a
flag-shaped required-package name), mirroring the sibling hook's own
defeat test for the identical call site.

Issue #1725.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Rq64ZNBDoyLYzDdDaE2DG
@tvna
tvna deployed to ruleset-verify September 4, 2026 00:23 — with GitHub Actions Active
_publish() lost its only call site when the stacked-PR test fixture was
fixed to stop pre-publishing the parent branch's own change (the fix in
the previous commit) -- left behind as unused code, which codecov/patch
correctly flagged as 4 uncovered lines. Deleted; the fixture's own
docstring comment referencing it is corrected to describe the actual
prior fixture shape instead.

Issue #1725.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Rq64ZNBDoyLYzDdDaE2DG
@tvna
tvna deployed to ruleset-verify September 4, 2026 00:33 — with GitHub Actions Active
@tvna
tvna marked this pull request as draft September 4, 2026 00:46
@tvna
tvna marked this pull request as ready for review September 4, 2026 02:49
@tvna
tvna merged commit 7512778 into main Sep 4, 2026
44 of 45 checks passed
@tvna
tvna deleted the claude/pr-1725-prep-74n4o8 branch September 4, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gate-proposal-umbrella: consolidated local PR-body gate check (skill-audit-disclosure + provenance-disclosure)

2 participants