feat(gates): add consolidated local PR-body preflight check - #1741
Merged
Merged
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1741 +/- ##
========================================
Coverage 99.67% 99.68%
========================================
Files 156 158 +2
Lines 25572 25939 +367
Branches 3118 3150 +32
========================================
+ Hits 25490 25857 +367
Misses 82 82 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Adds a single command and PreToolUse hook that run every PR-body- affecting local gate together (skill-audit-disclosure, provenance- disclosure, ASCII-only, provenance-marker scan) before a create_pull_request/update_pull_request call, so an editing session no longer has to remember which individual script applies to a given body edit. Closes #1725 Refs #1707, #1711
Adds hypothesis property coverage for check_ascii_only, direct test coverage for CheckResult.status/_run/run_all_checks, and a pipefail disclosure on the module docstring's shell-pipe usage example -- closes the local preflight gaps gitapex_gate_local_preflight.py's own pre-push run surfaced against the prior commit. Refs #1725
CI coverage-combine flagged gitapex_gate_pr_body_preflight.py below the 90 percent floor gitapex_gate_evals_scripts_coverage.py enforces. Adds direct tests for the missing-sibling-script error paths, the check-diff end-to-end entry point, invalid UTF-8 body handling, and a sub-check timeout. Refs #1725
Independent review found three issues: the skill-audit-disclosure sub-check had no dependency precondition probe, matching the fix hooks/check-pr-skill-audit-disclosure.sh already carries for the identical pydantic-import risk; the new hook duplicated that same sub-check verdict alongside the already-wired skill-audit-disclosure hook on every PR write call; and the new registry entry did not disclose the transitive pydantic dependency. Adds a precondition probe, a skip flag so a caller can avoid a redundant sub-check, wires the hook to skip skill-audit-disclosure since a dedicated hook already covers it, and declares the precondition in the registry entry. Refs #1725
Fan-out review across correctness/blast-radius/reuse/convention/security axes against PR #1741's diff surfaced real defects, all fixed here: - run_all_checks aborted the whole aggregate report when one sub-check's own setup failed (e.g. an unresolvable --check-diff ref), contradicting this module's own "always reports the whole set" docstring. Every sub-check dispatch now goes through a new _isolated helper, matching gitapex_gate_local_preflight.py's own per-check isolation. - check-pr-body-preflight.sh fell back to the default branch on a stacked PR with no explicit base, dragging the parent branch's own changes into --check-diff's scope -- the same false-deny class hooks/check-pr-skill- audit-disclosure.sh's own base_is_explicit gate already exists to prevent. Mirrors that fix: narrows to the two body-only sub-checks instead of denying against the wrong ancestor. - The hook failed OPEN when the preflight script itself crashed without a recognized FAIL line, inverting the sibling hook's own explicit fail-closed policy for the identical situation (PR #1213). Now denies, same as the sibling. - _missing_packages_report called gitapex_check_python_precondition.py without the "--" argv separator the sibling hook's own tier-1 call already adds for this exact reason (issue #1566) -- a future hyphen- leading required-package name would otherwise be read as an option. - SUBPROCESS_TIMEOUT_SECONDS (120s) exceeded the hook's own wired 30s PreToolUse timeout, starving the internal graceful-timeout path of any chance to fire before the harness kills the process. Lowered to 8s. Also: corrected an inaccurate comment claiming every sub-check this hook runs flags an empty body (verified false for all three); extracted a small _temp_text_file helper, closing a duplicated mkstemp/write/cleanup shape between main() and check_provenance_disclosure; completed the pr-body-preflight registry entry's own script array to list every cooperating file, matching the skill-audit-disclosure entry's precedent; added hooks/test_gitapex_check_pr_body_preflight_shell.py (previously no shell-level coverage at all for this hook's own bash logic). Deliberately not fixed: the _run() helper's own duplication with gitapex_gate_local_preflight.py's identical helper. Closing it would mean extracting a shared module and importing it into that second, stable, already-tested gate script -- out of this fix round's own narrow scope. Issue #1725. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Rq64ZNBDoyLYzDdDaE2DG
A fresh 5-axis review against this fix round's own diff surfaced more issues, fixed here: - _isolated's own exception scope (PrBodyPreflightError, TimeoutExpired only) was narrower than gitapex_gate_local_preflight.py's own run_check, the comparison its docstring drew -- an uncaught OSError (e.g. git itself missing from PATH inside build_diff_added_corpus) still crashed the whole aggregate run. Widened to match run_check's scope (OSError, ValueError, subprocess.SubprocessError too), and deduplicated the exception-to-CheckResult conversion (previously written out twice, once in _isolated and once in run_all_checks's own diff_added_corpus handling) into one shared _error_result helper. - _run's own new timeout parameter used a plain default-argument value (`timeout: float = SUBPROCESS_TIMEOUT_SECONDS`), which freezes that module global's value at function-definition time -- silently breaking every test (and every real call) that relies on monkeypatching SUBPROCESS_TIMEOUT_SECONDS afterward. Live-confirmed via _run.__defaults__ staying stale post-monkeypatch. Switched to a None-sentinel resolved at call time. - The prior round's SUBPROCESS_TIMEOUT_SECONDS=8 could still sum to 32s across the hook's own up-to-four sequential _run calls, exceeding its 30s harness timeout. Lowered to 5s (20s worst case); empirically verified against this repo's own largest tracked Markdown files that 5s carries roughly 90x headroom over real git diff -U1000000 runtimes. _missing_packages_report's own call (never part of that sequential chain -- always --skip'd by the hook) gets its own longer, dedicated timeout instead, so it isn't cut short before gitapex_check_python_precondition.py's own slower per-package probe. - check-pr-body-preflight.sh's own --check-diff array was expanded unconditionally even when empty, which raises "unbound variable" under `set -u` on bash older than 4.4 (e.g. macOS's shipped bash 3.2) -- hooks/check-pr-skill-audit-disclosure.sh already guards its own optional array this way; mirrored here via a length-guarded build. - Corrected an inaccurate claim, in four places (this module's own docstring and --check-diff help text, the hook's own header comment and runtime Notice message, CONTRIBUTING.md), that omitting --check-diff skips two sub-checks. Only skill-audit-disclosure is actually skipped; provenance-disclosure always runs, in a body-only mode. The hook's own stacked-PR test fixture was also fixed: it previously published the parent branch's change to origin/main before forking the stacked branch, which meant the "wrong" and "right" ancestor computations produced the same diff scope regardless of the fix under test -- a false reassurance. The fixture now proves the risk is real (the wrong-ancestor diff genuinely includes the parent's own change) before checking the fix. Also added: a regression test for the "--" argv separator fix (a flag-shaped required-package name), mirroring the sibling hook's own defeat test for the identical call site. Issue #1725. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Rq64ZNBDoyLYzDdDaE2DG
_publish() lost its only call site when the stacked-PR test fixture was fixed to stop pre-publishing the parent branch's own change (the fix in the previous commit) -- left behind as unused code, which codecov/patch correctly flagged as 4 uncovered lines. Deleted; the fixture's own docstring comment referencing it is corrected to describe the actual prior fixture shape instead. Issue #1725. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Rq64ZNBDoyLYzDdDaE2DG
tvna
marked this pull request as draft
September 4, 2026 00:46
tvna
marked this pull request as ready for review
September 4, 2026 02:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a single consolidated local PR-body preflight check (CLI + PreToolUse hook) so an editing session no longer has to remember which individual PR-body-affecting gate scripts apply to a given body edit before calling
create_pull_request/update_pull_request.Facts
skill-audit-disclosurealready has a PreToolUse hook (hooks/check-pr-skill-audit-disclosure.sh) on bothcreate_pull_requestandupdate_pull_request.provenance-disclosure(.gitapex/ssot.jsongateprovenance-disclosure) has no PreToolUse hook today -- CI-only.skills/outward-artifact-preflight/scripts/gitapex_scan_provenance.py) are documented only as a manual checklist (skills/outward-artifact-preflight/SKILL.md), never wired as an automated gate.review-personafan-out (correctness, blast-radius, reuse-and-simplification, convention, security): once against the initial implementation (3 confirmed findings), once against the fix round that followed (many more, spanning a real correctness bug, a security-tier fail-open/fail-closed inversion, and a stacked-PR false-deny risk). All confirmed findings from both rounds were fixed; see the## Independent review verdictsection below for the full breakdown and the audit trail of what was and was not fixed.Assumptions
AskUserQuestionin this session, not re-derived from the issue text alone.Risk / blast radius
create_pull_request/update_pull_requestcalls -- scoped to this repository's own agent-mediated PR-write path, same blast radius class as the existing skill-audit-disclosure hook it mirrors.Rollback
Verification
Acceptance Criteria Map (from issue #1725):
Result, per criterion:
.github/scripts/gitapex_gate_pr_body_preflight.py'scheck_skill_audit_disclosurewraps this gate;skills/drafting-a-pr-to-merge/SKILL.mdStep 8 names the consolidated command explicitly.tests/test_gitapex_gate_pr_body_preflight.py::test_check_skill_audit_disclosure_reconstructs_1707_regex_breakreconstructs the stray-comma defect (fails against it, passes once corrected). This PR body itself was preflighted with the new command before every push, per the new Step 8 guidance..github/scripts/gitapex_gate_pr_body_preflight.pyruns all four checks in one command;hooks/check-pr-body-preflight.shwires it as a PreToolUse hook oncreate_pull_request/update_pull_request.tests/test_gitapex_gate_pr_body_preflight.py::test_check_provenance_disclosure_reconstructs_1711_false_positiveandtest_main_exits_nonzero_reconstructing_1711reconstruct the exact false-positive paragraph shape (fail against it), plus a clean-body pass.Local verification commands run (all green on the final head commit):
uv run --frozen python3 -m pytest tests/test_gitapex_gate_pr_body_preflight.py tests/test_gitapex_gate_pr_body_preflight_properties.py hooks/test_gitapex_check_pr_body_preflight_shell.py -q-- 95 passed (includes 100% line coverage on the new Python module)uv run --frozen python3 -m pytest -q(full suite) -- 8324 passeduv run --frozen python3 .github/scripts/gitapex_gate_local_preflight.py-- all 46 wired gates PASSChecklist
skills/*/SKILL.md, adocs/superpowers/specs/*.mddesign doc, a security-relevant skill, or a deterministic checker script (skills/*/scripts/*.py,evals/scripts/*.py,.github/scripts/*.py), a## Skill audit evidencesection discloses the required verdicts/waivers (see.github/scripts/gitapex_gate_skill_audit_disclosure.py)evals/*/split.md, that entry discloses a Transfer check line (see.github/scripts/gitapex_gate_transfer_check_disclosure.py) -- not applicable, noevals/*/split.mdtouchedskills/*/SKILL.md's Stop-boundary bullets or named dispatch branches,evals/<skill>/tasks/*.yamlgained at least as many new fixtures (see.github/scripts/gitapex_gate_skill_branch_fixture_coverage.py) -- not applicable, this PR adds a paragraph of prose to an existing step, not a new Stop-boundary bullet or dispatch branchSkill audit evidence
skills/drafting-a-pr-to-merge/SKILL.md's frontmatterdescription:line is unchanged; the diff adds one paragraph of prose inside Step 8 naming a new consolidated local command, with no new Stop-boundary bullet, dispatch branch, or behavior-affecting decision logic for this audit to exercise..github/scripts/gitapex_gate_pr_body_preflight.py(new checker script) andhooks/check-pr-body-preflight.sh(new hook) were independently reviewed twice by freshreview-personadispatches against correctness, blast-radius, reuse, convention, and security; see Facts above and the## Independent review verdictsection for the full findings and fixes.pr-body-preflightgate (.gitapex/ssot.json,hooks/check-pr-body-preflight.sh,.github/scripts/gitapex_gate_pr_body_preflight.py) was read againstskills/evaluating-deterministic-gate-quality/references/dimensions.md, in particular dimension 15 (fail-closed default on incomplete/malformed input -- the hook fails open only on inconclusive local git state that keeps the preflight script from ever running, and fails closed once the script actually runs but exits unrecognizably, matchinghooks/check-pr-skill-audit-disclosure.sh's own documented posture after a review-round fix corrected an initial fail-open inversion) and shape check 6 (explicit, proportionate timeout:SUBPROCESS_TIMEOUT_SECONDS = 5per sub-check, sized against and empirically verified under the hook's own 30s harness-level timeout inhooks.json, with a separate longer dedicated timeout for the one call site never part of that sequential budget).test_ascii_clean_text_always_passeswas confirmed to fail against that narrowed regex before being restored; the two reconstruction tests above (test_check_skill_audit_disclosure_reconstructs_1707_regex_break,test_check_provenance_disclosure_reconstructs_1711_false_positive) are themselves defeat tests built from the two original production defects; the review round added further defeat tests (a reconstructed_isolated-bypass crash, a flag-shaped required-package name, a live-confirmed monkeypatch-staleness reproduction for_run's own timeout parameter), none merely happy-path exercises.Independent review verdict
request_copilot_review) twice (once against an earlier head commit, once against this one) and posted no response within 30 minutes of either request -- treated as unreachable for this step, per drafting-a-pr-to-merge's own Step 8 disclosure requirement. CodeRabbit auto-skips review on this repository ("fewer than 10 stars"). Neither outer-layer mechanism produced a usable signal for this PR.reviewing-an-artifact's own 5-axis fan-out (correctness, blast-radius, reuse-and-simplification, convention, security) ran twice against this PR's own diff, at low effort -- once against the initial implementation, once against the fix round that followed. Every axis's own candidate findings were independently re-verified (FABRICATED pre-check, direct code read, counterfactual check) before being treated as confirmed; several were personally reproduced (a live monkeypatch demonstration for the_rundefault-argument staleness bug, a direct timing measurement ofgit diff -U1000000against this repository's own largest tracked Markdown files for the timeout-margin question, a directpython3 -creproduction of the uncaughtFileNotFoundErrorpath). All confirmed findings across both rounds were fixed and are covered by new or changed tests; the audit trail below lists every candidate raised, confirmed or not.Round 1 (initial implementation, 3 confirmed findings, all fixed): no dependency-precondition probe before invoking skill-audit-disclosure's own gate script (risked a misleading FAIL on a missing
pydantic); the new hook duplicated skill-audit-disclosure's own verdict already covered by a separate existing hook; the new registry entry did not disclose its own transitivepydanticdependency.Round 2 (fix round, 5 axes, confirmed findings all fixed):
run_all_checksaborted the whole aggregate report when one sub-check's own setup failed, contradicting this module's own "always reports the whole set" docstring -- fixed via a new_isolatedper-check wrapper.base, dragging the parent branch's own changes into--check-diff's scope (a false-deny risk the sibling hook already guards against) -- fixed via the samebase_is_explicitpattern; the hook's own regression fixture for this was itself corrected, since its first version published the parent change toorigin/mainbefore forking the stacked branch, which made the wrong- and right-ancestor computations produce the same diff scope regardless of the fix under test.FAILline, inverting the sibling hook's own explicit fail-closed policy for the identical situation (PR fix(hooks): fail closed on missing/malformed jq in four PreToolUse gates #1213) -- fixed to deny, matching the sibling.--argv separator the sibling hook's own identical call already carries (issue gate-proposal-umbrella: environment/repo-state preconditions #1566's own fix) -- fixed, with a defeat test for a flag-shaped package name mirroring the sibling's own.git diff -U1000000runtimes; the one call site never part of that sequential chain (the precondition probe) got its own longer, dedicated timeout instead._isolated's own exception scope was narrower than therun_checkhelper its docstring compared itself to, and a duplicate exception-to-result conversion existed alongside it -- widened to match, and deduplicated into one shared helper; found in the course of that fix, atimeoutparameter added to_runusing a plain default-argument value silently froze the module constant's value at function-definition time, breaking every test that monkeypatches it afterward -- fixed with a call-time-resolved sentinel, live-confirmed via_run.__defaults__before and after.--check-diffhelp text, the hook's own header comment and runtime notice, and CONTRIBUTING.md) inaccurately claimed omitting--check-diffskips two sub-checks; only skill-audit-disclosure is actually skipped, provenance-disclosure always runs in a body-only mode -- corrected in all four.--check-diffarray was expanded unconditionally even when empty, which raises "unbound variable" underset -uon bash older than 4.4 (e.g. macOS's shipped bash 3.2) -- fixed via the same length-guarded build the sibling hook already uses for its own optional array..gitapex/ssot.jsonregistry entry's ownscriptarray under-listed its cooperating files relative to the sibling entry's own precedent.Deliberately not fixed (disclosed, not silently dropped): the
_runsubprocess helper's own duplication withgitapex_gate_local_preflight.py's identical helper, and the hook's own bash-level duplication withhooks/check-pr-skill-audit-disclosure.sh(jq guards,deny(), base-branch resolution, the fail-closed backstop) -- both would require extracting a shared module/library and modifying that second, stable, already-tested file, judged out of this fix round's own narrow scope rather than bundled in.Unconfirmed concerns (security-tier, reported per policy though not independently reproduced as reachable): a theoretical dash-leading-ref argument-injection shape in
build_diff_added_corpus's owngit diffinvocation, confirmed unreachable in the current call graph (the hook always supplies a computed SHA, never attacker-controlled text) but flagged for a hypothetical future caller of the standalone--check-diffCLI flag; whether this harness's own PreToolUse hook timeout truly discards output on expiry (fail-open) was not independently verified in this environment, cited from this repository's ownthreat-model-and-authorization.md.Related Issue
Closes #1725
Refs #1707, #1711
Execution log