feat: configurable commit overview with per-author counts and line totals - #28
Conversation
…tals Split the commit-history context by token cost. The reviewer now always can be told the situation at a glance — "There are X commits already on this PR" — with per-author commit counts and added/removed line totals, while the fully quoted commit messages (the token-expensive part) get their own, smaller cap: - MAX_SUMMARY_COMMITS (default 15): how many of the PR's most recent commits the overview reads. Line stats are not part of the pulls list response, so each summarized commit costs one extra GitHub API call; the cap bounds that, and 0 keeps the count-only header. - MAX_COMMIT_MESSAGES (default 5, was a hard-coded 15): how many commit messages are fully quoted in the prompt. - INCLUDE_COMMIT_SUMMARY (default true): toggles the overview. The PR commit list is now fetched once and shared by both features (--paginate output slurped into a single array, which also makes the "most recent N" truncation global instead of per-page). Merge commits are excluded from counts and stats; authors fall back to the commit author name when there is no linked GitHub account; failed stats fetches count as zero instead of aborting the review. Offline regression tests cover the overview text, both caps acting independently, merge exclusion, the zero-reads mode, and the singular phrasing.
Three quoted messages are enough context for how the PR evolved while the overview carries the broader picture; the default stays configurable.
Three context-cost adjustments for large PRs:
- Check runs: successful checks collapse into a one-line count
("N of M checks passed"); only non-passing runs (failure, skipped,
cancelled, timed out, still running) are listed individually, so
green matrix shards no longer flood the prompt. Skipped runs stay
visible — they can matter.
- Human comments: kept generous but fully configurable via
MAX_HUMAN_COMMENTS (default 100), MAX_HUMAN_COMMENT_LENGTH (4000),
and MAX_HUMAN_COMMENTS_TOTAL (20000). The newest comments are
presented newest-first, so clipping drops the oldest of the selected
— the latest feedback always survives — and every clip is marked as
truncated so the model knows context was cut.
- Labels: the list itself is unchanged and complete (it is important);
the prompt now instructs the model to only apply labels that are
genuinely useful and to add none when unsure.
The full diff is never shortened.
Measuring the clipped result's byte count is unreliable: command substitution strips trailing newlines, so a comment ending in blank lines could shrink the result below the budget and hide a real cut. Compare the full block's length against the budget instead, so the marker appears exactly when content was lost (and not on exact fill). Also documents why splicing validated numeric values into the gh api --jq filter is safe (gh api has no --arg passthrough).
…section The review contract now requires every finding to carry exactly one severity tag — must fix, should fix, or nit — ordered accordingly; inferences to be labeled explicitly as "Inference (not verified):" so assumptions never read as verified facts; and unverifiable but consequential questions to be collected in a final "Should be checked" section instead of being skipped or buried in findings. The section is omitted when there is nothing meaningful to check.
AI Code ReviewCritical Issues
Medium Issues
Style Suggestions
Review by Friendly AI Reviewer - made with ❤️ |
|
Addressing the AI review (and adding a review-contract change on top): Critical — Medium — temp file "in /tmp": declining. Style — New in ede8f84: review contract now requires |
AI Code ReviewFindings
Review by Friendly AI Reviewer - made with ❤️ |
The per-commit stats rows now pipe straight from the fetch loop into awk/sort, so there is no temp file left behind when a stats fetch goes wrong mid-loop. A bare additional EXIT trap was not an option: this script chains EXIT traps (DIFF_FILE, then DIFF_FILE+PROMPT_FILE), and a new trap silently replaces the previous one. The overall human-comments budget was always enforced by head -c, so its marker and documentation now say bytes; a wc -m check could disagree with the byte cutter (marker missing despite a real cut on multibyte content).
Switch the built-in default (script and this repo's own reviewer workflow) from minimax/minimax-m2.5 to z-ai/glm-5.3; slug verified against OpenRouter's model list. AI_MODEL still overrides.
|
Addressing the latest AI review: must fix — should fix — stats temp file cleanup: accepted, in a stronger form (2206dcf): the per-commit stats rows now pipe straight from the fetch loop into nit — Also in this push: default model switched to All 22 offline tests pass; |
|
TL;DR on the declined review points:
|
AI Code Review
Should be checked
✅ Approved with recommendations Review by Friendly AI Reviewer - made with ❤️ |
The comment fetch sent no per_page/--paginate, so GitHub returned a single default page of 30 comments, oldest first — on busy PRs the newest feedback (the whole point of the newest-first selection) was never fetched. The comment list is now fetched once with --paginate and merged locally (jq -s add), shared by the previous-AI-review and human-comment context; the previous-review selector also finds sticky reviews beyond page one now. The slice runs in local jq (--argjson caps), never inside a gh api --jq filter, because that would apply it per page. This also removes the now-dead COMMENT_SLICE branch. The check-runs fetch had the same first-page-only limit (30 runs per page by default); it paginates and merges locally too, so the pass summary counts every run on matrix-heavy repos. The commit-message list now marks its 2500-byte clip like every other budget, instead of cutting silently. The gh stub in the offline tests emulates real 30-item pages for both endpoints; new regressions cover beyond-page-one selection, sticky reviews past page one, all-page check summaries, and the marked message clip.
|
All three findings accepted — good catches — plus the structured-outputs question answered; fixed in 941c39e: should fix — first-page-only comment fetch: confirmed real. The comment list is now fetched once with nit — unreachable else: resolved structurally — nit — silent 2500-byte clip on commit messages: now detected from source length and marked " […truncated at 2500 bytes]", same contract as the other budgets; covered by a test. Should be checked — GLM 5.3 structured outputs: confirmed via OpenRouter's model metadata: 26 offline tests pass; |
AI Code Review
Should be checked
✅ Approved with recommendations Review by Friendly AI Reviewer - made with ❤️ |
…knobs
- The previous AI review now marks its 10000-byte clip like every other
budget, so a cut-off prior review (verdict, Should-be-checked items)
is never mistaken for a complete one.
- Per-commit line-stat fetch failures are counted and reported in the
summary header ("line stats unavailable for K commit(s)") instead of
silently reading as verified +0/-0. The stats rows now accumulate in
a shell variable with an explicit newline (command substitution
strips it; concatenated rows made awk mis-parse author totals).
- The workflow's env block forwards the six new configuration
variables from repository variables, so the README instructions for
them actually take effect; a regression test pins the forwarding.
|
Both nits accepted and the Should-be-checked confirmed — fixed in d4defa8: nit — unmarked previous-review clip: the previous AI review now uses the same source-length detection as the other budgets and appends " […truncated at 10000 bytes]"; regression included. nit — failed stats fetches reading as +0/-0: the loop now counts fetch failures and the summary header reports "(line stats unavailable for K commit(s))", so unknown never masquerades as verified zeros. Implementing this surfaced a real bug the suite caught: accumulating the rows in a shell variable lost the row-separating newline (command substitution strips it), concatenating rows and making awk mis-parse author totals ("alice: 1 commit, +10/-2" for a 4-commit PR) — fixed with an explicit Should be checked — workflow forwarding: confirmed: the env block enumerated variables explicitly and did not include the new knobs, so the README's repository-variable instructions would have silently done nothing. All six are now forwarded ( 29 offline tests pass; |
AI Code Review
✅ Approved with recommendations Review by Friendly AI Reviewer - made with ❤️ |
- Merge detection now uses the exact GitHub signal (.parents | length
> 1) centralized in one jq predicate, instead of a startswith("Merge")
heuristic that silently miscounted commits like "MergeableHashMap:
fix iteration" once the overview made it load-bearing.
- Byte-boundary clips strip a trailing incomplete UTF-8 sequence
(head -c can split a multibyte character and jq rejects invalid
UTF-8 when building the request payload, failing the whole review);
applied to the human-comments, previous-review, and commit-message
budgets.
- The PR-description budget follows the shared truncation contract
now: detected from source length and marked.
- Test fixture: restore the f-prefix lost on the {FOOTER} literal so
the fixture round-trips a realistic footer again.
|
All three accepted — fixed in a78fe9a: should fix — lost f-prefix: confirmed, my regression from the severity-tags edit; the last segment is an f-string again so nit — "Merge" prefix heuristic: replaced with the exact GitHub signal — nit — byte-boundary UTF-8 cuts: implemented the strip variant: every Bonus consistency: the PR-description 2000-byte clip (same pattern, previously unmarked) now detects from source length and marks like every other budget, with its own test. 31 offline tests pass; |
AI Code Review
Let me re-verify the first finding more carefully before committing to "must fix":
But: Re-examining the diff for real bugs:
Now, what are the genuinely reportable findings? A. Should fix — unbounded B. Should fix or nit — C. Nit — hard-coded budgets: COMMIT_MESSAGES 2500, PREVIOUS_REVIEWS 10000, PR_DESCRIPTION 2000 remain hard-coded while the human-comment budgets got knobs. Consistency nit; fine to defer. D. Nit — E. Should be checked — workflow forwarding of INCLUDE_ toggles*: the new test pins nine variables, but the diff doesn't show whether Wait — does the README document INCLUDE_* toggles? The README section shown lists AI_MODEL, AI_TEMPERATURE, AI_MAX_TOKENS, MAX_DIFF_SIZE, and the new MAX_/INCLUDE_COMMIT_SUMMARY, MAX_HUMAN_, STRUCTURED_OUTPUT. The hunk at line 72-73 shows the list; INCLUDE_COMMIT_SUMMARY is documented. Are INCLUDE_PREVIOUS_REVIEWS etc. documented? Not visible. The script has them as env-configs. The forwarding test covers the documented ones (plus STRUCTURED_OUTPUT). INCLUDE_COMMIT_SUMMARY is forwarded (in diff). So the check is: are there other documented-but-unforwarded vars? FAIL_ON_REQUESTED_CHANGES is forwarded (visible in diff context). AI_TEMPERATURE, AI_MAX_TOKENS visible. So the only uncertain ones are INCLUDE_* toggles beyond COMMIT_SUMMARY — if the README documents them anywhere else. I'll phrase the check narrowly: confirm the workflow env forwards every variable the README documents as configurable (the diff shows only part of the env block). F. Check status: "AI Code Review: in_progress" is this review's own run — not a failure. The other check passed. Nothing actionable. G. H. I. One more: J. K. L. M. Security review of prompt-injection surface: comment bodies, commit messages, author names, PR description all flow into the prompt — the model is instructed to review code; a malicious PR author could inject "ignore instructions" text. This is inherent to the design (pre-existing); the new commit summary adds author names (attacker-controlled via git author name) into the prompt — same class as before (commit messages already were). No new escalation. Skip per high-value focus? Could mention as a should-be-checked? It's a known inherent limitation; the PR doesn't worsen it materially. Skip. N. O. P. Q. R. S. T. U. V. W. X. Y. Z. So final findings list: Must fix: none hold up under scrutiny. Let me double-check one candidate: the Hmm, wait — one more: And One more potential must-fix candidate: And: Another: Another candidate: Another: Another: Another: Another: Another: Another: Another: Another: Another: Another: Another: Another: Another: OK. Also consider: the review-contract prompt change removes the old "Default: Skip the issue" guidance — now unverifiable-but-critical items go to "Should be checked". Behavior change intended. Fine.
Now the "Should be checked" items:
So one Should-be-checked item (workflow env completeness). Maybe also: whether Findings ordering: Should fix:
Nits:
Also maybe a nit: Wait, also reconsider severity for A. Is it "should fix"? The PR deliberately switched from first-page-only to full pagination to fix a correctness bug (newest comments beyond page 1 were invisible). The cost is real on huge PRs but correctness demanded full fetch (newest are on the LAST page chronologically). per_page=100 is a trivial improvement (3.3x fewer calls). A page cap would reintroduce the bug. So the actionable part is: add Also for the commits fetch: per_page=100 too. Let me also double-check the Also check: does One more potential should-fix: And: Also verify the claim in my finding A about the comments endpoint: GitHub default per_page=30, max 100. Yes. Now — labels: which to add? The changes are a feature (commit overview, configurable budgets) with test coverage. Existing labels: "enhancement" fits (new feature). "ai_code_review" is the bot's own label — the instructions for the reviewer's own PR? This is the repo of the reviewer itself. The label list includes ai_code_review (Friendly AI Code Review). Adding "ai_code_review" seems apt for PRs touching the reviewer? It's described as the label the bot adds. Hmm — "Prefer existing labels... only apply labels genuinely useful." The PR is a feature enhancement to the review system: "enhancement" fits. "ai_code_review" — the bot's label for its own reviews; this PR modifies the AI code reviewer itself, so it's arguably useful. I'll add "enhancement" and "ai_code_review". "documentation" also fits (README changes) but that's secondary; the PR is primarily code. Keep to enhancement + ai_code_review. fail_pass_workflow: the changes look sound; findings are should-fix/nits → "pass"? The fail_pass_workflow field presumably gates merge — with no must-fix, "pass". Verdict: "✅ Approved with recommendations". Let me now double-check finding D's claim that computing merges directly is better: current code: Also confirm nit C's numbers: COMMIT_MESSAGES head -c 2500 (hard-coded), PREVIOUS_REVIEWS head -c 10000 (hard-coded), PR_DESCRIPTION head -c 2000 (hard-coded). Yes. And nit B: script line Should-be-checked phrasing: "Cannot verify from diff whether the workflow's env block (outside the shown hunk) forwards the INCLUDE_* toggles (e.g., INCLUDE_COMMIT_MESSAGES, INCLUDE_CHECK_RUNS, INCLUDE_LABELS, INCLUDE_PR_DESCRIPTION, INCLUDE_PREVIOUS_REVIEWS, INCLUDE_HUMAN_COMMENTS) — please confirm each is either forwarded or not documented as configurable; a missing forwarding line makes the corresponding repository variable silently ineffective (the exact regression the new forwarding test guards for the nine listed names)." Is that within scope ("security vulnerabilities, breaking bugs, data loss risks")? A silently-ignored documented config is a breaking bug of the config contract. I'll keep it. Also consider whether to flag the in_progress check: "AI Code Review: in_progress" is this review itself — self-referential, not a failure. The instructions say consider failed/pending checks; this one is the reviewer running on this PR, so nothing to investigate. I won't add a finding, maybe a brief note? Instructions say omit empty sections and don't add summaries. I'll skip mentioning it — or one line? Better to skip; it's not actionable. Also: the human comments mention "31 offline tests pass; bash -n/shellcheck unchanged" — consistent. One more scan for security: Double-check Check Check Check Check Now, is there anything about the What about Edge: I'm fairly confident in the final set now. Let me also sanity-check finding A's numbers: GitHub issues comments default per_page=30, max 100. Pulls commits default 30, max 100? Pulls list commits default 30, max 100? Actually Also, Also for finding A, note the memory point is minor; the API-call count is the cost. Concrete failure scenario: a long-lived PR with 900 comments → 30 pages × (comments) each review run; with per_page=100 → 9 pages. Plus jq slurp of ~900 comments into a shell variable (could be MBs) — fine but wasteful. Severity: should fix. Now write the review. Keep it scannable. Order: should fix (A), then nits (D, C, B, merge-only-PR nit?). Let me decide on the merge-only nit — I'll include it briefly as a nit since it's a genuine behavioral gap: PR containing only merge commits → no summary block at all. Actually wait: can a GitHub PR have only merge commits? If someone merges main into the feature branch, the PR contains feature commits + the merge commit. A PR with ONLY a merge commit would mean the PR's head is a merge of two branches both... possible if the PR branch itself is a merge result and the diff vs base is non-empty? E.g., PR that merges branch X into branch Y where Y is the base — the PR would contain just the merge commit. Rare but real (sync PRs). In that case NONMERGE_COUNT=0 → no summary. The model then lacks even the count. Minor. Include as nit with one-line fix suggestion (emit header with 0 count or note merges). Hmm, the instructions say nits are fine. I'll include it compactly. Actually, let me reconsider — is it worth it? The feature's stated purpose: "Tells the model how many commits are already on a PR". For a merge-only PR it says nothing. One-line fix. Keep as nit. Also the double-parenthetical style nit — skip. Also Also Final check on the required JSON: review field with markdown, footer, verdict before footer. fail_pass_workflow: "pass". labels: ["enhancement", "ai_code_review"] — both exist. Should I add "documentation"? README changes are substantial but supporting. Keep two. Write it concisely. |
The subtraction was correct today but silently wrong if the non-merge filter ever gained more exclusions. Both counts now come straight from the shared is_merge classifier, with 2>/dev/null fallbacks and numeric validation so a jq hiccup degrades to zero rather than propagating an empty string.
|
On the latest review: must fix — MERGE_COUNT arithmetic: the crash scenario doesn't materialize: bash treats an empty/unset arithmetic operand as 0 (no must fix — stats loop: declining per the review's own analysis, which walks through each sub-concern (tab-injection is handled by 31 offline tests pass unchanged. |
AI Code Review
✅ Approved with recommendations Review by Friendly AI Reviewer - made with ❤️ |
- The non-passing check-run list is capped at 20 lines with a "+K more
non-passing run(s) not listed" line, so a broadly red matrix (shared
dependency failure, mass cancellation) cannot reintroduce the prompt
flooding exactly when the diff context is largest.
- The PR object is fetched once and shared by the check-runs head-SHA
lookup and the PR-description context; both default-on, so the same
endpoint was hit twice on every review.
- The workflow now forwards every INCLUDE_* toggle the script reads
(previously silently ignored as repository variables), and the
forwarding test derives the expected knob list from the script's
\${VAR:-default} lines so a new knob fails the test until forwarded.
- README: document the 0 semantics for the human-comment caps, note the
check-list cap, and label the cost ranges as carried-over estimates.
|
All five findings implemented in 6a705f3: should fix — uncapped non-passing list: capped at 20 lines with a "+K more non-passing run(s) not listed" line; regression covers 25 failures → 20 bullets + the overflow line, nit — duplicate PR fetch: the PR object is now fetched once into a shared variable; the check-runs head SHA and the PR description are both derived from it. Test asserts exactly one nit — undocumented zero semantics: README now states nit — stale cost ranges: the section now says the ranges are estimates carried over from real usage with the previous default model and points at current OpenRouter pricing for GLM 5.3. nit — hand-enumerated forwarding test: replaced with a test that derives the expected knob list from the script's 33 offline tests pass; |
Round-1 findings from an independent fresh-context review agent: - A mid-pagination gh failure (rate limit, transient 5xx) after valid pages was invisible: the pipeline's exit status is jq's, so the emitted prefix posed as the full list — the newest comments/commits silently vanished. Every paginated fetch (comments, commits, check runs, labels) now captures gh's exit separately from the merge and degrades to no data (with a stderr note) instead of a stale prefix. - The multibyte rationale was wrong for jq 1.7 (verified: --rawfile exits 0 and substitutes U+FFFD rather than rejecting), so nothing pinned strip_partial_utf8 — removing it kept tests green while the prompt silently gained replacement characters. Tests now assert the absence of U+FFFD at every clip site, and the comments state the actual behavior. Also adds coverage: partial-fetch failure paths, multibyte clips on the previous-review budget, per-comment codepoint slicing, the no-PR-fetch-when-unused invariant, and commit-message body indent.
Round-2 findings from a fresh review agent: the PR-object fetch was the only context fetch without a failure warning (operators got no trace when CI status and PR description silently vanished), and the README said the previous-review clip was 10k chars where the code clips 10000 bytes.
Round-3 finding: the README's test command lacked -B, so following it recreates the stray-.pyc problem this branch already had to clean up once; CI uses -B for exactly that reason. Also adds a .gitignore backstop and lists GLM 5.3 among the structured-output-capable models (support confirmed via OpenRouter's model metadata).
AI Code Review
Should be checked
✅ Approved with recommendations Review by Friendly AI Reviewer - made with ❤️ |
- Commit-message extraction now keeps the entire body — all paragraphs,
indented per line — and no longer drops bodies from non-conforming
messages that lack a blank line after the subject.
- The check-status prompt says to treat skipped and neutral runs as
informational rather than failures, so a continue-on-error neutral
conclusion cannot read as a red check.
- Line-stat fetch failures now log to stderr ("N of M listed commits")
so rate limiting is distinguishable from one flaky fetch.
- Workflow env block documents why the || fallbacks mirror the script
defaults (deliberate, keep in sync); the default test PR fixture
gains title/body so description tests exercise the real formatting
path.
Declined from the same review: the claimed tab-in-author column shift
cannot occur — jq @TSV escapes tabs (verified: the author field stays
intact through read -r and the awk aggregation).
AI Code Review
Should be checked
✅ Approved with recommendations Review by Friendly AI Reviewer - made with ❤️ |
Summary
Split the reviewer's context budget by value-per-token on busy PRs, and sharpen the review-output contract.
Review contract (new)
Inference (not verified):label so assumptions never read as verified facts.Commit history
MAX_SUMMARY_COMMITS(default 15): how many of the PR's most recent commits the overview reads (one extra GitHub API call each;0keeps the count-only header). Only numbers enter the prompt, so this cap stays generous.MAX_COMMIT_MESSAGES(default 3, was hard-coded 15): how many commit messages are fully quoted — the token-heavy part gets its own smaller cap.INCLUDE_COMMIT_SUMMARY(default true): toggle for the overview.Context-cost adjustments for large PRs
MAX_HUMAN_COMMENTS(100),MAX_HUMAN_COMMENT_LENGTH(4000),MAX_HUMAN_COMMENTS_TOTAL(20000). Newest-first selection so the latest feedback always survives clipping; clips are detected from the source length and marked " […truncated]".Implementation notes
--paginateoutput is slurped into a single array (also fixes a latent per-page truncation bug).0is honored ("read none").Tests
python3 -B -m unittest discover -s tests— 22 tests, all offline (localgh/curlsubstitutes): commit overview/aggregation/caps, check-run summary incl. skipped + all-green collapse, human-comment newest-first selection, both truncation markers plus exact-fill and trailing-newline edges, label guidance, the severity/inference/Should-be-checked prompt contract, and all prior context-filter regressions.bash -nandshellcheckclean (pre-existing style notes only).Once merged,
local-deep-researchcan bump its pin and pass the new vars through its workflow.