Repository navigation
perf(ci): fail held merge-queue pull requests before the lanes run - #10838
Conversation
ci-status's pr-unblock step applied its own jq and awk copy of the rule merge-holds.mjs states as unblockStops. It now hands the head's statuses to the base commit's unblockStops, so the rule is stated once. A missing base config or a base module that cannot judge the list fails closed. Refs: #10747 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A merge group for a pull request carrying a hold label, a head a pr-unblock lane stopped, or a lanes App pull request under the kill switch ran every lane leg before ci-status failed it. select-tests, which every leg needs, now judges those holds first in merge_group runs by the base commit's merge-holds.mjs queueHolds, so the group fails in seconds and the legs skip. ci-status keeps judging each hold from its own reads. A read the early step cannot make, or a base module without queueHolds, leaves the hold to ci-status. The step reads the pull request and its head's statuses through the api-budget wrapper and ends with its api-calls notice; select-tests gains statuses: read for it. The pr-unblock step's base-module call is written as one if, which shellcheck reads cleanly. Refs: #10747 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci-status's hold steps each read the same pull request facts again: the events in the cost-hold and hold-lift steps, the pull in the review-lane, pr-unblock and lanes App steps, the comments in the hold-lift and lanes App steps. The read helper now writes pr_read, which answers a read the job already made with the same args and keeps each successful answer in $RUNNER_TEMP/pr-reads; a failed read keeps nothing, so the next step reads again and fails closed as before. The cost-hold step keeps its own no-wait reads, keeps their answers for the later steps, and drops every kept answer after it writes to the pull request. Readers that filtered with gh's --jq now read the raw page and filter locally, so they share the same kept answer. A dry run of the cost-hold, human-gated and hold-lift steps on one pull_request run sends 3 requests where it sent 4. Refs: #10747 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cost-hold step acts on what it reads, so it no longer answers from a read an earlier step kept; it still keeps its own answers for the later steps and drops them all after it writes. Refs: #10747 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Incremental security review of critical-findings: 0 What changed, and the trust controls it touches
What I checked
No manifest fields are in this scope. No billable resource changes, so there is no cost hold. The change can only reduce API request spend, which is not a hold. |
|
claude-review reviewed Incremental review of What I checked:
No cost or critical-security hold. The change spends no billed resources. |
… step check-lane-coverage refuses a continue-on-error step whose outcome no aggregator row reads. The base-rules read now never fails the job and says what it read through an `ok` output, and the setup-node step is gone: queueHolds needs nothing newer than the runner's Node. Refs: #10747 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The pr-verify verifier failed head This pull request leaves the merge queue, and ci-status holds it out while it carries |
Comments, reviews, runs, jobs and annotations are read live again, so a member's veto or a new review is never answered from an earlier step's read. The cost-hold step keeps only its pull and events answers, and the lanes App step drops every kept read after it disarms auto-merge. The pr-unblock step's comment says why it imports unblockStops inline rather than through holds_js. Refs: #10747 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs: #10747
This delivers part of #10747: the pr-unblock rule stated once (ci-status's step now runs
merge-holds.mjsunblockStops) and the early merge-queue step inselect-tests. The remaining part, switching ci-status's hold-label and kill-switch steps tolabelHolds/switchHold, is tracked in #10835 and can land only once this is on main.Owner merge approval required. This changes gate and trust hold paths (
.github/workflows/pr-require-checks.yml,.github/scripts/pr-merge/merge-holds.mjs, both on the hold-path floor). No hold is loosened: every state that fails ci-status today still fails it.Summary
Fail a head a pr-unblock lane stoppedstep now judges the head's statuses with the base commit'smerge-holds.mjsunblockStopsinstead of its own jq/awk copy, so that rule is stated once.select-tests(which every lane legneeds) first judges the holds that cost no lane work: hold labels (do-not-merge, the tracker's human-gated label,pr-verify-failed), apr-unblock/*stop on the queued head, and the lanes kill switch on a lanes App pull request. A held group failsselect-testsin seconds and the legs skip. ci-status keeps all its own hold steps, so the aggregate still fails on the hold.pr_readcache in the read helper.Fix
Shared rules.
merge-holds.mjsgainshumanGatedLabel,labelHolds,switchHoldandqueueHolds(which composes them with the existingunblockStops). The reasons they return are ci-status's current error texts.Early step. Two steps at the top of
select-tests, merge_group only:git fetchof the group's base commit, thengit showof itsmerge-holds.mjs,api-budget.bashand.work-item-tracker.json. This step never fails the job; it setsok=trueonly when the module and wrapper read. The rules always come from the base, never from the queued pull request's code. The runner's Node runs the module (queueHoldsneeds nothing newer than the image's).pulls/N, which carries its labels and author) and its head's statuses through the api-budget gh wrapper, then run the base'squeueHolds. Only a positive hold fails the job. A failed read, an unparsable ref, or a base module withoutqueueHoldslogs a warning or notice and leaves the decision to ci-status, which retries reads and fails closed. That way a transient API error cannot become a new reason for the queue to eject a pull request. Nothing from this job feeds a ci-status hold decision, because this job runs pull request code.select-testsgainsstatuses: read. "List commit statuses for a reference" needs the "Commit statuses" read permission (permissions required for GitHub Apps). Labels come from the pulls response, whichpull-requests: readcovers.Skips cannot pass. When select-tests fails, each skipped leg stays
skippedinCheck each skipped lane against scope(that step passes a skip only when select-tests succeeded), andtreat-skipped-asisfailin merge_group. The existing case "a skip behind a failed scope stays skipped" inscripts/ci-ordered-skips.test.shcovers this.Sequencing (deviation from the issue's done-when). ci-status loads
merge-holds.mjsfrom the merge group's base. A ci-status that calledlabelHoldsorswitchHoldin this PR would fail closed in this PR's own merge group, and the PR could never merge. So ci-status keeps its bash label and kill-switch checks for now, and the follow-up issue linked below switches them, the same two-step path #10737 took forunblockStops. Until then the label and kill-switch rules exist in two places. The early step also does nothing in this PR's own merge group, because the base lacksqueueHoldsand the step logs that. It takes effect for groups whose base contains this commit.Read cache (separate commits). The coordinator asked for it: ci-status was the top GITHUB_TOKEN spender, at 528 requests an hour measured 23:55 to 00:55Z (39 runs, median 17 calls each), with each run reading the same pull request facts again. The read helper writes
pr_read, used for three reads only: the pull request, its labels and its issue events. The step answers from a read an earlier step of the same job made, else reads live and keeps the answer in$RUNNER_TEMP/pr-reads. Comments, reviews, runs, jobs and annotations are always read live, so a member's veto or a new review is never missed. A failed read keeps nothing, so the next step reads again and fails closed as before. The cost-hold step reads everything live, because it acts on its reads, and keeps only its pull and events answers. Every step that writes to the pull request (the cost-hold step's comment and label removal, the lanes App step's auto-merge disarm) drops every kept answer after the write. check-contract writes before anything is kept. Measured saving: in a dry run of the step bodies against a gh stub, a pull_request run sends 3 requests instead of 4 (the events read is shared) and a merge_group run sends 2 instead of 3 (the pull read is shared).I did not replace the review-lane step's REST run and job reads with one GraphQL query. The step filters runs by
head_repository.full_name == this repo, so a fork's run on the same SHA never counts as the review. GraphQLWorkflowRunhas no head-repository field (introspection:checkSuite createdAt databaseId deploymentReviews displayTitle event file id pendingDeploymentRequests resourcePath runAttempt runNumber updatedAt url workflow), andCheckSuite.repositoryis the base repository. The swap would drop that filter.Verification
Measurement: merge-group runs failing on a designed hold. I took 10-27 and 10-28 UTC (10-29 was partial and is excluded). There were 385 merge-group runs and 31 failures. Of the failures, 13 were designed holds (about 6.5 a day; the issue's 20 to 26 a day did not reproduce). 9 of those 13 are holds the early step catches (about 4.5 a day): 3 needs-human, 3 pr-unblock stops and 3 kill-switch stops on lanes App pull requests; the other 4 are pr-verify verdict holds, which still need ci-status's lanes-hold step. The 9 caught runs spent 283.6 job-minutes in lane legs (excluding ci-status, mark-lanes and select-tests), which works out to about 142 lane job-minutes a day saved. Each held group's wall time drops from 5 to 10 minutes to the select-tests runtime plus about 0.5 minutes of ci-status. Commands:
API calls.
api-callsnotice. At 140 to 245 merge-group runs a day, that is 280 to 490 a day.Tests.
actionlintpasses on the workflow.scripts/ci-early-queue-holds.test.shis new. It runs the early step body for each hold (labels, the tracker's own label, a pr-unblock stop, the kill switch), for the pass-to-ci-status paths, and for theapi-callsnotice of 2.queueHoldsdropping the kill switch,unblockStopsignoring the status creator (killed after adding the "another account's failure does not hold" case), the human-gated label ignoring the tracker, and an unreadable pull failing closed.merge-holds.test.mjsgetsqueueHoldscases.ci-hold-pr-unblock-status.test.shadds cases for a missing base config, a base module withoutunblockStops, and a module answering no list (each fails closed).ci-gh-read.test.shaddspr_readcases.ci-hold-cost-decision.test.shchecks that the step reads live even with a kept answer, keeps its events read, and drops kept reads after writing.log-api-budget.test.mjs, the pr-merge node tests, and reconcile.ci-review-plugin-source.test.shandresolve-config/callers.test.mjswere not run locally because packages are missing here (yaml,ajv). CI runs the full selected corpus.Related
🤖 Generated with Claude Code