From bce6294733a44ca7a6dc0575c24023adf72de9ca Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 20 Oct 2026 03:29:58 -0400 Subject: [PATCH 1/3] feat(pr-pipeline): trust a reconciler-recorded re-run of a lost pr-verify run The lost-run reconciler could re-run CI workflows but only list lost lane runs for a person, because check-trusted-trigger refuses every re-run github-actions[bot] starts. pr-verify runs lost to an outage therefore still needed a manual re-run. Trust rule: with reconciler-workflow set (pr-verify only), a re-run whose triggering actor is github-actions[bot] proceeds only when it is attempt 2 of this run, started by github-actions[bot] (numeric id), and an unexpired artifact named reconcile-rerun--2- exists, created no later than that attempt started, by a run of the reconciler workflow in this repository on the default branch, started by its schedule or a listed human's dispatch, whose head is the default branch tip or behind it with no same-named tag, and that started at most 10 minutes before this attempt. The artifact's contents are never read. Every refusal is untrusted-rerunner; the original event's actors and the PR author are still checked as before. The reconciler now plans first, uploads one record per pr-verify run it will re-run, and only then re-runs them, re-reading each run (and, for PR CI, the PR head) before the write. Hardening from the security review (F1-F4, N1-N2): - the record binds run id, attempt 2 and head sha, and must come from a default-branch reconciler run started by its schedule or a listed human; - a record whose run cannot be read does not vouch, and the check reads at most 20 candidate runs before refusing, so forged same-named records cannot drain the token quota; - a pr-verify run whose first attempt started its judge step is never re-run, so a head is never judged twice; - a run none of whose jobs started (a pending duplicate concurrency cancelled) is never re-run; - the plan's record names are validated before upload, and verify re-runs share the in-flight cap with PR CI sets; - a pr-verify run that ended startup_failure with no jobs is held for a person. Review, refine and feedback-dispatch lanes still refuse these re-runs. Refs: #8468 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT --- .../actions/check-trusted-trigger/README.md | 38 +- .../actions/check-trusted-trigger/action.yml | 13 +- .../check-trusted-trigger.mjs | 183 ++++++++-- .../check-trusted-trigger.test.mjs | 298 +++++++++++++++ .../reconcile.mjs | 342 +++++++++++++----- .../reconcile.test.mjs | 286 +++++++++++---- .../maintenance-reconcile-lost-runs.yml | 80 +++- .github/workflows/pr-verify.yml | 4 +- 8 files changed, 1046 insertions(+), 198 deletions(-) diff --git a/.github/actions/check-trusted-trigger/README.md b/.github/actions/check-trusted-trigger/README.md index 92fcb15ad0..8805d1353b 100644 --- a/.github/actions/check-trusted-trigger/README.md +++ b/.github/actions/check-trusted-trigger/README.md @@ -28,6 +28,7 @@ condition 1 and the actor and author parts of condition 2). | `require-pr` | `true` | `false` checks only the actors and reads no PR | | `require-kind` | empty | `human` narrows the actor-only checks to actors listed as `kind: human` | | `app-label` | empty | With `require-kind: human`, the one label the lanes App may apply to an issue to start the lane | +| `reconciler-workflow` | empty | The lost-run reconciler's workflow path: a `github-actions[bot]` re-run proceeds only as its recorded re-run; empty stops every such re-run | ## Rules @@ -91,7 +92,7 @@ condition 1 and the actor and author parts of condition 2). a first `completed` seen after the 24th wait refuses at once. Each wait logs `dispatcher record: absent, re-read N` with the two run ids. - A failed read, a missing or unparseable field, or any failed condition stops with + A failed read, a missing or unparsable field, or any failed condition stops with `untrusted-actor`. The re-runner of a dispatched run is read by id from this run's `triggering_actor`: `github-actions[bot]` on attempt 1, otherwise a listed id that is not denied, or the lane stops with `untrusted-rerunner`. The PR author check still applies. @@ -132,6 +133,37 @@ condition 1 and the actor and author parts of condition 2). unlisted or denied re-runner stops with `untrusted-rerunner`, which `report-check-run` maps to no skip reason, so the check posts failure: a re-run cannot turn an earlier red check on the same SHA neutral. +- **Reconciler re-run (`reconciler-workflow`).** On the `pull_request` and `workflow_run` paths, + when `triggering-actor` is `github-actions[bot]` and `reconciler-workflow` is set, the re-runner + check is replaced by a check of the lost-run reconciler's record; the event's actors and the PR + author are still checked as above. It proceeds only when every one of these holds, and any + failed read or condition stops with `untrusted-rerunner`: + - `github-actions[bot]` (41898282) is not in `denied-actor-ids`, `GITHUB_RUN_ATTEMPT` is `2`, + and the payload names the repository's default branch; + - this run's attempt 2, read through `GET /actions/runs/{github.run_id}/attempts/2`, has that + `id` and `run_attempt`, `triggering_actor.id` 41898282, this repository, a 40-hex `head_sha` + and a `run_started_at`; + - `GET /actions/artifacts?name=reconcile-rerun--2-&per_page=100`, up to ten + pages, lists an + artifact with exactly that name, `expired` `false`, and a `created_at` no later than the + attempt's `run_started_at`, whose `workflow_run.id` names a run that has that `id`, this + repository, `path` equal to `reconciler-workflow`, `head_branch` the default branch, `event` + `schedule` or `workflow_dispatch` by an `actor` listed as `kind: human`, any re-run of it by a + `triggering_actor` listed as `kind: human`, and a `run_started_at` no later than the attempt's + and at most ten minutes before it; + - that run's `head_sha` passes the same tag and ancestry checks as a dispatcher run. + + Any run can upload an artifact with that name, so each listed artifact's run is read and + judged, and one that passes is enough; one whose run cannot be read does not vouch. At most 20 + candidate runs are read per check; a 21st candidate stops it with `untrusted-rerunner`. The run id and the attempt bind the record to one run + attempt, which starts once, so a record cannot vouch for another run or a later attempt. The + reconciler uploads the record before it calls the re-run, so an absent record is read once more + after 5 s and then refused. + + Residuals: whether one run can write another run's artifacts is unverified, as for the + dispatcher record. A re-run that waits in a queue past the ten-minute window is refused, red; + a person re-runs it. More than 1,000 forged artifacts of one name would push the real record + past the pages read, which refuses the re-run: a denial, never a pass. - **Denied ids.** When `sender` or a `workflow_run` actor has an id in `denied-actor-ids`, the lane stops with `bot-actor`, even though that actor is listed. `pr-run-activity-write.yml` denies the lanes App bot, so the bot's own pushes and dispatches never chain a write activity. @@ -151,7 +183,9 @@ any App token exists, so the job token is the only one available, and it needs read-only, so no write token reaches the model. A token that cannot read the PR stops the lane with `no-pr`. A dispatcher-sent run also reads two run records, a comparison and the dispatcher run's artifact list; on this public repository the job token reads them without `actions: read`, -and a token that cannot stops the lane with `untrusted-actor`. +and a token that cannot stops the lane with `untrusted-actor`. A reconciler re-run reads its own +attempt, the artifact list by name, the recording run and a comparison the same way, and a token +that cannot stops it with `untrusted-rerunner`. ## Outputs diff --git a/.github/actions/check-trusted-trigger/action.yml b/.github/actions/check-trusted-trigger/action.yml index 6f6366a7dd..e0186ee54f 100644 --- a/.github/actions/check-trusted-trigger/action.yml +++ b/.github/actions/check-trusted-trigger/action.yml @@ -31,8 +31,8 @@ inputs: github-token: description: > Token that can read the PR, used on workflow_dispatch and workflow_run - to fetch it, and on a dispatcher-sent run to read both run records and - compare commits. + to fetch it, and on a dispatcher-sent run or a reconciler re-run to read + the run records and artifacts and compare commits. required: false default: ${{ github.token }} denied-actor-ids: @@ -98,6 +98,14 @@ inputs: every dispatcher-sent run. required: false default: "" + reconciler-workflow: + description: > + The workflow path of the lost-run reconciler. A re-run github-actions[bot] + started proceeds only as that workflow's recorded re-run of this run's + first attempt (the README's reconciler section). Empty keeps every + github-actions[bot] re-run stopped with untrusted-rerunner. + required: false + default: "" require-kind: description: > Empty accepts any listed actor. `human` narrows the actor-only checks @@ -162,6 +170,7 @@ runs: EXPECTED_UPSTREAM_HEAD: ${{ inputs.expected-upstream-head }} LANES_APP_ID: ${{ inputs.lanes-app-id }} DISPATCHED_WORKFLOW: ${{ inputs.dispatched-workflow }} + RECONCILER_WORKFLOW: ${{ inputs.reconciler-workflow }} REQUIRE_KIND: ${{ inputs.require-kind }} APP_LABEL: ${{ inputs.app-label }} REQUIRE_PR: ${{ inputs.require-pr }} diff --git a/.github/actions/check-trusted-trigger/check-trusted-trigger.mjs b/.github/actions/check-trusted-trigger/check-trusted-trigger.mjs index 1104a154c7..920baaf3e6 100644 --- a/.github/actions/check-trusted-trigger/check-trusted-trigger.mjs +++ b/.github/actions/check-trusted-trigger/check-trusted-trigger.mjs @@ -35,7 +35,20 @@ const RUN_RECORD_NAME = /^pr-feedback-dispatched-([1-9][0-9]{0,14})-([1-9][0-9]{ // The record can land after this run's gate starts: re-read every 5 s while // the dispatcher run is still going, for at most 2 min (judgment). const RECORD_WAIT_MS = 5000; +// Pages of same-name artifacts read for a reconciler record (judgment). +const RECORD_MAX_PAGES = 10; +// Candidate runs read per check: anyone's run can upload a record of the +// predictable name, so each forged one costs a run read (judgment). +const RECORD_MAX_CANDIDATES = 20; const RECORD_MAX_SLEEPS = 24; +// The lost-run reconciler re-runs a lane run once, on its first attempt, so +// the attempt it starts is always the second. +const RERUN_ATTEMPT = 2; +const RECONCILER_EVENTS = new Set(["schedule", "workflow_dispatch"]); + +// The artifact the reconciler run uploads before it re-runs a lane run: the +// run, the attempt the re-run starts, and the run's head SHA. +export const rerunRecordName = (runId, headSha) => `reconcile-rerun-${runId}-${RERUN_ATTEMPT}-${headSha}`; const defaultSleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); @@ -198,19 +211,42 @@ function checkRerunnerLogin({ ids, logins }, deniedIds, triggeringActor, pool = const timeOf = (value) => (typeof value === "string" ? Date.parse(value) : Number.NaN); -async function readOrRefuse(github, apiPath) { +async function readOrRefuse(github, apiPath, reason = "untrusted-actor") { let body; try { body = await github("GET", apiPath); } catch { - throw new Stop("untrusted-actor"); + throw new Stop(reason); } if (!isPlainObject(body)) { - throw new Stop("untrusted-actor"); + throw new Stop(reason); } return body; } +// The run at headSha ran the default branch's own code: no tag shares the +// branch's name (a run on that tag would report the same head_branch; only a +// 404 proves it absent), and headSha is the branch tip or behind it. +async function requireOnDefaultBranch(github, repository, defaultBranch, headSha, reason) { + let tagAbsent = false; + try { + await github("GET", `/repos/${repository}/git/ref/tags/${defaultBranch}`); + } catch (error) { + tagAbsent = error instanceof GitHubError && error.status === 404; + } + if (!tagAbsent) { + throw new Stop(reason); + } + const tip = await readOrRefuse(github, `/repos/${repository}/git/ref/heads/${defaultBranch}`, reason); + if (tip.ref !== `refs/heads/${defaultBranch}` || !SHA.test(tip.object?.sha ?? "")) { + throw new Stop(reason); + } + const comparison = await readOrRefuse(github, `/repos/${repository}/compare/${headSha}...${tip.object.sha}`, reason); + if (comparison.status !== "ahead" && comparison.status !== "identical") { + throw new Stop(reason); + } +} + // What a record name says about this run: null when it does not list it, // otherwise true, or for a workflow_run dispatcher's record the upstream run id. function readRecordName(name, runId, viaRun) { @@ -357,28 +393,7 @@ async function verifyDispatcher({ if (!trusted) { throw new Stop("untrusted-actor"); } - // A tag with the branch's name would give a run on that tag the same - // head_branch, so one must not exist; only a 404 proves that. - let tagAbsent = false; - try { - await github("GET", `/repos/${repository}/git/ref/tags/${defaultBranch}`); - } catch (error) { - tagAbsent = error instanceof GitHubError && error.status === 404; - } - if (!tagAbsent) { - throw new Stop("untrusted-actor"); - } - const tip = await readOrRefuse(github, `/repos/${repository}/git/ref/heads/${defaultBranch}`); - if (tip.ref !== `refs/heads/${defaultBranch}` || !SHA.test(tip.object?.sha ?? "")) { - throw new Stop("untrusted-actor"); - } - const comparison = await readOrRefuse( - github, - `/repos/${repository}/compare/${dispatcher.head_sha}...${tip.object.sha}`, - ); - if (comparison.status !== "ahead" && comparison.status !== "identical") { - throw new Stop("untrusted-actor"); - } + await requireOnDefaultBranch(github, repository, defaultBranch, dispatcher.head_sha, "untrusted-actor"); if (expectedUpstreamHead !== "" && (!viaRun || !SHA.test(expectedUpstreamHead))) { throw new Stop("untrusted-actor"); } @@ -462,6 +477,114 @@ function checkDispatchedRerunner({ ids }, deniedIds, own) { } } +// A re-run github-actions[bot] started names no human, so it is trusted only +// as the lost-run reconciler's re-run of this run: this run's attempt +// RERUN_ATTEMPT, started by github-actions[bot], and an unexpired +// rerunRecordName artifact, created no later than that attempt started, of a +// reconcilerWorkflow run in this repository on the default branch, started by +// its schedule or a listed human's dispatch, any re-run of it by a listed +// human, its head SHA the default branch tip or behind it, and this attempt +// started between that run's start and WINDOW_MS after it. A run id names one +// run and its second attempt starts once, so a record vouches for one attempt. +// The reconciler uploads the record before it re-runs; one more read follows +// RECORD_WAIT_MS for any lag in the artifact list. Every refusal is +// untrusted-rerunner, as for an unlisted re-runner. +async function verifyReconcilerRerun({ github, repository, event, list, deniedIds, reconcilerWorkflow, runId, runAttempt, sleep }) { + const refuse = "untrusted-rerunner"; + const defaultBranch = event.repository?.default_branch; + if ( + deniedIds.has(ACTIONS_BOT_ID) || + !RUN_ID.test(runId) || + runAttempt !== String(RERUN_ATTEMPT) || + typeof defaultBranch !== "string" || + !HEAD_REF.test(defaultBranch) + ) { + throw new Stop(refuse); + } + const own = await readOrRefuse(github, `/repos/${repository}/actions/runs/${runId}/attempts/${RERUN_ATTEMPT}`, refuse); + const ownStarted = timeOf(own.run_started_at); + if ( + String(own.id) !== runId || + own.run_attempt !== RERUN_ATTEMPT || + own.triggering_actor?.id !== ACTIONS_BOT_ID || + own.repository?.full_name !== repository || + !SHA.test(own.head_sha ?? "") || + Number.isNaN(ownStarted) + ) { + throw new Stop(refuse); + } + const name = rerunRecordName(runId, own.head_sha); + const isListedHuman = (user) => + Number.isInteger(user?.id) && list.humans.has(user.id) && !deniedIds.has(user.id); + // Anyone's run can upload an artifact of this name, so each candidate's run + // is read and judged, at most RECORD_MAX_CANDIDATES of them; one that cannot + // be read does not vouch. + let candidates = 0; + const vouches = async (artifact) => { + if ( + !isPlainObject(artifact) || + artifact.name !== name || + artifact.expired !== false || + !Number.isSafeInteger(artifact.workflow_run?.id) || + !(timeOf(artifact.created_at) <= ownStarted) + ) { + return false; + } + if (candidates >= RECORD_MAX_CANDIDATES) { + console.log(`check-trusted-trigger: reconciler record: over ${RECORD_MAX_CANDIDATES} candidate runs, refusing (run ${runId})`); + throw new Stop(refuse); + } + candidates += 1; + let run; + try { + run = await readOrRefuse(github, `/repos/${repository}/actions/runs/${artifact.workflow_run.id}`, refuse); + } catch { + return false; + } + const started = timeOf(run.run_started_at); + const trusted = + run.id === artifact.workflow_run.id && + run.repository?.full_name === repository && + run.path === reconcilerWorkflow && + run.head_branch === defaultBranch && + RECONCILER_EVENTS.has(run.event) && + (run.event === "schedule" || isListedHuman(run.actor)) && + SHA.test(run.head_sha ?? "") && + Number.isInteger(run.run_attempt) && + run.run_attempt >= 1 && + (run.run_attempt === 1 || isListedHuman(run.triggering_actor)) && + started <= ownStarted && + ownStarted <= started + WINDOW_MS; + if (!trusted) { + return false; + } + await requireOnDefaultBranch(github, repository, defaultBranch, run.head_sha, refuse); + return true; + }; + const listingPath = `/repos/${repository}/actions/artifacts?name=${name}&per_page=100`; + for (let read = 0; read < 2; read += 1) { + if (read > 0) { + console.log(`check-trusted-trigger: reconciler record: absent, re-read (run ${runId})`); + await sleep(RECORD_WAIT_MS); + } + for (let page = 1; page <= RECORD_MAX_PAGES; page += 1) { + const body = await readOrRefuse(github, `${listingPath}&page=${page}`, refuse); + if (!Array.isArray(body.artifacts)) { + throw new Stop(refuse); + } + for (const artifact of body.artifacts) { + if (await vouches(artifact)) { + return; + } + } + if (body.artifacts.length < 100) { + break; + } + } + } + throw new Stop(refuse); +} + // app-label: the lanes App applying exactly that label to an issue starts the // lane; the label alone admits the issue. Any other App event stops. The // re-runner is the App itself or a listed human. @@ -528,6 +651,8 @@ export async function evaluateTrigger({ requireKind = "", appLabel = "", runId = "", + runAttempt = "", + reconcilerWorkflow = "", dispatchedWorkflow = "", github, sleep = defaultSleep, @@ -609,7 +734,11 @@ export async function evaluateTrigger({ if (!actors.every((actor) => isListed(ids, actor))) { throw new Stop("untrusted-actor"); } - checkRerunnerLogin(list, deniedIds, triggeringActor); + if (reconcilerWorkflow !== "" && String(triggeringActor).toLowerCase() === "github-actions[bot]") { + await verifyReconcilerRerun({ github, repository, event, list, deniedIds, reconcilerWorkflow, runId, runAttempt, sleep }); + } else { + checkRerunnerLogin(list, deniedIds, triggeringActor); + } } if (!isListed(ids, pull.user)) { throw new Stop("untrusted-author"); @@ -671,6 +800,8 @@ export async function main({ env = process.env, github, sleep = defaultSleep } = appLabel: env.APP_LABEL ?? "", dispatchedWorkflow: env.DISPATCHED_WORKFLOW ?? "", runId: env.GITHUB_RUN_ID ?? "", + runAttempt: env.GITHUB_RUN_ATTEMPT ?? "", + reconcilerWorkflow: env.RECONCILER_WORKFLOW ?? "", github: github ?? createGitHub({ token: env.GITHUB_TOKEN, apiUrl: env.GITHUB_API_URL }), sleep, }); diff --git a/.github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs b/.github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs index a3989d1894..8885bc5f43 100644 --- a/.github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs +++ b/.github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs @@ -87,6 +87,8 @@ async function gate({ dispatchedWorkflow, requireKind, appLabel, + reconcilerWorkflow, + runAttempt, sleep = fakeSleep(), }) { const outputPath = writeJson("output", ""); @@ -115,6 +117,8 @@ async function gate({ ...(dispatchedWorkflow === undefined ? {} : { DISPATCHED_WORKFLOW: dispatchedWorkflow }), ...(requireKind === undefined ? {} : { REQUIRE_KIND: requireKind }), ...(appLabel === undefined ? {} : { APP_LABEL: appLabel }), + ...(reconcilerWorkflow === undefined ? {} : { RECONCILER_WORKFLOW: reconcilerWorkflow }), + ...(runAttempt === undefined ? {} : { GITHUB_RUN_ATTEMPT: runAttempt }), }, github, sleep, @@ -884,6 +888,300 @@ test("dispatcher: a listed human's own dispatch reads no record", async () => { assert.equal(artifactReads(github), 0); }); +// A lost lane run re-run by the lost-run reconciler: attempt 2, started by +// github-actions[bot], trusted only through the reconciler run's record +// reconcile-rerun--2-, uploaded before the re-run. Run and +// artifact shapes follow the recorded fixtures above. + +const RECONCILER_WORKFLOW = ".github/workflows/maintenance-reconcile-lost-runs.yml"; +const RECONCILER_RUN_ID = 39320000001; +const RECONCILER_HEAD = "7777777777777777777777777777777777777777"; +const LOST_RUN_ID = "39310000002"; +const LOST_HEAD = "6666666666666666666666666666666666666666"; +const RERUN_RECORD = `reconcile-rerun-${LOST_RUN_ID}-2-${LOST_HEAD}`; +const attemptRoute = `GET /repos/${REPOSITORY}/actions/runs/${LOST_RUN_ID}/attempts/2`; +const rerunArtifactsRoute = `GET /repos/${REPOSITORY}/actions/artifacts?name=${RERUN_RECORD}&per_page=100&page=1`; +const reconcilerRoute = `GET /repos/${REPOSITORY}/actions/runs/${RECONCILER_RUN_ID}`; +const reconcilerCompare = `GET /repos/${REPOSITORY}/compare/${RECONCILER_HEAD}...${TIP_SHA}`; + +const reconcilerRun = (overrides = {}) => ({ + id: RECONCILER_RUN_ID, + name: "maintenance-reconcile-lost-runs", + path: RECONCILER_WORKFLOW, + event: "schedule", + status: "in_progress", + head_branch: "main", + head_sha: RECONCILER_HEAD, + run_attempt: 1, + created_at: "2026-10-19T19:04:00Z", + run_started_at: "2026-10-19T19:04:10Z", + updated_at: "2026-10-19T19:04:50Z", + actor: KYLE, + triggering_actor: KYLE, + repository: { id: 1, full_name: REPOSITORY }, + head_repository: { id: 1, full_name: REPOSITORY }, + ...overrides, +}); +const lostAttempt = (overrides = {}) => ({ + id: Number(LOST_RUN_ID), + name: "pr-verify", + path: ".github/workflows/pr-verify.yml", + event: "workflow_run", + head_branch: "main", + head_sha: LOST_HEAD, + run_attempt: 2, + created_at: "2026-10-19T18:40:00Z", + run_started_at: "2026-10-19T19:04:45Z", + actor: KYLE, + triggering_actor: ACTIONS_BOT, + repository: { id: 1, full_name: REPOSITORY }, + ...overrides, +}); +const rerunRecord = (overrides = {}) => ({ + id: 11800000001, + name: RERUN_RECORD, + expired: false, + created_at: "2026-10-19T19:04:40Z", + updated_at: "2026-10-19T19:04:40Z", + expires_at: "2026-10-20T19:04:39Z", + workflow_run: { + id: RECONCILER_RUN_ID, + repository_id: 1, + head_repository_id: 1, + head_branch: "main", + head_sha: RECONCILER_HEAD, + }, + ...overrides, +}); + +// No tagRoute: the fake answers 404, so no tag shares the branch name. +function rerunRoutes({ attempt = {}, reconciler = {}, artifacts = listing(rerunRecord()), compare = { status: "ahead" }, extra = {} } = {}) { + return { + [commitPulls]: [fixture("api-pull-42.json")], + [attemptRoute]: { ...lostAttempt(), ...attempt }, + [rerunArtifactsRoute]: artifacts, + [reconcilerRoute]: typeof reconciler === "function" ? reconciler : reconcilerRun(reconciler), + [tipRoute]: TIP_REF, + [reconcilerCompare]: compare, + ...extra, + }; +} + +function rerunEvent() { + const event = fixture("event-workflow-run.json"); + event.repository = { ...event.repository, default_branch: "main" }; + return event; +} + +function reconcilerRerun({ github, event = rerunEvent(), eventName = "workflow_run", ...rest }) { + return gate({ + eventName, + event, + github, + triggeringActor: "github-actions[bot]", + reconcilerWorkflow: RECONCILER_WORKFLOW, + runId: LOST_RUN_ID, + runAttempt: "2", + ...rest, + }); +} + +test("reconciler re-run: a valid record vouches for github-actions[bot] as re-runner", async () => { + const github = fakeGitHub(rerunRoutes()); + assert.deepEqual(await reconcilerRerun({ github }), PROCEED); + assert.ok(github.calls.includes(rerunArtifactsRoute)); + assert.ok(github.calls.includes(reconcilerCompare)); +}); + +test("reconciler re-run: a valid record vouches on a pull_request lane too", async () => { + const event = fixture("event-pull-request.json"); + event.repository = { ...event.repository, default_branch: "main" }; + const github = fakeGitHub(rerunRoutes({ attempt: { event: "pull_request", head_sha: LOST_HEAD } })); + assert.deepEqual(await reconcilerRerun({ github, event, eventName: "pull_request" }), PROCEED); +}); + +test("reconciler re-run: a reconciler run a listed human dispatched vouches", async () => { + const github = fakeGitHub(rerunRoutes({ reconciler: { event: "workflow_dispatch" } })); + assert.deepEqual(await reconcilerRerun({ github }), PROCEED); +}); + +test("reconciler re-run: one valid record among forged ones of the same name vouches", async () => { + const forged = rerunRecord({ id: 11800000009, workflow_run: { id: 39320000009, head_branch: "fix/x" } }); + const github = fakeGitHub( + rerunRoutes({ + artifacts: listing(forged, rerunRecord()), + extra: { [`GET /repos/${REPOSITORY}/actions/runs/39320000009`]: reconcilerRun({ id: 39320000009, event: "pull_request", head_branch: "fix/x" }) }, + }), + ); + assert.deepEqual(await reconcilerRerun({ github }), PROCEED); +}); + +test("reconciler re-run: a forged record whose run cannot be read does not stop the check", async () => { + // No route answers for run 39320000009, so its read fails. + const forged = rerunRecord({ id: 11800000009, workflow_run: { id: 39320000009 } }); + const github = fakeGitHub(rerunRoutes({ artifacts: listing(forged, rerunRecord()) })); + assert.deepEqual(await reconcilerRerun({ github }), PROCEED); +}); + +test("reconciler re-run: a valid record past a full page of forged ones vouches", async () => { + const forged = Array.from({ length: 100 }, (_, i) => rerunRecord({ id: 11900000000 + i, expired: true })); + const github = fakeGitHub( + rerunRoutes({ + artifacts: { total_count: 101, artifacts: forged }, + extra: { [rerunArtifactsRoute.replace(/page=1$/, "page=2")]: { total_count: 101, artifacts: [rerunRecord()] } }, + }), + ); + assert.deepEqual(await reconcilerRerun({ github }), PROCEED); +}); + +test("reconciler re-run: 21 forged candidates before the real record refuse after 20 run reads", async () => { + const forgedIds = Array.from({ length: 21 }, (_, i) => 39330000000 + i); + const forged = forgedIds.map((id, i) => rerunRecord({ id: 11900000000 + i, workflow_run: { id } })); + const github = fakeGitHub(rerunRoutes({ artifacts: listing(...forged, rerunRecord()) })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); + const runReads = github.calls.filter((call) => /\/actions\/runs\/[0-9]+$/.test(call)); + assert.equal(runReads.length, 20); + assert.ok(!github.calls.includes(reconcilerRoute)); +}); + +test("reconciler re-run: with no record, github-actions[bot] stays an untrusted re-runner after one re-read", async () => { + const sleep = fakeSleep(); + const github = fakeGitHub(rerunRoutes({ artifacts: listing() })); + assert.deepEqual(await reconcilerRerun({ github, sleep }), STOPPED("untrusted-rerunner")); + assert.equal(github.calls.filter((call) => call === rerunArtifactsRoute).length, 2); + assert.deepEqual(sleep.waits, [5000]); +}); + +test("reconciler re-run: without reconciler-workflow the bot re-runner stops and nothing is read", async () => { + const github = fakeGitHub(rerunRoutes()); + assert.deepEqual( + await reconcilerRerun({ github, reconcilerWorkflow: "" }), + STOPPED("untrusted-rerunner"), + ); + assert.deepEqual(github.calls, [commitPulls]); +}); + +test("reconciler re-run: an unlisted re-runner with no record still stops", async () => { + const github = fakeGitHub(rerunRoutes({ artifacts: listing() })); + assert.deepEqual( + await reconcilerRerun({ github, triggeringActor: "stranger" }), + STOPPED("untrusted-rerunner"), + ); + assert.ok(!github.calls.includes(rerunArtifactsRoute)); +}); + +test("reconciler re-run: a record does not vouch for an unlisted PR author", async () => { + const pull = fixture("api-pull-42.json"); + pull.user = STRANGER; + const github = fakeGitHub({ ...rerunRoutes(), [commitPulls]: [pull] }); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-author")); +}); + +test("reconciler re-run: a record does not vouch for an unlisted upstream actor", async () => { + const event = rerunEvent(); + event.workflow_run.actor = STRANGER; + const github = fakeGitHub(rerunRoutes()); + assert.deepEqual(await reconcilerRerun({ github, event }), STOPPED("untrusted-actor")); + assert.ok(!github.calls.includes(rerunArtifactsRoute)); +}); + +for (const runAttempt of ["1", "3", ""]) { + test(`reconciler re-run: attempt ${runAttempt || "(none)"} is never vouched for and nothing is read`, async () => { + const github = fakeGitHub(rerunRoutes()); + assert.deepEqual(await reconcilerRerun({ github, runAttempt }), STOPPED("untrusted-rerunner")); + assert.deepEqual(github.calls, [commitPulls]); + }); +} + +test("reconciler re-run: a denied github-actions[bot] stops before any record read", async () => { + const github = fakeGitHub(rerunRoutes()); + assert.deepEqual( + await reconcilerRerun({ github, deniedActorIds: "41898282" }), + STOPPED("untrusted-rerunner"), + ); + assert.ok(!github.calls.includes(rerunArtifactsRoute)); +}); + +for (const [label, attempt] of [ + ["the attempt was started by someone else", { triggering_actor: KYLE }], + ["the attempt names another run", { id: 39310000003 }], + ["the attempt is another attempt", { run_attempt: 3 }], + ["the attempt belongs to another repository", { repository: { id: 2, full_name: "stranger/claude-code-plugins" } }], + ["the attempt has no head SHA", { head_sha: "main" }], + ["the attempt started before the reconciler run", { run_started_at: "2026-10-19T19:04:09Z" }], + ["the attempt started more than ten minutes after the reconciler run", { run_started_at: "2026-10-19T19:14:11Z" }], + ["the attempt started before the record was written", { run_started_at: "2026-10-19T19:04:39Z" }], +]) { + test(`reconciler re-run: refused when ${label}`, async () => { + const github = fakeGitHub(rerunRoutes({ attempt })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); + }); +} + +test("reconciler re-run: a record naming another run or another head does not vouch", async () => { + // The listing for this run's name is empty; records for other names exist. + const others = { + [`GET /repos/${REPOSITORY}/actions/artifacts?name=reconcile-rerun-39310000099-2-${LOST_HEAD}&per_page=100&page=1`]: listing(rerunRecord({ name: `reconcile-rerun-39310000099-2-${LOST_HEAD}` })), + [`GET /repos/${REPOSITORY}/actions/artifacts?name=reconcile-rerun-${LOST_RUN_ID}-2-${HEAD_SHA}&per_page=100&page=1`]: listing(rerunRecord({ name: `reconcile-rerun-${LOST_RUN_ID}-2-${HEAD_SHA}` })), + }; + const github = fakeGitHub(rerunRoutes({ artifacts: listing(), extra: others })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); +}); + +for (const [label, artifacts] of [ + ["the listed artifact has another name", listing(rerunRecord({ name: `reconcile-rerun-39310000099-2-${LOST_HEAD}` }))], + ["the record has expired", listing(rerunRecord({ expired: true }))], + ["the record has no run id", listing(rerunRecord({ workflow_run: null }))], + ["the record was written after the re-run started", listing(rerunRecord({ created_at: "2026-10-19T19:04:46Z" }))], + ["the listing has no artifacts array", { total_count: 1 }], +]) { + test(`reconciler re-run: refused when ${label}`, async () => { + const github = fakeGitHub(rerunRoutes({ artifacts })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); + }); +} + +for (const [label, reconciler] of [ + ["the recording run is another workflow", { path: ".github/workflows/pr-merge.yml" }], + ["the recording run is a PR branch's run", { event: "pull_request", head_branch: "fix/forge" }], + ["the recording run is a dispatch on a PR branch", { event: "workflow_dispatch", head_branch: "fix/forge" }], + ["the recording run is a push run", { event: "push" }], + ["the recording run is in another repository", { repository: { id: 2, full_name: "stranger/claude-code-plugins" } }], + ["the recording run was dispatched by an unlisted user", { event: "workflow_dispatch", actor: STRANGER }], + ["the recording run was re-run by an unlisted user", { run_attempt: 2, triggering_actor: STRANGER }], + ["the recording run reports another id", { id: 39320000002 }], + ["the recording run's head is not a SHA", { head_sha: "main" }], + ["the recording run started after the attempt", { run_started_at: "2026-10-19T19:04:46Z" }], +]) { + test(`reconciler re-run: refused when ${label}`, async () => { + const github = fakeGitHub(rerunRoutes({ reconciler })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); + }); +} + +test("reconciler re-run: a recording run not on the default branch's history does not vouch", async () => { + const github = fakeGitHub(rerunRoutes({ compare: { status: "diverged" } })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); +}); + +test("reconciler re-run: a tag sharing the default branch name does not vouch", async () => { + const github = fakeGitHub({ ...rerunRoutes(), [tagRoute]: { ref: "refs/tags/main" } }); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); +}); + +test("reconciler re-run: a payload without the default branch does not vouch", async () => { + const github = fakeGitHub(rerunRoutes()); + assert.deepEqual( + await reconcilerRerun({ github, event: fixture("event-workflow-run.json") }), + STOPPED("untrusted-rerunner"), + ); +}); + +test("reconciler re-run: a failed attempt read refuses", async () => { + const github = fakeGitHub(rerunRoutes({ attempt: undefined, extra: { [attemptRoute]: new GitHubError(502, "attempt") } })); + assert.deepEqual(await reconcilerRerun({ github }), STOPPED("untrusted-rerunner")); +}); + // actor-only mode (require-pr: false) const ACTORS_OK = { ...STOPPED("ok"), proceed: "true" }; diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs index 845ecbae4b..527f61c5fb 100755 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs @@ -1,41 +1,55 @@ #!/usr/bin/env node -// reconcile.mjs: re-runs PR CI runs a GitHub outage ended without a result. -// A CI workflow's newest run on an open PR's current head that ended -// cancelled, timed_out or startup_failure, on its first attempt, is re-run -// once with the job's own token. Its attempt number is the record: a run at -// attempt 2 or later was already re-run, by this script or a person, and is -// never touched again. No model. +// reconcile.mjs: re-runs PR CI and lane runs a GitHub outage ended without a +// result. A run that ended cancelled, timed_out or startup_failure, on its +// first attempt, is re-run once with the job's own token. Its attempt number is +// the record: a run at attempt 2 or later was already re-run, by this script +// or a person, and is never touched again. No model. +// +// Two kinds are re-run: +// - CI: a CI workflow's newest run on an open PR's current head. These +// workflows check no re-runner. +// - Lane: a pr-verify run. Its trigger check trusts a re-run github-actions[bot] +// starts only through this workflow's record of it, the artifact +// rerunRecordName(run id, head SHA) (check-trusted-trigger's README). The +// plan step names each record, the workflow uploads them, and only then does +// the act step re-run the run, so the record exists before the re-run. +// Other lost lane runs (the review lanes, refine) are listed in the summary for +// a person: their gates or the lanes token broker refuse a re-run +// github-actions[bot] starts. A merge-group run is not covered: a re-run puts +// no PR back in the queue, and pr-merge re-enqueues its own PRs once their +// checks pass. // // Nothing is read from GitHub while githubstatus.com shows Actions as anything // but operational, or the status cannot be read, so a re-run never adds load -// mid-incident. At most MAX_IN_FLIGHT PRs have a re-run in flight at once -// (re-runs already running count), merge-critical PRs (auto-merge set) first, -// then oldest first; writes are at least WRITE_GAP_MS apart. The first -// rate-limit error stops the run. -// -// Only the CI workflows below are re-run. A lane workflow's trigger check -// refuses a re-run github-actions[bot] starts, so a lost lane run is listed in -// the summary for a person instead. A merge-group run is not covered: a re-run -// puts no PR back in the queue, and pr-merge re-enqueues its own PRs once -// their checks pass. +// mid-incident. At most MAX_IN_FLIGHT re-runs are in flight at once (PR CI +// sets and pr-verify runs, counting those already running): merge-critical PRs +// (auto-merge set) first, then oldest PR, then the oldest pr-verify runs. +// Writes are at least WRITE_GAP_MS apart. The first rate-limit error stops the +// run. // // Environment: +// MODE `plan` (read, choose, name records) or `act` (re-run the plan) +// PLAN_PATH the plan file plan writes and act reads +// RECORD_PATH the file plan writes for the record uploads // GH_TOKEN the job's GITHUB_TOKEN (actions: write, pull-requests: read) // GITHUB_REPOSITORY owner/repo // KILL_SWITCH `proceed` from the kill-switch step; anything else stops // STATUS_URL the status page's components endpoint (default githubstatus.com) +// GITHUB_OUTPUT plan writes record-1 to record-3 there (empty when unused) // GITHUB_STEP_SUMMARY the outcome is listed there when set // GITHUB_API_URL -import { appendFileSync } from "node:fs"; +import { appendFileSync, readFileSync, writeFileSync } from "node:fs"; import process from "node:process"; import { setTimeout as delay } from "node:timers/promises"; import { pathToFileURL } from "node:url"; -import { createGitHub, paginate } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; +import { writeOutputs } from "../../actions/check-kill-switch/check-kill-switch.mjs"; +import { createGitHub, paginate, rerunRecordName } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; const STATUS_URL = "https://www.githubstatus.com/api/v2/components.json"; const REPO = /^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/; const SHA = /^[0-9a-f]{40}$/; +const RECORD = /^reconcile-rerun-[1-9][0-9]{0,19}-2-[0-9a-f]{40}$/; export const LOST = new Set(["cancelled", "timed_out", "startup_failure"]); // Workflows with no trigger-actor check, which a github-actions[bot] re-run passes. export const CI_WORKFLOWS = new Set([ @@ -43,6 +57,17 @@ export const CI_WORKFLOWS = new Set([ ".github/workflows/pr-test-windows.yml", ".github/workflows/pr-check-managed-files-hosted.yml", ]); +// Lane workflows whose trigger check takes this workflow's record +// (reconciler-workflow) and that mint no broker token. Each starts on +// workflow_run from main, so its runs are listed by workflow, not by PR head. +export const RECORDED_LANES = [".github/workflows/pr-verify.yml"]; +// The lane step that judges the head. A run that reached it may have judged, +// and whoever can cancel it (actions: write) can stop or delete the verdict +// before it lands, so a re-run would let them have the same head judged again: +// it is left alone. So is a run none of whose jobs started: GitHub cancels a +// pending run when a newer one for the same PR head queues behind it, and that +// newer run judges the head. +export const JUDGE_STEP = "Judge the pull request"; export const MAX_IN_FLIGHT = 3; export const WRITE_GAP_MS = 1000; // An outage loss is re-run within a day; an older one is left to a person (judgment). @@ -77,6 +102,8 @@ function newestByWorkflow(runs) { } const isLost = (r, since) => r.status === "completed" && LOST.has(r.conclusion) && r.run_attempt === 1 && Date.parse(r.created_at) >= since; +const isLostLane = (r, since) => r.event === "workflow_run" && r.head_branch === "main" && SHA.test(r.head_sha ?? "") && isLost(r, since); +const lostNow = (item, r, since) => r?.head_sha === item.head && (item.kind === "lane" ? isLostLane(r, since) : isLost(r, since)); // What one PR's head holds: CI runs to re-run, lost lane runs, and whether a // CI re-run is already in flight. @@ -91,6 +118,164 @@ export function classify(runs, since) { } const byMergeCriticality = (a, b) => Number(b.auto_merge != null) - Number(a.auto_merge != null) || a.number - b.number; +const byAge = (a, b) => Date.parse(a.created_at) - Date.parse(b.created_at) || a.id - b.id; + +// Why the run's first attempt must not be re-run, or undefined when it may be: +// a job started JUDGE_STEP, or no job started any step. Step data, unlike an +// artifact, cannot be deleted. An unreadable listing throws. +async function holdReason(call, repo, runId) { + const listing = await call("GET", `/repos/${repo}/actions/runs/${runId}/attempts/1/jobs?per_page=100`); + if (!Array.isArray(listing?.jobs) || listing.total_count > 100) { + throw new Error("its job listing is not one complete page"); + } + const started = listing.jobs + .flatMap((job) => (Array.isArray(job?.steps) ? job.steps : [])) + .filter((s) => s?.conclusion != null && s.conclusion !== "skipped"); + if (started.some((s) => s.name === JUDGE_STEP)) { + return "after its judge step started"; + } + return started.length === 0 ? "before any of its jobs started" : undefined; +} + +// Reads every open PR's head and the recorded lanes' runs, and returns the +// re-runs to make in order, with a record name on each lane re-run. +async function plan({ repo, call, say, since }) { + const pulls = (await paginate(call, `/repos/${repo}/pulls?state=open&base=main`, 5)) + .filter((p) => p.state === "open" && p.draft !== true && p.base?.ref === "main" && p.head?.repo?.full_name === repo && SHA.test(p.head?.sha ?? "")) + .sort(byMergeCriticality); + let failed = 0; + let inFlight = 0; + const ready = []; + const noteFailure = (what, error) => { + if (error instanceof RateLimited) { + throw error; + } + failed += 1; + console.log(`::error title=reconcile::${what}: ${escapeCommand(error?.message ?? error)}`); + }; + for (const pull of pulls) { + try { + const listing = await call("GET", `/repos/${repo}/actions/runs?head_sha=${pull.head.sha}&per_page=100`); + if (!Array.isArray(listing?.workflow_runs) || listing.total_count > 100) { + throw new Error("its run listing is not one complete page"); + } + const { rerun, lane, inFlight: busy } = classify(listing.workflow_runs, since); + inFlight += busy ? 1 : 0; + for (const r of lane) { + say(`#${pull.number}: lane run ${r.id} (${r.path}) ended ${r.conclusion}; a re-run of this lane needs a person`); + } + if (rerun.length > 0 && !busy) { + ready.push(rerun.map((r) => ({ kind: "ci", pr: pull.number, id: r.id, path: r.path, conclusion: r.conclusion, head: pull.head.sha }))); + } + } catch (error) { + noteFailure(`#${pull.number}`, error); + } + } + const created = encodeURIComponent(`>=${new Date(since).toISOString().replace(/\.\d{3}Z$/, "Z")}`); + const lanes = []; + for (const lanePath of RECORDED_LANES) { + try { + const file = lanePath.slice(lanePath.lastIndexOf("/") + 1); + const listing = await call("GET", `/repos/${repo}/actions/workflows/${file}/runs?created=${created}&per_page=100`); + if (!Array.isArray(listing?.workflow_runs) || listing.total_count > 100) { + throw new Error("its run listing is not one complete page"); + } + const runs = listing.workflow_runs.filter((r) => r.path === lanePath); + inFlight += runs.filter((r) => r.status !== "completed" && r.run_attempt > 1).length; + const unpublished = []; + for (const r of runs.filter((run) => isLostLane(run, since))) { + try { + const held = await holdReason(call, repo, r.id); + if (held) { + say(`lane run ${r.id} (${r.path}) ended ${r.conclusion} ${held}; a re-run needs a person`); + } else { + unpublished.push(r); + } + } catch (error) { + noteFailure(`lane run ${r.id}`, error); + } + } + lanes.push( + ...unpublished + .sort(byAge) + .map((r) => ({ kind: "lane", id: r.id, path: r.path, conclusion: r.conclusion, head: r.head_sha, created_at: r.created_at, record: rerunRecordName(r.id, r.head_sha) })), + ); + } catch (error) { + noteFailure(lanePath, error); + } + } + + // Every item takes one place: a PR's CI set, or one lane run. + let room = Math.max(0, MAX_IN_FLIGHT - inFlight); + const items = []; + for (const set of ready) { + if (room > 0) { + items.push(...set); + room -= 1; + } else { + say(`#${set[0].pr}: deferred, ${MAX_IN_FLIGHT} re-runs in flight`); + } + } + for (const lane of lanes) { + if (room > 0) { + items.push(lane); + room -= 1; + } else { + say(`lane run ${lane.id} (${lane.path}): deferred, ${MAX_IN_FLIGHT} re-runs in flight`); + } + } + return { items, failed }; +} + +// Re-runs each planned item still lost on a fresh read. +async function act({ repo, call, say, since, sleep, items }) { + let failed = 0; + let wrote = false; + for (const item of items) { + const where = item.kind === "ci" ? `#${item.pr}` : "lane"; + try { + if (item.kind === "lane" && item.record !== rerunRecordName(item.id, item.head)) { + throw new Error("the plan names no record for it"); + } + const r = await call("GET", `/repos/${repo}/actions/runs/${item.id}`); + if (!lostNow(item, r, since)) { + say(`${where}: run ${item.id} changed since the listing; left alone`); + continue; + } + if (item.kind === "ci") { + // A push since the listing makes this run obsolete, and re-running it + // would cancel the new head's CI in the PR's concurrency group. + const current = await call("GET", `/repos/${repo}/pulls/${item.pr}`); + if (current?.state !== "open" || current.draft === true || current.head?.sha !== item.head) { + say(`${where}: the PR changed since the listing; run ${item.id} left alone`); + continue; + } + } + if (wrote) { + await sleep(WRITE_GAP_MS); + } + wrote = true; + await call("POST", `/repos/${repo}/actions/runs/${item.id}/rerun`); + say(`${where}: re-ran run ${item.id} (${item.path}), which ended ${item.conclusion}`); + } catch (error) { + if (error instanceof RateLimited) { + throw error; + } + failed += 1; + console.log(`::error title=reconcile::${where} run ${item.id}: ${escapeCommand(error?.message ?? error)}`); + } + } + return failed; +} + +// The plan's record-N outputs, each a record name or empty. +function recordOutputs(items) { + const names = items.filter((item) => item.kind === "lane").map((item) => item.record); + if (names.length > MAX_IN_FLIGHT || !names.every((name) => RECORD.test(name))) { + throw new Error("the plan's record names are malformed"); + } + return Object.fromEntries(Array.from({ length: MAX_IN_FLIGHT }, (_, i) => [`record-${i + 1}`, names[i] ?? ""])); +} export async function main({ env = process.env, github, fetchStatus: readStatus = fetchStatus, sleep = delay, now = Date.now() } = {}) { const lines = []; @@ -100,7 +285,7 @@ export async function main({ env = process.env, github, fetchStatus: readStatus }; const finish = (code) => { if (env.GITHUB_STEP_SUMMARY && lines.length > 0) { - appendFileSync(env.GITHUB_STEP_SUMMARY, `### Lost-run reconciler\n\n${lines.join("\n")}\n`); + appendFileSync(env.GITHUB_STEP_SUMMARY, `### Lost-run reconciler (${env.MODE})\n\n${lines.join("\n")}\n`); } return code; }; @@ -108,21 +293,12 @@ export async function main({ env = process.env, github, fetchStatus: readStatus if (!REPO.test(env.GITHUB_REPOSITORY ?? "")) { throw new Error("GITHUB_REPOSITORY is not owner/repo"); } - if (env.KILL_SWITCH !== "proceed") { - say("the lanes kill switch stops this run; nothing read or re-run"); - return finish(0); + if (env.MODE !== "plan" && env.MODE !== "act") { + throw new Error("MODE is neither plan nor act"); } - let status; - try { - status = actionsStatus(await readStatus(env.STATUS_URL || STATUS_URL)); - } catch (error) { - status = `unreadable (${error?.message ?? error})`; + if (!env.PLAN_PATH) { + throw new Error("PLAN_PATH is not set"); } - if (status !== "operational") { - say(`Actions is ${status ?? "missing from the status page"}; idle until it is operational`); - return finish(0); - } - const repo = env.GITHUB_REPOSITORY; let last; github ??= createGitHub({ token: env.GH_TOKEN, apiUrl: env.GITHUB_API_URL || undefined, onResponse: (r) => (last = r) }); @@ -136,77 +312,53 @@ export async function main({ env = process.env, github, fetchStatus: readStatus throw limited ? new RateLimited(`rate limited at ${args[0]} ${args[1]}`) : error; } }; - const since = now - LOOKBACK_HOURS * 3_600_000; - const pulls = (await paginate(call, `/repos/${repo}/pulls?state=open&base=main`, 5)) - .filter((p) => p.state === "open" && p.draft !== true && p.base?.ref === "main" && p.head?.repo?.full_name === repo && SHA.test(p.head?.sha ?? "")) - .sort(byMergeCriticality); - let failed = 0; - let inFlight = 0; - const ready = []; - for (const pull of pulls) { - try { - const listing = await call("GET", `/repos/${repo}/actions/runs?head_sha=${pull.head.sha}&per_page=100`); - if (!Array.isArray(listing?.workflow_runs) || listing.total_count > 100) { - throw new Error("its run listing is not one complete page"); - } - const { rerun, lane, inFlight: busy } = classify(listing.workflow_runs, since); - inFlight += busy ? 1 : 0; - for (const r of lane) { - say(`#${pull.number}: lane run ${r.id} (${r.path}) ended ${r.conclusion}; a lane re-run needs a person`); - } - if (rerun.length > 0 && !busy) { - ready.push({ pull, rerun }); - } - } catch (error) { - if (error instanceof RateLimited) { - throw error; - } - failed += 1; - console.log(`::error title=reconcile::#${pull.number}: ${escapeCommand(error?.message ?? error)}`); + + if (env.MODE === "act") { + const planned = JSON.parse(readFileSync(env.PLAN_PATH, "utf8")); + if (!Array.isArray(planned?.items) || !Number.isInteger(planned.failed)) { + throw new Error("the plan file does not hold a plan"); } + const failed = await act({ repo, call, say, since, sleep, items: planned.items }); + return finish(planned.failed + failed === 0 ? 0 : 1); } - const room = Math.max(0, MAX_IN_FLIGHT - inFlight); - for (const { pull } of ready.slice(room)) { - say(`#${pull.number}: deferred, ${MAX_IN_FLIGHT} PR CI sets in flight`); - } - let wrote = false; - for (const { pull, rerun } of ready.slice(0, room)) { - for (const listed of rerun) { - try { - // A fresh read, so a run re-run since the listing is left alone. - const r = await call("GET", `/repos/${repo}/actions/runs/${listed.id}`); - if (r?.head_sha !== pull.head.sha || !isLost(r, since)) { - say(`#${pull.number}: run ${listed.id} changed since the listing; left alone`); - continue; - } - // A push since the listing makes this run obsolete, and re-running it - // would cancel the new head's CI in the PR's concurrency group. - const current = await call("GET", `/repos/${repo}/pulls/${pull.number}`); - if (current?.state !== "open" || current.draft === true || current.head?.sha !== pull.head.sha) { - say(`#${pull.number}: the PR changed since the listing; run ${listed.id} left alone`); - continue; - } - if (wrote) { - await sleep(WRITE_GAP_MS); - } - wrote = true; - await call("POST", `/repos/${repo}/actions/runs/${listed.id}/rerun`); - say(`#${pull.number}: re-ran run ${listed.id} (${listed.path}), which ended ${listed.conclusion}`); - } catch (error) { - if (error instanceof RateLimited) { - throw error; - } - failed += 1; - console.log(`::error title=reconcile::#${pull.number} run ${listed.id}: ${escapeCommand(error?.message ?? error)}`); - } + let result = { items: [], failed: 0 }; + const write = () => { + writeFileSync(env.PLAN_PATH, JSON.stringify(result)); + const outputs = recordOutputs(result.items); + if (outputs["record-1"] !== "") { + writeFileSync(env.RECORD_PATH, JSON.stringify(result.items.filter((item) => item.kind === "lane"))); + } + if (env.GITHUB_OUTPUT) { + writeOutputs(outputs, env.GITHUB_OUTPUT); } + }; + if (env.KILL_SWITCH !== "proceed") { + say("the lanes kill switch stops this run; nothing read or re-run"); + write(); + return finish(0); + } + let status; + try { + status = actionsStatus(await readStatus(env.STATUS_URL || STATUS_URL)); + } catch (error) { + status = `unreadable (${error?.message ?? error})`; + } + if (status !== "operational") { + say(`Actions is ${status ?? "missing from the status page"}; idle until it is operational`); + write(); + return finish(0); + } + result = await plan({ repo, call, say, since }); + if (result.items.some((item) => item.kind === "lane") && !env.RECORD_PATH) { + throw new Error("RECORD_PATH is not set"); } - if (lines.length === 0) { + write(); + if (lines.length === 0 && result.items.length === 0) { say("nothing lost"); } - return finish(failed === 0 ? 0 : 1); + return finish(0); } catch (error) { console.log(`::error title=reconcile::${escapeCommand(error?.message ?? error)}; the run stops`); return finish(1); diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs index 85d956d33b..1b7a7d5549 100644 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs @@ -5,13 +5,20 @@ // re-run while githubstatus.com shows Actions as anything but operational. // Run and PR shapes follow a recorded REST response (a cancelled // pr-require-checks run, an open PR); fixtures/components.json is a recorded -// githubstatus.com components response, trimmed. +// githubstatus.com components response, trimmed. A lost pr-verify run is +// re-run only after the plan names its record +// reconcile-rerun--2- for the upload steps; pr-verify run +// shapes follow a recorded workflow_run run (head_branch main, head_sha the +// main tip, no pull_requests), and only when a job of its first attempt started +// but none started the judge step. import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; import { test } from "node:test"; import { createGitHub, GitHubError } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; -import { main } from "./reconcile.mjs"; +import { JUDGE_STEP, main } from "./reconcile.mjs"; const REPO = "melodic-software/claude-code-plugins"; const NOW = Date.parse("2026-10-19T20:00:00Z"); @@ -19,6 +26,8 @@ const STATUS = JSON.parse(readFileSync(new URL("./fixtures/components.json", imp const CHECKS = ".github/workflows/pr-require-checks.yml"; const WINDOWS = ".github/workflows/pr-test-windows.yml"; const REVIEW = ".github/workflows/pr-review-hosted.yml"; +const VERIFY = ".github/workflows/pr-verify.yml"; +const MAIN_TIP = "e".repeat(40); const sha = (n) => String(n).padStart(40, "0"); const statusWith = (actions) => ({ @@ -67,6 +76,16 @@ function fakeGitHub({ pulls, runs = [], fail = {} }) { const found = runs.filter((r) => r.head_sha === listing[1]); return { total_count: found.length, workflow_runs: found }; } + const byWorkflow = /^\/repos\/[^/]+\/[^/]+\/actions\/workflows\/([a-z0-9-]+\.yml)\/runs\?created=%3E%3D2026-10-18T20%3A00%3A00Z&per_page=100$/.exec(path); + if (method === "GET" && byWorkflow) { + const found = runs.filter((r) => r.path === `.github/workflows/${byWorkflow[1]}`); + return { total_count: found.length, workflow_runs: found }; + } + const jobs = /^\/repos\/[^/]+\/[^/]+\/actions\/runs\/([0-9]+)\/attempts\/1\/jobs\?per_page=100$/.exec(path); + if (method === "GET" && jobs) { + const found = byId.get(Number(jobs[1]))?.jobs ?? []; + return { total_count: found.length, jobs: found }; + } const one = /^\/repos\/[^/]+\/[^/]+\/actions\/runs\/([0-9]+)(\/rerun)?$/.exec(path); if (one && method === "GET") { return byId.get(Number(one[1])); @@ -79,33 +98,69 @@ function fakeGitHub({ pulls, runs = [], fail = {} }) { return { github, calls }; } -async function reconcile({ pulls, runs, status = STATUS, env = {}, fail } = {}) { +// Runs the workflow's two steps: plan, then, when the plan step succeeds, +// act. The upload steps between them read the plan's record-N outputs. +async function reconcile({ pulls, runs, status = STATUS, env = {}, fail, github: wrap = (g) => g } = {}) { const { github, calls } = fakeGitHub({ pulls, runs, fail }); const sleeps = []; const order = []; - const tracked = async (method, path, body) => { + const tracked = wrap(async (method, path, body) => { order.push(method === "GET" ? "read" : "write"); return github(method, path, body); - }; - const code = await main({ - env: { GITHUB_REPOSITORY: REPO, KILL_SWITCH: "proceed", ...env }, - github: tracked, - fetchStatus: async () => { - if (status instanceof Error) { - throw status; - } - return status; - }, - sleep: async (ms) => { - sleeps.push(ms); - order.push(`sleep ${ms}`); - }, - now: NOW, }); - const reruns = calls.filter((c) => c.startsWith("POST ")).map((c) => Number(/runs\/([0-9]+)\/rerun$/.exec(c)[1])); - return { code, calls, reruns, sleeps, order }; + const dir = mkdtempSync(path.join(tmpdir(), "reconcile-")); + try { + const files = { + PLAN_PATH: path.join(dir, "plan.json"), + RECORD_PATH: path.join(dir, "record.json"), + GITHUB_OUTPUT: path.join(dir, "output"), + }; + writeFileSync(files.GITHUB_OUTPUT, ""); + const step = (mode) => + main({ + env: { GITHUB_REPOSITORY: REPO, KILL_SWITCH: "proceed", ...files, ...env, MODE: mode }, + github: tracked, + fetchStatus: async () => { + if (status instanceof Error) { + throw status; + } + return status; + }, + sleep: async (ms) => { + sleeps.push(ms); + order.push(`sleep ${ms}`); + }, + now: NOW, + }); + const planCode = await step("plan"); + const planWrites = calls.filter((c) => c.startsWith("POST ")).length; + const outputs = Object.fromEntries( + [...readFileSync(files.GITHUB_OUTPUT, "utf8").matchAll(/^([a-z0-9-]+)<<(\S+)\n([^\n]*)\n\2$/gm)].map((m) => [m[1], m[3]]), + ); + const code = planCode === 0 ? await step("act") : planCode; + const reruns = calls.filter((c) => c.startsWith("POST ")).map((c) => Number(/runs\/([0-9]+)\/rerun$/.exec(c)[1])); + return { code, planCode, planWrites, outputs, calls, reruns, sleeps, order }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } } +// A first-attempt job of pr-verify, lost before (null) or after its judge step +// started; by default it was lost in the trigger check, before the judge step. +const verifyJob = (judgeConclusion = null) => [ + { + name: "verify", + steps: [ + { name: "Set up job", status: "completed", conclusion: "success" }, + { name: "Check the trigger", status: "completed", conclusion: judgeConclusion === null ? "cancelled" : "success" }, + { name: "Judge the pull request", status: judgeConclusion === null ? "pending" : "completed", conclusion: judgeConclusion }, + { name: "Upload the verdict", status: "pending", conclusion: null }, + ], + }, +]; +const verifyRun = (id, opts = {}) => ({ ...run(id, MAIN_TIP, { path: VERIFY, event: "workflow_run", ...opts }), head_branch: "main", jobs: verifyJob() }); +const records = (outputs) => [outputs["record-1"], outputs["record-2"], outputs["record-3"]]; + test("a cancelled CI run on a PR's head is re-run once", async () => { const { code, reruns } = await reconcile({ pulls: [pr(1)], runs: [run(101, sha(1))] }); assert.equal(code, 0); @@ -127,26 +182,20 @@ test("a run already re-fired is not re-fired again", async () => { assert.equal(calls.filter((c) => c.startsWith("POST ")).length, 0); }); +// The fresh read before the write sees the run already re-run. +const rerunSinceListing = (id) => (github) => async (method, apiPath, body) => { + if (method === "GET" && apiPath.endsWith(`/actions/runs/${id}`)) { + const listed = await github(method, apiPath, body); + return { ...listed, run_attempt: 2, status: "queued", conclusion: null }; + } + return github(method, apiPath, body); +}; + test("a run re-fired between the listing and the write is left alone", async () => { - const listed = run(101, sha(1)); - const { github, calls } = fakeGitHub({ pulls: [pr(1)], runs: [listed] }); - let reads = 0; - const code = await main({ - env: { GITHUB_REPOSITORY: REPO, KILL_SWITCH: "proceed" }, - github: async (method, path, body) => { - if (method === "GET" && path.endsWith("/actions/runs/101")) { - reads += 1; - return { ...listed, run_attempt: 2, status: "queued", conclusion: null }; - } - return github(method, path, body); - }, - fetchStatus: async () => STATUS, - sleep: async () => {}, - now: NOW, - }); + const { code, calls, reruns } = await reconcile({ pulls: [pr(1)], runs: [run(101, sha(1))], github: rerunSinceListing(101) }); assert.equal(code, 0); - assert.equal(reads, 1); - assert.equal(calls.filter((c) => c.startsWith("POST ")).length, 0); + assert.equal(calls.filter((c) => c.endsWith("/actions/runs/101")).length, 1); + assert.deepEqual(reruns, []); }); test("a degraded Actions status produces no dispatch and reads nothing from GitHub", async () => { @@ -253,34 +302,149 @@ test("a refused re-run is reported and the rest go ahead", async () => { test("a re-run GitHub accepts with its empty 201 response counts as done", async () => { // The re-run endpoint answers 201 with no body; the real client parses it. - const { github: fake, calls } = fakeGitHub({ pulls: [pr(1)], runs: [run(101, sha(1))] }); - const fetchImpl = async (url, init) => { - const answer = await fake(init.method, url.slice("https://api.example".length)); - return init.method === "POST" ? new Response(null, { status: 201 }) : Response.json(answer); - }; - const code = await main({ - env: { GITHUB_REPOSITORY: REPO, KILL_SWITCH: "proceed" }, - github: createGitHub({ token: "t", apiUrl: "https://api.example", fetchImpl }), - fetchStatus: async () => STATUS, - sleep: async () => {}, - now: NOW, + const { code, reruns } = await reconcile({ + pulls: [pr(1)], + runs: [run(101, sha(1))], + github: (fake) => + createGitHub({ + token: "t", + apiUrl: "https://api.example", + fetchImpl: async (url, init) => { + const answer = await fake(init.method, url.slice("https://api.example".length)); + return init.method === "POST" ? new Response(null, { status: 201 }) : Response.json(answer); + }, + }), }); assert.equal(code, 0); - assert.ok(calls.includes(`POST /repos/${REPO}/actions/runs/101/rerun`)); + assert.deepEqual(reruns, [101]); }); test("a PR whose head moved, or that closed or went draft, after the listing is left alone", async () => { for (const now of [{ head: { ...pr(1).head, sha: sha(2) } }, { state: "closed" }, { draft: true }]) { - const { github, calls } = fakeGitHub({ pulls: [pr(1)], runs: [run(101, sha(1))] }); - const code = await main({ - env: { GITHUB_REPOSITORY: REPO, KILL_SWITCH: "proceed" }, - github: async (method, path, body) => - method === "GET" && path === `/repos/${REPO}/pulls/1` ? { ...pr(1), ...now } : github(method, path, body), - fetchStatus: async () => STATUS, - sleep: async () => {}, - now: NOW, + const { code, reruns } = await reconcile({ + pulls: [pr(1)], + runs: [run(101, sha(1))], + github: (fake) => async (method, p, body) => + method === "GET" && p === `/repos/${REPO}/pulls/1` ? { ...pr(1), ...now } : fake(method, p, body), }); assert.equal(code, 0, JSON.stringify(now)); - assert.equal(calls.filter((c) => c.startsWith("POST ")).length, 0, JSON.stringify(now)); + assert.deepEqual(reruns, [], JSON.stringify(now)); + } +}); + +test("a lost verify run is re-run after the plan names its record, and the plan writes nothing", async () => { + const { code, planWrites, outputs, reruns } = await reconcile({ pulls: [pr(1)], runs: [verifyRun(201)] }); + assert.equal(code, 0); + assert.equal(planWrites, 0); + assert.deepEqual(records(outputs), [`reconcile-rerun-201-2-${MAIN_TIP}`, "", ""]); + assert.deepEqual(reruns, [201]); +}); + +test("a verify run already at attempt 2 is never re-run or recorded", async () => { + const { outputs, reruns } = await reconcile({ pulls: [], runs: [verifyRun(201, { attempt: 2 })] }); + assert.deepEqual(reruns, []); + assert.deepEqual(records(outputs), ["", "", ""]); +}); + +test("a verify run re-run between the plan and the write is left alone", async () => { + const { code, reruns } = await reconcile({ pulls: [], runs: [verifyRun(201)], github: rerunSinceListing(201) }); + assert.equal(code, 0); + assert.deepEqual(reruns, []); +}); + +test("verify runs that succeeded, failed, are older than the lookback, or ran off main are left alone", async () => { + const { reruns, outputs } = await reconcile({ + pulls: [], + runs: [ + verifyRun(201, { conclusion: "success" }), + verifyRun(202, { conclusion: "failure" }), + verifyRun(203, { created: "2026-10-17T19:00:00Z" }), + { ...verifyRun(204), head_branch: "fix/x" }, + verifyRun(205, { event: "workflow_dispatch" }), + ], + }); + assert.deepEqual(reruns, []); + assert.deepEqual(records(outputs), ["", "", ""]); +}); + +test("verify re-runs share the cap of 3 with PR CI sets, CI first, then the oldest verify run", async () => { + // One verify re-run is already in flight, so two places are left. + const { reruns, outputs } = await reconcile({ + pulls: [pr(1)], + runs: [ + run(101, sha(1)), + verifyRun(209, { status: "in_progress", attempt: 2 }), + verifyRun(202, { created: "2026-10-19T19:20:00Z" }), + verifyRun(201, { created: "2026-10-19T19:10:00Z" }), + ], + }); + assert.deepEqual(reruns, [101, 201]); + assert.deepEqual(records(outputs), [`reconcile-rerun-201-2-${MAIN_TIP}`, "", ""]); +}); + +test("at most 3 verify runs are recorded and re-run in one run", async () => { + const { reruns, outputs } = await reconcile({ + pulls: [], + runs: [verifyRun(201), verifyRun(202), verifyRun(203), verifyRun(204)].map((r, i) => ({ ...r, created_at: `2026-10-19T19:0${i}:00Z` })), + }); + assert.deepEqual(reruns, [201, 202, 203]); + assert.deepEqual(records(outputs), [201, 202, 203].map((id) => `reconcile-rerun-${id}-2-${MAIN_TIP}`)); +}); + +test("pr-verify still names its judge step as the reconciler reads it", () => { + const workflow = readFileSync(new URL("../../workflows/pr-verify.yml", import.meta.url), "utf8"); + const stepAt = (name) => { + const escaped = name.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + return workflow.search(new RegExp(`^[ ]*- name: ${escaped}$`, "m")); + }; + const judge = stepAt(JUDGE_STEP); + assert.ok(judge >= 0); + assert.ok(judge < stepAt("Upload the verdict")); +}); + +test("a verify run whose judge step started is never re-run or recorded", async () => { + for (const conclusion of ["success", "failure", "cancelled"]) { + const { code, reruns, outputs } = await reconcile({ pulls: [], runs: [{ ...verifyRun(201), jobs: verifyJob(conclusion) }] }); + assert.equal(code, 0, conclusion); + assert.deepEqual(reruns, [], conclusion); + assert.deepEqual(records(outputs), ["", "", ""], conclusion); + } +}); + +test("a verify run lost after it started but before its judge step is re-run", async () => { + const { reruns } = await reconcile({ + pulls: [], + runs: [ + { ...verifyRun(201), jobs: verifyJob(null) }, + { ...verifyRun(202), jobs: verifyJob("skipped"), created_at: "2026-10-19T19:00:00Z" }, + ], + }); + assert.deepEqual(reruns, [201, 202]); +}); + +test("a verify run none of whose first-attempt jobs started is never re-run or recorded", async () => { + // A pending run concurrency cancelled for a newer one on the same PR head. + for (const jobs of [[], [{ name: "verify", status: "completed", conclusion: "cancelled", steps: [] }]]) { + const { code, reruns, outputs } = await reconcile({ pulls: [], runs: [{ ...verifyRun(201), jobs }] }); + assert.equal(code, 0); + assert.deepEqual(reruns, []); + assert.deepEqual(records(outputs), ["", "", ""]); } }); + +test("a verify run whose jobs cannot be read is not re-run", async () => { + const { code, reruns, outputs } = await reconcile({ + pulls: [], + runs: [verifyRun(201)], + fail: { [`GET /repos/${REPO}/actions/runs/201/attempts/1/jobs?per_page=100`]: new GitHubError(500, "boom") }, + }); + assert.equal(code, 1); + assert.deepEqual(reruns, []); + assert.deepEqual(records(outputs), ["", "", ""]); +}); + +test("a non-operational Actions status records nothing", async () => { + const { outputs, calls } = await reconcile({ pulls: [], runs: [verifyRun(201)], status: statusWith("major_outage") }); + assert.deepEqual(calls, []); + assert.deepEqual(records(outputs), ["", "", ""]); +}); diff --git a/.github/workflows/maintenance-reconcile-lost-runs.yml b/.github/workflows/maintenance-reconcile-lost-runs.yml index 94fe8b5fb5..2223f6a397 100644 --- a/.github/workflows/maintenance-reconcile-lost-runs.yml +++ b/.github/workflows/maintenance-reconcile-lost-runs.yml @@ -1,19 +1,34 @@ name: maintenance-reconcile-lost-runs -# Re-runs PR CI runs a GitHub outage ended without a result, so nobody has to -# re-run them by hand. Every 30 minutes it looks at each open, non-draft pull -# request into main from a branch of this repository. A CI workflow whose -# newest run on the PR's current head ended cancelled, timed_out or +# Re-runs PR CI and verify runs a GitHub outage ended without a result, so +# nobody has to re-run them by hand. Every 30 minutes it looks at each open, +# non-draft pull request into main from a branch of this repository, and at +# the last day's pr-verify runs. A CI workflow whose newest run on the PR's +# current head, or a pr-verify run, ended cancelled, timed_out or # startup_failure, on its first attempt, is re-run once with this job's own # token. The attempt number is the record: a run at attempt 2 was already -# re-run and is never touched again. +# re-run and is never touched again. A pr-verify run whose judge step started, +# or none of whose jobs started, is left to a person, so a head is never judged +# twice. A pr-verify run that ended startup_failure with no jobs is held for +# a person, not re-run. +# +# pr-verify's trigger check trusts a re-run github-actions[bot] starts only +# through this run's record of it: the plan step names an artifact +# reconcile-rerun--2- for each pr-verify run it will re-run, +# the upload steps below write them, and only then does the act step re-run +# those runs. That no other run can upload into this run's artifacts is +# UNVERIFIED: GitHub does not document that it scopes the runtime upload token +# to its own run, and the REST API has no endpoint that creates an artifact. +# A forged record still needs a github-actions[bot] attempt-2 re-run of the +# lane, and the check's actor checks still apply +# (.github/actions/check-trusted-trigger/README.md). # # It does nothing while githubstatus.com shows Actions as anything but # operational, or the page cannot be read, so it adds no load mid-incident. At -# most 3 PRs have a re-run in flight at once, auto-merge PRs first, and its -# writes are at least a second apart. A lost lane run (a review, verify or -# refine lane) is listed in the run summary for a person: those lanes refuse a -# re-run github-actions[bot] starts. No model, and no PR code is checked out. +# most 3 re-runs are in flight at once, auto-merge PRs first, and its writes +# are at least a second apart. A lost review or refine lane run is listed in +# the run summary for a person: those lanes, or the lanes token broker, refuse +# a re-run github-actions[bot] starts. No model, and no PR code is checked out. # Rules: .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs. # # How to stop it: set the organization or repository lanes kill-switch @@ -40,7 +55,7 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 10 permissions: - actions: write # list runs and re-run a lost CI run + actions: write # list runs and re-run a lost run contents: read # check out main pull-requests: read # list the open PRs steps: @@ -64,9 +79,52 @@ jobs: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} - - name: Re-run lost CI runs + - name: Find lost runs + id: plan env: + MODE: plan + PLAN_PATH: ${{ runner.temp }}/reconcile-plan.json + RECORD_PATH: ${{ runner.temp }}/reconcile-record/record.json GH_TOKEN: ${{ github.token }} GITHUB_REPOSITORY: ${{ github.repository }} KILL_SWITCH: ${{ steps.kill.outputs.proceed == 'true' && 'proceed' || '' }} + run: | + mkdir -p "$RUNNER_TEMP/reconcile-record" + node .base/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs + + # One record per pr-verify run the act step re-runs, uploaded first. A + # failed upload fails the job, so that run is not re-run. + - name: Record lane re-run 1 + if: steps.plan.outputs.record-1 != '' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-1 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Record lane re-run 2 + if: steps.plan.outputs.record-2 != '' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-2 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Record lane re-run 3 + if: steps.plan.outputs.record-3 != '' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-3 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Re-run lost runs + env: + MODE: act + PLAN_PATH: ${{ runner.temp }}/reconcile-plan.json + GH_TOKEN: ${{ github.token }} + GITHUB_REPOSITORY: ${{ github.repository }} run: node .base/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs diff --git a/.github/workflows/pr-verify.yml b/.github/workflows/pr-verify.yml index 59f4ab9689..632812b3bd 100644 --- a/.github/workflows/pr-verify.yml +++ b/.github/workflows/pr-verify.yml @@ -87,7 +87,8 @@ jobs: echo "::notice::the lanes kill switch stops this run (reason: ${REASON:-unreadable}); no verdict" # The upstream run's actors, the re-runner and the PR author must all be - # on the trusted-actor list. + # on the trusted-actor list. A re-run github-actions[bot] started passes + # only as the lost-run reconciler's recorded re-run of this run. - name: Check the trigger id: gate if: steps.kill.outputs.proceed == 'true' @@ -95,6 +96,7 @@ jobs: with: trusted-actors-path: .base/.github/standards/trusted-actors/trusted-actors.json triggering-actor: ${{ github.triggering_actor }} + reconciler-workflow: .github/workflows/maintenance-reconcile-lost-runs.yml - name: Resolve the pull request id: pr From ce17844cd597f30ddaeaaea8307207aa64213dcc Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 20 Oct 2026 03:33:15 -0400 Subject: [PATCH 2/3] docs(pr-pipeline): say the reconciler record does not bind the pull request The record name carries the pr-verify run's own head_sha, which is main's tip; the live gate binds the re-run to the pull request. Say so where the record is built, so nobody relies on its SHA for that. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT --- .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs index 527f61c5fb..b35dc86759 100755 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs @@ -195,6 +195,9 @@ async function plan({ repo, call, say, since }) { noteFailure(`lane run ${r.id}`, error); } } + // The record's head is the pr-verify run's own head_sha (main's tip), not + // the pull request's head: the PR binding is the live gate's job + // (check-trusted-trigger and resolve-pr), never this record's. lanes.push( ...unpublished .sort(byAge) From 74d0dbdb2f847790d2c0892107bc725538f4b2e3 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 20 Oct 2026 09:36:33 -0400 Subject: [PATCH 3/3] fix(pr-pipeline): pass the reconciler workflow to check-intent Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT --- .github/workflows/pr-verify.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/pr-verify.yml b/.github/workflows/pr-verify.yml index c318c93521..e47f046253 100644 --- a/.github/workflows/pr-verify.yml +++ b/.github/workflows/pr-verify.yml @@ -108,6 +108,7 @@ jobs: with: trusted-actors-path: .base/.github/standards/trusted-actors/trusted-actors.json triggering-actor: ${{ github.triggering_actor }} + reconciler-workflow: .github/workflows/maintenance-reconcile-lost-runs.yml - name: Resolve the pull request id: pr