Skip to content

fix(ci): re-run ci-status on lanes App pull requests once the kill switch is lifted - #10282

Merged
kyle-sexton merged 8 commits into
mainfrom
fix/rerun-ci-status-after-kill-switch
Oct 26, 2026
Merged

kyle-sexton merged 8 commits into
mainfrom
fix/rerun-ci-status-after-kill-switch

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #9522

Summary

While CLAUDE_LANES_DISABLED or CLAUDE_LANES_REPO_DISABLED is on, ci-status fails every lanes App pull request (the lanes-hold step in pr-require-checks.yml). Nothing re-ran it once the switch was lifted, so open lane PRs stayed red until a push or a manual re-run. The existing reconciler did not cover the case: its flake re-run skips a run where only ci-status failed (infra-failure.mjs firstFailure).

Fix

maintenance-reconcile-lost-runs gains one more re-run kind, switch. It reuses the reconciler's half-hourly schedule, record uploads, cap and act-time re-check. It adds no new workflow, token, permission or secret.

  • It applies to an open lanes App pull request (author id from the trusted list by merge.agent-identity, type Bot) whose newest pr-require-checks pull_request run meets all of these:
    • it failed on attempt 1;
    • ci-status was its only failed job;
    • ci-status's check-run annotations include the lanes kill switch is on.
  • The run then has its failed jobs re-run. The re-run is recorded as reconcile-rerun-<id>-2-<head>, so pr-verify trusts the attempt 2.
  • No lookback applies, because the switch may stay on for days.
  • The reconciler still reads nothing while the switch is on, so nothing polls any pull request then.
  • A run already at attempt 2 or later is left for a person, because pr-verify trusts only a recorded attempt 2.
  • These re-runs share the cap of 3 per run, merge-critical first. With N such pull requests, all are re-run within ceil(N/3) half-hourly runs.

Criteria:

  • Lifting the kill switch leads to a ci-status re-run on each open lanes App pull request the switch failed, at most 3 per reconciler run (one half-hourly interval for up to 3 pull requests).
  • No pull request is read while the switch is on.
  • A ci-status failure for any other reason (a body or member hold, a lane failure, a title) is not re-run by this kind.

Verification

  • node --test in .github/scripts/maintenance-reconcile-lost-runs: 106 pass, 0 fail (reconcile 82, including 7 new tests).
    • The new tests cover:
      • a re-run past the lookback, with its record;
      • nothing read while the switch is on;
      • no re-run for a missing note, another author, a non-Bot author, attempt 2, or no lane identity or trusted list;
      • a failed contract-only newest run;
      • a lane also failing, which takes the existing flake re-run once;
      • the cap and the act-time re-check.
    • One test pins the note text and the exit 1 in pr-require-checks.yml's lanes-hold step.
  • actionlint and zizmor --offline on maintenance-reconcile-lost-runs.yml: clean.
  • Suites that read the reconciler or workflows, each with 0 failures:
    • check-trusted-trigger: 362 pass
    • rate-guard: 62 pass
    • fix-ci-dispatch: 48 pass
    • workflow-gates: 25 pass
    • resolve-config callers: 182 pass
    • check-killswitch-hoist.test.sh: 29 pass
    • workflow-self-paths.test.sh: 35 pass
    • check-changelog-fragments.test.sh: 68 pass
  • check-changelog-fragments.sh --check and --check-required origin/main: pass.
  • Post-merge: after the next time the kill switch is lifted, the next reconciler run's summary lists "ci-status failed while the lanes kill switch was on" for each such pull request.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits October 25, 2026 23:42
…itch is lifted

While CLAUDE_LANES_DISABLED or CLAUDE_LANES_REPO_DISABLED is on, ci-status
fails every lanes App pull request, and nothing re-ran it once the switch
was lifted. The lost-run reconciler, which reads nothing while the switch
is on, now re-runs the failed ci-status of each open lanes App pull request
whose newest pr-require-checks run failed on its first attempt only in
ci-status, with ci-status's kill-switch annotation. The re-run is recorded
as attempt 2 so pr-verify trusts it, and shares the cap of 3 per run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T13:53:38.630614Z 7dc759b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed bad00d9bc082853774f6bd912968341f1a7bbc22: 0 finding(s), 0 as inline comments.

No security findings. critical-findings: 0

What I read: the whole diff (6 files) and the PR context. I also read loadTrustedList in check-trusted-trigger.mjs and the AGGREGATE_JOBS and FAILED exports in infra-failure.mjs.

Trusted-path touch: the only trusted-path file changed is .github/workflows/pr-require-checks.yml, in the lanes-hold step.

  • The change is limited to comments and to the text of the ::error:: message.
  • The condition [ "${LANES_DISABLED:-}" != false ] || [ "${LANES_REPO_DISABLED:-}" != false ] is unchanged, and so is the exit 1 that follows it.
  • The kill switch still fails closed, so no merge hold is weakened.
  • A new test pins the note text and the exit 1 in that step.

New switch re-run kind in reconcile.mjs: I checked it for widened authority and found none.

  • Author check: it applies only when the PR author's id equals the id that trusted.logins.get(LANE_IDENTITY) returns from the trusted-actor list, and the author type is Bot. If LANE_IDENTITY or TRUSTED_ACTORS_PATH is unset, laneApp stays undefined and nothing is re-run (fails closed).
  • Run conditions: the newest pr-require-checks pull_request run must have failed on attempt 1, with ci-status as its only failed job.
  • Annotation text: ci-status's failure annotation must contain the kill-switch text. Another hold's message could echo that text, but a re-run only re-evaluates every hold under the current variables. So a spoofed note cannot make a held PR pass.
  • Kill switch: the reconciler reads nothing unless the KILL_SWITCH step outputs proceed.
  • Head check: the existing act-time check requires the head to be unchanged and the run to still be a failed attempt 1. A stale or moved head is not re-run.
  • Cap and record: the re-run shares the cap of 3, and it reuses the existing record name reconcile-rerun-<id>-2-<head>, so pr-verify trusts it by the same rule as the existing infra re-run.
  • Permissions: the change adds no token, permission, secret or workflow trigger. It uses the same rerun-failed-jobs call and the same actions: write token.
  • No lookback: the missing lookback only applies to runs that meet all of the conditions above, and the cap bounds the cost.

Manifests, instruction surfaces, hold: the diff has no plugin-manifest changes and no instruction-surface deletions. No CRITICAL finding, so no hold. The only added spend is Actions re-runs within the existing cap, which falls under prepaid or included usage.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed bad00d9bc082853774f6bd912968341f1a7bbc22: 0 finding(s), 0 as inline comments.

No findings. I read the full diff, the PR context and closing issue #9522, and the surrounding code in reconcile.mjs, infra-failure.mjs, maintenance-reconcile-lost-runs.yml and the pr-require-checks.yml lanes-hold step. I could not run the tests in this lane (no shell). The pass counts in the PR body are author-claimed and unverified.

Closing issue #9522 ("Lifting the kill switch leads to a ci-status re-run on each open lanes App pull request within one scheduled interval ... with no per-PR polling while the switch is on"): met.

  • plan adds the switch kind. It re-runs the failed jobs of the newest pr-require-checks run when all of these hold:
    • the PR author is the lanes App (id from the trusted list via LANE_IDENTITY, type Bot);
    • the run failed on attempt 1;
    • ci-status was the only failed job;
    • its annotations include the lanes kill switch is on.
  • The re-run is recorded as reconcile-rerun-<id>-2-<head> through withRecord, and act re-checks the run and the PR head before it re-runs.
  • The reconciler's check-kill-switch step reads the same two variables as the lanes-hold step (CLAUDE_LANES_DISABLED and CLAUDE_LANES_REPO_DISABLED, both must be exactly false). So no PR is read while the switch is on, and a re-run does not spend the one attempt 2 while the switch is still on.
  • The tests could fail if this broke: re-run past the lookback with its record, nothing read while the switch is on, the negative cases, and the cap of 3 with the act-time re-check.
  • One test pins the note text and the exit 1 in the workflow step.

Checks I made:

  • onlyAggregateFailed accepts a run only when every failed job is ci-status, and the set.length === 0 guard keeps the existing flake and infra re-runs ahead of switch.
  • The withAnnotations fallback fetches annotations when ci-status steps list in full. Reading the same annotation endpoint explicitly in that case is correct.
  • scripts/ci-lanes-pr-hold.test.sh matches the kill-switch error by substring, so the reworded ::error:: message in pr-require-checks.yml does not break it.
  • The new switch kind is added to the act-time lostNow and the rerun-failed-jobs branch. It has no other consumers to update.

Spend: this adds at most 3 ci-status re-runs per half-hourly reconciler run, which use no model and no paid resource. It is not a hold.

The changed paths under the trusted globs are .github/workflows/pr-require-checks.yml and docs/conventions/pr-pipeline/README.md. The workflow change is a comment and an error-message text edit. The README change is documentation. Neither changes what the step gates.

Comment thread .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs
Comment thread docs/conventions/pr-pipeline/README.md
@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 26, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane stopped on this pull request and needs a person: the merge stages a change under .github/workflows/, which the lane does not push.

  • Reason: workflow-staged
  • Resume key: 7dc759bc700b6913cac12b30ae2cb76a2a95f040 (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=10282 and activity=resolve-conflicts:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=10282 -f activity=resolve-conflicts

kyle-sexton and others added 5 commits October 26, 2026 12:44
…fter-kill-switch

reconcile.mjs takes both import sets: the PR's AGGREGATE_JOBS and FAILED
for the switch kind, and main's LOST for #10269's lost-again summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lanes-hold step's comment and error text in pr-require-checks.yml now
name maintenance-reconcile-lost-runs as what re-runs ci-status once both
switches read false, as #9522's criteria require. The pinned prefix
"the lanes kill switch is on (" and the following exit 1 are unchanged.

Refs: #9522

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 26, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit f8768ad Oct 26, 2026
64 of 65 checks passed
@kyle-sexton
kyle-sexton deleted the fix/rerun-ci-status-after-kill-switch branch October 26, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ci): re-run ci-status on open lanes App pull requests when the kill switch is lifted

1 participant