From 6f46d68adb8ed0e43470de4cc55d330b70307496 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Wed, 12 Aug 2026 17:25:32 +0200 Subject: [PATCH 1/3] ci: fix the #7205 relapse on the scheduled gate arm and reap dead PR runs #7856 moved ten expensive gates' main-line arm from `push: branches: [main]` to a staggered six-hourly `schedule:`. Their concurrency groups key on `github.event_name == 'push' && github.sha || github.ref` -- #7205's fix, guarded on the event being `push`. With the main-line arm now `schedule`, the guard stops matching and the group falls through to `github.ref`, constant `refs/heads/main`. GitHub keeps at most one PENDING run per concurrency group and cancels the previously pending one when a new run enters, regardless of `cancel-in-progress`. Measured 2026-08-12, identically on all ten gates: oldest run `queued` holding the group, the next two `cancelled` with `jobs: 0`, newest `pending`. `gate-freshness` itself -- the alarm for this -- was cancelled the same way. Groups are now keyed on `github.run_id` for every non-pull-request event. PR runs keep the shared per-ref group and keep coalescing. `gc_gate_wiring_check.py` gains `check_schedule_group`, swept over all 31 workflows rather than just the GC gates. It found ten further workflows with the same latent constant group -- including `test.yml`'s nightly safety net and `soak-autofix`, whose group was a bare constant string -- all fixed here. Five new self-test cases; the first is the sabotage case, since the existing CLEAN fixture carries the bad shape. Capacity half: 1,529 runs queued against 12-14 concurrent, of which 794 were `pull_request` runs over 63 head branches -- 61 of which no longer existed. Roughly 790 runs, 51% of the queue, were dead work for already-merged PRs sitting in front of the `main` gates. `scripts/reap_stale_ci_runs.py` + `ci-queue-reaper.yml` cancel QUEUED pull-request runs with no open PR; dry-run by default, `--max` capped, and structurally unable to touch a push/schedule/tag/dispatch run. `zizmor`'s `push: main` arm is path-filtered to `.github/**` and gains the concurrency block it never had. Refs #7966 --- .github/workflows/auto-opt-app-patterns.yml | 14 +- .github/workflows/benchmark.yml | 7 +- .github/workflows/ci-queue-reaper.yml | 86 ++++++++ .github/workflows/container-tests.yml | 7 +- .github/workflows/coverage.yml | 7 +- .github/workflows/eh-transport.yml | 14 +- .github/workflows/feature-matrix.yml | 7 +- .github/workflows/gate-freshness.yml | 14 +- .github/workflows/gc-moving-witnesses.yml | 14 +- .github/workflows/gc-native-roots.yml | 14 +- .github/workflows/gc-parse-churn-gate.yml | 14 +- .../workflows/gc-ptr-shape-off-witness.yml | 14 +- .github/workflows/gc-ratchet.yml | 14 +- .github/workflows/gc-root-dominance.yml | 14 +- .github/workflows/llvm-inprocess.yml | 14 +- .github/workflows/node-compat-matrix.yml | 7 +- .github/workflows/node-core-subset.yml | 7 +- .github/workflows/node-suite-guard.yml | 7 +- .github/workflows/npm-package-sweep.yml | 7 +- .github/workflows/security-audit.yml | 14 +- .github/workflows/soak-autofix.yml | 7 +- .github/workflows/test.yml | 7 +- .github/workflows/tls-budget.yml | 14 +- .github/workflows/zizmor.yml | 18 ++ changelog.d/7966-gate-starvation.md | 63 ++++++ docs/src/testing/ci-gate-scheduling.md | 92 +++++++- gc-handoff/GATES-NOTES.md | 126 +++++++++++ scripts/gc_gate_wiring_check.py | 116 +++++++++- scripts/reap_stale_ci_runs.py | 200 ++++++++++++++++++ 29 files changed, 909 insertions(+), 30 deletions(-) create mode 100644 .github/workflows/ci-queue-reaper.yml create mode 100644 changelog.d/7966-gate-starvation.md create mode 100644 gc-handoff/GATES-NOTES.md create mode 100755 scripts/reap_stale_ci_runs.py diff --git a/.github/workflows/auto-opt-app-patterns.yml b/.github/workflows/auto-opt-app-patterns.yml index 05dac73f1c..27f6a276d8 100644 --- a/.github/workflows/auto-opt-app-patterns.yml +++ b/.github/workflows/auto-opt-app-patterns.yml @@ -63,7 +63,19 @@ permissions: contents: read concurrency: - group: auto-opt-app-patterns-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: auto-opt-app-patterns-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 615386c92c..18a8310f2c 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -13,7 +13,12 @@ on: # Don't cancel in-progress benchmark runs — we want complete samples concurrency: - group: bench-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: bench-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false env: diff --git a/.github/workflows/ci-queue-reaper.yml b/.github/workflows/ci-queue-reaper.yml new file mode 100644 index 0000000000..fbdc2b48d5 --- /dev/null +++ b/.github/workflows/ci-queue-reaper.yml @@ -0,0 +1,86 @@ +name: CI Queue Reaper + +# Cancels QUEUED pull-request runs whose pull request is already closed. +# +# WHY: GitHub does not reliably cancel a queued run when its PR merges and the +# branch auto-deletes. Perry squash-merges, auto-deletes branches, and fans each +# PR out to ~11 workflows, so a busy day leaves hundreds of runs queued against +# branches that no longer exist. They cannot gate anything -- the code is +# already in `main` -- but they hold runner slots ahead of the six-hourly `main` +# gates, which is how those gates go dark. +# +# Measured 2026-08-12 (#7966): 1,529 queued runs; 794 were `pull_request` runs +# across 63 head branches, of which 61 no longer existed. ~790 runs -- 51% of +# the whole queue -- were dead work sitting in front of ten `main` gates that +# had not completed in 32+ hours. +# +# THIS WORKFLOW IS NOT A GATE. It cannot fail a merge and is not a required +# context. It is a janitor. The policy it enforces, its guard rails and its +# self-test live in scripts/reap_stale_ci_runs.py -- read that before changing +# anything here. In particular it only ever touches `event == pull_request` +# runs in state `queued` whose branch has no OPEN PR, so a `push`, `schedule`, +# tag or dispatch run is structurally out of reach. +# +# BOOTSTRAP NOTE: this job queues like everything else, so it cannot dig the +# repo out of an already-saturated queue on its own. The first drain is a +# manual `python3 scripts/reap_stale_ci_runs.py --apply` (or a +# `workflow_dispatch` of this workflow with `apply=true`); the schedule then +# keeps the queue clear. + +on: + schedule: + # Every 30 minutes, off the hour and half-hour -- :00 and :30 are the most + # contended slots on GitHub's cron scheduler and the most likely to be + # dropped or delayed. + - cron: "13,43 * * * *" + workflow_dispatch: + inputs: + apply: + description: "Actually cancel (unchecked = dry run)" + type: boolean + default: false + +permissions: {} + +concurrency: + # #7966: keyed per RUN. A constant group across scheduled runs lets only the + # first one execute -- GitHub keeps at most one PENDING run per group and + # cancels the rest with `jobs: 0`, regardless of `cancel-in-progress`. That is + # the exact bug this workflow exists to clean up after; reproducing it here + # would be its own joke. Enforced by scripts/gc_gate_wiring_check.py. + group: ci-queue-reaper-${{ github.run_id }} + cancel-in-progress: false + +jobs: + reap: + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + # Cancelling a run is an Actions write. `pull-requests: read` backs the + # open-PR list that protects every live PR from being reaped. + actions: write + pull-requests: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # The policy check runs unconditionally, including on a dry run. A janitor + # that cancels things must be able to prove its guard rails still bind + # before it is allowed to touch the queue. + - name: Self-test the reaping policy + run: python3 scripts/reap_stale_ci_runs.py --self-test + + - name: Reap stale queued pull-request runs + env: + GH_TOKEN: ${{ github.token }} + # Scheduled sweeps apply; a manual dispatch applies only if asked. + APPLY: ${{ github.event_name == 'schedule' || inputs.apply }} + run: | + set -euo pipefail + if [[ "$APPLY" == "true" ]]; then + python3 scripts/reap_stale_ci_runs.py --apply + else + python3 scripts/reap_stale_ci_runs.py + fi diff --git a/.github/workflows/container-tests.yml b/.github/workflows/container-tests.yml index 216562e6b9..d8bf1dbd75 100644 --- a/.github/workflows/container-tests.yml +++ b/.github/workflows/container-tests.yml @@ -63,7 +63,12 @@ on: options: ["true", "false"] concurrency: - group: container-tests-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: container-tests-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: true env: diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index d80962e1ea..3bcdc86df8 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -16,7 +16,12 @@ permissions: contents: read concurrency: - group: coverage-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: coverage-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: true env: diff --git a/.github/workflows/eh-transport.yml b/.github/workflows/eh-transport.yml index 07e0047706..b037a7e3ad 100644 --- a/.github/workflows/eh-transport.yml +++ b/.github/workflows/eh-transport.yml @@ -41,7 +41,19 @@ on: workflow_dispatch: concurrency: - group: eh-transport-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: eh-transport-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} # One group per main COMMIT, cancelling PR runs only. `cancel-in-progress: # false` alone does not protect a `main` run: GitHub allows at most one # PENDING run per group and cancels the previously pending one when a new run diff --git a/.github/workflows/feature-matrix.yml b/.github/workflows/feature-matrix.yml index 3c8e4a253e..12d53a196e 100644 --- a/.github/workflows/feature-matrix.yml +++ b/.github/workflows/feature-matrix.yml @@ -18,7 +18,12 @@ permissions: contents: read concurrency: - group: feature-matrix-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: feature-matrix-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false env: diff --git a/.github/workflows/gate-freshness.yml b/.github/workflows/gate-freshness.yml index 4b74dbc660..0366f0b02a 100644 --- a/.github/workflows/gate-freshness.yml +++ b/.github/workflows/gate-freshness.yml @@ -43,7 +43,19 @@ concurrency: # One sweep at a time. Unlike the gates this watches, coalescing is correct here: # the freshness verdict is a function of "now", so a superseded run had nothing # unique to say. PR runs supersede themselves; scheduled runs queue. - group: gate-freshness-${{ github.event_name }}-${{ github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gate-freshness-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: diff --git a/.github/workflows/gc-moving-witnesses.yml b/.github/workflows/gc-moving-witnesses.yml index 6c12b7a603..ae9db34abf 100644 --- a/.github/workflows/gc-moving-witnesses.yml +++ b/.github/workflows/gc-moving-witnesses.yml @@ -130,7 +130,19 @@ concurrency: # enters, regardless of that setting. That is #7205, measured on gc-ratchet — # three consecutive `main` runs cancelled with `jobs: []`, zero executions. # Keying push runs on the SHA gives every merged commit a group of its own. - group: gc-moving-witnesses-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gc-moving-witnesses-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/gc-native-roots.yml b/.github/workflows/gc-native-roots.yml index a652fbf2d7..eaac32e1df 100644 --- a/.github/workflows/gc-native-roots.yml +++ b/.github/workflows/gc-native-roots.yml @@ -127,7 +127,19 @@ on: workflow_dispatch: concurrency: - group: gc-native-roots-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gc-native-roots-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} # Same shape as llvm-inprocess (#7357), and for the same measured reason. # # This workflow had NO concurrency group at all, so nothing ever superseded a diff --git a/.github/workflows/gc-parse-churn-gate.yml b/.github/workflows/gc-parse-churn-gate.yml index 1dcd7eb859..3b88af1414 100644 --- a/.github/workflows/gc-parse-churn-gate.yml +++ b/.github/workflows/gc-parse-churn-gate.yml @@ -97,7 +97,19 @@ concurrency: # cancel a pending `main` run the moment a new one enters the SAME group -- # #7205, measured on gc-ratchet with three consecutive `main` runs # cancelled and zero executed). PR runs are cancelled on superseding pushes. - group: gc-parse-churn-gate-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gc-parse-churn-gate-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/gc-ptr-shape-off-witness.yml b/.github/workflows/gc-ptr-shape-off-witness.yml index d892fcd183..894ef0e766 100644 --- a/.github/workflows/gc-ptr-shape-off-witness.yml +++ b/.github/workflows/gc-ptr-shape-off-witness.yml @@ -92,7 +92,19 @@ permissions: concurrency: # One group per main COMMIT, cancelling PR runs only — see # gc-moving-witnesses.yml's comment for the #7205 rationale this mirrors. - group: gc-ptr-shape-off-witness-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gc-ptr-shape-off-witness-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/gc-ratchet.yml b/.github/workflows/gc-ratchet.yml index b4ec936d36..fac8cfc322 100644 --- a/.github/workflows/gc-ratchet.yml +++ b/.github/workflows/gc-ratchet.yml @@ -77,7 +77,19 @@ concurrency: # eventually gets its own answer. PR runs keep sharing a per-ref group and # keep superseding themselves, which is still correct — only the head # commit's result gates the merge. - group: gc-ratchet-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gc-ratchet-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/gc-root-dominance.yml b/.github/workflows/gc-root-dominance.yml index d8b96be8cd..6038b925a0 100644 --- a/.github/workflows/gc-root-dominance.yml +++ b/.github/workflows/gc-root-dominance.yml @@ -88,7 +88,19 @@ concurrency: # gc-ratchet, whose shape this file copied). Keying push runs on the SHA gives # every merged commit a group of its own, so no two main runs can contend. # Same reasoning as gc-ratchet.yml, which carries the full writeup. - group: gc-root-dominance-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: gc-root-dominance-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/llvm-inprocess.yml b/.github/workflows/llvm-inprocess.yml index 7eb5208bb6..bb0ecb3789 100644 --- a/.github/workflows/llvm-inprocess.yml +++ b/.github/workflows/llvm-inprocess.yml @@ -29,7 +29,19 @@ on: workflow_dispatch: concurrency: - group: llvm-inprocess-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: llvm-inprocess-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} # One group per main COMMIT, cancelling PR runs only. `cancel-in-progress: # false` alone does not protect a `main` run: GitHub allows at most one # PENDING run per group and cancels the previously pending one when a new run diff --git a/.github/workflows/node-compat-matrix.yml b/.github/workflows/node-compat-matrix.yml index 25d7b4bae8..ee40e1e55f 100644 --- a/.github/workflows/node-compat-matrix.yml +++ b/.github/workflows/node-compat-matrix.yml @@ -29,7 +29,12 @@ permissions: contents: read concurrency: - group: node-compat-matrix-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: node-compat-matrix-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false env: diff --git a/.github/workflows/node-core-subset.yml b/.github/workflows/node-core-subset.yml index cb78b0ad93..761733d544 100644 --- a/.github/workflows/node-core-subset.yml +++ b/.github/workflows/node-core-subset.yml @@ -32,7 +32,12 @@ permissions: contents: read concurrency: - group: node-core-subset-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: node-core-subset-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false env: diff --git a/.github/workflows/node-suite-guard.yml b/.github/workflows/node-suite-guard.yml index 3e10b87526..cb8eeb9298 100644 --- a/.github/workflows/node-suite-guard.yml +++ b/.github/workflows/node-suite-guard.yml @@ -23,7 +23,12 @@ permissions: contents: read concurrency: - group: node-suite-guard-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: node-suite-guard-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false env: diff --git a/.github/workflows/npm-package-sweep.yml b/.github/workflows/npm-package-sweep.yml index 7e291b0954..6c8766bce1 100644 --- a/.github/workflows/npm-package-sweep.yml +++ b/.github/workflows/npm-package-sweep.yml @@ -23,7 +23,12 @@ permissions: contents: read concurrency: - group: npm-package-sweep-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: npm-package-sweep-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false env: diff --git a/.github/workflows/security-audit.yml b/.github/workflows/security-audit.yml index 8eb269943c..b492494247 100644 --- a/.github/workflows/security-audit.yml +++ b/.github/workflows/security-audit.yml @@ -21,7 +21,19 @@ concurrency: # group and cancels the previously pending one when a new run enters, # regardless of that setting (#7205). Keying push runs on the SHA gives every # merged commit a group of its own. - group: security-audit-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: security-audit-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: diff --git a/.github/workflows/soak-autofix.yml b/.github/workflows/soak-autofix.yml index 9aa044154e..55380b78f2 100644 --- a/.github/workflows/soak-autofix.yml +++ b/.github/workflows/soak-autofix.yml @@ -19,7 +19,12 @@ permissions: {} # One run at a time: overlapping runs would race on the bot branch # force-push and the open-PR check. concurrency: - group: soak-autofix + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: soak-autofix-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: false jobs: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 49babe9233..5c08ebe694 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -46,7 +46,12 @@ on: # only backstop for integration-suite regressions a scoped PR run can't see, so # it must always reach a conclusion. concurrency: - group: test-${{ github.event_name }}-${{ github.ref }} + # #7966: keyed per RUN for non-PR events. A group that is constant across + # scheduled runs lets only the first one execute -- GitHub keeps at most one + # PENDING run per group and cancels the rest with `jobs: 0`, regardless of + # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded + # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. + group: test-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/tls-budget.yml b/.github/workflows/tls-budget.yml index eb0be91407..01cc46964e 100644 --- a/.github/workflows/tls-budget.yml +++ b/.github/workflows/tls-budget.yml @@ -72,7 +72,19 @@ permissions: contents: read concurrency: - group: tls-budget-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} + # ***#7966: KEY EVERY MAIN-LINE RUN ON `github.run_id`, NOT `github.sha`.*** + # The previous expression read `github.event_name == 'push' && github.sha || + # github.ref`. That was #7205's fix and it keyed on the event being `push` -- + # correct while the main-line arm WAS `push: branches: [main]`. #7856 moved the + # main-line arm to `schedule:`, which falls through to `github.ref` (constant + # `refs/heads/main`), so every scheduled run shared one group again and #7205 + # came straight back. Measured 2026-08-12 on all ten scheduled gates, the same + # shape every time: oldest run `queued` holding the group, the two after it + # `cancelled` with `jobs: 0`, newest `pending`. `github.run_id` is unique per + # run, so schedule / tag-push / workflow_dispatch each get a group of their own + # and none can supersede another. PR runs keep the shared per-ref group and + # keep superseding themselves, which is still what we want. + group: tls-budget-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: diff --git a/.github/workflows/zizmor.yml b/.github/workflows/zizmor.yml index c4231bb49a..582c70a11c 100644 --- a/.github/workflows/zizmor.yml +++ b/.github/workflows/zizmor.yml @@ -3,11 +3,29 @@ name: zizmor on: push: branches: [main] + # #7966: the push arm was UNFILTERED while the PR arm was already scoped to + # `.github/**`, so every merge queued a zizmor run whose subject could not + # have changed. Measured 2026-08-12: 90 queued `push`/`main` zizmor runs, + # ~6% of a 1,529-deep queue that had ten `main` gates dark behind it. This + # is a static analyser over workflow files; a merge that touches no workflow + # file has nothing for it to say. The PR arm (unchanged) still scans every + # workflow edit before it can merge, and this arm still covers a direct + # admin push to `main`, which is the case the PR arm structurally misses. + paths: ['.github/**'] pull_request: paths: ['.github/**'] permissions: {} +concurrency: + # This workflow had NO concurrency block, so superseded PR pushes never + # coalesced and every push to a busy branch added a run that nothing would + # ever read. Per-ref for PRs (supersede), per-RUN for `main` pushes -- a + # constant group would let only the first main-line run execute and cancel the + # rest with `jobs: 0` (#7205/#7966). Enforced by scripts/gc_gate_wiring_check.py. + group: zizmor-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + jobs: zizmor: runs-on: ubuntu-latest diff --git a/changelog.d/7966-gate-starvation.md b/changelog.d/7966-gate-starvation.md new file mode 100644 index 0000000000..1d5e1af6ac --- /dev/null +++ b/changelog.d/7966-gate-starvation.md @@ -0,0 +1,63 @@ +### CI: eleven post-merge gates were dark — #7205 relapsed on the arm #7856 created + +`gate-freshness.yml` filed #7966 naming eleven gates with no recent successful `main` +run. The issue's own hypothesis was starvation. That is half right, and the half it +misses is a bug we have now shipped three times. + +**The relapse.** Every scheduled gate carried +`group: -${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }}`. +The `github.sha` arm was #7205's fix, and it is guarded on the event being `push` — +correct while the main-line arm *was* `push: branches: [main]`. #7856 moved that arm to +`schedule:`, the guard stopped matching, and the expression fell through to `github.ref`, +constant `refs/heads/main`. All scheduled runs of a gate shared one group again. + +GitHub allows at most one PENDING run per concurrency group and cancels the previously +pending one when a new run enters, *regardless of `cancel-in-progress`* — the finding +#7205 was measured by. Observed 2026-08-12, identically on all ten gates: oldest run +`queued` holding the group (20h in the runner queue), the next two `cancelled` with +`jobs: 0`, newest `pending`. `gate-freshness` itself — the alarm, documented as built so +it "cannot be starved by the condition it is alarming about" — was cancelled the same +way. Concurrency groups are now keyed on `github.run_id` for every non-pull-request +event, so schedule / tag-push / dispatch runs can never supersede one another. PR runs +keep the shared per-ref group and keep coalescing. + +`scripts/gc_gate_wiring_check.py` gains `check_schedule_group`, swept over all 31 +workflow files rather than just the GC gates, since the hazard reached `gate-freshness` +and `test.yml`'s nightly safety net too. It found ten further workflows already carrying +the same latent constant group; all are fixed here. Five new self-test cases, the first +of which is the sabotage case — the existing CLEAN fixture *has* the bad shape, so a +checker that could not fail on it would be worthless. `lint` is a required context, so a +fourth relapse is a red build. + +**The capacity half.** Measured queue: 1,529 runs queued against 12–14 concurrent. 794 +were `pull_request` runs spread over 63 head branches — and 61 of those branches no +longer existed. GitHub does not reliably cancel a queued run when its PR merges and the +branch auto-deletes, so roughly 790 runs (51% of the entire queue) were dead work pinned +in front of ten six-hourly `main` gates. New `scripts/reap_stale_ci_runs.py` + +`ci-queue-reaper.yml` cancel QUEUED `pull_request` runs that have no open PR: dry-run by +default, `--max` cap, and structurally unable to touch a `push`, `schedule`, tag or +dispatch run. Keyed on open PRs rather than branch existence so fork PRs stay protected. +`zizmor`'s `push: main` arm is path-filtered to `.github/**` (the PR arm already was) and +gains the concurrency block it never had. + +**What was actually unprotected.** gc-ratchet dark 33.6h across 37 collector-touching +merges; gc-root-dominance dark 33.6h across 44 codegen merges with an empty allowlist; +three more dark 57.3h. But the honest finding is that **gc-ratchet would not have caught +#7965 even had it run**: its gating metric set has no full-mark-sweep count, the counter +that found the regression (`collection_kind:"full"` 0 → 2) is not one of its metrics and +no gate in the repo ratchets one, the two dimensions that did move (wall time, RSS) are +explicitly `"gating": false` in the `shared_ci` profile CI uses, and its probe corpus is +not the gc-handoff workloads that showed it. That class stays uncovered; #7965's third +ask is still open. + +Two of the eleven are not starved at all. `gc-native-roots` has **never had a successful +run** on any branch — three of four arms fail with three distinct causes. `llvm-inprocess` +failed its last three `main` runs and, worse, its PR "successes" show +`native-backend: skipped` — the path filter skips the only real job and the workflow +reports green. Both are filed separately; neither is fixed here. + +Not closed: the reaper queues like everything else and cannot dig out an already-full +queue, so the first drain is a manual `--apply`. And `parity` / `compile-smoke` are +required contexts whose jobs never run on a pull request (`test.yml`'s `push:` is tags +only), so every merge needs an admin bypass — which is why pending gates stopped looking +unusual. That is branch-protection state, not a file in the tree. diff --git a/docs/src/testing/ci-gate-scheduling.md b/docs/src/testing/ci-gate-scheduling.md index 44d0c8e579..b7404df9e7 100644 --- a/docs/src/testing/ci-gate-scheduling.md +++ b/docs/src/testing/ci-gate-scheduling.md @@ -55,13 +55,37 @@ watching — aged out. were the obvious reading, and both are wrong. Getting this right matters, because each wrong reading has a "fix" that would make things worse. -**Not the concurrency block.** `gc-ratchet.yml`'s `concurrency:` comment records two -prior attempts (#7205): a shared group with unconditional `cancel-in-progress` -cancelled three consecutive `main` runs, and scoping `cancel-in-progress` to pull -requests did not fix it either, because GitHub allows at most one *pending* run per -group. Keying the group on `github.sha` for push events **did** fix cancellation. -The failure mode simply moved: runs stopped cancelling each other and started -queueing forever instead. **Those blocks are correct. Do not "fix" them again.** +**Not the concurrency block — *as it stood in #7856*.** `gc-ratchet.yml`'s +`concurrency:` comment records two prior attempts (#7205): a shared group with +unconditional `cancel-in-progress` cancelled three consecutive `main` runs, and +scoping `cancel-in-progress` to pull requests did not fix it either, because GitHub +allows at most one *pending* run per group. Keying the group on `github.sha` for push +events **did** fix cancellation. The failure mode simply moved: runs stopped +cancelling each other and started queueing forever instead. + +> **⚠️ SUPERSEDED BY #7966 — this paragraph used to end "Those blocks are correct. Do +> not 'fix' them again." That sentence was true when written and false three days +> later, and it is exactly the sentence that would send the next reader past the real +> bug.** +> +> The `github.sha` arm is guarded on `github.event_name == 'push'`. #7856 — the change +> this very document describes — moved the main-line arm from `push: branches: [main]` +> to `schedule:`. The guard stopped matching, the expression fell through to +> `github.ref` (constant `refs/heads/main`), and **#7205 came straight back on the arm +> #7856 created.** Measured 2026-08-12, identically on all ten gates: oldest run +> `queued` holding the group, the next two `cancelled` with `jobs: 0`, newest +> `pending`. `gate-freshness` itself was cancelled the same way. +> +> Groups are now keyed on `github.run_id` for every non-pull-request event, which is +> the only context value unconditionally distinct across scheduled runs. +> `scripts/gc_gate_wiring_check.py` (in the required `lint` context) now rejects a +> `schedule:` workflow whose concurrency group lacks `github.run_id`, so this cannot +> relapse a fourth time silently. +> +> **The lesson is about the sentence, not the YAML.** "Do not fix this again" is a +> claim about the future, and a scheduling change three days later invalidated it. A +> repaired invariant should be written down as an *executable check*, not as an +> instruction to the next human to stop looking. **Not macOS capacity.** This was the natural inference — the gates that went dark are the macOS ones — but the job-level numbers refute it. At the moment of @@ -173,3 +197,57 @@ stale gate, a fresh gate, a gate with no successful run at all, and a gate whose recent success is a `pull_request` run (the exact shape that made `gc-root-dominance` look healthy while its `main` arm was dark), and asserts the verdict for each. A green `--self-test` means the detector works, not that nothing was tried. + +## The queue in front of the schedule (#7966) + +A six-hourly sweep only helps if the queue drains faster than six hours. On +2026-08-12 it did not, and the reason was not the gates: + +| metric | value | +|---|---| +| queued runs | 1,529 | +| concurrent runs observed | 12–14 | +| queued by event | 794 `pull_request`, 181 `push`, 19 `schedule` | +| distinct head branches among queued PR runs | 63 | +| **branches that still existed** | **2** | + +GitHub does not reliably cancel a queued run when its pull request merges and the +branch auto-deletes. Perry squash-merges, auto-deletes branches, and fans each PR out +to ~11 workflows, so roughly **790 runs — 51% of the entire queue — were work for +already-merged PRs**, holding runner slots ahead of ten `main` gates that had not +completed in 32+ hours. No amount of scheduling cadence recovers from that; the +garbage has to be removed. + +`ci-queue-reaper.yml` runs `scripts/reap_stale_ci_runs.py` every 30 minutes. It +cancels a run only when all of these hold: `event == "pull_request"`, `status == +"queued"`, and the head branch has **no open pull request**. A `push`, `schedule`, +tag or `workflow_dispatch` run is therefore structurally out of reach, and an +in-flight run is left alone because it has already consumed the scarce thing. The +predicate keys on open PRs rather than on branch existence, which is what keeps fork +PRs safe — a fork's head branch never appears in this repo's refs. + +**It cannot bootstrap.** The reaper queues like everything else, so it will not dig +the repo out of an already-saturated queue. The first drain is a manual +`python3 scripts/reap_stale_ci_runs.py --apply` (dry run is the default); the +schedule keeps it clear afterwards. + +## Why "the gate was dark" is not the same as "the gate would have caught it" + +#7966 landed alongside #7965, a 2.2–4.8× regression that reached `main` while +`gc-ratchet` was dark. The tempting conclusion — the dark gate let it through — does +not survive checking, and recording why matters more than the incident: + +- `gc-ratchet`'s gating metrics are heap/cycle/copy/promote/freed counts. **There is + no full-mark-sweep or major-cycle count**, and the counter that actually found + #7965 (`collection_kind: "full"` going 0 → 2) is not one of them. No gate in the + repo ratchets a collection-*kind* count. +- The two dimensions that did move — wall time and RSS — are explicitly + `"gating": false` in the `shared_ci` profile CI runs, with the rationale recorded + in `tolerances.json`. They gate only under `pinned_host`, which CI never uses. +- The workloads that showed it (`retain`, `deeplist`, …) are the gc-handoff corpus, + not `gc-ratchet`'s fixed 14 probes. + +So the human counter census was not a lucky substitute for a starved gate; **it was +the only instrument that covered that dimension at all.** Restoring gate freshness +does not close #7965's third ask, and a freshness dashboard that is entirely green +would still not have caught it. diff --git a/gc-handoff/GATES-NOTES.md b/gc-handoff/GATES-NOTES.md new file mode 100644 index 0000000000..fa57e6f68f --- /dev/null +++ b/gc-handoff/GATES-NOTES.md @@ -0,0 +1,126 @@ +# Gate starvation investigation — issue #7966 + +Measured 2026-08-12 ~15:00Z from `origin/main` @ a769fafc6. + +## Headline + +The #7856 fix (move the ten expensive gates' main-line arm from `push: branches:[main]` +to a staggered six-hourly `schedule:`) was correct in its diagnosis and **re-introduced +#7205 on the arm it created**. The gates are now dark for a *different* reason than the +issue text assumes. + +## Measured queue state + +| metric | value | +|---|---| +| queued runs | **1,529** | +| in-progress runs | 14 | +| queued by event | 794 `pull_request`, 181 `push`, 19 `schedule` | +| distinct head branches among queued PR runs | 63 | +| **of those branches still existing on the remote** | **2** | +| open PRs | 8 | +| lifetime success / failure / cancelled | 19,824 / 6,482 / **15,283** | +| last 100 completed runs | **100 cancelled, 0 executed** | + +**~790 of 1,529 queued runs (51%) are for 61 branches that no longer exist** — PRs that +already merged and auto-deleted. GitHub does not reliably cancel queued runs on branch +deletion, so they hold runner slots ahead of the scheduled `main` gates forever. + +## The #7205 relapse (this is the fixable bug) + +Every one of the ten gates carries: + +```yaml +group: -${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} +cancel-in-progress: ${{ github.event_name == 'pull_request' }} +``` + +The `github.sha` arm was #7205's fix, and it keys on the event being **`push`**. #7856 +then moved the main-line arm to **`schedule`**, which falls through to `github.ref` — +constant `refs/heads/main` for every scheduled run. So all scheduled runs of a gate +share one group again. + +Per the repo's own measured finding (quoted in gc-ratchet.yml:63-67): GitHub allows at +most one PENDING run per group and cancels the previously pending one when a new run +enters, *regardless of `cancel-in-progress`*. + +Observed, identically across all ten gates: + +``` +2026-08-12T13:37Z schedule pending <- newest, blocked on the group +2026-08-12T07:46Z schedule cancelled <- jobs: 0 (never reached a runner) +2026-08-12T02:37Z schedule cancelled <- jobs: 0 +2026-08-11T19:15Z schedule queued <- holds the group, 20h in the runner queue +``` + +`jobs: 0` is the exact zero-execution signature #7205 was measured by. The oldest run +holds the group and is stuck behind the 1,529-deep queue; every newer scheduled run is +cancelled on arrival. **The gate cannot run again until that one run drains.** + +`Gate Freshness` itself was cancelled at 2026-08-12T13:38Z — same shape +(`gate-freshness-${{ github.event_name }}-${{ github.ref }}`). The alarm designed so it +"cannot be starved by the condition it is alarming about" is now starved by it. + +## Verdict per cause + +- **#7205 relapse on the schedule arm** — 10 gates + gate-freshness. Fixable in one line each. +- **Capacity / zombie queue** — 51% of the queue is dead PR work. Needs a reaper. +- Neither is "the gate is broken". No gate content is at fault. + +--- + +## Per-gate verdict (the issue assumed one cause; there are three) + +| gate | verdict | evidence | +|---|---|---| +| gc-ratchet | STARVED + #7205 relapse | sched runs: oldest `queued` 20h, two `cancelled` w/ `jobs: 0`, newest `pending` | +| gc-root-dominance | STARVED + relapse | same shape | +| tls-budget | STARVED + relapse | same shape | +| gc-ptr-shape-off-witness | STARVED + relapse | same shape | +| gc-parse-churn-gate | STARVED + relapse | same shape | +| gc-moving-witnesses | STARVED + relapse | same shape | +| auto-opt-app-patterns | STARVED + relapse | same shape | +| eh-transport | STARVED + relapse | same shape | +| security-audit | STARVED | 90 queued `push`/`main` runs; required context, so merges bypass | +| **gc-native-roots** | **BROKEN** | **never had a single successful run, any branch, any event.** 3 of 4 arms fail with 3 distinct causes: aarch64-linux SIGSEGV (139) under `PERRY_STACKMAP_WALKER=verify`; windows Rust panic (101); macos-14 `gc_evacuation_liveness_assert.py` reports 0 copying minors / 0 objects copied | +| **llvm-inprocess** | **BROKEN + VACUOUS-GREEN** | last 3 `main` runs `failure`. Worse: sampled PR "successes" show `changes=success, native-backend=skipped` — the path filter skips the only real job and the workflow reports green. Hazard 4 | + +## Structural finding not in the issue: required contexts that can never pass + +Required contexts on `main` are: +`lint, cargo-test, parity, compile-smoke, api-docs-drift, security-audit, conformance-smoke-complete` + +`parity` and `compile-smoke` carry +`if: github.event_name == 'push' || (workflow_dispatch && inputs.run_extended_tests) || (pull_request && contains(labels, 'run-extended-tests'))` +and `push:` in test.yml is **tags only**. So on an ordinary PR they never report at all, and a required context that never reports blocks the merge button forever. Meanwhile `security-audit` sits `queued`. + +**Consequence: every merge needs an admin bypass**, which bypasses the required contexts that DO work. This is why ~20 PRs merged today with checks pending, and it is upstream of the whole incident: bypass is the normal path, so nothing about a pending gate looks unusual. Fixing this is a branch-protection edit (server-side state, not in the tree) and is NOT in this PR. + +## What was unprotected, and what got through + +- **#7965 (2.2-4.8x regression)** — introduced by #7902 (`1bd5eeb6b`), merged in #7944 at 10:17Z, fixed by #7968 at 14:27Z, ~4.2h on main. + **gc-ratchet would NOT have caught it even if it had run.** Its gating metric set has no full-mark-sweep / major-cycle count; the counter that found the bug (`collection_kind:"full"` 0 -> 2) is not a gc-ratchet metric, and no gate in the repo ratchets one. The two dimensions that did move (wall time, RSS) are explicitly `"gating": false` in the `shared_ci` profile CI uses. The workloads (`retain`, `deeplist`) are the gc-handoff corpus, not gc-ratchet's. **This regression class is structurally uncovered by all 11 gates**, independent of the starvation. #7965's ask #3 (gate on the full count) is still open. +- **#7843's seven red rows** — retired by #7921 re-pinning the baseline, not by a code fix. The re-pin was measured at `98e9ecdb5`, which contains #7888 but NOT #7901/#7902 — so `changelog.d/7921-gc-ratchet-repin.md`'s "the merged bounded untraced-promotion changes" is inaccurate. **19 collector-touching PRs merged after that pin with zero gc-ratchet runs completing; whether the new pin holds on current main is unmeasured.** +- **gc-ratchet dark 33.6h** across 37 collector-touching merges. **gc-root-dominance dark 33.6h** across 44 codegen merges with an EMPTY allowlist (every new hit is meant to be a red build). **gc-moving-witnesses / gc-parse-churn-gate / gc-ptr-shape-off-witness dark 57.3h.** + +## Capacity math + +- Arrival: ~58 merges/day, each PR fanning out to ~11 workflows, plus a per-merge `push` arm for security-audit + zizmor. +- Drain: 12-14 concurrent runs observed; lifetime cancelled (15,283) is approaching lifetime success (19,824). +- Standing queue 1,529 with ~790-1,200 of it dead PR work. + +Reaping dead PR runs is worth ~51% of the queue immediately and is the only lever that does not trade away coverage. The six-hourly schedule (#7856) already cut ~19 jobs/merge; it cannot help further while the queue in front of it is half garbage. + +## What this PR changes + +1. **Concurrency group keyed per RUN for non-PR events** across 22 workflows + the new one. `${{ github.event_name == 'pull_request' && github.ref || github.run_id }}`. PR coalescing preserved; main-line runs can no longer supersede each other. +2. **`scripts/gc_gate_wiring_check.py` gains `check_schedule_group`**, swept over ALL 31 workflows (not just the GC gates -- the hazard took out `gate-freshness` too). Requires `github.run_id` in the group of any workflow with a `schedule:` trigger. 5 new self-test cases incl. the CLEAN fixture as the sabotage case. `lint` is required, so a fourth relapse is now a red build. +3. **`scripts/reap_stale_ci_runs.py` + `ci-queue-reaper.yml`** — cancels QUEUED `pull_request` runs with no open PR. Dry-run by default, `--apply` to act, `--max` cap, self-tested incl. two sabotage cases. Only `event==pull_request` + `status==queued` + no open PR; a push/schedule/tag/dispatch run is structurally unreachable. +4. **zizmor**: `push: main` arm path-filtered to `.github/**` (it was unfiltered while the PR arm was already scoped), plus the concurrency block it never had. + +## NOT closed by this PR + +- The **bootstrap**: the reaper queues like everything else, so it cannot dig out an already-saturated queue. First drain must be a manual `python3 scripts/reap_stale_ci_runs.py --apply`. I did not run it -- it cancels ~780 runs on shared infrastructure and is a maintainer call. +- **Branch protection**: `parity` / `compile-smoke` required-but-never-reporting. Server-side; admin only. +- **gc-native-roots** and **llvm-inprocess** are broken, not starved. Filed separately. +- **No gate ratchets collection KIND.** #7965's class stays uncovered. diff --git a/scripts/gc_gate_wiring_check.py b/scripts/gc_gate_wiring_check.py index 6bb291867c..bbec208220 100644 --- a/scripts/gc_gate_wiring_check.py +++ b/scripts/gc_gate_wiring_check.py @@ -295,6 +295,51 @@ def check_gate(text: str, job_id: str, wf_name: str) -> list[str]: # --------------------------------------------------------------------------- # Self-test: the checker must be able to fail, too. # --------------------------------------------------------------------------- +def check_schedule_group(text: str, wf_name: str) -> list[str]: + """A scheduled workflow's concurrency group must vary per RUN. + + Hazard 3 in CLAUDE.md, third relapse (#7966). `cancel-in-progress: false` + does not protect a main-line run: GitHub allows at most one PENDING run per + concurrency group and cancels the previously pending one when a new run + enters, regardless of that setting. So a group expression that evaluates to + a CONSTANT for scheduled runs lets exactly one run — whichever grabbed the + group first — ever execute, and silently cancels every later one with + `jobs: 0`. + + #7205 fixed this for the `push: branches: [main]` arm by keying the group on + `github.sha`, guarded by `github.event_name == 'push'`. #7856 then moved the + main-line arm of ten gates from `push` to `schedule`, and the guard stopped + matching: the expression fell through to `github.ref`, constant + `refs/heads/main`, and #7205 came straight back on the new arm. Measured + 2026-08-12: all ten gates showed the identical signature — oldest run + `queued` holding the group, the next two `cancelled` with zero jobs, newest + `pending` — and `gate-freshness` itself, the alarm for exactly this, was + cancelled the same way. + + `github.run_id` is unique per run and is the only context value that is + unconditionally distinct for scheduled runs, so that is what this requires. + A workflow that genuinely wants scheduled runs to coalesce has to say so by + failing this check and arguing the exemption in review. + """ + if "schedule" not in workflow_triggers(text): + return [] + conc = _block(text, "concurrency", 0) + if not conc: + return [] + group = scalar(conc, "group", 2) + if not group: + return [] + if "github.run_id" in group: + return [] + return [ + f"{wf_name}: has a `schedule:` trigger but its concurrency group does " + f"not contain `github.run_id`, so it is CONSTANT across scheduled runs. " + f"GitHub keeps at most one pending run per group and cancels the rest " + f"with zero jobs, so only one scheduled run can ever execute (#7205, " + f"relapsed as #7966). group: {group}" + ] + + CLEAN = """\ name: X on: @@ -427,6 +472,62 @@ def expect(name: str, text: str, want_substr: str | None): if not got or "not found" not in got[0]: failures.append(f"missing job: expected a not-found problem, got {got}") + # ---- hazard 3 relapse: constant concurrency group on a scheduled run ---- + # (#7966) These exercise check_schedule_group, not check_gate, so they get + # their own harness. The sabotage case is first: a checker that cannot fail + # on the real shape is worth nothing, and CLEAN carries that exact shape. + def expect_group(name: str, text: str, want_problem: bool): + nonlocal cases + cases += 1 + got = check_schedule_group(text, "fixture.yml") + if want_problem and not got: + failures.append(f"{name}: expected a constant-group problem, got none") + if not want_problem and got: + failures.append(f"{name}: expected clean, got {got}") + + # CLEAN is `group: x-${{ github.ref }}` with a schedule trigger -- constant + # across scheduled runs, which is precisely the #7966 shape. + expect_group("constant ref group under schedule", CLEAN, True) + + # The #7205 spelling that #7856 invalidated: guarded on `push`, so a + # scheduled run falls through to the constant ref. + expect_group( + "push-guarded sha group under schedule", + CLEAN.replace( + " group: x-${{ github.ref }}", + " group: x-${{ github.event_name == 'push' && github.sha || github.ref }}", + ), + True, + ) + + # The fix. + expect_group( + "run_id group under schedule", + CLEAN.replace( + " group: x-${{ github.ref }}", + " group: x-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}", + ), + False, + ) + + # No schedule trigger -> the hazard does not apply. + expect_group( + "constant group without a schedule trigger", + CLEAN.replace(" schedule:\n - cron: '0 4 * * *'", " push:\n tags: ['v*']"), + False, + ) + + # No concurrency block at all -> nothing can supersede anything. + expect_group( + "schedule with no concurrency block", + CLEAN.replace( + "concurrency:\n group: x-${{ github.ref }}\n" + " cancel-in-progress: ${{ github.event_name == 'pull_request' }}\n", + "", + ), + False, + ) + if failures: for f in failures: print(f"SELF-TEST FAIL: {f}", file=sys.stderr) @@ -463,6 +564,15 @@ def main() -> int: continue problems.extend(check_gate(path.read_text(), job, wf)) + # The constant-group hazard is not specific to the GC gates -- it hits any + # scheduled workflow, and it took out `gate-freshness` (the alarm) too. So + # this arm sweeps every workflow file rather than just GATES. + wf_dir = REPO_ROOT / ".github" / "workflows" + scanned = 0 + for path in sorted(wf_dir.glob("*.yml")): + scanned += 1 + problems.extend(check_schedule_group(path.read_text(), path.name)) + if problems: print("GC GATE WIRING: one or more gates cannot fail where it matters.\n", file=sys.stderr) for p in problems: @@ -473,7 +583,11 @@ def main() -> int: ) return 1 - print(f"GC gate wiring OK ({len(GATES)} gates main-line-reachable and able to fail)") + print( + f"GC gate wiring OK ({len(GATES)} gates main-line-reachable and able to " + f"fail; {scanned} workflows checked for constant scheduled-run " + f"concurrency groups)" + ) return 0 diff --git a/scripts/reap_stale_ci_runs.py b/scripts/reap_stale_ci_runs.py new file mode 100755 index 0000000000..f4b8ce02c9 --- /dev/null +++ b/scripts/reap_stale_ci_runs.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Cancel QUEUED pull-request workflow runs whose pull request is already closed. + +WHY THIS EXISTS +--------------- +GitHub does not reliably cancel a workflow run that is still sitting in the +queue when its pull request merges and its branch auto-deletes. Perry squash- +merges and auto-deletes branches, and every PR fans out to ~11 workflows, so a +busy day leaves hundreds of runs queued against branches that no longer exist. +They cannot gate anything -- the PR they were measuring is already in `main` -- +but they hold runner slots ahead of the scheduled `main` gates, which is how +those gates go dark. + +Measured on 2026-08-12 (#7966): 1,529 queued runs, of which 794 were +`pull_request` runs spread over 63 head branches. Two of those branches still +existed. The other 61 were merged-and-deleted, accounting for roughly 790 runs +-- 51% of the entire queue -- pinned in front of ten six-hourly `main` gates +that had not completed a run in over 32 hours. + +WHAT IT WILL AND WILL NOT TOUCH +------------------------------- +Cancelling runs is destructive and this script is deliberately timid: + + * Only `event == "pull_request"` runs. A `push`, `schedule`, + `workflow_dispatch` or tag run is never a candidate -- those ARE the + main-line gates this exists to protect. + * Only `status == "queued"`. A run that already reached a runner is left + alone: it has consumed the scarce thing (a slot) and killing it mid-flight + just wastes the work. + * Only when the head branch has no OPEN pull request. Keying on open PRs + rather than on branch existence is what makes fork PRs safe -- a fork's + head branch never appears in this repo's refs, but its PR does appear in + the open-PR list, so it is never a candidate. + * `--dry-run` is the DEFAULT. Cancelling requires an explicit `--apply`. + * `--max` caps one invocation, so a bug cannot empty the queue in one go. + +Usage: + python3 scripts/reap_stale_ci_runs.py # report only + python3 scripts/reap_stale_ci_runs.py --apply # actually cancel + python3 scripts/reap_stale_ci_runs.py --self-test # check the checker +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys + +REPO = "PerryTS/perry" + +# Events that carry a main-line verdict. A run with one of these events is +# never a reaping candidate, whatever its branch looks like. +MAIN_LINE_EVENTS = frozenset({"push", "schedule", "workflow_dispatch", "release"}) + + +def should_reap(run: dict, open_pr_branches: set[str]) -> bool: + """The whole policy, in one testable function. + + `run` needs `event`, `status` and `head_branch`. + """ + if run.get("event") != "pull_request": + return False + if run.get("status") != "queued": + return False + branch = run.get("head_branch") + if not branch: + # No branch to reason about -> refuse. "Unknown" must never mean "cancel". + return False + return branch not in open_pr_branches + + +def _gh_json(args: list[str]) -> object: + out = subprocess.run( + ["gh", *args], capture_output=True, text=True, check=True + ).stdout + return json.loads(out) if out.strip() else [] + + +def open_pr_branches() -> set[str]: + rows = _gh_json( + ["pr", "list", "--repo", REPO, "--state", "open", "--limit", "500", + "--json", "headRefName"] + ) + return {r["headRefName"] for r in rows} # type: ignore[index] + + +def queued_runs() -> list[dict]: + runs: list[dict] = [] + page = 1 + while True: + batch = _gh_json( + ["api", f"repos/{REPO}/actions/runs?status=queued&per_page=100&page={page}", + "-q", ".workflow_runs"] + ) + if not batch: + break + runs.extend(batch) # type: ignore[arg-type] + if len(batch) < 100: # type: ignore[arg-type] + break + page += 1 + if page > 30: # 3000 runs is far past any sane queue; stop rather than spin + break + return runs + + +def cancel(run_id: int) -> bool: + r = subprocess.run( + ["gh", "api", "-X", "POST", f"repos/{REPO}/actions/runs/{run_id}/cancel"], + capture_output=True, text=True, + ) + return r.returncode == 0 + + +def _self_test() -> int: + open_prs = {"feat/live-one", "fork-contributor-branch"} + cases = [ + ("merged PR, branch gone", + {"event": "pull_request", "status": "queued", "head_branch": "fix/7843-gc-ratchet"}, True), + ("open PR is protected", + {"event": "pull_request", "status": "queued", "head_branch": "feat/live-one"}, False), + ("fork PR is protected by its OPEN pr, not by branch existence", + {"event": "pull_request", "status": "queued", "head_branch": "fork-contributor-branch"}, False), + ("a scheduled main gate is never reaped", + {"event": "schedule", "status": "queued", "head_branch": "main"}, False), + ("a push to main is never reaped", + {"event": "push", "status": "queued", "head_branch": "main"}, False), + ("a tag/dispatch run is never reaped", + {"event": "workflow_dispatch", "status": "queued", "head_branch": "main"}, False), + ("an already-running PR run is left alone", + {"event": "pull_request", "status": "in_progress", "head_branch": "fix/7843-gc-ratchet"}, False), + ("a completed run is not a candidate", + {"event": "pull_request", "status": "completed", "head_branch": "fix/7843-gc-ratchet"}, False), + ("missing branch means refuse, not cancel", + {"event": "pull_request", "status": "queued", "head_branch": None}, False), + ] + failures = [] + for name, run, want in cases: + got = should_reap(run, open_prs) + if got != want: + failures.append(f"{name}: want {want}, got {got}") + + # Sabotage: a policy that reaped everything would pass a test that only + # checked the positive case. Assert the guard rails actually bind. + if should_reap({"event": "schedule", "status": "queued", "head_branch": "gone"}, set()): + failures.append("SABOTAGE: a schedule run was reaped with an empty open-PR set") + if not should_reap( + {"event": "pull_request", "status": "queued", "head_branch": "gone"}, set() + ): + failures.append("SABOTAGE: the detector cannot fire at all") + + if failures: + print("reap_stale_ci_runs self-test FAILED", file=sys.stderr) + for f in failures: + print(f" - {f}", file=sys.stderr) + return 1 + print(f"reap_stale_ci_runs self-test: OK ({len(cases) + 2} cases)") + return 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--apply", action="store_true", + help="actually cancel (default is a dry run)") + ap.add_argument("--max", type=int, default=400, + help="cap cancellations for one invocation (default 400)") + ap.add_argument("--self-test", action="store_true") + args = ap.parse_args() + + if args.self_test: + return _self_test() + + protected = open_pr_branches() + runs = queued_runs() + victims = [r for r in runs if should_reap(r, protected)] + + print(f"queued runs: {len(runs)}") + print(f"open PR branches: {len(protected)}") + print(f"reapable (stale PR):{len(victims)}") + + by_branch: dict[str, int] = {} + for r in victims: + by_branch[r["head_branch"]] = by_branch.get(r["head_branch"], 0) + 1 + for br, n in sorted(by_branch.items(), key=lambda kv: -kv[1]): + print(f" {n:4d} {br}") + + if not args.apply: + print("\nDRY RUN — nothing cancelled. Re-run with --apply to cancel.") + return 0 + + done = 0 + for r in victims[: args.max]: + if cancel(r["id"]): + done += 1 + print(f"\ncancelled {done} of {len(victims)} stale queued runs") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 915c6d90f8fcfda8f9d838ee0a850a39477f77a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Wed, 12 Aug 2026 17:26:54 +0200 Subject: [PATCH 2/3] chore: name the changelog fragment after the PR (#7969) --- changelog.d/{7966-gate-starvation.md => 7969-gate-starvation.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelog.d/{7966-gate-starvation.md => 7969-gate-starvation.md} (100%) diff --git a/changelog.d/7966-gate-starvation.md b/changelog.d/7969-gate-starvation.md similarity index 100% rename from changelog.d/7966-gate-starvation.md rename to changelog.d/7969-gate-starvation.md From 74f8906a8a58006a3c2a3fd701184eb84f94ef95 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Wed, 12 Aug 2026 17:27:45 +0200 Subject: [PATCH 3/3] docs: record the filed follow-ups (#7970, #7971) in the gate notes --- gc-handoff/GATES-NOTES.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/gc-handoff/GATES-NOTES.md b/gc-handoff/GATES-NOTES.md index fa57e6f68f..e553cce4d4 100644 --- a/gc-handoff/GATES-NOTES.md +++ b/gc-handoff/GATES-NOTES.md @@ -122,5 +122,15 @@ Reaping dead PR runs is worth ~51% of the queue immediately and is the only leve - The **bootstrap**: the reaper queues like everything else, so it cannot dig out an already-saturated queue. First drain must be a manual `python3 scripts/reap_stale_ci_runs.py --apply`. I did not run it -- it cancels ~780 runs on shared infrastructure and is a maintainer call. - **Branch protection**: `parity` / `compile-smoke` required-but-never-reporting. Server-side; admin only. -- **gc-native-roots** and **llvm-inprocess** are broken, not starved. Filed separately. +- **gc-native-roots** (#7970) and **llvm-inprocess** (#7971) are broken, not starved. Filed, not fixed. - **No gate ratchets collection KIND.** #7965's class stays uncovered. + +## Outcome + +- PR **#7969** (this work) — concurrency relapse fix across 22 workflows + the new reaper, + `check_schedule_group` guard in the required `lint` context, zizmor push-arm filter, + scheduling doc corrected. +- Issue **#7970** — `gc-native-roots` has never been green; 3 of 4 arms fail with 3 distinct causes. +- Issue **#7971** — `llvm-inprocess` reports green on PRs while skipping its only real job. +- Issue **#7966** left OPEN deliberately: `gate-freshness.yml` maintains it in place and + closes it itself once every gate is fresh. Closing it by hand would be reverted.