Skip to content

fix(ci): use the previous master tip as base, and wait for its run before retagging (#1549) - #1947

Merged
groupsky merged 4 commits into
masterfrom
fix/1549-ci-base-ref
Nov 11, 2026
Merged

groupsky merged 4 commits into
masterfrom
fix/1549-ci-base-ref

Conversation

@groupsky

@groupsky groupsky commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Refs #1549

What

  • One base for detection and reuse. A master push now compares against github.event.before (the tip before the push), not HEAD^1. scripts/ci/master-base-ref.sh falls back to HEAD^1 for workflow_dispatch, the null SHA (branch creation), a commit not in the clone, or a force-push. Detect Changes exports this as base_ref, and Stage 5B uses the same value instead of git rev-parse HEAD^.
  • Rapid merges. New job wait-for-base-run (it starts after this run's own builds and tests) waits (up to 55 min) until every ci-unified.yml run of the base commit has finished. Stage 5B needs it, so it reads the base's final SHA tags instead of failing while the base run is still going. It never fails itself; 5B still checks the tags and fails loudly when they are missing.

Decision: a waiting retag step, not queue-by-branch concurrency

  • A queue-by-branch group (ci-master + cancel-in-progress: false) keeps only one pending run; GitHub cancels the older pending ones. A cancelled run tags nothing, so its commit is never deployable, and its successor again finds no base images. That breaks the chain in the same case it should fix.
  • The wait keeps every run building and testing in parallel. Only the cheap retag waits. One job polls the API (not each of the ~20 matrix legs), every 30 s.
  • Limit: a long burst (like 49 merges) can still pass 55 min or the parallel-job limit; then 5B fails loudly as before, and force_rebuild=true repairs. Large batches should still go one at a time.
  • No fallback to :latest is added: the reasons in .github/workflows/CLAUDE.md ("Stage 5B sources ONLY the base commit's SHA tag") still hold.

Tests

  • scripts/tests/master-base-ref.bats (runs in Deployment Scripts Tests): a two-commit push with the change in the first commit, a single commit, empty before, null SHA, force-push, and an unknown SHA.

Progress

  • Master pushes compare against github.event.before, with the HEAD^1 fallbacks
  • Stage 5B uses the same base
  • Test for a two-commit push with the change in the first commit
  • Decision recorded (this PR, and docs)
  • Docs updated (.github/workflows/CLAUDE.md)
  • Review (2 rounds; fixed: wait job starts only after own tests, query limited to master, API errors shown)
  • Green CI, merge; widen the issue's title and body
  • Two PRs merged within 30 s of each other both leave master green — waits for an answer from Geno on CI base for master runs: multi-commit pushes and rapid successive merges left master with no SHA-tagged images #1549

…n before retagging (#1549)

Detection and Stage 5B now share one base: github.event.before, falling
back to HEAD^1 for dispatch, branch creation and force-push. A new job
waits until the base commit's runs are finished, so rapid merges no
longer fail Stage 5B on tags that are not pushed yet.
@groupsky
groupsky marked this pull request as ready for review November 11, 2026 10:57
@groupsky
groupsky merged commit ee65c50 into master Nov 11, 2026
25 checks passed
@groupsky
groupsky deleted the fix/1549-ci-base-ref branch November 11, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant