Skip to content

ci: never prune a manifest a tag still references - #57

Merged
eetu merged 1 commit into
mainfrom
fix/ghcr-prune-keeps-referenced-manifests
Sep 4, 2026
Merged

eetu merged 1 commit into
mainfrom
fix/ghcr-prune-keeps-referenced-manifests

Conversation

@eetu

@eetu eetu commented Sep 4, 2026

Copy link
Copy Markdown
Owner

ghcr.io/eetu/dice:1 is currently unpullable on every platform. The prune added in #56 deleted it.

What happened

The step assumed untagged means build leftover:

Untagged versions are the intermediate manifests each multi-arch build leaves behind; the tagged ones (1.0.8, 1.0, 1, latest, main) are never candidates.

For a multi-arch image that isn't true. The tag goes on the index; the index's per-architecture child manifests are separate, untagged package versions referenced only by digest. They are the image. Delete them and the tag still resolves to an index, but every platform it points at is gone.

With 2 children per build and KEEP: "5", the window covered barely two builds, so children of still-tagged indexes fell out and were deleted. Current state:

ghcr.io/eetu/dice:1        BROKEN: 2/2 children MISSING   ← index intact, both children gone

continue-on-error: true meant none of it ever showed up as a red build.

Why it matters beyond a warning

The invinite-tech host has been failing podman-auto-update nightly since 00:10 today:

Error: checking image updates for container ...: fetching target platform image
selected from image index: reading manifest sha256:2b5b2d19... in ghcr.io/eetu/dice:
manifest unknown

The container is still up only because the image is already in local storage. A server recreate, an image prune, or any forced re-pull would have left dice unable to start — silently, since the app looks healthy right now.

The fix

Collect every digest referenced by a tagged index and exclude those, then keep the 5 newest of what genuinely is unreferenced. The registry read uses docker buildx imagetools inspect, already set up and logged in earlier in the job, so no extra token handling.

It also fails closed: if tagged multi-arch images exist but no referenced digests could be read, it refuses to prune rather than treating a blank answer as "nothing is referenced" — the same class of mistake that caused this.

Verified against the live package

Both logics delete nothing today (the damage is done, only 15 versions remain), so I re-ran with the window tightened to KEEP=0 to simulate it having slid:

OLD logic candidates: 5
  ...REFERENCED by a live tag (deleting these breaks the tag):
     BREAKS: sha256:64c39d6872bb0b6028c     ← child of :main
     BREAKS: sha256:5dedf9788c7ad3fd9b3     ← child of :main

NEW logic candidates: 3
  ...referenced (must be none):
     (nothing above = safe)

The keep-set resolved 20 referenced digests across 13 tags.

This does not yet fix :1

Merging this only rebuilds :main — the semver tags (1, 1.0, latest) are type=semver, so they publish only on a v* tag. Restoring :1 needs a release. I'll follow up with just release patch → v1.0.9 once this is in, which republishes 1.0.9/1.0/1/latest with the fixed prune already live. The CHANGELOG entry here covers it.

I'd also suggest a workflow_dispatch trigger as a separate change, so a future "republish this tag" doesn't need a version bump — happy to add it.

The inlined GHCR prune assumed "untagged" meant "build leftover". It doesn't: a
multi-arch tag puts the tag on the INDEX, and the index's per-architecture child
manifests are separate, untagged versions referenced only by digest. They are the
image, not leftovers.

With two children per build and KEEP=5, the window covered barely two builds, so the
children of still-tagged indexes fell out of it and were deleted. On 2026-09-04 both
children of the `1` index were gone: `ghcr.io/eetu/dice:1` still resolved to an
index but was unpullable on every platform, and the invinite-tech host's
podman-auto-update failed nightly. The container kept running only because the image
was already in local storage — a server recreate would have left dice unable to start.
continue-on-error meant none of this surfaced as a red build.

The prune now collects every digest referenced by a tagged index (via buildx
imagetools, which is already set up and logged in) and excludes those, then keeps the
5 newest of what genuinely is unreferenced. It also fails closed: if tagged
multi-arch images exist but no referenced digests could be read, it refuses to delete
on the strength of a blank answer.

Verified against the live package with the window tightened to KEEP=0: the old logic
selects 5 candidates, 2 of which are children of the live `main` tag and would break
it; the new logic selects 3 and none are referenced.
@eetu
eetu merged commit 266b1ae into main Sep 4, 2026
6 checks passed
@eetu
eetu deleted the fix/ghcr-prune-keeps-referenced-manifests branch September 4, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant