ci: continuous drift verification of released artifacts - #324
Open
robinnsc wants to merge 1 commit into
Open
Conversation
A daily anonymous audit of everything the registries owe users: every released version tag resolves to its recorded digest, every signature cryptographically verifies against the committed key (transparency log included where entries exist), and latest points at the highest release shipped to each registry. Reachable-and-wrong (drift) is distinguished from unreachable (outage); alerting is a self-resolving registry-drift issue rather than author-only run emails. Logic lives in devtools/ and runs identically locally: the full 34-check matrix passes against the live registries, and a corrupted-digest negative test fails with DRIFT as required.
robinnsc
requested review from
LeeroyHannigan,
amrith,
c33howard,
jcshepherd,
pdf-amzn and
yesyayen
as code owners
August 31, 2026 04:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Continuous drift verification for released container artifacts, in three pieces:
docker/released-artifacts.json— the source of truth for what everyregistry owes users: per image, each released version's digest, its Rekor
status, and (where narrower than the default) which registries it shipped
to. One appended entry per future release.
devtools/verify-released-artifacts— asserts, through anonymousreads only, that every released version tag resolves to its recorded
digest, every signature tag exists and cryptographically verifies
against the committed public key (transparency log included for ≥0.1.7
releases), and
lateston each registry points at the highest releaseshipped there. Bounded retries absorb blips; DRIFT (reachable and
wrong) and UNREACHABLE (persistent registry errors) are distinct
outcomes, because an outage is not a security event.
.github/workflows/registry-drift.yml— runs it daily (and ondispatch) with checksum-pinned crane/cosign. No credentials, no
environment,
issues: writeonly: drift creates or comments on aregistry-driftissue carrying the full report, and a clean runauto-closes open ones (author-only scheduled-failure emails are how these
alerts die).
Why
Every guarantee we make is currently verified at release time only; after
that, nothing watches. Signature tags are deletable, tags are movable by
anyone with a credential, and registries can diverge after partial manual
intervention — today a user's failing
cosign verifywould be thedetector. Two historical incidents are exactly cells in this matrix: the
GHCR
latestthat sat a release behind for days after v0.1.5, and theECR half-state (signature present, version tags absent) from the v0.1.8
propagation failure.
Building the manifest also surfaced a live docs-vs-reality gap, reported
separately to the release owner: the published v0.1.9/v0.1.10 release pages
reference container images that were never released (both Aug 25 container
runs still sit unapproved at their publish gates; registries top out at
0.1.8, internally consistent and fully verified). This PR's manifest
records reality; it gains a line when 0.1.10 containers actually ship.
Testing done
registries × versions × {tag, signature-verify, latest}) passes against
the live registries — output in the run log of this branch's author, and
reproducible by anyone:
./devtools/verify-released-artifactsneeds onlycrane, cosign, jq, and no credentials.
DRIFTonall seven affected cells and exit 1; restoring it re-verifies clean.
yaml.safe_loadpasses on the workflow; crane pinned by the officialrelease checksum; the script is
set -u-safe on bash 4 (workstation)and 5 (runner).
Checklist
ADR / RFC: n/a — CI tooling only.
Breaking changes
None. Read-only; gates nothing.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.