Skip to content

ci: continuous drift verification of released artifacts - #324

Open
robinnsc wants to merge 1 commit into
mainfrom
ci/registry-drift-verification
Open

ci: continuous drift verification of released artifacts#324
robinnsc wants to merge 1 commit into
mainfrom
ci/registry-drift-verification

Conversation

@robinnsc

Copy link
Copy Markdown
Collaborator

What

Continuous drift verification for released container artifacts, in three pieces:

  • docker/released-artifacts.json — the source of truth for what every
    registry owes users: per image, each released version's digest, its Rekor
    status, and (where narrower than the default) which registries it shipped
    to. One appended entry per future release.
  • devtools/verify-released-artifacts — asserts, through anonymous
    reads only
    , that every released version tag resolves to its recorded
    digest, every signature tag exists and cryptographically verifies
    against the committed public key (transparency log included for ≥0.1.7
    releases), and latest on each registry points at the highest release
    shipped there. Bounded retries absorb blips; DRIFT (reachable and
    wrong) and UNREACHABLE (persistent registry errors) are distinct
    outcomes, because an outage is not a security event.
  • .github/workflows/registry-drift.yml — runs it daily (and on
    dispatch) with checksum-pinned crane/cosign. No credentials, no
    environment, issues: write only: drift creates or comments on a
    registry-drift issue carrying the full report, and a clean run
    auto-closes open ones (author-only scheduled-failure emails are how these
    alerts die).

Why

Every guarantee we make is currently verified at release time only; after
that, nothing watches. Signature tags are deletable, tags are movable by
anyone with a credential, and registries can diverge after partial manual
intervention — today a user's failing cosign verify would be the
detector. Two historical incidents are exactly cells in this matrix: the
GHCR latest that sat a release behind for days after v0.1.5, and the
ECR half-state (signature present, version tags absent) from the v0.1.8
propagation failure.

Building the manifest also surfaced a live docs-vs-reality gap, reported
separately to the release owner: the published v0.1.9/v0.1.10 release pages
reference container images that were never released (both Aug 25 container
runs still sit unapproved at their publish gates; registries top out at
0.1.8, internally consistent and fully verified). This PR's manifest
records reality; it gains a line when 0.1.10 containers actually ship.

Testing done

  • Positive, against production: the full 34-check matrix (2 images × 3
    registries × versions × {tag, signature-verify, latest}) passes against
    the live registries — output in the run log of this branch's author, and
    reproducible by anyone: ./devtools/verify-released-artifacts needs only
    crane, cosign, jq, and no credentials.
  • Negative: corrupting one digest in the manifest produces DRIFT on
    all seven affected cells and exit 1; restoring it re-verifies clean.
  • yaml.safe_load passes on the workflow; crane pinned by the official
    release checksum; the script is set -u-safe on bash 4 (workstation)
    and 5 (runner).

Checklist

  • Tests / fmt / clippy — not applicable, CI + manifest only
  • I have added or updated tests for new functionality — live positive run + corrupted-digest negative test above
  • I have updated documentation if behavior changed — headers document semantics; manifest is self-describing
  • Breaking changes are noted below (if any)

ADR / RFC: n/a — CI tooling only.

Breaking changes

None. Read-only; gates nothing.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

A daily anonymous audit of everything the registries owe users:
every released version tag resolves to its recorded digest, every
signature cryptographically verifies against the committed key
(transparency log included where entries exist), and latest points
at the highest release shipped to each registry. Reachable-and-wrong
(drift) is distinguished from unreachable (outage); alerting is a
self-resolving registry-drift issue rather than author-only run
emails. Logic lives in devtools/ and runs identically locally: the
full 34-check matrix passes against the live registries, and a
corrupted-digest negative test fails with DRIFT as required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant