ci: publish the PostgreSQL container image for amd64 and arm64 on a version tag - #251
ci: publish the PostgreSQL container image for amd64 and arm64 on a version tag#251LeeroyHannigan wants to merge 3 commits into
Conversation
PR #241 landed a production-shaped PostgreSQL image and a 334-line container smoke test, but nothing in .github/workflows referenced either, so both the Dockerfile and ci/smoke-test-container.sh only ran when someone remembered to run them locally. This adds the release workflow that makes them mandatory. The step order is the substance of the change: gate, build, test, log in, push. * The image is built with `load: true` into the local store and NOT pushed. The smoke test then runs against it via EXTENDDB_IMAGE, which puts the script into prebuilt mode: it reads VERSION / VCS_REF / BUILD_DATE back off the image labels, runs `compose up --no-build`, and asserts every container is running that exact image ID. `docker push` then uploads the same local image by ID, so the artifact that is published is provably the artifact that passed, with no rebuild in between. * Login happens after the smoke test rather than before it, so no registry credential exists in the job while the freshly built image is executing. Three gates run before anything is built: * The tag version must equal the workspace version in Cargo.toml. main has already been at 0.1.3 while the newest tag claimed 0.1.2; this makes that divergence a build failure instead of a silently mislabelled artifact. * The tagged commit must be an ancestor of origin/main, so a tag pushed on an unmerged branch cannot publish unreviewed code to a public namespace. * devtools/generate-software-license-notices --check must pass. A stale SOFTWARE-LICENSE-NOTICES.html is a compliance problem in a distributed image in a way it is not in source, and nothing enforced it before. All four actions are pinned to commit SHAs rather than floating tags, matching the discipline the Dockerfile already applies to its four base images by digest. Each pin is the newest release within the major the workflow is written against, so the input contracts are known rather than assumed. Two omissions are deliberate and documented in the file. There are no provenance or SBOM attestations, because those require the registry exporter while `load: true` requires the local docker exporter; getting both those attestations and the test-equals-ship guarantee means pushing to a staging tag and promoting the digest with `imagetools create`, which needs a staging repository. The build is also single-architecture for the same reason: an amd64 runner cannot execute an arm64 image, so publishing one would mean publishing something never tested. Structured for a second backend without being generalised prematurely: the image name is a top-level env var and the concurrency group names the image, so the future split into a `workflow_call` workflow plus one thin caller per backend is mechanical. A matrix is the wrong shape there, because a protected environment prompts once per job and a partial failure publishes one backend under a version tag and not the other.
The first version of this workflow published linux/amd64 only, which was the
wrong default for a local DynamoDB emulator: a large share of the audience
develops on Apple Silicon, where an amd64-only image runs under emulation with a
platform-mismatch warning and a real performance cost.
Nothing prevented arm64. Verified before writing this: all three pinned base
image digests are OCI indexes that include linux/arm64/v8, the Dockerfile has no
architecture-specific paths since the binary is copied out of its own build
stage, and the version-pinned apt packages resolve on arm64 (tini=0.19.0-1+b3 is
the arm64 candidate, so the binNMU suffix is not a problem). An emulated arm64
image was then built locally and the full container smoke test run against it:
41m18s compile, then healthy, CreateTable / PutItem / GetItem, restart
persistence, graceful SIGTERM and migration bootstrap all passed. The image
reports arch=arm64 and the binary inside is a 64-bit aarch64 ELF, so it was not
a silent amd64 fallback.
The workflow is restructured into three jobs rather than one:
gate Runs once. The version, ancestry and licence checks need no build, so
a bad tag now fails in seconds instead of after two image builds.
build One job per architecture on its NATIVE runner: ubuntu-latest for
amd64, ubuntu-24.04-arm for arm64. No qemu, so the arm64 image is
genuinely executed rather than published untested. Each job builds
with `load: true`, asserts the image really reports the expected
architecture, smoke tests it, and saves it as a workflow artifact.
`docker save`/`load` preserves the image ID, so the artifact is
bit-for-bit what was tested. These jobs hold no credentials.
publish Runs once, after both architectures pass, and is the only job that
references the `dockerhub` environment. So the token never exists in
a job that is executing a freshly built image, and never exists at
all unless both architectures passed. It also keeps this to a single
approval, where an environment on a matrix job would prompt twice.
Publishing pushes the two per-architecture tags, combines them into a manifest
list for both :VERSION and :latest, and then re-inspects the published tag to
assert both platforms are actually present. The per-arch tags stay visible on
Docker Hub, which is the conventional cost of this pattern; the tags users
consume are proper multi-arch lists.
Also added a prerequisites step, because ci/smoke-test-container.sh requires the
AWS CLI and it is not guaranteed on every runner image, least of all the newer
arm64 ones. It installs the correct bundle for the host architecture rather than
assuming the tool is present.
Still no provenance or SBOM attestations. Those come from the registry exporter
while `load: true` requires the local docker exporter, and `load` is precisely
what allows the image to be smoke tested before any credential exists in the
job. Adding attestations would mean pushing before testing, or pushing to a
staging repository and promoting the digest, which are worse trades than losing
the attestation for now.
LeeroyHannigan
changed the title
|
Nice. Definitely a lot in this workflow that I think we should preserve: native amd64 and ARM64 builds, actually running both images before publication, keeping Docker Hub credentials out of the build/test jobs transferring the tested images as workflow artifacts instead of rebuilding them in the publisher, pinning Actions to full commit SHAs, and checking generated license notices. Then I think the main decision would be how much release infrastructure do we want to build before the first public image. Id say there's two main options:
Option 1: Controlled manual process for the initial releaseFor the first release, it may be reasonable to keep the automation relatively small and have maintainers perform and record some of the verification and promotion steps manually. The workflow would still build and test the images, but it would stop after publishing an immutable release candidate rather than immediately creating 0.1.3 and latest. A possible flow would be: workflow builds and locally tests amd64 + ARM64 This would require fewer changes than fully automating every registry and promotion gate, but I think there are still some minimum changes we should make to this PR. Minimum workflow changes for the manual option The release should be dispatched using the workflow definition from protected main, with an existing version tag such as v0.1.3 as input. I suggest avoiding direct push: tags: v* publication because that executes the workflow definition from the tagged revision, which may be older than the current protected release workflow. The workflow should:
The current direct interpolation: REF_NAME='${{ github.event.inputs.ref || github.ref_name }}' should be changed because GitHub context and dispatch input values are untrusted shell input. The current wall-clock build date should also be changed: date -u +%Y-%m-%dT%H:%M:%SZ Rerunning the same release currently produces a different image config and digest. A commit- or tag-derived timestamp would make the result deterministic. Manual candidate verification Once the candidate is published, maintainers would manually:
Docker Hub can remain canonical: docker.io/extenddb/extenddb-postgres with mirrors at: ghcr.io/extenddb/extenddb-postgres The registry index digests may differ if a registry normalizes the manifest representation, so we should record each index digest while confirming that the underlying platform configs and layers match. ECR does not necessarily have to block the first release. We can release Docker Hub and GHCR first and add ECR after the branded public.ecr.aws/extenddb alias is active, provided we mirror the existing artifact instead of rebuilding it. Manual stable promotion After the candidate has been verified, a maintainer would manually promote the exact candidate digest to 0.1.3. The promotion procedure should:
This also avoids an older manual workflow invocation moving latest backward. The existing concurrency group prevents simultaneous runs but does not enforce version ordering. The steps and outputs should be captured in a release checklist or GitHub Release so the manual process remains auditable. Signing under the manual option This PR is currently unsigned. Docker Hub authentication, environment approval, and the resulting image digest are useful controls, but they are not a cryptographic publisher signature. For the first release we could consistently publish all registry copies unsigned: Docker Hub: unsigned If we do that, the release notes should explicitly state that the images are unsigned and provide immutable digests. Keyless signatures can be attached to those exact digests later without rebuilding. Alternatively, we can sign the first release, but keyless production signing should still happen in a protected GitHub Actions job rather than on a developer machine. The manual process would become: maintainers verify candidates We should avoid signing only one registry unless there is a deliberate documented reason. Option 2: Build the robust automated release process nowThe other option is to automate the complete process in this PR or a coordinated series of PRs. The workflow would look approximately like: gate This would automate:
It is more work up front, but it gives us a repeatable release mechanism for 0.1.3 and every subsequent version. If we add signing, the workflow would use the same GitHub OIDC identity for all registries while using separate registry credentials to store each signature. The signing order would be: candidate published Signing should not be added to the current direct-to-stable push as a final command. The image should be signed only after the exact registry candidate has passed testing and before it receives a stable tag. Common requirements for either option The AWS CLI installation should also be addressed under either option. The workflow currently downloads the mutable AWS CLI installer and executes it with sudo without pinning a version or verifying a checksum/signature. We should either use a reviewed runner-provided version or pin and verify the installer. |
…om main
Implements the reviewed minimum changes for the controlled manual initial
release (option 1 in the review discussion) and the runbook's workflow
hardening list:
- Trigger is workflow_dispatch only. The push: tags: v* trigger is removed
because a tag-triggered run executes the workflow definition from the
tagged revision, which can be older than the reviewed workflow on
protected main. The dockerhub environment's deployment rule must migrate
from 'tag: v*' to 'branch: main' immediately after this merges.
- The tag input is strictly validated: semver shape check before it reaches
any git command, must resolve to an existing tag, full 40-char SHA,
contained in origin/main, workspace version match checked at that commit.
- Untrusted input hardening: the dispatch input and gate outputs reach shell
scripts only through env vars, never by direct interpolation.
- Deterministic metadata: BUILD_DATE is the tagged commit's own timestamp
and VCS_REF is the full commit SHA, so re-running the same release
reproduces the same image config instead of minting a new digest.
- Candidate-only publication: pushes sha-<full-commit>-{amd64,arm64} and the
sha-<full-commit> manifest list. No version tag, no latest. Promotion,
mirroring, verification, and signing are manual runbook steps for the
initial release.
- Overwrite refusal: the publish job fails if the candidate tag already
exists in the registry.
- The AWS CLI fallback install is pinned to 2.31.6 and verified against
recorded SHA-256 checksums for both architectures before execution.
Preserved: native per-arch runners with architecture assertion, smoke test
before any credential exists, docker save/load artifact transfer so the
pushed bytes are what was tested, full-SHA action pins, licence-notice gate,
single-approval dockerhub environment, concurrency group.
|
Thanks, this landed exactly where the release runbook did: option 1, controlled manual initial release. Pushed 75d2826 implementing your minimum-changes list in full: Dispatch-only. Strict input validation. Semver shape check before the input reaches any git command, tag must exist, resolves to a full 40-char SHA, ancestry-checked against origin/main, and the workspace version is read at that commit ( Untrusted input. The dispatch input and all gate outputs reach shell only via env vars; the Deterministic metadata. BUILD_DATE is now the tagged commit's own timestamp and VCS_REF is the full SHA, so re-running the same release reproduces the same image config and digest. Candidate-only. The publish job (renamed publish-candidate) pushes AWS CLI. The fallback install is pinned to 2.31.6 with recorded SHA-256 checksums for both architectures, verified before execution. On signing: agreed on not signing from a developer machine as a workflow habit, and per the runbook the first release signs each registry-specific digest via the KMS-backed procedure that was dry-run on the dev key, between candidate verification and promotion. On ECR ordering: also agreed it does not block the first release if the alias timing is awkward, mirroring the exact artifact later is fine. Everything you flagged to preserve is untouched: native per-arch runners with the architecture assertion, smoke before any credential exists, save/load artifact transfer, full-SHA pins, licence gate, single-approval environment. |
2 participants
What
Adds
.github/workflows/release-image.yml, which builds, tests and publishes thePostgreSQL container image to Docker Hub as
extenddb/extenddb-postgresforlinux/amd64 and linux/arm64 when a
v*tag is pushed.Three jobs rather than one:
gateruns once. The version, ancestry and licence checks need no build, so abad tag fails in seconds instead of after two image builds.
buildruns one job per architecture on its native runner (ubuntu-latestfor amd64,
ubuntu-24.04-armfor arm64). No qemu, so the arm64 image is genuinelyexecuted rather than published untested. Each job builds with
load: true, assertsthe image really reports the expected architecture, smoke tests it via
EXTENDDB_IMAGE, and saves it as a workflow artifact.docker save/loadpreserves the image ID, so the artifact is bit-for-bit what was tested. These
jobs hold no registry credentials.
publishruns once, after both architectures pass, and is the only job thatreferences the
dockerhubenvironment. The token therefore never exists in a jobthat is executing a freshly built image, and never exists at all unless both
architectures passed. It also keeps this to a single approval, where putting the
environment on a matrix job would prompt twice.
Publishing pushes the two per-architecture tags, combines them into a manifest list
for both
:VERSIONand:latest, then re-inspects the published tag to assert bothplatforms are actually present.
Three gates run before anything is built:
Cargo.toml.origin/main, so a tag pushed on anunmerged branch cannot publish unreviewed code to a public namespace.
devtools/generate-software-license-notices --checkmust pass.All six actions are pinned to commit SHAs rather than floating tags, matching the
discipline the
Dockerfilealready applies to its base images by digest.Why
PR #241 landed a production-shaped PostgreSQL image and a 334-line container smoke
test, but nothing in
.github/workflowsreferenced theDockerfile,ci/smoke-test-container.shorcargo-about. Both the hardening proof and thelicence-notice currency check therefore only ran when someone remembered to run them
locally, which on a hardening change means container regressions and a stale
SOFTWARE-LICENSE-NOTICES.htmlcould ship silently.arm64 is included from the start rather than added later because an amd64-only image
is the wrong default for a local DynamoDB emulator: a large share of the audience
develops on Apple Silicon, where such an image runs under emulation with a
platform-mismatch warning and a real performance cost.
The version gate exists for a concrete reason:
mainhas already sat at 0.1.3 whilethe newest tag claimed 0.1.2. This turns that divergence into a build failure rather
than a silently mislabelled artifact.
Closes #
Testing done
arm64 was validated locally end to end before this was written, because adding a
platform without executing it would be a guess:
linux/arm64/v8, sothe digest pins do not lock the build to one architecture.
tini=0.19.0-1+b3is the arm64candidate, so the Debian binNMU suffix is not a problem as I first suspected.
container smoke test run against it: health, CreateTable / PutItem / GetItem,
restart persistence, graceful SIGTERM and the migration bootstrap path all passed,
and the trap cleanup left zero containers and zero volumes.
arch=arm64and the binary inside is a 64-bit aarch64 ELF. The workflow nowasserts this same property itself, so a future platform fallback fails the build.
Also verified:
publishjob only.--checkand pinscargo-aboutto exactly0.9.0, and the smoke test doessupport a prebuilt image via
EXTENDDB_IMAGE, reading the three label values backoff it.
ubuntu-24.04-armis a standard GitHub-hosted arm64 runner, free and unlimited onpublic repositories, which this repository is.
Not verified:
because it triggers only on
v*tags. Its first real execution is its own firstverification, so expect an iteration or two rather than a clean first pass.
not pin a platform. The ExtendDB arm64 image is validated; a fully-arm64 stack is
what the native runner will exercise.
tag: v*ref restriction are configured buthave not been exercised by a real run.
Checklist
cargo test --workspace) — not applicable, no Rust source changedcargo fmt --check) — not applicable, no Rust source changedcargo clippy -- -W clippy::pedantic) — not applicable, no Rust source changedStoragetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
On tests: there is no test harness for workflow files in this repo, and the gates are
themselves the assertions. The architecture assertion and the published-manifest
check are both real tests that fail loudly.
On documentation:
docker/README.mddoes not yet describe how images are published,and separately its prerequisites list omits that a current Docker Compose needs
buildx 0.17 or later for this build (a local run with buildx 0.12.1 fails with
compose build requires buildx 0.17.0 or laterbefore compiling anything). Both areworth adding; say so and I will fold them into this PR.
ADR / RFC: n/a for an RFC, since this changes no wire protocol,
Storagetrait, authmodel, on-disk format or CLI surface. No ADR either, on the grounds that this adds
release plumbing rather than changing documented behaviour. If you would rather the
publish mechanism and its one deliberate omission were recorded as a decision, I will
add one.
Breaking changes
None. This adds a workflow and changes no existing behaviour, no existing workflow,
and no source.
Notes for review
One deliberate omission. There are no provenance or SBOM attestations. Those are
produced by the registry/OCI exporter, while
load: truerequires the local dockerexporter, and
loadis precisely what allows the image to be smoke tested before anycredential exists in the job. Adding attestations would mean pushing before testing,
or pushing to a staging repository and promoting the digest afterwards. Both are worse
trades than losing the attestation for now.
A known cosmetic cost. The per-architecture tags (
:VERSION-amd64,:VERSION-arm64) remain visible on Docker Hub, because a manifest list can onlyreference images that already exist in the registry. The tags users consume,
:VERSIONand:latest, are proper multi-arch lists.Neither the smoke test nor the licence check runs on pull requests. This gates
releases, not merges; a PR-triggered job would stop both regressing in between.
Structured for a second backend without being generalised prematurely: the image name
is a top-level
envvar and the concurrency group names the image, so the futuresplit into a
workflow_callworkflow plus one thin caller per backend is mechanical.A matrix across backends would be the wrong shape, because a partial failure would
publish one backend under a version tag and not the other.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.