Skip to content

fix(image): pin musl in the runtime apk transaction so the image builds reproducibly - #197

Merged
antoinetoussaint-byte merged 2 commits into
mainfrom
fix/runtime-image-pin-musl
Oct 27, 2026
Merged

antoinetoussaint-byte merged 2 commits into
mainfrom
fix/runtime-image-pin-musl

Conversation

@antoinetoussaint-byte

Copy link
Copy Markdown
Contributor

What broke

Since 2026-10-26 every PR (#194, #195, #196) has failed image at "Verify runtime image candidate". All three built the same digest, sha256:e9b799a7…, against the locked sha256:20295012…, and ci was skipped. None of them touches the Dockerfile.

Root cause

The two indexes differ in exactly one layer per platform: the runtime stage's apk add --no-cache --upgrade layer. The pgvector COPY layer and every base layer are byte-identical, so the builder stage and build-base are not the cause. Inside that layer the only difference in lib/apk/db/installed is:

- musl=1.2.6-r2
+ musl=1.2.6-r3

The new layer also carries lib/ld-musl-*.so.1. apk add --upgrade upgrades the dependencies of every package it names. libcrypto3, libssl3, libuuid and su-exec all depend on musl, and musl was not named, so it floated with the Alpine v3.24 index. When Alpine published musl 1.2.6-r3, it replaced the base image's 1.2.6-r2. apk add --simulate against the pinned base confirms that musl is the only unnamed package the transaction changes.

Fix

  • Every package this apk add changes is now named at an exact version: musl=1.2.6-r3 joins libcrypto3, libssl3, libuuid and su-exec. If Alpine rebuilds musl again, the build fails with "unable to select packages" instead of drifting silently. This is the same failure mode the other pins already have.
  • TestRuntimeDockerfilePinsReproducibleBuildInputs asserts the new pin.
  • The runtime-image-relock skill now says to pin the whole transaction and shows how to list it (apk add --simulate).
  • runtime-image.json is re-locked through the skill's loop: CI publishes the candidate by digest, then the lock takes the built digest. The tag string does not change (postgres 17.10, pgvector 0.8.5, alpine 3.24).

Classification: fix, at the place that owns the behaviour (the runtime image's package pins). It is not a digest bump: the input that moved is now pinned. Reused: the repo's existing pin test, its relock loop and CI's own reproducible build flags. No new mechanism.

Evidence

Not verified / remaining risk

  • The pgvector builder stage still installs about 45 unpinned transitive packages under build-base/clang21/llvm21-dev (gcc, binutils, musl-dev, …). They also moved on 2026-10-26, and the .so came out byte-identical, so they did not cause this break. Their output is still not pinned, though. If a toolchain rebuild ever changes the pgvector bytes, this same check will catch it.
  • I did not push a candidate from my machine. CI's publish is the one that lands in ghcr.

🤖 Generated with Claude Code

antoinetoussaint-byte and others added 2 commits October 27, 2026 10:11
…ds reproducibly

`apk add --upgrade` upgrades the dependencies of every package it names.
libcrypto3, libssl3, libuuid and su-exec all depend on musl, which was not
named, so musl floated with the Alpine v3.24 index. Alpine published
musl 1.2.6-r3 around 2026-10-26, the runtime layer picked it up in place of
the base image's 1.2.6-r2, and the image digest moved with no Dockerfile
change (sha256:e9b799a7… on every PR, against the locked sha256:20295012…).

Name musl=1.2.6-r3 so every package the transaction changes carries an
exact version, assert it in TestRuntimeDockerfilePinsReproducibleBuildInputs,
and record the whole-transaction rule in the runtime-image-relock skill.
runtime-image.json moves in the next commit, once CI has published this
candidate by digest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI published the candidate for 5e8b184 by digest and built
sha256:1d7680adb926a17a378b4f56e4b33ed98b64c8dabf39f2301b6d5d71ff5c75e3,
the same digest two local --no-cache builds produce with CI's buildkit
and flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antoinetoussaint-byte
antoinetoussaint-byte merged commit 9b1032e into main Oct 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant