Supply chain: digest-pin the base, keep apt unpinned, scan for CVEs (#395)

[Ultimate-Storm]

Closes #395. Supersedes #435 (which I'm closing).

The state of main is not what the Dockerfile says

0 of 87 apt packages have been pinned since 2026-07-03. PR #387 merged the cron's temporary "WIP: remove apt versions for rebuild" strip commit; the PR carrying the correct re-add (#385) was closed. Dockerfile_ODELIA has been claiming "version-pinned for security" ever since.

Why ratify that rather than restore the pins

• They gave no reproducibility. The base was pinned by tag — mutable, re-pushable — so we froze 87 leaf packages on top of a floating multi-GB foundation.
• They gave no security; they blocked it. A pin freezes an old patch version. Restore the apt version pins dropped by a half-finished CI run (#395) #435 proved this against its own intent: restoring 9-day-old pins broke CI because five had vanished from the archive, and the fix (3db2815) was to bump them to exactly what unpinned apt already installs. All five were security updates.

What this does

base image	tag → @sha256: digest (the only pin here that makes a build reproducible)
apt	stays unpinned, and the comment now says so honestly
pip	unchanged, but the 3 load-bearing pins now say why (none of the 112 did): numpy==1.26.4 predates the NumPy-2 ABI break; torchvision/torchaudio are the exact companions of the base image's torch 2.2.2
cron	retired. It was a silent no-op for 11 days (its change-check compared a commit range while the regenerated pins sat uncommitted → green tick, push+PR skipped)
replacement	daily job that builds from current sources, proves it still trains, CVE-scans the image, retains the SBOM — and fails hard
dependabot	added (there was none)

Answers @oleschwen's "isn't digest-pinning the base while unpinning apt inconsistent?" — no, that's the crux: pin what is mutable but persistent (the base), float what is immutable but garbage-collected (apt). Sites never build, so build reproducibility was never the requirement; image-level reproducibility (digest + SBOM) is, and the registry gives it free.

CVE scan fails only on fixable HIGH/CRITICAL — unfixed CVEs in the CUDA base are reported, not blocking, so the gate stays actionable.

Verification: confirmed the digest resolves and that a real docker build off this exact ARG/FROM succeeds. Unit suite: 232 passed. Dockerfile_STAMP untouched to avoid conflicting with #421 (follow-up).

[oleschwen]

Can we shorten the list of apt packages to those we actually need (excluding their dependencies)? Currently, it's a mixed list of desired packages and (possibly outdated, possibly incomplete by now) dependencies the desired packages had at some point in the past.

[Ultimate-Storm]

@oleschwen is right, and I'd go a step further — the list is worse than cosmetic once the pins are gone.

Counting it: of the 87 names, about eight are actual intent — ca-certificates, curl, gnupg, openssl, unzip, zip, apt-transport-https, lsb-release. The rest is transitive closure: libkrb5support0, libpam-modules-bin, libext2fs2, logsave, libc6… Nobody chose those; they're the residue of a "which packages had CVE updates that day" snapshot.

One subtlety before anyone just deletes them. The list isn't only documentation — naming a transitive package explicitly makes apt upgrade it to the newest available even when the installed version already satisfies dependencies. That was the CVE-remediation mechanism. Dropping the 79 without replacing it would quietly reduce patch coverage, so the PR would be weaker than it looks.

But the mechanism is wrong in a way that bites this PR specifically. I checked: 48 of the 87 carry a version in the name — libpython3.10-minimal, libpython3.10-stdlib, python3.10, libapt-pkg6.0, libssl3, libldap-2.5-0, libnghttp2-14, libreadline8, libsasl2-2, libssh-4, libcurl4, libgnutls30, libkrb5-3, … Those names are tied to the distro/Python release of the base image. This PR's whole point is to make base bumps deliberate via the digest — and the first time that digest moves to a newer Ubuntu, a couple of dozen of these stop existing and the build dies on packages nobody ever intended to request. The PR promotes base bumps to a first-class operation while leaving a landmine that only detonates on base bumps.

Suggestion that satisfies Ole and strengthens the PR's own thesis:

RUN apt-get update \
 && apt-get upgrade -y \
 && apt-get install -y --no-install-recommends \
      apt-transport-https ca-certificates curl gnupg lsb-release openssl unzip zip \
 && rm -rf /var/lib/apt/lists/*

apt-get upgrade covers the entire base closure rather than a hand-picked subset, so it is strictly more security coverage than today, it can never go stale (Ole's "possibly incomplete by now" disappears by construction), and it survives base bumps. The explicit list then means exactly what it says.

This is also more consistent with the PR's argument: it already decided the CVE scan + SBOM is the control and apt should float. An 79-name pseudo-manifest is a leftover of the pinning philosophy this PR is retiring.

Caveats, honestly:

• apt-get upgrade, not dist-upgrade — avoid it adding/removing packages to satisfy new deps.
• Less deterministic, but that ship sailed with unpinned apt; the digest fixes the starting point and the scan is the gate.
• Needs real verification: image builds, size delta sane, and training still passes. The plausible failure is an upgraded base library conflicting with the CUDA / torch 2.2.2 stack — and the new daily job (build → prove it trains → CVE-scan) is exactly the right harness for it.

Two notes on merging, since main moved under this PR:

1. ✅ I verified the pinned digest sha256:923f6877… is exactly what pytorch/pytorch:2.2.2-cuda12.1-cudnn8-runtime resolves to right now, i.e. the same base the just-released jefftud/odelia:1.6.0 was built from. So merging does not silently move the base under the shipped release — good, and worth stating in the PR body as a checked fact.
2. Dockerfile_ODELIA now carries ARG NVF_VERSION=2.8.0 (backbone: NVFlare 2.7.2 → 2.8.0 (#392 P2) [DRAFT — gated on canary] #462), so this will want a rebase.

[oleschwen]

Yes, that sounds good.

• Doing update, upgrade and installation in one step gives us a consistent "latest" state rather than updating only selected packages.
• This will give us a more consistent state of pinned base image + pinned python packages with explicit dependencies (where I would expect possible incompatibilities by silent updates) vs. "latest" apt system packages (where I would expect benefit of having the latest patches, but few incompatibility problems).
• We could, at some point, also update python packages to a newer state—but that's beyond this PR.