Skip to content

Restore the apt version pins dropped by a half-finished CI run (#395) - #435

Closed
Ultimate-Storm wants to merge 2 commits into
mainfrom
fix/restore-apt-pins
Closed

Restore the apt version pins dropped by a half-finished CI run (#395)#435
Ultimate-Storm wants to merge 2 commits into
mainfrom
fix/restore-apt-pins

Conversation

@Ultimate-Storm

Copy link
Copy Markdown
Contributor

Spotted by @oleschwen in review of 518670c — and yes, it was unintended.

main currently pins zero of its 87 apt packages, while the comment directly above the block still says "version-pinned for security". Before 518670c there were 87 pins.

Cause: a half-finished automation run got merged

scripts/ci/update_apt_versions.sh is a three-step: strip the pins (commit literally titled "WIP: remove apt versions for rebuild", line 14) → rebuildre-add the resolved pins (line 29).

PR #387 (ci/apt-update) merged with only the WIP commit. The rebuild step evidently failed, so step 3 never ran and the intermediate de-pinned state landed on main. The CVE pinning has been silently absent since 2026-07-03.

This PR

Restores the apt block verbatim from the last pinned state (518670c~1).

  • Verified the diff is the exact inverse of 518670c and touches nothing else.
  • Package set is identical — 87 packages, pins only, no package added or removed.
  • Spot-checked apt=2.4.14, libc6=2.35-0ubuntu3.13, python3=3.10.6-1~22.04.1 against the Ubuntu archive: all still published. The pins are only ~9 days old, so this should build — but the build in CI is the real test, and I can't verify it locally.

Follow-up (not this PR)

The root cause is that a failed rebuild can still force-push a branch and open a mergeable PR, with nothing guarding against merging a commit named "WIP". That plus the base image being pinned by tag, not digest is tracked in #395, where I've posted a plan (digest-pin the base + snapshot.ubuntu.com so the pin list stops needing daily regeneration at all).

main currently installs 87 apt packages with NO version pins, while the comment
directly above the block still claims "version-pinned for security".

Cause: scripts/ci/update_apt_versions.sh is a three-step automation --
strip the pins (commit literally titled "WIP: remove apt versions for rebuild",
line 14), rebuild, then re-add the resolved pins (line 29). PR #387
(ci/apt-update) merged with ONLY the WIP commit (518670c): the rebuild step
evidently failed, so the re-pin never ran and the intermediate state landed on
main. The CVE pinning that block exists to provide has been silently absent
since 2026-07-03.

This restores the apt block verbatim from the last pinned state (518670c~1).
Verified the diff is the exact inverse of 518670c and touches nothing else; the
package set is identical (87 packages, pins only). Spot-checked apt=2.4.14,
libc6=2.35-0ubuntu3.13 and python3=3.10.6-1~22.04.1 against the Ubuntu archive
-- all still published -- but the build is the real test.

This is the stop-gap. The underlying fragility (a failed rebuild can still open
a mergeable PR; base image pinned by tag not digest) is tracked in #395.

Reported by Ole Schwen in review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Ultimate-Storm Ultimate-Storm self-assigned this Jul 11, 2026
@Ultimate-Storm
Ultimate-Storm requested a review from oleschwen July 11, 2026 08:38
The straight revert failed CI: five of the restored pins no longer exist in the
Ubuntu archive, superseded in the 9 days the pins were absent from main.

  libpython3.10-minimal / libpython3.10-stdlib / python3.10 / python3.10-minimal
      3.10.12-1~22.04.15 -> 3.10.12-1~22.04.16
  libnghttp2-14
      1.43.0-1ubuntu0.3  -> 1.43.0-1ubuntu0.4

Both are security updates, so taking the newer versions is what the pin block is
for. Resolved every one of the 87 packages against apt's candidate version inside
the real base image (pytorch/pytorch:2.2.2-cuda12.1-cudnn8-runtime) rather than
spot-checking, then dry-ran the exact apt command with --simulate: exit 0, all 87
installable.

This is also a live demonstration of why #395 matters: pins rot within days, and
the daily regenerate-and-force-push cron is a treadmill, not a fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Ultimate-Storm

Copy link
Copy Markdown
Contributor Author

The straight revert failed CI — and the failure is itself the argument for #395.

Five of the restored pins had already been superseded in the 9 days the pins were missing from main:

E: Version '3.10.12-1~22.04.15' for 'libpython3.10-minimal' was not found
E: Version '3.10.12-1~22.04.15' for 'python3.10' was not found
E: Version '1.43.0-1ubuntu0.3'  for 'libnghttp2-14' was not found       (+2 more)

I'd spot-checked 3 of the 87 pins against Launchpad and called it good; the 5 that rotated were among the 84 I didn't check. My mistake — the build was the real test and it disagreed with me.

Now fixed properly. Rather than spot-check, I resolved all 87 packages against apt's candidate version inside the real base image (pytorch/pytorch:2.2.2-cuda12.1-cudnn8-runtime), found exactly the 5 stale ones, and bumped them:

package was now
libpython3.10-minimal / -stdlib / python3.10 / python3.10-minimal 3.10.12-1~22.04.15 3.10.12-1~22.04.16
libnghttp2-14 1.43.0-1ubuntu0.3 1.43.0-1ubuntu0.4

Both are security updates, which is exactly what this pin block exists to track. Then I dry-ran the exact apt command from the Dockerfile in the base image with --simulate: exit 0, all 87 installable.

@oleschwen — worth noting the pins rotted within 9 days. That's the treadmill #395 describes: the daily regenerate-and-force-push cron doesn't fix it, it just re-runs it (and, as here, can merge a half-finished "WIP" state). The durable fix is digest-pinning the base + snapshot.ubuntu.com.

@oleschwen oleschwen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@oleschwen

Copy link
Copy Markdown
Collaborator

Yes, we should remove pinning and the daily action via a future PR. This PR only reverts to the previously intended, but not fully working starte.

@Ultimate-Storm

Copy link
Copy Markdown
Contributor Author

Superseded by #446 — closing.

Digging into this, your own second commit here is the best argument against the PR. 3db2815 records that the straight revert failed CI because five of the restored pins no longer exist in the Ubuntu archive — superseded in the nine days the pins were absent from main — and the fix was to bump them to the versions that an unpinned apt install would have selected automatically. All five were security updates.

That is the whole case in one commit: the pins did not hold anything still. They just made the build fail until a human re-typed what apt already knew.

#446 instead:

  • digest-pins the base (it was pinned by tag — mutable — so we were freezing 87 leaf packages on top of a floating multi-GB foundation),
  • leaves apt unpinned and fixes the comment that claimed otherwise,
  • retires the cron (which had been a silent no-op for 11 days — its change-check compared a commit range while the regenerated pins sat uncommitted, so it green-ticked while skipping its own push and PR steps; it is also what put main in this state, by offering two near-identically-named branches where the mergeable one contained only a commit titled "WIP"),
  • and replaces it with a daily build-verify + CVE scan that fails hard — the security control this repo never actually had.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants