Port pti blackbox tests to bakery - #819
Draft
bschwedler wants to merge 14 commits into
Draft
bschwedler wants to merge 14 commits into
bschwedler wants to merge 14 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Image build tests build the committed rendered files, never freshly rendered ones, so a macro change that isn't re-rendered into a fixture still passes CI. The new test re-renders each fixture context into a temp dir and fails if anything differs from what is committed. Also assert in check_build_artifacts, which previously called docker.image.exists() and discarded the result.
Port pti's blackbox tests to Bakery: build images that use the Jinja2 macros and assert their behavior with goss, instead of only asserting rendered shell text. One image per macro module (syspkg, python, r, quarto), one variant per scenario, across Ubuntu 22.04/24.04/26.04 and Rocky Linux 9/10 so the dnf macros and rhel paths are covered. - goss.yaml includes test/scenarios/<variant>.yaml at runtime, since non-Containerfile templates render once per version with no variant or OS. OS differences branch on IMAGE_OS_FAMILY inside scenarios. - Per-OS Containerfiles are identical and branch on Image.OS, so every scenario runs the same macros on every OS. A unit test enforces that, and that each variant has a scenario file. - Assertions avoid arch-specific paths so arm64 can be added later by listing it under each OS's platforms. - Python and R pin two versions each so multi-version macro loops are exercised. - check-yaml now skips goss scenario files, which use Go template blocks that are not valid YAML until goss renders them.
python.run_install_packages() and r.run_install_packages() emit one RUN per version, and each RUN ended by deleting the requirements/package list files. Every version after the first then failed because its input files were already gone. Only the last version's RUN now removes them. r.run_install_packages() also gains the clean parameter that python.run_install_packages() already had. Found by the macros-functional python/packages scenario, which pins two Python versions; pti only ever tested one.
Add a Blackbox Tests job that builds and goss-tests the macros-functional context through the reusable bakery-build.yml workflow. It has no needs, so it runs in parallel with the Test job, and fans out one runner per image. The CI gate requires it. - blackbox-clean-caches prunes this context's build caches, matching the other suites. - just test-macros runs the same build and dgoss locally and accepts bakery filter flags. - README documents the layout, adding scenarios, adding arm64, and the pti tests that were dropped.
A full local run builds 55 images with several GB of layers, and left them behind along with their build cache. The recipe now builds with a dedicated buildx builder and, on exit (pass or fail), removes the images it built and that builder, which discards its build cache without touching any other builder's cache. Arguments are now passed positionally so filter values with spaces, such as --image-os 'Rocky Linux 9', survive quoting.
tlmgr is a Perl script that needs core modules (File::Find) missing
from the perl-base in Debian-like base images and absent entirely on
Rocky. Without perl, tlmgr fails, and quarto's --update-path only warns
("Unable to determine a path to use when installing TeX Live"), so the
build succeeds with TinyTeX off the PATH and tlmgr unusable.
Found by the macros-functional quarto/tinytex scenario. with-macros
had the same broken install; its goss never ran tlmgr. connect-content
works only because its package lists install perl themselves.
`just test` deselects `image_build` tests, not `slow` tests. Also document the new `just test-macros` recipe.
CI built the with-macros image three times per run: two pytest image_build scenarios and the `bakery` job. The blackbox job covers the same ground on a wider macro surface, five OSes, and the same bakery-build.yml workflow, so all three are redundant. - Drop the `bakery` CI job and its cache-clean job - Drop the with-macros build and dgoss pytest scenarios - Keep with-macros as a render fixture for update/create tests and the drift check `release` needed `bakery` so that a tag only publishes once the tagged bakery works end to end in bakery-build.yml, the reusable workflow product repos call. `blackbox` is now the only job that runs bakery-build.yml, so it takes over that gate.
Run all macro scenarios for an OS on one runner so builds reuse base image layers instead of repeating those pulls in image-specific jobs. Name each job for its OS and guard the matrix against fixture drift.
Use a short, scope-neutral job name so the suite can grow beyond macro scenarios while remaining distinct from unit tests and image builds.
Retain existing cleanup behavior in the shared Python and R macros, and avoid adding Perl to every TinyTeX image. Exercise file-based installs on one version and install Perl only in the TinyTeX test fixture.
Preserve shared package-list files for every version in multi-version Python and R installs, then remove them after the final RUN. Expose the same clean control for R. Keep Perl out of Quarto's TinyTeX install path and limit its blackbox scenario to verifying TinyTeX installation.
bschwedler
force-pushed
the
port-pti-blackbox-tests
branch
from
September 30, 2026 16:56
a65de1f to
27da00a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add functional tests for Bakery's Jinja2 macros that build real images and run goss against them, replacing the pti blackbox tests. The unit tests only check rendered shell text, and the with-macros build only checked that one Ubuntu image built, so macro regressions showed up in product repos first.
The new
macros-functionalcontext builds four images (macros-syspkg,macros-python,macros-r,macros-quarto), with one variant per scenario, on Ubuntu 22.04, 24.04, 26.04 and Rocky Linux 9, 10 (amd64). It runs in a newblackboxCI job viabakery-build.yml, and locally withjust test-macros. Pro Drivers and Jupyter kernel registration aren't ported because no macro exists for them.Bugs it surfaced, fixed here:
python.run_install_packagesandr.run_install_packagesdeleted their package files after the first version, which broke multi-version installs.quarto.install()with TinyTeX didn't installperl, sotlmgrfailed and--update-pathonly warned.check_build_artifactsdidn't assert.The three with-macros image builds (two pytest scenarios and the
bakeryCI job) are removed as redundant.releasenow needsblackboxinstead ofbakery, so tags are still gated onbakery-build.ymlworking end to end.Product repos will see new rendered output after upgrading:
perlin TinyTeX installs, and package-file cleanup only after the last version.