test(e2e): migrate the features and drivers suites to testkit by matthewgrossman · Pull Request #5137 · NVIDIA/OpenShell

matthewgrossman · 2026-11-02T18:14:05Z

Summary

Port the remaining suites that still use crates/openshell-conformance (features/oci-image, features/provider-refresh/keycloak and the Podman user-namespace suite in drivers/podman) to testkit, and run them through e2e-suites.yml with explicit CI rows. This is a parity change: each suite keeps its current environments and installers. The last commit deletes crates/openshell-conformance.

Related Issue

Part of #3954

Changes

Support crates (tests/support): container-engine builds images into the gateway's image store through OPENSHELL_TEST_CONTAINER_ENGINE. podman reads OPENSHELL_TEST_PODMAN_USERNS and computes Podman's own UID map for a mode. testkit gains Context::expect_compute_driver and testkit::base_image() (OPENSHELL_TEST_BASE_IMAGE, or the default sandbox image). The support unit tests run in the Rust test lists.
OCI image (tests/suites/features/oci-image, package renamed to oci-image): rewritten as #[testkit::test] tests. The Docker, rootful Podman and rootless Podman environments publish OPENSHELL_TEST_CONTAINER_ENGINE (docker, sudo -n podman, podman). mise run e2e:oci-image(:podman) and the ARM64 Docker lane use the new package name.
Keycloak provider refresh: the testkit suite (keycloak-provider-refresh, from test(features): run feature tests against environment services, starting with Keycloak #4891) already matches the old test. This PR adds the fedora-podman-{rootful,rootless}-keycloak environments for it and deletes the old provider-refresh/keycloak crate. On Podman, the keycloak role now starts the container with --restart always and enables podman-restart.service (the system unit when rootful, tmachine's user unit when rootless), so Keycloak comes back when a cached disk boots. Docker keeps unless-stopped.
Podman user namespaces (tests/suites/drivers/podman, package podman-driver, because podman is the support crate's name): one test compares the sandbox's UID map with Podman's for the configured mode and checks that the non-root workload owns and can write /sandbox. As in the old playbooks, the reference container runs and is removed before the sandbox is created: in auto mode Podman gives each running container its own range, so the sandbox gets the range the reference released.
- Eight environments, fedora-podman-{rootful,rootless}-userns-{default,auto,keep-id,private}, run playbooks/drivers/podman/userns.yaml. It writes the mode's [openshell.drivers.podman] settings to /etc/openshell-test/podman-driver.toml and publishes OPENSHELL_TEST_PODMAN_USERNS. In default mode it removes that file, so gateway.toml has no userns, uidmap or gidmap settings (the old baseline asserted this).
- The openshell_gateway role (binaries installer) adds those settings to gateway.toml before the gateway starts. The packaged installers fail on such an environment instead of testing without the mode.
- This replaces the in-VM chaining of default-userns-baseline.yaml, tests.yaml and the userns-*.yaml playbooks.
CI:
- feature-specific-integration (Branch E2E, Release Dev, Release Qualification; job ID kept) now calls e2e-suites.yml with family: features. Branch E2E's separate e2e-suites job is folded into it.
- driver-specific-integration (Branch E2E) calls it with family: drivers.
- The tmachine e2e-podman row is unchanged and moves to its own podman-e2e-integration job, which is added to the Core E2E result.
- The manual Integration Tests workflow defaults to the e2e-podman row.
Removed: the tmachine oci-image, provider-refresh and driver-podman testsuites, their playbooks, fixtures and test archives.
tests/suites/drivers is added to ALIGNED_WORKSPACES. The fmt/clippy lists (Branch Checks Format and Lint steps, rust:lint, rust:format, rust:format:check) now run once per test workspace (conformance, drivers, features) instead of once per package, so new suite crates need no lint-list edit. This also lints features/kubernetes-updates for the first time; it raises no warnings. TESTING.md and CI.md are updated.

CI rows after this PR (e2e-suites.yml):

Family	Package	Environments	Installer
features	`keycloak-provider-refresh`	`ubuntu-docker-rootful-keycloak`	deb
features	`keycloak-provider-refresh`	`fedora-podman-{rootful,rootless}-keycloak`	rpm
features	`oci-image`	`ubuntu-docker-rootful`	deb
features	`oci-image`	`fedora-podman-{rootful,rootless}`	rpm
drivers	`podman-driver`	`fedora-podman-{rootful,rootless}-userns-{default,auto,keep-id,private}`	binaries

The drivers family runs only in Branch E2E, as driver-specific-integration does today. The Docker Keycloak row now also runs in Release Dev and Release Qualification, because it belongs to the features family.

Still to come in this PR: nothing. The last commit deletes crates/openshell-conformance, its Cargo.lock entry and its AGENTS.md architecture row. No other code used the crate. tmachine's testsuites list is now just shell (interactive) and e2e-podman. nextest.yaml stays because the runner rows still run nextest in the guest.

Cleanup of providers, settings and fixture resources stays inline, as in the old suites, so a panic can leak them. Names are unique and the VMs are throwaway.

Testing

Checks appropriate to the affected code and behavior pass: cargo fmt and cargo clippy -D warnings for every list entry, including each whole test workspace, cargo check --workspace --locked, lockfile checks and alignment, mise run rust:deny:policy, mise run license:check, markdownlint-cli2, actionlint (no new findings), zizmor (no new findings against main), ansible-playbook --syntax-check on the changed playbooks, and nix build .#tmachine-config
Unit tests added/updated (if applicable): tests/support UID-map normalization
E2E tests added/updated (if applicable)

Local runs:

mise run e2e:oci-image against a local Docker gateway: 6/6 passed, with no images or sandboxes left behind. Changing an expected UID made the test fail and still clean up.
The gateway-podman.toml.j2 template rendered for the default and private modes and parsed as TOML.
sudo -n podman compared with the old playbooks' context, in a Fedora 44 container: sudo -n sets HOME=/root and drops XDG_RUNTIME_DIR, but rootful Podman's graph root, run root, temp directory and socket are identical to the old HOME=/root XDG_RUNTIME_DIR=/run/user/0 context. Rootless runs as tmachine over SSH, where PAM sets XDG_RUNTIME_DIR.

Not verified locally (tmachine needs an amd64 KVM host): every tmachine row in the table above, including the Podman OCI rows, the eight user-namespace rows, the Fedora Keycloak rows, and Keycloak restarting after a cached-disk boot.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable)

copy-pr-bot · 2026-11-02T18:14:09Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-11-02T18:14:49Z

🌿 Preview your docs: https://nvidia-preview-pr-5137.docs.buildwithfern.com/openshell

matthewgrossman · 2026-11-02T18:20:48Z

/ok to test b07fbc4

github-actions · 2026-11-02T18:21:20Z

Label test:e2e applied for b07fbc4. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

matthewgrossman · 2026-11-02T19:33:05Z

/ok to test 78af387

matthewgrossman · 2026-11-02T21:06:37Z

/ok to test 7a8d936

matthewgrossman · 2026-11-02T23:07:14Z

/ok to test e9281a5

Add the container-engine and podman support crates for suites that build images into the gateway's store or compare against Podman's own containers, and add expect_compute_driver and base_image to testkit. Run the support crates' unit tests in the Rust test lists. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Rewrite tests/suites/features/oci-image as testkit tests (package oci-image) that build their images through OPENSHELL_TEST_CONTAINER_ENGINE, which the Docker and Podman environments now publish. Run it through e2e-suites.yml on the Docker rootful, Podman rootful and Podman rootless rows, and retire the tmachine oci-image testsuite, its playbook and its archive. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add fedora-podman-{rootful,rootless}-keycloak environments and run keycloak-provider-refresh there with the rpm installer, replacing the old provider-refresh crate, its tmachine testsuite and its archive. Podman has no daemon to restart Keycloak at boot, so start the container with --restart always and enable podman-restart.service for the user that runs it. feature-specific-integration now runs the features family through e2e-suites.yml in Branch E2E, Release Dev and Release Qualification, and absorbs Branch E2E's separate e2e-suites job. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Rewrite tests/suites/drivers/podman as a testkit test (package podman-driver) that compares a sandbox's UID map with the one Podman gives a container in the same user-namespace mode. Each fedora-podman-{rootful,rootless}-userns-<mode> environment configures that mode for the binaries installer's gateway and publishes it; the packaged installers refuse such environments rather than test without the mode. driver-specific-integration now runs the drivers family through e2e-suites.yml, and the e2e-podman tmachine row moves to its own podman-e2e-integration job. Retire the driver-podman testsuite, its playbooks, fixtures and archive. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Every suite that used the CLI conformance library now runs on testkit, so remove the crate, its lockfile entry and its row in the AGENTS.md architecture table. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…andbox In auto mode Podman gives each running container its own UID range. The ported test ran the reference container while the sandbox still held the first range, so the reference got the next one and the auto rows failed. Run the reference first, as the old playbooks did, and let it pull the base image if it is missing. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The old default-userns baseline asserted that gateway.toml had no userns, uidmap or gidmap settings. The driver fragment is now their only source, so remove it in default mode instead of only skipping the write. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-03T02:29:58Z

/ok to test d0d6e10

Replace the per-package cargo fmt and cargo clippy lines for the conformance, drivers and features test workspaces with one line per workspace, in the Branch Checks Format and Lint steps and in the rust:lint, rust:format and rust:format:check tasks. New suite crates are covered without editing the lint lists, and the features workspace's kubernetes-updates crate is now linted; it raises no warnings. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-03T06:26:30Z

/ok to test 988d5c3

matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 2, 2026

matthewgrossman force-pushed the test/3954-migrate-features-drivers/matthewgrossman branch from 78af387 to 7a8d936 Compare November 2, 2026 21:06

matthewgrossman mentioned this pull request Nov 2, 2026

test(e2e): run the testkit conformance suite in the host lanes #5078

Merged

4 of 6 tasks

matthewgrossman force-pushed the test/3954-migrate-features-drivers/matthewgrossman branch from 7a8d936 to e9281a5 Compare November 2, 2026 23:07

Base automatically changed from test/3954-host-lanes-to-testkit/matthewgrossman to main November 3, 2026 02:26

matthewgrossman added 7 commits November 2, 2026 18:27

test(conformance): delete the openshell-conformance crate

efd2aae

Every suite that used the CLI conformance library now runs on testkit, so remove the crate, its lockfile entry and its row in the AGENTS.md architecture table. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/3954-migrate-features-drivers/matthewgrossman branch from e9281a5 to d0d6e10 Compare November 3, 2026 02:29

matthewgrossman marked this pull request as ready for review November 3, 2026 06:16

matthewgrossman requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners November 3, 2026 06:16

Navigation Menu

Repository navigation

test(e2e): migrate the features and drivers suites to testkit - #5137

test(e2e): migrate the features and drivers suites to testkit#5137
matthewgrossman wants to merge 8 commits into
mainfrom
test/3954-migrate-features-drivers/matthewgrossman

matthewgrossman commented Nov 2, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

matthewgrossman commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

matthewgrossman commented Nov 2, 2026 •

edited

Loading