test(suites): port the universal conformance suites onto testkit and run them from the plan by matthewgrossman · Pull Request #4774 · NVIDIA/OpenShell

matthewgrossman · 2026-10-20T23:38:28Z

Summary

Ports the universal conformance suites (CLI, lifecycle, file-transfer, persistent-workspace) onto testkit and runs them from the cpu-tests plan on Docker, Podman (rootful and rootless) and k3s (single and HA). This is PR 2 of the plan in conformance-plan.md, stacked on the runner.

It adds the new suites without removing anything: the old suites, the hand-written release lanes (conformance-integration) and the old crates are untouched. Switching conformance-integration to the runner, and deleting the old CLI suite, is PR 3, after each environment has been shown green here a few times.

The lifecycle, persistent-workspace and file-transfer ports start from @elezar's draft #4629.

Stack

Part of a stack; each PR targets the one below it. The tmachine move (#4726) and the testkit library (#4724) have merged.

feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: the test runner, plans, tmachine Session API and a smoke suite running on CI.
test(suites): port the universal conformance suites onto testkit and run them from the plan #4774: ports the universal conformance suites and runs them in the plan on five environments.

Still to come: (3) ship openshell-conformance as one program with testkit::main(), point the host lanes at it, delete the old CLI suite and switch conformance-integration to the runner; (4) migrate features and drivers (oci-image, Keycloak providers, Podman userns, e2e-podman) and delete crates/openshell-conformance.

Related Issue

Part of #3954

Changes

tests/suites/conformance/sandbox: test files lifecycle, cli, persistent_workspace and file_transfer next to smoke. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start (MXC isn't conformant until it can restart a stopped sandbox).
cx.track and cx.adopt in testkit, for sandboxes a test creates through the CLI.
tests/environments: adds Podman (Fedora 44, rootful and rootless, RPM) and k3s and HA k3s (Ubuntu 24.04), amd64 only.
tests/plans/cpu-tests.yaml: the whole conformance family on all five environments.
Plan validation: a conformance entry must select the same suites in every environment, or the runner refuses the plan and names the environments that differ.
The k3s installer now waits for the Agent Sandbox controller to reconcile (k3s-controller-ready.yaml, moved out of the old conformance playbook, which includes it too). Tests don't wait for it.
e2e-suites.yml: the tmachine disk cache is keyed per environment. The k3s environments skip the cache, as integration-runner.yml does; its history doesn't say why, so that is still to find out.
Git is installed in every tmachine test environment (test-tools.yaml), for the .gitignore filtering test.

Old test to new test

Old	New	Notes
`smoke` (scenario: status, create, get Ready, list, exec, delete, list empty)	`sandbox/tests/cli.rs::cli_create_exec_list_delete`	Same CLI steps. `smoke.rs::create_exec_delete` is the SDK variant (in #4773).
`sandbox_lifecycle` stop_start_preserves_workspace	`persistent_workspace.rs::persistent_workspace_survives_stop_start` (SDK) and `cli.rs::cli_stop_start_delete` (CLI)	Also `lifecycle.rs::restart_stopped_sandbox`.
`sandbox_lifecycle` stopped_can_be_deleted	`lifecycle.rs::stopped_sandbox_can_be_deleted`	Also at the end of `cli_stop_start_delete`.
`file_transfer::round_trip`	`file_transfer.rs::upload_download_round_trip`
`file_transfer::git_filtering`	`file_transfer.rs::upload_respects_gitignore`	Needs `git` where the tests run.
`file_transfer::path_safety`	`file_transfer.rs::download_stays_inside_workspace`

New with no old counterpart (from #4629): lifecycle.rs::long_running_exec_delete, run_to_completion_exit_delete, default_command_sandbox_is_listed_ready.

policy_advisor has no port: it is a filter (binary(policy_advisor)) over existing conformance tests that no workflow runs, so it goes with the old suite.

Not in this PR

Switching conformance-integration: PR 3 per the plan, after each environment is shown green several times via workflow_dispatch (fallback: switch Docker first).
oci-image, Keycloak providers, Podman userns, e2e-podman: PR 4, in features and drivers. Earlier drafts of this PR ported them; they were removed because the runner has no way yet for a VM to hand a suite extra inputs (the plan's decision 8).
Find out why integration-runner.yml skips caching k3s before caching it here.
arm64: not defined; needs an arm64 KVM runner and arm64 build inputs.
The six open PRs that extend the old suite (test(conformance): move scenario files into Cargo tests #4120, test(conformance): organize stories into area modules #4536, test(conformance): migrate create-upload tests to file_transfer stories #4606, test(conformance): migrate workspace lifecycle tests to workspace stories #4607, test(conformance): migrate sandbox template tests to template stories #4608, test(conformance): migrate lifecycle main-result tests to lifecycle stories #4628): their tests will be ported later, after they land or are reworked.

Testing

Checks appropriate to the affected code and behavior pass
- Locally: fmt, clippy, the runner tests (48), testkit and lockfile alignment checks, and the plan dry run listing the suites in all five environments.
- On CI: a Branch E2E workflow_dispatch on this head is running; results will be posted here. The ports passed on Docker on an earlier head (21 tests); Podman and k3s have not run through the runner yet.
Unit tests added/updated
E2E tests added/updated

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (TESTING.md, tests/plans/README.md)

🤖 Generated with Claude Code

copy-pr-bot · 2026-10-20T23:38:32Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-10-20T23:39:04Z

🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell

copy-pr-bot · 2026-10-21T20:37:55Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

`#[testkit::test]` takes an optional `id = "t-0123abcd"` argument that gives a test a stable identity surviving renames and moves. The attribute's parsing moves into a new testkit-spec crate, which the test runner's catalog also uses to list tests from source without compiling them. The scanner refuses testkit tests inside inline modules. `#[should_panic]` stays rejected. There are no per-environment requirements: a test that cannot pass everywhere belongs in another family, and a broken one is `#[ignore]`d with an issue link. The testkit lockfile is also realigned with the root lockfile. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add a library API that boots an environment, runs commands in it with timeouts, copies files in and out, collects diagnostics and tears it down, so a caller can drive a machine without writing an Ansible playbook per suite. SSH and exec timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from leaving QEMU running. Add the `test-runner` flake package, which runs tests/runner with the toolchains and VM tools it needs. The Session API and test-runner package are adapted from #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…like the root `cargo --locked` only checks a lockfile against its own manifest. A workspace outside the root that builds root crates through path dependencies, such as the testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two lockfiles can resolve a shared crate to different versions without any check failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link against the AWS-LC that Nix provides. check_cargo_lockfile_alignment.py fails when an aligned workspace locks a registry crate that the root lockfile also contains at a version the root lockfile does not have. With --fix it re-locks the workspace starting from the root lockfile. tests/testkit is the first aligned workspace. The script and its tests come from #4629 (a37ebe9). Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add the `sandbox` crate to the conformance workspace with one testkit test that creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone. It is deliberately small: it exists so a plan can run a real test through the runner on CI before the existing suites move to testkit. The workspace's other crate, the CLI conformance suite, keeps running as it does today. The conformance workspace joins the lockfile alignment check, so its lockfile is realigned with the root lockfile. The test is adapted from the `sandbox-lifecycle` suite in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

`openshell-test-runner` expands a plan (tests/plans/*.yaml) into one job per environment, builds the family's nextest archive for the guest, boots the environment through the tmachine Session API, runs nextest in it and brings back the JUnit report and logs. No per-suite Ansible playbook is involved. The runner points the tests at the system under test with environment variables: OPENSHELL_GATEWAY (the gateway the installer registered, from the environment's `gateway` field) and OPENSHELL_BIN (the installed CLI). Nextest runs with `--test-threads 1` and a slow-timeout that terminates a hung test. The JUnit report is checked against the catalog, so a missing or skipped expected test fails the job; `#[ignore]` tests are not expected and nextest's skipped case for them is accepted. `--strict` fails on an unsupported environment, a family with no tests, or an empty selection. `--emit-github-matrix` prints the GitHub Actions matrix for a plan, so the plan is the only list of what runs where. One environment is defined, amd64 Ubuntu with rootful Docker and the Debian package, which is the one CI can provision today; tests/plans/cpu-tests.yaml runs the conformance family on it. Further environments (Podman, k3s, arm64) are added with the suites that need them. There are no waivers or per-test requirements: the plan picks suites, and a broken test is `#[ignore]`d. Most of this comes from #4629, adapted to testkit finding the gateway the way the CLI does instead of reading a target file. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and clippy for the new `sandbox` suite crate, wherever tests/testkit already runs them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they run through `mise run e2e:testkit` against a local Docker gateway and, on CI, through the test runner rather than the unit-test job. The lockfile task and the Cargo lockfiles job also run the alignment check, and its own tests run with `mise run test`. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add a reusable workflow that takes a plan. A planning job runs `openshell-test-runner test --plan <plan> --emit-github-matrix`, and one job per environment, on the runner label the matrix gives, runs `nix run .#test-runner -- test --plan <plan> --env <id> --strict`. The jobs use the same integration inputs as the existing tmachine lanes, cache the tmachine disks keyed on the Ansible tree, upload the JUnit reports and logs whether or not the run passes, and an aggregator job reports the overall result. Branch E2E calls it for tests/plans/cpu-tests.yaml when the test:e2e label is set. The job is not in the Core E2E result's needs yet, so it starts out advisory. The existing conformance, feature and driver lanes are unchanged. Modeled on the E2E Suites workflow in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Describe how a plan runs on CI and locally in TESTING.md and a new tests/plans/README.md, add a README for the sandbox suite, point the testkit README at the runner, and add rows for tests/suites, tests/tmachine, tests/runner, tests/plans and tests/environments to the project-structure tables in AGENTS.md and CONTRIBUTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

`mise run e2e:testkit` runs the conformance sandbox suite against a Docker-backed gateway started by e2e/with-docker-gateway.sh, the same way the runner runs it in a guest. It is the quick way to run the suite locally. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…ependency Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add cx.track and cx.adopt, so a test that creates a sandbox through the CLI still cleans it up after a failure: track the name before the command, adopt it afterwards for an SDK handle. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…ent-workspace suites to testkit Add the lifecycle, cli, persistent_workspace and file_transfer test files to the sandbox crate. The lifecycle, persistent-workspace and file-transfer tests are adapted from #4629. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start, and the one that can't yet (MXC) isn't conformant until it can. The policy-advisor suite needs no port: it is a filter over the existing conformance tests that no workflow runs. The OCI image, Keycloak and Podman userns suites are ported separately, in the features and drivers families. The old conformance/cli crate stays until the release lanes move to plans. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The k3s installer now waits until the Agent Sandbox controller reconciles a gated probe Sandbox, so a test that starts right after boot doesn't race it. The check moves out of the old conformance playbook into tests/tmachine/ansible/playbooks/k3s-controller-ready.yaml, which the installer and the old playbook both include. Tests themselves don't wait for the controller. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

A plan can't run part of the conformance family in one environment and a different part in another, or conformance would mean different things in different places. The runner refuses such a plan and names the environments that differ. A test that can't pass everywhere belongs in another family. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add the Podman (rootful and rootless, RPM) and k3s and HA k3s environments and put them, with Docker, in the cpu-tests plan, which now runs the whole conformance family (smoke, lifecycle, cli, persistent-workspace and file-transfer) on all five. The tmachine disk cache in e2e-suites.yml is keyed per environment; the k3s environments skip it as integration-runner.yml does. The old suites and lanes still run unchanged. arm64 is not defined. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman added this pull request to stack #4776 October 20, 2026 23:38

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch 2 times, most recently from 6d64703 to e3d9f01 Compare October 21, 2026 20:40

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from e3d9f01 to 5e2cbc5 Compare October 21, 2026 20:49

This was referenced Oct 21, 2026

test(testkit): add the testkit library for end-to-end suites #4724

Merged

feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773

Open

feat(runner): add the test runner, plans and tmachine Session API #4775

Closed

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 5e2cbc5 to cda205f Compare October 21, 2026 23:45

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from cda205f to 7c75fbf Compare October 22, 2026 04:51

matthewgrossman removed this pull request from stack #4776 October 22, 2026 04:51

matthewgrossman added this pull request to stack #4824 October 22, 2026 04:51

matthewgrossman mentioned this pull request Oct 22, 2026

test(sandbox): run a testkit suite in Branch E2E #4822

Closed

5 tasks done

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 7c75fbf to bb39723 Compare October 22, 2026 05:29

matthewgrossman changed the title ~~test(suites): move the conformance, oci-image, provider-refresh and podman suites to testkit~~ test(suites): port the old suites onto testkit and run them from the plan Oct 22, 2026

matthewgrossman removed this pull request from stack #4824 October 22, 2026 05:29

matthewgrossman added this pull request to stack #4825 October 22, 2026 05:30

matthewgrossman removed this pull request from stack #4825 October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from bb39723 to 1cb1dc8 Compare October 22, 2026 16:01

matthewgrossman added this pull request to stack #4836 October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 1cb1dc8 to fbcd389 Compare October 22, 2026 16:37

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 986fdcf to 2fdcd59 Compare October 22, 2026 17:31

matthewgrossman mentioned this pull request Oct 22, 2026

feat(tests): let external compute drivers run the OpenShell conformance suite #4848

Open

2 tasks done

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 2fdcd59 to 33b8dc1 Compare October 23, 2026 00:01

matthewgrossman changed the title ~~test(suites): port the old suites onto testkit and run them from the plan~~ test(suites): port the universal conformance suites onto testkit and run them from the plan Oct 23, 2026

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 33b8dc1 to db85da9 Compare October 23, 2026 03:37

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from db85da9 to 6f5c621 Compare October 23, 2026 04:00

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 6f5c621 to b9e6bfe Compare October 23, 2026 04:46

matthewgrossman and others added 10 commits October 22, 2026 22:28

build(cargo): re-lock the conformance suite without testkit's tonic d…

be103d6

…ependency Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman and others added 6 commits October 22, 2026 22:29

test(tmachine): install Git in every test environment

32c85f6

The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from b9e6bfe to 919f767 Compare October 23, 2026 05:30

Navigation Menu

Repository navigation

test(suites): port the universal conformance suites onto testkit and run them from the plan - #4774

test(suites): port the universal conformance suites onto testkit and run them from the plan#4774
matthewgrossman wants to merge 16 commits into
test/3954-first-testkit-lane/matthewgrossmanfrom
test/3954-migrate-suites-to-testkit/matthewgrossman

matthewgrossman commented Oct 20, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

matthewgrossman commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Related Issue

Changes

Old test to new test

Not in this PR

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

matthewgrossman commented Oct 20, 2026 •

edited

Loading