[matthewgrossman]

Summary

Turn the conformance tests into one program, openshell-test, that anyone can run against their own gateway without a Rust toolchain, cargo or nextest. #[testkit::test] now also registers each test, and testkit::main() turns a binary into a test program with list, ready and run. The conformance suite becomes a library, and tests/openshell-test/ links it into the program. Existing suites still run under cargo test and nextest.

Switching CI rows or host lanes to the program, publishing it, and adding other families are later PRs.

Related Issue

Part of #3954

Changes

Three commits (the crate rename to conformance landed in #5078):

1. feat(testkit): register tests and add a standalone test program.
   • #[testkit::test] keeps its #[cfg(test)] #[tokio::test] wrapper and also adds a link-time registry entry (linkme) with the test's family, name, description and #[ignore] reason.
     • The family is the directory under tests/suites/. The name is the same name libtest and nextest use.
     • The hidden function gets only cfg and lint attributes (allow, warn, deny, forbid). #[should_panic], #[cfg_attr] and #[expect] are rejected with an error explaining why.
   • testkit::main():
     • With no subcommand, the program takes libtest's arguments (via libtest-mimic), so it also works as a harness = false target under cargo test and nextest.
     • list [FILTER]... prints family, name and description, with ignored tests marked.
     • ready [--timeout SECS] waits for the gateway, sharing its check with testkit-ready. It exits 2 if the gateway isn't ready in time.
     • run [FILTER]... [--include-ignored] [--timeout SECS] [--grace SECS] [--junit PATH] runs tests one at a time. Each test is a new process (<name> --exact --nocapture) in its own process group, so nothing a test starts outlives it. A test that runs past its timeout (900s by default) gets SIGTERM, then SIGKILL after the grace period (10s). Interrupting run stops the current test and runs no more tests; a second interrupt skips the grace period. Output is shown live and kept (the last 4 MiB per stream) for the JUnit report. run exits 1 when a test failed and 2 when tests couldn't run or the run was interrupted. It is Unix-only; on other platforms it says to use cargo test or nextest.
   • tests/testkit/README.md documents the program and its limits.
2. refactor(conformance): move the conformance suites into a library. Cargo builds each file in tests/ as its own test executable, which no other program can link. tests/suites/conformance/tests/*.rs move to src/ as modules of a lib.rs; test code is unchanged. cargo test and nextest run the same 15 tests as the library's unit tests.
   • Nextest names change. conformance::smoke create_exec_delete (binary conformance::smoke) becomes binary conformance, test smoke::create_exec_delete. Filters that picked a suite by binary now pick it by test name, e.g. --filter 'test(/^policy_advisor::/)'. CI.md, e2e/policy-advisor/README.md, the runner's --filter help and the suite README are updated.
   • The host lanes (e2e/support/conformance.sh), mise run e2e:testkit and the tmachine runner select the suite by workspace or package, so they still select every test. The kind(test) archive filters in tests/tmachine/artifacts.nix aren't used for conformance.
3. feat(test): add the openshell-test program.
   • tests/openshell-test/ is its own workspace (later families live in other workspaces). Its main calls testkit::main() and names the suite crate with use conformance as _;. A test checks that list shows the conformance tests.
   • Rust only links a dependency that is named, so a suite the program forgets to name would silently go missing. Any testkit program that links no tests now exits 2 with an error under every command.
   • Its lockfile is aligned with the root's (ALIGNED_WORKSPACES), and Branch Checks and the mise tasks format, lint and test it. TESTING.md and the testkit README mention it.

Limits:

• Plaintext and mTLS gateways only.
• A test stopped by timeout or signal doesn't run its cleanup yet, so its sandboxes stay on the gateway.
• Not published, and CI doesn't run the program against a gateway yet.

Release Notes

For discussion with @elezar

This reverses two decisions from #4629: "no custom harness" and "one binary per suite". The reason is that openshell-test should run where the user has no cargo or nextest. The design still keeps the standard tooling (the libtest interface, so nextest and cargo test work as before), per-test process isolation, and nextest archives in CI. I'd like to agree on this here before the program becomes what CI runs.

Testing

• Checks appropriate to the affected code and behavior pass:
  • fmt and clippy (--all-targets -D warnings) are clean for testkit, conformance, features, support, the runner and openshell-test.
  • nextest: testkit 31 passed, 3 skipped (ignored macro fixtures); openshell-test 1/1; runner 10/10. testkit doctests pass.
  • openshell-test list matches cargo nextest list for the conformance workspace (15 tests, none ignored) in debug and release builds, before and after rebasing onto the rename. Removing the suite's use line gives the empty-program error.
  • Linux aarch64 (rust:1.95-bookworm): testkit cargo test, openshell-test clippy and test, and a release list (15 tests).
  • Cargo lockfile alignment and the lockfile check pass, and license headers are in place.
• Unit tests added/updated: the program checks (testkit/tests/program.rs) run fixture tests through the subcommands. They cover pass, fail, timeout, SIGTERM ignored, interrupt during and between tests, leftover helper processes, JUnit output, and a program with no tests. testkit/tests/attribute.rs checks that the registry agrees with libtest's --list and --list --ignored.
• E2E (before rebasing onto the rename): against a live mTLS Docker gateway (e2e/with-docker-gateway.sh), nextest ran the conformance workspace with --no-tests fail: 15/15 passed. openshell-test ready exited 0, and openshell-test run passed 15/15 and exited 0. Its JUnit report has one conformance suite with 15 test cases.

CI coverage: Rust format, Rust lint and Rust test (x86_64-linux, aarch64-linux, aarch64-darwin) in Branch Checks build and test tests/testkit and tests/openshell-test and lint the conformance suite. Branch E2E's conformance lanes run the moved suite under its new names; on f945820cf (the same three patches before #5078 merged) all five tmachine conformance rows and every host lane that runs conformance ran 15/15.

Not verified locally: Linux x86_64 and musl, the non-Unix run path, and a plaintext gateway.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable): tests/testkit/README.md, TESTING.md, tests/suites/conformance/README.md

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[matthewgrossman]

/ok to test ca412d5

[github-actions]

Label test:e2e applied for ca412d5. Open Branch E2E Checks, find the run for commit ca412d5, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5138.docs.buildwithfern.com/openshell

[matthewgrossman]

/ok to test 592a748

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

openshell-test-runner replaces --family conformance --package sandbox --filter 'binary_id(=sandbox::policy_advisor)' with --family conformance --package conformance --filter 'test(/^policy_advisor::/)'.

• Impact: Existing contributor and downstream CI invocations fail archive creation because the sandbox package no longer exists, while updating only the package leaves legacy suite filters selecting no tests.
• Action: Preserve the sandbox package selector and legacy suite filters for v0.1.7, or defer their removal to a minor train with explicit migration notice.
• PR: feat(testkit): register tests and add the openshell-test program #5138

Review details

• Execution: complete
• Candidate: PR #5138 (2e1b00deb0a0b0a3e8f07214380be9fb6bddf768)
• Baseline: 81cdd4ef4c100e701e5172f60538774fe9533f8c (81cdd4ef4c100e701e5172f60538774fe9533f8c)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[matthewgrossman]

/ok to test 2fecde6

[matthewgrossman]

/ok to test f945820

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

With unchanged environment, installer, family and package arguments, openshell-test-runner --filter 'binary_id(=conformance::policy_advisor)' must become openshell-test-runner --filter 'test(/^policy_advisor::/)' because the suite binary becomes a library module.

• Impact: Contributor and downstream CI scripts using the documented binary selector no longer select the policy-advisor tests and fail with runner exit code 2.
• Action: Preserve legacy conformance suite selection in v0.1.7, or defer its removal to a minor train with explicit notice and actionable migration guidance.
• PR: feat(testkit): register tests and add the openshell-test program #5138

Review details

• Execution: complete
• Candidate: PR #5138 (116373046b31e9d809523502be42254bf9f54f27)
• Baseline: aa7ce2e0bc1a4e1e69a3dba762f76b3dedf5f376 (aa7ce2e0bc1a4e1e69a3dba762f76b3dedf5f376)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[matthewgrossman]

/ok to test 8835ce6