Skip to content

test(e2e): migrate the features and drivers suites to testkit #5137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
matthewgrossman wants to merge 9 commits into
base: main
Choose a base branch
from
+5,387 −5,677
Open

test(e2e): migrate the features and drivers suites to testkit #5137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8ba3a52
test(support): add container engine and Podman support crates
matthewgrossman Nov 2, 2026
7ea9eb3
test(features): port the OCI image suite to testkit
matthewgrossman Nov 2, 2026
3e910dd
test(features): run the Keycloak provider-refresh suite on Fedora Podman
matthewgrossman Nov 2, 2026
bfb2256
test(drivers): port the Podman user-namespace suite to testkit
matthewgrossman Nov 2, 2026
efd2aae
test(conformance): delete the openshell-conformance crate
matthewgrossman Nov 2, 2026
2b6e438
test(drivers): take the Podman userns reference before creating the s…
matthewgrossman Nov 2, 2026
d0d6e10
test(drivers): keep Podman userns settings unset in default mode
matthewgrossman Nov 2, 2026
988d5c3
ci(tests): format and lint each test workspace as a whole
matthewgrossman Nov 3, 2026
c41046c
refactor(tests): name driver suites after the driver
matthewgrossman Nov 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/branch-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,8 +233,9 @@ jobs:
cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check
cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance -- --check
cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml --all -- --check
cargo fmt --manifest-path tests/suites/drivers/Cargo.toml --all -- --check
cargo fmt --manifest-path tests/suites/features/Cargo.toml --all -- --check
cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check

cargo-lockfiles:
Expand Down Expand Up @@ -297,8 +298,9 @@ jobs:
cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml -p conformance --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/suites/drivers/Cargo.toml --workspace --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/suites/features/Cargo.toml --workspace --all-targets -- -D warnings
cargo clippy --locked --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings

rust-test:
Expand Down Expand Up @@ -389,6 +391,7 @@ jobs:
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/testkit/Cargo.toml
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/tmachine/Cargo.toml
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/runner/Cargo.toml
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/support/Cargo.toml

rust-postgres-test:
name: Rust PostgreSQL tests
Expand Down
56 changes: 23 additions & 33 deletions .github/workflows/branch-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,65 +255,55 @@ jobs:
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

# Run feature-specific integration tests:
# Run the testkit feature suites (tests/suites/features) on tmachine. The
# environments and suites are listed in e2e-suites.yml.
feature-specific-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
uses: ./.github/workflows/e2e-suites.yml
with:
category: feature-specific
family: features
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"oci-image"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"oci-image"}
]
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

# Run driver-specific integration tests:
# These suites require the binary installer's CLI path, local HTTP gateway,
# and system service configuration for their Podman user-namespace fixtures.
# Run the testkit driver suites (tests/suites/drivers) on tmachine. The
# environments and suites are listed in e2e-suites.yml.
driver-specific-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
uses: ./.github/workflows/e2e-suites.yml
with:
category: driver-specific
family: drivers
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"driver-podman"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
]
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

# Run the testkit feature suites in tests/suites/ on tmachine. The
# environments and suites are listed in e2e-suites.yml; the conformance rows
# run in conformance-integration.
e2e-suites:
needs: [pr_metadata, prepare-integration]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
# Run the Podman e2e suite on tmachine. It needs the binary installer's CLI
# path and local HTTP gateway.
podman-e2e-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-suites.yml
uses: ./.github/workflows/integration-runner.yml
with:
family: features
category: driver-specific
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
test-matrix: >-
[
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"}
]

docker-e2e:
needs: [pr_metadata, build-binaries, build-images]
Expand Down Expand Up @@ -573,9 +563,9 @@ jobs:
needs:
- pr_metadata
- conformance-integration
- e2e-suites
- feature-specific-integration
- driver-specific-integration
- podman-e2e-integration
- docker-e2e
- podman-e2e
- podman-additional-ca-e2e
Expand Down
17 changes: 15 additions & 2 deletions .github/workflows/e2e-suites.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,14 +62,27 @@ jobs:
# Only amd64 for now: CI has no arm64 runner with KVM for tmachine,
# and the caller only builds amd64 inputs. The gateway is the name the
# installer registers with the CLI: `openshell` for the packages,
# `tmachine` for the k3s installers.
# `tmachine` for the k3s and binaries installers.
rows='[

@elezar elezar Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a follow-up, maybe we can consider maintaining this in some local-tracked file (although I suppose this would just be the "plan" or "matrix" that we were talking about in the initial discussions). Something like:

{
  "defaults": { "runner": "ubuntu-24.04" },
  "suites": {
    "conformance": [
      { "environment": "ubuntu-docker-rootful",  "installer": "deb" },
      { "environment": "fedora-podman-rootful",  "installer": "rpm" },
      { "environment": "fedora-podman-rootless", "installer": "rpm" },
      { "environment": "ubuntu-k3s",             "installer": "k3s" },
      { "environment": "ubuntu-k3s",             "installer": "k3s-ha" }
    ],
    "features/keycloak-provider-refresh": [
      { "environment": "ubuntu-docker-rootful-keycloak",  "installer": "deb" },
      { "environment": "fedora-podman-rootful-keycloak",  "installer": "rpm" },
      { "environment": "fedora-podman-rootless-keycloak", "installer": "rpm" }
    ],
    "features/oci-image": [
      { "environment": "ubuntu-docker-rootful",  "installer": "deb" },
      { "environment": "fedora-podman-rootful",  "installer": "rpm" },
      { "environment": "fedora-podman-rootless", "installer": "rpm" }
    ],
    "drivers/podman": [
      { "environment": "fedora-podman-rootful-userns-default",  "installer": "binaries" },
      { "environment": "fedora-podman-rootful-userns-auto",     "installer": "binaries" },
      { "environment": "fedora-podman-rootful-userns-keep-id",  "installer": "binaries" },
      { "environment": "fedora-podman-rootful-userns-private",  "installer": "binaries" },
      { "environment": "fedora-podman-rootless-userns-default", "installer": "binaries" },
      { "environment": "fedora-podman-rootless-userns-auto",    "installer": "binaries" },
      { "environment": "fedora-podman-rootless-userns-keep-id", "installer": "binaries" },
      { "environment": "fedora-podman-rootless-userns-private", "installer": "binaries" }
    ]
  }
}

although we could also consider a packages entry instead of defining these explicitly.

{"environment":"ubuntu-docker-rootful","installer":"deb","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}
{"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful-keycloak","installer":"rpm","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless-keycloak","installer":"rpm","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful-userns-default","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful-userns-auto","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful-userns-keep-id","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful-userns-private","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless-userns-default","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless-userns-auto","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless-userns-keep-id","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless-userns-private","installer":"binaries","family":"drivers","package":"podman","gateway":"tmachine","runner":"ubuntu-24.04"}
]'
matrix=$(jq -c --arg family "${FAMILY}" \
'{include: map(select($family == "" or .family == $family))}' <<< "$rows")
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/integration-runner.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@ on:
type: string
default: >-
[
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"}
]

permissions:
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/integration-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,14 @@ on:
description: Integration-test category, used for the job name and concurrency key
required: true
type: string
default: feature-specific
default: driver-specific
test-matrix:
description: JSON array of environment, installer, and testsuite combinations
required: true
type: string
default: >-
[
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"}
]

permissions:
Expand Down
14 changes: 4 additions & 10 deletions .github/workflows/release-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,19 +132,13 @@ jobs:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
uses: ./.github/workflows/e2e-suites.yml
with:
category: feature-specific
family: features
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"oci-image"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"oci-image"}
]
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

docker-e2e:
needs: [build-binaries, build-images]
Expand Down
14 changes: 4 additions & 10 deletions .github/workflows/release-qualification.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,13 @@ jobs:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
uses: ./.github/workflows/e2e-suites.yml
with:
category: feature-specific
family: features
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"oci-image"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"oci-image"}
]
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

docker-e2e:
permissions:
Expand Down
1 change: 0 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ Do not rely on this file for a full inventory. The detailed public and contribut
| Path | Components | Purpose |
|------|-----------|---------|
| `crates/openshell-cli/` | CLI binary | User-facing command-line interface |
| `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner |
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
| `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
| `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
Expand Down
11 changes: 7 additions & 4 deletions CI.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,10 +165,13 @@ selected. DEB and RPM packages build only when integration qualification is sele
the DEB on Ubuntu with Docker and installs the CLI and gateway RPMs on Fedora with
rootful and rootless Podman. These lanes run conformance using the matching runtime
images. Release Dev, Release, and Pre-release use the same package installers.
Fedora provider-refresh tests also use RPMs. The Podman driver-specific suites
retain the binary installer because their fixtures configure its system service,
local HTTP gateway, and CLI path. The manual Integration Tests workflow defaults
to the package installers and downloads the packages selected by its matrix.
The feature suites (`tests/suites/features`) run the same way, on the
package installers. The Podman driver suite (`tests/suites/drivers`) and the
`e2e-podman` lane retain the binary installer because they configure its system
service, local HTTP gateway, and CLI path. The driver suite and `e2e-podman` run
only in `Branch E2E Checks`. The manual Integration Tests workflow runs the
remaining tmachine testsuites, such as `e2e-podman`, and downloads the packages
selected by its matrix.

Three opt-in labels enable the long-running E2E suites:

Expand Down
11 changes: 0 additions & 11 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading