diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index f6f9caaa1..9dcceab86 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -69,12 +69,12 @@ and guest artifacts only when their inputs changed. Leave `test-openvmm-unit`, `test-openvmm`, `build-guest --guest all`, `verify-guest-determinism`, and `benchmark` to CI. -- `nvx.py sandbox`, `nvx.py run`, and the microVM scenarios that drive them run - `nvx-rs`, which `nvx.py` builds with Cargo into `nvx-rs/target/release/` the - first time that a command needs it, in about a minute. Rebuild it with - `cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml` - after changing `nvx-rs/`, or set `NVX_RS` to another build (see - [Usage](../doc/usage.md#nvx-rs)). The `managed-lifecycle` scenario runs an +- `nvx-rs/target/release/nvx-rs run` and `nvx-rs/target/release/nvx-rs sandbox` + run microVMs and sandboxes. Build `nvx-rs` there, in about a minute, with + `cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml`, + and again after changing `nvx-rs/`. The microVM scenarios run it too, and build + it there the first time that they need it, unless `NVX_RS` names another build + (see [Usage](../doc/usage.md#nvx-rs)). The `managed-lifecycle` scenario runs an `nvx-rs` test with Cargo. - Build trees are large. Check `df -h .` before a build, keep one OpenVMM build profile at a time, and delete trees you no longer need with `cargo clean` or diff --git a/.github/skills/nvx-debug/SKILL.md b/.github/skills/nvx-debug/SKILL.md index 96c88b7ba..20f633996 100644 --- a/.github/skills/nvx-debug/SKILL.md +++ b/.github/skills/nvx-debug/SKILL.md @@ -43,13 +43,13 @@ Set OpenVMM logging in the selected execution shell and rerun the same command: ```bash export OPENVMM_LOG=debug backend=kvm # Use mshv when required; append the original options. -python3 scripts/nvx.py run --hypervisor "$backend" +nvx-rs/target/release/nvx-rs run --hypervisor "$backend" ``` ```powershell $env:OPENVMM_LOG = 'debug' # Append the options from the original failing command when needed. -python scripts\nvx.py run --hypervisor whp +nvx-rs\target\release\nvx-rs.exe run --hypervisor whp ``` Use `trace` only when `debug` does not expose the failing component. Keep the guest @@ -57,9 +57,9 @@ interactive; do not pipe the command through a logger that can detach standard input. The terminal transcript is the primary debug record. Exit a responsive guest with `/sbin/nvx-exit 0`. -Use `scripts/nvx.py run --dry-run` to inspect the generated OpenVMM command. Invoke -that printed command directly only when low-level debugging requires options the NVX -wrapper does not expose, and record every deviation. +Use `nvx-rs run --dry-run` to inspect the generated OpenVMM command. Invoke that +printed command directly only when low-level debugging requires options that +`nvx-rs` does not expose, and record every deviation. ## 4. Reduce a Benchmark Failure diff --git a/.github/skills/nvx-host-connect/SKILL.md b/.github/skills/nvx-host-connect/SKILL.md index c8d46844e..a167f9706 100644 --- a/.github/skills/nvx-host-connect/SKILL.md +++ b/.github/skills/nvx-host-connect/SKILL.md @@ -174,6 +174,9 @@ Choose artifact preparation according to the task: the package. Report the selected release as the artifact provenance. - For guest, OpenVMM, or build changes, use the relevant `build-guest`, `build-openvmm`, or `build` command. +- `nvx-rs`, which runs NVX's microVMs and sandboxes, comes with the package only + when its release was built from the checkout's `nvx-rs` sources. Otherwise, or + for `nvx-rs` changes, build it with Cargo from the repository root. - For coordinator-only changes, existing matching release artifacts may be reused after reporting their provenance. @@ -183,6 +186,8 @@ Linux examples: python3 scripts/nvx.py download --hypervisor kvm python3 scripts/nvx.py download --hypervisor mshv python3 scripts/nvx.py build +cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml \ + --target-dir nvx-rs/target ``` Windows examples: @@ -190,6 +195,8 @@ Windows examples: ```powershell python scripts\nvx.py download --hypervisor whp python scripts\nvx.py build +cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml ` + --target-dir nvx-rs/target ``` Do not download over artifacts the user wants to preserve. Verify the generated @@ -197,11 +204,11 @@ command and required files without starting a VM: ```bash backend=kvm # Use the resolved profile backend. -python3 scripts/nvx.py run --hypervisor "$backend" --dry-run +nvx-rs/target/release/nvx-rs run --hypervisor "$backend" --dry-run ``` ```powershell -python scripts\nvx.py run --hypervisor whp --dry-run +nvx-rs\target\release\nvx-rs.exe run --hypervisor whp --dry-run ``` ## Handoff diff --git a/.github/skills/nvx-openvmm-promote/SKILL.md b/.github/skills/nvx-openvmm-promote/SKILL.md index 8fc4d0f4f..01a29ea54 100644 --- a/.github/skills/nvx-openvmm-promote/SKILL.md +++ b/.github/skills/nvx-openvmm-promote/SKILL.md @@ -98,10 +98,22 @@ must follow. Copy both files unchanged into `nvx-rs/tests/vectors/openvmm/`, rec revision in [the vectors' README](../../../nvx-rs/tests/vectors/README.md#openvmms-vectors), and run the `nvx-rs` tests, which may need coordinated changes too. +`nvx-rs run` also reads a copy of the identities of OpenVMM's built-in CPU profiles, from +which it guides a failed cold boot on a CPU that none serves. Compare it with the revision's +profiles too: + +```console +cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_cpu_profiles -- --ignored +``` + +A failure means that the revision adds, revises, or removes a built-in profile. Update +`nvx-rs/src/host/cli/cpu_generations.json` and the copy of `scripts/nvx_tools/time_abi.py`, +which `scripts/test_time_abi.py` compares with the same profiles, as coordinated changes. + Before committing, require `git diff --name-only` to contain exactly `openvmm` unless coordinated NVX changes were explicitly requested, or the revision needs them, -as a change to OpenVMM's protocol vectors does. Stage only the gitlink and those -changes, and name the coordinated changes in the pull request. +as a change to OpenVMM's protocol vectors or CPU profiles does. Stage only the gitlink and +those changes, and name the coordinated changes in the pull request. Commit only when the user requested it. Follow the repository's subject convention, sign the commit with the user's existing configuration, and stop rather than falling back to an unsigned commit if signing fails. Never amend, force-push, or rewrite an @@ -168,5 +180,5 @@ head SHA matches the current pull-request head. Treat older runs as superseded a Report the source PR and revision, signature result, dependency state, equivalence method and result, NVX target commit, final changed-file set, local validation, the -comparison of OpenVMM's protocol vectors, pull-request state and URL, exact-head -workflow URL, and every blocked or pending gate. +comparisons of OpenVMM's protocol vectors and CPU profiles, pull-request state and URL, +exact-head workflow URL, and every blocked or pending gate. diff --git a/.github/skills/nvx-run/SKILL.md b/.github/skills/nvx-run/SKILL.md index 570ff3864..88e36e891 100644 --- a/.github/skills/nvx-run/SKILL.md +++ b/.github/skills/nvx-run/SKILL.md @@ -31,10 +31,17 @@ For SSH execution, load and complete repository directory, backend, revision, and artifact provenance. Keep standard input attached to the guest; use `ssh -tt` if opening a direct interactive command. -Use `scripts/nvx.py run` rather than invoking OpenVMM directly. If artifacts are -missing, choose an explicit `download --hypervisor ` for packaged artifacts -or the relevant build command for checkout changes. Do not overwrite artifacts the -user wants to preserve. +Use `nvx-rs run` rather than invoking OpenVMM directly: the checkout's +`nvx-rs/target/release/nvx-rs`, or `nvx-rs\target\release\nvx-rs.exe` on Windows. +If artifacts are missing, choose an explicit `download --hypervisor ` for +packaged artifacts or the relevant build command for checkout changes. If `download` +installed no `nvx-rs`, or the task changes `nvx-rs`, build it from the repository root: + +```console +cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target +``` + +Do not overwrite artifacts the user wants to preserve. ## 2. Assemble the Run @@ -42,25 +49,25 @@ Start with an explicit backend and add only options required by the task: ```bash backend=kvm # Use mshv when required. -python3 scripts/nvx.py run --hypervisor "$backend" --dry-run +nvx-rs/target/release/nvx-rs run --hypervisor "$backend" --dry-run ``` ```powershell -python scripts\nvx.py run --hypervisor whp --dry-run +nvx-rs\target\release\nvx-rs.exe run --hypervisor whp --dry-run ``` Supported examples: ```bash -python3 scripts/nvx.py run --hypervisor kvm \ +nvx-rs/target/release/nvx-rs run --hypervisor kvm \ --memory-mib 256 --net 10.0.0.2/24 --network-profile portable \ --cmdline "quiet loglevel=0" -python3 scripts/nvx.py run --hypervisor mshv \ +nvx-rs/target/release/nvx-rs run --hypervisor mshv \ --mount "/mnt/host,/absolute/host/share,rw" ``` ```powershell -python scripts\nvx.py run --hypervisor whp ` +nvx-rs\target\release\nvx-rs.exe run --hypervisor whp ` --mount "/mnt/host,C:\Users\me\microvm-share,rw" ``` @@ -89,12 +96,12 @@ current session did not start. ```bash backend=kvm # Use mshv when required; append the approved options. -python3 scripts/nvx.py run --hypervisor "$backend" +nvx-rs/target/release/nvx-rs run --hypervisor "$backend" ``` ```powershell # Append the options approved from the dry run when needed. -python scripts\nvx.py run --hypervisor whp +nvx-rs\target\release\nvx-rs.exe run --hypervisor whp ``` Success requires the guest's `NVX-GUEST-BOOT-OK: ` marker, such as diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b1df337b7..db8005b73 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -130,11 +130,12 @@ jobs: with: python-version: "3.12" - # nvx.py runs nvx-rs for its run and sandbox commands. A static musl - # build runs on every self-hosted Linux runner, so the microVM jobs take - # this one, as they take OpenVMM from its build jobs. It goes where - # `cargo build --release` puts it, where nvx.py, and so the CLI tests, - # look for it, and NVX_RS names it for the tests that run it directly. + # The microVM tests run their microVMs and sandboxes with nvx-rs. A + # static musl build runs on every self-hosted Linux runner, so the + # microVM jobs take this one, as they take OpenVMM from its build jobs. + # It goes where `cargo build --release` puts it, where nvx.py, and so + # the CLI tests, look for it, and NVX_RS names it for the tests that run + # it directly. - name: Build nvx-rs shell: bash run: | @@ -169,6 +170,15 @@ jobs: cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored + # nvx-rs run reads a copy of the identities of OpenVMM's built-in CPU + # profiles, to guide a failed cold boot on a CPU that none serves, which + # this job compares with the submodule's profiles as it does the vectors. + - name: Compare OpenVMM's CPU profiles + shell: bash + run: >- + cargo test --locked --manifest-path nvx-rs/Cargo.toml + --no-default-features --test openvmm_cpu_profiles -- --ignored + - name: Upload nvx-rs executable uses: actions/upload-artifact@v7 with: @@ -266,6 +276,15 @@ jobs: --test openvmm_vectors -- --ignored if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + # As on Linux. + - name: Compare OpenVMM's CPU profiles + shell: powershell + run: | + $ErrorActionPreference = "Stop" + cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features ` + --test openvmm_cpu_profiles -- --ignored + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + - name: Upload nvx-rs executable uses: actions/upload-artifact@v7 with: diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index d3cdd7e3c..81520bbae 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -154,9 +154,9 @@ jobs: shell: bash run: chmod +x openvmm/target/release/openvmm - # nvx.py runs nvx-rs for its sandbox command, which the scenarios and - # the steps below drive, and for the egress and artifact steps of its run - # command. It looks for nvx-rs where `cargo build --release` puts it. + # The scenarios and the steps below run their sandboxes and microVMs + # with nvx-rs, which they find where `cargo build --release` puts it, + # and whose artifact root is then this checkout. - name: Download nvx-rs executable uses: actions/download-artifact@v8 with: @@ -309,7 +309,7 @@ jobs: cp --reflink=auto build/ubuntu-smoke-scratch.ext4 "${scratch}" # The probe checks the security profile and that the 32 MiB memory # and 8-process limits hold. - python3 scripts/nvx.py sandbox \ + nvx-rs/target/release/nvx-rs sandbox \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ --scratch "${scratch}" \ @@ -342,7 +342,7 @@ jobs: printf 'secret\n' >"${share}/nvx-denied/secret" printf 'host-outside\n' >"${outside}" chmod 0777 "${share}" "${share}/nvx-links" - python3 scripts/nvx.py sandbox \ + nvx-rs/target/release/nvx-rs sandbox \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ --scratch "${scratch}" \ @@ -368,7 +368,7 @@ jobs: rm -rf "${share}/nvx-guest-marker" "${share}/nvx-guest-directory" \ "${share}/nvx-tool" "${share}/nvx-links" cp --reflink=auto build/ubuntu-smoke-scratch.ext4 "${scratch}" - python3 scripts/nvx.py sandbox \ + nvx-rs/target/release/nvx-rs sandbox \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ --scratch "${scratch}" \ @@ -398,7 +398,7 @@ jobs: state=$(mktemp -d) cleanup() { if [ -f "${state}/runtime.json" ]; then - python3 scripts/nvx.py sandbox stop --state-dir "${state}" --timeout 30 || true + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" --timeout 30 || true fi mkdir -p "${logs}" cp "${state}/openvmm.log" "${logs}/managed-share-openvmm.log" 2>/dev/null || true @@ -412,7 +412,7 @@ jobs: printf 'secret\n' >"${share}/nvx-denied/secret" printf 'host-outside\n' >"${outside}" chmod 0777 "${share}" "${share}/nvx-links" - python3 scripts/nvx.py sandbox provision \ + nvx-rs/target/release/nvx-rs sandbox provision \ --state-dir "${state}" \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ @@ -420,13 +420,13 @@ jobs: --mount "/workspace,${share},rw" \ --mount-deny nvx-denied \ --workload-user 65534:65534 - python3 scripts/nvx.py sandbox start --state-dir "${state}" - python3 scripts/nvx.py sandbox exec \ + nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox exec \ --state-dir "${state}" \ --entrypoint /sbin/nvx-sandbox-smoke \ --arg /workspace \ --arg rw - python3 scripts/nvx.py sandbox stop --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" # A clean ext4 journal shows that stop unmounted the share, overlay, # and scratch in order instead of powering off with them mounted. python3 - "${state}/outcome.json" "${scratch}" <<'EOF' @@ -454,7 +454,7 @@ jobs: test ! -L "${share}/nvx-links/marker" test "$(cat "${outside}")" = host-outside test "$(cat "${share}/nvx-denied/secret")" = secret - python3 scripts/nvx.py sandbox deprovision --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox deprovision --state-dir "${state}" rm -rf "${share}" "${scratch}" "${outside}" - name: Run Ubuntu sandbox caller-owned live-share test on Linux @@ -472,7 +472,7 @@ jobs: state=$(mktemp -d) cleanup() { if [ -f "${state}/runtime.json" ]; then - python3 scripts/nvx.py sandbox stop --state-dir "${state}" --timeout 30 || true + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" --timeout 30 || true fi mkdir -p "${logs}" cp "${state}/openvmm.log" "${logs}/caller-share-openvmm.log" 2>/dev/null || true @@ -488,7 +488,7 @@ jobs: # with EPERM instead of running as the runner. The OpenVMM unit tests # cover the identity of each request, including guest root squash. cp --reflink=auto build/ubuntu-smoke-scratch.ext4 "${scratch}" - python3 scripts/nvx.py sandbox \ + nvx-rs/target/release/nvx-rs sandbox \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ --scratch "${scratch}" \ @@ -505,7 +505,7 @@ jobs: # mode squashes to the share's owner, before any workload runs, so # that request fails with EPERM too, and the sandbox never starts. cp --reflink=auto build/ubuntu-smoke-scratch.ext4 "${scratch}" - python3 scripts/nvx.py sandbox provision \ + nvx-rs/target/release/nvx-rs sandbox provision \ --state-dir "${state}" \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ @@ -522,7 +522,7 @@ jobs: if config["format"] != 7 or config["export"]["owner"] != "caller": sys.exit(f"managed sandbox did not persist caller ownership: {config}") EOF - if python3 scripts/nvx.py sandbox start --state-dir "${state}"; then + if nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}"; then echo "the caller-owned managed sandbox reached its share as the runner" >&2 exit 1 fi @@ -531,7 +531,7 @@ jobs: grep -q 'NVX-MANAGED-ERROR: stage=host-mappings' "${state}/openvmm.log" test ! -e "${state}/runtime.json" test "$(ls -A "${share}")" = nvx-host-marker - python3 scripts/nvx.py sandbox deprovision --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox deprovision --state-dir "${state}" rm -rf "${share}" "${scratch}" - name: Run Ubuntu sandbox concurrent read-write and read-only shares on Linux @@ -552,7 +552,7 @@ jobs: state=$(mktemp -d) cleanup() { if [ -f "${state}/runtime.json" ]; then - python3 scripts/nvx.py sandbox stop --state-dir "${state}" --timeout 30 || true + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" --timeout 30 || true fi mkdir -p "${logs}" cp "${state}/openvmm.log" "${logs}/shares-openvmm.log" 2>/dev/null || true @@ -611,7 +611,7 @@ jobs: --arg /results/output.txt --arg file-rw ) prepare_shares - python3 scripts/nvx.py sandbox \ + nvx-rs/target/release/nvx-rs sandbox \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ --scratch "${scratch}" \ @@ -622,7 +622,7 @@ jobs: check_shares # A managed sandbox persists every share and reattaches them on start. prepare_shares - python3 scripts/nvx.py sandbox provision \ + nvx-rs/target/release/nvx-rs sandbox provision \ --state-dir "${state}" \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ @@ -660,10 +660,10 @@ jobs: if config["format"] != 7 or "mounts" in config or shares != expected: sys.exit(f"managed sandbox did not persist every share: {config}") EOF - python3 scripts/nvx.py sandbox start --state-dir "${state}" - python3 scripts/nvx.py sandbox exec --state-dir "${state}" \ + nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox exec --state-dir "${state}" \ "${smoke_arguments[@]}" - python3 scripts/nvx.py sandbox stop --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" # A clean ext4 journal shows that stop unmounted every share, the # overlay, and scratch in order. python3 - "${state}/outcome.json" "${scratch}" <<'EOF' @@ -682,7 +682,7 @@ jobs: sys.exit("managed stop left the scratch filesystem mounted") EOF check_shares - python3 scripts/nvx.py sandbox deprovision --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox deprovision --state-dir "${state}" rm -rf "${workspace}" "${toolcache}" "${files}" "${scratch}" "${outside}" - name: Run Ubuntu sandbox share access policy on Linux @@ -700,7 +700,7 @@ jobs: state=$(mktemp -d) cleanup() { if [ -f "${state}/runtime.json" ]; then - python3 scripts/nvx.py sandbox stop --state-dir "${state}" --timeout 30 || true + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" --timeout 30 || true fi mkdir -p "${logs}" cp "${state}/openvmm.log" "${logs}/policy-openvmm.log" 2>/dev/null || true @@ -743,7 +743,7 @@ jobs: --arg /workspace --arg policy ) prepare_share - python3 scripts/nvx.py sandbox \ + nvx-rs/target/release/nvx-rs sandbox \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ --scratch "${scratch}" \ @@ -755,7 +755,7 @@ jobs: # A managed sandbox persists the access policy and reapplies it on # start. prepare_share - python3 scripts/nvx.py sandbox provision \ + nvx-rs/target/release/nvx-rs sandbox provision \ --state-dir "${state}" \ --hypervisor "${{ inputs.backend }}" \ --layer "distro,build/ubuntu-distro.erofs,${uuid}" \ @@ -796,10 +796,10 @@ jobs: if config["format"] != 7 or "mounts" in config or shares != expected: sys.exit(f"managed sandbox did not persist the access policy: {config}") EOF - python3 scripts/nvx.py sandbox start --state-dir "${state}" - python3 scripts/nvx.py sandbox exec --state-dir "${state}" \ + nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox exec --state-dir "${state}" \ "${smoke_arguments[@]}" - python3 scripts/nvx.py sandbox stop --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox stop --state-dir "${state}" # A clean ext4 journal shows that stop unmounted the share, the # overlay, and scratch in order. python3 - "${state}/outcome.json" "${scratch}" <<'EOF' @@ -818,7 +818,7 @@ jobs: sys.exit("managed stop left the scratch filesystem mounted") EOF check_share - python3 scripts/nvx.py sandbox deprovision --state-dir "${state}" + nvx-rs/target/release/nvx-rs sandbox deprovision --state-dir "${state}" rm -rf "${share}" "${scratch}" - name: Run Ubuntu sandbox layer smoke test on Windows @@ -831,7 +831,7 @@ jobs: Copy-Item build\ubuntu-smoke-scratch.ext4 $scratch # The probe checks the security profile and that the 32 MiB memory # and 8-process limits hold. - python scripts\nvx.py sandbox ` + nvx-rs\target\release\nvx-rs.exe sandbox ` --hypervisor "${{ inputs.backend }}" ` --layer "distro,build\ubuntu-distro.erofs,$($manifest.uuid)" ` --scratch $scratch ` @@ -862,7 +862,7 @@ jobs: [IO.File]::WriteAllText((Join-Path $share "nvx-host-marker"), "host-to-guest`n") [IO.File]::WriteAllText((Join-Path $share "nvx-denied\secret"), "secret`n") [IO.File]::WriteAllText($outside, "host-outside`n") - python scripts\nvx.py sandbox ` + nvx-rs\target\release\nvx-rs.exe sandbox ` --hypervisor "${{ inputs.backend }}" ` --layer "distro,build\ubuntu-distro.erofs,$($manifest.uuid)" ` --scratch $scratch ` @@ -905,7 +905,7 @@ jobs: if ([IO.File]::ReadAllText((Join-Path $share "nvx-denied\secret")) -ne "secret`n") { throw "a guest link reached a denied file" } Remove-Item -Recurse -Force (Join-Path $share "nvx-guest-marker"), (Join-Path $share "nvx-guest-directory"), (Join-Path $share "nvx-tool"), (Join-Path $share "nvx-links") Copy-Item -Force build\ubuntu-smoke-scratch.ext4 $scratch - python scripts\nvx.py sandbox ` + nvx-rs\target\release\nvx-rs.exe sandbox ` --hypervisor "${{ inputs.backend }}" ` --layer "distro,build\ubuntu-distro.erofs,$($manifest.uuid)" ` --scratch $scratch ` @@ -927,9 +927,9 @@ jobs: run: | $ErrorActionPreference = "Stop" function Invoke-Nvx { - python scripts\nvx.py @args + nvx-rs\target\release\nvx-rs.exe @args if ($LASTEXITCODE -ne 0) { - throw "nvx.py $($args[0]) $($args[1]) failed with exit code $LASTEXITCODE" + throw "nvx-rs $($args[0]) $($args[1]) failed with exit code $LASTEXITCODE" } } $manifest = Get-Content -Raw build\ubuntu-distro.erofs.manifest.json | @@ -965,7 +965,7 @@ jobs: Invoke-Nvx sandbox stop --state-dir $state } finally { if (Test-Path (Join-Path $state "runtime.json")) { - python scripts\nvx.py sandbox stop --state-dir $state --timeout 30 + nvx-rs\target\release\nvx-rs.exe sandbox stop --state-dir $state --timeout 30 } New-Item -ItemType Directory -Force $logs | Out-Null $log = Join-Path $state "openvmm.log" @@ -1023,9 +1023,9 @@ jobs: run: | $ErrorActionPreference = "Stop" function Invoke-Nvx { - python scripts\nvx.py @args + nvx-rs\target\release\nvx-rs.exe @args if ($LASTEXITCODE -ne 0) { - throw "nvx.py $($args[0]) $($args[1]) failed with exit code $LASTEXITCODE" + throw "nvx-rs $($args[0]) $($args[1]) failed with exit code $LASTEXITCODE" } } $manifest = Get-Content -Raw build\ubuntu-distro.erofs.manifest.json | @@ -1127,7 +1127,7 @@ jobs: Invoke-Nvx sandbox stop --state-dir $state } finally { if (Test-Path (Join-Path $state "runtime.json")) { - python scripts\nvx.py sandbox stop --state-dir $state --timeout 30 + nvx-rs\target\release\nvx-rs.exe sandbox stop --state-dir $state --timeout 30 } New-Item -ItemType Directory -Force $logs | Out-Null $log = Join-Path $state "openvmm.log" @@ -1160,9 +1160,9 @@ jobs: run: | $ErrorActionPreference = "Stop" function Invoke-Nvx { - python scripts\nvx.py @args + nvx-rs\target\release\nvx-rs.exe @args if ($LASTEXITCODE -ne 0) { - throw "nvx.py $($args[0]) $($args[1]) failed with exit code $LASTEXITCODE" + throw "nvx-rs $($args[0]) $($args[1]) failed with exit code $LASTEXITCODE" } } $manifest = Get-Content -Raw build\ubuntu-distro.erofs.manifest.json | @@ -1238,7 +1238,7 @@ jobs: Invoke-Nvx sandbox stop --state-dir $state } finally { if (Test-Path (Join-Path $state "runtime.json")) { - python scripts\nvx.py sandbox stop --state-dir $state --timeout 30 + nvx-rs\target\release\nvx-rs.exe sandbox stop --state-dir $state --timeout 30 } New-Item -ItemType Directory -Force $logs | Out-Null $log = Join-Path $state "openvmm.log" diff --git a/README.md b/README.md index a2fb2fc9b..05a6decce 100644 --- a/README.md +++ b/README.md @@ -22,22 +22,22 @@ Python 3.10 or newer and Git are required. The commands below download the prebu for the selected platform from the cloned revision's release when available, or from its nearest released first-parent ancestor otherwise, so the guest and OpenVMM need no build. -`nvx.py run` also runs [`nvx-rs`](doc/usage.md#nvx-rs), which `download` installs only from a -release built from the checkout's `nvx-rs` sources. Otherwise, `nvx.py` builds `nvx-rs` with -Cargo the first time that a command needs it, so install Rust with [rustup](https://rustup.rs/). -A clone of the latest `dev` often needs this build, because `nvx-rs` may have changed since its -nearest release. To run without Rust, check out the tag of a +NVX runs its microVMs with [`nvx-rs`](doc/usage.md#nvx-rs), the Rust command-line interface at +`nvx-rs/target/release/nvx-rs`, which `download` installs only from a release built from the +checkout's `nvx-rs` sources. Otherwise, the commands below build it with Cargo, so install Rust +with [rustup](https://rustup.rs/). A clone of the latest `dev` often needs this build, because +`nvx-rs` may have changed since its nearest release. To run without Rust, check out the tag of a [development release](https://github.com/microsoft/nvx/releases) before `download`. A successful boot prints `NVX-GUEST-BOOT-OK: alpine` and opens a root shell. The package also includes the Ubuntu and Azure Linux userlands: boot either with the same NVX kernel by adding -`--guest ubuntu` or `--guest azurelinux` to `nvx.py run`. Exit cleanly from the guest with +`--guest ubuntu` or `--guest azurelinux` to `nvx-rs run`. Exit cleanly from the guest with `/sbin/nvx-exit 0`. See the [setup](doc/setup.md) and [run](doc/run.md) guides for detailed prerequisites and runtime options. To build NVX from source instead, first [initialize](doc/setup.md#initialize) the `openvmm` submodule, which the clone below omits. -> ℹ️ For direct OpenVMM integration without the `scripts/nvx.py` runtime harness, -see [Run OpenVMM directly](doc/run.md#run-openvmm-directly). +> ℹ️ For direct OpenVMM integration without `nvx-rs`, see +[Run OpenVMM directly](doc/run.md#run-openvmm-directly). ### Linux / KVM @@ -46,7 +46,10 @@ Requires [KVM configured with read/write access to `/dev/kvm`](doc/setup.md#linu ```bash git clone https://github.com/microsoft/nvx.git && cd nvx python3 scripts/nvx.py download -python3 scripts/nvx.py run +test -x nvx-rs/target/release/nvx-rs || + cargo build --release --locked --features cli \ + --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target +nvx-rs/target/release/nvx-rs run ``` ### Linux / MSHV @@ -56,7 +59,10 @@ Requires [MSHV configured with read/write access to `/dev/mshv`](doc/setup.md#li ```bash git clone https://github.com/microsoft/nvx.git && cd nvx python3 scripts/nvx.py download --hypervisor mshv -python3 scripts/nvx.py run --hypervisor mshv +test -x nvx-rs/target/release/nvx-rs || + cargo build --release --locked --features cli \ + --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target +nvx-rs/target/release/nvx-rs run --hypervisor mshv ``` ### Windows / WHP @@ -66,7 +72,11 @@ Requires [Windows Hypervisor Platform enabled](doc/setup.md#windows--whp). ```powershell git clone https://github.com/microsoft/nvx.git; Set-Location nvx python scripts\nvx.py download -python scripts\nvx.py run +if (-not (Test-Path nvx-rs\target\release\nvx-rs.exe)) { + cargo build --release --locked --features cli ` + --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target +} +nvx-rs\target\release\nvx-rs.exe run ``` ## Documentation @@ -77,7 +87,8 @@ python scripts\nvx.py run - [Build](doc/build.md) - Instructions for building NVX. - [Run](doc/run.md) - Instructions for running NVX. - [Benchmark](doc/benchmarks.md) - Instructions for benchmarking NVX. -- [Command-line reference](doc/usage.md) - Complete `scripts/nvx.py` command and option reference. +- [Command-line reference](doc/usage.md) - Complete `nvx-rs` and `scripts/nvx.py` command and option + reference. - [Rust API](aci_edge_sandboxes/README.md) - The `aci_edge_sandboxes` crate for driving the sandbox lifecycle from Rust. - [NVX Rust crate](nvx-rs/README.md) - The `nvx` crate that holds the NVX mechanics: the OpenVMM diff --git a/aci_edge_sandboxes/src/openvmm/config.rs b/aci_edge_sandboxes/src/openvmm/config.rs index 5b12e1981..b8ebce172 100644 --- a/aci_edge_sandboxes/src/openvmm/config.rs +++ b/aci_edge_sandboxes/src/openvmm/config.rs @@ -14,7 +14,7 @@ use crate::error::{Error, Result}; /// Construct it with [`OpenVmmConfig::discover`], [`OpenVmmConfig::from_artifacts`], /// [`OpenVmmConfig::new`], [`OpenVmmConfig::from_release_dir`], or /// [`OpenVmmConfig::from_repo_layout`], then adjust the public fields. Defaults match -/// `scripts/nvx.py sandbox`. Artifacts named by path, as by [`OpenVmmConfig::new`], also need +/// `nvx-rs sandbox`. Artifacts named by path, as by [`OpenVmmConfig::new`], also need /// their [`runtime_identity`](Self::runtime_identity). #[derive(Debug, Clone, PartialEq, Eq)] #[non_exhaustive] diff --git a/doc/adversarial-testing.md b/doc/adversarial-testing.md index 10b838af7..6f35169b8 100644 --- a/doc/adversarial-testing.md +++ b/doc/adversarial-testing.md @@ -155,11 +155,11 @@ metadata and summary record the absolute target artifact root. ### nvx-rs on the target Besides the guest and OpenVMM artifacts, a target needs nvx-rs, the Rust -command-line interface that `nvx.py` runs ([Usage](usage.md#nvx-rs)). Of the -catalogued scenarios, `managed-lifecycle` runs the nvx-rs test -`managed_lifecycle` with Cargo, and `l3-l4-egress-policy` compiles its egress -policy with nvx-rs and runs `nvx.py run`, which runs nvx-rs too. They find -nvx-rs where `nvx.py` does: at `NVX_RS`, then at +command-line interface that runs NVX's sandboxes and microVMs +([Usage](usage.md#nvx-rs)). Of the catalogued scenarios, `managed-lifecycle` +runs the nvx-rs test `managed_lifecycle` with Cargo, and +`l3-l4-egress-policy` compiles its egress policy with nvx-rs and runs +`nvx-rs run`. They find nvx-rs at `NVX_RS`, then at `nvx-rs/target/release/nvx-rs[.exe]`, and otherwise they build it there with Cargo. diff --git a/doc/build.md b/doc/build.md index 6501f42db..eb456218f 100644 --- a/doc/build.md +++ b/doc/build.md @@ -127,8 +127,8 @@ input configuration, generated configuration, and output hash; bind the initramfs and package manifest to the pinned Alpine inputs, Rust guest inputs, and source files; and bind OpenVMM to the exact clean gitlink revision and executable hash. Packaging rejects missing, dirty, stale, or mismatched -provenance. `build/nvx-rs.provenance.json` likewise binds the `nvx-rs` -executable that `nvx.py` runs to the digest of its sources and its build's +provenance. `build/nvx-rs.provenance.json` likewise binds the checkout's +`nvx-rs` executable to the digest of its sources and its build's Rust release, target, and features; `nvx.py` writes it when it builds `nvx-rs`, and [`record-nvx-rs-provenance`](usage.md#record-nvx-rs-provenance) writes it for a build made otherwise. diff --git a/doc/ci.md b/doc/ci.md index 25410675f..8345e12cf 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -326,7 +326,7 @@ twice the limit and the pids controller stops the workload at eight processes. The live-share runs below repeat the profile checks. Each backend also runs the public `managed-exec-config`, `sandbox-lifecycle`, and `sandbox-proxy-only` scenarios of `test-microvm`, which drive -`nvx.py sandbox provision`, `start`, +`nvx-rs sandbox provision`, `start`, `exec`, `stop`, and `deprovision` as subprocesses (see [Run](run.md#managed-lifecycle)): request-scoped working directories and environments, timeouts, nonzero exits, stdout and stderr, bounded output and @@ -439,12 +439,12 @@ The producer handoff uses one-day workflow artifacts rather than caches. Each consumer downloads both the normalized executable and its build provenance, then restores executable permissions on Linux. -`nvx.py` runs [`nvx-rs`](../nvx-rs/README.md#the-nvx-rs-cli) for its `sandbox` -command and for the egress-policy and artifact steps of its `run` command, so -the microVM test jobs take `nvx-rs` the same way, and every release package -ships it. Each NVX CLI job builds it for its operating system with +The microVM test jobs run their sandboxes and microVMs with +[`nvx-rs`](../nvx-rs/README.md#the-nvx-rs-cli), and compile their egress +policies with it, so they take `nvx-rs` the same way, and every release +package ships it. Each NVX CLI job builds it for its operating system with `--release --locked --features cli`, puts it at `nvx-rs/target/release/`, -where `nvx.py` looks for it, and runs the CLI tests with it, with `NVX_RS` +where the microVM tests look for it, and runs the CLI tests with it, with `NVX_RS` naming it for the tests that run it directly: `nvx-cli-linux` builds a static `x86_64-unknown-linux-musl` executable, which runs on every self-hosted Linux runner, and `nvx-cli-windows` an `x86_64-pc-windows-msvc` executable with a @@ -457,8 +457,8 @@ the `nvx-rs-windows-msvc-*` pair. The workflows that take them name the pair by its base name, `nvx-rs-linux-musl` or `nvx-rs-windows-msvc`, as they name the OpenVMM artifacts. Each microVM test job of that operating system downloads the executable into `nvx-rs/target/release/` and restores its executable -permission on Linux. The platform jobs run neither `nvx.py sandbox` nor -`nvx.py run`, but their +permission on Linux, where the scenarios and the sandbox steps run it. The +platform jobs run no sandbox or microVM with it, but their [`package-release`](../.github/actions/package-release/action.yml) step downloads both, the provenance into `build/`, so that each package ships the `nvx-rs` of its operating system. The `managed-lifecycle` scenario also builds @@ -542,12 +542,15 @@ hypervisor job. Each checks out OpenVMM, whose pinned CPU profiles a test reads, and uses Python 3.12, the release that the self-hosted runners provide; the Windows job also installs the pinned Rust release with the minimal profile for the curl shim test and the `nvx-rs` build. Each job first builds the -`nvx-rs` executable that `nvx.py` runs, tests with it, and hands it to the +`nvx-rs` executable that the microVM tests run, tests with it, and hands it to the microVM test and platform jobs of its operating system, as described above. Each also runs the ignored `nvx-rs` test `tests/openvmm_vectors.rs`, which compares the submodule's control-session protocol vectors with the copies that the protocol tests read in the `nvx-rs` job, which cannot check out OpenVMM; see [OpenVMM's vectors](../nvx-rs/tests/vectors/README.md#openvmms-vectors). +The ignored test `tests/openvmm_cpu_profiles.rs` likewise compares the +submodule's built-in CPU profiles with the copy of their identities from which +`nvx-rs run` guides a failed cold boot on a CPU that none serves. Fork pull requests lack the OpenVMM deploy key, so the CLI jobs run, like the hypervisor jobs, only for pushes, full runs, and same-repository pull requests. Because the microVM and platform jobs of each operating system wait for its CLI @@ -1277,7 +1280,7 @@ unless none of that suite's jobs runs it, so a new module selects both until the classifier excludes it. The always-run quality and CLI jobs catch import and static-analysis failures, so `nvx.py` importing a module at startup does not alone make it a suite input. `microvm_tests.py` and the modules that only -it, `nvx.py sandbox`, or `nvx.py test-aci-edge-sandboxes` use, including +it or `nvx.py test-aci-edge-sandboxes` use, including `microvm_durations.py`, with the `microvm_test_scripts/` that it runs in guests and the partition of its [shards](#microvm-test-shards), `microvm_shards.json`, are inputs of the microVM tests alone. @@ -1839,8 +1842,8 @@ through the `restore-only` input of [`build-guest-artifacts`](../.github/actions/build-guest-artifacts/action.yml) and the KVM OpenVMM binary that `build-openvmm` caches for the pinned revision, so agents can run microVM tests without rebuilding either. It doesn't build -`nvx-rs`: `nvx.py` builds it with Cargo, and records its provenance, the first -time that a command needs it. The guest artifact +`nvx-rs`: the microVM tests build it with Cargo, and record its provenance, +the first time that they need it. The guest artifact restore computes its keys with the same action that saves them. The binary restore repeats the key, paths, and gzip compression of the cache entry that `build-openvmm` saves, and a test in `scripts/test_nvx_tools.py` fails when diff --git a/doc/contribute.md b/doc/contribute.md index bed907b46..5d2d992af 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -78,7 +78,7 @@ the Python, shell, and link checks. | `aci_edge_sandboxes/` | rustfmt, Clippy and rustdoc with warnings denied, tests, and the minimum supported Rust version; see [Rust crates](ci.md#rust-crates) | [check-aci-edge-sandboxes](../.github/actions/check-aci-edge-sandboxes/action.yml) | | `nvx-rs/` | rustfmt, Clippy and rustdoc with warnings denied, tests including a bundled-release fixture, the minimum supported Rust version, a macOS check, and, through `nvx.py`, the musl lint, tests, build, digest manifest, and static-ELF check of the guest binaries; see [Rust crates](ci.md#rust-crates) | [check-nvx-rs](../.github/actions/check-nvx-rs/action.yml) | | `tests/runtime-adapter/` | rustfmt, then Clippy with warnings denied, the build, the tests, and the minimum supported Rust version of `nvx`, in the protocol-only, host-only, and, on Linux, static musl guest-only configurations; see [Rust crates](ci.md#rust-crates) | [check-nvx-rs](../.github/actions/check-nvx-rs/action.yml) | -| NVX CLI | Compiles `scripts/`, runs every `scripts/test_*.py` module, and checks command help, on Linux and Windows, and compares the copies of OpenVMM's protocol vectors in `nvx-rs/tests/vectors/openvmm/` with the submodule's files | [validate-nvx](../.github/actions/validate-nvx/action.yml), [ci.yml](../.github/workflows/ci.yml) | +| NVX CLI | Compiles `scripts/`, runs every `scripts/test_*.py` module, and checks command help, on Linux and Windows, and compares the copies of OpenVMM's protocol vectors in `nvx-rs/tests/vectors/openvmm/`, and of the identities of its built-in CPU profiles in `nvx-rs/src/host/cli/cpu_generations.json`, with the submodule's files | [validate-nvx](../.github/actions/validate-nvx/action.yml), [ci.yml](../.github/workflows/ci.yml) | check-quality names each shell script that it checks, and a CLI test fails unless validate-nvx runs every `scripts/test_*.py` module on both operating @@ -99,11 +99,11 @@ devices. Add those scenarios to `scripts/nvx_tools/microvm_tests.py` and run them with `scripts/nvx.py test-microvm --backend BACKEND`. Keep guest workloads in `scripts/nvx_tools/microvm_test_scripts` and retain complete failure logs. -The scenarios that drive the public `nvx.py sandbox` commands live in +The scenarios that drive the public `nvx-rs sandbox` commands live in `scripts/nvx_tools/managed_exec_tests.py`, `scripts/nvx_tools/sandbox_lifecycle_tests.py`, and `scripts/nvx_tools/sandbox_proxy_tests.py`, and `microvm_tests.py` -registers them. `nvx.py sandbox` and the control protocol belong to `nvx-rs` +registers them. `sandbox` and the control protocol belong to `nvx-rs` ([microsoft/nvx#359](https://github.com/microsoft/nvx/issues/359)), so the scenarios drive them as commands, compile egress policies with `nvx-rs egress compile`, and leave the control session to `nvx-rs`: the @@ -172,6 +172,10 @@ A pin promotion moves the `openvmm` gitlink to a reviewed OpenVMM revision: them, copy them into `nvx-rs/tests/vectors/openvmm/` unchanged and record the revision, in the same commit; see [OpenVMM's vectors](../nvx-rs/tests/vectors/README.md#openvmms-vectors). + `nvx-rs run` also reads a copy of the identities of OpenVMM's built-in CPU + profiles, `nvx-rs/src/host/cli/cpu_generations.json`, which the NVX CLI jobs + compare with the pinned profiles; a revision that adds, revises, or removes + a profile updates it in the same commit. - In the pull request, name the OpenVMM pull request and the exact pinned SHA, the scope and any prerequisite revisions, and the validation you ran. - A gitlink change is an input of every CI suite, so its pull request runs diff --git a/doc/distribution.md b/doc/distribution.md index 99307ff31..bb896770a 100644 --- a/doc/distribution.md +++ b/doc/distribution.md @@ -7,7 +7,8 @@ Every package is either binary-only or source-inclusive, and you must choose one. Every package ships the release build of `nvx-rs`, the Rust command-line -interface that `nvx.py` runs ([Usage](usage.md#nvx-rs)): `bin/nvx-rs`, a +interface that runs NVX's microVMs and sandboxes ([Usage](usage.md#nvx-rs)): +`bin/nvx-rs`, a static `x86_64-unknown-linux-musl` executable, in the Linux packages, and `bin/nvx-rs.exe`, an `x86_64-pc-windows-msvc` executable with a static C runtime, in the Windows package. `provenance/nvx-rs.provenance.json` binds it @@ -118,12 +119,12 @@ archive marks the executables in `bin/` executable. [`download`](usage.md#download) verifies the package against its `SHA256SUMS` and installs OpenVMM, the guest artifacts, their provenance, and `nvx-rs` into the checkout, `nvx-rs` at `nvx-rs/target/release/nvx-rs[.exe]` with its -provenance in `build/`, so that `nvx.py run` and `nvx.py sandbox` need no -Cargo. It installs `nvx-rs` only when its provenance describes the build of -the checkout's `nvx-rs` sources that `nvx.py` runs. Releases from before -packages shipped `nvx-rs` remain installable: `download` installs the rest and -leaves `nvx.py` to build `nvx-rs` with Cargo, as it does for a fallback -release whose `nvx-rs` sources differ from the checkout's. +provenance in `build/`, so that the checkout runs `nvx-rs` without a Cargo +build. It installs `nvx-rs` only when its provenance describes a build of +the checkout's `nvx-rs` sources. Releases from before packages shipped +`nvx-rs` remain installable: `download` installs the rest and leaves +`nvx-rs` to a Cargo build, as it does for a fallback release whose `nvx-rs` +sources differ from the checkout's. When a package includes `nvx-rs` and its provenance, `download` checks their consistency before deciding whether the build is compatible. A mismatched diff --git a/doc/guests/azurelinux-guest.md b/doc/guests/azurelinux-guest.md index baa028f63..4760b391f 100644 --- a/doc/guests/azurelinux-guest.md +++ b/doc/guests/azurelinux-guest.md @@ -1,7 +1,7 @@ # Azure Linux guest support NVX runs Azure Linux 3.0 userland on its own Linux kernel. It builds one -artifact, `initramfs-azurelinux.cpio.gz`, which `nvx.py run --guest azurelinux` +artifact, `initramfs-azurelinux.cpio.gz`, which `nvx-rs run --guest azurelinux` boots for an interactive shell or a workload. The artifact boots no Azure Linux kernel, firmware, or systemd, and it does not change the [machine and device ABI](../design/2-virtual-machine-architecture.md). diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index ace24b41d..7fb39bbe7 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -1,9 +1,9 @@ # Ubuntu guest support NVX runs Ubuntu userland on its own Linux kernel. It builds two artifacts from -one pinned Ubuntu Base root filesystem: an initramfs that `nvx.py run` boots +one pinned Ubuntu Base root filesystem: an initramfs that `nvx-rs run` boots for an interactive Ubuntu shell, and an EROFS `distro` layer that -`nvx.py sandbox` runs as a non-root workload. Neither artifact boots an Ubuntu +`nvx-rs sandbox` runs as a non-root workload. Neither artifact boots an Ubuntu kernel, firmware, or systemd, and neither changes the [machine and device ABI](../design/2-virtual-machine-architecture.md). @@ -15,8 +15,8 @@ describes how the pieces fit together and why. | Artifact | Used by | Boot supervisor | | --- | --- | --- | -| `initramfs-ubuntu.cpio.gz` | `nvx.py run --guest ubuntu` | The common NVX `/init` inside the Ubuntu initramfs | -| `ubuntu-distro.erofs` | `nvx.py sandbox --layer distro,...` | The Alpine control initramfs and its init agent | +| `initramfs-ubuntu.cpio.gz` | `nvx-rs run --guest ubuntu` | The common NVX `/init` inside the Ubuntu initramfs | +| `ubuntu-distro.erofs` | `nvx-rs sandbox --layer distro,...` | The Alpine control initramfs and its init agent | Alpine remains the default guest of `run` and the only control environment of `sandbox`. The sandbox launcher always boots the Alpine initramfs, and the diff --git a/doc/project-structure.md b/doc/project-structure.md index 7374021f9..659f7a9a8 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -145,7 +145,7 @@ nvx/ | | |-- microvm_test_scripts/ Guest workloads for the microVM tests | | |-- microvm_tests.py NVX microVM correctness tests | | |-- nightly_ci.py Nightly CI failure report -| | |-- nvx_rs.py Finds, builds, and runs nvx-rs, to which nvx.py delegates +| | |-- nvx_rs.py Finds and builds nvx-rs, and runs it for the microVM tests | | |-- openvmm_process.py Foreground OpenVMM process control for tests | | |-- performance.py CI performance collection, persistence, and gates | | |-- process_supervisor.py Linux supervisor for one owned process tree @@ -287,9 +287,9 @@ bundled-release staging, OpenVMM's command line, the lowering of network and filesystem policies, the compiler of `nvx.py`'s egress policy files, process and platform primitives for Linux and Windows hosts, the supervision of VM instances, the sandbox launch profile of -`nvx.py sandbox`, the `nvx-rs` command-line interface that ports -`nvx.py run`, `nvx.py sandbox`, egress policy compilation, and artifact -lookup, and a fake OpenVMM for tests. Its +`nvx-rs sandbox`, the `nvx-rs` command-line interface that runs NVX's +microVMs and sandboxes, compiles egress policies, and looks up artifacts, +and a fake OpenVMM for tests. Its `guest` feature builds the nine static executables that every guest image installs at `/sbin/nvx-*`. It is a standalone crate with its own `Cargo.lock`, like `aci_edge_sandboxes`, and diff --git a/doc/run.md b/doc/run.md index 6d500d808..63f982354 100644 --- a/doc/run.md +++ b/doc/run.md @@ -6,28 +6,31 @@ package, before the first run: ```bash python3 scripts/nvx.py download -python3 scripts/nvx.py run +nvx-rs/target/release/nvx-rs run ``` This installs the packaged kernel and initramfs under `build/` and OpenVMM under `openvmm/target/release/`, so neither needs a local build. It installs -the package's [`nvx-rs`](usage.md#nvx-rs), which runs `run`, only when the -release was built from the checkout's `nvx-rs` sources; otherwise, `nvx.py` -builds it with Cargo the first time that a command needs it. Set `GH_TOKEN` -to a token with contents read access when downloading from a private repository. -On Linux, pass `--hypervisor mshv` to both commands to use the MSHV package. +the package's [`nvx-rs`](usage.md#nvx-rs), which runs the microVMs and +sandboxes on this page, only when the release was built from the checkout's +`nvx-rs` sources; otherwise, build it with Cargo, as +[Usage](usage.md#nvx-rs) describes. Set `GH_TOKEN` to a token with contents +read access when downloading from a private repository. On Linux, pass +`--hypervisor mshv` to both commands to use the MSHV package. The examples +on this page use the POSIX spelling; on Windows, run +`nvx-rs\target\release\nvx-rs.exe` instead. The CLI chooses WHP on Windows and KVM on Linux: ```bash -python3 scripts/nvx.py run +nvx-rs/target/release/nvx-rs run ``` A successful Alpine boot prints both `ALPINE-MICROVM-BOOT-OK` and `NVX-GUEST-BOOT-OK: alpine`. Boot Ubuntu userland with the same NVX kernel: ```bash -python3 scripts/nvx.py run --guest ubuntu +nvx-rs/target/release/nvx-rs run --guest ubuntu ``` Ubuntu defaults to 512 MiB and prints `NVX-GUEST-BOOT-OK: ubuntu`. It is @@ -44,7 +47,7 @@ Exit cleanly from the guest with: Pass extra guest options without changing the generated device ABI: ```bash -python3 scripts/nvx.py run \ +nvx-rs/target/release/nvx-rs run \ --memory-mib 256 \ --net 10.0.0.2/24 \ --network-profile portable \ @@ -58,7 +61,7 @@ same in-process data plane on KVM, MSHV, and WHP; omitting either `--net` or Select an explicit processor count after building the matching specialized guest kernel: ```bash -python3 scripts/nvx.py run --machine microvm --processors 8 +nvx-rs/target/release/nvx-rs run --machine microvm --processors 8 ``` The microVM uses fixed device topology, reserves a shared interrupt-status page, and uses @@ -66,10 +69,11 @@ shared-status edge-triggered virtio interrupts with 1, 2, 4, or 8 vCPUs. ## Run OpenVMM directly -The platform release archives are self-contained; `scripts/nvx.py` is a -convenience wrapper and is not required at runtime. Extract the archive that -matches the host and run the following command from its -`nvx-VERSION-PLATFORM` directory. For Linux/KVM: +The platform release archives are self-contained: their `bin/nvx-rs` runs the +commands on this page on the archive's artifacts, and neither it nor +`scripts/nvx.py` is required to run OpenVMM. Extract the archive that matches +the host and run the following command from its `nvx-VERSION-PLATFORM` +directory. For Linux/KVM: ```bash ./bin/openvmm \ @@ -347,12 +351,12 @@ loopback in both directions without a port list, and preserves the existing guest-to-host mapping without publishing guest ports. Guest-originated traffic remains subject to egress policy. -Most `nvx.py run` options pass through unchanged: `--machine`, `--processors`, +Most `nvx-rs run` options pass through unchanged: `--machine`, `--processors`, `--cpu-profile`, `--mount`, `--net`, `--network-profile`, `--cmdline`, `--restore-snapshot`, `--restore-processors`, and `--restore-ready-path`. The -wrapper performs these translations and additions: +`nvx-rs run` performs these translations and additions: -| `nvx.py run` | Direct OpenVMM option | +| `nvx-rs run` | Direct OpenVMM option | | --- | --- | | `--guest alpine` | `--initrd .../initramfs.cpio.gz` on a fresh boot | | `--guest ubuntu` | `--initrd .../initramfs-ubuntu.cpio.gz` and a 512 MiB default on a fresh boot | @@ -483,7 +487,7 @@ A `/30` guest network holds only the gateway and the guest, so OpenVMM presents IPv4 host peers at the gateway instead. TCP still works, but the guest's UDP replies to host clients over `127.0.0.1` then do not return through the published port, so use a `/29` or larger network with -`--host-loopback allow`. The NVX Rust API and `nvx.py sandbox` reject +`--host-loopback allow`. The NVX Rust API and `nvx-rs sandbox` reject host-loopback access for a `/30`. The IPv6 subnet of a `/30` keeps an address for host peers. @@ -516,7 +520,7 @@ run retired ABI-1 guests or snapshots. Restore an existing microVM snapshot with an optional host-readiness endpoint: ```bash -python3 scripts/nvx.py run \ +nvx-rs/target/release/nvx-rs run \ --machine microvm \ --processors 8 \ --restore-snapshot /var/lib/nvx/snapshot \ @@ -553,13 +557,13 @@ The microVM has one virtio-fs slot, with the fixed tag `microvm`. On a cold boot with `--mount`, the initramfs mounts the mapping automatically: ```bash -python3 scripts/nvx.py run --mount "/mnt/host,/absolute/host/share,rw" +nvx-rs/target/release/nvx-rs run --mount "/mnt/host,/absolute/host/share,rw" ``` PowerShell example: ```powershell -python scripts\nvx.py run ` +nvx-rs\target\release\nvx-rs.exe run ` --mount "/mnt/host,C:\Users\me\microvm-share,rw" ``` @@ -568,7 +572,7 @@ It ends at the first comma, and the mode follows the last comma, so a host path may contain commas when the mode is given, which it then requires: ```powershell -python scripts\nvx.py run ` +nvx-rs\target\release\nvx-rs.exe run ` --mount "/mnt/alpha,C:\projects\alpha,beta,rw" ` --mount "/mnt/reports,C:\reports\2026, final,ro" ` --mount-deny "C:\projects\alpha,beta\secrets" ` @@ -580,7 +584,7 @@ access mode, for example a read-write workspace next to a read-only tool cache: ```bash -python3 scripts/nvx.py run \ +nvx-rs/target/release/nvx-rs run \ --mount "/workspace,/srv/checkout,rw" \ --mount "/opt/hostedtoolcache,/opt/hostedtoolcache,ro" ``` @@ -590,7 +594,7 @@ file at its target, but neither its directory nor its siblings, for example a read-only settings file next to a read-write results file: ```bash -python3 scripts/nvx.py run \ +nvx-rs/target/release/nvx-rs run \ --mount "/config/settings.json,/srv/app/config/settings.json,ro" \ --mount "/results/output.txt,/srv/app/results/output.txt,rw" ``` @@ -678,10 +682,9 @@ mount -t virtiofs microvm /mnt/host ## Experimental single-workload sandbox -The `sandbox` command launches the microVM with one compressed EROFS distro -layer and one preformatted ext4 scratch image. `nvx.py` runs it with -[`nvx-rs`](usage.md#nvx-rs), which it finds, or builds with Cargo, as -[Usage](usage.md#nvx-rs) describes: +The `sandbox` command of [`nvx-rs`](usage.md#nvx-rs) launches the microVM +with one compressed EROFS distro layer and one preformatted ext4 scratch +image. Build the distro layer with `nvx.py`: ```bash python3 scripts/nvx.py build-distro-layer \ @@ -695,7 +698,7 @@ Read the deterministic UUID from control initramfs: ```bash -python3 scripts/nvx.py sandbox \ +nvx-rs/target/release/nvx-rs sandbox \ --layer distro,build/ubuntu-distro.erofs,11111111-1111-1111-1111-111111111111 \ --scratch /var/lib/nvx/scratch.ext4 \ --entrypoint /bin/sh \ @@ -826,7 +829,7 @@ and policy on the host side, so edits are visible in both directions without staging or copy-back: ```bash -python3 scripts/nvx.py sandbox \ +nvx-rs/target/release/nvx-rs sandbox \ --layer distro,build/ubuntu-distro.erofs,11111111-1111-1111-1111-111111111111 \ --scratch /var/lib/nvx/scratch.ext4 \ --mount /workspace,/srv/checkout,rw \ @@ -964,7 +967,7 @@ that it spills below them, which stay readable, and lets the workload write only to `agent` and `cache`: ```bash -python3 scripts/nvx.py sandbox \ +nvx-rs/target/release/nvx-rs sandbox \ --layer distro,build/ubuntu-distro.erofs,11111111-1111-1111-1111-111111111111 \ --scratch /var/lib/nvx/scratch.ext4 \ --mount /tmp/mcp-runtime,/tmp/mcp-runtime,rw \ @@ -1006,7 +1009,7 @@ operations: permitted`) instead of running as OpenVMM. ```bash -python3 scripts/nvx.py sandbox \ +nvx-rs/target/release/nvx-rs sandbox \ --layer distro,/var/lib/nvx/distro.erofs,11111111-1111-1111-1111-111111111111 \ --scratch /var/lib/nvx/scratch.ext4 \ --mount /workspace,/srv/checkout,rw \ @@ -1030,7 +1033,7 @@ so the example passes the user's own back: ```bash sudo setpriv --reuid="$(id -u)" --regid="$(id -g)" --init-groups \ --inh-caps=+setuid,+setgid --ambient-caps=+setuid,+setgid -- \ - env HOME="$HOME" python3 scripts/nvx.py sandbox ... --mount-owner caller + env HOME="$HOME" nvx-rs/target/release/nvx-rs sandbox ... --mount-owner caller ``` A service manager can grant the same set, such as systemd's @@ -1081,7 +1084,7 @@ gateway. Every other destination is denied, the proxy's port over UDP and the host's other loopback services included: ```bash -python3 scripts/nvx.py sandbox \ +nvx-rs/target/release/nvx-rs sandbox \ --layer distro,build/ubuntu-distro.erofs,11111111-1111-1111-1111-111111111111 \ --scratch /var/lib/nvx/scratch.ext4 \ --net 10.0.0.2/24 \ @@ -1129,7 +1132,7 @@ rather than the typed policy; deprovision it and provision it again. The explicit `test-microvm --scenario sandbox-proxy-only --backend BACKEND` scenario is the acceptance for proxy-only networking. Like the -[managed lifecycle](#managed-lifecycle) scenarios, it runs `nvx.py sandbox` as +[managed lifecycle](#managed-lifecycle) scenarios, it runs `nvx-rs sandbox` as subprocesses with the guest-artifact build's Ubuntu layer and scratch template, so it also runs only when named. It checks that `provision` refuses a proxy beside `--network-egress allow`, and one at an address other than the @@ -1157,23 +1160,23 @@ start it once, run multiple workloads in the same warm guest, stop it while retaining configuration, and finally deprovision it: ```bash -python3 scripts/nvx.py sandbox provision \ +nvx-rs/target/release/nvx-rs sandbox provision \ --state-dir /run/user/1000/nvx-example \ --layer distro,/var/lib/nvx/distro.erofs,11111111-1111-1111-1111-111111111111 \ --scratch /var/lib/nvx/scratch.ext4 -python3 scripts/nvx.py sandbox start \ +nvx-rs/target/release/nvx-rs sandbox start \ --state-dir /run/user/1000/nvx-example -python3 scripts/nvx.py sandbox exec \ +nvx-rs/target/release/nvx-rs sandbox exec \ --state-dir /run/user/1000/nvx-example \ --entrypoint /usr/bin/python3 --arg=/work/agent.py --cwd /work \ --environment-file /run/user/1000/nvx-example-environment.json \ --outcome-report /run/user/1000/nvx-example-exec.json -python3 scripts/nvx.py sandbox exec \ +nvx-rs/target/release/nvx-rs sandbox exec \ --state-dir /run/user/1000/nvx-example \ --entrypoint /bin/sh --arg=-c --arg='cat /tmp/previous-result' -python3 scripts/nvx.py sandbox stop \ +nvx-rs/target/release/nvx-rs sandbox stop \ --state-dir /run/user/1000/nvx-example -python3 scripts/nvx.py sandbox deprovision \ +nvx-rs/target/release/nvx-rs sandbox deprovision \ --state-dir /run/user/1000/nvx-example ``` @@ -1185,7 +1188,7 @@ resolved path, so for a layer given through a link to another name, it also verifies the manifest beside the link's target. `start` checks the binding from metadata alone, without reading the layer again, and refuses a layer or manifest that was replaced, modified, added, or removed since `provision`, and a -configuration without a binding, as an older `nvx.py` wrote; deprovision such a +configuration without a binding, as older releases wrote; deprovision such a sandbox and provision it again. `provision` and `start` refuse a manifest path that they cannot look up, as through a link loop, rather than take it for no manifest. Keep provisioned layers and their manifests read-only, and never @@ -1314,7 +1317,7 @@ the `output-limit` category. The explicit `test-microvm --scenario managed-exec-config --backend BACKEND` scenario is the authoritative acceptance for these public options. It invokes -`scripts/nvx.py sandbox provision`, `start`, `exec`, `stop`, and `deprovision` +`nvx-rs sandbox provision`, `start`, `exec`, `stop`, and `deprovision` as subprocesses with an Alpine control guest and an Ubuntu workload layer. It checks sequential distinct CWD and exact-environment requests, an environment layered over the defaults, and then omitted defaults. It resolves the selected @@ -1406,7 +1409,7 @@ and no evidence holds the control capability. Decoder, helper, and direct control-session tests remain useful supplemental coverage for protocol boundaries and guest implementation details. They do not -replace or establish support through the public `nvx.py sandbox` interface. +replace or establish support through the public `nvx-rs sandbox` interface. `run --outcome-report PATH` and one-shot `sandbox run --outcome-report PATH` forward OpenVMM's bounded local JSON report. Managed `sandbox exec` writes only diff --git a/doc/setup.md b/doc/setup.md index 731c6a025..400120162 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -32,11 +32,10 @@ python3 scripts/nvx.py init Python 3.10 or newer and Git are required on every platform. Building OpenVMM or the Rust crates also requires [rustup](https://rustup.rs/), which installs the Rust release that [`rust-toolchain.toml`](../rust-toolchain.toml) pins on -first use. `nvx.py run` and `nvx.py sandbox` run `nvx-rs`, which +first use. NVX runs its microVMs and sandboxes with `nvx-rs`, which [`download`](usage.md#download) installs from a release built from the -checkout's `nvx-rs` sources. Otherwise, unless `NVX_RS` names an executable, -`nvx.py` builds it with that release the first time that a command needs it; -see [Usage](usage.md#nvx-rs). +checkout's `nvx-rs` sources. Otherwise, build it with Cargo and that +release; see [Usage](usage.md#nvx-rs). By default, a microVM boots on the built-in [CPU profile](usage.md#cpu-profiles) of its host's CPU: four Intel @@ -44,7 +43,7 @@ generations and AMD's EPYC Milan, Genoa, and Turin, on a hypervisor that supports that profile. On another Intel or AMD CPU, `--cpu-profile auto`, the default, falls back to a development profile derived from the host and warns at every cold boot, asking for the CPU's fingerprint so that a built-in -profile can serve it; `nvx.py run --cpu-profile host` selects that profile +profile can serve it; `nvx-rs run --cpu-profile host` selects that profile explicitly, for example where the hypervisor does not support the built-in profile (`E_PROFILE_UNSUPPORTED`, whose message names this option). OpenVMM verifies a host profile as it verifies a built-in profile: the cold boot diff --git a/doc/usage.md b/doc/usage.md index 6b9fb560f..bc9481f38 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1,61 +1,72 @@ # Command-line usage -`scripts/nvx.py` is the supported command-line entry point for building, -running, benchmarking, and packaging NVX. Run it from the repository root with -Python 3.10 or newer: +NVX has two command-line interfaces, which run from the repository root: -```text -python3 scripts/nvx.py COMMAND [OPTIONS] -``` +- [`nvx-rs`](#nvx-rs) runs NVX microVMs and sandboxes: [`run`](#run) boots or + restores a microVM on the terminal, and [`sandbox`](#sandbox) runs or + manages workloads over EROFS layers and private ext4 scratch. +- `scripts/nvx.py`, with Python 3.10 or newer, builds, tests, benchmarks, and + packages NVX, and downloads its releases: + + ```text + python3 scripts/nvx.py COMMAND [OPTIONS] + ``` The examples on this page use the POSIX spelling. On Windows, use -`python scripts\nvx.py` instead. Every command accepts `-h` or `--help`, -including nested commands: +`nvx-rs\target\release\nvx-rs.exe` and `python scripts\nvx.py` instead. +Every command accepts `-h` or `--help`, including nested commands: ```console +nvx-rs/target/release/nvx-rs run --help python3 scripts/nvx.py --help -python3 scripts/nvx.py run --help python3 scripts/nvx.py performance gate --help ``` ## nvx-rs -`nvx.py` runs [`nvx-rs`](../nvx-rs/README.md#the-nvx-rs-cli), the Rust -implementation of NVX's runtime steps, for the [`run`](#run) and -[`sandbox`](#sandbox) commands, which it runs entirely, for the artifact lookup -of the CPU-profile guidance of `run`, and for the egress policies of the -[`test-microvm`](#test-microvm) scenarios. It runs, in this order: +[`nvx-rs`](../nvx-rs/README.md#the-nvx-rs-cli) is the Rust command-line +interface that runs NVX's microVMs and sandboxes. A checkout gets it at +`nvx-rs/target/release/nvx-rs`, or `nvx-rs.exe` on Windows, in one of two +ways: + +- [`download`](#download) installs the `nvx-rs` of a release there when the + release was built from the checkout's `nvx-rs` sources. A checkout whose + `nvx-rs` sources differ from those of its nearest release, as those of the + latest `dev` often do, gets none. +- Cargo builds it there from the repository root, with the Rust release that + `rust-toolchain.toml` pins; [rustup](https://rustup.rs) installs Cargo. + `--target-dir` keeps the executable there even where `CARGO_TARGET_DIR` + names another directory. Build it again after changing `nvx-rs`: + + ```console + cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target + ``` + +Without `--root`, `nvx-rs` uses the artifacts of the checkout that holds it: +the guest artifacts in `build/` and OpenVMM in `openvmm/target/release/`. The +`nvx-rs` of an extracted release package, `bin/nvx-rs`, uses the release's +artifacts; see [Artifact root](../nvx-rs/README.md#artifact-root). + +The [`test-microvm`](#test-microvm) scenarios run `nvx-rs` too. They run, in +this order: 1. the executable that the `NVX_RS` environment variable names; -2. `nvx-rs/target/release/nvx-rs`, or `nvx-rs.exe` on Windows, in the - checkout, where `cargo build --release` puts it, and where - [`download`](#download) installs the `nvx-rs` of a release; or else -3. that executable, after it builds it from the repository root with - `cargo build --release --locked --features cli --manifest-path - nvx-rs/Cargo.toml --target-dir nvx-rs/target`, the Rust release that - `rust-toolchain.toml` pins, and Cargo's output on standard error, so that - it never mixes with the output of the command. - -Every release package ships `bin/nvx-rs`, or `bin/nvx-rs.exe`, which -`download` installs only when the release was built from the checkout's -`nvx-rs` sources, so that `nvx.py` then needs no Cargo. A checkout whose -`nvx-rs` sources differ from those of its nearest release, as those of the -latest `dev` often do, needs Cargo to build it. Without `NVX_RS`, a -built or installed executable, or Cargo, such a command fails and names -these choices; [rustup](https://rustup.rs) installs Cargo. `nvx.py` rebuilds -nothing on its own once the executable exists, so run the build command again, -or delete the executable, after changing `nvx-rs`. The `managed-lifecycle` -scenario of [`test-microvm`](#test-microvm) also needs Cargo, which runs its -`nvx-rs` test. - -The provenance of the `nvx-rs` that `nvx.py` runs, -`build/nvx-rs.provenance.json`, binds the SHA-256 of its executable to a -digest of the `nvx-rs` sources that built it, the crate's manifest, lock -file, build script, and `src/` tree and `rust-toolchain.toml`, and to the -Rust release, target, and Cargo features of the build. `nvx.py` records it -when it builds `nvx-rs`, `download` installs a release's, and -[`record-nvx-rs-provenance`](#record-nvx-rs-provenance) records it for a -build made otherwise. [`package`](#package) and the +2. `nvx-rs/target/release/nvx-rs[.exe]` in the checkout; or else +3. that executable, after they build it with the command above, with Cargo's + output on standard error, so that it never mixes with their output. + +Without `NVX_RS`, a built or installed executable, or Cargo, they fail and +name these choices. They rebuild nothing once the executable exists. The +`managed-lifecycle` scenario also needs Cargo, which runs its `nvx-rs` test. + +The provenance of a checkout's `nvx-rs`, `build/nvx-rs.provenance.json`, binds +the SHA-256 of its executable to a digest of the `nvx-rs` sources that built +it, the crate's manifest, lock file, build script, and `src/` tree and +`rust-toolchain.toml`, and to the Rust release, target, and Cargo features of +the build. `nvx.py` records it when it builds `nvx-rs` for the scenarios, +`download` installs a release's, and +[`record-nvx-rs-provenance`](#record-nvx-rs-provenance) records it for a build +made otherwise, such as with the command above. [`package`](#package) and the [adversarial executor](adversarial-testing.md#nvx-rs-on-the-target) refuse an `nvx-rs` that it does not describe as built from the checkout's sources, so a stale build never ships or runs unnoticed. @@ -64,7 +75,7 @@ stale build never ships or runs unnoticed. `nvx-rs` reports each error on standard error as `error[CODE]: MESSAGE`, and exits with status 1, and each usage error as the usage of its parser and then -`PROG: error[CODE]: MESSAGE`, such as `nvx.py run: error[argument-invalid]: +`PROG: error[CODE]: MESSAGE`, such as `nvx-rs run: error[argument-invalid]: ...`, and exits with status 2. Each error is one line: a line break in a value that its message names, such as a path, prints as `\n` or `\r`. `CODE` is a stable code, which tells errors apart where their messages may change: callers @@ -107,6 +118,201 @@ each code for callers of the `nvx` library. | `not-found` | 1 | Other state that the command names does not exist on the host. | | `other` | 1 | Any other failure. | +### `run` + +```text +nvx-rs/target/release/nvx-rs run + [--guest {alpine,ubuntu,azurelinux}] + [--hypervisor {auto,whp,kvm,mshv}] + [--machine {microvm}] + [--memory-mib MIB] + [--memory-capacity-mib MIB] + [--processors {1,2,4,8}] + [--cpu-profile ID] + [--mount GUEST_TARGET,HOST_PATH[,ro|rw]]... + [--mount-deny HOST_PATH]... + [--mount-allow HOST_PATH]... + [--mount-write HOST_PATH]... + [--mount-owner {vmm,caller}] + [--net IPV4/PREFIX] + [--network-profile {portable}] + [--network-egress {allow,deny}] + [--network-ingress {allow,deny}] + [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]... + [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]... + [--network-egress-policy-file PATH] + [--host-loopback {allow,deny}] + [--network-proxy IPV4:TCP-PORT] + [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]... + [--outcome-report PATH] + [--cmdline TEXT] + [--restore-snapshot PATH] + [--restore-processors {1,2,4,8}] + [--restore-memory-mib MIB] + [--restore-ready-path PATH] + [--dry-run] +``` + +| Option | Default | Description | +| --- | --- | --- | +| `--guest {alpine,ubuntu,azurelinux}` | `alpine` | Select Alpine, Ubuntu, or Azure Linux userland with the same NVX kernel. This option is not used for snapshot restore. | +| `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the OpenVMM hypervisor. `auto` chooses WHP on Windows and KVM elsewhere. | +| `--machine {microvm}` | `microvm` | Select the fixed-topology microVM with shared-status edge interrupts. | +| `--memory-mib MIB` | guest-specific | Set guest memory in MiB. Defaults to 128 for Alpine, 512 for Ubuntu, and 512 for Azure Linux. | +| `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot. | +| `--processors {1,2,4,8}` | `1` | Select the microVM processor count. | +| `--cpu-profile ID` | `auto` | Select the guest's [CPU profile](#cpu-profiles): `auto` for the built-in profile of the host's CPU, which falls back with a warning to a development profile derived from this host on an Intel or AMD CPU that no built-in profile serves, a built-in profile ID, or `host` for that development profile. A restore uses the snapshot's profile, which `auto` and, for a host profile, `host` also name. | +| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose a host directory, or a single regular host file without its directory, at the absolute guest target. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, as the children of an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets; a shared file always uses the aggregate. Targets and host paths must not overlap, and the shares' kernel command-line tokens must fit its 1024-byte budget. An `rw` mapping accepts guest-created symbolic links, which the host never follows. Active snapshot restore requires the same mappings in the same order, with the same canonical paths, kinds, targets, modes, and ownership mode; a dormant-slot restore may attach one new directory mapping that the resumed guest mounts explicitly. | +| `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a mounted host root; repeat to deny multiple paths. It may name the root itself when `--mount-allow` paths inside it remain. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | +| `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path again; the hidden directories on the way list only the entries that lead to it. Repeat to allow multiple paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. See [Access policy](run.md#access-policy). | +| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` mapping; the guest's writes anywhere else in it fail with `EROFS`. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. Active snapshot restore requires the same denied, allowed, and writable paths. | +| `--mount-owner {vmm,caller}` | `vmm` | Select the host identity of the guest's operations on every `--mount` directory or file. `caller` performs each one as the guest caller's UID and GID and squashes guest root to the owner of the shared directory or file; it requires a Linux host. See [Run](run.md#file-ownership). | +| `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. | +| `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net`. | +| `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | +| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture and restore reject `allow`. | +| `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | +| `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | +| `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions from JSON. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | +| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture and restore reject `allow`. | +| `--network-proxy IPV4:TCP-PORT` | none | Allow one explicit host TCP proxy endpoint. | +| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. | +| `--outcome-report PATH` | none | Write a bounded local JSON outcome report. | +| `--cmdline TEXT` | empty | Append kernel parameters; `nvx_*` and `tsc=` tokens are reserved. | +| `--restore-snapshot PATH` | none | Restore the immutable machine contract and saved state from a snapshot directory. | +| `--restore-processors {1,2,4,8}` | none | Bring this contiguous processor prefix online before restore readiness. Requires an opt-in microVM snapshot and cannot exceed `--processors` capacity. | +| `--restore-memory-mib MIB` | none | Select the 128 MiB-aligned RAM target for an expansion-capable snapshot restore. | +| `--restore-ready-path PATH` | none | Publish one restore-readiness event to an existing Unix socket or Windows named pipe. | +| `--dry-run` | off | Print the generated OpenVMM command without running it. | + +`nvx-rs run` checks its options, prints the OpenVMM command after `>> `, and +unless `--dry-run` is given, runs OpenVMM with the standard streams. It exits +with status 1 for an error, 2 for a usage error, and 130 after Ctrl-C, and +otherwise with the status that OpenVMM returns. Its errors and usage errors +carry their [error codes](#error-codes), as `error[CODE]: MESSAGE` and +`nvx-rs run: error[CODE]: MESSAGE`. A cold boot requires the OpenVMM release +binary, `build/vmlinux`, and the selected `build/initramfs*.cpio.gz`, and a +restore only OpenVMM. An option takes its value from the next argument, or +after `=` in its own, which a value that starts with `-` needs, and its name +cannot be abbreviated. Ubuntu selection never falls back to Alpine. See +[Run](run.md) for host setup, guest shutdown, networking, and virtio-fs +examples. + +If no built-in CPU profile serves an Intel or AMD host's CPU, OpenVMM falls +back to a host profile and warns with `NVX-CPU-PROFILE-FALLBACK:` at every +cold boot, as [CPU profiles](#cpu-profiles) describes. After a failed cold +boot on such a CPU, `run` names the CPU, lists the CPUs that the built-in +profiles cover, says that `auto` falls back on it, and asks for the CPU's +fingerprint through a [CPU profile request](#cpu-profiles), with the OpenVMM +command that writes the fingerprint. OpenVMM's +own error, above the guidance, names the cause of the failure, and says so +where the host profile cannot boot. On a CPU of another vendor that no +built-in profile serves, OpenVMM exits with `E_PROFILE_HOST_UNKNOWN` before it +creates the VM, and `run` explains that host profiles cannot boot there +either. A host whose hypervisor does not support its built-in profile fails +with `E_PROFILE_UNSUPPORTED` instead, and on an Intel or AMD CPU OpenVMM's +error itself names `--cpu-profile host`. `nvx-rs run` identifies the CPU with +CPUID, as OpenVMM does, derives the guidance from its copy of the CPUs that the +built-in profiles serve, and prints it only after OpenVMM fails a cold boot, +never after a dry run, a restore, an option that `nvx-rs` refuses, or Ctrl-C. + +### `sandbox` + +```text +nvx-rs/target/release/nvx-rs sandbox + [{run,provision,start,exec,stop,deprovision}] + [--layer ROLE,PATH,EROFS_UUID]... + [--scratch PATH] + [--state-dir PATH] + [--entrypoint PATH] + [--arg VALUE]... + [--hostname NAME] + [--workload-user UID:GID] + [--memory-max BYTES] + [--pids-max COUNT] + [--memory-mib MIB] + [--timeout SECONDS] + [--exec-timeout-ms MILLISECONDS] + [--cwd GUEST_PATH] + [--environment KEY=VALUE]... + [--environment-file PATH] + [--inherit-default-environment] + [--hypervisor {auto,whp,kvm,mshv}] + [--mount GUEST_TARGET,HOST_PATH[,ro|rw]]... + [--mount-deny HOST_PATH]... + [--mount-allow HOST_PATH]... + [--mount-write HOST_PATH]... + [--mount-owner {vmm,caller}] + [--net IPV4/PREFIX] + [--network-profile {portable}] + [--network-egress {allow,deny}] + [--network-ingress {allow,deny}] + [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]... + [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]... + [--network-egress-policy-file PATH] + [--host-loopback {allow,deny}] + [--network-proxy IPV4:TCP-PORT] + [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]... + [--outcome-report PATH] + [--cmdline TEXT] + [--dry-run] +``` + +Network policy and live-share options configure only the `run` and `provision` +launches, which check the network options as one typed policy before anything +launches or is recorded; see [Sandbox networking](run.md#sandbox-networking). + +`nvx-rs sandbox` runs the operation, `run` by default, with the options +given. It exits with status 1 for an error, 2 for a usage error, and 130 after +Ctrl-C, and otherwise with the status that the workload or OpenVMM returns. Its +errors and usage errors carry their [error codes](#error-codes), as +`error[CODE]: MESSAGE` and `nvx-rs sandbox: error[CODE]: MESSAGE`; an `exec` +prints its workload's standard output and error before any error of its own, +which starts a new line. Options take their values as those of [`run`](#run) +do, and their names cannot be abbreviated either. + +| Option | Default | Description | +| --- | --- | --- | +| `{run,provision,start,exec,stop,deprovision}` | `run` | Select a one-shot run or a managed lifecycle operation. | +| `--layer ROLE,PATH,EROFS_UUID` | required for `run` and `provision` | Attach the `distro` EROFS layer. Only `distro` is accepted. The UUID must be lowercase hexadecimal in canonical hyphenated form; braces, prefixes, uppercase, and unhyphenated forms are refused. | +| `--scratch PATH` | required for `run` and `provision` | Attach a preformatted ext4 scratch image as the writable overlay. | +| `--state-dir PATH` | required for managed operations | Select persistent sandbox state. One-shot `run` rejects this option. | +| `--entrypoint PATH` | `/bin/sh` | Select an absolute workload entrypoint without whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. | +| `--arg VALUE` | none | Append one nonempty entrypoint argument without whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. Repeat to pass multiple arguments. | +| `--hostname NAME` | `nvx-sandbox` | Set the workload UTS hostname. | +| `--workload-user UID:GID` | `65534:65534` | Select the fixed non-root workload identity for `run` or `provision`. Both IDs must be ASCII decimal integers from 1 to 4294967294, without signs, whitespace, or underscores. | +| `--memory-max BYTES` | none | Set the workload cgroup memory limit to an ASCII decimal integer from 1 to 18446744073709551615, without signs, whitespace, or underscores. | +| `--pids-max COUNT` | none | Set the workload cgroup process limit to an ASCII decimal integer from 1 to 4194303, without signs, whitespace, or underscores. The guest adds one process for init. | +| `--memory-mib MIB` | `256` | Set guest memory in MiB. | +| `--timeout SECONDS` | `60` | Set the timeout for each managed control request and its response in `start`, `exec`, and `stop`, and the longest time that `provision`, `start`, `stop`, and `deprovision` wait for another lifecycle transition of the sandbox. It must be finite and greater than 0. | +| `--exec-timeout-ms MILLISECONDS` | `0` | Set the managed `exec` guest workload timeout in the unsigned 32-bit range `0..4294967295`; zero disables the workload deadline. This is separate from the finite host `--timeout` control deadline. | +| `--cwd GUEST_PATH` | `/` | Set an absolute working directory inside the workload root for managed `exec`. It is resolved after the workload identity and root are applied; missing, inaccessible, or non-directory paths fail the workload launch. Default and layered environments point `PWD` at this directory; an exact replacement environment controls `PWD` itself. | +| `--environment KEY=VALUE` | omitted | Set the exact managed `exec` environment. Repeat for multiple entries. Empty values, spaces, additional equals signs, and UTF-8 are preserved. Inline values are visible in the invoking host process arguments; use `--environment-file` for sensitive values. | +| `--environment-file PATH` | omitted | Read the exact managed `exec` environment from a strict UTF-8 JSON array of `KEY=VALUE` strings, limited to 1 MiB of input. Invalid JSON, non-finite numbers, and unpaired surrogate escapes are refused. An empty array requests an empty environment. This option is mutually exclusive with `--environment`; omitting both preserves guest defaults. | +| `--inherit-default-environment` | off | Layer the managed `exec` environment from `--environment` or `--environment-file` over the guest default environment instead of replacing it; an entry replaces the default variable of the same name. Without either option the workload already receives the defaults, so the flag has no effect. | +| `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the host hypervisor. | +| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Live-share a host directory, or a single regular host file without its directory, at the absolute target inside the container rootfs for `run` or `provision`; defaults to `ro`. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to attach more shares, each with its own target and mode, for example a read-write workspace and a read-only tool cache, as the children of an aggregate that the guest binds at their targets; a shared file always uses the aggregate, and so does every share of a managed sandbox. Targets and host paths must not overlap. A one-shot `run` describes its shares in kernel command-line tokens that count toward the sandbox's 1024-byte budget; a managed sandbox announces only their number and sends their mapping table once its guest agent is ready, for at most 256 shares. An `rw` share accepts guest-created symbolic links, which the host never follows. `/`, `/etc`, `/run`, and `/run/nvx`, and the `/etc/machine-id`, `/proc`, `/sys`, `/dev`, `/.nvx-agent`, and `/run/nvx/time` trees, are reserved. | +| `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a `--mount` host directory; relative paths are resolved inside it. It may name the directory itself when `--mount-allow` paths inside it remain. With several shares, it applies to the `--mount` before it, which must share a directory. Repeat to deny multiple paths. | +| `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path of the same share again, for example a readable `mcp-payloads` directory inside hidden logs. The hidden directories on the way list only the entries that lead to it, and the workload cannot modify them. Relative paths are resolved inside the share; with several shares, it applies to the `--mount` before it. Repeat to allow multiple paths. See [Access policy](run.md#access-policy). | +| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` share; the workload's writes anywhere else in the share fail with `EROFS`. Relative paths are resolved inside the share; with several shares, it applies to the `--mount` before it, which must be `rw`. Repeat to declare multiple writable paths. `provision` persists the denied, allowed, and writable paths. | +| `--mount-owner {vmm,caller}` | `vmm` | Select the host identity of the shares' file operations for `run` or `provision`. `vmm` performs them as OpenVMM; `caller` performs them as the workload's UID and GID, squashes guest root to the owner of each shared host directory or file, and fails them with `EPERM` when OpenVMM cannot assume that identity. `caller` requires a Linux host and is persisted by `provision`. See [File ownership](run.md#file-ownership). | +| `--net IPV4/PREFIX` | none | Enable virtio-net with a static guest address. The guest also gets the IPv6 address that embeds it in `fd00::/96`. The other network options need it. A policy that lets nothing through, `--network-egress deny` without allow rules, with ingress denied and `--host-loopback deny`, attaches no network device. | +| `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net`. | +| `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy for `run` or `provision`. | +| `--network-ingress {allow,deny}` | `deny` | Set the host ingress policy for `run` or `provision`. `allow` publishes every TCP and UDP port on which the workload listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). | +| `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. Requires explicit `--network-egress`. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | +| `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Requires explicit `--network-egress`; deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | +| `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions for `run` or `provision`. Managed provision persists the compiled rules in its network policy, not this path. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | +| `--host-loopback {allow,deny}` | guest to host | Control access between the workload and host loopback services for `run` or `provision`. Without the option, the workload reaches host loopback through its gateway, subject to the egress policy, and no port is published; `deny` closes both directions. Explicit `allow` is bidirectional: the workload reaches host loopback through its gateway, and every TCP and UDP port on which the workload listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; see [Host-loopback access](run.md#host-loopback-access). | +| `--network-proxy IPV4:TCP-PORT` | none | Make the network proxy-only: the workload reaches the HTTP or HTTPS proxy that listens on host `127.0.0.1:TCP-PORT` at `IPV4:TCP-PORT`, where `IPV4` must be the guest gateway, and nothing else. `TCP-PORT` can't be 53, where OpenVMM answers DNS at the gateway. Requires `--network-egress deny` and `--host-loopback deny`, and refuses egress rules, `--network-ingress allow`, and forwards. NVX gives the workload no proxy settings; see [Proxy-only networking](run.md#proxy-only-networking). | +| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Requires `--host-loopback allow`. A UDP forward also requires egress allowed by default and no deny rule that covers the guest network, which carries its replies; OpenVMM lets the replies on a forwarded TCP connection through whatever the egress policy. | +| `--outcome-report PATH` | none | Write a bounded local JSON outcome report for one-shot `run` or managed `exec`. | +| `--cmdline TEXT` | empty | Append non-sandbox kernel parameters; `nvx_*`, `virtfs_*`, and `tsc=` tokens are reserved. | +| `--dry-run` | off | For one-shot `run` only, print the generated OpenVMM microVM command without running it. Managed operations reject this option. | + +See [Run](run.md) for artifact preparation, the security boundary, and current +snapshot/configuration limitations. + ## Commands | Command | Description | @@ -148,8 +354,6 @@ each code for callers of the `nvx` library. | `test-adversarial` | Run a brokered Copilot-driven adversarial campaign. | | `build` | Build the guest artifacts and OpenVMM. | | `download` | Install the selected platform package built from this checkout or a released first-parent ancestor. | -| `run` | Run an OpenVMM microVM. | -| `sandbox` | Run or manage workloads over EROFS layers and private ext4 scratch. | | `benchmark` | Run the OpenVMM-native benchmark coordinator. | | `performance` | Collect, gate, and persist CI performance results, and check benchmark hosts. | | `collect-sources` | Materialize verified Linux, Alpine, and Ubuntu release sources. | @@ -534,8 +738,8 @@ python3 scripts/nvx.py record-nvx-rs-provenance [--target TARGET] | --- | --- | --- | | `--target TARGET` | the host target that `rustc -vV` reports | Rust target that built `nvx-rs`. | -Records `build/nvx-rs.provenance.json` for the `nvx-rs` that `nvx.py` runs -without building it, the one that `NVX_RS` names or else +Records `build/nvx-rs.provenance.json` for the `nvx-rs` that the checkout +runs without building it, the one that `NVX_RS` names or else `nvx-rs/target/release/nvx-rs[.exe]`; see [nvx-rs](#nvx-rs). The record binds the executable's SHA-256 to the digest of this checkout's `nvx-rs` sources, the Rust release that `rust-toolchain.toml` pins, the target, and the `cli` @@ -809,9 +1013,10 @@ against the public OpenVMM CLI. Repeat `--scenario` to select a subset; without it, every scenario supported by the selected guest runs, except four that run only when named. `smp-lapic` repeats `smp` and also asserts that every CPU uses the one-shot counting LAPIC. `managed-exec-config`, `sandbox-lifecycle`, -and `sandbox-proxy-only` drive the public `sandbox` lifecycle commands (see -[Run](run.md#managed-lifecycle)) and also require the guest-artifact build's -Ubuntu layer and scratch template, which packages omit. Alpine remains the +and `sandbox-proxy-only` drive the public `nvx-rs sandbox` lifecycle +commands (see [Run](run.md#managed-lifecycle)) and also require the +guest-artifact build's Ubuntu layer and scratch template, which packages +omit. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`, @@ -839,8 +1044,8 @@ the boot to the managed agent, and runs the ignored `nvx-rs` test `tests/managed_lifecycle.rs` with Cargo, which starts OpenVMM with the scenario's command line and drives the guest's control session; the scenario then checks OpenVMM's outcome report and the guest console. The scenarios that -drive `run` or `sandbox`, or that compile egress policies, run -[`nvx-rs`](#nvx-rs). The +run microVMs or sandboxes as the public commands do, or that compile egress +policies, run [`nvx-rs`](#nvx-rs). The command requires `build/vmlinux` (with `--debug-kernel`, `build/vmlinux-debug` and its `build/vmlinux-debug.config`), the selected initramfs, and `openvmm/target/release/openvmm[.exe]`. @@ -1076,7 +1281,7 @@ the run summary records their absolute location. See [Copilot-driven adversarial testing](adversarial-testing.md) for the trust boundary, wrapper contract, artifacts, and CI policy. -## Download and run +## Releases ### `download` @@ -1107,19 +1312,18 @@ removes any known guest artifact that the release does not declare, such as the Azure Linux guest that source-inclusive packages omit. It installs the packaged OpenVMM at `openvmm/target/release/openvmm[.exe]` and the packaged `nvx-rs` at `nvx-rs/target/release/nvx-rs[.exe]`, both executable, with their -provenance in `build/`, so that [`run`](#run) and [`sandbox`](#sandbox) need -no Cargo. +provenance in `build/`, so that [`nvx-rs`](#nvx-rs) runs without a Cargo +build. `download` installs the packaged `nvx-rs` only when its provenance describes -the build of the checkout's `nvx-rs` sources that `nvx.py` runs, with the Rust -release that the checkout pins, the release target, and the `cli` feature, -because `nvx.py` runs that executable without rebuilding it. A fallback -release built from other `nvx-rs` sources, and a release from before packages -shipped `nvx-rs`, still install everything else: `download` then removes any -earlier `nvx-rs` and its provenance and reports why, and `nvx.py` builds -`nvx-rs` from the checkout with Cargo when a command needs it; see -[nvx-rs](#nvx-rs). A package whose `nvx-rs` does not match its own provenance -fails the download before it replaces anything. +a build of the checkout's `nvx-rs` sources, with the Rust release that the +checkout pins, the release target, and the `cli` feature, because the +checkout then runs that executable as its own. A fallback release built from +other `nvx-rs` sources, and a release from before packages shipped `nvx-rs`, +still install everything else: `download` then removes any earlier `nvx-rs` +and its provenance and reports why, and the checkout's `nvx-rs` needs a Cargo +build; see [nvx-rs](#nvx-rs). A package whose `nvx-rs` does not match its own +provenance fails the download before it replaces anything. Windows release downloads support WHP. Linux release downloads support KVM and MSHV. `download` first uses `GH_TOKEN` or `GITHUB_TOKEN` when configured. @@ -1128,214 +1332,6 @@ retries without credentials so public releases remain downloadable. Private repositories require a token with read access to the repository contents that is authorized for the organization when it enforces single sign-on. -### `run` - -```text -python3 scripts/nvx.py run - [--guest {alpine,ubuntu,azurelinux}] - [--hypervisor {auto,whp,kvm,mshv}] - [--machine {microvm}] - [--memory-mib MIB] - [--memory-capacity-mib MIB] - [--processors {1,2,4,8}] - [--cpu-profile ID] - [--mount GUEST_TARGET,HOST_PATH[,ro|rw]]... - [--mount-deny HOST_PATH]... - [--mount-allow HOST_PATH]... - [--mount-write HOST_PATH]... - [--mount-owner {vmm,caller}] - [--net IPV4/PREFIX] - [--network-profile {portable}] - [--network-egress {allow,deny}] - [--network-ingress {allow,deny}] - [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]... - [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]... - [--network-egress-policy-file PATH] - [--host-loopback {allow,deny}] - [--network-proxy IPV4:TCP-PORT] - [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]... - [--outcome-report PATH] - [--cmdline TEXT] - [--restore-snapshot PATH] - [--restore-processors {1,2,4,8}] - [--restore-memory-mib MIB] - [--restore-ready-path PATH] - [--dry-run] -``` - -| Option | Default | Description | -| --- | --- | --- | -| `--guest {alpine,ubuntu,azurelinux}` | `alpine` | Select Alpine, Ubuntu, or Azure Linux userland with the same NVX kernel. This option is not used for snapshot restore. | -| `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the OpenVMM hypervisor. `auto` chooses WHP on Windows and KVM elsewhere. | -| `--machine {microvm}` | `microvm` | Select the fixed-topology microVM with shared-status edge interrupts. | -| `--memory-mib MIB` | guest-specific | Set guest memory in MiB. Defaults to 128 for Alpine, 512 for Ubuntu, and 512 for Azure Linux. | -| `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot. | -| `--processors {1,2,4,8}` | `1` | Select the microVM processor count. | -| `--cpu-profile ID` | `auto` | Select the guest's [CPU profile](#cpu-profiles): `auto` for the built-in profile of the host's CPU, which falls back with a warning to a development profile derived from this host on an Intel or AMD CPU that no built-in profile serves, a built-in profile ID, or `host` for that development profile. A restore uses the snapshot's profile, which `auto` and, for a host profile, `host` also name. | -| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose a host directory, or a single regular host file without its directory, at the absolute guest target. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, as the children of an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets; a shared file always uses the aggregate. Targets and host paths must not overlap, and the shares' kernel command-line tokens must fit its 1024-byte budget. An `rw` mapping accepts guest-created symbolic links, which the host never follows. Active snapshot restore requires the same mappings in the same order, with the same canonical paths, kinds, targets, modes, and ownership mode; a dormant-slot restore may attach one new directory mapping that the resumed guest mounts explicitly. | -| `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a mounted host root; repeat to deny multiple paths. It may name the root itself when `--mount-allow` paths inside it remain. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | -| `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path again; the hidden directories on the way list only the entries that lead to it. Repeat to allow multiple paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. See [Access policy](run.md#access-policy). | -| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` mapping; the guest's writes anywhere else in it fail with `EROFS`. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. Active snapshot restore requires the same denied, allowed, and writable paths. | -| `--mount-owner {vmm,caller}` | `vmm` | Select the host identity of the guest's operations on every `--mount` directory or file. `caller` performs each one as the guest caller's UID and GID and squashes guest root to the owner of the shared directory or file; it requires a Linux host. See [Run](run.md#file-ownership). | -| `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. | -| `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net`. | -| `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | -| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture and restore reject `allow`. | -| `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | -| `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | -| `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions from JSON. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | -| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture and restore reject `allow`. | -| `--network-proxy IPV4:TCP-PORT` | none | Allow one explicit host TCP proxy endpoint. | -| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. | -| `--outcome-report PATH` | none | Write a bounded local JSON outcome report. | -| `--cmdline TEXT` | empty | Append kernel parameters; `nvx_*` and `tsc=` tokens are reserved. | -| `--restore-snapshot PATH` | none | Restore the immutable machine contract and saved state from a snapshot directory. | -| `--restore-processors {1,2,4,8}` | none | Bring this contiguous processor prefix online before restore readiness. Requires an opt-in microVM snapshot and cannot exceed `--processors` capacity. | -| `--restore-memory-mib MIB` | none | Select the 128 MiB-aligned RAM target for an expansion-capable snapshot restore. | -| `--restore-ready-path PATH` | none | Publish one restore-readiness event to an existing Unix socket or Windows named pipe. | -| `--dry-run` | off | Print the generated OpenVMM command without running it. | - -`run` runs `nvx-rs run` (see [nvx-rs](#nvx-rs)) with the options given, and -its exit status is the command's: 1 for an error, 2 for a usage error, 130 -after Ctrl-C, and otherwise the status that OpenVMM returns. The errors and -usage errors that `nvx-rs` reports carry their [error codes](#error-codes), as -`error[CODE]: MESSAGE` and `nvx.py run: error[CODE]: MESSAGE`. `nvx-rs` checks -the options, prints the OpenVMM command after `>> `, and unless `--dry-run` is -given, runs OpenVMM with the standard streams. A cold boot requires the OpenVMM -release binary, `build/vmlinux`, and the selected `build/initramfs*.cpio.gz`, -and a restore only OpenVMM. `nvx.py` parses the options only for its help and -for the usage errors of unknown options, invalid choices of `--guest`, -`--hypervisor`, `--machine`, `--mount-owner`, and the network options, and -missing values, so it still accepts abbreviated options, which it passes to -`nvx-rs` in full. `nvx-rs` checks their values, the integers and processor -counts included, and its usage and messages name `nvx.py run`; its usage text -wraps at 80 columns whatever the terminal's width. On a command line with two -errors, the one that `nvx.py` reports wins over a value that `nvx-rs` would -refuse, even when that value comes first, and so does a request for help. -Ubuntu selection never falls back to Alpine. See [Run](run.md) for host setup, -guest shutdown, networking, and virtio-fs examples. - -If no built-in CPU profile serves an Intel or AMD host's CPU, OpenVMM falls -back to a host profile and warns with `NVX-CPU-PROFILE-FALLBACK:` at every -cold boot, as [CPU profiles](#cpu-profiles) describes. After a failed cold -boot on such a CPU, `run` names the CPU, lists the CPUs that the built-in -profiles cover, says that `auto` falls back on it, and asks for the CPU's -fingerprint through a [CPU profile request](#cpu-profiles), with the OpenVMM -command that writes the fingerprint. OpenVMM's -own error, above the guidance, names the cause of the failure, and says so -where the host profile cannot boot. On a CPU of another vendor that no -built-in profile serves, OpenVMM exits with `E_PROFILE_HOST_UNKNOWN` before it -creates the VM, and `run` explains that host profiles cannot boot there -either. A host whose hypervisor does not support its built-in profile fails -with `E_PROFILE_UNSUPPORTED` instead, and on an Intel or AMD CPU OpenVMM's -error itself names `--cpu-profile host`. `nvx.py` derives the guidance from its -CPU catalog, and `nvx-rs run` prints it only after OpenVMM fails a cold boot, -never after a dry run, a restore, an option that `nvx-rs` refuses, or Ctrl-C. - -### `sandbox` - -```text -python3 scripts/nvx.py sandbox - [{run,provision,start,exec,stop,deprovision}] - [--layer ROLE,PATH,EROFS_UUID]... - [--scratch PATH] - [--state-dir PATH] - [--entrypoint PATH] - [--arg VALUE]... - [--hostname NAME] - [--workload-user UID:GID] - [--memory-max BYTES] - [--pids-max COUNT] - [--memory-mib MIB] - [--timeout SECONDS] - [--exec-timeout-ms MILLISECONDS] - [--cwd GUEST_PATH] - [--environment KEY=VALUE]... - [--environment-file PATH] - [--inherit-default-environment] - [--hypervisor {auto,whp,kvm,mshv}] - [--mount GUEST_TARGET,HOST_PATH[,ro|rw]]... - [--mount-deny HOST_PATH]... - [--mount-allow HOST_PATH]... - [--mount-write HOST_PATH]... - [--mount-owner {vmm,caller}] - [--net IPV4/PREFIX] - [--network-profile {portable}] - [--network-egress {allow,deny}] - [--network-ingress {allow,deny}] - [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]... - [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]... - [--network-egress-policy-file PATH] - [--host-loopback {allow,deny}] - [--network-proxy IPV4:TCP-PORT] - [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]... - [--outcome-report PATH] - [--cmdline TEXT] - [--dry-run] -``` - -Network policy and live-share options configure only the `run` and `provision` -launches, which check the network options as one typed policy before anything -launches or is recorded; see [Sandbox networking](run.md#sandbox-networking). - -`sandbox` runs `nvx-rs sandbox` (see [nvx-rs](#nvx-rs)) with the operation and -the options given, and its exit status is the command's: 1 for an error, 2 for -a usage error, 130 after Ctrl-C, and otherwise the status that the workload or -OpenVMM returns. The errors and usage errors that `nvx-rs` reports carry their -[error codes](#error-codes), as `error[CODE]: MESSAGE` and -`nvx.py sandbox: error[CODE]: MESSAGE`; an `exec` prints its workload's -standard output and error before any error of its own, which starts a new -line. `nvx.py` parses the -options only for its help and for the -usage errors of unknown options, invalid choices, and missing values, so it -still accepts abbreviated options, which it passes to `nvx-rs` in full. -`nvx-rs` checks their values, and its usage and messages name -`nvx.py sandbox`; its usage text wraps at 80 columns whatever the terminal's -width. As for `run`, an error that `nvx.py` reports, or a request for help, -wins over a value that `nvx-rs` would refuse, even when that value comes first. - -| Option | Default | Description | -| --- | --- | --- | -| `{run,provision,start,exec,stop,deprovision}` | `run` | Select a one-shot run or a managed lifecycle operation. | -| `--layer ROLE,PATH,EROFS_UUID` | required for `run` and `provision` | Attach the `distro` EROFS layer. Only `distro` is accepted. The UUID must be lowercase hexadecimal in canonical hyphenated form; braces, prefixes, uppercase, and unhyphenated forms are refused. | -| `--scratch PATH` | required for `run` and `provision` | Attach a preformatted ext4 scratch image as the writable overlay. | -| `--state-dir PATH` | required for managed operations | Select persistent sandbox state. One-shot `run` rejects this option. | -| `--entrypoint PATH` | `/bin/sh` | Select an absolute workload entrypoint without whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. | -| `--arg VALUE` | none | Append one nonempty entrypoint argument without whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. Repeat to pass multiple arguments. | -| `--hostname NAME` | `nvx-sandbox` | Set the workload UTS hostname. | -| `--workload-user UID:GID` | `65534:65534` | Select the fixed non-root workload identity for `run` or `provision`. Both IDs must be ASCII decimal integers from 1 to 4294967294, without signs, whitespace, or underscores. | -| `--memory-max BYTES` | none | Set the workload cgroup memory limit to an ASCII decimal integer from 1 to 18446744073709551615, without signs, whitespace, or underscores. | -| `--pids-max COUNT` | none | Set the workload cgroup process limit to an ASCII decimal integer from 1 to 4194303, without signs, whitespace, or underscores. The guest adds one process for init. | -| `--memory-mib MIB` | `256` | Set guest memory in MiB. | -| `--timeout SECONDS` | `60` | Set the timeout for each managed control request and its response in `start`, `exec`, and `stop`, and the longest time that `provision`, `start`, `stop`, and `deprovision` wait for another lifecycle transition of the sandbox. It must be finite and greater than 0. | -| `--exec-timeout-ms MILLISECONDS` | `0` | Set the managed `exec` guest workload timeout in the unsigned 32-bit range `0..4294967295`; zero disables the workload deadline. This is separate from the finite host `--timeout` control deadline. | -| `--cwd GUEST_PATH` | `/` | Set an absolute working directory inside the workload root for managed `exec`. It is resolved after the workload identity and root are applied; missing, inaccessible, or non-directory paths fail the workload launch. Default and layered environments point `PWD` at this directory; an exact replacement environment controls `PWD` itself. | -| `--environment KEY=VALUE` | omitted | Set the exact managed `exec` environment. Repeat for multiple entries. Empty values, spaces, additional equals signs, and UTF-8 are preserved. Inline values are visible in the invoking host process arguments; use `--environment-file` for sensitive values. | -| `--environment-file PATH` | omitted | Read the exact managed `exec` environment from a strict UTF-8 JSON array of `KEY=VALUE` strings, limited to 1 MiB of input. Invalid JSON, non-finite numbers, and unpaired surrogate escapes are refused. An empty array requests an empty environment. This option is mutually exclusive with `--environment`; omitting both preserves guest defaults. | -| `--inherit-default-environment` | off | Layer the managed `exec` environment from `--environment` or `--environment-file` over the guest default environment instead of replacing it; an entry replaces the default variable of the same name. Without either option the workload already receives the defaults, so the flag has no effect. | -| `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the host hypervisor. | -| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Live-share a host directory, or a single regular host file without its directory, at the absolute target inside the container rootfs for `run` or `provision`; defaults to `ro`. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to attach more shares, each with its own target and mode, for example a read-write workspace and a read-only tool cache, as the children of an aggregate that the guest binds at their targets; a shared file always uses the aggregate, and so does every share of a managed sandbox. Targets and host paths must not overlap. A one-shot `run` describes its shares in kernel command-line tokens that count toward the sandbox's 1024-byte budget; a managed sandbox announces only their number and sends their mapping table once its guest agent is ready, for at most 256 shares. An `rw` share accepts guest-created symbolic links, which the host never follows. `/`, `/etc`, `/run`, and `/run/nvx`, and the `/etc/machine-id`, `/proc`, `/sys`, `/dev`, `/.nvx-agent`, and `/run/nvx/time` trees, are reserved. | -| `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a `--mount` host directory; relative paths are resolved inside it. It may name the directory itself when `--mount-allow` paths inside it remain. With several shares, it applies to the `--mount` before it, which must share a directory. Repeat to deny multiple paths. | -| `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path of the same share again, for example a readable `mcp-payloads` directory inside hidden logs. The hidden directories on the way list only the entries that lead to it, and the workload cannot modify them. Relative paths are resolved inside the share; with several shares, it applies to the `--mount` before it. Repeat to allow multiple paths. See [Access policy](run.md#access-policy). | -| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` share; the workload's writes anywhere else in the share fail with `EROFS`. Relative paths are resolved inside the share; with several shares, it applies to the `--mount` before it, which must be `rw`. Repeat to declare multiple writable paths. `provision` persists the denied, allowed, and writable paths. | -| `--mount-owner {vmm,caller}` | `vmm` | Select the host identity of the shares' file operations for `run` or `provision`. `vmm` performs them as OpenVMM; `caller` performs them as the workload's UID and GID, squashes guest root to the owner of each shared host directory or file, and fails them with `EPERM` when OpenVMM cannot assume that identity. `caller` requires a Linux host and is persisted by `provision`. See [File ownership](run.md#file-ownership). | -| `--net IPV4/PREFIX` | none | Enable virtio-net with a static guest address. The guest also gets the IPv6 address that embeds it in `fd00::/96`. The other network options need it. A policy that lets nothing through, `--network-egress deny` without allow rules, with ingress denied and `--host-loopback deny`, attaches no network device. | -| `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net`. | -| `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy for `run` or `provision`. | -| `--network-ingress {allow,deny}` | `deny` | Set the host ingress policy for `run` or `provision`. `allow` publishes every TCP and UDP port on which the workload listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). | -| `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. Requires explicit `--network-egress`. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | -| `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Requires explicit `--network-egress`; deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | -| `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions for `run` or `provision`. Managed provision persists the compiled rules in its network policy, not this path. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | -| `--host-loopback {allow,deny}` | guest to host | Control access between the workload and host loopback services for `run` or `provision`. Without the option, the workload reaches host loopback through its gateway, subject to the egress policy, and no port is published; `deny` closes both directions. Explicit `allow` is bidirectional: the workload reaches host loopback through its gateway, and every TCP and UDP port on which the workload listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; see [Host-loopback access](run.md#host-loopback-access). | -| `--network-proxy IPV4:TCP-PORT` | none | Make the network proxy-only: the workload reaches the HTTP or HTTPS proxy that listens on host `127.0.0.1:TCP-PORT` at `IPV4:TCP-PORT`, where `IPV4` must be the guest gateway, and nothing else. `TCP-PORT` can't be 53, where OpenVMM answers DNS at the gateway. Requires `--network-egress deny` and `--host-loopback deny`, and refuses egress rules, `--network-ingress allow`, and forwards. NVX gives the workload no proxy settings; see [Proxy-only networking](run.md#proxy-only-networking). | -| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Requires `--host-loopback allow`. A UDP forward also requires egress allowed by default and no deny rule that covers the guest network, which carries its replies; OpenVMM lets the replies on a forwarded TCP connection through whatever the egress policy. | -| `--outcome-report PATH` | none | Write a bounded local JSON outcome report for one-shot `run` or managed `exec`. | -| `--cmdline TEXT` | empty | Append non-sandbox kernel parameters; `nvx_*`, `virtfs_*`, and `tsc=` tokens are reserved. | -| `--dry-run` | off | For one-shot `run` only, print the generated OpenVMM microVM command without running it. Managed operations reject this option. | - -See [Run](run.md) for artifact preparation, the security boundary, and current -snapshot/configuration limitations. - ## CPU profiles Every microVM presents a CPU profile to its guest: the complete CPUID surface @@ -1377,8 +1373,9 @@ presents on Genoa, which a bare-metal Genoa host's KVM does not report. On an Intel or AMD development host that no built-in profile serves, `--cpu-profile auto` falls back to a host profile, `intel.host.v1` or -`amd.host.v1`, which `run --cpu-profile host` also selects, for example where -the hypervisor does not support the built-in profile. OpenVMM fingerprints the +`amd.host.v1`, which `nvx-rs run --cpu-profile host` also selects, for +example where the hypervisor does not support the built-in profile. OpenVMM +fingerprints the hypervisor on this host and applies the built-in profiles' derivation policy to it, so the guest sees the same kind of filtered CPU surface, and verifies it as it verifies a built-in profile: a cold boot still fails with diff --git a/nvx-rs/README.md b/nvx-rs/README.md index c3675ddbf..df991cbe6 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -6,11 +6,11 @@ microVMs. Its package and library are both called `nvx`; the package was called `nvx-rs`, like the directory that holds it. The [`aci_edge_sandboxes` crate](../aci_edge_sandboxes/README.md) builds on it and re-exports it as `aci_edge_sandboxes::nvx`. The crate also holds the programs -that run in NVX's Linux guests, and the `nvx-rs` command-line interface, the -backend to which `nvx.py` delegates its `run` and `sandbox` commands, -egress-policy compilation, and artifact lookup, as the phases of -[microsoft/nvx#356](https://github.com/microsoft/nvx/issues/356) moved them to -Rust. +that run in NVX's Linux guests, and the `nvx-rs` command-line interface, which +runs NVX's microVMs and sandboxes, compiles egress policies, and looks up +artifacts, the runtime steps that the phases of +[microsoft/nvx#356](https://github.com/microsoft/nvx/issues/356) moved from +`nvx.py` to Rust. An NVX sandbox is a microVM that OpenVMM runs with the managed lifecycle: the guest's init starts a managed agent, which runs the workloads that the host @@ -89,8 +89,8 @@ nvx-rs/ | `nvx::host::supervisor` | The supervisor of a sandbox's VMM, beneath the CLI, `Instance`, and external orchestrators, which keep the configuration, the authorization, and the application protocol. `launch` runs the store's launch steps under the sandbox's lock: it writes the journal's marker, spawns OpenVMM with the claimed log, records its `ProcessRef` right after, and returns a `Starting` launch that the caller commits once the VMM serves its control endpoint, or abandons. A launch that fails never leaves an unrecorded VMM running. `VmmCommand::inherit` lists the descriptors (Linux) or handles (Windows) that the VMM inherits under their numbers, besides its standard streams, and it inherits nothing else of the caller's: on Linux, they must be close-on-exec, a flag that only the VMM's own process clears; on Windows, they join the VMM's explicit handle list. `Vmm::stop` runs one graceful step of the caller's and waits for the VMM's exit event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. `Vmm::terminate` takes that forced step alone, within the caller's bound, for a VM whose guest the caller must not ask anything, such as one of another runtime identity. `reattach` recovers an interrupted launch, and holds the recorded VMM again after the orchestrator restarts within the same host boot: `Live`, `Mismatched` with another runtime identity, `Exited`, or `Indeterminate` | | `nvx::host::seal` | `FileSeal`: what the host reports about an open file from its metadata alone, without reading it: the platform scheme, the device or volume serial number, the file ID, the kind, the length, the modification time, and on Linux the change time, with a versioned serde form. A seal detects accidental change, not tampering; see [Distro bindings](#distro-bindings) | | `nvx::host::session`, `platform`, `report` | The authenticated control session, which carries NVX's control protocol, `NVXC`, one frame per DATA record of a `HostConnection`; Linux and Windows primitives; and OpenVMM's outcome report | -| `nvx::host::sandbox` | The launch profile of `nvx.py sandbox`, with native input validation: `SandboxLaunch`, with the distro `SandboxLayer`, an EROFS layer or a GPT disk, the one layer that a sandbox takes, the ext4 scratch disk, the entrypoint, arguments, hostname, workload identity, and `SandboxSettings` resource limits (`Option` for memory and `Option` for processes), and the `SandboxPlan` that it makes of them. `plan`: `SandboxPlan`, the typed plan of a cold layered launch, which is persistable in a versioned serde form: one read-only `distro` block device, an EROFS filesystem or a GPT disk with the identity that the guest expects of it, if any, an upper, the ext4 scratch disk or RAM, the launch settings, the `ExportPlan` of its live shares, and optionally a `SandboxNetwork`, the guest network and its typed `NetworkPolicy`, whose proxy's URL as the guest sees it the plan reports. The plan of an adapter that boots its own guest, `SandboxPlan::adapter`, holds no launch settings and no export plan. A plan knows only the `distro` and `scratch` `BlockRole`s, and gives, for every combination, its `--microvm-sandbox-block` arguments, with `--microvm-workload-identity` when it has launch settings, with no scratch block device for a RAM upper, and those of its network, which `nvx::host::network` lowers, and its kernel command line, with the boot tokens of `nvx::protocol::root`, within the 1024-byte budget that OpenVMM leaves it, refusing one whose root tokens the guest's parser refuses. Both deliver the live shares as the managed lifecycle does: the arguments export them at `nvx::host::filesystem::GUEST_EXPORT`, the command line announces the number of their mappings with `nvx_maps=`, and `mapping_table` gives the ordered table that `start` sends once the guest agent is ready; the CLI's one-shot `run` still renders them as boot tokens. An adapter's plan renders the caller's tokens before those of its root, and refuses tokens that its root or OpenVMM's shares own. `mount`: `SandboxMount` requests of live shares of host directories and files, their guest targets and the reserved ones, their access policies and owner, and the checks of their host paths; `plan_mounts`, which plans them through `nvx::host::filesystem` for the container root, and `boot_shares`, which renders a plan as the OpenVMM arguments and boot tokens of one share or of the aggregate of several, as the one-shot lifecycle delivers them. `options`: the checks that `nvx.py sandbox` and `nvx.py run` apply to their options: the attribution of `--mount-deny`, `--mount-allow`, and `--mount-write` to shares, the systemd entrypoint check and the explicit verification of the distro layer, the targets of `nvx.py run` shares, and the managed exec environment; see [Sandbox input contract](#sandbox-input-contract). `distro`: the `DistroBinding` of a distro layer, an EROFS layer or a GPT disk, which explicit verification makes or a staging caller imports, and its check from metadata alone, which returns the layer open; see [Distro bindings](#distro-bindings). `scratch`: the `ScratchBinding` of a scratch image, its identity, kind, and length, and its check from metadata alone, which claims the image for one sandbox at a time and returns it open. `prepared`: `PreparedLaunch`, the launch of a plan with the checked files of its block devices, which renders the plan's OpenVMM arguments and kernel command line, naming the checked files as `/proc/self/fd/N` on Linux and by their final paths on Windows, and lists the descriptors that the VMM inherits through `VmmCommand::inherit`; see [Checked handoff](#checked-handoff) | -| `nvx::testing::fake_openvmm` | A fake OpenVMM that serves the control console and runs scripted workloads without a VM, for direct launches and for the managed and one-shot sandboxes of `nvx-rs sandbox` and `nvx.py sandbox` | +| `nvx::host::sandbox` | The launch profile of `nvx-rs sandbox`, with native input validation: `SandboxLaunch`, with the distro `SandboxLayer`, an EROFS layer or a GPT disk, the one layer that a sandbox takes, the ext4 scratch disk, the entrypoint, arguments, hostname, workload identity, and `SandboxSettings` resource limits (`Option` for memory and `Option` for processes), and the `SandboxPlan` that it makes of them. `plan`: `SandboxPlan`, the typed plan of a cold layered launch, which is persistable in a versioned serde form: one read-only `distro` block device, an EROFS filesystem or a GPT disk with the identity that the guest expects of it, if any, an upper, the ext4 scratch disk or RAM, the launch settings, the `ExportPlan` of its live shares, and optionally a `SandboxNetwork`, the guest network and its typed `NetworkPolicy`, whose proxy's URL as the guest sees it the plan reports. The plan of an adapter that boots its own guest, `SandboxPlan::adapter`, holds no launch settings and no export plan. A plan knows only the `distro` and `scratch` `BlockRole`s, and gives, for every combination, its `--microvm-sandbox-block` arguments, with `--microvm-workload-identity` when it has launch settings, with no scratch block device for a RAM upper, and those of its network, which `nvx::host::network` lowers, and its kernel command line, with the boot tokens of `nvx::protocol::root`, within the 1024-byte budget that OpenVMM leaves it, refusing one whose root tokens the guest's parser refuses. Both deliver the live shares as the managed lifecycle does: the arguments export them at `nvx::host::filesystem::GUEST_EXPORT`, the command line announces the number of their mappings with `nvx_maps=`, and `mapping_table` gives the ordered table that `start` sends once the guest agent is ready; the CLI's one-shot `run` still renders them as boot tokens. An adapter's plan renders the caller's tokens before those of its root, and refuses tokens that its root or OpenVMM's shares own. `mount`: `SandboxMount` requests of live shares of host directories and files, their guest targets and the reserved ones, their access policies and owner, and the checks of their host paths; `plan_mounts`, which plans them through `nvx::host::filesystem` for the container root, and `boot_shares`, which renders a plan as the OpenVMM arguments and boot tokens of one share or of the aggregate of several, as the one-shot lifecycle delivers them. `options`: the checks that `nvx-rs sandbox` and `nvx-rs run` apply to their options: the attribution of `--mount-deny`, `--mount-allow`, and `--mount-write` to shares, the systemd entrypoint check and the explicit verification of the distro layer, the targets of `nvx-rs run` shares, and the managed exec environment; see [Sandbox input contract](#sandbox-input-contract). `distro`: the `DistroBinding` of a distro layer, an EROFS layer or a GPT disk, which explicit verification makes or a staging caller imports, and its check from metadata alone, which returns the layer open; see [Distro bindings](#distro-bindings). `scratch`: the `ScratchBinding` of a scratch image, its identity, kind, and length, and its check from metadata alone, which claims the image for one sandbox at a time and returns it open. `prepared`: `PreparedLaunch`, the launch of a plan with the checked files of its block devices, which renders the plan's OpenVMM arguments and kernel command line, naming the checked files as `/proc/self/fd/N` on Linux and by their final paths on Windows, and lists the descriptors that the VMM inherits through `VmmCommand::inherit`; see [Checked handoff](#checked-handoff) | +| `nvx::testing::fake_openvmm` | A fake OpenVMM that serves the control console and runs scripted workloads without a VM, for direct launches and for the managed and one-shot sandboxes of `nvx-rs sandbox` | | `nvx::host::cli` | Feature `cli`: the `nvx-rs` executable; see [The nvx-rs CLI](#the-nvx-rs-cli) | | `nvx::guest` | The programs that run in Linux guests; see [Guest binaries](#guest-binaries) | | `nvx::guest::mappings` | `Mapper`: applies host mappings for any guest init or agent beneath the descriptor of a `TargetRoot`, the root of a mount of a whole file system, which `statmount` tells from a bind of a directory below it, such as a layered root's overlay or a guest's `/`, from a `SourceRoot`, such as the guest's mount of the host's export. A `Mapping` names a relative source, an absolute target, its `Access`, and optionally the `Kind` that its source must have. The mapper opens each component with `openat2`, through no symbolic link or magic link, creates what a target lacks only on the root's own mounts, which `statx` tells apart by their unique mount IDs, never inside another mapping, and checks kinds, overlaps, and the targets that the root profile's `Reserved` set holds, with the `/run/nvx/time` tree and its ancestors always. It attaches each mapping as a detached copy of its source's mount, `open_tree`, restricted `nosuid`, `nodev`, and read-only when asked, `mount_setattr`, before `move_mount` puts it at its target, then checks that it is there. Zero mappings do no work; the first that fails is reported with its index as an `ApplyError`, and latches the mapper. `Mapper::teardown` detaches a mapping that a rename moved and that the attach failed to detach, then unmounts the mappings in reverse order, and reports each failure. `bind` applies one runtime bind through the same walk and attach. Linux 6.8 or later. NVX's managed agent applies its host mapping table through it, beneath `/` in direct mode and beneath the container's assembled root in sandbox mode, and a layered `Root` its live shares and mappings | @@ -120,7 +120,7 @@ fields. ## Egress policy files `nvx::host::network::policy` compiles the egress policy files that -`nvx.py run` and `nvx.py sandbox` take with `--network-egress-policy-file` +`nvx-rs run` and `nvx-rs sandbox` take with `--network-egress-policy-file` ([format](../doc/run.md#run-openvmm-directly)). It replaced the former `scripts/nvx_tools/egress_policy.py` when `nvx.py` delegated to `nvx-rs`, and compiles the policies that it compiled to the same rules, but reads policy @@ -392,10 +392,10 @@ export planner applies containment checks. The guest lookups of `nvx::host::artifacts` replaced Python code that `nvx.py` ran, so they keep its contract byte for byte: its limits, option semantics, outputs, and messages, which quote values as `repr` does. The `nvx-rs` CLI, -a faithful port of the runtime steps of `nvx.py`, also checks values as Python -does: whitespace is what `str.isspace` accepts, integers are what `int` reads, -and JSON what `json.loads` reads, failing with its messages. Host paths are -normalized as `pathlib` normalizes them, with backslashes on Windows, and +a faithful port of the former runtime steps of `nvx.py`, also checks values as +Python does: whitespace is what `str.isspace` accepts, integers are what `int` +reads, and JSON what `json.loads` reads, failing with its messages. Host paths +are normalized as `pathlib` normalizes them, with backslashes on Windows, and resolved as `Path.resolve` resolves them; file identities are those of `os.stat`, which on Windows are the volume serial number and the 128-bit file ID. The private `python` module implements these semantics after CPython 3.14, @@ -417,21 +417,23 @@ have [scenario tests](#scenario-tests). ## The nvx-rs CLI -The `cli` feature builds the `nvx-rs` executable, a faithful port of the -runtime steps of `nvx.py` that `nvx.py` delegates to it -([microsoft/nvx#359](https://github.com/microsoft/nvx/issues/359) and -[microsoft/nvx#638](https://github.com/microsoft/nvx/issues/638)): +The `cli` feature builds the `nvx-rs` executable, which runs NVX's microVMs +and sandboxes, a faithful port of the former runtime steps of `nvx.py` +([microsoft/nvx#359](https://github.com/microsoft/nvx/issues/359), +[microsoft/nvx#638](https://github.com/microsoft/nvx/issues/638), and +[microsoft/nvx#725](https://github.com/microsoft/nvx/issues/725)): ```text -nvx-rs [-h] [--root PATH] [--prog PROG] artifacts -nvx-rs [-h] [--root PATH] [--prog PROG] egress compile PATH -nvx-rs [-h] [--root PATH] [--prog PROG] run [OPTIONS] -nvx-rs [-h] [--root PATH] [--prog PROG] sandbox [run|provision|start|exec|stop|deprovision] [OPTIONS] +nvx-rs [-h] [--root PATH] artifacts +nvx-rs [-h] [--root PATH] egress compile PATH +nvx-rs [-h] [--root PATH] run [OPTIONS] +nvx-rs [-h] [--root PATH] sandbox [run|provision|start|exec|stop|deprovision] [OPTIONS] ``` ```bash cd nvx-rs -cargo build --release --features cli # target/release/nvx-rs +cargo build --release --locked --features cli # target/release/nvx-rs +target/release/nvx-rs run # the checkout's artifacts ``` Every NVX release package ships the release build as `bin/nvx-rs`, a static @@ -442,20 +444,17 @@ provenance in `provenance/nvx-rs.provenance.json` extracted release runs as is: without `--root`, it uses the release that holds it; see [Artifact root](#artifact-root). [`nvx.py download`](../doc/usage.md#download) installs it in a checkout of the -sources that built it, so that `nvx.py` runs it without Cargo. +sources that built it, so that the checkout runs it without a Cargo build; +see [Usage](../doc/usage.md#nvx-rs). -`nvx.py` runs the executable that `NVX_RS` names, else -`nvx-rs/target/release/nvx-rs[.exe]` of its checkout, and else builds that one -with Cargo first, with its output on standard error; see +The scenarios of `nvx.py test-microvm` run the executable that `NVX_RS` names, +else `nvx-rs/target/release/nvx-rs[.exe]` of their checkout, and else build +that one with Cargo first, with its output on standard error; see [Usage](../doc/usage.md#nvx-rs). `nvx.py` records the provenance of the builds that it makes and installs, and `nvx.py record-nvx-rs-provenance` that of a -build made otherwise. It runs `nvx-rs --prog nvx.py --root REPOSITORY run -OPTIONS` and `nvx-rs --prog nvx.py --root REPOSITORY sandbox OPERATION OPTIONS` -with the options that the user gave, in full, and the standard streams and exit -status of the command. For `run`, it adds the `--cold-boot-guidance` of a host -whose CPU no built-in CPU profile serves, which names the OpenVMM that -`nvx-rs --root REPOSITORY artifacts` gives; see [`run`](#run). It also runs -`nvx-rs egress compile PATH` for the egress scenarios of `nvx.py test-microvm`. +build made otherwise. The scenarios run their microVMs and sandboxes with +`nvx-rs --root ROOT run OPTIONS` and `nvx-rs --root ROOT sandbox OPERATION +OPTIONS`, and compile their egress policies with `nvx-rs egress compile PATH`. Errors print `error[CODE]: MESSAGE` to standard error and exit with status 1, as `nvx.py` reports a `ScriptError`, where `CODE` is the stable @@ -474,16 +473,11 @@ own, even when the workload left its standard error without a line end. Each error is one line: a line break in a value that its message names, such as a path, prints as `\n` or `\r`. -`--prog PROG` names the program in usage, help, and error messages, `nvx-rs` by -default. `nvx.py` passes its own name, so that the usage errors of `nvx.py run` -and `nvx.py sandbox` that `nvx-rs` reports, such as an invalid `--processors` -or `--layer`, read `usage: nvx.py run ...` and `nvx.py run: error[CODE]: ...`, -or those of `nvx.py sandbox`, as they did before `nvx.py` delegated the -commands, but for their codes. -The usage wraps for the name as `argparse` wraps it for 80 columns. A name -with square brackets or control characters, which could read as the code of an -error or end its line, is refused with `argument-invalid`, under the name -`nvx-rs`. +Usage errors and help start with the usage of the parser, on one line, such as +`usage: nvx-rs egress compile [-h] PATH`. The usages of `run` and `sandbox`, +which take many options, name them `[OPTIONS]`, as in +`usage: nvx-rs run [OPTIONS]`, and their help lists every option. A usage error +names the parser after the usage, as in `nvx-rs run: error[CODE]: MESSAGE`. ### Artifact root @@ -561,14 +555,13 @@ message. ### `run` -`nvx-rs run` keeps the contract of `nvx.py run` -([Usage](../doc/usage.md#run)): every option of its parser, with the same -names, defaults, and value checks, and the same checks, messages, outputs, and -exit statuses. It prints `>> COMMAND`, OpenVMM's command line, quoted as -`sandbox run` quotes it, and unless `--dry-run` is given, it runs OpenVMM with -the standard streams, so that the guest's console and the terminal settings -that OpenVMM restores behave as they do for `nvx.py run`, and exits with -OpenVMM's status. +`nvx-rs run` ([Usage](../doc/usage.md#run)) has the options, with their names, +defaults, and value checks, and the checks, messages, outputs, and exit +statuses of the `nvx.py run` that it replaced. It prints `>> COMMAND`, +OpenVMM's command line, quoted as `sandbox run` quotes it, and unless +`--dry-run` is given, it runs OpenVMM with the standard streams, so that the +guest's console and the terminal settings that OpenVMM restores behave as +they do for OpenVMM itself, and exits with OpenVMM's status. - A cold boot runs the kernel and the initramfs of `--guest`, `alpine` by default, with `--memory-mib` or else the guest's default, 128 MiB for Alpine @@ -584,21 +577,26 @@ OpenVMM's status. `--mount-write` name host paths, which OpenVMM attributes to the share that holds them, and which must be absolute with an aggregate, unlike those of `sandbox`, which apply to the `--mount` before them. -- `--cold-boot-guidance=TEXT`, which neither the usage nor the help shows, - holds what to print to standard error after OpenVMM fails a cold boot: the - CPU-profile guidance that `nvx.py run` passes for a host whose CPU no - built-in profile serves, which `nvx.py` derives from the CPU catalog that - `doctor` and `benchmark` also use. It prints only after OpenVMM exits with a - status other than 0 from a cold boot, and never after a dry run, a restore, - an error or usage error of `nvx-rs`, or Ctrl-C, so that it follows only - OpenVMM's own refusals, whose statuses those of `nvx-rs` share. +- After OpenVMM exits with a status other than 0 from a cold boot, on a host + whose CPU no built-in CPU profile serves, it prints guidance to standard + error: what `--cpu-profile auto` does on that CPU, and, on an Intel or AMD + CPU, how to request a built-in profile for it with the fingerprint that the + OpenVMM of the run writes. It identifies the CPU with CPUID, as OpenVMM does, + and reads the CPUs that the built-in profiles serve from + `src/host/cli/cpu_generations.json`, a copy of the identities of the profiles + that OpenVMM pins. It prints none for `--cpu-profile` with a profile ID, nor + for `host` on an Intel or AMD CPU, whose failures OpenVMM explains, and never + after a dry run, a restore, an error or usage error of `nvx-rs`, or Ctrl-C, + so that it follows only OpenVMM's own refusals, whose statuses those of + `nvx-rs` share. ### `sandbox` -`nvx-rs sandbox` keeps the contract of `nvx.py sandbox`: every option of its -parser, with the same names, defaults, and value checks, and the same -operations, checks, messages, outputs, exit statuses, and outcome reports, -except as [Other differences from nvx.py](#other-differences-from-nvxpy) lists. +`nvx-rs sandbox` ([Usage](../doc/usage.md#sandbox)) has the options, with +their names, defaults, and value checks, and the operations, checks, +messages, outputs, exit statuses, and outcome reports of the `nvx.py sandbox` +that it replaced, except as +[Other differences from nvx.py](#other-differences-from-nvxpy) lists. The operation defaults to `run`. Options take their values as `--option VALUE` or `--option=VALUE`, repeated options accumulate, and `--mount-deny`, `--mount-allow`, and `--mount-write` apply to the `--mount` @@ -730,14 +728,12 @@ guest agents of NVX advertise bits 0, 4, and 6 in sandbox mode. #### Other differences from nvx.py -- Long options cannot be abbreviated; `nvx.py` passes them in full. -- Help and usage are formatted for 80 columns, as `argparse` formats them for a - pipe, and an invalid choice lists the choices quoted, as Python 3.14.5 and - later do. -- `nvx.py` checks the choices of its string options, unknown options, missing - values, and `--help` before `nvx-rs` checks the values, so on a command line - with two errors, the one that `nvx.py` reports wins, where the Python - implementation reported the first. +- Long options cannot be abbreviated, where `argparse` accepted any prefix that + named one option. +- The usage is one line, which names `[OPTIONS]` for the options of `run` and + `sandbox`, and the program is always `nvx-rs`, where `argparse` listed every + option and wrapped the usage for 80 columns. An invalid choice lists the + choices quoted, as Python 3.14.5 and later do. - Where `nvx.py` would end with a Python traceback, as for a configuration without a field that `start` reads, `nvx-rs` prints `error[CODE]: MESSAGE`, with the message of the exception, and exits with status 1. @@ -795,7 +791,7 @@ serves the sandbox profile: its managed lifecycle, with the capability on its standard input, the control console, and an outcome report at stop, and its one-shot runs, which run `/bin/echo`, `/bin/sleep`, `/bin/true`, or `/bin/false`, write an outcome report if asked, and exit with the workload's -status. It also runs the microVMs of `nvx.py run`, without a guest: a cold +status. It also runs the microVMs of `nvx-rs run`, without a guest: a cold boot, or the restore of a snapshot directory, prints that it booted or restored the microVM, writes an outcome report if asked, and exits with status 0, and a restore whose snapshot directory is missing fails with status 1. Kernel @@ -832,9 +828,7 @@ checks that refuse a command line before anything changes, dry runs, one-shot runs, the lifecycles of sandboxes of every configuration, their executions, the refusals of lifecycle operations, configurations of every format, from the hand-written `config.json` files of `tests/fixtures/state/`, `egress compile`, -and `artifacts`. `tests/fixtures/run-help.txt` holds the help of `nvx-rs run`, -which `nvx.py run` prints too, since it parses the options of `nvx-rs run` -before it runs it: a test of each program checks its help against it. +and `artifacts`. They replaced the scenarios that `scripts/nvx_rs_fixtures.py` and `scripts/nvx_rs_run_fixtures.py` recorded from the Python implementations of @@ -1162,6 +1156,10 @@ OpenVMM's vectors in `tests/vectors/openvmm/` with the files of the OpenVMM submodule, which the `nvx-rs` job cannot check out. The NVX CLI jobs run it once they check the submodule out, and so does an OpenVMM pin promotion; see [OpenVMM's vectors](tests/vectors/README.md#openvmms-vectors). +`tests/openvmm_cpu_profiles.rs` is an ignored test that compares +`src/host/cli/cpu_generations.json`, the identities of the built-in CPU +profiles from which [`run`](#run) guides a failed cold boot, with the profiles +that the submodule pins, and runs where that one does. `tests/managed_lifecycle.rs` is an ignored test of the managed lifecycle of a guest whose init hands the boot to the managed agent, on a real hypervisor: diff --git a/nvx-rs/src/host/artifacts.rs b/nvx-rs/src/host/artifacts.rs index 89d2af315..5009b1d83 100644 --- a/nvx-rs/src/host/artifacts.rs +++ b/nvx-rs/src/host/artifacts.rs @@ -392,7 +392,7 @@ const MAX_METADATA_LEN: u64 = 1 << 20; /// A Linux guest that `nvx.py` builds and runs, as `scripts/nvx_tools/guests.py` describes it. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub enum Guest { - /// Alpine Linux: the default guest of `nvx.py run`, and the one that runs sandboxes. + /// Alpine Linux: the default guest of `nvx-rs run`, and the one that runs sandboxes. Alpine, /// Ubuntu Base, which also provides the distro layer of sandbox workloads. Ubuntu, @@ -451,10 +451,10 @@ impl Guest { } } - /// Returns the RAM, in MiB, with which `nvx.py run` boots the guest without `--memory-mib`: + /// Returns the RAM, in MiB, with which `nvx-rs run` boots the guest without `--memory-mib`: /// the `default_memory_mib` of its descriptor in `scripts/nvx_tools/guests.py`, which - /// `nvx.py test-microvm` also uses, and which a test of `nvx.py` pins to the help of - /// `nvx-rs run` in `tests/fixtures/run-help.txt`. + /// `nvx.py test-microvm` also uses, and which a test of `nvx.py` checks against the help of + /// `nvx-rs run`. pub const fn default_memory_mib(self) -> u32 { match self { Self::Alpine => 128, @@ -630,7 +630,7 @@ impl ArtifactDir { required(self.kernel(), "Linux direct kernel") } - /// Checks that a guest's initramfs is a regular file and returns its path, as `nvx.py run` + /// Checks that a guest's initramfs is a regular file and returns its path, as `nvx-rs run` /// does before it boots the guest. /// /// Fails with [`ErrorCode::ArtifactMissing`] and the message `DISTRIBUTION initramfs not @@ -642,7 +642,7 @@ impl ArtifactDir { ) } - /// Returns the artifacts of a sandbox, as `nvx.py sandbox run` and `nvx.py sandbox start` + /// Returns the artifacts of a sandbox, as `nvx-rs sandbox run` and `nvx-rs sandbox start` /// require them: the OpenVMM executable, the kernel, and the Alpine initramfs, in that order, /// located in this directory. /// diff --git a/nvx-rs/src/host/cli/args.rs b/nvx-rs/src/host/cli/args.rs index 8630a5d73..4d190ecd9 100644 --- a/nvx-rs/src/host/cli/args.rs +++ b/nvx-rs/src/host/cli/args.rs @@ -23,18 +23,17 @@ //! the top-level parser. //! - `-h` and `--help` stop the parse for the help as soon as they are consumed. //! -//! Unlike `argparse`, it does not expand unambiguous prefixes of option names: `nvx.py` passes -//! canonical options, so `--dry` is an unknown option rather than `--dry-run`. +//! Unlike `argparse`, it does not expand unambiguous prefixes of option names, so `--dry` is an +//! unknown option rather than `--dry-run`. //! -//! [`Syntax`] holds what a parser's usage and help show, and formats its usage for any program -//! name as `argparse` wraps it for 80 columns. +//! [`Syntax`] holds what a parser's usage and help show. The usage is one line, which names the +//! parser's few options, or `[OPTIONS]` for many, and the help lists every option. use crate::ErrorCode; use crate::host::python::builtins::{Int, decimal_value, repr}; -/// The width of usage and help: 80 columns, less the margin of 2 that `argparse` keeps, as for a -/// pipe, whose `shutil.get_terminal_size()` is 80 columns. -const WIDTH: usize = 78; +/// The name of the program in usage and error messages. +const PROGRAM: &str = "nvx-rs"; /// What usage messages start with. const USAGE_PREFIX: &str = "usage: "; @@ -44,104 +43,51 @@ pub(crate) struct Syntax { /// The command of the parser after the program's name, such as `sandbox`, or nothing for the /// top-level parser. pub(crate) command: &'static str, - /// The usage parts of the options, such as `[-h]` and `[--root PATH]`, in order. - pub(crate) optionals: &'static [&'static str], - /// The usage parts of the positionals, such as `PATH`, in order. - pub(crate) positionals: &'static [&'static str], + /// What the usage shows after the parser's name: its options, such as `[-h] [--root PATH]`, + /// or `[OPTIONS]` for a parser with many, and its positionals, such as `PATH`. + pub(crate) arguments: &'static str, /// The help after the usage and its blank line: the description and the sections of the /// arguments, without a final line end. pub(crate) help: &'static str, } impl Syntax { - /// Returns the parser's name for the program `program`, such as `nvx-rs sandbox`: what - /// `argparse` calls the `prog` of a subparser. - pub(crate) fn prog(&self, program: &str) -> String { + /// Returns the parser's name, such as `nvx-rs sandbox`, which starts its usage errors. + pub(crate) fn prog(&self) -> String { if self.command.is_empty() { - program.to_owned() + PROGRAM.to_owned() } else { - format!("{program} {}", self.command) + format!("{PROGRAM} {}", self.command) } } - /// Returns the usage for the program `program`, `usage: PROG ...`, without a line end. - pub(crate) fn usage(&self, program: &str) -> String { - format_usage(&self.prog(program), self.optionals, self.positionals) + /// Returns the usage, `usage: PROG ARGUMENTS`, on one line without a line end. + pub(crate) fn usage(&self) -> String { + format!("{USAGE_PREFIX}{} {}", self.prog(), self.arguments) } - /// Returns the help for the program `program`: the usage, a blank line, and the rest. - pub(crate) fn help(&self, program: &str) -> String { - format!("{}\n\n{}", self.usage(program), self.help) + /// Returns the help: the usage, a blank line, and the rest. + pub(crate) fn help(&self) -> String { + format!("{}\n\n{}", self.usage(), self.help) } -} -/// Returns the usage of the parser `prog` with the usage parts `optionals` and `positionals`, as -/// `HelpFormatter._format_usage` of CPython 3.14 formats it for [`WIDTH`]: on one line if it -/// fits, and otherwise wrapped at the parts, after the name if the name is short and on the lines -/// after it if not. -fn format_usage(prog: &str, optionals: &[&str], positionals: &[&str]) -> String { - let length = |text: &str| text.chars().count(); - let parts: Vec<&str> = optionals.iter().chain(positionals).copied().collect(); - let full = std::iter::once(prog) - .chain(parts.iter().copied()) - .collect::>() - .join(" "); - if USAGE_PREFIX.len() + length(&full) <= WIDTH { - return format!("{USAGE_PREFIX}{full}"); - } - // Wraps `parts` into lines after `indent`, the first of which continues the prefix if given. - let lines_of = |parts: &[&str], indent: &str, prefix: Option<&str>| -> Vec { - let mut lines = Vec::new(); - let mut line: Vec<&str> = Vec::new(); - let indent_length = length(indent); - let mut line_length = prefix.map_or(indent_length, length) - 1; - for part in parts { - if line_length + 1 + length(part) > WIDTH && !line.is_empty() { - lines.push(format!("{indent}{}", line.join(" "))); - line.clear(); - line_length = indent_length - 1; - } - line.push(part); - line_length += length(part) + 1; - } - if !line.is_empty() { - lines.push(format!("{indent}{}", line.join(" "))); - } - if prefix.is_some() - && let Some(first) = lines.first_mut() - { - *first = first[indent.len()..].to_owned(); - } - lines - }; - let prog_length = length(prog); - // A name of at most three quarters of the width is followed by the parts. - let lines = if 4 * (USAGE_PREFIX.len() + prog_length) <= 3 * WIDTH { - let indent = " ".repeat(USAGE_PREFIX.len() + prog_length + 1); - if optionals.is_empty() { - let leading: Vec<&str> = std::iter::once(prog) - .chain(positionals.iter().copied()) - .collect(); - lines_of(&leading, &indent, Some(USAGE_PREFIX)) - } else { - let leading: Vec<&str> = std::iter::once(prog) - .chain(optionals.iter().copied()) - .collect(); - let mut lines = lines_of(&leading, &indent, Some(USAGE_PREFIX)); - lines.extend(lines_of(positionals, &indent, None)); - lines - } - } else { - let indent = " ".repeat(USAGE_PREFIX.len()); - let mut lines = lines_of(&parts, &indent, None); - if lines.len() > 1 { - lines = lines_of(optionals, &indent, None); - lines.extend(lines_of(positionals, &indent, None)); - } - lines.insert(0, prog.to_owned()); - lines - }; - format!("{USAGE_PREFIX}{}", lines.join("\n")) + /// Asserts that the help lists `-h, --help` and each of `options`, and no other option: the + /// options that the help describes on lines of their own, indented by two spaces. + #[cfg(test)] + pub(crate) fn assert_lists(&self, options: &[Spec]) { + let mut listed: Vec<&str> = self + .help + .lines() + .filter(|line| line.starts_with(" -")) + .filter_map(|line| line.split_whitespace().find(|word| word.starts_with("--"))) + .collect(); + let mut expected: Vec<&str> = std::iter::once("--help") + .chain(options.iter().map(|option| option.name)) + .collect(); + listed.sort_unstable(); + expected.sort_unstable(); + assert_eq!(listed, expected, "{}", self.prog()); + } } /// How a parse ends early: with a usage error, or with the help that `-h` asked for. @@ -816,65 +762,27 @@ mod tests { ); } - // The expected usages are what CPython 3.14's argparse prints for 80 columns. #[test] - fn usages_wrap_as_argparse_wraps_them() { - const OPTIONALS: &[&str] = &[ - "[-h]", - "[--layer ROLE,PATH,EROFS_UUID]", - "[--scratch SCRATCH]", - "[--state-dir STATE_DIR]", - "[--entrypoint ENTRYPOINT]", - "[--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]", - ]; - const POSITIONALS: &[&str] = &["[{run,provision}]"]; - let long = "a-program-whose-name-is-long-enough-to-take-its-own-line.py"; - let syntax = |command, optionals, positionals| Syntax { - command, - optionals, - positionals, - help: "", - }; - // A usage that fits stays on one line. - assert_eq!( - syntax("egress compile", &["[-h]"], &["PATH"]).usage("nvx-rs"), - "usage: nvx-rs egress compile [-h] PATH" - ); - // A short name is followed by the options, and the positionals start a line of their own. - assert_eq!( - syntax("", OPTIONALS, POSITIONALS).usage("tool"), - "usage: tool [-h] [--layer ROLE,PATH,EROFS_UUID] [--scratch SCRATCH]\n \ - [--state-dir STATE_DIR] [--entrypoint ENTRYPOINT]\n \ - [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]\n \ - [{run,provision}]" - ); - // A long name takes a line of its own, and so do the options and the positionals if all - // of them do not fit on one line. - assert_eq!( - syntax("", OPTIONALS, POSITIONALS).usage(long), - format!( - "usage: {long}\n [-h] [--layer ROLE,PATH,EROFS_UUID] [--scratch SCRATCH]\n \ - [--state-dir STATE_DIR] [--entrypoint ENTRYPOINT]\n \ - [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]\n \ - [{{run,provision}}]" - ) - ); - assert_eq!( - syntax("egress compile", &["[-h]"], &["PATH"]).usage(long), - format!("usage: {long} egress compile\n [-h] PATH") - ); - assert_eq!( - syntax("", &["[-h]", "[--root PATH]"], &["path"]).usage(long), - format!("usage: {long}\n [-h] [--root PATH] path") - ); - let help = Syntax { + fn usages_name_the_parser_on_one_line() { + let compile = Syntax { + command: "egress compile", + arguments: "[-h] PATH", help: "options:\n -h, --help show this help message and exit", - ..syntax("artifacts", &["[-h]"], &[]) }; - assert_eq!(help.prog("nvx.py"), "nvx.py artifacts"); + assert_eq!(compile.prog(), "nvx-rs egress compile"); + assert_eq!(compile.usage(), "usage: nvx-rs egress compile [-h] PATH"); assert_eq!( - help.help("nvx.py"), - "usage: nvx.py artifacts [-h]\n\noptions:\n -h, --help show this help message and exit" + compile.help(), + "usage: nvx-rs egress compile [-h] PATH\n\noptions:\n -h, --help show this help \ + message and exit" ); + compile.assert_lists(&[]); + let top = Syntax { + command: "", + arguments: "[OPTIONS]", + ..compile + }; + assert_eq!(top.prog(), "nvx-rs"); + assert_eq!(top.usage(), "usage: nvx-rs [OPTIONS]"); } } diff --git a/nvx-rs/src/host/cli/artifacts.rs b/nvx-rs/src/host/cli/artifacts.rs index c2a5fc81d..a4a56b0f3 100644 --- a/nvx-rs/src/host/cli/artifacts.rs +++ b/nvx-rs/src/host/cli/artifacts.rs @@ -1,5 +1,5 @@ -//! `nvx-rs artifacts`: the paths of the artifacts that `nvx.py` runs, as JSON, and the artifact -//! root that every command uses. +//! `nvx-rs artifacts`: the paths of the NVX artifacts, as JSON, and the artifact root that every +//! command uses. //! //! The root is the directory that `--root` names, in the layout that its built artifacts reveal, //! as [`ArtifactDir::from_dir`] tells a release from a repository checkout. Without `--root`, it @@ -21,10 +21,9 @@ use crate::host::python::dumps::{self, Value}; pub(crate) const OPENVMM_VARIABLE: &str = "NVX_OPENVMM"; /// The usage and help of the parser. -const SYNTAX: Syntax = Syntax { +pub(super) const SYNTAX: Syntax = Syntax { command: "artifacts", - optionals: &["[-h]"], - positionals: &[], + arguments: "[-h]", help: "\ Print the paths of the NVX artifacts as a JSON object: the layout and root of the artifact directory, the OpenVMM executable, the guest kernel, and for each @@ -35,12 +34,11 @@ options: -h, --help show this help message and exit", }; -/// Parses the arguments of `artifacts` for the program `program`, and returns those that it does -/// not recognize. -pub(super) fn parse(arguments: &[String], program: &str) -> Result> { +/// Parses the arguments of `artifacts`, and returns those that it does not recognize. +pub(super) fn parse(arguments: &[String]) -> Result> { Parser::new(&[], &[]) .parse(arguments, &mut |_: &'static str, _: Option<&str>| Ok(())) - .map_err(|stop| Exit::from_stop(stop, &SYNTAX, program)) + .map_err(|stop| Exit::from_stop(stop, &SYNTAX)) } /// Prints the paths of the artifacts of the root. diff --git a/nvx-rs/src/host/cli/cpu_generations.json b/nvx-rs/src/host/cli/cpu_generations.json new file mode 100644 index 000000000..745972edc --- /dev/null +++ b/nvx-rs/src/host/cli/cpu_generations.json @@ -0,0 +1,61 @@ +[ + { + "id": "intel.skylake-sp.v1", + "vendor": "GenuineIntel", + "generation": { + "name": "skylake-sp", + "cpus": [{ "family": 6, "model": 85, "steppings": [0, 4] }] + } + }, + { + "id": "intel.icelake-sp.v1", + "vendor": "GenuineIntel", + "generation": { + "name": "icelake-sp", + "cpus": [{ "family": 6, "model": 106, "steppings": [0, 15] }] + } + }, + { + "id": "intel.emeraldrapids.v1", + "vendor": "GenuineIntel", + "generation": { + "name": "emeraldrapids", + "cpus": [{ "family": 6, "model": 207, "steppings": [0, 15] }] + } + }, + { + "id": "intel.alderlake.v1", + "vendor": "GenuineIntel", + "generation": { + "name": "alderlake", + "cpus": [ + { "family": 6, "model": 151, "steppings": [0, 15] }, + { "family": 6, "model": 154, "steppings": [0, 15] } + ] + } + }, + { + "id": "amd.milan.v1", + "vendor": "AuthenticAMD", + "generation": { + "name": "milan", + "cpus": [{ "family": 25, "model": 1, "steppings": [0, 15] }] + } + }, + { + "id": "amd.genoa.v1", + "vendor": "AuthenticAMD", + "generation": { + "name": "genoa", + "cpus": [{ "family": 25, "model": 17, "steppings": [0, 15] }] + } + }, + { + "id": "amd.turin.v1", + "vendor": "AuthenticAMD", + "generation": { + "name": "turin", + "cpus": [{ "family": 26, "model": 2, "steppings": [0, 15] }] + } + } +] diff --git a/nvx-rs/src/host/cli/cpu_profile.rs b/nvx-rs/src/host/cli/cpu_profile.rs new file mode 100644 index 000000000..e99a449e4 --- /dev/null +++ b/nvx-rs/src/host/cli/cpu_profile.rs @@ -0,0 +1,703 @@ +//! The CPU-profile guidance that `nvx-rs run` prints after OpenVMM fails a cold boot on a host +//! whose CPU no built-in CPU profile serves. +//! +//! OpenVMM presents each guest a CPU profile, by default the built-in profile of the host CPU's +//! generation (`doc/usage.md`, "CPU profiles"). `cpu_generations.json` copies the identity of +//! every profile that OpenVMM pins, from its document in `vmm_core/cpu_profile/profiles/` of the +//! OpenVMM submodule: its ID, the vendor of its CPUs, and its generation's name and CPU models, +//! each a display family, a display model, and the first and last of its steppings. +//! `tests/openvmm_cpu_profiles.rs` compares the copy with the submodule's documents. +//! [`HostCpu::current`] identifies the host CPU with CPUID, as OpenVMM does. +//! +//! On an Intel or AMD CPU that no built-in profile serves, `--cpu-profile auto`, the default, falls +//! back to a profile derived from the host, which OpenVMM warns about, and on another vendor's +//! CPU, `auto` and `host` fail with `E_PROFILE_HOST_UNKNOWN`. OpenVMM keeps its standard error on +//! the terminal, so `nvx-rs run` cannot read the code that ended a cold boot: [`guidance`] says +//! what `auto` does on such a CPU, and on an Intel or AMD CPU, asks for the CPU's fingerprint, +//! from which NVX can add a built-in profile for it. An explicit profile ID fails with its own +//! code, and `host` on an Intel or AMD CPU with OpenVMM's own explanation, which need none. + +use serde::Deserialize; + +/// The identities of the built-in CPU profiles, in the order in which [`guidance`] lists their +/// generations. +const CATALOG: &str = include_str!("cpu_generations.json"); +/// The `--cpu-profile` of a profile that OpenVMM derives from the host that it runs on. +const HOST_PROFILE: &str = "host"; +/// The code with which OpenVMM fails a cold boot where no CPU profile serves the host's CPU. +const PROFILE_HOST_UNKNOWN: &str = "E_PROFILE_HOST_UNKNOWN"; +/// The marker that leads OpenVMM's warning of each cold boot whose `--cpu-profile auto` falls back +/// to a host profile. +const FALLBACK_MARKER: &str = "NVX-CPU-PROFILE-FALLBACK:"; +/// NVX's issue form that requests a built-in CPU profile for a CPU. +const REQUEST_URL: &str = "https://github.com/microsoft/nvx/issues/new?template=cpu-profile.yml"; +/// The vendors whose CPUs host profiles serve, by their CPUID vendor strings, each with its name +/// and the issue that tracks built-in profiles for more of its CPUs. +const HOST_PROFILE_VENDORS: [(&str, &str, &str); 2] = [ + ( + "GenuineIntel", + "Intel", + "https://github.com/microsoft/nvx/issues/408", + ), + ( + "AuthenticAMD", + "AMD", + "https://github.com/microsoft/nvx/issues/409", + ), +]; + +/// A CPU model of a generation. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +#[serde(deny_unknown_fields)] +struct CpuModel { + /// The display family. + family: u32, + /// The display model. + model: u32, + /// The first and the last stepping that the generation covers. + steppings: [u32; 2], +} + +impl CpuModel { + /// Returns the model as the guidance names it: `6/106`, or `6/85 steppings 0-4` if the + /// generation covers only some of its steppings. + fn describe(&self) -> String { + let name = format!("{}/{}", self.family, self.model); + match self.steppings { + [0, 15] => name, + [first, last] => format!("{name} steppings {first}-{last}"), + } + } +} + +/// The CPU generation that a built-in profile serves. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +#[serde(deny_unknown_fields)] +struct Generation { + /// The generation's name, such as `skylake-sp`. + name: String, + /// The generation's CPU models. + cpus: Vec, +} + +/// The identity of a built-in CPU profile. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +#[serde(deny_unknown_fields)] +struct Profile { + /// The profile's ID, such as `intel.skylake-sp.v1`. + id: String, + /// The CPUID vendor string of the generation's CPUs, such as `GenuineIntel`. + vendor: String, + /// The generation that the profile serves. + generation: Generation, +} + +impl Profile { + /// Returns whether the profile serves `cpu`. + fn serves(&self, cpu: &HostCpu) -> bool { + self.vendor == cpu.vendor + && self.generation.cpus.iter().any(|model| { + (model.family, model.model) == (cpu.family, cpu.model) + && (model.steppings[0]..=model.steppings[1]).contains(&cpu.stepping) + }) + } + + /// Returns the profile's generation and its CPU models, such as `alderlake 6/151 and 6/154`. + fn describe(&self) -> String { + let cpus: Vec = self + .generation + .cpus + .iter() + .map(CpuModel::describe) + .collect(); + format!("{} {}", self.generation.name, cpus.join(" and ")) + } +} + +/// Returns the built-in CPU profiles of [`CATALOG`]. +fn catalog() -> Vec { + serde_json::from_str(CATALOG).expect("cpu_generations.json lists the built-in CPU profiles") +} + +/// A host CPU, as CPUID identifies it. Only x86-64 hosts have CPUID, so elsewhere, nothing +/// identifies one. +#[derive(Debug, Clone, PartialEq, Eq)] +#[cfg_attr(not(target_arch = "x86_64"), allow(dead_code))] +pub(crate) struct HostCpu { + /// The vendor string of CPUID leaf 0, such as `GenuineIntel`. + vendor: String, + /// The display family. + family: u32, + /// The display model. + model: u32, + /// The stepping. + stepping: u32, + /// The brand string of CPUID leaves 0x80000002 to 0x80000004, or nothing. + brand: String, +} + +impl HostCpu { + /// Identifies the CPU that this process runs on, or returns `None` on a host without CPUID, + /// which is not an x86-64 host. + pub(crate) fn current() -> Option { + #[cfg(target_arch = "x86_64")] + { + Some(Self::from_cpuid(cpuid)) + } + #[cfg(not(target_arch = "x86_64"))] + { + None + } + } + + /// Identifies the CPU from `cpuid`, which returns EAX, EBX, ECX, and EDX of a leaf, as + /// OpenVMM's `cpu_profile` crate identifies it: Intel extends the model for families 6 and 15, + /// and AMD only for family 15. + #[cfg_attr(not(target_arch = "x86_64"), allow(dead_code))] + fn from_cpuid(mut cpuid: impl FnMut(u32) -> [u32; 4]) -> Self { + let [max_basic, ebx, ecx, edx] = cpuid(0); + let vendor = register_text(&[ebx, edx, ecx]); + let signature = if max_basic >= 1 { cpuid(1)[0] } else { 0 }; + let brand = if cpuid(0x8000_0000)[0] >= 0x8000_0004 { + let registers: Vec = (0x8000_0002..=0x8000_0004).flat_map(&mut cpuid).collect(); + register_text(®isters) + } else { + String::new() + }; + let stepping = signature & 0xf; + let base_model = (signature >> 4) & 0xf; + let base_family = (signature >> 8) & 0xf; + let family = if base_family == 0xf { + base_family + ((signature >> 20) & 0xff) + } else { + base_family + }; + let extended = base_family == 0xf || (base_family == 6 && vendor == "GenuineIntel"); + let model = if extended { + (((signature >> 16) & 0xf) << 4) | base_model + } else { + base_model + }; + Self { + vendor, + family, + model, + stepping, + brand, + } + } + + /// Returns the CPU's vendor and display family, model, and stepping, such as + /// `GenuineIntel 6/173/1`. + fn signature(&self) -> String { + format!( + "{} {}/{}/{}", + self.vendor, self.family, self.model, self.stepping + ) + } + + /// Returns the CPU as the guidance names it: its signature and its brand string, if any, such + /// as `GenuineIntel 6/173/1 (Intel(R) Xeon(R) 6973P-C)`. + fn describe(&self) -> String { + if self.brand.is_empty() { + self.signature() + } else { + format!("{} ({})", self.signature(), self.brand) + } + } + + /// Returns the link to the CPU profile request form for the CPU, which prefills the form's + /// title and CPU fields, as OpenVMM's fallback warning does. + fn request_url(&self) -> String { + let mut url = format!( + "{REQUEST_URL}&title={}&signature={}", + encode_query_value(&format!("CPU profile: {}", self.describe())), + encode_query_value(&self.signature()) + ); + if !self.brand.is_empty() { + url.push_str("&cpu="); + url.push_str(&encode_query_value(&self.brand)); + } + url + } +} + +/// Runs CPUID with `leaf` and subleaf 0, and returns EAX, EBX, ECX, and EDX. +#[cfg(target_arch = "x86_64")] +#[allow(unsafe_code, unused_unsafe)] +fn cpuid(leaf: u32) -> [u32; 4] { + // SAFETY: CPUID executes in user mode on every x86-64 processor and accesses no memory. The + // intrinsic became a safe function after the crate's minimum Rust version. + let result = unsafe { core::arch::x86_64::__cpuid_count(leaf, 0) }; + [result.eax, result.ebx, result.ecx, result.edx] +} + +/// Returns the text of the little-endian bytes of `registers`, as CPUID's vendor and brand +/// strings hold it, without the NULs that pad it or the spaces around it. +#[cfg_attr(not(target_arch = "x86_64"), allow(dead_code))] +fn register_text(registers: &[u32]) -> String { + let bytes: Vec = registers + .iter() + .flat_map(|register| register.to_le_bytes()) + .collect(); + String::from_utf8_lossy(&bytes) + .trim_end_matches('\0') + .trim() + .to_owned() +} + +/// Percent-encodes `value` for a URL query, as OpenVMM's fallback warning does: every byte but +/// ASCII letters, digits, `-`, `.`, `_`, and `~`. +fn encode_query_value(value: &str) -> String { + let mut encoded = String::with_capacity(value.len()); + for byte in value.bytes() { + if byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'.' | b'_' | b'~') { + encoded.push(char::from(byte)); + } else { + encoded.push_str(&format!("%{byte:02X}")); + } + } + encoded +} + +/// Returns what to print after OpenVMM failed a cold boot with `--cpu-profile cpu_profile`, or +/// with its default, `auto`, if `None`, on the CPU `host` with the `hypervisor` backend, where +/// `openvmm` is the OpenVMM executable, quoted for the shell. Returns `None` unless no built-in +/// profile serves the CPU and the profile is `auto`, or `host` on a CPU of a vendor that host +/// profiles do not serve either, and where the CPU is unknown. +pub(crate) fn guidance( + cpu_profile: Option<&str>, + host: Option<&HostCpu>, + openvmm: &str, + hypervisor: &str, +) -> Option { + let host = host?; + let catalog = catalog(); + if catalog.iter().any(|profile| profile.serves(host)) { + return None; + } + let vendor = HOST_PROFILE_VENDORS + .iter() + .find(|(vendor, ..)| *vendor == host.vendor); + match cpu_profile { + None | Some("auto") => {} + Some(HOST_PROFILE) if vendor.is_none() => {} + Some(_) => return None, + } + let unserved = format!( + "nvx: no built-in CPU profile serves this host's CPU, {}", + host.describe() + ); + let generations: Vec = catalog.iter().map(Profile::describe).collect(); + let generations = format!("the built-in profiles cover {}.", generations.join(", ")); + let Some((_, name, issue)) = vendor else { + return Some(format!( + "{unserved}, so a cold boot with --cpu-profile auto, the default, fails on it with \ + {PROFILE_HOST_UNKNOWN}; {generations}\n\ + nvx: --cpu-profile {HOST_PROFILE} cannot boot on it either: host CPU profiles serve \ + only Intel and AMD CPUs." + )); + }; + Some( + [ + format!( + "{unserved}, so --cpu-profile auto, the default, falls back on it to a CPU profile \ + derived from this host, which OpenVMM warns about with {FALLBACK_MARKER}; \ + {generations}" + ), + "nvx: OpenVMM's error above names the cause of this failure, and says so where the \ + host profile cannot boot; doc/usage.md (\"CPU profiles\") explains a host profile's \ + limits." + .to_owned(), + "nvx: Help NVX add a built-in profile for this CPU: run".to_owned(), + format!( + "nvx: {openvmm} --hypervisor {hypervisor} --cpu-fingerprint fingerprint.json" + ), + "nvx: and attach fingerprint.json to a CPU profile request:".to_owned(), + format!("nvx: {}", host.request_url()), + format!("nvx: {issue} tracks CPU profiles for more {name} CPUs."), + ] + .join("\n"), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Returns the CPU of `vendor` with the display `family`, `model`, and `stepping`, and the + /// brand string `brand`. + fn cpu(vendor: &str, family: u32, model: u32, stepping: u32, brand: &str) -> HostCpu { + HostCpu { + vendor: vendor.to_owned(), + family, + model, + stepping, + brand: brand.to_owned(), + } + } + + /// Returns the guidance for `host` with `--cpu-profile cpu_profile`, for `/nvx/openvmm` on KVM. + fn guide(cpu_profile: Option<&str>, host: &HostCpu) -> Option { + guidance(cpu_profile, Some(host), "/nvx/openvmm", "kvm") + } + + /// Returns the name of the generation that serves the CPU of `vendor` with `family`, `model`, + /// and `stepping`, if any. + fn generation(vendor: &str, family: u32, model: u32, stepping: u32) -> Option { + let host = cpu(vendor, family, model, stepping, ""); + catalog() + .into_iter() + .find(|profile| profile.serves(&host)) + .map(|profile| profile.generation.name) + } + + #[test] + fn the_catalog_lists_the_built_in_profiles() { + let catalog = catalog(); + let ids: Vec<&str> = catalog.iter().map(|profile| profile.id.as_str()).collect(); + assert_eq!( + ids, + [ + "intel.skylake-sp.v1", + "intel.icelake-sp.v1", + "intel.emeraldrapids.v1", + "intel.alderlake.v1", + "amd.milan.v1", + "amd.genoa.v1", + "amd.turin.v1", + ] + ); + let generations: Vec = catalog.iter().map(Profile::describe).collect(); + assert_eq!( + generations.join(", "), + "skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, emeraldrapids 6/207, alderlake \ + 6/151 and 6/154, milan 25/1, genoa 25/17, turin 26/2" + ); + } + + #[test] + fn cpus_map_to_the_generations_of_the_built_in_profiles() { + let intel = |model, stepping| generation("GenuineIntel", 6, model, stepping); + assert_eq!(intel(85, 4).as_deref(), Some("skylake-sp")); + assert_eq!(intel(106, 6).as_deref(), Some("icelake-sp")); + assert_eq!(intel(207, 2).as_deref(), Some("emeraldrapids")); + // Alder Lake-S and Alder Lake-P and -H share one profile. + assert_eq!(intel(151, 2).as_deref(), Some("alderlake")); + assert_eq!(intel(154, 3).as_deref(), Some("alderlake")); + // Cascade Lake and Cooper Lake share model 85 but have no profile, and neither have + // Sapphire Rapids, Tiger Lake, and Raptor Lake. + for (model, stepping) in [(85, 7), (85, 11), (143, 8), (140, 1), (183, 1)] { + assert_eq!(intel(model, stepping), None, "6/{model}/{stepping}"); + } + // Milan's profile serves every stepping of AMD's family 25 model 1, Genoa's every stepping + // of model 17, and Turin's every stepping of family 26 model 2. + for (family, model, expected) in [(25, 1, "milan"), (25, 17, "genoa"), (26, 2, "turin")] { + for stepping in [0, 1, 2, 15] { + assert_eq!( + generation("AuthenticAMD", family, model, stepping).as_deref(), + Some(expected) + ); + } + } + // Vermeer and Raphael (Ryzen), Turin Dense, and Granite Ridge have no profile, and neither + // has another vendor's CPU with the signature of an AMD generation. + for (vendor, family, model) in [ + ("AuthenticAMD", 1, 1), + ("AuthenticAMD", 25, 33), + ("AuthenticAMD", 25, 97), + ("AuthenticAMD", 26, 17), + ("AuthenticAMD", 26, 68), + ("GenuineIntel", 25, 1), + ("HygonGenuine", 25, 1), + ("GenuineIntel", 26, 2), + ] { + assert_eq!( + generation(vendor, family, model, 1), + None, + "{vendor} {family}/{model}" + ); + } + } + + /// Returns the registers of a CPUID leaf that hold `text`, as the vendor and brand leaves + /// hold it. + fn words(text: &[u8]) -> Vec { + text.chunks(4) + .map(|chunk| u32::from_le_bytes(chunk.try_into().unwrap())) + .collect() + } + + /// Returns the CPUID of a CPU of `vendor` with the CPUID.1:EAX `signature` and, if not empty, + /// the 48-byte `brand`. + fn fake_cpuid(vendor: &[u8; 12], signature: u32, brand: &[u8]) -> impl FnMut(u32) -> [u32; 4] { + let vendor = words(vendor); + let brand = words(brand); + let max_extended = if brand.is_empty() { + 0x8000_0001 + } else { + 0x8000_0004 + }; + move |leaf| match leaf { + 0 => [0x1b, vendor[0], vendor[2], vendor[1]], + 1 => [signature, 0, 0, 0], + 0x8000_0000 => [max_extended, 0, 0, 0], + 0x8000_0002..=0x8000_0004 if !brand.is_empty() => { + let first = (leaf - 0x8000_0002) as usize * 4; + [ + brand[first], + brand[first + 1], + brand[first + 2], + brand[first + 3], + ] + } + _ => [0; 4], + } + } + + #[test] + fn cpuid_identifies_the_cpu_as_openvmm_does() { + let skylake = HostCpu::from_cpuid(fake_cpuid( + b"GenuineIntel", + 0x0005_0654, + b"Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz\0\0\0\0\0\0", + )); + assert_eq!( + skylake, + cpu( + "GenuineIntel", + 6, + 85, + 4, + "Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz" + ) + ); + // Older Intel brand strings start with spaces, which are trimmed as NULs are. + let padded = HostCpu::from_cpuid(fake_cpuid( + b"GenuineIntel", + 0x000c_06f2, + b" Intel(R) Xeon(R) Platinum 8573C\0\0\0\0\0\0\0\0\0", + )); + assert_eq!((padded.family, padded.model, padded.stepping), (6, 207, 2)); + assert_eq!(padded.brand, "Intel(R) Xeon(R) Platinum 8573C"); + // AMD extends the family and the model of family 15 alone. + let genoa = HostCpu::from_cpuid(fake_cpuid(b"AuthenticAMD", 0x00a1_0f11, b"")); + assert_eq!(genoa, cpu("AuthenticAMD", 25, 17, 1, "")); + let k7 = HostCpu::from_cpuid(fake_cpuid(b"AuthenticAMD", 0x0001_0661, b"")); + assert_eq!((k7.family, k7.model, k7.stepping), (6, 6, 1)); + let p6 = HostCpu::from_cpuid(fake_cpuid(b"GenuineIntel", 0x0001_0661, b"")); + assert_eq!((p6.family, p6.model, p6.stepping), (6, 22, 1)); + } + + /// Returns the host CPU as the host OS reports it, without its brand string: the first + /// processor of `/proc/cpuinfo` on Linux, and `PROCESSOR_IDENTIFIER` on Windows, such as + /// `Intel64 Family 6 Model 85 Stepping 4, GenuineIntel`. + #[cfg(all(target_arch = "x86_64", any(target_os = "linux", windows)))] + fn reported_cpu() -> Option { + if cfg!(windows) { + let identifier = std::env::var("PROCESSOR_IDENTIFIER").ok()?; + let (description, vendor) = identifier.split_once(", ")?; + let words: Vec<&str> = description.split_whitespace().collect(); + let number = |name: &str| { + let position = words.iter().position(|word| *word == name)?; + words.get(position + 1)?.parse().ok() + }; + return Some(cpu( + vendor.trim(), + number("Family")?, + number("Model")?, + number("Stepping")?, + "", + )); + } + let cpuinfo = std::fs::read_to_string("/proc/cpuinfo").ok()?; + let field = |name: &str| { + cpuinfo.lines().find_map(|line| { + let (key, value) = line.split_once(':')?; + (key.trim() == name).then(|| value.trim().to_owned()) + }) + }; + Some(cpu( + &field("vendor_id")?, + field("cpu family")?.parse().ok()?, + field("model")?.parse().ok()?, + field("stepping")?.parse().ok()?, + "", + )) + } + + #[cfg(all(target_arch = "x86_64", any(target_os = "linux", windows)))] + #[test] + fn the_host_cpu_is_the_one_that_the_host_os_reports() { + let current = HostCpu::current().unwrap(); + let reported = reported_cpu().expect("the host OS reports the CPU"); + assert_eq!( + HostCpu { + brand: String::new(), + ..current.clone() + }, + reported + ); + if cfg!(target_os = "linux") && !current.brand.is_empty() { + let cpuinfo = std::fs::read_to_string("/proc/cpuinfo").unwrap(); + assert!( + cpuinfo + .lines() + .any(|line| line.split_once(':').is_some_and(|(key, value)| { + key.trim() == "model name" && value.trim() == current.brand + })), + "{current:?}" + ); + } + } + + #[cfg(not(target_arch = "x86_64"))] + #[test] + fn hosts_without_cpuid_identify_no_cpu() { + assert_eq!(HostCpu::current(), None); + } + + #[test] + fn request_links_prefill_the_cpu() { + let granite_rapids = cpu("GenuineIntel", 6, 173, 1, "Intel(R) Xeon(R) 6973P-C"); + assert_eq!(granite_rapids.signature(), "GenuineIntel 6/173/1"); + assert_eq!( + granite_rapids.describe(), + "GenuineIntel 6/173/1 (Intel(R) Xeon(R) 6973P-C)" + ); + // The link that OpenVMM's fallback warning gives the same CPU. + assert_eq!( + granite_rapids.request_url(), + "https://github.com/microsoft/nvx/issues/new?template=cpu-profile.yml\ + &title=CPU%20profile%3A%20GenuineIntel%206%2F173%2F1%20%28Intel%28R%29\ + %20Xeon%28R%29%206973P-C%29&signature=GenuineIntel%206%2F173%2F1\ + &cpu=Intel%28R%29%20Xeon%28R%29%206973P-C" + ); + // A CPU without a brand string prefills only its signature. + let brandless = cpu("GenuineIntel", 6, 173, 1, ""); + assert_eq!(brandless.describe(), "GenuineIntel 6/173/1"); + assert_eq!( + brandless.request_url(), + format!( + "{REQUEST_URL}&title=CPU%20profile%3A%20GenuineIntel%206%2F173%2F1\ + &signature=GenuineIntel%206%2F173%2F1" + ) + ); + assert!( + cpu("GenuineIntel", 6, 173, 1, "&=?#%+ é") + .request_url() + .ends_with("&cpu=%26%3D%3F%23%25%2B%20%C3%A9") + ); + } + + #[test] + fn unserved_intel_and_amd_cpus_get_the_fallback_and_a_profile_request() { + let tiger_lake = cpu( + "GenuineIntel", + 6, + 140, + 1, + "11th Gen Intel(R) Core(TM) i7-1185G7", + ); + let guidance = guide(None, &tiger_lake).unwrap(); + let lines: Vec<&str> = guidance.lines().collect(); + assert!( + lines.iter().all(|line| line.starts_with("nvx: ")), + "{guidance}" + ); + // auto falls back on the CPU, so the guidance says what auto does there, not which code + // ended this run, which nvx-rs run cannot read. + assert_eq!( + lines[0], + "nvx: no built-in CPU profile serves this host's CPU, GenuineIntel 6/140/1 (11th Gen \ + Intel(R) Core(TM) i7-1185G7), so --cpu-profile auto, the default, falls back on it to \ + a CPU profile derived from this host, which OpenVMM warns about with \ + NVX-CPU-PROFILE-FALLBACK:; the built-in profiles cover skylake-sp 6/85 steppings \ + 0-4, icelake-sp 6/106, emeraldrapids 6/207, alderlake 6/151 and 6/154, milan 25/1, \ + genoa 25/17, turin 26/2." + ); + assert!(lines[1].contains("OpenVMM's error above names the cause")); + assert!(!guidance.contains(PROFILE_HOST_UNKNOWN)); + // It asks for the CPU's fingerprint through the CPU profile request form, which the link + // prefills with the CPU. + let request = format!("nvx: {}", tiger_lake.request_url()); + assert_eq!( + lines[2..], + [ + "nvx: Help NVX add a built-in profile for this CPU: run", + "nvx: /nvx/openvmm --hypervisor kvm --cpu-fingerprint fingerprint.json", + "nvx: and attach fingerprint.json to a CPU profile request:", + request.as_str(), + "nvx: https://github.com/microsoft/nvx/issues/408 tracks CPU profiles for more \ + Intel CPUs.", + ] + ); + assert_eq!(guide(Some("auto"), &tiger_lake).as_ref(), Some(&guidance)); + // A host profile that fails on this CPU fails for a reason that OpenVMM's own error + // explains. + assert_eq!(guide(Some(HOST_PROFILE), &tiger_lake), None); + + // Host profiles serve AMD CPUs too: an AMD CPU without a built-in profile, such as + // Raphael (Ryzen 7000), falls back as an Intel one does, with the issue that tracks + // profiles for more AMD CPUs. + let raphael = cpu("AuthenticAMD", 25, 97, 2, ""); + let guidance = guide(None, &raphael).unwrap(); + assert!(guidance.contains( + "this host's CPU, AuthenticAMD 25/97/2, so --cpu-profile auto, the default, falls \ + back on it" + )); + assert!(guidance.contains(&raphael.request_url())); + assert!(guidance.ends_with( + "\nnvx: https://github.com/microsoft/nvx/issues/409 tracks CPU profiles for more AMD \ + CPUs." + )); + assert!(!guidance.contains("issues/408")); + assert_eq!(guide(Some(HOST_PROFILE), &raphael), None); + } + + #[test] + fn unserved_cpus_of_other_vendors_get_no_profile_request() { + // Another vendor's CPU fails with either profile, and gets no request form and no issue: + // the time ABI does not serve its CPUs. + let hygon = cpu("HygonGenuine", 24, 0, 1, ""); + let guidance = guide(Some("auto"), &hygon).unwrap(); + assert_eq!( + guidance, + "nvx: no built-in CPU profile serves this host's CPU, HygonGenuine 24/0/1, so a cold \ + boot with --cpu-profile auto, the default, fails on it with E_PROFILE_HOST_UNKNOWN; \ + the built-in profiles cover skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, \ + emeraldrapids 6/207, alderlake 6/151 and 6/154, milan 25/1, genoa 25/17, turin \ + 26/2.\nnvx: --cpu-profile host cannot boot on it either: host CPU profiles serve only \ + Intel and AMD CPUs." + ); + assert_eq!(guide(None, &hygon).as_ref(), Some(&guidance)); + assert_eq!(guide(Some(HOST_PROFILE), &hygon).as_ref(), Some(&guidance)); + } + + #[test] + fn served_cpus_explicit_profiles_and_unknown_cpus_get_none() { + let tiger_lake = cpu("GenuineIntel", 6, 140, 1, ""); + let raphael = cpu("AuthenticAMD", 25, 97, 2, ""); + let hygon = cpu("HygonGenuine", 24, 0, 1, ""); + let alder_lake = cpu("GenuineIntel", 6, 154, 3, ""); + let milan = cpu("AuthenticAMD", 25, 1, 1, ""); + let genoa = cpu("AuthenticAMD", 25, 17, 1, ""); + let turin = cpu("AuthenticAMD", 26, 2, 1, ""); + for (cpu_profile, host) in [ + (None, &alder_lake), + (Some(HOST_PROFILE), &alder_lake), + (None, &milan), + (Some(HOST_PROFILE), &milan), + (None, &genoa), + (None, &turin), + (Some(HOST_PROFILE), &turin), + (Some("intel.alderlake.v1"), &tiger_lake), + (Some("intel.alderlake.v1"), &raphael), + (Some("intel.skylake-sp.v1"), &milan), + (Some("intel.skylake-sp.v1"), &hygon), + ] { + assert_eq!(guide(cpu_profile, host), None, "{cpu_profile:?} {host:?}"); + } + assert_eq!(guidance(None, None, "/nvx/openvmm", "kvm"), None); + } +} diff --git a/nvx-rs/src/host/cli/egress.rs b/nvx-rs/src/host/cli/egress.rs index ced091d57..48b9677ba 100644 --- a/nvx-rs/src/host/cli/egress.rs +++ b/nvx-rs/src/host/cli/egress.rs @@ -1,4 +1,4 @@ -//! `nvx-rs egress compile PATH`: compiles an egress policy file, as `nvx.py run` and `nvx.py +//! `nvx-rs egress compile PATH`: compiles an egress policy file, as `nvx-rs run` and `nvx-rs //! sandbox` compile their `--network-egress-policy-file`, and prints the native OpenVMM rules as //! a JSON object, `{"allow": [...], "deny": [...]}`, with the rules of each action in the order //! in which OpenVMM receives them. @@ -12,10 +12,9 @@ use crate::host::network::policy::compile_policy_file; use crate::host::python::dumps::{self, Value}; /// The usage and help of the parser. -const SYNTAX: Syntax = Syntax { +pub(super) const SYNTAX: Syntax = Syntax { command: "egress", - optionals: &["[-h]"], - positionals: &["{compile} ..."], + arguments: "[-h] {compile} ...", help: "\ Compile egress policy files. @@ -29,10 +28,9 @@ options: /// The actions, in the order in which usage messages list them. const ACTIONS: [&str; 1] = ["compile"]; /// The usage and help of the `compile` parser. -const COMPILE_SYNTAX: Syntax = Syntax { +pub(super) const COMPILE_SYNTAX: Syntax = Syntax { command: "egress compile", - optionals: &["[-h]"], - positionals: &["PATH"], + arguments: "[-h] PATH", help: "\ Compile the egress policy file at PATH, as --network-egress-policy-file does, and print its native rules as a JSON object: {\"allow\": [...], \"deny\": [...]}. @@ -45,17 +43,16 @@ options: -h, --help show this help message and exit", }; -/// Parses the arguments of `egress` for the program `program`, and returns the policy file and -/// the arguments that no parser recognizes. -pub(super) fn parse(arguments: &[String], program: &str) -> Result<(String, Vec)> { +/// Parses the arguments of `egress`, and returns the policy file and the arguments that no parser +/// recognizes. +pub(super) fn parse(arguments: &[String]) -> Result<(String, Vec)> { let parser = Parser::new(&[], &[Spec::one("action")]); let (index, mut extras) = parser .split_command(arguments, &mut |_: &'static str, _: Option<&str>| Ok(())) - .map_err(|stop| Exit::from_stop(stop, &SYNTAX, program))?; + .map_err(|stop| Exit::from_stop(stop, &SYNTAX))?; if arguments[index] != "compile" { return Err(Exit::usage( &SYNTAX, - program, ErrorCode::ArgumentInvalid, format!( "argument action: {}", @@ -72,7 +69,7 @@ pub(super) fn parse(arguments: &[String], program: &str) -> Result<(String, Vec< Ok(()) }, ) - .map_err(|stop| Exit::from_stop(stop, &COMPILE_SYNTAX, program))?; + .map_err(|stop| Exit::from_stop(stop, &COMPILE_SYNTAX))?; extras.extend(unrecognized); Ok((path, extras)) } diff --git a/nvx-rs/src/host/cli/mod.rs b/nvx-rs/src/host/cli/mod.rs index 5b65219ce..32dc7c2fd 100644 --- a/nvx-rs/src/host/cli/mod.rs +++ b/nvx-rs/src/host/cli/mod.rs @@ -1,37 +1,35 @@ -//! The `nvx-rs` command-line interface, the backend of the runtime steps of `nvx.py`. +//! The `nvx-rs` command-line interface, which runs NVX's microVMs and sandboxes. //! //! ```text -//! nvx-rs [-h] [--root PATH] [--prog PROG] artifacts -//! nvx-rs [-h] [--root PATH] [--prog PROG] egress compile PATH -//! nvx-rs [-h] [--root PATH] [--prog PROG] run [OPTIONS] -//! nvx-rs [-h] [--root PATH] [--prog PROG] sandbox [run|provision|start|exec|stop|deprovision] [OPTIONS] +//! nvx-rs [-h] [--root PATH] artifacts +//! nvx-rs [-h] [--root PATH] egress compile PATH +//! nvx-rs [-h] [--root PATH] run [OPTIONS] +//! nvx-rs [-h] [--root PATH] sandbox [run|provision|start|exec|stop|deprovision] [OPTIONS] //! ``` //! -//! - `artifacts` prints the paths of the artifacts that `nvx.py` runs as JSON. +//! - `artifacts` prints the paths of the artifacts of the artifact root as JSON. //! - `egress compile PATH` compiles an egress policy file of `--network-egress-policy-file` and //! prints its native rules as JSON. -//! - `run` is `nvx.py run`: the same options, checks, outputs, and exit statuses, and it boots or -//! restores a microVM on the terminal. -//! - `sandbox` is `nvx.py sandbox`: the same options, outputs, exit statuses, outcome reports, -//! and state directories. +//! - `run` boots or restores a microVM on the terminal, with the options, checks, outputs, and +//! exit statuses of the `nvx.py run` that it replaced. +//! - `sandbox` runs or manages a sandbox, with the options, outputs, exit statuses, outcome +//! reports, and state directories of the `nvx.py sandbox` that it replaced. //! //! The artifact root is `--root`, or else the release or repository checkout that holds the //! executable, and the `NVX_OPENVMM` environment variable names another OpenVMM executable. -//! `--prog` names the program in usage and error messages, `nvx-rs` by default: `nvx.py`, which -//! runs `nvx-rs` for its `run` and `sandbox` commands, passes its own name, so that its users see -//! the messages that it printed before it delegated the commands. A name with square brackets or -//! control characters, which could read as the code of an error or end its line, is refused. //! //! Errors print `error[CODE]: MESSAGE` to standard error and exit with status 1, as `nvx.py` //! reports a `ScriptError`, with the stable [`ErrorCode`] of the failure. Usage errors print the -//! usage and `PROG: error[CODE]: MESSAGE`, with one of the `argument-*` codes, and exit with -//! status 2, as `argparse` does. Each error is one line: a line break in a value that its message -//! names, such as a path, prints as `\n` or `\r`. Ctrl-C prints `Interrupted` and exits with -//! status 130. Text ends its lines as Python's `print` ends them: `\r\n` on Windows, and `\n` -//! elsewhere. +//! one-line usage of the parser and `PROG: error[CODE]: MESSAGE`, where `PROG` is the parser's +//! name, such as `nvx-rs run`, with one of the `argument-*` codes, and exit with status 2, as +//! `argparse` does. Help prints the usage and every option of the parser, and exits with status 0. +//! Each error is one line: a line break in a value that its message names, such as a path, prints +//! as `\n` or `\r`. Ctrl-C prints `Interrupted` and exits with status 130. Text ends its lines as +//! Python's `print` ends them: `\r\n` on Windows, and `\n` elsewhere. mod args; mod artifacts; +mod cpu_profile; mod deadline; mod egress; mod network; @@ -49,18 +47,17 @@ use crate::host::platform; use crate::host::python::builtins::repr; use crate::host::python::fs::os_error_message; -/// The name of the program in usage messages unless `--prog` names another. -const PROGRAM: &str = "nvx-rs"; /// The commands, in the order in which usage messages list them. const COMMANDS: [&str; 4] = ["artifacts", "egress", "run", "sandbox"]; +/// The options of the top-level parser. +const OPTIONS: [Spec; 1] = [Spec::one("--root")]; /// The usage and help of the top-level parser. const SYNTAX: Syntax = Syntax { command: "", - optionals: &["[-h]", "[--root PATH]", "[--prog PROG]"], - positionals: &["{artifacts,egress,run,sandbox} ..."], + arguments: "[-h] [--root PATH] {artifacts,egress,run,sandbox} ...", help: "\ -Run the NVX steps that nvx.py delegates: artifact lookup, egress policy -compilation, microVM runs, and sandboxes. +Run NVX microVMs and sandboxes, look up NVX artifacts, and compile egress +policies. positional arguments: {artifacts,egress,run,sandbox} @@ -75,9 +72,6 @@ options: --root PATH the NVX repository checkout or extracted release whose artifacts to use (default: the release or checkout that holds this executable) - --prog PROG the program name that usage and error messages show, - such as nvx.py, which runs nvx-rs for its commands - (default: nvx-rs) environment: NVX_OPENVMM the OpenVMM executable to run instead of the one of the @@ -131,23 +125,21 @@ impl Exit { } } - /// Returns the usage error `message` of `code` of the parser of `syntax` for the program - /// `program`. - pub(crate) fn usage(syntax: &Syntax, program: &str, code: ErrorCode, message: String) -> Self { + /// Returns the usage error `message` of `code` of the parser of `syntax`. + pub(crate) fn usage(syntax: &Syntax, code: ErrorCode, message: String) -> Self { Self::Usage { - usage: syntax.usage(program), - prog: syntax.prog(program), + usage: syntax.usage(), + prog: syntax.prog(), code, message, } } - /// Returns the end of a parse that `stop` ended, for the parser of `syntax` and the program - /// `program`. - pub(crate) fn from_stop(stop: Stop, syntax: &Syntax, program: &str) -> Self { + /// Returns the end of a parse that `stop` ended, for the parser of `syntax`. + pub(crate) fn from_stop(stop: Stop, syntax: &Syntax) -> Self { match stop { - Stop::Usage(code, message) => Self::usage(syntax, program, code, message), - Stop::Help => Self::Help(syntax.help(program)), + Stop::Usage(code, message) => Self::usage(syntax, code, message), + Stop::Help => Self::Help(syntax.help()), } } @@ -282,29 +274,12 @@ enum Command { struct TopLevel { /// `--root`, if given. root: Option, - /// `--prog`, if given. - prog: Option, } impl Receiver for TopLevel { - fn take(&mut self, name: &'static str, value: Option<&str>) -> std::result::Result<(), String> { - let value = value.map(str::to_owned); - match name { - "--root" => self.root = value, - _ => { - // A usage error names the program before its code, so the name may hold nothing - // that reads as a code or ends the error's line. - if value.as_deref().is_some_and(|program| { - program.contains(['[', ']']) || program.contains(char::is_control) - }) { - return Err( - "the program name may hold no square brackets or control characters" - .to_owned(), - ); - } - self.prog = value; - } - } + fn take(&mut self, _: &'static str, value: Option<&str>) -> std::result::Result<(), String> { + // `--root` is the only option. + self.root = value.map(str::to_owned); Ok(()) } } @@ -316,7 +291,6 @@ fn run(arguments: &[OsString]) -> Result { let text = argument.to_str().ok_or_else(|| { Exit::usage( &SYNTAX, - PROGRAM, ErrorCode::ArgumentInvalid, format!( "argument {} is not valid Unicode: {}", @@ -327,29 +301,25 @@ fn run(arguments: &[OsString]) -> Result { })?; strings.push(text.to_owned()); } - let parser = Parser::new( - &[Spec::one("--root"), Spec::one("--prog")], - &[Spec::one("command")], - ); + let parser = Parser::new(&OPTIONS, &[Spec::one("command")]); let mut top = TopLevel::default(); - let split = parser.split_command(&strings, &mut top); - // A usage error names the program that the options before it named. - let program = top.prog.clone().unwrap_or_else(|| PROGRAM.to_owned()); - let usage = |code, message: String| Exit::usage(&SYNTAX, &program, code, message); - let (index, mut extras) = split.map_err(|stop| Exit::from_stop(stop, &SYNTAX, &program))?; + let usage = |code, message: String| Exit::usage(&SYNTAX, code, message); + let (index, mut extras) = parser + .split_command(&strings, &mut top) + .map_err(|stop| Exit::from_stop(stop, &SYNTAX))?; let rest = &strings[index + 1..]; let (command, unrecognized) = match strings[index].as_str() { - "artifacts" => (Command::Artifacts, artifacts::parse(rest, &program)?), + "artifacts" => (Command::Artifacts, artifacts::parse(rest)?), "egress" => { - let (path, unrecognized) = egress::parse(rest, &program)?; + let (path, unrecognized) = egress::parse(rest)?; (Command::EgressCompile(path), unrecognized) } "run" => { - let (options, unrecognized) = run::parse(rest, &program)?; + let (options, unrecognized) = run::parse(rest)?; (Command::Run(Box::new(options)), unrecognized) } "sandbox" => { - let (options, unrecognized) = sandbox::parse(rest, &program)?; + let (options, unrecognized) = sandbox::parse(rest)?; (Command::Sandbox(Box::new(options)), unrecognized) } other => { @@ -393,31 +363,37 @@ mod tests { } #[test] - fn program_names_that_could_read_as_codes_or_end_lines_are_refused() { - for program in [ - "error[io]: wrapper", - "wrapper: error[io]: alias", - "a]", - "a\nb", - "a\tb", + fn usages_are_one_line_and_helps_list_every_option() { + for (syntax, options) in [ + (&SYNTAX, &OPTIONS[..]), + (&artifacts::SYNTAX, &[][..]), + (&egress::SYNTAX, &[][..]), + (&egress::COMPILE_SYNTAX, &[][..]), + (&run::SYNTAX, &run::OPTIONS[..]), + (&sandbox::SYNTAX, &sandbox::OPTIONS[..]), ] { - let arguments = ["--prog", program, "egress", "compile"].map(OsString::from); - let Err(Exit::Usage { - prog, - code, - message, - .. - }) = run(&arguments) - else { - panic!("{program:?} was accepted"); - }; - assert_eq!(prog, PROGRAM, "{program:?}"); - assert_eq!(code, ErrorCode::ArgumentInvalid, "{program:?}"); - assert_eq!( - message, - "argument --prog: the program name may hold no square brackets or control \ - characters" - ); + let usage = syntax.usage(); + assert!(usage.starts_with("usage: nvx-rs"), "{usage}"); + assert!(!usage.contains('\n'), "{usage}"); + syntax.assert_lists(options); + } + assert_eq!(run::SYNTAX.usage(), "usage: nvx-rs run [OPTIONS]"); + assert_eq!( + sandbox::SYNTAX.usage(), + "usage: nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] [OPTIONS]" + ); + } + + #[test] + fn the_run_help_states_the_memory_default_of_each_guest() { + let help = run::SYNTAX + .help + .split_whitespace() + .collect::>() + .join(" "); + for guest in crate::host::artifacts::Guest::ALL { + let default = format!("{} {}", guest.name(), guest.default_memory_mib()); + assert!(help.contains(&default), "{default}: {help}"); } } diff --git a/nvx-rs/src/host/cli/run.rs b/nvx-rs/src/host/cli/run.rs index 7ad7b0b74..94e2f7cb8 100644 --- a/nvx-rs/src/host/cli/run.rs +++ b/nvx-rs/src/host/cli/run.rs @@ -1,10 +1,10 @@ -//! `nvx-rs run`: `nvx.py run`, ported from `command_run` and the `run` parser of `scripts/nvx.py`, -//! with its options, defaults, checks, outputs, and exit statuses. +//! `nvx-rs run`: the port of the former `nvx.py run`, from `command_run` and the `run` parser of +//! `scripts/nvx.py`, with its options, defaults, checks, outputs, and exit statuses. //! //! It boots a guest in a microVM, or restores a microVM from a snapshot, on the terminal: it //! prints OpenVMM's command as `>> COMMAND`, then runs OpenVMM with the standard streams, unless //! `--dry-run` is given, and exits with OpenVMM's status. The parser reads the options of -//! `nvx.py run` with their names, defaults, types, and choices, and fails as `argparse` fails, +//! `nvx.py run` with their names, defaults, types, and choices, and fails as `argparse` failed, //! with status 2. //! //! Its shares differ from those of `sandbox`. `--mount-deny`, `--mount-allow`, and `--mount-write` @@ -14,15 +14,14 @@ //! the kernel command line, and a restore does not, since its guest keeps the binds that its boot //! made. //! -//! The hidden option `--cold-boot-guidance=TEXT`, which neither the usage nor the help shows, -//! holds what to print to standard error after OpenVMM fails a cold boot, such as the CPU-profile -//! guidance of `nvx.py run` for a host whose CPU no built-in CPU profile serves, which `nvx.py` -//! derives from the catalog of the CPUs that those profiles serve. It is printed only after -//! OpenVMM exits with a status other than 0 from a cold boot, and never after a dry run, a -//! restore, a refused option, or Ctrl-C. +//! After OpenVMM exits with a status other than 0 from a cold boot, on a host whose CPU no built-in +//! CPU profile serves, it prints the guidance of [`cpu_profile`] to standard error, which says +//! what `--cpu-profile auto` does on that CPU and how to request a built-in profile for it. It +//! never prints it after a dry run, a restore, a refused option, or Ctrl-C. use super::args::{Parser, Receiver, Spec, Syntax, choice, integer}; use super::artifacts::{Located, locate}; +use super::cpu_profile::{self, HostCpu}; use super::network::{NativeRules, NetworkOptions}; use super::sandbox::{HYPERVISORS, MOUNT_OWNERS, hypervisor, path_string, path_text, run_openvmm}; use super::{Exit, Result, check_interrupt, output}; @@ -40,42 +39,10 @@ use crate::host::sandbox::mount::{ use crate::host::sandbox::options::validate_run_share_targets; use crate::host::sandbox::{SANDBOX_COMMAND_LINE_MAX_SIZE, has_kernel_separator, require_artifact}; -/// The usage and help of the parser, without the hidden `--cold-boot-guidance`. -const SYNTAX: Syntax = Syntax { +/// The usage and help of the parser. +pub(super) const SYNTAX: Syntax = Syntax { command: "run", - optionals: &[ - "[-h]", - "[--guest {alpine,ubuntu,azurelinux}]", - "[--hypervisor {auto,whp,kvm,mshv}]", - "[--machine {microvm}]", - "[--memory-mib MEMORY_MIB]", - "[--memory-capacity-mib MEMORY_CAPACITY_MIB]", - "[--processors {1,2,4,8}]", - "[--cpu-profile ID]", - "[--mount GUEST_TARGET,HOST_PATH[,ro|rw]]", - "[--mount-deny MOUNT_DENY]", - "[--mount-allow HOST_PATH]", - "[--mount-write HOST_PATH]", - "[--mount-owner {vmm,caller}]", - "[--net IPV4/PREFIX]", - "[--network-profile {portable}]", - "[--network-egress {allow,deny}]", - "[--network-ingress {allow,deny}]", - "[--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]", - "[--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]", - "[--network-egress-policy-file PATH]", - "[--host-loopback {allow,deny}]", - "[--network-proxy IPV4:TCP-PORT]", - "[--host-loopback-forward HOST_LOOPBACK_FORWARD]", - "[--outcome-report OUTCOME_REPORT]", - "[--cmdline CMDLINE]", - "[--restore-snapshot RESTORE_SNAPSHOT]", - "[--restore-processors {1,2,4,8}]", - "[--restore-memory-mib RESTORE_MEMORY_MIB]", - "[--restore-ready-path RESTORE_READY_PATH]", - "[--dry-run]", - ], - positionals: &[], + arguments: "[OPTIONS]", help: "\ options: -h, --help show this help message and exit @@ -150,11 +117,9 @@ const MACHINES: [&str; 1] = ["microvm"]; /// The choices of `--processors` and `--restore-processors`, which `argparse` checks after it /// converts the value with `int`, so that it shows the converted value. const PROCESSORS: [&str; 4] = ["1", "2", "4", "8"]; -/// The hidden option that holds the guidance to print after a failed cold boot. -const GUIDANCE_OPTION: &str = "--cold-boot-guidance"; -/// The options of `nvx-rs run`, which parse with the defaults of `nvx.py run`. -const OPTIONS: [Spec; 30] = [ +/// The options of `nvx-rs run`, which parse with the defaults of the former `nvx.py run`. +pub(super) const OPTIONS: [Spec; 29] = [ Spec::one("--guest"), Spec::one("--hypervisor"), Spec::one("--machine"), @@ -184,10 +149,9 @@ const OPTIONS: [Spec; 30] = [ Spec::one("--restore-memory-mib"), Spec::one("--restore-ready-path"), Spec::flag("--dry-run"), - Spec::one(GUIDANCE_OPTION), ]; -/// The parsed options of `nvx-rs run`: the namespace of `nvx.py run`. +/// The parsed options of `nvx-rs run`: the namespace of the former `nvx.py run`. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct Options { /// `--guest`, Alpine by default. @@ -226,8 +190,6 @@ pub(crate) struct Options { pub(crate) restore_ready_path: Option, /// `--dry-run`. pub(crate) dry_run: bool, - /// The hidden `--cold-boot-guidance`. - pub(crate) cold_boot_guidance: Option, } impl Default for Options { @@ -251,7 +213,6 @@ impl Default for Options { restore_memory_mib: None, restore_ready_path: None, dry_run: false, - cold_boot_guidance: None, } } } @@ -298,20 +259,19 @@ impl Receiver for Options { "--restore-memory-mib" => self.restore_memory_mib = Some(integer(&text())?), "--restore-ready-path" => self.restore_ready_path = Some(text()), "--dry-run" => self.dry_run = true, - GUIDANCE_OPTION => self.cold_boot_guidance = Some(text()), _ => return self.network.take(name, &text()), } Ok(()) } } -/// Parses the arguments of `run` for the program `program`, and returns the options and the -/// arguments that the parser does not recognize. -pub(super) fn parse(arguments: &[String], program: &str) -> Result<(Options, Vec)> { +/// Parses the arguments of `run`, and returns the options and the arguments that the parser does +/// not recognize. +pub(super) fn parse(arguments: &[String]) -> Result<(Options, Vec)> { let mut options = Options::default(); let unrecognized = Parser::new(&OPTIONS, &[]) .parse(arguments, &mut options) - .map_err(|stop| Exit::from_stop(stop, &SYNTAX, program))?; + .map_err(|stop| Exit::from_stop(stop, &SYNTAX))?; Ok((options, unrecognized)) } @@ -367,7 +327,7 @@ fn memory_mib(options: &Options) -> Int { .unwrap_or_else(|| Int::from(options.guest.default_memory_mib())) } -/// Runs the microVM of `options`, as `command_run` does. +/// Runs the microVM of `options`, as `command_run` did. pub(super) fn run(root: Option<&str>, options: Options) -> Result { check(&options)?; let rules = options.network.egress_rules()?; @@ -420,10 +380,15 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result { let status = run_openvmm(&command, &[])?; if status != 0 && options.restore_snapshot.is_none() - && let Some(guidance) = &options.cold_boot_guidance + && let Some(guidance) = cpu_profile::guidance( + options.cpu_profile.as_deref(), + HostCpu::current().as_ref(), + &format_command(&command[..1]), + hypervisor(options.hypervisor), + ) { check_interrupt()?; - output::print_stderr(guidance); + output::print_stderr(&guidance); } Err(Exit::Status(status)) } @@ -432,7 +397,7 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result { mod specification; /// Appends the shares of `options`, their policy paths and owner, the network options with the -/// egress `rules`, and the outcome report to `command`, as `command_run` does, and returns the +/// egress `rules`, and the outcome report to `command`, as `command_run` did, and returns the /// kernel command line: the one of `--cmdline`, and on the cold boot of an aggregate, the /// `nvx_share=` tokens of its children, which with OpenVMM's tokens must fit the 1024-byte budget. fn extend_shares_and_network( diff --git a/nvx-rs/src/host/cli/run/specification.rs b/nvx-rs/src/host/cli/run/specification.rs index 51b24817a..d4a729092 100644 --- a/nvx-rs/src/host/cli/run/specification.rs +++ b/nvx-rs/src/host/cli/run/specification.rs @@ -55,7 +55,7 @@ proptest! { "--mount".to_owned(), format!("/second,{},rw", roots[1].display()), "--restore-snapshot".to_owned(), "snapshot".to_owned(), ]); - let (options, unrecognized) = parse(&arguments, "nvx-rs").unwrap(); + let (options, unrecognized) = parse(&arguments).unwrap(); prop_assert!(unrecognized.is_empty()); let rules = options.network.egress_rules().unwrap(); let mut command = Vec::new(); @@ -75,7 +75,7 @@ proptest! { "--mount", "/second,second", option, &relative, ].map(str::to_owned); - let (options, _) = parse(&arguments, "nvx-rs").unwrap(); + let (options, _) = parse(&arguments).unwrap(); let rules = options.network.egress_rules().unwrap(); prop_assert!(extend_shares_and_network(&mut Vec::new(), &options, &rules).is_err()); } diff --git a/nvx-rs/src/host/cli/sandbox/mod.rs b/nvx-rs/src/host/cli/sandbox/mod.rs index e18890d75..2deaa3028 100644 --- a/nvx-rs/src/host/cli/sandbox/mod.rs +++ b/nvx-rs/src/host/cli/sandbox/mod.rs @@ -1,5 +1,6 @@ -//! `nvx-rs sandbox`: `nvx.py sandbox`, ported from `command_sandbox` and the `sandbox` parser of -//! `scripts/nvx.py`, with its options, defaults, checks, outputs, and exit statuses. +//! `nvx-rs sandbox`: the port of the former `nvx.py sandbox`, from `command_sandbox` and the +//! `sandbox` parser of `scripts/nvx.py`, with its options, defaults, checks, outputs, and exit +//! statuses. //! //! - `run`, the default, boots a one-shot sandbox: it prints the OpenVMM command as `>> COMMAND`, //! then runs it on the terminal, unless `--dry-run`, and exits with OpenVMM's status. It runs @@ -11,7 +12,7 @@ //! `--state-dir`; see [`lifecycle`]. //! //! The parser reads the options of `nvx.py sandbox` with their names, defaults, types, and -//! choices, and fails as `argparse` fails, with status 2. The `--mount-deny`, `--mount-allow`, +//! choices, and fails as `argparse` failed, with status 2. The `--mount-deny`, `--mount-allow`, //! and `--mount-write` options name paths of the `--mount` before them. mod lifecycle; @@ -57,47 +58,9 @@ pub(crate) use self::lifecycle::ProvisionConfig; use self::state::RecordedDistro; /// The usage and help of the parser. -const SYNTAX: Syntax = Syntax { +pub(super) const SYNTAX: Syntax = Syntax { command: "sandbox", - optionals: &[ - "[-h]", - "[--layer ROLE,PATH,EROFS_UUID]", - "[--scratch SCRATCH]", - "[--state-dir STATE_DIR]", - "[--entrypoint ENTRYPOINT]", - "[--arg SANDBOX_ARG]", - "[--hostname HOSTNAME]", - "[--workload-user UID:GID]", - "[--memory-max MEMORY_MAX]", - "[--pids-max PIDS_MAX]", - "[--memory-mib MEMORY_MIB]", - "[--timeout TIMEOUT]", - "[--exec-timeout-ms EXEC_TIMEOUT_MS]", - "[--cwd CWD]", - "[--environment KEY=VALUE]", - "[--environment-file PATH]", - "[--inherit-default-environment]", - "[--hypervisor {auto,whp,kvm,mshv}]", - "[--mount GUEST_TARGET,HOST_PATH[,ro|rw]]", - "[--mount-deny HOST_PATH]", - "[--mount-allow HOST_PATH]", - "[--mount-write HOST_PATH]", - "[--mount-owner {vmm,caller}]", - "[--net IPV4/PREFIX]", - "[--network-profile {portable}]", - "[--network-egress {allow,deny}]", - "[--network-ingress {allow,deny}]", - "[--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]", - "[--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]", - "[--network-egress-policy-file PATH]", - "[--host-loopback {allow,deny}]", - "[--network-proxy IPV4:TCP-PORT]", - "[--host-loopback-forward HOST_LOOPBACK_FORWARD]", - "[--outcome-report OUTCOME_REPORT]", - "[--cmdline CMDLINE]", - "[--dry-run]", - ], - positionals: &["[{run,provision,start,exec,stop,deprovision}]"], + arguments: "[{run,provision,start,exec,stop,deprovision}] [OPTIONS]", help: "\ positional arguments: {run,provision,start,exec,stop,deprovision} @@ -192,8 +155,9 @@ pub(super) const HYPERVISORS: [&str; 4] = ["auto", "whp", "kvm", "mshv"]; /// The choices of `--mount-owner`. pub(super) const MOUNT_OWNERS: [&str; 2] = ["vmm", "caller"]; -/// The options of `nvx-rs sandbox`, which parse with the defaults of `nvx.py sandbox`. -const OPTIONS: [Spec; 35] = [ +/// The options of `nvx-rs sandbox`, which parse with the defaults of the former +/// `nvx.py sandbox`. +pub(super) const OPTIONS: [Spec; 35] = [ Spec::one("--layer"), Spec::one("--scratch"), Spec::one("--state-dir"), @@ -231,7 +195,7 @@ const OPTIONS: [Spec; 35] = [ Spec::flag("--dry-run"), ]; -/// The parsed options of `nvx-rs sandbox`: the namespace of `nvx.py sandbox`. +/// The parsed options of `nvx-rs sandbox`: the namespace of the former `nvx.py sandbox`. #[derive(Debug, Clone, PartialEq)] pub(crate) struct Options { /// The operation, `run` by default. @@ -412,13 +376,13 @@ impl Receiver for Options { } } -/// Parses the arguments of `sandbox` for the program `program`, and returns the options and the -/// arguments that the parser does not recognize. -pub(super) fn parse(arguments: &[String], program: &str) -> Result<(Options, Vec)> { +/// Parses the arguments of `sandbox`, and returns the options and the arguments that the parser +/// does not recognize. +pub(super) fn parse(arguments: &[String]) -> Result<(Options, Vec)> { let mut options = Options::default(); let unrecognized = Parser::new(&OPTIONS, &[Spec::optional("sandbox_operation")]) .parse(arguments, &mut options) - .map_err(|stop| Exit::from_stop(stop, &SYNTAX, program))?; + .map_err(|stop| Exit::from_stop(stop, &SYNTAX))?; Ok((options, unrecognized)) } @@ -530,7 +494,7 @@ pub(crate) struct Artifacts<'a> { impl Artifacts<'_> { /// Returns the OpenVMM executable, kernel, and Alpine initramfs, checked in that order as - /// `nvx.py sandbox run` and `start` check them. + /// `nvx.py sandbox run` and `start` checked them. pub(crate) fn require(&self) -> Result<[String; 3]> { let Located { dir, openvmm } = locate(self.root)?; require_artifact(&openvmm, "OpenVMM release binary")?; @@ -589,7 +553,7 @@ pub(crate) fn error_with_cause(error: crate::Error) -> Exit { } } -/// Runs the operation of `options`, as `command_sandbox` does. +/// Runs the operation of `options`, as `command_sandbox` did. pub(super) fn run(root: Option<&str>, options: Options) -> Result { let operation = options.operation; let checks = SandboxOptions { diff --git a/nvx-rs/src/host/cli/sandbox/tests.rs b/nvx-rs/src/host/cli/sandbox/tests.rs index 7135c8042..4e298d356 100644 --- a/nvx-rs/src/host/cli/sandbox/tests.rs +++ b/nvx-rs/src/host/cli/sandbox/tests.rs @@ -17,7 +17,7 @@ fn resource_limits_are_bounded_ascii_decimal() { ("--pids-max", "4194303"), ] { let arguments = [option.to_owned(), value.to_owned()]; - let (options, _) = parse(&arguments, "nvx-rs").unwrap(); + let (options, _) = parse(&arguments).unwrap(); let actual = if option == "--memory-max" { options.memory_max } else { @@ -39,10 +39,10 @@ fn resource_limits_are_bounded_ascii_decimal() { "18446744073709551616", ] { let arguments = [option.to_owned(), value.to_owned()]; - assert!(parse(&arguments, "nvx-rs").is_err(), "{option} {value:?}"); + assert!(parse(&arguments).is_err(), "{option} {value:?}"); } } - assert!(parse(&["--pids-max".to_owned(), "4194304".to_owned()], "nvx-rs").is_err()); + assert!(parse(&["--pids-max".to_owned(), "4194304".to_owned()]).is_err()); } proptest! { @@ -71,7 +71,7 @@ proptest! { path, }); } - let (options, unrecognized) = parse(&arguments, "nvx-rs").unwrap(); + let (options, unrecognized) = parse(&arguments).unwrap(); prop_assert!(unrecognized.is_empty()); prop_assert_eq!(options.mounts.len(), mounts); prop_assert_eq!(options.policy_paths, expected); diff --git a/nvx-rs/src/host/cli/session.rs b/nvx-rs/src/host/cli/session.rs index e79a584bf..8b7c14fa0 100644 --- a/nvx-rs/src/host/cli/session.rs +++ b/nvx-rs/src/host/cli/session.rs @@ -1,4 +1,4 @@ -//! The control session of `nvx.py sandbox`, a port of the former +//! The control session of `nvx-rs sandbox`, a port of the former //! `scripts/nvx_tools/control_session.py` on the framing, sequencing, attachment, and transports //! of [`crate::host::session`]. //! diff --git a/nvx-rs/src/host/mod.rs b/nvx-rs/src/host/mod.rs index 3e00d9d4f..d4543fe1e 100644 --- a/nvx-rs/src/host/mod.rs +++ b/nvx-rs/src/host/mod.rs @@ -26,15 +26,15 @@ //! - [`process`] names processes by verified references: their ID, start time, and host boot, //! through which a pidfd or a process handle holds them, so that waits sleep until a process //! exits and termination reaches only it, whether or not the caller started it. -//! - [`sandbox`] is the launch profile of `nvx.py sandbox`: a container over read-only EROFS +//! - [`sandbox`] is the launch profile of `nvx-rs sandbox`: a container over read-only EROFS //! layers and a private ext4 scratch disk, with live shares, and the checks of its options. //! [`sandbox::distro`] binds a distro layer to what explicit verification found, so that a //! launch checks the layer through [`seal`], which seals open files from their metadata alone, //! and [`sandbox::scratch`] binds and claims a scratch image. A //! [`PreparedLaunch`](sandbox::PreparedLaunch) hands the VMM the files that those checks //! opened, through the [`supervisor`]'s allowlist on Linux. -//! - `cli` (feature `cli`) is the `nvx-rs` command-line interface, which `nvx.py` runs for the -//! steps that it delegates: artifact lookup, egress policy compilation, and sandboxes. +//! - `cli` (feature `cli`) is the `nvx-rs` command-line interface, which runs microVMs and +//! sandboxes, looks up artifacts, and compiles egress policies. //! //! The ports of `nvx.py` code keep its Python semantics, which the private `python` module //! provides. @@ -42,7 +42,9 @@ //! The host modules deny unsafe code. On Unix, the private `sys` module confines the system calls //! that the standard library lacks behind documented safe functions, which call `nix`. The //! Windows platform module, the Windows module of [`seal`], and the module of the Win32 calls that -//! Python's filesystem queries make, still call `windows-sys` directly. +//! Python's filesystem queries make, still call `windows-sys` directly, and the CLI's identification +//! of the host CPU executes `CPUID` through its `core::arch` intrinsic, which the crate's minimum +//! Rust version still marks unsafe. #![deny(unsafe_code)] pub mod artifacts; diff --git a/nvx-rs/src/host/network/policy/mod.rs b/nvx-rs/src/host/network/policy/mod.rs index 1dce3ccd2..ddee7d5b0 100644 --- a/nvx-rs/src/host/network/policy/mod.rs +++ b/nvx-rs/src/host/network/policy/mod.rs @@ -1,4 +1,4 @@ -//! Egress policy files, which `nvx.py run` and `nvx.py sandbox` accept with +//! Egress policy files, which `nvx-rs run` and `nvx-rs sandbox` accept with //! `--network-egress-policy-file`, and their compilation into OpenVMM's native rules. //! //! The compiler replaced the former `scripts/nvx_tools/egress_policy.py` (microsoft/nvx#359): it diff --git a/nvx-rs/src/host/sandbox/layer.rs b/nvx-rs/src/host/sandbox/layer.rs index acbc0e69b..7b2191c14 100644 --- a/nvx-rs/src/host/sandbox/layer.rs +++ b/nvx-rs/src/host/sandbox/layer.rs @@ -9,7 +9,7 @@ use crate::error::Result; use crate::host::python::path::{Flavor, PurePath}; use crate::uuid::Uuid; -/// The role of a read-only layer of `nvx.py sandbox`. +/// The role of a read-only layer of `nvx-rs sandbox`. /// /// A sandbox takes exactly one layer, the distro layer, which its plan attaches as the `distro` /// [`BlockRole`](super::BlockRole). diff --git a/nvx-rs/src/host/sandbox/mod.rs b/nvx-rs/src/host/sandbox/mod.rs index b0081cc8c..4e2e72f20 100644 --- a/nvx-rs/src/host/sandbox/mod.rs +++ b/nvx-rs/src/host/sandbox/mod.rs @@ -1,7 +1,7 @@ //! The sandbox launch profile: a microVM that runs one workload in a container over a read-only //! distro image and a writable upper, with live virtio-fs shares. //! -//! This is the profile of `nvx.py sandbox`: its documented limits, option semantics, OpenVMM +//! This is the profile of `nvx-rs sandbox`: its documented limits, option semantics, OpenVMM //! arguments, and guest kernel tokens, with native Rust input validation and diagnostics: //! //! - [`plan`]: the typed, persistable [`SandboxPlan`] of a cold layered launch: one read-only @@ -14,18 +14,18 @@ //! [`protocol::root`](crate::protocol::root) defines. An adapter that runs its own guest plans //! the same root with [`SandboxPlan::adapter`], without the profile's workload settings. //! - [`SandboxLayer`]: the distro layer of a sandbox, the one [`LayerRole`] that a sandbox takes: -//! an EROFS filesystem, as `nvx.py sandbox` names it with `ROLE,PATH,EROFS_UUID`, or a GPT disk. +//! an EROFS filesystem, as `nvx-rs sandbox` names it with `ROLE,PATH,EROFS_UUID`, or a GPT disk. //! - [`SandboxMount`]: a requested live share of a host directory or regular file at a guest //! target, with its access mode, owner, and access-policy paths; [`mount`] holds the rules of //! guest targets and the checks of host paths, plans the shares of a sandbox through the planner //! of every NVX caller, [`host::filesystem`](crate::host::filesystem), and renders the plan as //! the guest's shell reads it at boot. -//! - [`SandboxLaunch`]: the launch of `nvx.py sandbox`, its layers over its scratch image with +//! - [`SandboxLaunch`]: the launch of `nvx-rs sandbox`, its layers over its scratch image with //! its settings and the export plan of its shares, which plans its distro layer, in the layer's //! format, over the scratch image. -//! - [`options`]: the checks that `nvx.py sandbox` and `nvx.py run` apply to their options: the +//! - [`options`]: the checks that `nvx-rs sandbox` and `nvx-rs run` apply to their options: the //! attribution of access-policy paths to shares, the systemd entrypoint check and the preflight -//! that verifies the distro layer, the targets of `nvx.py run` shares, and the managed exec +//! that verifies the distro layer, the targets of `nvx-rs run` shares, and the managed exec //! environment. //! - [`distro`]: the binding of the distro layer, which explicit verification makes from the //! layer and its manifest, and which a launch checks from metadata alone. @@ -207,7 +207,7 @@ fn reject_container_path(path: &PurePath, shown: &str) -> Result<()> { } /// The launch settings of a [`SandboxPlan`] or a [`SandboxLaunch`], besides its block devices and -/// its shares, with the defaults of `nvx.py sandbox`. +/// its shares, with the defaults of `nvx-rs sandbox`. #[derive(Clone, Debug, PartialEq, Eq)] pub struct SandboxSettings { /// The absolute path of the workload's program in the container, without whitespace. @@ -294,7 +294,7 @@ fn validate_settings(settings: &SandboxSettings) -> Result<()> { Ok(()) } -/// The launch of a sandbox of `nvx.py sandbox`: its read-only layers, its ext4 scratch image, its +/// The launch of a sandbox of `nvx-rs sandbox`: its read-only layers, its ext4 scratch image, its /// settings, the export plan of its shares, and its network, which it plans as its distro layer, /// in the layer's format, over the scratch image. /// diff --git a/nvx-rs/src/host/sandbox/options.rs b/nvx-rs/src/host/sandbox/options.rs index 6a0720181..e1ae7ebf3 100644 --- a/nvx-rs/src/host/sandbox/options.rs +++ b/nvx-rs/src/host/sandbox/options.rs @@ -1,6 +1,6 @@ -//! The contracts that `nvx.py sandbox` and `nvx.py run` apply to their options: +//! The contracts that `nvx-rs sandbox` and `nvx-rs run` apply to their options: //! -//! - [`SandboxOptions::check`]: the checks of `command_sandbox` before it builds the launch: the +//! - [`SandboxOptions::check`]: the checks of `nvx-rs sandbox` before it builds the launch: the //! operations that each option is valid for, the options that need others, and the managed exec //! environment, from `--environment` or `--environment-file`. //! - [`sandbox_mounts`]: the shares of the `--mount` options, with each `--mount-deny`, @@ -8,7 +8,7 @@ //! - [`validate_sandbox_entrypoint`]: the refusal of systemd entrypoints, and //! [`validate_sandbox_systemd_policy`]: the preflight of `run` and `provision`, which also //! verifies the distro layer and returns its [`DistroBinding`]. -//! - [`validate_run_share_targets`]: the guest targets of several `nvx.py run` shares, or of a +//! - [`validate_run_share_targets`]: the guest targets of several `nvx-rs run` shares, or of a //! shared file. //! - [`validate_exec_environment`] and [`exec_environment_from_file`]: the environment of a //! managed exec. @@ -36,7 +36,7 @@ pub const MAX_EXEC_ENVIRONMENT: usize = 256; /// The longest entry of a managed exec environment, in UTF-8 bytes. pub const MAX_EXEC_ENVIRONMENT_BYTES: usize = 4096; -/// An operation of `nvx.py sandbox`. +/// An operation of `nvx-rs sandbox`. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub enum SandboxOperation { /// `run`: a one-shot sandbox, the default. @@ -90,7 +90,7 @@ pub struct PolicyPathOption { pub path: String, } -/// The options of `nvx.py sandbox` that [`SandboxOptions::check`] examines, as given. +/// The options of `nvx-rs sandbox` that [`SandboxOptions::check`] examines, as given. #[derive(Clone, Debug, PartialEq, Eq)] pub struct SandboxOptions { /// The operation. @@ -131,7 +131,7 @@ pub struct SandboxOptions { } impl SandboxOptions { - /// Returns the options of an operation with the defaults of `nvx.py sandbox`. + /// Returns the options of an operation with the defaults of `nvx-rs sandbox`. pub fn new(operation: SandboxOperation) -> Self { Self { operation, @@ -153,7 +153,7 @@ impl SandboxOptions { } } - /// Applies the checks of `command_sandbox` that precede the launch, in its order, and + /// Applies the checks of `nvx-rs sandbox` that precede the launch, in its order, and /// returns the managed exec environment of an `exec`, if one was given. /// /// It refuses `--dry-run` except for `run`, both `--environment` and `--environment-file`, @@ -307,9 +307,9 @@ pub(crate) fn sandbox_mounts_with( .collect() } -/// Refuses the guest targets of `nvx.py run` shares that the guest cannot bind from an +/// Refuses the guest targets of `nvx-rs run` shares that the guest cannot bind from an /// aggregate: targets that would not survive the kernel command line, or that hide each other or -/// the aggregate itself. `nvx.py run` checks them when it attaches several shares, or a file. +/// the aggregate itself. `nvx-rs run` checks them when it attaches several shares, or a file. /// /// Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported). pub fn validate_run_share_targets(targets: &[&str]) -> Result<()> { @@ -351,7 +351,7 @@ pub fn validate_sandbox_entrypoint(entrypoint: &str) -> Result<()> { } /// Refuses a launch that would run systemd, and returns the binding of its distro layer: the -/// preflight of `nvx.py sandbox run` and `provision`. +/// preflight of `nvx-rs sandbox run` and `provision`. /// /// It refuses a systemd entrypoint with [`validate_sandbox_entrypoint`], and then verifies the /// distro layer explicitly with [`DistroBinding::verify`], which refuses a layer whose manifest, diff --git a/nvx-rs/src/host/sandbox/options/tests.rs b/nvx-rs/src/host/sandbox/options/tests.rs index 63abdcae3..b7e127201 100644 --- a/nvx-rs/src/host/sandbox/options/tests.rs +++ b/nvx-rs/src/host/sandbox/options/tests.rs @@ -858,7 +858,7 @@ impl Shares { path } - /// Runs the steps of `nvx.py sandbox --dry-run` that build the OpenVMM arguments and the + /// Runs the steps of `nvx-rs sandbox --dry-run` that build the OpenVMM arguments and the /// kernel command line of these shares. fn run( &self, diff --git a/nvx-rs/src/testing/fake_openvmm.rs b/nvx-rs/src/testing/fake_openvmm.rs index b453b1055..077cdf452 100644 --- a/nvx-rs/src/testing/fake_openvmm.rs +++ b/nvx-rs/src/testing/fake_openvmm.rs @@ -52,7 +52,7 @@ //! kernel, and each connection to the control endpoint that the fake accepts adds a line to //! `fake-openvmm--sessions.log` there, so that a test can tell whether a host connected. //! -//! The fake also serves the sandbox profile of `nvx.py sandbox` and `nvx-rs sandbox`, which a +//! The fake also serves the sandbox profile of `nvx-rs sandbox`, which a //! `--microvm-sandbox-block` option selects: read-only `distro`, `runtime`, and `custom` layers //! and a `scratch` disk, each an existing file, with `nvx_sandbox=1`, `nvx_entrypoint=`, and //! `nvx_hostname=` on the kernel command line. Like OpenVMM, it opens each block device at the @@ -74,7 +74,7 @@ //! of operation `run` if `--microvm-report` asks for one, and exits with the workload's status, //! or with `STATUS` if the kernel command line has `fake_exit_status=STATUS`. //! -//! The fake also runs the microVMs of `nvx.py run` and `nvx-rs run`, whose command lines have +//! The fake also runs the microVMs of `nvx-rs run`, whose command lines have //! neither a `--microvm-lifecycle` nor a sandbox block device, on the terminal. Such a command //! line takes `--processors` and at most one `--cpu-profile`, shares through one `--mount` or the //! children of an aggregate at `/run/nvx/shares`, whose relative policy paths name paths in the @@ -194,8 +194,8 @@ enum Lifecycle { /// `--microvm-lifecycle one-shot`: the guest runs its entrypoint once, and OpenVMM exits with /// its status. Only the sandbox profile uses it. OneShot, - /// Neither a lifecycle nor a sandbox block device: the microVM that `nvx.py run` and - /// `nvx-rs run` boot, or restore from a snapshot, on the terminal, which OpenVMM runs until + /// Neither a lifecycle nor a sandbox block device: the microVM that `nvx-rs run` + /// boots, or restores from a snapshot, on the terminal, which OpenVMM runs until /// the guest shuts down. Run, } @@ -206,7 +206,7 @@ struct Options { program: String, /// The lifecycle. lifecycle: Lifecycle, - /// Whether the command line launches the sandbox profile of `nvx.py sandbox`: a container + /// Whether the command line launches the sandbox profile of `nvx-rs sandbox`: a container /// over EROFS layers and an ext4 scratch disk. sandbox: bool, /// The snapshot directory that a run restores, or `None` for a cold boot. @@ -471,7 +471,7 @@ fn run_one_shot(options: &Options) -> i32 { status } -/// Boots or restores the microVM of a run of `nvx.py run` or `nvx-rs run`, as OpenVMM runs it on +/// Boots or restores the microVM of a run of `nvx-rs run`, as OpenVMM runs it on /// the terminal, and returns the status with which OpenVMM exits. /// /// No guest runs. After the `fake_boot_delay_ms` delay, a cold boot prints `PROGRAM: booted the @@ -547,7 +547,7 @@ fn parse(program: &str, arguments: &[String]) -> Result { "--host-loopback", "--host-loopback-forward", ]; - // The options that only the microVMs of `nvx.py run` take. + // The options that only the microVMs of `nvx-rs run` take. const RUN_VALUED: [&str; 7] = [ "--processors", "--cpu-profile", @@ -557,7 +557,7 @@ fn parse(program: &str, arguments: &[String]) -> Result { "--restore-memory", "--restore-ready-path", ]; - // `nvx.py run` boots or restores its microVM without a lifecycle, and is no sandbox. + // `nvx-rs run` boots or restores its microVM without a lifecycle, and is no sandbox. let mut option_arguments = arguments.iter(); let mut run = true; while let Some(argument) = option_arguments.next() { @@ -595,7 +595,7 @@ fn parse(program: &str, arguments: &[String]) -> Result { } }; expect("--machine", "microvm")?; - // The sandbox profile of `nvx.py sandbox` puts its layers and scratch disk on sandbox block + // The sandbox profile of `nvx-rs sandbox` puts its layers and scratch disk on sandbox block // devices, and may run one-shot. let sandbox = values.contains_key("--microvm-sandbox-block"); let lifecycle = if run { @@ -889,7 +889,7 @@ fn check_boot(values: &BTreeMap<&str, Vec<&str>>) -> Result<(u32, PathBuf), Stri Ok((workload_uid, kernel)) } -/// Checks how a run of `nvx.py run` starts its microVM, and returns the path next to which the +/// Checks how a run of `nvx-rs run` starts its microVM, and returns the path next to which the /// fake records the command line, and the snapshot directory of a restore. /// /// A run takes `--processors`, a positive count, and at most one `--cpu-profile`. A cold boot diff --git a/nvx-rs/tests/cli.rs b/nvx-rs/tests/cli.rs index 85940bb4d..c4333aabc 100644 --- a/nvx-rs/tests/cli.rs +++ b/nvx-rs/tests/cli.rs @@ -2250,8 +2250,7 @@ fn a_relative_openvmm_override_is_relative_to_the_working_directory() { #[test] fn top_level_usage_errors_exit_with_status_2() { - let usage = "usage: nvx-rs [-h] [--root PATH] [--prog PROG]\n \ - {artifacts,egress,run,sandbox} ...\n"; + let usage = "usage: nvx-rs [-h] [--root PATH] {artifacts,egress,run,sandbox} ...\n"; for (arguments, message) in [ ( &[][..], @@ -2270,6 +2269,16 @@ fn top_level_usage_errors_exit_with_status_2() { &["artifacts", "--bogus"][..], "nvx-rs: error[argument-unrecognized]: unrecognized arguments: --bogus\n", ), + // No option names the program any more. + ( + &["--prog", "nvx.py", "sandbox"][..], + "nvx-rs: error[argument-invalid]: argument command: invalid choice: 'nvx.py' \ + (choose from 'artifacts', 'egress', 'run', 'sandbox')\n", + ), + ( + &["--prog=nvx.py", "artifacts"][..], + "nvx-rs: error[argument-unrecognized]: unrecognized arguments: --prog=nvx.py\n", + ), ] { let output = Command::new(NVX_RS).args(arguments).output().unwrap(); assert_eq!(output.status.code(), Some(2), "{arguments:?}"); @@ -2292,151 +2301,218 @@ fn top_level_usage_errors_exit_with_status_2() { ); } -/// What `nvx.py sandbox` printed for a refused `--workload-user` before it ran `nvx-rs`, as -/// CPython 3.14 prints it for 80 columns: the usage of its parser, wrapped after its name. -const NVX_PY_SANDBOX_USAGE_ERROR: &str = "\ -usage: nvx.py sandbox [-h] [--layer ROLE,PATH,EROFS_UUID] [--scratch SCRATCH] - [--state-dir STATE_DIR] [--entrypoint ENTRYPOINT] - [--arg SANDBOX_ARG] [--hostname HOSTNAME] - [--workload-user UID:GID] [--memory-max MEMORY_MAX] - [--pids-max PIDS_MAX] [--memory-mib MEMORY_MIB] - [--timeout TIMEOUT] [--exec-timeout-ms EXEC_TIMEOUT_MS] - [--cwd CWD] [--environment KEY=VALUE] - [--environment-file PATH] - [--inherit-default-environment] - [--hypervisor {auto,whp,kvm,mshv}] - [--mount GUEST_TARGET,HOST_PATH[,ro|rw]] - [--mount-deny HOST_PATH] [--mount-allow HOST_PATH] - [--mount-write HOST_PATH] [--mount-owner {vmm,caller}] - [--net IPV4/PREFIX] [--network-profile {portable}] - [--network-egress {allow,deny}] - [--network-ingress {allow,deny}] - [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]] - [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]] - [--network-egress-policy-file PATH] - [--host-loopback {allow,deny}] - [--network-proxy IPV4:TCP-PORT] - [--host-loopback-forward HOST_LOOPBACK_FORWARD] - [--outcome-report OUTCOME_REPORT] [--cmdline CMDLINE] - [--dry-run] - [{run,provision,start,exec,stop,deprovision}] -nvx.py sandbox: error[argument-invalid]: argument --workload-user: --workload-user UID must be \ -between 1 and 4294967294 -"; - #[test] -fn messages_name_the_program_that_prog_names() { - // nvx.py passes its own name, so that usage errors read as they did before it delegated. - let output = Command::new(NVX_RS) - .args(["--prog", "nvx.py", "sandbox", "--workload-user", "0:0"]) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(2)); - assert_eq!(text(&output.stderr), NVX_PY_SANDBOX_USAGE_ERROR); - assert!(output.stdout.is_empty()); - // The usage wraps for the name as argparse wraps it: a long name takes a line of its own. - let long = "a-program-whose-name-is-long-enough-to-take-its-own-line.py"; - let output = Command::new(NVX_RS) - .args(["--prog", long, "egress", "compile"]) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(2)); - assert_eq!( - text(&output.stderr), - format!( - "usage: {long} egress compile\n [-h] PATH\n\ - {long} egress compile: error[argument-missing]: the following arguments are \ - required: PATH\n" - ) - ); - // The top-level parser names it as soon as it has read it, and so does the help. - let output = Command::new(NVX_RS) - .args(["--prog=nvx.py", "build"]) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(2)); - assert_eq!( - text(&output.stderr), - "usage: nvx.py [-h] [--root PATH] [--prog PROG]\n \ - {artifacts,egress,run,sandbox} ...\n\ - nvx.py: error[argument-invalid]: argument command: invalid choice: 'build' (choose from \ - 'artifacts', 'egress', 'run', 'sandbox')\n" - ); +fn usage_errors_print_the_one_line_usage_of_their_parser() { + for (arguments, expected) in [ + ( + &["sandbox", "--workload-user", "0:0"][..], + "usage: nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] [OPTIONS]\n\ + nvx-rs sandbox: error[argument-invalid]: argument --workload-user: --workload-user \ + UID must be between 1 and 4294967294\n", + ), + ( + &["run", "--processors", "3"][..], + "usage: nvx-rs run [OPTIONS]\n\ + nvx-rs run: error[argument-invalid]: argument --processors: invalid choice: '3' \ + (choose from '1', '2', '4', '8')\n", + ), + ] { + let output = Command::new(NVX_RS).args(arguments).output().unwrap(); + assert_eq!(output.status.code(), Some(2), "{arguments:?}"); + assert_eq!(text(&output.stderr), expected, "{arguments:?}"); + assert!(output.stdout.is_empty(), "{arguments:?}"); + } + // The help starts with the same usage and lists the options after it. let output = Command::new(NVX_RS) - .args(["--prog", "nvx.py", "artifacts", "--help"]) + .args(["artifacts", "--help"]) .output() .unwrap(); assert!(output.status.success()); - assert!(text(&output.stdout).starts_with("usage: nvx.py artifacts [-h]\n\nPrint the paths")); -} - -#[test] -fn program_names_that_could_hide_the_code_of_an_error_are_refused() { - // A usage error names the program before its code, so a name that holds a code's brackets, or - // a line end, could change the code that callers read: it is refused under the default name. - for program in [ - "error[io]: wrapper", - "wrapper: error[io]: alias", - "two\nlines", - ] { - let output = Command::new(NVX_RS) - .args(["--prog", program, "egress", "compile"]) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(2), "{program:?}"); - assert_eq!( - text(&output.stderr), - "usage: nvx-rs [-h] [--root PATH] [--prog PROG]\n \ - {artifacts,egress,run,sandbox} ...\n\ - nvx-rs: error[argument-invalid]: argument --prog: the program name may hold no \ - square brackets or control characters\n", - "{program:?}" + assert!(text(&output.stdout).starts_with("usage: nvx-rs artifacts [-h]\n\nPrint the paths")); +} + +/// The host CPU as the host OS reports it, independently of the CPUID that `nvx-rs` reads: its +/// vendor and display family, model, and stepping, from the first processor of `/proc/cpuinfo` on +/// Linux, and from `PROCESSOR_IDENTIFIER`, such as `Intel64 Family 6 Model 85 Stepping 4, +/// GenuineIntel`, on Windows. +fn reported_cpu() -> (String, u32, u32, u32) { + if cfg!(windows) { + let identifier = std::env::var("PROCESSOR_IDENTIFIER").unwrap(); + let (description, vendor) = identifier.split_once(", ").unwrap(); + let words: Vec<&str> = description.split_whitespace().collect(); + let number = |name: &str| -> u32 { + let position = words.iter().position(|word| *word == name).unwrap(); + words[position + 1].parse().unwrap() + }; + return ( + vendor.trim().to_owned(), + number("Family"), + number("Model"), + number("Stepping"), ); } + let cpuinfo = fs::read_to_string("/proc/cpuinfo").unwrap(); + let field = |name: &str| -> String { + cpuinfo + .lines() + .find_map(|line| { + let (key, value) = line.split_once(':')?; + (key.trim() == name).then(|| value.trim().to_owned()) + }) + .unwrap() + }; + ( + field("vendor_id"), + field("cpu family").parse().unwrap(), + field("model").parse().unwrap(), + field("stepping").parse().unwrap(), + ) } -/// The CPU-profile guidance that `nvx.py` passes to `nvx-rs run` on a host whose CPU no built-in -/// profile serves, in two lines. -const GUIDANCE: &str = "nvx: no built-in CPU profile serves this host's CPU\nnvx: see doc/usage.md"; - -/// Returns the arguments of `nvx-rs run` with the guidance of `nvx.py`, then `arguments`. -fn guided_run(arguments: &[&str]) -> Vec { - ["run", &format!("--cold-boot-guidance={GUIDANCE}")] - .into_iter() - .chain(arguments.iter().copied()) - .map(str::to_owned) - .collect() +/// Returns whether a built-in CPU profile serves the CPU of `vendor` with the display `family`, +/// `model`, and `stepping`, by the copy of OpenVMM's catalog that `nvx-rs` reads. +fn served(vendor: &str, family: u32, model: u32, stepping: u32) -> bool { + let catalog: Value = + serde_json::from_str(include_str!("../src/host/cli/cpu_generations.json")).unwrap(); + catalog.as_array().unwrap().iter().any(|profile| { + profile["vendor"] == vendor + && profile["generation"]["cpus"] + .as_array() + .unwrap() + .iter() + .any(|cpu| { + let steppings = cpu["steppings"].as_array().unwrap(); + cpu["family"] == family + && cpu["model"] == model + && steppings[0].as_u64().unwrap() <= u64::from(stepping) + && u64::from(stepping) <= steppings[1].as_u64().unwrap() + }) + }) } #[test] -fn run_prints_its_guidance_only_after_a_failed_cold_boot() { +fn run_guides_failed_cold_boots_on_cpus_that_no_built_in_profile_serves() { + // nvx-rs run identifies the host's CPU with CPUID, as OpenVMM does and as the host OS reports + // it, and guides a failed cold boot only where no built-in profile serves the CPU: what + // follows OpenVMM's output depends on the host that runs the test. + let (vendor, family, model, stepping) = reported_cpu(); + let unserved = !served(&vendor, family, model, stepping); + let host_profiles = ["GenuineIntel", "AuthenticAMD"].contains(&vendor.as_str()); let root = Root::new(); - let snapshot = root.path.join("snapshot"); - fs::create_dir(&snapshot).unwrap(); - let snapshot = snapshot.to_str().unwrap(); + let arguments = |arguments: &[&str]| -> Vec { + std::iter::once("run") + .chain(arguments.iter().copied()) + .map(str::to_owned) + .collect() + }; + // The guidance asks for a fingerprint from the OpenVMM that the run ran, as the run quoted it. + let dry_run = text(&run(&root, &arguments(&["--dry-run"])).stdout); + let (openvmm, _) = dry_run + .strip_prefix(">> ") + .and_then(|command| command.split_once(" --single-process ")) + .unwrap(); + let default = if cfg!(windows) { "whp" } else { "kvm" }; // OpenVMM fails the cold boot, or refuses its command line with the status of a usage error; - // the guidance follows what OpenVMM printed. - for (arguments, status, stderr) in [ - (&["--cmdline", "fake_exit_status=1"][..], 1, ""), - (&["--cmdline", "fake_exit_status=2"][..], 2, ""), + // the guidance follows what OpenVMM printed, for auto, the default, on every unserved CPU, and + // for host on a CPU of a vendor that host profiles do not serve either. + for (options, status, printed, guided, hypervisor) in [ + ( + &["--cmdline", "fake_exit_status=1"][..], + 1, + "", + unserved, + default, + ), + ( + &["--cmdline", "fake_exit_status=2"][..], + 2, + "", + unserved, + default, + ), ( &["--host-loopback-forward", "tcp:8080:80"][..], 2, "openvmm: --host-loopback-forward requires explicit --host-loopback allow\n", + unserved, + default, + ), + ( + &[ + "--cpu-profile", + "auto", + "--hypervisor", + "mshv", + "--cmdline", + "fake_exit_status=1", + ][..], + 1, + "", + unserved, + "mshv", + ), + ( + &["--cpu-profile", "host", "--cmdline", "fake_exit_status=1"][..], + 1, + "", + unserved && !host_profiles, + default, + ), + ( + &[ + "--cpu-profile", + "intel.alderlake.v1", + "--cmdline", + "fake_exit_status=1", + ][..], + 1, + "", + false, + default, ), ] { - let output = run(&root, &guided_run(arguments)); - assert_eq!(output.status.code(), Some(status), "{arguments:?}"); + let output = run(&root, &arguments(options)); + assert_eq!(output.status.code(), Some(status), "{options:?}"); + assert!(text(&output.stdout).starts_with(">> "), "{options:?}"); + let stderr = text(&output.stderr); + let guidance = stderr + .strip_prefix(printed) + .unwrap_or_else(|| panic!("{options:?}: {stderr}")); + if !guided { + assert_eq!(guidance, "", "{options:?}"); + continue; + } + assert!( + guidance.starts_with(&format!( + "nvx: no built-in CPU profile serves this host's CPU, {vendor} \ + {family}/{model}/{stepping}" + )), + "{options:?}: {guidance}" + ); + assert!( + guidance.ends_with('\n') && guidance.lines().all(|line| line.starts_with("nvx: ")), + "{options:?}: {guidance}" + ); + let fingerprint = format!( + "\nnvx: {openvmm} --hypervisor {hypervisor} --cpu-fingerprint fingerprint.json\n" + ); assert_eq!( - text(&output.stderr), - format!("{stderr}{GUIDANCE}\n"), - "{arguments:?}" + guidance.contains(&fingerprint), + host_profiles, + "{options:?}: {guidance}" + ); + assert_eq!( + guidance.contains("E_PROFILE_HOST_UNKNOWN"), + !host_profiles, + "{options:?}: {guidance}" ); - assert!(text(&output.stdout).starts_with(">> "), "{arguments:?}"); } // A cold boot that succeeds, a dry run, the restores that fail, the options that nvx-rs // refuses, its usage errors, and the help print none. - for (arguments, status) in [ + let snapshot = root.path.join("snapshot"); + fs::create_dir(&snapshot).unwrap(); + let snapshot = snapshot.to_str().unwrap(); + for (options, status) in [ (&[][..], 0), (&["--cmdline", "fake_exit_status=1", "--dry-run"][..], 0), ( @@ -2458,28 +2534,24 @@ fn run_prints_its_guidance_only_after_a_failed_cold_boot() { (&["--bogus"][..], 2), (&["--help"][..], 0), ] { - let output = run(&root, &guided_run(arguments)); - assert_eq!(output.status.code(), Some(status), "{arguments:?}"); + let output = run(&root, &arguments(options)); + assert_eq!(output.status.code(), Some(status), "{options:?}"); assert!( - !text(&output.stderr).contains(GUIDANCE), - "{arguments:?}: {}", + !text(&output.stderr).contains("nvx: "), + "{options:?}: {}", text(&output.stderr) ); - assert!( - !text(&output.stdout).contains("cold-boot-guidance"), - "{arguments:?}" - ); } // Nor does OpenVMM that does not start. let environment = BTreeMap::from([( "NVX_OPENVMM".to_owned(), root.path.join("single.txt").to_str().unwrap().to_owned(), )]); - let output = nvx_rs(&root, &guided_run(&[]), &environment); + let output = nvx_rs(&root, &arguments(&[]), &environment); assert_eq!(output.status.code(), Some(1)); let stderr = text(&output.stderr); assert!( - stderr.starts_with("error[io]: ") && !stderr.contains(GUIDANCE), + stderr.starts_with("error[io]: ") && !stderr.contains("nvx: "), "{stderr}" ); } @@ -2537,41 +2609,6 @@ fn run_shares_relative_roots_with_absolute_policy_paths() { ); } -#[test] -fn run_messages_name_the_program_that_prog_names() { - let output = Command::new(NVX_RS) - .args(["--prog", "nvx.py", "run", "--processors", "3"]) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(2)); - let stderr = text(&output.stderr); - assert!( - stderr.starts_with( - "usage: nvx.py run [-h] [--guest {alpine,ubuntu,azurelinux}]\n \ - [--hypervisor {auto,whp,kvm,mshv}] [--machine {microvm}]\n" - ), - "{stderr}" - ); - assert!( - stderr.ends_with( - " [--restore-ready-path RESTORE_READY_PATH] [--dry-run]\n\ - nvx.py run: error[argument-invalid]: argument --processors: invalid choice: '3' \ - (choose from '1', '2', '4', '8')\n" - ), - "{stderr}" - ); - // The help shows the options of nvx.py run, without the guidance that nvx.py passes. - let output = Command::new(NVX_RS) - .args(["--prog", "nvx.py", "run", "--help"]) - .output() - .unwrap(); - assert!(output.status.success()); - let stdout = text(&output.stdout); - assert!(stdout.starts_with("usage: nvx.py run [-h]"), "{stdout}"); - assert!(stdout.ends_with("\n --dry-run\n"), "{stdout}"); - assert!(!stdout.contains("guidance"), "{stdout}"); -} - #[cfg(unix)] mod interrupts { use std::time::Instant; @@ -2661,9 +2698,9 @@ mod interrupts { fn interrupting_a_cold_boot_ends_it_without_guidance() { // The fake OpenVMM delays its boot, which Ctrl-C ends, alone or with OpenVMM, as the // terminal's does; nvx-rs run then exits as interrupted, without the guidance that it - // prints after a failed cold boot. + // prints after a failed cold boot on a CPU that no built-in profile serves. let root = Root::new(); - let arguments = guided_run(&["--cmdline", "fake_boot_delay_ms=60000"]); + let arguments = ["run", "--cmdline", "fake_boot_delay_ms=60000"].map(str::to_owned); let started = root.path.join("build").join("fake-openvmm-.json"); for group in [false, true] { let _ = fs::remove_file(&started); @@ -2812,6 +2849,11 @@ mod interrupts { #[test] fn interrupting_help_ends_its_write_to_a_full_pipe() { // Help also ends as interrupted, as `KeyboardInterrupt` ends Python's. + let help = Command::new(NVX_RS) + .args(["sandbox", "--help"]) + .output() + .unwrap(); + assert!(help.stdout.len() > 4096, "the help fits in one page"); let (reader, writer) = std::io::pipe().unwrap(); // A pipe of one page, which the help, longer than that, fills. nix::fcntl::fcntl(&writer, nix::fcntl::FcntlArg::F_SETPIPE_SZ(4096)).unwrap(); diff --git a/nvx-rs/tests/cli/one_shot.rs b/nvx-rs/tests/cli/one_shot.rs index 2f82567b2..44a4713de 100644 --- a/nvx-rs/tests/cli/one_shot.rs +++ b/nvx-rs/tests/cli/one_shot.rs @@ -151,6 +151,20 @@ fn workload(user: &str, entrypoint: &str, arguments: &[&str]) -> Options { one_shot().sandbox_kernel(user, &tokens) } +/// Returns whether the launch is a cold boot of `nvx-rs run` that fails, which `run` follows with +/// its CPU-profile guidance on a host whose CPU no built-in profile serves, as +/// `run_guides_failed_cold_boots_on_cpus_that_no_built_in_profile_serves` checks. +fn fails_a_cold_boot(case: &Launch) -> bool { + case.arguments + .first() + .is_some_and(|command| command == "run") + && case.status != 0 + && !case + .arguments + .iter() + .any(|argument| argument == "--restore-snapshot") +} + /// Returns `nvx-rs run OPTIONS`. fn run(options: &[&str]) -> Vec { ["run"] @@ -540,6 +554,8 @@ fn one_shot_runs_exit_with_the_workload_or_openvmm_status() { .into_iter() .chain(run_launches()) .collect(); + let (vendor, family, model, stepping) = crate::reported_cpu(); + let unserved = !crate::served(&vendor, family, model, stepping); in_parallel(&cases, |case| { let root = Root::new(); for (relative, content) in &case.files { @@ -560,12 +576,19 @@ fn one_shot_runs_exit_with_the_workload_or_openvmm_status() { let (command, rest) = outcome.command(); // The workload's output follows the command byte for byte, as does OpenVMM's. assert_eq!(rest, case.stdout.as_bytes(), "{}", outcome.describe()); - assert_eq!( - outcome.stderr(), - root.expand(case.stderr), - "{}", - outcome.describe() - ); + let stderr = outcome.stderr(); + let guidance = stderr + .strip_prefix(&root.expand(case.stderr)) + .unwrap_or_else(|| panic!("{}", outcome.describe())); + if unserved && fails_a_cold_boot(case) { + assert!( + guidance.starts_with("nvx: no built-in CPU profile serves this host's CPU"), + "{}", + outcome.describe() + ); + } else { + assert_eq!(guidance, "", "{}", outcome.describe()); + } let (program, arguments) = command.split_first().unwrap(); assert_eq!( *program, diff --git a/nvx-rs/tests/cli/usage.rs b/nvx-rs/tests/cli/usage.rs index 1cab501b4..0ae15979a 100644 --- a/nvx-rs/tests/cli/usage.rs +++ b/nvx-rs/tests/cli/usage.rs @@ -2,10 +2,6 @@ use crate::harness::{LAYER, LINE_END, Root, SCRATCH}; -/// The help of `nvx-rs run`, which `nvx.py run` prints too, since it parses the options of -/// `nvx-rs run` before it runs it: a test of `nvx.py` checks its help against this one. -const RUN_HELP: &str = include_str!("../fixtures/run-help.txt"); - /// A command line that a parser of `nvx-rs` refuses. struct UsageError { /// The name of the case. @@ -474,7 +470,7 @@ struct Help { name: &'static str, /// The arguments after `--root ROOT`. arguments: &'static [&'static str], - /// How its usage starts. + /// Its usage, the first line of the help. usage: &'static str, /// Options that it describes. options: &'static [&'static str], @@ -486,7 +482,7 @@ const HELP: [Help; 5] = [ Help { name: "sandbox", arguments: &["sandbox", "-h"], - usage: "usage: nvx-rs sandbox [-h]", + usage: "usage: nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] [OPTIONS]", options: &[ "--layer ROLE,PATH,EROFS_UUID", "--state-dir STATE_DIR", @@ -496,13 +492,13 @@ const HELP: [Help; 5] = [ Help { name: "sandbox after an operation", arguments: &["sandbox", "exec", "--help"], - usage: "usage: nvx-rs sandbox [-h]", + usage: "usage: nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] [OPTIONS]", options: &["--environment KEY=VALUE", "--outcome-report OUTCOME_REPORT"], }, Help { name: "run", arguments: &["run", "-h"], - usage: "usage: nvx-rs run [-h]", + usage: "usage: nvx-rs run [OPTIONS]", options: &[ "--guest {alpine,ubuntu,azurelinux}", "--restore-snapshot", @@ -512,13 +508,13 @@ const HELP: [Help; 5] = [ Help { name: "run after options", arguments: &["run", "--guest", "ubuntu", "--help"], - usage: "usage: nvx-rs run [-h]", + usage: "usage: nvx-rs run [OPTIONS]", options: &["--memory-mib", "--processors"], }, Help { name: "run before an invalid value", arguments: &["run", "--help", "--memory-mib", "x"], - usage: "usage: nvx-rs run [-h]", + usage: "usage: nvx-rs run [OPTIONS]", options: &["--memory-mib"], }, ]; @@ -530,7 +526,11 @@ fn help_prints_the_usage_and_the_options_and_exits_0() { let outcome = root.nvx_rs(case.arguments); outcome.assert_success(); let help = outcome.stdout(); - assert!(help.starts_with(case.usage), "{}: {help}", case.name); + assert!( + help.starts_with(&format!("{}{LINE_END}{LINE_END}", case.usage)), + "{}: {help}", + case.name + ); assert!( help.contains("\n -h, --help ") && help.ends_with(LINE_END), "{}: {help}", @@ -545,10 +545,3 @@ fn help_prints_the_usage_and_the_options_and_exits_0() { } } } - -#[test] -fn run_prints_the_help_of_nvx_py_run() { - let outcome = Root::new().nvx_rs(&["run", "--help"]); - outcome.assert_success(); - assert_eq!(outcome.stdout(), RUN_HELP.replace('\n', LINE_END)); -} diff --git a/nvx-rs/tests/fake_openvmm.rs b/nvx-rs/tests/fake_openvmm.rs index 54980df34..a07217991 100644 --- a/nvx-rs/tests/fake_openvmm.rs +++ b/nvx-rs/tests/fake_openvmm.rs @@ -538,7 +538,7 @@ fn records_that_name_another_process_never_end_it() { } } -/// Runs the fake in `directory` as `nvx.py run` runs OpenVMM on the terminal: with two +/// Runs the fake in `directory` as `nvx-rs run` runs OpenVMM on the terminal: with two /// processors and the backend of this host, and then `arguments`. fn run_microvm(directory: &Path, arguments: &[&str]) -> std::process::Output { let hypervisor = Hypervisor::platform_default().expect("tests run on Linux or Windows"); @@ -645,8 +645,8 @@ fn runs_boot_or_restore_a_microvm_on_the_terminal() { #[test] fn runs_find_absolute_policy_paths_in_relative_share_roots() { - // nvx.py run and nvx-rs run pass the host path of each share as given, which may be relative - // to the working directory, and with several shares, absolute policy paths, which OpenVMM + // nvx-rs run passes the host path of each share as given, which may be relative to the + // working directory, and with several shares, absolute policy paths, which OpenVMM // attributes to the share whose root contains them once it has resolved both. let directory = tempfile::tempdir().unwrap(); let root = directory.path(); diff --git a/nvx-rs/tests/fixtures/run-help.txt b/nvx-rs/tests/fixtures/run-help.txt deleted file mode 100644 index 09ce1d629..000000000 --- a/nvx-rs/tests/fixtures/run-help.txt +++ /dev/null @@ -1,83 +0,0 @@ -usage: nvx-rs run [-h] [--guest {alpine,ubuntu,azurelinux}] - [--hypervisor {auto,whp,kvm,mshv}] [--machine {microvm}] - [--memory-mib MEMORY_MIB] - [--memory-capacity-mib MEMORY_CAPACITY_MIB] - [--processors {1,2,4,8}] [--cpu-profile ID] - [--mount GUEST_TARGET,HOST_PATH[,ro|rw]] - [--mount-deny MOUNT_DENY] [--mount-allow HOST_PATH] - [--mount-write HOST_PATH] [--mount-owner {vmm,caller}] - [--net IPV4/PREFIX] [--network-profile {portable}] - [--network-egress {allow,deny}] - [--network-ingress {allow,deny}] - [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]] - [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]] - [--network-egress-policy-file PATH] - [--host-loopback {allow,deny}] - [--network-proxy IPV4:TCP-PORT] - [--host-loopback-forward HOST_LOOPBACK_FORWARD] - [--outcome-report OUTCOME_REPORT] [--cmdline CMDLINE] - [--restore-snapshot RESTORE_SNAPSHOT] - [--restore-processors {1,2,4,8}] - [--restore-memory-mib RESTORE_MEMORY_MIB] - [--restore-ready-path RESTORE_READY_PATH] [--dry-run] - -options: - -h, --help show this help message and exit - --guest {alpine,ubuntu,azurelinux} - --hypervisor {auto,whp,kvm,mshv} - --machine {microvm} - --memory-mib MEMORY_MIB - guest RAM in MiB (default by --guest: alpine 128, - ubuntu 512, azurelinux 512) - --memory-capacity-mib MEMORY_CAPACITY_MIB - --processors {1,2,4,8} - --cpu-profile ID OpenVMM CPU profile: auto (OpenVMM's default) for the - built-in profile of the host's CPU, a built-in profile - ID, or host to derive a development profile from this - host (doc/usage.md) - --mount GUEST_TARGET,HOST_PATH[,ro|rw] - live-share a host directory, or a single regular host - file without its directory; the mode follows the last - comma, so HOST_PATH may contain commas when the mode - is given; repeat for more shares, each with its own - guest target and mode, as children of one virtio-fs - device - --mount-deny MOUNT_DENY - --mount-allow HOST_PATH - expose one existing path inside a --mount-deny path - again - --mount-write HOST_PATH - make one existing path one of the only writable parts - of a read-write --mount - --mount-owner {vmm,caller} - host identity of the share's file operations: vmm - (default) or caller, with guest root squashed to the - owner of the shared directory or file (Linux only) - --net IPV4/PREFIX - --network-profile {portable} - --network-egress {allow,deny} - --network-ingress {allow,deny} - --network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]] - allow guest egress to an IPv4 or IPv6 CIDR, optionally - only for PROTOCOL tcp, udp, or icmp, and for tcp or - udp only on PORT or on the inclusive range PORT-PORT; - repeat to add rules - --network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]] - deny guest egress to an IPv4 or IPv6 CIDR, optionally - only for PROTOCOL tcp, udp, or icmp, and for tcp or - udp only on PORT or on the inclusive range PORT-PORT; - repeat to add rules - --network-egress-policy-file PATH - load bounded MXC-shaped IPv4 and IPv6 egress rules - from JSON - --host-loopback {allow,deny} - --network-proxy IPV4:TCP-PORT - --host-loopback-forward HOST_LOOPBACK_FORWARD - --outcome-report OUTCOME_REPORT - write a bounded local JSON outcome report - --cmdline CMDLINE - --restore-snapshot RESTORE_SNAPSHOT - --restore-processors {1,2,4,8} - --restore-memory-mib RESTORE_MEMORY_MIB - --restore-ready-path RESTORE_READY_PATH - --dry-run diff --git a/nvx-rs/tests/openvmm_cpu_profiles.rs b/nvx-rs/tests/openvmm_cpu_profiles.rs new file mode 100644 index 000000000..0e18a8886 --- /dev/null +++ b/nvx-rs/tests/openvmm_cpu_profiles.rs @@ -0,0 +1,93 @@ +//! Compares `src/host/cli/cpu_generations.json`, the copy of the identities of OpenVMM's built-in +//! CPU profiles from which `nvx-rs run` guides a failed cold boot, with the profile documents of +//! the OpenVMM submodule's `vmm_core/cpu_profile/profiles/`. +//! +//! `nvx-rs` reads the copy, because the `nvx-rs` job cannot check out the submodule, whose deploy +//! key fork pull requests lack. This test is ignored by default: the NVX CLI jobs, which check out +//! the submodule, run it, and so does an OpenVMM pin promotion. + +use std::collections::BTreeMap; +use std::fs; +use std::path::Path; + +use serde_json::{Value, json}; + +/// Returns the identity of the profile `document`, with the fields that the copy holds: its ID, +/// its vendor, and its generation's name and CPU models. +fn identity(document: &Value) -> Value { + let generation = &document["generation"]; + let cpus: Vec = generation["cpus"] + .as_array() + .unwrap_or_else(|| panic!("{document} names no CPU models")) + .iter() + .map(|cpu| { + json!({ + "family": cpu["family"], + "model": cpu["model"], + "steppings": cpu["steppings"], + }) + }) + .collect(); + json!({ + "id": document["id"], + "vendor": document["vendor"], + "generation": {"name": generation["name"], "cpus": cpus}, + }) +} + +/// Returns `profiles` by their IDs, failing on an ID that two of them share. +fn by_id(profiles: impl IntoIterator) -> BTreeMap { + let mut identities = BTreeMap::new(); + for profile in profiles { + let id = profile["id"].as_str().unwrap().to_owned(); + assert!( + identities.insert(id.clone(), profile).is_none(), + "{id} is listed twice" + ); + } + identities +} + +#[test] +#[ignore = "needs the OpenVMM submodule, which the NVX CLI jobs check out"] +fn the_copy_matches_the_submodules_profiles() { + let crate_root = Path::new(env!("CARGO_MANIFEST_DIR")); + let directory = crate_root.join("../openvmm/vmm_core/cpu_profile/profiles"); + let entries = fs::read_dir(&directory).unwrap_or_else(|error| { + panic!( + "cannot read the CPU profiles of the OpenVMM submodule, which `scripts/nvx.py init` \ + checks out: {error}" + ) + }); + let mut pinned = Vec::new(); + for entry in entries { + let path = entry.unwrap().path(); + if path + .extension() + .is_some_and(|extension| extension == "json") + { + let document: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + pinned.push(identity(&document)); + } + } + let copy: Vec = serde_json::from_slice( + &fs::read(crate_root.join("src/host/cli/cpu_generations.json")).unwrap(), + ) + .unwrap(); + // The copy holds nothing but the identities, which nvx-rs reads with these fields alone. + assert_eq!( + copy, + copy.iter().map(identity).collect::>(), + "src/host/cli/cpu_generations.json holds other fields" + ); + assert!( + !pinned.is_empty(), + "the OpenVMM submodule pins no CPU profile" + ); + assert!( + by_id(copy) == by_id(pinned), + "src/host/cli/cpu_generations.json differs from the CPU profiles that the OpenVMM \ + submodule pins: copy the ID, the vendor, and the generation of each profile, and keep \ + the order in which nvx-rs run lists the generations" + ); +} diff --git a/scripts/nvx.py b/scripts/nvx.py index ce4e12b7f..dbf731c19 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Build, run, benchmark, and package the OpenVMM/NVX distribution.""" +"""Build, test, benchmark, and package the OpenVMM/NVX distribution.""" from __future__ import annotations @@ -11,7 +11,6 @@ import sys from collections.abc import Sequence from pathlib import Path -from typing import Any, cast from nvx_tools import nvx_rs from nvx_tools.aci_edge_sandboxes_tests import ( @@ -113,68 +112,11 @@ from nvx_tools.shard_plans import parse_shard from nvx_tools.time_abi import ( benchmark_cpu_profile_refusal, - cpu_profile_guidance_applies, - host_cpu_profile_guidance, openvmm_cpu_generations, ) DEFAULT_RELEASE_REPOSITORY = "microsoft/nvx" HYPERVISORS = ("auto", "whp", "kvm", "mshv") -# `vmm` performs every share operation as OpenVMM; `caller` performs each as the -# guest caller's identity, with guest root squashed to the share owner. -MOUNT_OWNERS = ("vmm", "caller") -NETWORK_PROFILES = ("portable",) -NETWORK_EGRESS_RULE_METAVAR = "CIDR[:PROTOCOL[:PORT[-PORT]]]" -NETWORK_EGRESS_RULE_HELP = ( - "{action} guest egress to an IPv4 or IPv6 CIDR, optionally only for PROTOCOL " - "tcp, udp, or icmp, and for tcp or udp only on PORT or on the inclusive range " - "PORT-PORT; repeat to add rules" -) -SANDBOX_OPERATIONS = ("run", "provision", "start", "exec", "stop", "deprovision") -# The namespace attribute in which the run and sandbox parsers record their -# options for nvx-rs. -NVX_RS_ARGUMENTS = "nvx_rs_arguments" - - -class _NvxRsOption(argparse.Action): - """Record an option of `nvx.py run` or `nvx.py sandbox` for nvx-rs, which - checks and uses every value. - - argparse resolves abbreviations and the --option=VALUE form, so nvx-rs - receives each option under its canonical name, in command-line order, - which attributes each --mount-deny, --mount-allow, and --mount-write of - `sandbox` to the --mount before it, and with its value attached, so that a - value that starts with "-" stays a value. With `keep`, the value, or `True` - for a flag, also goes to the option's destination, where nvx.py reads it. - """ - - def __init__( - self, - option_strings: Sequence[str], - dest: str, - nargs: int | str | None = None, - *, - keep: bool = False, - **kwargs: Any, - ) -> None: - super().__init__( - option_strings, dest, nargs=nargs, default=argparse.SUPPRESS, **kwargs - ) - self.keep = keep - - def __call__( - self, - parser: argparse.ArgumentParser, - namespace: argparse.Namespace, - values: str | Sequence[object] | None, - option_string: str | None = None, - ) -> None: - name = self.option_strings[0] - argument = name if self.nargs == 0 else f"{name}={cast(str, values)}" - recorded = cast(list[str], getattr(namespace, NVX_RS_ARGUMENTS, None) or []) - setattr(namespace, NVX_RS_ARGUMENTS, [*recorded, argument]) - if self.keep: - setattr(namespace, self.dest, True if self.nargs == 0 else values) def _run( @@ -440,53 +382,6 @@ def command_download(args: argparse.Namespace) -> None: download_checkout_release(args.repository, _release_platform(args.hypervisor)) -def _format_command(command: list[str]) -> str: - return subprocess.list2cmdline(command) if os.name == "nt" else shlex.join(command) - - -def _cold_boot_guidance(args: argparse.Namespace) -> list[str]: - """Return the hidden --cold-boot-guidance option of nvx-rs run, which - holds the guidance that nvx-rs prints after OpenVMM fails a cold boot on a - host whose CPU no built-in CPU profile serves, or nothing. - - The CPU catalog of the guidance stays in time_abi.py, which doctor and - benchmark use too. A dry run and a restore never boot the guest, and only - a guidance that applies needs the OpenVMM executable, which the guidance's - command names, so nvx.py looks it up only then.""" - if getattr(args, "dry_run", False) or hasattr(args, "restore_snapshot"): - return [] - cpu_profile = cast(str | None, getattr(args, "cpu_profile", None)) - host = host_cpu_signature() - if not cpu_profile_guidance_applies(cpu_profile, host): - return [] - guidance = host_cpu_profile_guidance( - cpu_profile, - host, - _format_command([os.fspath(nvx_rs.artifacts().openvmm)]), - _hypervisor(cast(str, getattr(args, "hypervisor", "auto"))), - ) - return [] if guidance is None else [f"--cold-boot-guidance={guidance}"] - - -def command_run(args: argparse.Namespace) -> int: - """Run the run command with nvx-rs, which checks its options, builds the - OpenVMM command line, and runs OpenVMM on the standard streams, and return - the status that nvx.py exits with.""" - return nvx_rs.run( - ["run", *getattr(args, NVX_RS_ARGUMENTS), *_cold_boot_guidance(args)], - program=args.program, - ) - - -def command_sandbox(args: argparse.Namespace) -> int: - """Run the sandbox command with nvx-rs, which checks its options and runs it - on the standard streams, and return the status that nvx.py exits with.""" - return nvx_rs.run( - ["sandbox", args.sandbox_operation, *getattr(args, NVX_RS_ARGUMENTS)], - program=args.program, - ) - - def command_collect_sources(_: argparse.Namespace) -> None: collect_release_sources(DockerBuildConfig()) @@ -1042,259 +937,6 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: download.add_argument("--hypervisor", choices=HYPERVISORS, default="auto") download.set_defaults(handler=command_download) - run = subparsers.add_parser("run", help="run an OpenVMM microVM") - - # nvx-rs checks the values and applies the defaults; these options define - # the command line and its help, and argparse checks only the choices of - # those that take a string. nvx.py keeps the options that decide on its - # CPU-profile guidance. - def run_option(*names: str, flag: bool = False, **kwargs: Any) -> None: - run.add_argument( - *names, action=_NvxRsOption, nargs=0 if flag else None, **kwargs - ) - - run_option("--guest", choices=GUEST_NAMES) - run_option("--hypervisor", choices=HYPERVISORS, keep=True) - run_option("--machine", choices=("microvm",)) - run_option( - "--memory-mib", - help=( - "guest RAM in MiB (default by --guest: " - + ", ".join( - f"{name} {guest_descriptor(name).default_memory_mib}" - for name in GUEST_NAMES - ) - + ")" - ), - ) - run_option("--memory-capacity-mib") - # nvx-rs converts the processor counts before it checks their choices, as - # argparse checks those of an option of type int. - run_option("--processors", metavar="{1,2,4,8}") - run_option( - "--cpu-profile", - keep=True, - metavar="ID", - help=( - "OpenVMM CPU profile: auto (OpenVMM's default) for the built-in " - "profile of the host's CPU, a built-in profile ID, or host to " - "derive a development profile from this host (doc/usage.md)" - ), - ) - run_option( - "--mount", - metavar="GUEST_TARGET,HOST_PATH[,ro|rw]", - help=( - "live-share a host directory, or a single regular host file without " - "its directory; the mode follows the last comma, so HOST_PATH may " - "contain commas when the mode is given; repeat for more shares, " - "each with its own guest target and mode, as children of one " - "virtio-fs device" - ), - ) - run_option("--mount-deny") - run_option( - "--mount-allow", - metavar="HOST_PATH", - help="expose one existing path inside a --mount-deny path again", - ) - run_option( - "--mount-write", - metavar="HOST_PATH", - help=( - "make one existing path one of the only writable parts of a " - "read-write --mount" - ), - ) - run_option( - "--mount-owner", - choices=MOUNT_OWNERS, - help=( - "host identity of the share's file operations: vmm (default) or " - "caller, with guest root squashed to the owner of the shared " - "directory or file (Linux only)" - ), - ) - run_option("--net", metavar="IPV4/PREFIX") - run_option("--network-profile", choices=NETWORK_PROFILES) - run_option("--network-egress", choices=("allow", "deny")) - run_option("--network-ingress", choices=("allow", "deny")) - run_option( - "--network-egress-allow", - metavar=NETWORK_EGRESS_RULE_METAVAR, - help=NETWORK_EGRESS_RULE_HELP.format(action="allow"), - ) - run_option( - "--network-egress-deny", - metavar=NETWORK_EGRESS_RULE_METAVAR, - help=NETWORK_EGRESS_RULE_HELP.format(action="deny"), - ) - run_option( - "--network-egress-policy-file", - metavar="PATH", - help="load bounded MXC-shaped IPv4 and IPv6 egress rules from JSON", - ) - run_option("--host-loopback", choices=("allow", "deny")) - run_option("--network-proxy", metavar="IPV4:TCP-PORT") - run_option("--host-loopback-forward") - run_option("--outcome-report", help="write a bounded local JSON outcome report") - run_option("--cmdline") - run_option("--restore-snapshot", keep=True) - run_option("--restore-processors", metavar="{1,2,4,8}") - run_option("--restore-memory-mib") - run_option("--restore-ready-path") - run_option("--dry-run", flag=True, keep=True) - run.set_defaults(handler=command_run, program=parser.prog, **{NVX_RS_ARGUMENTS: []}) - - sandbox = subparsers.add_parser( - "sandbox", - help="run or manage workloads over EROFS layers and private ext4 scratch", - ) - - # nvx-rs checks the values and applies the defaults; these options define - # the command line and its help. - def sandbox_option(*names: str, flag: bool = False, **kwargs: Any) -> None: - sandbox.add_argument( - *names, action=_NvxRsOption, nargs=0 if flag else None, **kwargs - ) - - sandbox.add_argument( - "sandbox_operation", - nargs="?", - choices=SANDBOX_OPERATIONS, - default="run", - ) - sandbox_option("--layer", metavar="ROLE,PATH,EROFS_UUID") - sandbox_option("--scratch") - sandbox_option("--state-dir") - sandbox_option("--entrypoint") - sandbox_option("--arg", dest="sandbox_arg") - sandbox_option("--hostname") - sandbox_option( - "--workload-user", - metavar="UID:GID", - help="fixed non-root workload identity (default: 65534:65534)", - ) - sandbox_option("--memory-max") - sandbox_option("--pids-max") - sandbox_option("--memory-mib") - sandbox_option( - "--timeout", - help=( - "control operation timeout, and the longest wait for another " - "lifecycle transition of the sandbox, in seconds (default: 60)" - ), - ) - sandbox_option( - "--exec-timeout-ms", - help="guest workload timeout in milliseconds; zero disables it", - ) - sandbox_option( - "--cwd", - help="absolute guest working directory for managed exec (default: /)", - ) - sandbox_option( - "--environment", - metavar="KEY=VALUE", - help="set the exact managed exec environment; repeat for multiple entries", - ) - sandbox_option( - "--environment-file", - metavar="PATH", - help="read the exact managed exec environment from a JSON string array", - ) - sandbox_option( - "--inherit-default-environment", - flag=True, - help=( - "layer the managed exec environment over the guest default " - "environment instead of replacing it" - ), - ) - sandbox_option("--hypervisor", choices=HYPERVISORS) - sandbox_option( - "--mount", - metavar="GUEST_TARGET,HOST_PATH[,ro|rw]", - help=( - "live-share a host directory, or a single regular host file without " - "its directory, inside the container rootfs; the mode follows the " - "last comma, so HOST_PATH may contain commas when the mode is given; " - "repeat to attach more shares, each with its own target and mode, as " - "children of one virtio-fs device" - ), - ) - sandbox_option( - "--mount-deny", - metavar="HOST_PATH", - help=( - "hide one existing path inside a --mount host directory; with " - "several --mount options, it applies to the --mount before it" - ), - ) - sandbox_option( - "--mount-allow", - metavar="HOST_PATH", - help=( - "expose one existing path inside a --mount-deny path of the same " - "share again; the hidden directories on the way list only it" - ), - ) - sandbox_option( - "--mount-write", - metavar="HOST_PATH", - help=( - "make one existing path one of the only writable parts of a " - "read-write --mount; the rest of the share is read-only" - ), - ) - sandbox_option( - "--mount-owner", - choices=MOUNT_OWNERS, - help=( - "host identity of the share's file operations: vmm (default) or " - "caller, the workload identity with guest root squashed to the " - "owner of the shared directory or file (Linux only)" - ), - ) - sandbox_option("--net", metavar="IPV4/PREFIX") - sandbox_option("--network-profile", choices=NETWORK_PROFILES) - sandbox_option("--network-egress", choices=("allow", "deny")) - sandbox_option("--network-ingress", choices=("allow", "deny")) - sandbox_option( - "--network-egress-allow", - metavar=NETWORK_EGRESS_RULE_METAVAR, - help=NETWORK_EGRESS_RULE_HELP.format(action="allow"), - ) - sandbox_option( - "--network-egress-deny", - metavar=NETWORK_EGRESS_RULE_METAVAR, - help=NETWORK_EGRESS_RULE_HELP.format(action="deny"), - ) - sandbox_option( - "--network-egress-policy-file", - metavar="PATH", - help=( - "load bounded IPv4 and IPv6 ranges and rule-local exclusions for " - "run/provision" - ), - ) - sandbox_option("--host-loopback", choices=("allow", "deny")) - sandbox_option("--network-proxy", metavar="IPV4:TCP-PORT") - sandbox_option("--host-loopback-forward") - sandbox_option( - "--outcome-report", - help="write a bounded local JSON outcome report for run or exec", - ) - sandbox_option("--cmdline") - sandbox_option( - "--dry-run", - flag=True, - help="print the one-shot run command without launching it", - ) - sandbox.set_defaults( - handler=command_sandbox, program=parser.prog, **{NVX_RS_ARGUMENTS: []} - ) - benchmark = subparsers.add_parser( "benchmark", help="run the OpenVMM-native benchmark coordinator", diff --git a/scripts/nvx_tools/adversarial_executor.py b/scripts/nvx_tools/adversarial_executor.py index 8b2bd2fce..51ad008eb 100644 --- a/scripts/nvx_tools/adversarial_executor.py +++ b/scripts/nvx_tools/adversarial_executor.py @@ -59,9 +59,9 @@ ), } # The variables that the scenarios inherit. Besides the platform's own, they -# are the paths that nvx-rs needs: NVX_RS, the executable that nvx.py runs, and -# CARGO_HOME and RUSTUP_HOME, the Rust toolchain that builds it and runs the -# managed-lifecycle test. RUSTUP_TOOLCHAIN stays out, so that +# are the paths that nvx-rs needs: NVX_RS, the executable that the scenarios +# run, and CARGO_HOME and RUSTUP_HOME, the Rust toolchain that builds it and +# runs the managed-lifecycle test. RUSTUP_TOOLCHAIN stays out, so that # rust-toolchain.toml selects the Rust release, as do Cargo's other settings. _ENVIRONMENT_ALLOWLIST = frozenset( { @@ -178,8 +178,8 @@ def _artifact_metadata() -> dict[str, object]: artifact_path(OpenVMMBuildConstants.PROVENANCE_NAME), "OpenVMM provenance", ), - # The nvx-rs that the scenarios run through nvx.py, which never - # rebuilds it, so a stale one would run without its provenance check. + # The nvx-rs that the scenarios run, which they never rebuild, so a + # stale one would run without its provenance check. "nvx_rs": nvx_rs.find_binary() or require_file(nvx_rs.binary_path(), "nvx-rs executable"), "nvx_rs_provenance": require_file( diff --git a/scripts/nvx_tools/managed_exec_tests.py b/scripts/nvx_tools/managed_exec_tests.py index c3cffff66..98040195c 100644 --- a/scripts/nvx_tools/managed_exec_tests.py +++ b/scripts/nvx_tools/managed_exec_tests.py @@ -5,7 +5,6 @@ import json import shutil import subprocess -import sys import tempfile from dataclasses import dataclass from pathlib import Path @@ -177,11 +176,7 @@ def run_managed_exec_configuration( state = root / "state" scratch = root / "scratch.ext4" shutil.copyfile(scratch_template, scratch) - base = [ - sys.executable, - str(BuildConstants.REPO_ROOT / "scripts" / "nvx.py"), - "sandbox", - ] + base = nvx_rs.command_line(("sandbox",)) def invoke( operation: str, *arguments: str, expected: int = 0 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 7647e318f..ecc4e2a3a 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -148,7 +148,7 @@ "snapshot-tiers", ) ) -# The public `nvx.py sandbox` scenarios need the Ubuntu scratch template that +# The public `nvx-rs sandbox` scenarios need the Ubuntu scratch template that # the guest-artifact build produces and packages omit, so they run only when # named. PUBLIC_SANDBOX_SCENARIOS = frozenset( @@ -2055,9 +2055,7 @@ def serve_allowed() -> None: policy_file=policy_path, ) command = [ - sys.executable, - str(Path(__file__).parents[1] / "nvx.py"), - "run", + *nvx_rs.command_line(("run",)), "--guest", guest, "--hypervisor", @@ -2185,7 +2183,7 @@ def serve_allowed() -> None: (output_dir / "l3-l4-egress-policy-results.json").write_text( json.dumps( { - "interface": "nvx.py run", + "interface": "nvx-rs run", "policy_file": policy_path.name, "tcp_ports": { "adjacent_low": tcp_ports[0], @@ -2423,9 +2421,7 @@ def run_l3_l4_egress_protocols( encoding="utf-8", ) command = [ - sys.executable, - str(Path(__file__).parents[1] / "nvx.py"), - "run", + *nvx_rs.command_line(("run",)), "--guest", guest, "--hypervisor", @@ -2504,7 +2500,7 @@ def run_l3_l4_egress_protocols( (output_dir / "l3-l4-egress-protocols-results.json").write_text( json.dumps( { - "interface": "nvx.py run", + "interface": "nvx-rs run", "ports": {"named": named_port, "other": other_port}, "cases": results, }, @@ -2603,9 +2599,7 @@ def run_l3_l4_egress_port_ranges( ABOVE_PORT=str(ports["above"]), ) public_run = [ - sys.executable, - str(Path(__file__).parents[1] / "nvx.py"), - "run", + *nvx_rs.command_line(("run",)), "--guest", guest, "--hypervisor", @@ -2766,7 +2760,7 @@ def run_l3_l4_egress_port_ranges( (output_dir / "l3-l4-egress-port-ranges-results.json").write_text( json.dumps( - {"interface": "nvx.py run", "ports": ports, "cases": results}, + {"interface": "nvx-rs run", "ports": ports, "cases": results}, indent=2, ) + "\n", @@ -3068,9 +3062,7 @@ def run_ipv6_egress_policy( ) native = compile_egress_policy(policy_path) command = [ - sys.executable, - str(Path(__file__).parents[1] / "nvx.py"), - "run", + *nvx_rs.command_line(("run",)), "--guest", guest, "--hypervisor", @@ -3167,7 +3159,7 @@ def run_ipv6_egress_policy( (output_dir / "ipv6-egress-policy-results.json").write_text( json.dumps( { - "interface": "nvx.py run", + "interface": "nvx-rs run", "guest_ipv6": ( f"{DIRECTIONAL_NETWORK_GUEST_IPV6}/" f"{DIRECTIONAL_NETWORK_IPV6_PREFIX_LENGTH}" @@ -6915,7 +6907,7 @@ def run_filesystem_comma_paths( output_dir: Path, guest: str = "alpine", ) -> None: - """Share host directories whose paths contain commas through `nvx.py run`: + """Share host directories whose paths contain commas through `nvx-rs run`: a read-write `alpha,beta` and a read-only `2026, final`, each with a denied subdirectory, first each alone, as OpenVMM's `--mount`, and then both, as the children of an aggregate. The guest must find each exact @@ -6944,9 +6936,7 @@ def run_filesystem_comma_paths( ("aggregate", (alpha, reports)), ): command = [ - sys.executable, - str(Path(__file__).parents[1] / "nvx.py"), - "run", + *nvx_rs.command_line(("run",)), "--guest", guest, "--hypervisor", @@ -7936,7 +7926,7 @@ def _run_scenarios( if "filesystem-comma-paths" in scenarios: report.start( "filesystem-comma-paths", - "Running public nvx.py shares of host paths with commas " + "Running public nvx-rs shares of host paths with commas " f"on OpenVMM/{args.backend}", ) run_filesystem_comma_paths( @@ -8041,7 +8031,7 @@ def _run_scenarios( if "l3-l4-egress-policy" in scenarios: report.start( "l3-l4-egress-policy", - "Running public nvx.py L3/L4 egress policy acceptance " + "Running public nvx-rs L3/L4 egress policy acceptance " f"on OpenVMM/{args.backend}", ) run_l3_l4_egress_policy( @@ -8057,7 +8047,7 @@ def _run_scenarios( if "l3-l4-egress-protocols" in scenarios: report.start( "l3-l4-egress-protocols", - "Running public nvx.py L3/L4 egress protocol acceptance " + "Running public nvx-rs L3/L4 egress protocol acceptance " f"on OpenVMM/{args.backend}", ) run_l3_l4_egress_protocols( @@ -8073,7 +8063,7 @@ def _run_scenarios( if "l3-l4-egress-port-ranges" in scenarios: report.start( "l3-l4-egress-port-ranges", - "Running public nvx.py L3/L4 egress port-range acceptance " + "Running public nvx-rs L3/L4 egress port-range acceptance " f"on OpenVMM/{args.backend}", ) run_l3_l4_egress_port_ranges( @@ -8089,7 +8079,7 @@ def _run_scenarios( if "ipv6-egress-policy" in scenarios: report.start( "ipv6-egress-policy", - "Running public nvx.py IPv6 egress policy acceptance " + "Running public nvx-rs IPv6 egress policy acceptance " f"on OpenVMM/{args.backend}", ) run_ipv6_egress_policy( diff --git a/scripts/nvx_tools/nvx_rs.py b/scripts/nvx_tools/nvx_rs.py index befd8c72c..38d9aa4e9 100644 --- a/scripts/nvx_tools/nvx_rs.py +++ b/scripts/nvx_tools/nvx_rs.py @@ -1,15 +1,15 @@ -"""Find, build, and run nvx-rs, the Rust command-line interface that nvx.py -delegates to. +"""Find, build, and run nvx-rs, the Rust command-line interface that runs +NVX's microVMs and sandboxes. -nvx.py runs nvx-rs for `nvx.py run` and `nvx.py sandbox`, for the artifact -lookup of the CPU-profile guidance of `nvx.py run`, and for the egress policies -of the microVM tests (nvx-rs/README.md). It runs, in this order: +The microVM tests run their microVMs and sandboxes with the command lines +that command_line() gives, and compile their egress policies with nvx-rs +(nvx-rs/README.md). They run, in this order: 1. the executable that the NVX_RS environment variable names; 2. nvx-rs/target/release/nvx-rs[.exe] in this checkout, where `cargo build --release` puts it, and where `nvx.py download` installs the nvx-rs of a release built from this checkout's nvx-rs sources; or else -3. that executable, after it builds it with Cargo, whose output goes to +3. that executable, after nvx.py builds it with Cargo, whose output goes to standard error so that it never mixes with the output of the command. Without Cargo, nvx.py fails and names these choices. @@ -28,21 +28,17 @@ from __future__ import annotations -import contextlib import hashlib import json import os import re import shlex import shutil -import signal import subprocess import sys -import threading -from collections.abc import Callable, Generator, Sequence +from collections.abc import Sequence from dataclasses import dataclass from pathlib import Path -from types import FrameType from typing import cast from .build_constants import BuildConstants, OpenVMMBuildConstants @@ -324,117 +320,13 @@ def require_binary() -> Path: return find_binary() or build_binary() -def _command(arguments: Sequence[str], program: str | None) -> list[str]: - command = [os.fspath(require_binary())] - if program is not None: - command.extend(("--prog", program)) - command.extend(("--root", os.fspath(ROOT), *arguments)) - return command - - -def _forward(process: subprocess.Popen[bytes], signum: int) -> None: - # Popen.send_signal signals nothing once the process has exited. - with contextlib.suppress(OSError): - process.send_signal(signum) - - -@contextlib.contextmanager -def _signals_handed_over() -> Generator[Callable[[subprocess.Popen[bytes]], None]]: - """Hand nvx-rs the signals that would end nvx.py while it runs, and yield - the function that names the nvx-rs process once it has started. - - nvx-rs handles Ctrl-C itself: it prints Interrupted and exits with status - 130, which nvx.py then exits with, rather than printing Interrupted again. - A terminal's Ctrl-C reaches both. On POSIX hosts, a SIGINT, SIGTERM, or - SIGHUP sent to nvx.py alone reaches nvx-rs too, a signal that arrives - before nvx-rs has started reaches it as it starts, and a second SIGINT - changes nothing for nvx-rs. On Windows, where Ctrl-C reaches every process - of the console, nvx.py only ignores it. - - A signal that nvx.py ignores stays ignored, so that nvx-rs inherits it - ignored rather than at its default, which a handler would leave it at - exec: a shell without job control starts a background command with SIGINT - ignored, and nohup starts one with SIGHUP ignored. - """ - started: list[subprocess.Popen[bytes]] = [] - pending: list[int] = [] - - def record(process: subprocess.Popen[bytes]) -> None: - started.append(process) - for signum in pending: - _forward(process, signum) - - if threading.current_thread() is not threading.main_thread(): - yield record - return - names = ("SIGINT",) if os.name == "nt" else ("SIGINT", "SIGTERM", "SIGHUP") - signums = [ - cast(int, getattr(signal, signal_name)) - for signal_name in names - if hasattr(signal, signal_name) - ] - - def hand_over(signum: int, _frame: FrameType | None) -> None: - if os.name == "nt": - return - if started: - _forward(started[0], signum) - else: - pending.append(signum) - - previous: list[tuple[int, Callable[[int, FrameType | None], object] | int]] = [] - try: - for signum in signums: - if signal.getsignal(signum) == signal.SIG_IGN: - continue - handler = signal.signal(signum, hand_over) - previous.append((signum, signal.SIG_DFL if handler is None else handler)) - yield record - finally: - for signum, handler in reversed(previous): - signal.signal(signum, handler) - - -def _wait(process: subprocess.Popen[bytes]) -> int: - while True: - try: - return process.wait() - except KeyboardInterrupt: - # A handler other than nvx.py's raised it; nvx-rs, which Ctrl-C - # reaches too, decides how the command ends. - if os.name != "nt": - _forward(process, signal.SIGINT) - - -def exit_status(returncode: int) -> int: - """Return the status that nvx.py exits with for nvx-rs's `returncode`. - - That is nvx-rs's own status, 128 + N if signal N ended nvx-rs, as a shell - reports it, and on Windows, a 32-bit status as the signed value that - SystemExit passes on with every Python release. - """ - if returncode < 0: - return 128 - returncode - if os.name == "nt" and returncode > 0x7FFF_FFFF: - return returncode - (1 << 32) - return returncode - - -def run(arguments: Sequence[str], *, program: str) -> int: - """Run `nvx-rs --prog PROGRAM --root ROOT ARGUMENTS` on nvx.py's standard - streams, and return the status that nvx.py exits with.""" - command = _command(arguments, program) - with _signals_handed_over() as started: - try: - process = subprocess.Popen(command) - except OSError as error: - raise ScriptError(f"failed to run {command[0]}: {error}") from error - started(process) - returncode = _wait(process) - if os.name != "nt" and returncode == -signal.SIGINT: - # Ctrl-C ended nvx-rs before it could handle it, as it would have. - print("Interrupted", file=sys.stderr) - return exit_status(returncode) +def command_line(arguments: Sequence[str], *, root: Path | None = None) -> list[str]: + """Return the command line `nvx-rs --root ROOT ARGUMENTS`, which runs the + nvx-rs that nvx.py runs, built first if needed, on the artifacts of `root`, + this checkout by default, as the microVM scenarios run their sandboxes and + microVMs.""" + root = ROOT if root is None else root + return [os.fspath(require_binary()), "--root", os.fspath(root), *arguments] # nvx-rs reports an error on standard error as `error[CODE]: MESSAGE`, and a @@ -507,7 +399,7 @@ def query(arguments: Sequence[str]) -> object: that nvx-rs reported, which nvx.py prints after `error: ` as it printed the message of the Python code that nvx-rs replaced. """ - command = _command(arguments, None) + command = command_line(arguments) try: completed = subprocess.run( command, stdin=subprocess.DEVNULL, capture_output=True, check=False @@ -530,13 +422,6 @@ def _object(value: object, what: str) -> dict[str, object]: return cast(dict[str, object], value) -def _string(document: dict[str, object], key: str, what: str) -> str: - value = document.get(key) - if not isinstance(value, str): - raise ScriptError(f"nvx-rs printed {what} without a {key} string") - return value - - def _strings(document: dict[str, object], key: str, what: str) -> tuple[str, ...]: value = document.get(key) if not isinstance(value, list) or not all( @@ -567,55 +452,3 @@ def compile_egress_policy(path: Path) -> EgressRules: allow=_strings(document, "allow", what), deny=_strings(document, "deny", what), ) - - -@dataclass(frozen=True) -class GuestArtifacts: - """The artifacts of one guest that `nvx-rs artifacts` lists.""" - - name: str - distribution: str - initramfs: Path - - -@dataclass(frozen=True) -class ArtifactPaths: - """The paths of the artifacts that nvx.py runs, which nvx-rs does not check.""" - - openvmm: Path - kernel: Path - guests: tuple[GuestArtifacts, ...] - - def guest(self, name: str) -> GuestArtifacts: - for guest in self.guests: - if guest.name == name: - return guest - raise ScriptError(f"nvx-rs printed no artifacts of the {name} guest") - - -def artifacts() -> ArtifactPaths: - """Return the paths of the artifacts of ROOT with `nvx-rs artifacts`. - - OpenVMM is the executable that NVX_OPENVMM names, if set, as for nvx.py - run and nvx.py sandbox, and otherwise the checkout's. - """ - what = "artifact paths" - document = _object(query(("artifacts",)), what) - raw_guests = document.get("guests") - if not isinstance(raw_guests, list): - raise ScriptError(f"nvx-rs printed {what} without a guests list") - guests: list[GuestArtifacts] = [] - for raw_guest in cast(list[object], raw_guests): - guest = _object(raw_guest, "a guest") - guests.append( - GuestArtifacts( - name=_string(guest, "name", "a guest"), - distribution=_string(guest, "distribution", "a guest"), - initramfs=Path(_string(guest, "initramfs", "a guest")), - ) - ) - return ArtifactPaths( - openvmm=Path(_string(document, "openvmm", what)), - kernel=Path(_string(document, "kernel", what)), - guests=tuple(guests), - ) diff --git a/scripts/nvx_tools/release.py b/scripts/nvx_tools/release.py index 8866391cf..25370552d 100644 --- a/scripts/nvx_tools/release.py +++ b/scripts/nvx_tools/release.py @@ -710,11 +710,11 @@ def _packaged_nvx_rs( the build of this checkout's nvx-rs sources that nvx.py runs, and otherwise None, with the reason. - nvx.py runs the nvx-rs that download installs without rebuilding it, so an - nvx-rs of other sources, such as that of an older release or of a checkout - whose nvx-rs changed, or with other features, would run with this - checkout's nvx.py. A package whose nvx-rs does not match its own - provenance is broken, and fails. + The checkout's users and microVM tests run the nvx-rs that download + installs without rebuilding it, so an nvx-rs of other sources, such as + that of an older release or of a checkout whose nvx-rs changed, or with + other features, would run in place of a build of this checkout. A package + whose nvx-rs does not match its own provenance is broken, and fails. """ name = nvx_rs.binary_path().name executable = package_root / "bin" / name @@ -787,13 +787,13 @@ def _install_release_archive(archive_path: Path) -> None: nvx_rs_provenance = artifact_path(nvx_rs.PROVENANCE_NAME) if packaged_nvx_rs is None: # Remove the nvx-rs that an earlier download or build left, which - # may not match this checkout, so that nvx.py builds one from it. + # may not match this checkout, so that only a build of it runs. nvx_rs.binary_path().unlink(missing_ok=True) nvx_rs_provenance.unlink(missing_ok=True) print( - f">> {nvx_rs_skipped}, so nvx.py builds nvx-rs from this checkout " - f"with Cargo when a command needs it, unless {nvx_rs.VARIABLE} " - "names one" + f">> {nvx_rs_skipped}, so build nvx-rs from this checkout with " + f"`{' '.join(nvx_rs.BUILD_COMMAND)}`, as the microVM tests do " + f"when they need it, unless {nvx_rs.VARIABLE} names one" ) else: executable, provenance = packaged_nvx_rs diff --git a/scripts/nvx_tools/sandbox_lifecycle_tests.py b/scripts/nvx_tools/sandbox_lifecycle_tests.py index 16bd388fa..4cd2b2a93 100644 --- a/scripts/nvx_tools/sandbox_lifecycle_tests.py +++ b/scripts/nvx_tools/sandbox_lifecycle_tests.py @@ -1,6 +1,6 @@ """Public CLI acceptance for the managed sandbox lifecycle and security profile. -The scenario runs `nvx.py sandbox` as subprocesses against the real kernel, the +The scenario runs `nvx-rs sandbox` as subprocesses against the real kernel, the Alpine control initramfs, the Ubuntu EROFS layer, and ext4 scratch copies. It checks identity admission, fail-closed lifecycle transitions, that overlapping starts launch one VM, warm-guest state across exec requests and restarts, the @@ -106,7 +106,7 @@ def __str__(self) -> str: # A start that OpenVMM ended names the category of its outcome report. REFUSED_START = Refusal("exited", (REFUSAL_CATEGORY,)) -# The files of a state directory, as nvx.py sandbox documents them. +# The files of a state directory, as nvx-rs sandbox documents them. CONFIG = "config.json" RUNTIME = "runtime.json" CAPABILITY = "control.capability" @@ -241,9 +241,36 @@ def terminate_process(pid: int) -> None: os.kill(pid, signal.SIGKILL) +def openvmm_threads_remain(runtime: dict[str, Any]) -> bool: + """Returns whether a thread of the OpenVMM process of a runtime record + remains on Linux, where nvx-rs counts a process as running until its last + thread exits: a killed process's main thread can be a zombie, whose command + line is empty, while its other threads still exit.""" + if sys.platform == "win32": + return False + pid = recorded_pid(runtime) + try: + stat = Path(f"/proc/{pid}/stat").read_bytes() + except (FileNotFoundError, ProcessLookupError): + return False + fields = stat[stat.rindex(b")") + 1 :].split() + if int(fields[19]) != int(runtime["process"]["startTime"]): + # The ID names another process. + return False + if fields[0] == b"X": + return False + if fields[0] != b"Z": + return True + try: + return len(os.listdir(f"/proc/{pid}/task")) > 1 + except (FileNotFoundError, ProcessLookupError): + return False + + def kill_openvmm(runtime: dict[str, Any], timeout: float, marker: str) -> None: - """Ends the OpenVMM process of a runtime record, as a crash would; marker - names the fixture, as for openvmm_running.""" + """Ends the OpenVMM process of a runtime record, as a crash would, and waits + until it has exited as nvx-rs tells; marker names the fixture, as for + openvmm_running.""" if openvmm_running(runtime, marker): try: terminate_process(recorded_pid(runtime)) @@ -252,7 +279,7 @@ def kill_openvmm(runtime: dict[str, Any], timeout: float, marker: str) -> None: if openvmm_running(runtime, marker): raise deadline = time.monotonic() + timeout - while openvmm_running(runtime, marker): + while openvmm_running(runtime, marker) or openvmm_threads_remain(runtime): if time.monotonic() >= deadline: raise RuntimeError( f"OpenVMM process {recorded_pid(runtime)} survived its kill" @@ -406,16 +433,9 @@ def run_cli( state: Path | None, root: Path | None = None, ) -> subprocess.CompletedProcess[bytes]: - """Runs one public operation with `nvx.py sandbox`, or, under the - artifact root `root`, with the nvx-rs that it runs.""" - if root is None: - command = [ - sys.executable, - str(BuildConstants.REPO_ROOT / "scripts" / "nvx.py"), - ] - else: - command = [str(nvx_rs.require_binary()), "--root", str(root)] - command.extend(("sandbox", operation)) + """Runs one public operation with `nvx-rs sandbox`, under the artifact + root `root`, or else under this checkout.""" + command = nvx_rs.command_line(("sandbox", operation), root=root) if state is not None: command.extend(("--state-dir", str(state))) command.extend(("--timeout", str(self.timeout), *arguments)) diff --git a/scripts/nvx_tools/sandbox_proxy_tests.py b/scripts/nvx_tools/sandbox_proxy_tests.py index 5a5b09780..9429298cc 100644 --- a/scripts/nvx_tools/sandbox_proxy_tests.py +++ b/scripts/nvx_tools/sandbox_proxy_tests.py @@ -1,6 +1,6 @@ """Public CLI acceptance for proxy-only sandbox networking. -The scenario provisions a sandbox through `nvx.py sandbox` with a proxy-only +The scenario provisions a sandbox through `nvx-rs sandbox` with a proxy-only network: egress, ingress, and host-loopback access are denied, and a proxy on host loopback, which the guest reaches at its gateway, is the guest's only way out. A managed exec request in the Ubuntu workload layer then requests a URL diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 089e14adb..f8c54eeb0 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -128,11 +128,12 @@ def describe(self) -> str: # A copy of the generations of OpenVMM's pinned CPU profiles # (openvmm/vmm_core/cpu_profile/profiles), which test_time_abi compares with -# the submodule. Doctor uses it only without OpenVMM (--no-openvmm), and run to -# explain a failed cold boot on a CPU that it lacks; benchmark reads the -# profiles of the OpenVMM checkout that it builds instead +# the submodule. Doctor uses it only without OpenVMM (--no-openvmm); benchmark +# reads the profiles of the OpenVMM checkout that it builds instead # (openvmm_cpu_generations), and otherwise OpenVMM reports the generation and -# the profile itself. +# the profile itself. nvx-rs run, which guides a failed cold boot on a CPU that +# no built-in profile serves, reads a copy of its own, +# nvx-rs/src/host/cli/cpu_generations.json. # Model 85 also covers Cascade Lake (steppings 5-7) and Cooper Lake (10-11), # which have no profile. AMD's family 25 model 1 is Milan's in every stepping, # Milan-X's 2 included, family 25 model 17 Genoa's, Genoa-X's included, and @@ -249,16 +250,10 @@ def openvmm_cpu_generations(openvmm_dir: Path) -> tuple[CpuGeneration, ...]: rf"{re.escape(CPU_PROFILE_FALLBACK_MARKER)} .*?, so --cpu-profile auto fell back" r" to ([a-z0-9-]+\.host\.v[1-9][0-9]*) \(sha256:[0-9a-f]{64}\), " ) -# The issues that track built-in CPU profiles for more Intel and more AMD CPUs. -# No issue tracks the CPUs of other vendors, which the time ABI does not serve -# (doc/design/2.2-cpu-and-topology.md, "CPU Profile Selection"). -CPU_SUPPORT_ISSUES: Mapping[str, tuple[str, str]] = { - "GenuineIntel": ("Intel", "https://github.com/microsoft/nvx/issues/408"), - "AuthenticAMD": ("AMD", "https://github.com/microsoft/nvx/issues/409"), -} # The issue form that requests a built-in CPU profile for a CPU # (.github/ISSUE_TEMPLATE/cpu-profile.yml), which OpenVMM's fallback warning, -# run, and benchmark link to, prefilling the CPU fields that the form defines. +# nvx-rs run, and benchmark link to, prefilling the CPU fields that the form +# defines. CPU_PROFILE_REQUEST_URL = ( "https://github.com/microsoft/nvx/issues/new?template=cpu-profile.yml" ) @@ -336,75 +331,6 @@ def parse_cpu_profile_fallback(output: str) -> str | None: return match.group(1) -def cpu_profile_guidance_applies(cpu_profile: str | None, host: HostCpu | None) -> bool: - """Return whether host_cpu_profile_guidance has guidance for a cold boot - with ``--cpu-profile cpu_profile``, or with its default, ``auto``, if - ``cpu_profile`` is None, on the CPU ``host``: whether no built-in profile - serves the CPU, and the profile is ``auto``, or ``host`` on a CPU of a - vendor that host profiles do not serve either. A caller can then skip - looking up the guidance's other inputs.""" - if host is None or ( - cpu_generation(host.vendor, host.family, host.model, host.stepping) is not None - ): - return False - if cpu_profile in (None, "auto"): - return True - return ( - cpu_profile == HOST_PROFILE_GENERATION - and host.vendor not in HOST_PROFILE_VENDORS - ) - - -def host_cpu_profile_guidance( - cpu_profile: str | None, host: HostCpu | None, openvmm: str, hypervisor: str -) -> str | None: - """Return the next steps after OpenVMM failed a cold boot with - ``--cpu-profile cpu_profile``, or with its default, ``auto``, if - ``cpu_profile`` is None, on the CPU ``host`` with the ``hypervisor`` - backend, or None unless no built-in profile serves the CPU. - - On such an Intel or AMD CPU, ``auto`` falls back to a host profile, so the - failure may not be the profile's: the guidance says what ``auto`` does on - the CPU, and asks for the CPU's fingerprint, which ``openvmm`` writes, to - add a built-in profile for it. OpenVMM's own error names this run's cause, - and says so where the host profile cannot boot. On another vendor's CPU, - ``auto`` and ``host`` fail with ``E_PROFILE_HOST_UNKNOWN``. An explicit - profile ID fails with its own code, and ``host`` on an Intel or AMD CPU - with OpenVMM's own explanation, which need no guidance. OpenVMM keeps its - standard error on the terminal, so the caller cannot read the code that - ended the run; cpu_profile_guidance_applies tells whether there is - guidance.""" - if host is None or not cpu_profile_guidance_applies(cpu_profile, host): - return None - host_profiles = host.vendor in HOST_PROFILE_VENDORS - unserved = f"nvx: no built-in CPU profile serves this host's CPU, {host.describe()}" - generations = f"the built-in profiles cover {describe_cpu_generations()}." - if not host_profiles: - return "\n".join( - ( - f"{unserved}, so a cold boot with --cpu-profile auto, the default, " - f"fails on it with {PROFILE_HOST_UNKNOWN}; {generations}", - f"nvx: --cpu-profile {HOST_PROFILE_GENERATION} cannot boot on it " - "either: host CPU profiles serve only Intel and AMD CPUs.", - ) - ) - vendor, issue = CPU_SUPPORT_ISSUES[host.vendor] - lines = [ - f"{unserved}, so --cpu-profile auto, the default, falls back on it to a " - "CPU profile derived from this host, which OpenVMM warns about with " - f"{CPU_PROFILE_FALLBACK_MARKER}; {generations}", - "nvx: OpenVMM's error above names the cause of this failure, and says " - 'so where the host profile cannot boot; doc/usage.md ("CPU profiles") ' - "explains a host profile's limits.", - ] - lines.extend( - f"nvx: {line}" - for line in cpu_profile_request_lines(host, openvmm, hypervisor).splitlines() - ) - lines.append(f"nvx: {issue} tracks CPU profiles for more {vendor} CPUs.") - return "\n".join(lines) - - def benchmark_cpu_profile_refusal( host: HostCpu | None, openvmm: str, diff --git a/scripts/setup/setup-linux-mshv.sh b/scripts/setup/setup-linux-mshv.sh index cb7ae3932..a3fffe7ee 100644 --- a/scripts/setup/setup-linux-mshv.sh +++ b/scripts/setup/setup-linux-mshv.sh @@ -320,7 +320,13 @@ check_environment() { check_build() { ( cd "$workspace" - python3 scripts/nvx.py run --hypervisor mshv --dry-run + RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN cargo build --release --locked --features cli \ + --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target + # Packaging and the adversarial executor take only an nvx-rs whose + # provenance binds it to the checkout's sources. + RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN NVX_RS=nvx-rs/target/release/nvx-rs \ + python3 scripts/nvx.py record-nvx-rs-provenance + nvx-rs/target/release/nvx-rs run --hypervisor mshv --dry-run ) } diff --git a/scripts/setup/setup-windows-whp.ps1 b/scripts/setup/setup-windows-whp.ps1 index 71f5ef2fd..cd4abdc5d 100644 --- a/scripts/setup/setup-windows-whp.ps1 +++ b/scripts/setup/setup-windows-whp.ps1 @@ -1673,8 +1673,29 @@ function Assert-Environment { -PathType Leaf)) { throw "missing OpenVMM release binary" } - Invoke-Native $python @( - "scripts\nvx.py", "run", + # nvx-rs runs NVX's microVMs; a current build changes nothing. + Invoke-Native (Get-RequiredCommand "cargo.exe") @( + "+$RustToolchain", "build", "--release", "--locked", + "--features", "cli", + "--manifest-path", "nvx-rs\Cargo.toml", + "--target-dir", "nvx-rs\target" + ) + $nvxRs = "$Workspace\nvx-rs\target\release\nvx-rs.exe" + # Packaging and the adversarial executor take only an nvx-rs whose + # provenance binds it to the checkout's sources. + $previousNvxRs = $env:NVX_RS + $previousToolchain = $env:RUSTUP_TOOLCHAIN + $env:NVX_RS = $nvxRs + $env:RUSTUP_TOOLCHAIN = $RustToolchain + try { + Invoke-Native $python @("scripts\nvx.py", "record-nvx-rs-provenance") + } + finally { + $env:NVX_RS = $previousNvxRs + $env:RUSTUP_TOOLCHAIN = $previousToolchain + } + Invoke-Native $nvxRs @( + "run", "--hypervisor", "whp", "--dry-run" ) diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 11f2970ac..f7715510b 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -53,7 +53,7 @@ def _compile(policy: dict[str, object]) -> nvx_rs.EgressRules: - """Compiles an egress policy with nvx-rs, as nvx.py run compiles its + """Compiles an egress policy with nvx-rs, as nvx-rs run compiles its --network-egress-policy-file.""" with tempfile.TemporaryDirectory(prefix="nvx-egress-policy-") as temporary: path = Path(temporary) / "policy.json" @@ -61,6 +61,13 @@ def _compile(policy: dict[str, object]) -> nvx_rs.EgressRules: return nvx_rs.compile_egress_policy(path) +# The nvx-rs that a scenario runs, in place of the one that nvx_rs finds or +# builds, where the test compiles no egress policy, and the start of the +# command line of the scenario's public runs. +_FAKE_NVX_RS = Path("nvx-rs") +_FAKE_PUBLIC_RUN = [os.fspath(_FAKE_NVX_RS), "--root", os.fspath(nvx_rs.ROOT), "run"] + + def _posix_shell() -> str | None: shell = shutil.which("sh") if shell is None: @@ -404,7 +411,7 @@ def run( ) -> subprocess.CompletedProcess[bytes]: commands.append(command) options.append(kwargs) - operation = command[3] + operation = command[4] state = Path(command[command.index("--state-dir") + 1]) if operation == "provision": state.mkdir(parents=True, exist_ok=True) @@ -550,6 +557,9 @@ def run( patch.object(managed_exec_tests, "require_file", side_effect=require), patch.object(managed_exec_tests.subprocess, "run", side_effect=run), patch.object(managed_exec_tests.shutil, "copyfile", side_effect=copyfile), + patch.object( + managed_exec_tests.nvx_rs, "require_binary", return_value=Path("nvx-rs") + ), ): managed_exec_tests.run_managed_exec_configuration( "whp", timeout=2, output_dir=output_dir @@ -576,15 +586,10 @@ def test_public_acceptance_observes_full_cli_contract(self): ) ) - exec_commands = [command for command in commands if command[3] == "exec"] + exec_commands = [command for command in commands if command[4] == "exec"] self.assertTrue( all( - command[:3] - == [ - sys.executable, - str(BuildConstants.REPO_ROOT / "scripts" / "nvx.py"), - "sandbox", - ] + command[:4] == ["nvx-rs", "--root", str(nvx_rs.ROOT), "sandbox"] for command in commands ) ) @@ -856,7 +861,7 @@ def test_evidence_failure_still_deprovisions_stopped_sandbox(self): ) ) - self.assertIn("deprovision", [command[3] for command in commands]) + self.assertIn("deprovision", [command[4] for command in commands]) self.assertEqual(records[-1]["operation"], "deprovision") fixture_root = Path(commands[0][commands[0].index("--state-dir") + 1]).parent self.assertFalse(fixture_root.exists()) @@ -871,7 +876,7 @@ def test_failed_start_deprovisions_safely_stopped_sandbox(self): command_log=commands, ) - self.assertIn("deprovision", [command[3] for command in commands]) + self.assertIn("deprovision", [command[4] for command in commands]) fixture_root = Path(commands[0][commands[0].index("--state-dir") + 1]).parent self.assertFalse(fixture_root.exists()) @@ -889,7 +894,7 @@ def test_failed_stop_attempts_guarded_deprovision(self): command_log=commands, ) - self.assertIn("deprovision", [command[3] for command in commands]) + self.assertIn("deprovision", [command[4] for command in commands]) fixture_root = Path(commands[0][commands[0].index("--state-dir") + 1]).parent self.assertIn(str(fixture_root), str(raised.exception)) self.assertTrue((fixture_root / "state" / "openvmm.log").is_file()) @@ -910,7 +915,7 @@ def test_failed_stop_attempts_guarded_deprovision(self): class _FakeSandboxCli: - """Emulates `nvx.py sandbox` and the OpenVMM processes that it starts. + """Emulates `nvx-rs sandbox` and the OpenVMM processes that it starts. Each fault names one regression of the public lifecycle that the acceptance must detect. @@ -941,14 +946,15 @@ def __call__( return self._dispatch(command) def _dispatch(self, command: list[str]) -> subprocess.CompletedProcess[bytes]: - # nvx.py runs `sandbox OPERATION`, and the acceptance runs nvx-rs - # itself, as `--root ROOT sandbox OPERATION`, under another artifact - # root. + # The acceptance runs `nvx-rs --root ROOT sandbox OPERATION`, under this + # checkout, or under another artifact root, whose runtime identity may + # differ from that of the sandbox's VM. position = command.index("sandbox") operation = command[position + 1] self.operations.append(operation) + root = command[command.index("--root") + 1] values: dict[str, str] = ( - {"--root": command[2]} if command[1] == "--root" else {} + {} if root == os.fspath(nvx_rs.ROOT) else {"--root": root} ) arguments: list[str] = [] index = position + 2 @@ -1058,7 +1064,7 @@ def _provision( return ( 2, b"", - b"nvx.py sandbox: error[argument-invalid]: argument --workload-user: " + b"nvx-rs sandbox: error[argument-invalid]: argument --workload-user: " b"--workload-user UID must be between 1 and 4294967294\n", ) if (state / "config.json").exists(): @@ -1489,6 +1495,10 @@ def fixture(prefix: str) -> str: "terminate_process", side_effect=fake.terminate, ), + # The fake's processes end at once, and name no process of this host. + patch.object( + sandbox_lifecycle_tests, "openvmm_threads_remain", return_value=False + ), ): sandbox_lifecycle_tests.run_sandbox_lifecycle( "whp", timeout=2, output_dir=output_dir @@ -1695,9 +1705,13 @@ def processes(marker: str) -> tuple[int, ...]: self.assertEqual(marker, "nvx-public-lifecycle-kill") return (child.pid,) if child.poll() is None else () + start_time = 1 + if sys.platform != "win32": + stat = Path(f"/proc/{child.pid}/stat").read_bytes() + start_time = int(stat[stat.rindex(b")") + 1 :].split()[19]) runtime: dict[str, object] = { "format": 4, - "process": {"pid": child.pid, "startTime": 1, "boot": "b" * 32}, + "process": {"pid": child.pid, "startTime": start_time, "boot": "b" * 32}, "endpoint": "unused", } try: @@ -1717,12 +1731,48 @@ def processes(marker: str) -> tuple[int, ...]: runtime, "nvx-public-lifecycle-kill" ) ) + self.assertFalse( + sandbox_lifecycle_tests.openvmm_threads_remain(runtime) + ) self.assertNotEqual(child.wait(timeout=30), 0) finally: if child.poll() is None: child.kill() child.wait(timeout=30) + @unittest.skipIf(sys.platform == "win32", "Windows has no /proc") + def test_a_killed_openvmm_runs_until_its_last_thread_exits(self): + # As nvx-rs tells, a process runs while a thread of it remains, even + # once its main thread is a zombie, whose command line is empty. + runtime: dict[str, object] = { + "process": {"pid": 4242, "startTime": 99, "boot": "b" * 32} + } + + def stat(state: str, start_time: int) -> bytes: + fields = [state, *["0"] * 18, str(start_time), *["0"] * 30] + return b"4242 (open vmm) " + " ".join(fields).encode() + + for line, threads, remain in ( + (stat("S", 99), 40, True), + (stat("Z", 99), 3, True), + (stat("Z", 99), 1, False), + (stat("X", 99), 1, False), + # Another process has the ID. + (stat("S", 100), 40, False), + ): + with ( + self.subTest(line=line, threads=threads), + patch.object(Path, "read_bytes", return_value=line), + patch.object( + sandbox_lifecycle_tests.os, "listdir", return_value=["t"] * threads + ), + ): + self.assertEqual( + sandbox_lifecycle_tests.openvmm_threads_remain(runtime), remain + ) + with patch.object(Path, "read_bytes", side_effect=FileNotFoundError): + self.assertFalse(sandbox_lifecycle_tests.openvmm_threads_remain(runtime)) + def test_control_endpoint_presence_follows_the_endpoint(self): if sys.platform == "win32": from multiprocessing.connection import Listener @@ -1777,7 +1827,7 @@ def test_process_scan_finds_openvmm_processes_naming_the_fixture(self): class _FakeProxySandboxCli(_FakeSandboxCli): - """Emulates `nvx.py sandbox` with the scenario's open and proxy-only + """Emulates `nvx-rs sandbox` with the scenario's open and proxy-only networks, and the guest's probes, whose traffic reaches the host's listeners as OpenVMM would carry it: from the gateway to host loopback, and elsewhere unchanged. @@ -1984,6 +2034,11 @@ def fixture(prefix: str) -> str: sandbox_proxy_tests, "host_address", return_value=self.HOST_ADDRESS ), patch.object(sandbox_proxy_tests, "ipv6_loopback", return_value=ipv6), + patch.object( + sandbox_lifecycle_tests.nvx_rs, + "require_binary", + return_value=Path("nvx-rs"), + ), ): sandbox_proxy_tests.run_sandbox_proxy_only( "kvm", timeout=2, output_dir=output_dir @@ -3123,7 +3178,7 @@ def test_scenario_is_a_default_correctness_scenario(self): class FilesystemCommaPathsScenarioTests(unittest.TestCase): - """The filesystem-comma-paths scenario with `nvx.py run` and the guest + """The filesystem-comma-paths scenario with `nvx-rs run` and the guest simulated.""" @staticmethod @@ -3157,6 +3212,7 @@ def guest( with ( tempfile.TemporaryDirectory() as temporary, patch.object(microvm_tests, "run_guest_script", side_effect=guest), + patch.object(nvx_rs, "require_binary", return_value=_FAKE_NVX_RS), ): microvm_tests.run_filesystem_comma_paths( "kvm", memory_mib=128, timeout=1.0, output_dir=Path(temporary) @@ -3171,9 +3227,7 @@ def test_scenario_shares_each_directory_alone_and_both_together(self): for (command, script), shares in zip( launched, ((alpha,), (reports,), (alpha, reports)), strict=True ): - self.assertEqual(command[0], sys.executable) - self.assertEqual(Path(command[1]).name, "nvx.py") - self.assertEqual(command[2], "run") + self.assertEqual(command[:4], _FAKE_PUBLIC_RUN) self.assertEqual(command[command.index("--hypervisor") + 1], "kvm") mounts = self._values(command, "--mount") denied = self._values(command, "--mount-deny") @@ -6885,6 +6939,7 @@ def is_alive(self) -> bool: "Thread", side_effect=ImmediateThread, ), + patch.object(nvx_rs, "require_binary", return_value=_FAKE_NVX_RS), ): wait = openvmm_process.return_value.__enter__.return_value.wait wait.side_effect = ( @@ -6910,13 +6965,10 @@ def is_alive(self) -> bool: ) policy_path = output_dir / "l3-l4-requested-policy.json" - nvx_path = str(Path(microvm_tests.__file__).parents[1] / "nvx.py") self.assertEqual( run_guest_script.call_args.args[0], [ - sys.executable, - nvx_path, - "run", + *_FAKE_PUBLIC_RUN, "--guest", guest, "--hypervisor", @@ -6950,7 +7002,7 @@ def is_alive(self) -> bool: encoding="utf-8" ) ) - self.assertEqual(results["interface"], "nvx.py run") + self.assertEqual(results["interface"], "nvx-rs run") self.assertEqual( results["observed"]["allowed"], ["tcp:start", "tcp:end", "udp:start", "udp:end"], @@ -7238,15 +7290,13 @@ def test_l3_l4_egress_protocols_run_every_case_through_public_nvx_run(self): ) ] self.assertEqual(run_guest_script.call_count, len(names)) - nvx_path = str(Path(microvm_tests.__file__).parents[1] / "nvx.py") + public_run = nvx_rs.command_line(("run",)) for call, name in zip(run_guest_script.call_args_list, names, strict=True): policy_path = output_dir / f"l3-l4-egress-{name}-policy.json" self.assertEqual( call.args[0], [ - sys.executable, - nvx_path, - "run", + *public_run, "--guest", "alpine", "--hypervisor", @@ -7724,9 +7774,7 @@ def test_l3_l4_egress_port_ranges_run_every_case_through_public_nvx_run(self): ] self.assertEqual(run_guest_script.call_count, len(names)) public_run = [ - sys.executable, - str(Path(microvm_tests.__file__).parents[1] / "nvx.py"), - "run", + *nvx_rs.command_line(("run",)), "--guest", "alpine", "--hypervisor", @@ -8294,15 +8342,13 @@ def test_ipv6_egress_policy_runs_every_case_through_public_nvx_run(self): 20001, ) self.assertEqual(run_guest_script.call_count, len(cases)) - nvx_path = str(Path(microvm_tests.__file__).parents[1] / "nvx.py") + public_run = nvx_rs.command_line(("run",)) for call, case in zip(run_guest_script.call_args_list, cases, strict=True): policy_path = output_dir / f"ipv6-egress-{case.name}-policy.json" self.assertEqual( call.args[0], [ - sys.executable, - nvx_path, - "run", + *public_run, "--guest", "alpine", "--hypervisor", diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index fc95766cf..9239bf4d4 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -88,7 +88,6 @@ CpuModel, HostCpu, cpu_profile_request_url, - host_cpu_profile_guidance, ) @@ -800,11 +799,7 @@ def test_guest_selection_includes_azure_linux(self): "initramfs-azurelinux.cpio.gz", ) - def test_run_help_and_usage_state_each_guest_memory_default(self): - stdout = io.StringIO() - with contextlib.redirect_stdout(stdout), self.assertRaises(SystemExit): - nvx.parse_args(["run", "--help"]) - help_text = " ".join(stdout.getvalue().split()) + def test_usage_states_each_guest_memory_default(self): usage = (BuildConstants.REPO_ROOT / "doc" / "usage.md").read_text( encoding="utf-8" ) @@ -817,37 +812,8 @@ def test_run_help_and_usage_state_each_guest_memory_default(self): for name in guests.GUEST_NAMES: default = guests.guest_descriptor(name).default_memory_mib with self.subTest(guest=name): - self.assertIn(f"{name} {default}", help_text) self.assertIn(f"{default} for {labels[name]}", row) - def test_run_help_is_the_help_of_nvx_rs_run(self): - # nvx.py run parses the options of nvx-rs run before it runs it, so it - # prints the help that the scenario tests of nvx-rs pin in - # run-help.txt, which gives the default memory of each guest that - # guests.py also gives test-microvm. - help_of_nvx_rs = ( - nvx_rs.DIRECTORY / "tests" / "fixtures" / "run-help.txt" - ).read_text(encoding="utf-8") - stdout = io.StringIO() - with ( - patch.dict(os.environ, {"COLUMNS": "80", "NO_COLOR": "1"}), - patch.object(sys, "argv", ["nvx.py"]), - contextlib.redirect_stdout(stdout), - self.assertRaises(SystemExit), - ): - nvx.parse_args(["run", "--help"]) - expected = help_of_nvx_rs.replace("nvx-rs run", "nvx.py run", 1) - if sys.version_info >= (3, 14): - self.assertEqual(stdout.getvalue(), expected) - else: - # Older releases of argparse wrap some of the help differently. - self.assertEqual(stdout.getvalue().split(), expected.split()) - help_text = " ".join(help_of_nvx_rs.split()) - for name in guests.GUEST_NAMES: - default = guests.guest_descriptor(name).default_memory_mib - with self.subTest(guest=name): - self.assertIn(f"{name} {default}", help_text) - def test_benchmark_exposes_device_restore_profile(self): args = nvx.parse_args( [ @@ -1218,9 +1184,6 @@ def test_guest_selection_cli_contract(self): self.assertEqual(distro.output, Path("ubuntu.erofs")) self.assertTrue(distro.replace) - with self.assertRaises(SystemExit): - nvx.parse_args(["run", "--guest", "all"]) - def test_docker_initramfs_rejects_native_guest(self): with self.assertRaisesRegex( common.ScriptError, "Alpine Linux.*do not require Docker" @@ -1495,111 +1458,6 @@ def test_record_openvmm_provenance_command_is_exposed(self): self.assertIs(args.handler, nvx.command_record_openvmm_provenance) -SCRIPTS_DIRECTORY = Path(__file__).parent -# A stand-in for nvx-rs that the process tests run in its place: it acts as its -# first argument asks. -FAKE_NVX_RS = r""" -import os -import signal -import sys -import time - -mode = sys.argv[1] -if mode == "streams": - os.write(1, b"standard output\n") - os.write(2, b"standard error\n") - sys.exit(int(sys.argv[2])) -if mode == "interruptible": - # Like nvx-rs, which records each SIGINT, report the first one only. - reported = [] - - def interrupted(signum, frame): - if not reported: - reported.append(signum) - os.write(2, b"Interrupted\n") - os._exit(130) - - signal.signal(signal.SIGINT, interrupted) - open(sys.argv[2], "w").close() - time.sleep(60) - sys.exit(99) -if mode == "signaled": - signum = getattr(signal, sys.argv[2]) - signal.signal(signum, signal.SIG_DFL) - os.kill(os.getpid(), signum) - time.sleep(60) -if mode == "dispositions": - # How this process started with the signals that nvx.py hands nvx-rs: - # Python handles SIGINT at startup unless it starts ignored. - for name in ("SIGINT", "SIGTERM", "SIGHUP"): - handler = signal.getsignal(getattr(signal, name)) - if handler == signal.SIG_IGN: - state = "ignored" - elif handler == signal.SIG_DFL: - state = "default" - else: - state = "handled" - os.write(1, f"{name} {state}\n".encode()) - sys.exit(0) -sys.exit(f"unknown mode {mode}") -""" -# Runs nvx_rs.run in a fresh Python, with the stand-in in place of nvx-rs. -NVX_RS_DRIVER = r""" -import sys - -sys.path.insert(0, sys.argv[1]) -from nvx_tools import nvx_rs - -fake = sys.argv[2] -nvx_rs._command = lambda arguments, program: [sys.executable, fake, *arguments] -raise SystemExit(nvx_rs.run(sys.argv[3:], program="nvx.py")) -""" -# Runs nvx.main in a fresh Python, with the stand-in in place of nvx-rs: the -# arguments before "--" tell the stand-in what to do. -NVX_PY_DRIVER = r""" -import sys - -sys.path.insert(0, sys.argv[1]) -import nvx -from nvx_tools import nvx_rs - -separator = sys.argv.index("--") -fake = [sys.executable, sys.argv[2], *sys.argv[3:separator]] -nvx_rs._command = lambda arguments, program: fake -raise SystemExit(nvx.main(sys.argv[separator + 1 :])) -""" -# Runs nvx.main as nvx.py in a fresh Python, with the nvx-rs that NVX_RS names -# and the artifact root that the second argument names. -NVX_PY_REAL_DRIVER = r""" -import sys -from pathlib import Path - -sys.path.insert(0, sys.argv[1]) -sys.argv[0] = "nvx.py" -import nvx -from nvx_tools import nvx_rs - -nvx_rs.ROOT = Path(sys.argv[2]) -raise SystemExit(nvx.main(sys.argv[3:])) -""" -# Runs nvx.main as NVX_PY_REAL_DRIVER does, on a host whose CPU, a Tiger Lake, -# no built-in CPU profile serves. -NVX_PY_UNSERVED_CPU_DRIVER = r""" -import sys -from pathlib import Path - -sys.path.insert(0, sys.argv[1]) -sys.argv[0] = "nvx.py" -import nvx -from nvx_tools import nvx_rs -from nvx_tools.time_abi import HostCpu - -nvx_rs.ROOT = Path(sys.argv[2]) -nvx.host_cpu_signature = lambda: HostCpu("GenuineIntel", 6, 140, 1) -raise SystemExit(nvx.main(sys.argv[3:])) -""" - - @contextlib.contextmanager def _without_nvx_rs_variable() -> Generator[None]: with patch.dict(os.environ): @@ -1613,29 +1471,7 @@ def _completed( return subprocess.CompletedProcess(["nvx-rs"], returncode, stdout, stderr) -def _signal_process_group(pid: int, name: str) -> None: - """Send the POSIX signal `name` to the process group `pid`, as a terminal - sends Ctrl-C to its foreground group.""" - killpg = cast(Callable[[int, int], None], getattr(os, "killpg")) # noqa: B009 - killpg(pid, cast(int, getattr(signal, name))) - - class NvxRsTests(unittest.TestCase): - def _fake(self, directory: Path) -> Path: - fake = directory / "fake_nvx_rs.py" - fake.write_text(FAKE_NVX_RS, encoding="utf-8") - return fake - - def _driver(self, fake: Path, *arguments: str) -> list[str]: - return [ - sys.executable, - "-c", - NVX_RS_DRIVER, - str(SCRIPTS_DIRECTORY), - str(fake), - *arguments, - ] - def test_nvx_rs_variable_names_the_executable(self): with tempfile.TemporaryDirectory() as temporary: binary = Path(temporary) / "nvx-rs" @@ -1957,214 +1793,37 @@ def test_failed_builds_report_cargo(self): nvx_rs.require_binary() record.assert_not_called() - def test_commands_name_the_program_and_the_repository_root(self): + def test_command_lines_run_nvx_rs_on_the_checkout_or_another_root(self): + # The microVM scenarios run nvx-rs itself, on this checkout's + # artifacts, or on another artifact root, such as one of another + # runtime identity. binary = Path("bin") / "nvx-rs" with patch.object(nvx_rs, "require_binary", return_value=binary): self.assertEqual( - nvx_rs._command(["sandbox", "exec"], "nvx.py"), + nvx_rs.command_line(("sandbox", "exec")), [ os.fspath(binary), - "--prog", - "nvx.py", "--root", os.fspath(BuildConstants.REPO_ROOT), "sandbox", "exec", ], ) + other = Path("other-runtime") self.assertEqual( - nvx_rs._command(["artifacts"], None), - [ - os.fspath(binary), - "--root", - os.fspath(BuildConstants.REPO_ROOT), - "artifacts", - ], + nvx_rs.command_line(("run",), root=other), + [os.fspath(binary), "--root", os.fspath(other), "run"], ) - def test_exit_statuses_pass_through(self): - for returncode, status in ((0, 0), (1, 1), (2, 2), (130, 130), (255, 255)): - with self.subTest(returncode=returncode): - self.assertEqual(nvx_rs.exit_status(returncode), status) - # A signal that ends nvx-rs gives the status that a shell reports. - self.assertEqual(nvx_rs.exit_status(-15), 143) - self.assertEqual(nvx_rs.exit_status(-9), 137) - if os.name == "nt": - # Windows statuses are 32-bit; SystemExit takes them signed. - self.assertEqual(nvx_rs.exit_status(0xC000013A), -0x3FFFFEC6) - - def test_run_shares_the_standard_streams_and_passes_the_status_through(self): - with tempfile.TemporaryDirectory() as temporary: - fake = self._fake(Path(temporary)) - for status in (0, 1, 2, 7): - with self.subTest(status=status): - result = subprocess.run( - self._driver(fake, "streams", str(status)), - capture_output=True, - timeout=60, - ) - self.assertEqual(result.returncode, status) - self.assertEqual(result.stdout, b"standard output\n") - self.assertEqual(result.stderr, b"standard error\n") - - @unittest.skipIf(os.name == "nt", "Windows has no POSIX signals") - def test_ctrl_c_ends_the_command_as_nvx_rs_ends_it(self): - # A terminal's Ctrl-C reaches the whole process group, and a SIGINT sent - # to nvx.py alone reaches nvx-rs through nvx.py. Either way nvx-rs - # prints Interrupted, and nvx.py neither prints it again nor changes - # the status. - for group in (True, False): - with ( - self.subTest(group=group), - tempfile.TemporaryDirectory() as temporary, - ): - fake = self._fake(Path(temporary)) - started = Path(temporary) / "started" - process = subprocess.Popen( - self._driver(fake, "interruptible", str(started)), - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - start_new_session=True, - ) - try: - deadline = time.monotonic() + 30 - while not started.exists(): - self.assertLess(time.monotonic(), deadline) - self.assertIsNone(process.poll()) - time.sleep(0.02) - if group: - _signal_process_group(process.pid, "SIGINT") - else: - process.send_signal(signal.SIGINT) - stdout, stderr = process.communicate(timeout=30) - finally: - if process.poll() is None: - _signal_process_group(process.pid, "SIGKILL") - process.wait() - self.assertEqual(process.returncode, 130) - self.assertEqual(stderr, b"Interrupted\n") - self.assertEqual(stdout, b"") - - @unittest.skipIf(os.name == "nt", "Windows has no POSIX signals") - def test_signals_that_end_nvx_rs_give_shell_statuses(self): - with tempfile.TemporaryDirectory() as temporary: - fake = self._fake(Path(temporary)) - for name, status, stderr in ( - ("SIGTERM", 143, b""), - # SIGINT ended nvx-rs before it could print Interrupted. - ("SIGINT", 130, b"Interrupted\n"), - ): - with self.subTest(signal=name): - result = subprocess.run( - self._driver(fake, "signaled", name), - capture_output=True, - timeout=60, - ) - self.assertEqual(result.returncode, status) - self.assertEqual(result.stderr, stderr) - - def test_an_interrupted_wait_still_waits_for_nvx_rs(self): - process = MagicMock() - process.wait.side_effect = [KeyboardInterrupt, 130] - stderr = io.StringIO() - with ( - patch.object(nvx_rs, "_command", return_value=["nvx-rs", "sandbox"]), - patch("subprocess.Popen", return_value=process) as popen, - contextlib.redirect_stderr(stderr), - ): - self.assertEqual(nvx_rs.run(["sandbox"], program="nvx.py"), 130) - # The standard streams are shared with nvx-rs. - popen.assert_called_once_with(["nvx-rs", "sandbox"]) - self.assertEqual(process.wait.call_count, 2) - self.assertEqual(stderr.getvalue(), "") - if os.name != "nt": - process.send_signal.assert_called_once_with(signal.SIGINT) - - def test_signal_handlers_change_only_while_nvx_rs_runs(self): - # nvx.py handles SIGINT, unless it started with SIGINT ignored. - previous = signal.signal(signal.SIGINT, signal.default_int_handler) - self.addCleanup(signal.signal, signal.SIGINT, previous) - before = signal.getsignal(signal.SIGINT) - during: list[object] = [] - process = MagicMock() - - def wait() -> int: - during.append(signal.getsignal(signal.SIGINT)) - return 0 - - process.wait.side_effect = wait - with ( - patch.object(nvx_rs, "_command", return_value=["nvx-rs"]), - patch("subprocess.Popen", return_value=process), - ): - self.assertEqual(nvx_rs.run([], program="nvx.py"), 0) - self.assertEqual(len(during), 1) - self.assertIsNot(during[0], before) - self.assertIs(signal.getsignal(signal.SIGINT), before) - - def test_ignored_signals_stay_ignored_while_nvx_rs_runs(self): - # A handler would leave nvx-rs the default disposition at exec, so a - # signal that nvx.py ignores keeps no handler, and nvx-rs inherits it - # ignored. - previous = signal.signal(signal.SIGINT, signal.SIG_IGN) - self.addCleanup(signal.signal, signal.SIGINT, previous) - during: list[object] = [] - process = MagicMock() - - def wait() -> int: - during.append(signal.getsignal(signal.SIGINT)) - return 0 - - process.wait.side_effect = wait - with ( - patch.object(nvx_rs, "_command", return_value=["nvx-rs"]), - patch("subprocess.Popen", return_value=process), - ): - self.assertEqual(nvx_rs.run([], program="nvx.py"), 0) - self.assertEqual(during, [signal.SIG_IGN]) - self.assertIs(signal.getsignal(signal.SIGINT), signal.SIG_IGN) - - @unittest.skipIf(os.name == "nt", "Windows has no POSIX signals") - def test_nvx_rs_inherits_the_signals_that_nvx_py_ignores(self): - # A shell without job control starts a background command with SIGINT - # ignored, and nohup starts one with SIGHUP ignored. nvx-rs inherits - # them ignored, as it would without nvx.py, and so leaves SIGINT alone. - dispositions = ( - "import signal\n" - "signal.signal(signal.SIGINT, signal.{})\n" - "signal.signal(signal.SIGTERM, signal.SIG_DFL)\n" - "signal.signal(signal.SIGHUP, signal.{})\n" - ) - with tempfile.TemporaryDirectory() as temporary: - fake = self._fake(Path(temporary)) - for ignored, (sigint, sighup), expected in ( - ( - False, - ("default_int_handler", "SIG_DFL"), - b"SIGINT handled\nSIGTERM default\nSIGHUP default\n", - ), - ( - True, - ("SIG_IGN", "SIG_IGN"), - b"SIGINT ignored\nSIGTERM default\nSIGHUP ignored\n", - ), - ): - command = self._driver(fake, "dispositions") - command[2] = dispositions.format(sigint, sighup) + command[2] - with self.subTest(ignored=ignored): - result = subprocess.run(command, capture_output=True, timeout=60) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertEqual(result.stdout, expected) - def test_unstartable_nvx_rs_is_an_error(self): with ( - patch.object(nvx_rs, "_command", return_value=["nvx-rs"]), - patch("subprocess.Popen", side_effect=OSError("no such file")), + patch.object(nvx_rs, "require_binary", return_value=Path("nvx-rs")), + patch("subprocess.run", side_effect=OSError("no such file")), self.assertRaisesRegex( common.ScriptError, "failed to run nvx-rs: no such file" ), ): - nvx_rs.run([], program="nvx.py") + nvx_rs.query(["artifacts"]) def test_queries_report_the_codes_and_messages_of_nvx_rs(self): for stderr, code, message in ( @@ -2186,7 +1845,7 @@ def test_queries_report_the_codes_and_messages_of_nvx_rs(self): ): with ( self.subTest(stderr=stderr), - patch.object(nvx_rs, "_command", return_value=["nvx-rs"]), + patch.object(nvx_rs, "require_binary", return_value=Path("nvx-rs")), patch("subprocess.run", return_value=_completed(1, stderr=stderr)), self.assertRaises(nvx_rs.NvxRsError) as raised, ): @@ -2195,11 +1854,18 @@ def test_queries_report_the_codes_and_messages_of_nvx_rs(self): self.assertEqual(raised.exception.code, code) self.assertIsInstance(raised.exception, common.ScriptError) with ( - patch.object(nvx_rs, "_command", return_value=["nvx-rs"]), - patch("subprocess.run", return_value=_completed(0, stdout=b"{")), + patch.object(nvx_rs, "require_binary", return_value=Path("nvx-rs")), + patch("subprocess.run", return_value=_completed(0, stdout=b"{")) as run, self.assertRaisesRegex(common.ScriptError, "printed invalid JSON"), ): nvx_rs.query(["artifacts"]) + # A query runs nvx-rs on this checkout's artifacts. + run.assert_called_once_with( + ["nvx-rs", "--root", os.fspath(BuildConstants.REPO_ROOT), "artifacts"], + stdin=subprocess.DEVNULL, + capture_output=True, + check=False, + ) def test_reported_errors_are_the_last_that_nvx_rs_printed(self): # A workload's output precedes the error of sandbox exec. @@ -2235,24 +1901,13 @@ def test_reported_errors_are_the_last_that_nvx_rs_printed(self): ) self.assertEqual( nvx_rs.reported_error( - b"nvx.py sandbox: error[argument-invalid]: argument --layer: " + b"nvx-rs sandbox: error[argument-invalid]: argument --layer: " b"error[x]: y.erofs\n" ), nvx_rs.ReportedError( "argument-invalid", "argument --layer: error[x]: y.erofs" ), ) - # A program name may hold colons, but no brackets, which nvx-rs refuses. - self.assertEqual( - nvx_rs.reported_error( - b"usage: a: b egress compile [-h] PATH\n" - b"a: b egress compile: error[argument-missing]: the following " - b"arguments are required: PATH\n" - ), - nvx_rs.ReportedError( - "argument-missing", "the following arguments are required: PATH" - ), - ) # nvx-rs prints a line break of a value that a message names as `\n`, # so that the message holds no line in the form of an error. self.assertEqual( @@ -2267,7 +1922,8 @@ def test_reported_errors_are_the_last_that_nvx_rs_printed(self): ) self.assertEqual( nvx_rs.reported_error( - "usage: nvx.py sandbox [-h]\r\nnvx.py sandbox: error[argument-invalid]: " + "usage: nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] " + "[OPTIONS]\r\nnvx-rs sandbox: error[argument-invalid]: " "argument --workload-user: --workload-user UID must be between 1 and " "4294967294\r\n" ), @@ -2313,6 +1969,56 @@ def table(path: Path) -> list[tuple[str, str]]: with self.subTest(code=code): self.assertEqual(status, "2" if code.startswith("argument-") else "1") + @unittest.skipUnless( + os.environ.get("NVX_RS"), "set NVX_RS to an nvx-rs executable to run" + ) + def test_usage_documents_every_option_of_nvx_rs(self): + # The nvx-rs chapter of doc/usage.md documents each option of nvx-rs + # run and sandbox, as the rest documents those of each nvx.py command. + option = re.compile(r"(? subprocess.CompletedProcess[bytes]: - return subprocess.run( - [ - sys.executable, - "-c", - NVX_PY_REAL_DRIVER, - str(SCRIPTS_DIRECTORY), - str(root), - *arguments, - ], - cwd=root, - capture_output=True, - timeout=60, - ) - - result = nvx_py( - "sandbox", - "--dry", - "--layer", - f"distro,distro.erofs,{uuid}", - "--scratch", - "scratch.ext4", - "--arg=-c", - "--arg", - "true", - "--mount", - f"/data,{root / 'share'},rw", - "--mount-deny", - "hidden", - "--hyp", - "kvm", - "--memory-max", - "268435456", - ) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertEqual(result.stderr, b"") - command = result.stdout.decode("utf-8") - self.assertTrue(command.startswith(">> "), command) - # A relative policy path names a path in the share's host directory, - # and OpenVMM receives the path to which it resolves. - hidden = str(root / "share" / "hidden") - quoted = ( - subprocess.list2cmdline([hidden]) - if os.name == "nt" - else shlex.quote(hidden) - ) - for expected in ( - "--microvm-sandbox-block distro:file:distro.erofs,ro", - "--microvm-sandbox-block scratch:file:scratch.ext4", - "--hypervisor kvm", - "nvx_arg=-c nvx_arg=true nvx_memory_max=268435456", - f"--mount-deny {quoted}", - ): - self.assertIn(expected, command) - # A value that nvx-rs refuses is a usage error of nvx.py sandbox. - result = nvx_py("sandbox", "--workload-user", "0:0") - self.assertEqual(result.returncode, 2) - stderr = result.stderr.decode("utf-8").replace("\r\n", "\n") - self.assertTrue(stderr.startswith("usage: nvx.py sandbox [-h] "), stderr) - reported = nvx_rs.reported_error(result.stderr) - assert reported is not None, stderr - self.assertEqual(reported.code, "argument-invalid") - self.assertIn("--workload-user", reported.message) - self.assertIn("\nnvx.py sandbox: error[argument-invalid]: ", stderr) - result = nvx_py("sandbox", "exec", "--state-dir", "state") - self.assertEqual(result.returncode, 1) - reported = nvx_rs.reported_error(result.stderr) - assert reported is not None, result.stderr - self.assertEqual(reported.code, "sandbox-not-provisioned") - self.assertIn(str(root / "state"), reported.message) - - -def _artifact_root(directory: Path) -> Path: - """Create the artifacts of a cold boot below `directory`, which nvx-rs - checks but does not read, and return its resolved path.""" - root = directory.resolve() - openvmm = f"openvmm/target/release/openvmm{'.exe' if os.name == 'nt' else ''}" - for relative in ("build/vmlinux", "build/initramfs.cpio.gz", openvmm): - path = root / relative - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(b"artifact") - return root - - -def _run_nvx_py( - driver: str, - root: Path, - *arguments: str, - environment: Mapping[str, str] | None = None, -) -> subprocess.CompletedProcess[bytes]: - """Run nvx.py ARGUMENTS with `driver` and the artifact root `root`.""" - return subprocess.run( - [sys.executable, "-c", driver, str(SCRIPTS_DIRECTORY), str(root), *arguments], - cwd=root, - env={**os.environ, **(environment or {})}, - capture_output=True, - timeout=60, - check=False, - ) - - -def _text(data: bytes) -> str: - return data.decode("utf-8").replace("\r\n", "\n") - - -class RunDelegationTests(unittest.TestCase): - TIGER_LAKE = HostCpu("GenuineIntel", 6, 140, 1, "Intel(R) Core(TM) i7-1185G7") - - def setUp(self) -> None: - super().setUp() - # On a host whose CPU a built-in profile serves, nvx.py passes no - # guidance, and looks up no artifact. - patcher = patch.object( - nvx, "host_cpu_signature", return_value=HostCpu("GenuineIntel", 6, 154, 3) - ) - patcher.start() - self.addCleanup(patcher.stop) - - def test_run_records_canonical_options_in_command_line_order(self): - args = nvx.parse_args( - [ - "run", - "--gu", - "ubuntu", - "--memory-m", - "1_024", - "--memory-c=2048", - "--proc", - " 2 ", - "--cpu", - "host", - "--mount=/a,dir-a", - "--mount-d", - "x", - "--mount", - "/b,dir-b,rw", - "--mount-w=y", - "--mount-o", - "vmm", - "--network-prof", - "portable", - "--net", - "10.0.0.2/24", - "--network-egress-p", - "policy.json", - "--out", - "report.json", - "--cmdline=-x", - "--restore-s", - "snapshot", - "--restore-p=1", - "--dry", - "--hyp", - "kvm", - "--memory-mib=-5", - ] - ) - # Abbreviations are resolved, the order is kept, and every value is - # attached, so that a value that starts with "-" stays a value. - self.assertEqual( - args.nvx_rs_arguments, - [ - "--guest=ubuntu", - "--memory-mib=1_024", - "--memory-capacity-mib=2048", - "--processors= 2 ", - "--cpu-profile=host", - "--mount=/a,dir-a", - "--mount-deny=x", - "--mount=/b,dir-b,rw", - "--mount-write=y", - "--mount-owner=vmm", - "--network-profile=portable", - "--net=10.0.0.2/24", - "--network-egress-policy-file=policy.json", - "--outcome-report=report.json", - "--cmdline=-x", - "--restore-snapshot=snapshot", - "--restore-processors=1", - "--dry-run", - "--hypervisor=kvm", - "--memory-mib=-5", - ], - ) - # nvx-rs converts, checks, and defaults every value; nvx.py keeps the - # ones that its CPU-profile guidance reads, as given. - self.assertIs(args.handler, nvx.command_run) - self.assertEqual( - (args.cpu_profile, args.restore_snapshot, args.dry_run, args.hypervisor), - ("host", "snapshot", True, "kvm"), - ) - default = nvx.parse_args(["run"]) - self.assertEqual(default.nvx_rs_arguments, []) - for name in ("guest", "memory_mib", "processors", "cmdline", "dry_run"): - self.assertFalse(hasattr(default, name), name) - - def test_run_runs_nvx_rs_and_exits_with_its_status(self): - program = argparse.ArgumentParser().prog - with patch.object(nvx_rs, "run", return_value=7) as run: - self.assertEqual( - nvx.main(["run", "--guest", "ubuntu", "--processors", "3"]), 7 - ) - # argparse no longer converts the processor count, which nvx-rs - # converts and refuses itself. - run.assert_called_once_with( - ["run", "--guest=ubuntu", "--processors=3"], program=program - ) - - def test_run_usage_errors_stay_with_argparse(self): - for arguments in ( - ["run", "--bogus"], - ["run", "extra"], - ["run", "--guest", "debian"], - ["run", "--machine", "microvm-v2"], - ["run", "--mount-owner", "root"], - ["run", "--network-egress", "block"], - ["run", "--memory-mib"], - ["run", "--dry-run=1"], - ): - with ( - self.subTest(arguments=arguments), - patch.object(nvx_rs, "run") as run, - patch("sys.stderr", new_callable=io.StringIO), - self.assertRaises(SystemExit) as raised, - ): - nvx.main(arguments) - self.assertEqual(raised.exception.code, 2) - run.assert_not_called() - - def test_run_reports_a_missing_nvx_rs_as_an_error(self): - stderr = io.StringIO() - with ( - patch.object( - nvx_rs, "require_binary", side_effect=common.ScriptError("no nvx-rs") - ), - contextlib.redirect_stderr(stderr), - ): - self.assertEqual(nvx.main(["run", "--dry-run"]), 1) - self.assertEqual(stderr.getvalue(), "error: no nvx-rs\n") - - def _guidance( - self, - host: HostCpu | None, - arguments: list[str], - openvmm: Path = Path("/nvx/openvmm/target/release/openvmm"), - ) -> tuple[list[str], int]: - """Run `nvx.py run ARGUMENTS` on the CPU `host`, and return the - guidance options that nvx.py passes nvx-rs, and how many times it - looked up the artifacts.""" - artifacts = nvx_rs.ArtifactPaths( - openvmm=openvmm, kernel=Path("vmlinux"), guests=() - ) - with ( - patch.object(nvx, "host_cpu_signature", return_value=host), - patch.object(nvx_rs, "artifacts", return_value=artifacts) as lookup, - patch.object(nvx_rs, "run", return_value=1) as run, - ): - self.assertEqual(nvx.main(["run", *arguments]), 1) - passed = cast(list[str], run.call_args.args[0]) - guidance = [ - argument - for argument in passed - if argument.startswith("--cold-boot-guidance=") - ] - # The guidance follows the options that the user gave. - self.assertEqual(passed[: len(passed) - len(guidance)][0], "run") - self.assertEqual(passed[len(passed) - len(guidance) :], guidance) - return guidance, lookup.call_count - - def test_run_passes_the_cpu_profile_guidance_only_where_it_applies(self): - openvmm = Path("/nvx/openvmm/target/release/openvmm") - hygon = HostCpu("HygonGenuine", 24, 0, 1) - - def expected(cpu_profile: str | None, host: HostCpu, hypervisor: str) -> str: - guidance = host_cpu_profile_guidance( - cpu_profile, - host, - nvx._format_command([os.fspath(openvmm)]), - hypervisor, - ) - assert guidance is not None - return f"--cold-boot-guidance={guidance}" - - default = nvx._hypervisor("auto") - # A cold boot with auto on a CPU that no built-in profile serves, or - # with host on a CPU that host profiles do not serve either, gets the - # guidance, with the OpenVMM that nvx-rs runs and the backend. - for host, arguments, guidance in ( - (self.TIGER_LAKE, [], expected(None, self.TIGER_LAKE, default)), - ( - self.TIGER_LAKE, - ["--cpu-profile", "auto", "--hypervisor", "mshv"], - expected("auto", self.TIGER_LAKE, "mshv"), - ), - (hygon, [], expected(None, hygon, default)), - (hygon, ["--cpu-profile", "host"], expected("host", hygon, default)), - ): - with self.subTest(host=host, arguments=arguments): - self.assertEqual( - self._guidance(host, arguments, openvmm), ([guidance], 1) - ) - # A dry run, a restore, another profile, and a CPU that a built-in - # profile serves or that nvx.py cannot identify get none, and nvx.py - # does not look up the artifacts for them. - for host, arguments in ( - (self.TIGER_LAKE, ["--dry-run"]), - (self.TIGER_LAKE, ["--restore-snapshot", "snapshot"]), - (self.TIGER_LAKE, ["--restore-snapshot="]), - (self.TIGER_LAKE, ["--cpu-profile", "host"]), - (self.TIGER_LAKE, ["--cpu-profile", "intel.alderlake.v1"]), - (HostCpu("GenuineIntel", 6, 154, 3), []), - (HostCpu("AuthenticAMD", 25, 1, 1), []), - (None, []), - ): - with self.subTest(host=host, arguments=arguments): - self.assertEqual(self._guidance(host, arguments), ([], 0)) - - def test_run_quotes_the_openvmm_path_in_the_cpu_profile_guidance(self): - # The guidance's fingerprint command is for copying, so a path with a - # space is quoted as the shell of this platform reads it. - openvmm = Path("/nvx checkout") / "openvmm" - guidance, _ = self._guidance( - self.TIGER_LAKE, ["--hypervisor", "kvm"], openvmm=openvmm - ) - quoted = ( - subprocess.list2cmdline([str(openvmm)]) - if os.name == "nt" - else shlex.quote(str(openvmm)) - ) - self.assertNotEqual(quoted, str(openvmm)) - self.assertIn( - f"nvx: {quoted} --hypervisor kvm --cpu-fingerprint fingerprint.json\n", - guidance[0], - ) - - @unittest.skipUnless( - os.environ.get("NVX_RS"), "set NVX_RS to an nvx-rs executable to run" - ) - def test_the_real_nvx_rs_runs_the_options_that_nvx_py_reads(self): - with tempfile.TemporaryDirectory() as temporary: - root = _artifact_root(Path(temporary)) - result = _run_nvx_py( - NVX_PY_REAL_DRIVER, - root, - "run", - "--dry", - "--memory-m=256", - "--cmdline=-x", - "--hyp", - "kvm", - ) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertEqual(result.stderr, b"") - command = _text(result.stdout) - self.assertTrue(command.startswith(">> "), command) - for expected in ("--hypervisor kvm --memory 256M", "--cmdline -x\n"): - self.assertIn(expected, command) - # A value that nvx-rs refuses is a usage error of nvx.py run. - result = _run_nvx_py( - NVX_PY_REAL_DRIVER, root, "run", "--processors", "3", "--dry-run" - ) - self.assertEqual(result.returncode, 2) - stderr = _text(result.stderr) - self.assertTrue(stderr.startswith("usage: nvx.py run [-h] "), stderr) - self.assertIn("\nnvx.py run: error[argument-invalid]: ", stderr) - reported = nvx_rs.reported_error(result.stderr) - assert reported is not None, stderr - self.assertEqual(reported.code, "argument-invalid") - self.assertIn("--processors", reported.message) - result = _run_nvx_py( - NVX_PY_REAL_DRIVER, root, "run", "--net", "10.0.0.2/24", "--dry-run" - ) - self.assertEqual(result.returncode, 1) - reported = nvx_rs.reported_error(result.stderr) - assert reported is not None, result.stderr - self.assertEqual(reported.code, "unsupported") - for option in ("--net", "--network-profile"): - self.assertIn(option, reported.message) - - @unittest.skipUnless( - os.environ.get("NVX_RS"), "set NVX_RS to an nvx-rs executable to run" - ) - @unittest.skipIf(os.name == "nt", "needs an OpenVMM shell script") - def test_the_real_nvx_rs_guides_a_failed_cold_boot(self): - with tempfile.TemporaryDirectory() as temporary: - root = _artifact_root(Path(temporary)) - openvmm = root / "failing openvmm" - openvmm.write_text("#!/bin/sh\nexit 1\n", encoding="utf-8") - openvmm.chmod(0o755) - environment = {"NVX_OPENVMM": str(openvmm)} - - def run(*arguments: str) -> subprocess.CompletedProcess[bytes]: - return _run_nvx_py( - NVX_PY_UNSERVED_CPU_DRIVER, - root, - "run", - "--hypervisor", - "kvm", - *arguments, - environment=environment, - ) - - # OpenVMM fails the cold boot, and the guidance follows, with the - # OpenVMM that NVX_OPENVMM names. - result = run() - self.assertEqual(result.returncode, 1, result.stderr) - self.assertTrue( - _text(result.stdout).startswith(f">> {shlex.quote(str(openvmm))} "), - result.stdout, - ) - stderr = _text(result.stderr) - self.assertTrue( - stderr.startswith( - "nvx: no built-in CPU profile serves this host's CPU, " - "GenuineIntel 6/140/1, so --cpu-profile auto" - ), - stderr, - ) - self.assertIn( - f"nvx: {shlex.quote(str(openvmm))} --hypervisor kvm " - "--cpu-fingerprint fingerprint.json\n", - stderr, - ) - # A restore that fails, a dry run, and a refused option get none. - for arguments, status in ( - (["--restore-snapshot", str(root / "snapshot")], 1), - (["--dry-run"], 0), - (["--net", "10.0.0.2/24"], 1), - ): - with self.subTest(arguments=arguments): - result = run(*arguments) - self.assertEqual(result.returncode, status, result.stderr) - self.assertNotIn(b"nvx: ", result.stderr) @unittest.skipUnless( os.environ.get("NVX_RS"), "set NVX_RS to an nvx-rs executable to run" @@ -3090,8 +2099,8 @@ def run(*arguments: str) -> subprocess.CompletedProcess[bytes]: def test_run_leaves_openvmm_the_terminal_to_restore(self): """A guest-triggered shutdown can end OpenVMM while its console holds the terminal in raw mode, and OpenVMM restores the terminal settings - only when its standard error is a terminal, so nvx.py run, and the - nvx-rs run that it runs, must leave it one.""" + only when its standard error is a terminal, so nvx-rs run must leave + it one.""" if sys.platform != "linux": self.skipTest("needs a Linux pseudo-terminal") import pty @@ -3118,10 +2127,8 @@ def test_run_leaves_openvmm_the_terminal_to_restore(self): before = termios.tcgetattr(secondary) completed = subprocess.run( [ - sys.executable, - "-c", - NVX_PY_REAL_DRIVER, - str(SCRIPTS_DIRECTORY), + os.environ["NVX_RS"], + "--root", str(root), "run", "--restore-snapshot", @@ -3130,6 +2137,7 @@ def test_run_leaves_openvmm_the_terminal_to_restore(self): stdin=secondary, stdout=secondary, stderr=secondary, + cwd=root, env={**os.environ, "NVX_OPENVMM": str(openvmm)}, timeout=60, check=False, @@ -7820,7 +6828,9 @@ def reads(step: dict[str, Any], context: Mapping[str, object]) -> set[str]: if name in guest_files } for command in re.findall( - r"(?:nvx\.py|Invoke-Nvx)\s+([a-z][a-z-]*)", script + r"(?:nvx\.py|nvx-rs[/\\]target[/\\]release[/\\]nvx-rs(?:\.exe)?" + r"|Invoke-Nvx)\s+([a-z][a-z-]*)", + script, ): if command == "test-microvm": guest = re.search(r"--guest (\S+)", script) @@ -12386,43 +11396,49 @@ def test_ci_validates_the_cli_once_per_os_after_checking_out_openvmm(self): f" needs.{cli}.result == 'success' &&\n", job ) - def test_ci_compares_openvmms_protocol_vectors_in_the_cli_jobs(self): + def test_ci_compares_the_copies_of_openvmm_data_in_the_cli_jobs(self): # nvx-rs's protocol tests read copies of OpenVMM's control-session - # protocol vectors, because the nvx-rs job cannot check out OpenVMM. - # Each NVX CLI job checks it out, and runs the ignored test that - # compares the copies with the submodule's files. + # protocol vectors, and nvx-rs run a copy of the identities of its + # built-in CPU profiles, because the nvx-rs job cannot check out + # OpenVMM. Each NVX CLI job checks it out, and runs the ignored tests + # that compare the copies with the submodule's files. workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") - test = BuildConstants.REPO_ROOT / "nvx-rs" / "tests" / "openvmm_vectors.rs" - self.assertIn("#[ignore", test.read_text(encoding="utf-8")) - step_name = "Compare OpenVMM's protocol vectors" - for job_name in ci.REQUIRED_CI_CLI_JOBS: - with self.subTest(job=job_name): - job = _workflow_job(workflow, job_name) - step = _workflow_step(workflow, job_name, step_name) - command = " ".join(_yaml_scalar(step, "run").replace("`", " ").split()) - for argument in ( - "cargo test --locked", - "--manifest-path nvx-rs/Cargo.toml", - "--test openvmm_vectors -- --ignored", - ): - self.assertIn(argument, command) - for prerequisite in ( - "uses: ./.github/actions/checkout-openvmm", - "rustup toolchain install", - ): - self.assertLess( - job.index(prerequisite), job.index(f"- name: {step_name}") + for step_name, test_name in ( + ("Compare OpenVMM's protocol vectors", "openvmm_vectors"), + ("Compare OpenVMM's CPU profiles", "openvmm_cpu_profiles"), + ): + test = BuildConstants.REPO_ROOT / "nvx-rs" / "tests" / f"{test_name}.rs" + self.assertIn("#[ignore", test.read_text(encoding="utf-8")) + for job_name in ci.REQUIRED_CI_CLI_JOBS: + with self.subTest(job=job_name, step=step_name): + job = _workflow_job(workflow, job_name) + step = _workflow_step(workflow, job_name, step_name) + command = " ".join( + _yaml_scalar(step, "run").replace("`", " ").split() ) + for argument in ( + "cargo test --locked", + "--manifest-path nvx-rs/Cargo.toml", + f"--test {test_name} -- --ignored", + ): + self.assertIn(argument, command) + for prerequisite in ( + "uses: ./.github/actions/checkout-openvmm", + "rustup toolchain install", + ): + self.assertLess( + job.index(prerequisite), + job.index(f"- name: {step_name}"), + ) def test_ci_hands_nvx_rs_to_the_microvm_and_platform_jobs(self): - # nvx.py runs nvx-rs for its sandbox command and for run's egress and - # artifact steps, and finds it where `cargo build --release` puts it. - # Each NVX CLI job builds it for its OS, records its provenance, tests - # with it, and hands both to the microVM jobs, which run it, and to the - # platform jobs, which package it, as the OpenVMM build jobs hand - # over OpenVMM. + # The microVM scenarios and steps run nvx-rs, and find it where + # `cargo build --release` puts it. Each NVX CLI job builds it for its + # OS, records its provenance, tests with it, and hands both to the + # microVM jobs, which run it, and to the platform jobs, which package + # it, as the OpenVMM build jobs hand over OpenVMM. github = BuildConstants.REPO_ROOT / ".github" workflow = (github / "workflows" / "ci.yml").read_text(encoding="utf-8") microvm = (github / "workflows" / "run-nvx-microvm-tests.yml").read_text( @@ -26495,10 +25511,10 @@ def test_release_archive_installs_runtime_layout(self): run.assert_not_called() def test_release_archive_leaves_nvx_rs_of_other_sources_to_cargo(self): - # nvx.py runs the nvx-rs that download installs without rebuilding it, - # so download installs only the build of the checkout's nvx-rs sources - # that nvx.py runs, and removes an earlier one and its provenance, so - # that nvx.py builds nvx-rs from the checkout instead. Releases that + # The checkout runs the nvx-rs that download installs without + # rebuilding it, so download installs only the build of the checkout's + # nvx-rs sources, and removes an earlier one and its provenance, so + # that Cargo builds nvx-rs from the checkout instead. Releases that # predate nvx-rs install too. name = self.nvx_rs_path.name archive_name = ( @@ -26554,8 +25570,10 @@ def test_release_archive_leaves_nvx_rs_of_other_sources_to_cargo(self): self.assertFalse(stale_provenance.exists()) self.assertIn(reason, output) self.assertIn( - "so nvx.py builds nvx-rs from this checkout with Cargo when a " - "command needs it, unless NVX_RS names one", + "so build nvx-rs from this checkout with `cargo build " + "--release --locked --features cli --manifest-path " + "nvx-rs/Cargo.toml --target-dir nvx-rs/target`, as the " + "microVM tests do when they need it, unless NVX_RS names one", output, ) # Without Cargo, nvx.py names the ways to get nvx-rs. diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 4450c670c..ad8173f90 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -415,119 +415,6 @@ def test_tells_catalog_profiles_from_host_profiles(self): self.assertEqual(time_abi.is_catalog_profile_id(profile_id), catalog) self.assertEqual(time_abi.is_host_profile_id(profile_id), host) - def test_guides_hosts_that_no_built_in_profile_serves(self): - def guide(cpu_profile: str | None, host: time_abi.HostCpu | None) -> str | None: - return time_abi.host_cpu_profile_guidance( - cpu_profile, host, "/nvx/openvmm", "kvm" - ) - - tiger_lake = time_abi.HostCpu( - "GenuineIntel", 6, 140, 1, "11th Gen Intel(R) Core(TM) i7-1185G7" - ) - guidance = guide(None, tiger_lake) - assert guidance is not None - lines = guidance.splitlines() - self.assertTrue(all(line.startswith("nvx: ") for line in lines), guidance) - # auto falls back on the CPU, so the guidance says what auto does there, - # not which code ended this run, which run cannot read. - self.assertEqual( - lines[0], - "nvx: no built-in CPU profile serves this host's CPU, GenuineIntel " - "6/140/1 (11th Gen Intel(R) Core(TM) i7-1185G7), so --cpu-profile " - "auto, the default, falls back on it to a CPU profile derived from " - "this host, which OpenVMM warns about with NVX-CPU-PROFILE-FALLBACK:; " - f"the built-in profiles cover {time_abi.describe_cpu_generations()}.", - ) - self.assertIn("OpenVMM's error above names the cause", lines[1]) - self.assertNotIn("E_PROFILE_HOST_UNKNOWN", guidance) - self.assertNotIn("rerun with", guidance) - # It asks for the CPU's fingerprint through the CPU profile request - # form, which the link prefills with the CPU. - self.assertEqual( - lines[2:], - [ - "nvx: Help NVX add a built-in profile for this CPU: run", - "nvx: /nvx/openvmm --hypervisor kvm --cpu-fingerprint " - "fingerprint.json", - "nvx: and attach fingerprint.json to a CPU profile request:", - f"nvx: {time_abi.cpu_profile_request_url(tiger_lake)}", - "nvx: https://github.com/microsoft/nvx/issues/408 tracks CPU " - "profiles for more Intel CPUs.", - ], - ) - self.assertEqual(guidance, guide("auto", tiger_lake)) - # A host profile that fails on this CPU fails for a reason that - # OpenVMM's own error explains. - self.assertIsNone(guide("host", tiger_lake)) - - # Host profiles serve AMD CPUs too: an AMD CPU without a built-in - # profile, such as Raphael (Ryzen 7000), falls back as an Intel one - # does, with the issue that tracks profiles for more AMD CPUs. - raphael = time_abi.HostCpu("AuthenticAMD", 25, 97, 2) - guidance = guide(None, raphael) - assert guidance is not None - self.assertIn( - "this host's CPU, AuthenticAMD 25/97/2, so --cpu-profile auto, the " - "default, falls back on it", - guidance, - ) - self.assertIn(time_abi.cpu_profile_request_url(raphael), guidance) - self.assertIn( - "https://github.com/microsoft/nvx/issues/409 tracks CPU profiles for " - "more AMD CPUs.", - guidance, - ) - self.assertNotIn("issues/408", guidance) - self.assertIsNone(guide("host", raphael)) - # Another vendor's CPU fails with either request, and gets no request - # form and no issue: the time ABI does not serve its CPUs. - hygon = time_abi.HostCpu("HygonGenuine", 24, 0, 1) - guidance = guide("auto", hygon) - assert guidance is not None - self.assertIn( - "this host's CPU, HygonGenuine 24/0/1, so a cold boot with " - "--cpu-profile auto, the default, fails on it with " - "E_PROFILE_HOST_UNKNOWN;", - guidance, - ) - self.assertIn("host CPU profiles serve only Intel and AMD CPUs", guidance) - self.assertNotIn("falls back", guidance) - self.assertNotIn("github.com", guidance) - self.assertEqual(guidance, guide("host", hygon)) - - # A CPU that a built-in profile serves, an explicit profile, and an - # unknown CPU get none. - alder_lake = time_abi.HostCpu("GenuineIntel", 6, 154, 3) - milan = time_abi.HostCpu("AuthenticAMD", 25, 1, 1) - genoa = time_abi.HostCpu("AuthenticAMD", 25, 17, 1) - turin = time_abi.HostCpu("AuthenticAMD", 26, 2, 1) - for cpu_profile, host in ( - (None, alder_lake), - ("host", alder_lake), - (None, milan), - ("host", milan), - (None, genoa), - (None, turin), - ("host", turin), - ("intel.alderlake.v1", tiger_lake), - ("intel.alderlake.v1", raphael), - ("intel.skylake-sp.v1", milan), - ("intel.skylake-sp.v1", hygon), - (None, None), - ): - with self.subTest(cpu_profile=cpu_profile, host=host): - self.assertIsNone(guide(cpu_profile, host)) - # The guidance applies exactly where there is one, which a caller can - # tell before it looks up the OpenVMM executable that the guidance - # names. - for cpu_profile in (None, "auto", "host", "intel.alderlake.v1"): - for host in (tiger_lake, raphael, hygon, alder_lake, milan, None): - with self.subTest(cpu_profile=cpu_profile, host=host): - self.assertEqual( - time_abi.cpu_profile_guidance_applies(cpu_profile, host), - guide(cpu_profile, host) is not None, - ) - def test_names_cpus_and_prefills_cpu_profile_requests(self): granite_rapids = time_abi.HostCpu( "GenuineIntel", 6, 173, 1, "Intel(R) Xeon(R) 6973P-C"