Skip to content

Track sandbox network option bugs: provision accepts egress rules without --network-egress and loopback forwards without --host-loopback allow #756

Closed

Description

Tracking issue for confirmed bugs in how nvx.py sandbox and nvx-rs sandbox check their network options. Found against origin/dev at 2817f4eb; every link below is pinned to that commit.

1. provision records network options that OpenVMM refuses, so every start fails

Severity: medium. A sandbox provisioned this way can never start: every start fails with only OpenVMM exited during startup with status 2, and recovery takes deprovision and a new provision. No data loss and no security impact.

Where

Trigger

sandbox provision (or sandbox run) with either:

  • --network-egress-allow or --network-egress-deny but no --network-egress, or
  • --host-loopback-forward without --host-loopback allow.

Expected

doc/usage.md documents both requirements for sandbox: --network-egress-allow and --network-egress-deny each "Requires explicit --network-egress", and forwards need --host-loopback allow. The pinned OpenVMM enforces both (microsoft/openvmm@9f04d05c cli_args/microvm.rs L1236-L1240 and L1259-L1264). NVX should refuse both combinations before provision writes any state and before run launches, as it already does for --network-egress-policy-file, and as the fix for #733 did for resource limits that the guest cannot apply.

Actual

provision exits 0 and records the invalid options. Every later start launches OpenVMM, which exits with status 2. The cause appears only in openvmm.log. sandbox run --dry-run with the same options also succeeds and prints an OpenVMM command that OpenVMM refuses.

Reproduction

Windows, nvx-rs built from 2817f4eb (cargo test --no-run --all-features). ROOT is a scratch checkout layout holding SOURCE-MANIFEST.json, placeholder build/vmlinux, build/initramfs.cpio.gz, and image files, with the repository's nvx-fake-openvmm.exe, which mirrors OpenVMM's checks, installed as openvmm/target/release/openvmm.exe. nvx.py sandbox forwards the same options to nvx-rs unchanged.

> nvx-rs --root ROOT sandbox provision --state-dir ROOT\state --layer distro,ROOT\img\distro.erofs,11111111-1111-1111-1111-111111111111 --scratch ROOT\img\scratch.ext4 --net 10.0.0.2/24 --network-profile portable --network-egress-allow 192.0.2.1:tcp:443
provision exit=0
config.json: "network_egress": null, "network_egress_allow": ["192.0.2.1:tcp:443"]

> nvx-rs --root ROOT sandbox start --state-dir ROOT\state --timeout 10
error: OpenVMM exited during startup with status 2
start exit=1
openvmm.log: openvmm: --network-egress is required with egress rules

> nvx-rs --root ROOT sandbox start --state-dir ROOT\state --timeout 10
error: OpenVMM exited during startup with status 2
start exit=1

> nvx-rs --root ROOT sandbox provision --state-dir ROOT\state2 --layer ... --scratch ... --net 10.0.0.2/24 --network-profile portable --host-loopback-forward tcp:18080:8080
provision exit=0

> nvx-rs --root ROOT sandbox start --state-dir ROOT\state2 --timeout 10
error: OpenVMM exited during startup with status 2
start exit=1
openvmm.log: openvmm: --host-loopback-forward requires explicit --host-loopback allow

Suggested fix

For run and provision, refuse non-empty --network-egress-allow or --network-egress-deny without --network-egress, using OpenVMM's message, in resolve_network_egress_rules next to the policy-file check. Also refuse --host-loopback-forward unless --host-loopback allow is given. Validating through host::network::openvmm_arguments would let the CLI and the library share one set of checks. Add nvx-rs/tests/cli.rs cases that provision with each combination and assert the refusal and that no state directory was written, and a run --dry-run case.

Explored

  • Sandbox plan persistence (host/sandbox/plan/form.rs, e4214a0): 28 Windows path forms (UNC, \\?\, drive-relative, root-relative, trailing separators) round-trip for the distro, scratch, and share paths; the UUID, limits, and unknown-field rules hold.
  • nvx::uuid (0a7ac27) and --layer UUID normalization, which ports Python's uuid.UUID.
  • The release lookup retry of release: retry uploads that gh reports as "release not found" #716 in scripts/nvx_tools/development_release.py.
  • CI change classification (scripts/nvx_tools/ci.py, 9ad0468): no-renames diff, fallbacks, output checks. The guest-binary source identity and its dep-info guard (ci: key the guest images and VMM tests on the guest binaries' sources #710).
  • Windows-specific host code: sandbox store ACL privacy checks, file seals, process liveness and start times, CreateProcess argument quoting, the sandbox lifecycle lock's error classes.
  • The nvx.py to nvx-rs option seam (_SandboxOption, the argparse emulation in host/cli/args.rs), the Python int, float, list2cmdline, and shlex emulations, the timeout and memory options.
  • scripts/check_markdown_links.py anchors and links, compared against GitHub's own renderer on 29 headings and 18 link forms. No difference.
  • The host control session and NVXS connection, and aci_edge_sandboxes.

Checked, not filed

  • sandbox start deletes the previous outcome.json before it checks config.json, the distro binding, or the artifacts (lifecycle.rs L480-L482), so a start refused before launch loses the last VM-level report. Reproducible, but inherited from the Python _start, consistent with the nvx-rs README's "report of the last start", and low impact.
  • gh release view reports transient draft-lookup failures as release not found (release: retry development release uploads that gh reports as "release not found" while the draft exists #686), which the observations after create and upload in development_release.py treat as final. Already tracked by release: address the development release by its ID instead of its tag #688.
  • HostConnection::send ends the connection when write_all times out, even if nothing was written (nvxs.rs L259-L271), while HostConnectionError::TimedOut says that such a send leaves it usable. Ending is required, since write_all cannot report a partial write and the session already advanced its sequence, so only the wording is off.
  • ControlSession and the CLI Session stay open after a reply with a mismatched request ID. Their docs promise to end only after stream failures, and each CLI command exits on its first error.
  • --network-proxy values are not checked by the CLI either. Unconfirmed: not reproduced against OpenVMM's checks.

Validation limits

  • Reproduced on Windows x64 with nvx-rs built from 2817f4eb and the repository's fake OpenVMM. The real OpenVMM was not built or run in this session; its source at the pinned revision has the identical checks linked above.
  • Linux (KVM or MSHV) was not run. The defect is in platform-independent CLI option handling.

Activity

  1. akhil9tiet commented on Oct 22, 2026

    @akhil9tiet
    Contributor

    I'd like to work on this — please assign it to me.

  2. ppenna commented on Oct 29, 2026

    @ppenna
    ContributorAuthor

    Fixed by #753 (b4d76e32a, nvx-rs: lower the sandbox network options through the typed policy), which merged on 2026-10-23, after this issue was filed. That PR didn't reference this issue, which is why #781 was closed as superseded.

    sandbox run and sandbox provision now read their network options into one typed policy. They check it before anything launches and before provision records anything:

    Verification with the real OpenVMM

    I ran the same script on three test VMs: nvx-kvm-test (linux-kvm), nvx-mshv-test (linux-mshv), and nvx-whp-test (windows-whp). It used nvx-rs built from dev at d89d370 and from this issue's 2817f4e. OpenVMM came from the development releases v0.1.0-dev.8afc615d86b4 (KVM) and v0.1.0-dev.320666c6c096 (MSHV, WHP), both built from OpenVMM bd2b27a2. Its network option checks are identical to those of the current pin, c32c57d1, which changed only CPU profile code. All three platforms gave the same results:

    nvx-rs Invocation Result
    d89d370 sandbox provision, sandbox run --dry-run, and sandbox run, with --network-egress-allow 192.0.2.1:tcp:443 (or provision with --network-egress-deny 198.51.100.0/24), without --network-egress Status 1, empty stdout, error[unsupported]: --network-egress is required with --network-egress-allow or --network-egress-deny. No state directory and no OpenVMM process.
    d89d370 The same with --host-loopback-forward tcp:18080:8080 and either no --host-loopback or --host-loopback deny Status 1, empty stdout, error[unsupported]: host-loopback forwards need host-loopback access allowed. No state directory and no OpenVMM process.
    d89d370 provision, start, exec, stop, and deprovision, once with --network-egress deny --network-egress-allow 192.0.2.1:tcp:443 and once with --host-loopback allow --host-loopback-forward tcp:18080:8080 Every step exits 0. OpenVMM accepts the rendered options (--network-egress-allow 192.0.2.1/32:tcp:443, --host-loopback allow --host-loopback-forward tcp:18080:8080), the workload sees eth0, and OpenVMM listens on 127.0.0.1:18080 for the forward.
    2817f4e provision with each combination, then start twice provision exits 0 and records the options. Both starts fail with OpenVMM exited during startup: vmm-failure status 1, and openvmm.log holds OpenVMM's fatal error: --network-egress is required with --network-egress-allow or --network-egress-deny or fatal error: --host-loopback-forward requires explicit --host-loopback allow.

    So with the real OpenVMM the bug reproduces at 2817f4e, where OpenVMM exits with status 1 rather than the fake's 2, and is fixed at d89d370. At the current tip, 29c6854, the checks are unchanged: #978 refactored the lowering but kept the forward check. The sandbox network CLI tests above also pass there on Windows.

    Outside the scope of this issue, nvx-rs run still passes these options to OpenVMM as given, as its dry-run goldens expect. OpenVMM refuses them at once and prints its message on the terminal, and nothing is recorded.

    Closing as fixed by #753.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bug-huntFindings from focused bug huntsconfirmedIssue affects multiple people.

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions