nvx-rs: keep processes spawned during a launch from holding its VMM's log on Windows

Baseline: dev at 1853252c. Every link uses that revision.

Goal. On Windows, NVX makes a VMM's standard handles inheritable while it creates the VMM. A process that the caller of the launch spawns meanwhile without a handle list, as std::process::Command spawns one, inherits those handles, its log's among them, and keeps them until it exits. The launch's claim on its log then outlives the VMM. A recovery that finds the VMM gone cannot end the launch, which remains, with whatever it holds, until a recovery after that process exits. Tests that launch VMMs while other tests spawn processes fail intermittently on WHP because of it. Close that window to processes that the caller spawns through NVX, and make the tests spawn their processes that way.

Today:

spawn passes the VMM its standard handles: the read end of the pipe that carries its capability, and two handles of its log (spawn.rs#L86-L99).
- spawn_listed makes them inheritable (windows.rs#L694-L703), names them and the handles inherit in the VMM's handle list (#L704-L746), creates the VMM (#L751-L766), and only then closes them (#L772-L775).
- The list keeps the caller's other handles out of the VMM. It does not keep the VMM's handles out of other processes: a process that another thread creates meanwhile, with inheritance and no list, inherits every handle of the caller that is inheritable then. The docs of spawn_detached (#L591-L597) and of Inherited (inherit.rs#L36-L40) say so.
- INHERITANCE (windows.rs#L524-L529) serializes only NVX's spawns that allow handles inherit, and no other spawn.
- So NVX's own launches never take each other's handles, but processes that a test or an embedding application spawns with std::process can.
On Windows, a launch's claim on its log is the VMM's handles themselves: while any process holds the log, an open of it that shares nothing fails (store/mod.rs#L115-L129).
- Recovery checks a held claim again for up to a second (#L307-L313, claim_held at #L822-L850). That covers a process that holds the log on Linux, which it does only from its fork until its exec.
- Windows closes no handle when a process starts, so a process that inherited the log holds the claim until it exits.
- When the VMM has exited, end_identified (#L852-L887) finds no process that matches it, but a held claim, and reports the launch as indeterminate: "a process still holds its log, and no process matches its VMM process N".
That process also holds the read end of the capability pipe, from which it could read the capability before the VMM does.
Seen on WHP:
- runtime-adapter: start clones of templates #954's vm_clone test a_clone_whose_activation_fails_ends_before_it_is_ready launches a VM in the test process while other tests spawn runtime-adapter-host processes, which run for seconds to minutes.
  - It failed in the first attempt of CI run 40747944093 (job 123032287845, "NVX microVM tests / Windows / WHP / Guests and sandboxes") with "cannot activate it; the launch could not be ended, and remains for recovery: a process still holds its log, and no process matches its VMM process 7404". It passed on the rerun.
  - It failed in 2 of 4 WHP runs on one branch, and in every parallel WHP run on another. It passes with RUST_TEST_THREADS=1.
- nvx-rs: attribute a clone's failures, record them, and reattach it #951's library test host::template::clone::tests::restore::a_status_with_which_the_guest_ended_the_clone_s_vm_attributes_its_failure failed once, with Unresolved("a process still holds its log, and no process matches its VMM process …") instead of Exited. 5 of 5 reruns passed.
  - Meanwhile, other library tests spawn processes with std::process, such as link_dir's cmd /C mklink /J (store/lifecycle/tests.rs#L468-L488, template/store/tests.rs#L115-L136) and the minute-long sleeper (process.rs#L466-L482).

Implementation:

One lock over the whole window. spawn_listed takes a process-wide lock before it makes any handle inheritable. It releases the lock once it has closed the standard handles and restored the handles inherit. INHERITANCE becomes that lock; today it covers the handles inherit alone, and no spawn without them.
The caller's spawns under it. NVX exposes, on every host, a guard of that lock and a function that spawns a std::process::Command while it holds the lock, so that a process spawned that way cannot inherit a VMM's handles.
- On Linux, they need not lock anything: a process holds the log only until its exec, and the claim's recheck covers that.
- The docs of the guard, spawn_detached, Inherited, and the log claim say that on Windows, a process spawned without the guard can still hold a VMM's log until it exits, and leave its launch to a later recovery.
Tests spawn through it:
- nvx-rs's tests that spawn processes while others launch, such as link_dir and sleeper above;
- the runtime adapter's harness, which spawns runtime-adapter-host while a test launches a VM in its own process.

Not in scope: neither step covers a process that the caller spawns without the guard. A claim that only the VMM can hold would cover it, but either approach is a larger change:

OpenVMM could open its log itself, from a path that NVX passes, so that no handle of the log is ever inheritable in NVX's caller. That needs an OpenVMM option, and leaves the capability pipe.
A recovery could prove the VMM's exit through a job object of its launch, whatever holds its log. That changes the store's recovery.

Tests:

On Windows, a test of spawn_listed with a hook that holds a spawn in its window:
- A process that the guard's function spawns meanwhile does not inherit the spawned process's log: the log's claim is released once that process exits, while the other process still runs.
- The control: a process that std::process::Command spawns in that window holds the claim.
The two tests above pass in 10 consecutive parallel WHP runs, without RUST_TEST_THREADS=1.

Done when: merged, with:

the window under one lock;
the guard and its spawn function public and documented;
NVX's and the runtime adapter's tests spawning through them;
the affected tests passing in parallel WHP runs.

Also on dev. The push of e4ca7acaa (#959) failed the same way in CI run 40896635809, attempt 1 (job 123543947721, "NVX microVM tests / Windows / WHP / Guests and sandboxes" on azure-windows-8, step "Run runtime adapter scenarios on Windows"). Links below use that revision.

In vm_clone, 15 tests passed, and a_clone_whose_activation_fails_ends_before_it_is_ready failed in the Message::Start case of refuses_early (vm_clone.rs#L1126):

Start: cannot activate it; the launch could not be ended, and remains for recovery: a process still holds its log, and no process matches its VMM process 7596: the broker closed the NVXS session: the broker closed the NVXS session
  left: (Unknown, Protocol, Activation, Activating, Some(51), Unresolved("a process still holds its log, and no process matches its VMM process 7596"))
 right: (Unknown, Protocol, Activation, Activating, Some(51), Exited)

The test's Early control saw the VMM exit before the teardown, since the assertion on control.exited passed, so only the log's claim was still held.

The run's "Required status check" failed, and its "Publish development release", "Performance regression gate", and "Persist performance baseline" jobs were skipped.
Later dev pushes passed the job, but vm_clone.rs, tests/common/, and nvx-rs/src/host/ are unchanged from e4ca7acaa through 9a84f5722, so the window remains.
In vm_clone, refuses_early is the only test that launches in the test process (vm_clone.rs#L1087-L1090). Every other test runs runtime-adapter-host through Sandbox::start, which spawns it with std::process::Command (common/mod.rs#L234-L243), so step 3's harness change covers this binary.

Reproduced without a hypervisor. On a Windows host, with Rust 1.95.0 and nvx-rs at 9a84f5722, whose src/host/ matches e4ca7acaa. In each iteration, a binary creates a SandboxStore, begins a launch, spawns cmd /c exit 0 with the launch's log through supervisor::spawn, records the process, waits for it to exit, and abandons the launch, as a clone whose activation fails does. Meanwhile, another thread spawns ping -n 3 127.0.0.1, which runs for about 2 s, with std::process::Command, 5 ms apart:

Other thread's spawns	Iterations	`Exited`	`Indeterminate`
None	20	20	0
Under one mutex with the launch's spawn	40	40	0
Unsynchronized	40	6	34

Each indeterminate abandon took about 1.0 s, the claim's recheck, and reported "a process still holds its log, and no process matches its VMM process N". The second row has the shape of steps 1 and 2.

Reproduction source

Cargo.toml:

[package]
name = "inherit-repro"
version = "0.0.0"
edition = "2024"
publish = false

[dependencies]
nvx = { path = "<checkout>/nvx-rs", default-features = false, features = ["host"] }

[workspace]

src/main.rs, run as cargo run -q -- alone 20, cargo run -q -- serialized 40, and cargo run -q -- concurrent 40:

use std::env;
use std::fs;
use std::path::PathBuf;
use std::process::{Child, Command, Stdio};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, PoisonError};
use std::thread;
use std::time::{Duration, Instant};

use nvx::host::store::{ArtifactIdentity, LaunchId, Recovery, RuntimeIdentity, SandboxStore};
use nvx::host::supervisor::{self, VmmCommand};
use nvx::protocol::CAPABILITY_LEN;

const CAPABILITY: [u8; CAPABILITY_LEN] = [7; CAPABILITY_LEN];

static SPAWNS: Mutex<()> = Mutex::new(());

fn runtime() -> RuntimeIdentity {
    let artifact = |digit: char| ArtifactIdentity::sha256(&digit.to_string().repeat(64)).unwrap();
    RuntimeIdentity::new(artifact('1'), artifact('2'), artifact('3'))
}

fn main() {
    let mut args = env::args().skip(1);
    let mode = args.next().unwrap_or_else(|| "concurrent".to_owned());
    let iterations: usize = args.next().map_or(40, |count| count.parse().unwrap());
    assert!(
        matches!(mode.as_str(), "concurrent" | "serialized" | "alone"),
        "mode: concurrent | serialized | alone"
    );
    let serialized = mode == "serialized";
    let root = env::temp_dir().join(format!("nvx-inherit-repro-{}", std::process::id()));
    fs::create_dir_all(&root).unwrap();

    let stop = Arc::new(AtomicBool::new(false));
    let spawned = Arc::new(AtomicU64::new(0));
    let spawner = (mode != "alone").then(|| {
        let (stop, spawned) = (stop.clone(), spawned.clone());
        thread::spawn(move || {
            let mut children: Vec<Child> = Vec::new();
            while !stop.load(Ordering::Relaxed) {
                {
                    let _spawns =
                        serialized.then(|| SPAWNS.lock().unwrap_or_else(PoisonError::into_inner));
                    // About 2 s, longer than the store's 1 s CLAIM_RELEASE_WAIT, as a
                    // runtime-adapter-host command that another test starts.
                    children.push(
                        Command::new("ping")
                            .args(["-n", "3", "127.0.0.1"])
                            .stdin(Stdio::null())
                            .stdout(Stdio::null())
                            .stderr(Stdio::null())
                            .spawn()
                            .unwrap(),
                    );
                }
                spawned.fetch_add(1, Ordering::Relaxed);
                children.retain_mut(|child| child.try_wait().unwrap().is_none());
                thread::sleep(Duration::from_millis(5));
            }
            for mut child in children {
                let _ = child.wait();
            }
        })
    });
    thread::sleep(Duration::from_millis(500));

    let shell = env::var_os("ComSpec")
        .map_or_else(|| PathBuf::from(r"C:\Windows\System32\cmd.exe"), PathBuf::from);
    let (mut exited, mut held, mut other) = (0, 0, 0);
    for iteration in 0..iterations {
        let store = SandboxStore::create(root.join(format!("sandbox-{iteration}"))).unwrap();
        let mut locked = store.lock(None).unwrap();
        let mut launch = locked
            .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY)
            .unwrap();
        let log = launch.take_log().unwrap();
        let launched = {
            let _spawns =
                serialized.then(|| SPAWNS.lock().unwrap_or_else(PoisonError::into_inner));
            supervisor::spawn(
                &VmmCommand::new(
                    &shell,
                    vec!["/c".into(), "exit".into(), "0".into()],
                    store.dir(),
                ),
                &CAPABILITY,
                log,
            )
            .unwrap()
        };
        let process = launch.record_process(&launched).unwrap();
        let pid = launched.id();
        launched.detach();
        assert!(
            process
                .wait(Instant::now() + Duration::from_secs(10))
                .unwrap(),
            "the stand-in VMM {pid} did not exit"
        );
        drop(process);
        let started = Instant::now();
        let ended = launch.abandon();
        let waited = started.elapsed();
        match ended {
            Ok(Recovery::Exited) => exited += 1,
            Ok(Recovery::Indeterminate(reason)) if reason.contains("still holds its log") => {
                held += 1;
                println!("iteration {iteration}: Indeterminate after {waited:?}: {reason}");
            }
            ended => {
                other += 1;
                println!("iteration {iteration}: VMM {pid}: {ended:?}");
            }
        }
        drop(locked);
    }
    stop.store(true, Ordering::Relaxed);
    if let Some(spawner) = spawner {
        spawner.join().unwrap();
    }
    println!(
        "RESULT mode={mode} iterations={iterations} exited={exited} held={held} other={other} \
         concurrent_spawns={}",
        spawned.load(Ordering::Relaxed)
    );
    let _ = fs::remove_dir_all(&root);
}

Reopening: #1029 fixed this for processes that the caller spawns under the spawn lock, but not for processes that it spawns otherwise, which this issue left out of scope.

Another occurrence, before #1029. CI run 40989896779, attempt 1, a dev push of 2fe86fe0 (the merge of #1022), failed in job 123871462085 ("NVX microVM tests / Windows / WHP / Guests and sandboxes", runner azure-windows-9), step "Run runtime adapter scenarios on Windows":

test a_clone_whose_activation_fails_ends_before_it_is_ready ... FAILED
  left: (Unknown, Protocol, Activation, Activating, Some(51), Unresolved("a process still holds its log, and no process matches its VMM process 8656"))
 right: (Unknown, Protocol, Activation, Activating, Some(51), Exited)

2fe86fe0 predates #1029 (merged as 78fd0229): spawn_listed still made the VMM's standard handles inheritable outside INHERITANCE, and the harness started runtime-adapter-host with a plain Command::spawn. Only that job failed. "Required status check" failed because of it, and the run's performance gate, baseline, release, and crate jobs were skipped.

After #1029. The first dev run with it, 40994225569 at 78fd0229, passed the same job (123882816388): vm_clone 34/34, this test included. That is one sample of an intermittent race. On Windows without a hypervisor, host::platform::windows::tests passed 16/16 in 6 of 6 runs at fc6ccf62.

What remains. The control test a_process_spawned_without_the_spawn_lock_in_a_window_holds_the_vmms_log shows that a process spawned without the lock, as std::process::Command spawns one, still holds a VMM's log, and with it the launch's claim, for as long as it runs. end_identified then cannot end a launch whose VMM has exited. An application that embeds nvx and spawns processes on other threads without the lock, itself or through a library, still gets "a process still holds its log, and no process matches its VMM process N" from an abandon or a recovery, and the launch stays until that process exits. No open issue tracks this.

Plan.

A deterministic reproduction. Hold a launch's spawn in its window with the window hook of the platform tests, spawn a process there with std::process::Command, let the stand-in VMM exit, and abandon the launch through the store. Today that ends Indeterminate with the message above on every run. Run it on the Windows/WHP test host too, and check the real-VM scenarios there.
The fix: the job-object approach under "Not in scope". On Windows, the spawn creates the VMM in a job object of its own, named after the log's identity, through PROC_THREAD_ATTRIBUTE_JOB_LIST, so the VMM never runs outside it. A process that only inherits the log never joins the job. The job's handle is inheritable for the whole window in which the log's handles are, so a process that inherits the log also inherits the job's handle and keeps the job open. A claim check that opens the job decides by the job's active processes. One that finds no job falls back to today's exclusive open of the log, so a launch by an earlier version is still checked as before.

Navigation Menu

Repository navigation

nvx-rs: keep processes spawned during a launch from holding its VMM's log on Windows #966

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

nvx-rs: keep processes spawned during a launch from holding its VMM's log on Windows #966

Description

Activity

ppenna commented on Oct 29, 2026

ppenna commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions