tests: run the runtime adapter's executions in VMs through its host program by esaurez · Pull Request #870 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-24T09:11:31Z

Stacked on #868 (PR 2 of #805), which is stacked on #862 (PR 1 of #805), on #878 (PR 8 of #822), whose other PRs have merged into dev: #863 → #867 → #869 → #865 → #864 → #872 → #875. This PR retargets to dev once #878, #862 and #868 merge.

Part of #805: PR 3 of 3, the host's exec command, and executions in VMs.

What changed

The guest's worker runs executions in the assembled root, under their identities (#868), but the host program could not start one, so no test ran an execution in a VM, and nothing entered a real root.

exec::HostExecution is the host's side of one execution, in every configuration: the engine's Peer, with a sink that takes the program's output as it arrives, unless its caller stalls it, which withholds the guest's credit, and a source of the program's input. deliver hands it the guest's messages, serve returns the peer's, and deadline is the peer's next timer. serve takes the peer's messages before its events, so it still returns the cancellation of an execution that the peer gives up on in the step that ends it, as a peer without grace or settlement time does. The sink keeps at most exec::MAX_CAPTURE, 16 MiB, of each stream: once more of a stream arrives, it cancels the execution, and the peer discards the rest when the grace ends, so the outcome marks the stream truncated, with the cause cancelled. The guest's in-process tests (tests/guest_session.rs) now drive it in place of their own copy.
host::Executions (with host::ExecutionsError) is the host program's single-threaded driver of executions over one attachment of the client:
- start sends a start once it decodes as the guest decodes it. These starts fail before anything is sent and leave the client usable:
  - a start that does not decode, such as one with a relative working directory or an environment entry without a name;
  - a second start of an ID;
  - a start that says the host sends the program's input;
  - a start whose output windows count bytes, or more than host::MAX_OUTPUT_FRAMES (32) frames;
  - a start beyond exec::MAX_EXECUTIONS (8) executions;
  - any start once the driver ran: a driver runs one batch, whose executions it starts before it runs them. The guest may still send messages about the executions of a run that gave up on one or failed, so a later batch takes an attachment of its own, as each command of the host program does.
- run serves every execution, sends their messages, and receives until each finished. Each receive waits at most until the earliest timer of the executions, so timeouts fire whatever the guest sends. These end the run:
  - a guest message about no execution, about one that the driver did not start, whose body does not decode, or that breaks the engine's rules;
  - a deadline that passes first. In a receive, it leaves the client usable; in a send, which waits for the transport as the client's sends do, its timeout ends the client.
- Why the driver sends little before it reads: the session's broker passes the guest's credit for the host's records on only behind the guest's own records, which wait while the host does not read. So a single-threaded host that sends without reading can wait for credit that only its reading would free, even with a guest that reads. The driver therefore:
  - sends no program's input, which one thread could not send while it reads the programs' output;
  - starts at most 8 executions, whose answers are small and wait for the run, since no output flows before the run's credit does;
  - in the run, sends at most one credit for each message that it reads, besides one cancellation for each execution. With output windows of at most 32 frames, the guest's messages ahead of its own credit stay few, and those credits stay well within the guest's 64 KiB receive window.
exec::MAX_EXECUTIONS is the guest's limit of executions at once, which the worker's worker::MAX_EXECUTIONS now takes its value from.
Client::receive now also refuses the messages about executions that only the host sends: ExecStart, ExecInput, ExecInputEnd, and ExecCancel.
runtime-adapter-host exec STORE [--guest-build B] EXECUTION [';' EXECUTION]..., with EXECUTION = [--identity UID:GID [--groups GID,...] [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]..., reattaches and runs up to 8 executions at once over one session, in the sandbox that launch started: under runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's order, launch exits with 0 only once the guest started the sandbox, after the mapping table. It prints exec.N.launch, status, cause, truncated, settled, stdout, and stderr once all finished.
- It sends no standard input. Without --identity, a start names no identity, so the sandbox's workload identity applies.
- Each execution runs for at most 60 s, with a 1 s grace, a 5 s settlement time, 16 KiB chunks, and the credit of 8 chunks for its output, which both streams share.
- It keeps at most 16 MiB of each stream, and cancels an execution whose stream would exceed that.
- Output bytes that are not UTF-8 print as \xHH.
- It exits with 0 once every execution finished, whatever its outcome. It exits with 2 on a usage error, which it checks before it attaches, the start's encoding and the number of executions included.
Execution N of a command has the key (B + N, E): E is the attachment's epoch, and B the first 8 bytes of the broker instance's ID, as a little-endian number. The guest ends its session on a second start of a key, even in a later epoch, and keeps its history of generations across resets. Epochs only advance within a broker instance, while a broker restored from a snapshot has a new instance ID whose epochs start over, so B keeps different commands' keys apart.
launch ... --workload-identity UID:GID, for a layered root only, beside runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's --share, --deny, --reserved and --share-root, each ID from 1 to 4294967294, names the sandbox's workload identity in the root configuration. A workload identity whose account the image lacks fails the root's identity stage, and the launch exits with 4, as for any stage that fails.
The adapter's README and module docs describe the exec command, its keys and outputs, --workload-identity, the driver and its limits, HostExecution, and the new tests.

Tests

src/exec/tests.rs: with zero grace and settlement, the host's side still sends the cancellation of an execution that it gives up on at once, for a timeout and for its caller's cancellation. It keeps exactly 16 MiB of a stream that a program writes without end, then cancels the execution, which ends with that stream truncated and the cause cancelled.
tests/host_session.rs, in memory, on Linux and Windows. A fake guest runs NVX's execution engine's runner for each start, as the worker does, for:
- two executions at once: one that writes more than its credit holds, byte-exact, with its outcome, and one whose launch the guest refuses, with the guest's records whole or split;
- the starts that the driver refuses before it sends them (undecodable ones, a second start of an ID, one with input, ones whose output windows count bytes or more than 32 frames, one beyond 8 executions, and one once the driver ran), with a client that stays usable, and the widest output windows that it takes;
- the messages that end a run: about an execution that the driver did not start, about none, with a body that does not decode, or out of the engine's order, and the end of the stream;
- a timeout that fires while the guest says nothing, after which the execution is cancelled, then fenced;
- a run whose deadline passes first: in a receive, after which the client still answers a ping, or in a send to a guest that stopped reading, after which the client is ended.
The client also refuses the host-only messages about executions.
tests/layered_launch.rs: the exec and --workload-identity command lines that the host program refuses, with status 2, nine executions included.
tests/vm_layered.rs, three new VM scenarios after runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's 32, which CI runs on KVM, MSHV, and WHP:
- executions_run_in_the_root, without a workload identity. Over one session: four executions as nobody that meet only while all of them run, beside one as root that writes 1.29 MB to its standard output and some to its standard error, byte-exact. Over a later session, with new keys:
  - a program whose descendant leads a session of its own, as /proc shows before the program prints, and holds the program's output, which the program's end, as PID 1 of its namespace, ends, so that no output is lost;
  - nobody's IDs and groups, with empty permitted, effective, and bounding sets;
  - root with CAP_CHOWN alone in each of those sets;
  - exactly the start's environment, and its working directory;
  - the entry's default hostname, nvx-sandbox, as PID 1;
  - the stage-0 refusals of a start without an identity (EINVAL) and of one with CAP_SYS_ADMIN (EPERM), fenced and without output.
- executions_run_as_the_workload_identity: executions without an identity run as nobody, without capabilities, with exactly the start's environment, and the guest refuses a start that names an identity (EINVAL).
- accountless_workload_identity_fails_its_stage: the identity stage fails once the guest assembled the root, and the VM ends with 16, with the scratch unmounted.

Validation

This branch is rebased onto #868's current head (ca51780), which, with #862 (a8cc6e2), is on #878's current head (47f8bcd), the maintainer's re-commit of #878 on dev after #875, #907, #908 and #911 merged; the OpenVMM gitlink stays 2b681564. The rebase had no conflicts, and git range-diff shows the six commits' patches unchanged (4d7bb87 → bd2b901, d864b22 → 962245a, afa35d5 → 2e7814d, bc8f6f4 → 738924f, 25b905d → d37dc79, e4e5256 → ba8a5f6). Since the previous rebase, onto #875's rework, the first and fourth commits use its names and layout:

the host program's layered options hold sharing, a sharing::HostFsSharing with accessors, where this PR had shares: the launch's options keep workload_identity beside sharing, and the options that need --distro check --workload-identity beside --share, through sharing.shares();
src/host.rs declares the executions module beside tests: run the host's order of a layered root in the adapter's host program #875's map_reply and order_error, and re-exports Executions, ExecutionsError and, from the fourth commit, MAX_OUTPUT_FRAMES beside tests: run the host's order of a layered root in the adapter's host program #875's MapReply and OrderError.

The command lines of exec, its limits and its printed keys are unchanged. At ba8a5f6:

Each of this PR's six commits passes Clippy with warnings denied in the protocol-only, host-only and guest-only configurations.
Linux, on a cgroup v2 host:
- the adapter's steps of check-nvx-rs in its three configurations: rustfmt, Clippy with warnings denied, the build, the tests, and cargo +1.89 check (the MSRV);
- scripts/check_markdown_links.py;
- the guest-only tests as root: 50 library tests and 48 guest_session tests;
- 10 runs of guest_session as root and 10 unprivileged, and 3 runs of the library's tests, all passing, with no cgroup left.
Windows: rustfmt, Clippy, and the tests of the protocol-only and host-only configurations; 10 runs of host_session, all passing.
Real VMs, with a release build of this head's guest agent and the v0.1.0-dev.c2ace076c60a artifacts (OpenVMM 2b681564):
- WHP, locally: all 35 layered scenarios, the 3 restart tests and the 3 step tests pass.
- MSHV, on a Linux test host, as root, with OpenVMM's --cpu-profile host because no built-in profile serves its CPU, and under a short temporary directory, which keeps the VMs' control sockets' paths within SUN_LEN: all of them pass too.
- KVM: left to CI.
By hand on WHP, at ba8a5f6: stage, then launch with an upper in RAM, which printed sandbox=started, then exec of three executions over one session: one as nobody, which printed its IDs, the default hostname nvx-sandbox and a line on its standard error; one as root with CAP_CHOWN alone in its permitted and effective sets, in /tmp, which printed a byte that is not UTF-8 as \xff; and one without an identity, which the guest refused at stage 0 with EINVAL (exec.3.launch=failed 0 22), fenced. The first two settled reusable. Then stop, graceful, with a complete teardown.
Final reviews of the previous rebase, its resolutions and the six commits, and of this rebase found no issue.
fc0bc9f then adds a comment, in answer to a review: the host program's timers and limits of exec are values chosen for tests that make a valid configuration, within the limits that the guest and the host allow, which it names. It changes no code; rustfmt, and Clippy with warnings denied in the three configurations, pass at it.
Before marking this Ready, I ran three rounds of a preliminary code review and addressed every finding in the later commits: the cancellation that a peer without grace or settlement time dropped, the starts and the input that a single-threaded host could send without reading, the output windows, and a timing assumption in a new test.
Copilot's review found two more issues, fixed in d37dc79 (4de4cae, then c1be456, 431f1a6 and 25b905d, before the rebases): the host's side kept all of a stream's output, so a program that writes without end grew the host program's memory until its deadline, and the descendant scenario's program could end before its descendant called setsid(2).
At 4de4cae, whose host side the rebases leave as it was, in a temporary scenario that this PR doesn't include, two executions at once: one that wrote 17,000,000 bytes kept exactly 16,777,216 of them, with cause=cancelled and truncated=stdout, and one that wrote exactly 16 MiB exited 0, untruncated.

Copilot

🟡 Changes recommended

Unbounded output capture risks exhausting host memory, and the descendant-cleanup scenario needs deterministic synchronization.

2 open issues

What changed in this PR

Adds the host-command and VM-test portion of #805, connecting the runtime adapter’s execution engine to programs running in assembled guest roots.

Changes:

Adds reusable host-side execution handling and a single-batch session driver.
Introduces exec and --workload-identity, with argument validation and output reporting.
Expands protocol, driver, and VM tests and documents the new behavior.

File	Description
`tests/runtime-adapter/tests/vm_layered.rs`	Adds VM execution and workload-identity scenarios.
`tests/runtime-adapter/tests/layered_launch.rs`	Tests rejection of invalid execution and identity options.
`tests/runtime-adapter/tests/host_session.rs`	Tests execution driving, output, protocol failures, and deadlines.
`tests/runtime-adapter/tests/guest_session.rs`	Reuses the shared host-side execution implementation.
`tests/runtime-adapter/src/worker.rs`	Uses the shared concurrent-execution limit.
`tests/runtime-adapter/src/host/executions.rs`	Adds the single-batch execution driver.
`tests/runtime-adapter/src/host.rs`	Exports the driver and rejects host-only execution messages.
`tests/runtime-adapter/src/exec/tests.rs`	Tests cancellation delivery with zero grace and settlement time.
`tests/runtime-adapter/src/exec.rs`	Adds `HostExecution` and the shared execution limit.
`tests/runtime-adapter/src/bin/runtime-adapter-host.rs`	Implements execution commands, identity options, and result formatting.
`tests/runtime-adapter/README.md`	Documents commands, driver constraints, and test coverage.

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Flow control, cancellation, deadlines, and execution settlement across VM backends warrant final human review despite no additional confirmed defects.

No open issues 🎉

2 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Cross-platform flow control and execution lifecycle interactions warrant human review, with stacked-base integration and KVM validation confirmed before approval.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The interaction between single-threaded flow control, cancellation timing, and VM lifecycle behavior warrants human review alongside cross-backend validation.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Transport flow control, execution timers, and cross-platform VM settlement interact in ways that warrant final human review.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The interaction of flow control, execution deadlines, cancellation, and cleanup across VM backends warrants final human review despite no confirmed unresolved defects.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

No concrete defect was identified, but interactions among transport backpressure, execution timers, cancellation, and cross-platform VM sessions warrant final human review.

No open issues 🎉

_{Review effort: Balanced}

Enrique Saurez (esaurez) · 2026-10-26T23:21:13Z

+/// The chunks of an execution's input that the guest may hold for `exec`.
+const EXEC_INPUT_FRAMES: u32 = 4;
+/// The chunks of an execution's output, of both streams, that `exec` may hold.
+const EXEC_OUTPUT_FRAMES: u32 = 8;


how are this consts defined? why those specific numbers?

They're values chosen for tests that make a valid configuration, not derived from a formula. The 16 KiB chunk, the 1 s grace and the 5 s settlement time follow the example in nvx::exec's documentation, whose 64 KiB window the input's 4 chunks, and each stream's half of the 8 output chunks, match; the 60 s timeout matches the launch's START_TIMEOUT, ample for the tests' programs. What they must respect are the limits that the guest and the host enforce: chunks of at most exec::MAX_CHUNK bytes; at most worker::MAX_EXECUTION_MEMORY, 1 MiB, held in the guest for each execution (these hold 96 KiB: a chunk of each stream and the input's window); and output windows of at most host::MAX_OUTPUT_FRAMES, 32, frames. fc0bc9f adds a comment by the constants that says so.

Copilot

🔵 Needs a closer look

Cross-platform flow control and privileged namespace, identity, and cgroup lifecycle integration warrant final human review.

No open issues 🎉

_{Review effort: Balanced}

…rogram The guest's worker runs executions in the assembled root, under their identities, but the host program could not start one, so no test ran an execution in a VM, and nothing entered a real root. The host program's new exec command reattaches to a VM and runs several executions at once over one session. A single-threaded driver, host::Executions, serves the host's side of each, an exec::HostExecution: the engine's peer, with a sink of the program's output and a source of its input. The driver refuses a start that the guest would not decode, and a second start of an ID, before it sends anything, since the guest ends its session on either. Execution N's key is (B + N, E): E is the attachment's epoch and B the broker instance's ID, so no command repeats another's key, even after a broker restored from a snapshot starts its epochs over. The launch gains --workload-identity, the identity as which every execution of the sandbox runs. The client now also refuses the messages about executions that only the host sends. The VM scenarios run executions in the root over one session: output beyond a MiB, executions that meet only while all of them run, a descendant in a session of its own, the identities that starts name, root with one allowed capability, an exact environment, a working directory, the configured hostname, and the refusals of stage 0. Others run executions as the sandbox's workload identity, and fail the identity stage for one without an account. The in-memory tests of the host cover the driver's output, outcomes, refusals, and timers, and the in-process tests of the guest now drive the library's HostExecution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

…n at once A peer without grace or settlement time gives up on an execution in the step that ends it, for its timeout or its caller's cancellation. HostExecution::serve took the peer's events before its messages, and the last event, once taken, discarded the cancellation that waited to be sent, so the guest's program ran on after the host's side finished. serve now takes the messages first, as the example of nvx::exec does. The driver's sends wait for the transport, as the client's do, until the run's deadline, and no timer fires meanwhile. Its documentation now says so, and that a deadline that passes in a receive leaves the client usable, while one that passes in a send ends it, which the tests now check, the latter with a guest that stops reading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

The session's broker passes the guest's credit for the host's records on only behind the guest's own records, which wait while the host does not read, and takes no more of the host's records than that credit allows. So a single-threaded host that sends without reading can wait for credit that only its reading would free, even with a guest that reads. The exec command sent every start before it read anything, and a batch of hundreds, which the guest refuses beyond the executions that it runs at once, could fill the way back with refusals and hold the next start until its deadline. The driver now runs at most exec::MAX_EXECUTIONS executions that have not finished, as many as the guest runs at once. Their answers to their starts wait for the run, since no output flows before the run's credit does, and exec refuses more executions as a usage error. The driver also sends no program's input, which one thread could not send while it reads the programs' output: it refuses a start that says that the host sends input, and Executions::start no longer takes any. The worker's limit of executions now comes from that shared constant. The test of a send that waits for a guest that stops reading counts the run's deadline from the start's send, so that the deadline passes in the send that it checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

Small sends did not yet bound what the driver sends before the guest's credit for the host's records can reach the broker. An output window of many frames of few bytes lets the guest queue thousands of output messages ahead of its credit, while the driver sends a credit for each one that it reads. A start after a run may also wait for credit behind the output of executions that the run gave up on, or left running when its deadline passed. The driver now takes only output windows that count frames, at most host::MAX_OUTPUT_FRAMES of them, so that the credits for the guest's messages ahead of its own credit stay well within the guest's receive window. It also runs one batch: it starts every execution before it runs them and none after, so a later batch takes an attachment of its own, as each command of the host program does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

The host's side of an execution took all of a stream's output into memory, so a program that writes without end grew the host program's memory without bound until its deadline. HostExecution now keeps at most exec::MAX_CAPTURE, 16 MiB, of each stream. Once more of a stream arrives, it cancels the execution, and its peer discards the rest when the grace ends, so the outcome marks the stream truncated, with the cause cancelled. The VM scenario whose program leaves a descendant in a session of its own printed `started` as soon as it forked setsid, so the program could end before the descendant called setsid(2) or held its output. The program now waits until the descendant has printed its process ID and moved its output to the program's, and checks in /proc that it leads its session, before it prints `started`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

…s take The executions take nvx::guest::entry's default hostname, which no configuration changes, rather than a hostname that the kernel command line configures. The VM scenario that checks it now describes it so. Its check, nvx-sandbox, is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

The host program gives the executions of `exec` timers and limits whose values are chosen for tests, as a valid configuration within the limits that the guest and the host allow. A comment now says so, and names those limits: `exec::MAX_CHUNK`, the most bytes of a chunk; `worker::MAX_EXECUTION_MEMORY`, the most bytes that an execution holds in the guest; and `host::MAX_OUTPUT_FRAMES`, the most frames of an output window. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90

Enrique Saurez (esaurez) mentioned this pull request Oct 24, 2026

tests: run the runtime adapter's executions in the root, under their identity #868

Merged

Enrique Saurez (esaurez) marked this pull request as ready for review October 24, 2026 10:26

Enrique Saurez (esaurez) requested a balanced review from Copilot October 24, 2026 10:27

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 24, 2026 10:27

View session