Skip to content

tests: capture a template's VM in the runtime adapter's guest - #932

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
esaurez/840-pr2-template-mode-20261024
Oct 27, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
esaurez/840-pr2-template-mode-20261024

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

A platform template's snapshot must hold no root, so that every clone that a restore starts attaches its own image, upper, and export. #877 adds nvx::guest::clone, through which a guest init or agent prepares its guest for a capture before its root's assembly. This gives the runtime adapter's guest agent a template mode that uses it: in a VM whose kernel command line names the platform snapshot tier, the worker captures the VM before it assembles any root, and runs no application session before the capture.

Part of #840: the second of its two pull requests. #841 runs this mode in VMs, and #842 repairs a restored clone.

Chain: #886 → #877 → this PR → #884 → #887 → #841 PR 3, on dev; #879, #881 and #885 have merged.

What changed

The command line (tests/runtime-adapter/src/template.rs, in every configuration). template::boot reads a kernel command line, such as /proc/cmdline, as the kernel splits it, with nvx::protocol::cmdline::parameters. One in which the kernel finds no parameter nvx_snapshot_tier=platform boots cold, whatever else it holds. With it, the VM is a template's, and boot refuses, with a CommandLineError:

  • a double quote anywhere, which the kernel drops from a parameter;
  • a --, after which the kernel hands the rest of the command line to init;
  • the first parameter outside the platform tier's allowlist, template::is_platform_token.

The allowlist is the one that OpenVMM's capture of a platform snapshot enforces, in platform_command_line_token_is_invariant at the pinned OpenVMM: earlycon=xe9, console=hvc0, console=hvc1, reboot=t, panic=-1, nvx_sandbox=1, nvx_config=0xd0010000,65536, nvx_snapshot_tier=platform, and nvx_control_tty=hvc2 whole, and any value of nr_cpus=, maxcpus=, virtio_mmio.device=, virtnet_ip=, virtnet_mask=, virtnet_gw=, virtnet_dns=, virtnet_ip6=, and virtnet_gw6=. It compares names exactly, as OpenVMM does.

The capture (tests/runtime-adapter/src/worker/capture.rs, in the guest-only configuration on x86_64). worker::capture::capture_on_terminal:

  1. opens the control tty in raw mode, as the worker of a VM that boots cold does;
  2. attaches the control session with GuestConnection::attach, and waits until the tty has sent the attachment, so that the snapshot holds the guest attached. It reads nothing from the session, and serves no application session, before the capture;
  3. prepares the guest with CapturePreparation, for the worker's fence, Admission, and the template's devices(): the distro and scratch block devices, at the MMIO windows from which the worker assembles a root, and OpenVMM's host filesystem device, tagged microvm;
  4. requests the capture with CaptureBarrier::capture.

Admission implements AdmissionFence with #877's acquire contract: close fails with ResourceBusy while the fence is closed, whoever closed it, and leaves it closed.

The worker (tests/runtime-adapter/src/bin/runtime-adapter-guest.rs). On x86_64, runtime-adapter-guest worker reads /proc/cmdline before it opens its tty. A VM that boots cold goes on as before. A template's VM runs the capture, after which the worker, and with it the VM, ends with one of these statuses, after a diagnostic on standard error:

Status When
1 The control tty cannot be opened or put in raw mode, or the attachment cannot be sent
20 The command line names the platform tier, but holds another parameter, a double quote, or --, or the worker cannot read it
21–28 A stage of the capture failed: the fence, the mounts, the boot checks, the capture checks, the stall detectors' suppression, the VM generation ID, the capture request, or the release
29 A stage that this build of the worker does not know failed
30 The capture ended without a snapshot, and the template continued
31 A restore of the snapshot started this VM, a clone, which the worker does not repair yet

A committed capture ends the VM before the worker returns. The README's "Template mode" section describes the flow and the statuses, and the module documentation the API.

Design notes

  • Cold boot. A VM whose command line doesn't name the tier boots as before: the worker only reads the command line first. If it can't read it, it ends with 20, failing closed.
  • The kernel's view. boot decides on the parameters that the kernel sees, so a tier that an unclosed quote or a -- hides from the kernel boots cold, as the kernel does. OpenVMM splits the command line at ASCII whitespace, which leaves out the vertical tab and the byte 0xa0, at which the kernel splits too: virtnet_dns=10.0.2.3, a vertical tab, and quiet pass OpenVMM's check as one token, but the kernel sees quiet, which template mode refuses. The guest fails closed there, and its tests pin both separators.
  • Restored clones. Repair (nvx-rs: repair a restored clone in the guest, with an owned seed and one acknowledgment #842) hasn't landed, so a restored clone ends with 31, with its fence closed and its stall detectors suppressed.
  • Statuses. 21 to 29 are template::CAPTURE_STATUS_BASE plus the number of the stage that failed, in the order in which CaptureStage declares them. None collides with the root's stages, 11 to 16, the session's 1, or the time ABI's own 193 to 195.

Validation

At f28fdca, the head, on #877 at 060eefc.

Each of this PR's commits passes, on Linux x86_64 and Windows x86_64, nvx-rs's formatting, Clippy configurations, and host modules' layering test, and the runtime adapter's formatting and Clippy in each of its configurations, with template mode's tests: those of template everywhere, and those of worker::capture in the musl guest-only configuration on Linux.

At the head, on Linux x86_64 (toolchain 1.95.0, MSRV 1.89): check-nvx-rs's runtime adapter steps, in the protocol-only configuration, with 39 library tests passing, the host-only one, and the musl guest-only one, with 61 library tests passing, the 6 of template and the 5 of worker::capture among them, each with its MSRV check; and nvx-rs's cargo clippy --lib and host modules' layering test.

On Windows x86_64: nvx.py verify; nvx-rs's formatting, Clippy with all features and of the library, its tests with all features, and the layering test; and the runtime adapter's protocol-only and host-only configurations, with their MSRV checks.

The runtime adapter's VM suites at the head, whose VMs boot cold as before, with the artifacts of NVX's development release of dev at 60e6aab, whose guest, kernel, and OpenVMM inputs this PR shares, and its OpenVMM at the pinned e68871c: vm_layered 43, vm_restart 3, and vm_steps 3 passing, on Linux with MSHV, with OpenVMM's host CPU profile, and on Windows with WHP.

A CI run dispatched at the head passed every job but one: on Linux with MSHV, the runtime adapter's outbound_host_loopback_publishes_no_listener scenario, which this PR doesn't change, saw its guest listener exit at once with status 1, rather than wait out its 5 seconds. That is consistent with a collision of the listener's port, which the host draws from 49152 to 65535, with a guest port that the scenario's earlier outbound probes left in TIME_WAIT, which fails the listener's bind. The same job passed in #877's run, and the scenario passed with KVM and WHP in this run, and with MSHV and WHP above.

An earlier final review of the change found no issues, as did a final review of the rebase, its conflict resolutions, and the new maxcpus= token, and Copilot's review.

Enrique Saurez and others added 2 commits October 27, 2026 11:51
A platform template's snapshot must hold no root, so that every clone
that a restore starts attaches its own. The runtime adapter's guest now
has a template mode: on x86_64, its worker reads the kernel command line
before anything else, and in a VM whose command line names the platform
snapshot tier, it captures the VM before it assembles any root, through
nvx::guest::clone.

- template::boot splits the command line as the kernel does, through
  nvx::protocol::cmdline::parameters. A command line without
  nvx_snapshot_tier=platform boots cold, as before. With it, every
  parameter must be on the platform tier's allowlist, the tokens that
  OpenVMM's capture of a platform snapshot allows, and the command line
  may hold neither a double quote nor `--`.
- worker::capture attaches the control session with
  GuestConnection::attach, waits until the tty sent the attachment, and
  runs the capture's preparation and request behind the worker's fence
  of workload admission, which a close acquires. It reads nothing from
  the session, and serves no application session, before the capture.
- A refused command line, each failed stage of the capture, a capture
  that ended without a snapshot, and a restored clone, which the worker
  does not repair yet, end the VM with statuses 20 to 31, which the
  README tabulates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
A split at ASCII whitespace, as OpenVMM's checks of a platform snapshot's
command line make, keeps a vertical tab or the byte 0xa0 within a token,
but the kernel splits the command line there. Template mode reads the
command line as the kernel does, so it refuses a parameter that either
splits off a platform token's value, and finds a `--` that a vertical tab
separates. Pin both in its unit tests, and name them in the README.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The privileged capture path couples control-session state, guest shutdown, and time-ABI behavior, warranting human review supported by cross-backend VM validation.

No open issues 🎉

What changed in this PR

This PR adds template mode to the runtime adapter’s guest, completing the guest-side integration for #840. It captures a platform VM before root assembly so restored clones can attach their own roots.

Changes:

  • Detects template mode from kernel parameters and enforces the platform command-line allowlist.
  • Attaches and drains the control session before preparing and requesting capture, without serving application sessions.
  • Documents and tests capture ordering, admission fencing, and exit statuses, including temporary shutdown of restored clones pending #842.
File Description
tests/​runtime-adapter/​src/​worker/​capture.rs Implements capture preparation, session attachment, fencing, statuses, and unit tests.
tests/​runtime-adapter/​src/​worker.rs Exposes the x86_64 capture module and documents template behavior.
tests/​runtime-adapter/​src/​template.rs Defines template detection, the allowlist, shared statuses, and parser tests.
tests/​runtime-adapter/​src/​lib.rs Exports the template module in every configuration.
tests/​runtime-adapter/​src/​bin/​runtime-adapter-guest.rs Routes template workers to capture before normal root handling.
tests/​runtime-adapter/​README.md Documents template flow, failure statuses, and test coverage.

Review effort: Balanced

Base automatically changed from esaurez/840-pr1-capture-preparation-20261024 to dev October 27, 2026 19:44
@ppenna
Pedro Henrique Penna (ppenna) merged commit 4bd4068 into dev Oct 27, 2026
128 of 130 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the esaurez/840-pr2-template-mode-20261024 branch October 27, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants