runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822

Parent: #816. Sub-issue: #805 (executions). Needs: nothing outside #816; #813's adapter plan is merged. Unblocks: stage 1 of the snapshot/restore workstream (#834), PR by PR, as the order below shows; #859; #860's PR 2.

Goal. The in-repo test adapter runs real cold sandboxes on KVM, MSHV and WHP, through public APIs and its own framing:

• images staged and bound;
• a checked launch through the adapter plan;
• mapping tables from a host export;
• network policy;
• executions (runtime adapter: settle executions through cgroups and run them in a VM #805);
• stop and teardown.

#859 then runs its share of the consumer-contract scenarios on these VMs, and #624 counts it as the second adapter.

Today (dev d04d9fb):

• Plan:
  • runtime-adapter-host plans a layered VM with SandboxPlan::new(..., SandboxSettings::default()) (tests/runtime-adapter/src/layered.rs:70-75), not the adapter plan that sandbox: bind GPT disks, plan adapter launches, and stage registered images #813 added.
  • It passes OpenVMM the plan's paths, and --microvm-lifecycle managed (layered.rs:122-123), which OpenVMM accepts only with a workload identity (openvmm/openvmm_entry/src/cli_args/microvm.rs:1030-1034).
  • It binds no image, and doesn't hand OpenVMM checked files (PreparedLaunch).
• Session: after RootConfig, the guest answers RootReady or RootFailed, and the host pings and commits the launch (src/bin/runtime-adapter-host.rs:487-562).
• Mappings: the worker's mapping probes map guest-local sources under /tmp, in place of a host export. No host export reaches the guest:
  • nvx::guest::root mounts a share or an aggregate by binding its children (Root::mount_share, Root::mount_aggregate), but can't mount an export as the source of a mapping table;
  • NVX's shell does that (guest/common/nvx-init-agent:495-561), and the adapter replaces it.
• Network: none.
• VM tests, in CI on every backend (.github/workflows/run-nvx-microvm-tests.yml:1296-1331):
  • tests/vm_layered.rs, 19 scenarios: the four format and upper combinations, assembly faults, and probes of mappings and root profiles;
  • tests/vm_restart.rs: reattach, and a mismatched guest build left running.
• Executions run only in-process (runtime adapter: settle executions through cgroups and run them in a VM #805).

Decisions:

• No managed lifecycle in layered launches.
  • OpenVMM only checks arguments for --microvm-lifecycle managed and labels the outcome report with it. NVX's nvx-init-agent reads the token, and the adapter replaces that agent.
  • The adapter plan names no workload identity, and a platform template can carry neither token (openvmm/openvmm_helpers/src/snapshot/microvm.rs:2326-2350). So cold launches and template launches share one set of arguments.
  • Blockless launches keep the token: NVX's /init uses it to start the agent that the adapter replaces.
  • nvx-rs: decide whether adapters' layered launches need OpenVMM's workload lifecycle #861 decides whether adapters' layered launches need it at all.
• NVX's /init prelude. The adapter's VMs keep it: it mounts the pseudo-filesystems, runs the time step, configures the NIC from virtnet_* and starts the listener announcer, then hands over to /sbin/nvx-init-agent (guest/common/init:1-280). nvx-rs: run NVX's layered sandbox root from a Rust PID 1 on the guest library #817 keeps that hand-over.
• Identity. Each execution names its own identity, or the sandbox names one for all of them (runtime adapter: settle executions through cgroups and run them in a VM #805's PR 2), as the activation record allows (nvx-rs: a canonical activation record for platform clones #838).
• DNS inside the root is nvx-rs: give a layered root's workloads the DNS server that their network policy allows #860's.

Stacked PRs. Three stacks: A (PRs 1 to 3), B (PRs 4 to 8) and C (PRs 9 to 11). Each PR lists what it needs, lands with its own tests, and updates the adapter's README and module docs for what it changes.

Stack A: staging, the adapter plan and the checked handoff.

PR 1 of 11: staging and the adapter plan. Needs nothing. Host only, no VM.

• runtime-adapter-host stage STORE --distro FORMAT:PATH --distro-identity UUID [--upper scratch:PATH]:

  • verifies the image with DistroBinding::verify, which hashes it only when a manifest is beside it;
  • binds the scratch with ScratchBinding::bind;
  • records both bindings in the store.

  Staging is where hashing belongs; a launch never hashes.

• The launch plans with SandboxPlan::adapter, without NVX's sandbox settings, and drops --microvm-lifecycle managed.

• The image's identity becomes required, since a binding names it.

• The launch record moves to format 2, with the bindings.

• Tests (tests/layered_launch.rs, on Linux and Windows): the adapter plan's arguments, with no lifecycle and no workload identity; staging, and a manifest that doesn't match; the record.

PR 2 of 11: the checked handoff. Needs PR 1.

• launch:
  • checks the bindings from metadata (DistroBinding::check, ScratchBinding::check);
  • prepares a PreparedLaunch;
  • starts OpenVMM with its arguments, and inherited() as the VmmCommand's allowlist;
  • holds the checked files until the guest is ready, or the failed launch is torn down.
• A stale or missing binding fails before spawn.
• Tests, in VMs on KVM, MSHV and WHP: every scenario of vm_layered.rs and vm_restart.rs stages, then launches; a changed image or scratch, refused before spawn.

PR 3 of 11: replacement between the check and OpenVMM's open (C5). Needs PR 2.

• A test-only step hook pauses the host program, or exits it without unwinding, at a named launch step. runtime adapter: the consumer-contract scenarios C2, C3, C6 and C7 on real VMs #859 reuses it.
• Tests, in VMs: the image and the scratch replaced at their paths while the launch is paused after its checks.
  • On Linux, the guest assembles the checked originals.
  • On Windows, the replacement fails while the checked files are held.

Stack B: Ready, mapping tables and Start.

PR 4 of 11: the framing. Needs nothing.

• RootReady (kind 7) becomes Ready, which carries the guest agent's build. nvx-rs: the restricted control state and Activate after a restore, in the guest library and the test adapter's framing #843's PR 3 appends ReadyActivation (nvx-rs: a canonical activation record for platform clones #838).
• New kinds:
  • Maps, which carries the MAPS payloads of nvx::protocol (encode_maps_payloads, decode_guest_maps_payload, nvx-rs/src/protocol/maps.rs:41, :150);
  • Mapped and MapFailed;
  • Start and Started.
• The root configuration gains the mapping count.
• Tests, in every configuration: each new kind's encoding, split at every byte, and its refusals.

PR 5 of 11: the export as a mapping source, in nvx::guest. Needs nothing. This is #822's one library addition.

• Root::mount_export(tag, access):
  • mounts the export's virtio-fs device, nosuid and nodev, in the runtime directory, outside the root;
  • owns the mount, and unmounts it after the mappings at teardown;
  • returns its SourceRoot, for Root::map.
• A strict parser, in nvx::protocol, of the tokens that OpenVMM appends for an aggregate export (virtfs_dir=, virtfs_tag=, virtfs_mode=, virtfs_aggregate=1): all or none, each once, with the checks of NVX's shell (guest/common/nvx-init-agent:503-561).
• nvx-rs: run NVX's layered sandbox root from a Rust PID 1 on the guest library #817 replaces the shell's export mount with them, and nvx-rs: activate a clone's root for GPT images and either upper, with its export, network settings and workload view #845's PR 3 mounts a clone's export with them.
• Tests: the parser's vectors; with fakes, the mount, its teardown order, and a refused tag.

PR 6 of 11: the guest's order. Needs PRs 4 and 5.

• After assembly, the worker mounts the export when the command line names one, then answers Ready.
• It applies Maps in order with Root::map. A refused or failed mapping latches the root and answers MapFailed. Zero mappings do no work.
• It refuses every execution before Start, and Start before every announced mapping is applied.
• Tests, in guest_session.rs with a fake root: the order; partial tables; a failed mapping that latches the root; executions and Start refused out of order.

PR 7 of 11: the host's order. Needs PRs 2 and 6.

• launch options build an ExportRequest for filesystem::plan:

  • --share HOST:TARGET[:ro], the shares;
  • --deny PATH, host paths hidden inside them;
  • --reserved TARGET, the adapter's own reserved targets, beside the container root's (ReservedTargets, nvx-rs/src/host/filesystem.rs:83-110).

• --share-root DIR names the host directories that the adapter authorizes. A share outside them is refused before planning.

• Before the launch, ExportPlan::verify runs, and ExportPlan::openvmm_arguments attaches the export.

• The sequence:

  1. Ready, with the guest build checked against the host's;
  2. the table, in order;
  3. Start.

  The launch commits after Started. A failure, or another guest build, tears the VM down.

• Tests, in host_session.rs with a test broker: the sequence; each failure, torn down.

PR 8 of 11: mapping tables in VMs (C4). Needs PR 7.

• The mapping probes move from /tmp sources to the host export.
• Tests, on KVM, MSHV and WHP:
  • zero mappings, which do no work;
  • an ordered table in each of the four format and upper combinations, whose writes reach only the writable host sources;
  • a read-only mapping inside a writable one (nvx-rs: enforce read-only mappings nested in read-write mappings on the host #552);
  • a failed mapping, which blocks Start and tears the VM down;
  • a denied path; the adapter's reserved targets, and an unauthorized host path, refused before launch.

Stack C: network policy.

PR 9 of 11: the policy on the plan. Needs PR 1.

• launch --network FILE takes a SandboxNetwork in the plan's serde form (nvx-rs/src/host/sandbox/plan.rs:74-99), and puts it on the adapter plan, which renders it through network::openvmm_arguments. The record keeps it.
• Tests: the rendered arguments for each element of a policy; refusals before launch.

PR 10 of 11: the guest's settings. Needs PR 9, and #805's PR 2.

• The host gives executions the proxy's URL (SandboxPlan::proxy_url) in their environment.
• The root configuration says whether listeners are published, and executions enter with that Entry::publication. While listeners are published, Entry::check refuses NET_BIND_SERVICE and NET_RAW.
• The NIC, its addresses and the listener announcer stay with NVX's /init.
• Tests: the environment; the refusals at stage 0 while listeners are published.

PR 11 of 11: enforcement in VMs (C4). Needs PR 10, and #805's PR 3.

• Tests, on EROFS over scratch, on KVM, MSHV and WHP, as NVX's proxy-only scenario does (scripts/nvx_tools/sandbox_proxy_tests.py):
  • an open network first, as the positive control;
  • deny;
  • an allow rule;
  • proxy-only, with destinations that never arrive;
  • host loopback, outbound and allowed, with a listener that the host reaches;
  • a forward.

Order. The three stacks proceed in parallel:

• PRs 1, 4 and 5 need nothing.
• Across stacks, PR 7 needs PR 2, and PR 9 needs PR 1.
• PRs 10 and 11 also need runtime adapter: settle executions through cgroups and run them in a VM #805's PRs 2 and 3.

Stage 1 of the snapshot/restore workstream waits for these PRs:

Not in scope:

• the consumer-contract scenarios C2, C3, C6 and C7 on real VMs (runtime adapter: the consumer-contract scenarios C2, C3, C6 and C7 on real VMs #859);
• DNS inside the root (nvx-rs: give a layered root's workloads the DNS server that their network policy allows #860);
• NVX's adapter;
• library changes beyond PR 5.

Acceptance:

• On KVM, MSHV and WHP, in CI: staged images launched through the checked handoff, in all four format and upper combinations; mapping tables; network policy; executions (runtime adapter: settle executions through cgroups and run them in a VM #805); stop and teardown.
• It uses documented public APIs only, with default features off and its own lockfile.
• runtime adapter: settle executions through cgroups and run them in a VM #805 is done.

Carries over: #622's scope 8 (A4 PR 8).