Skip to content

nvx-rs: serve clones of each processor count from one template #926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into from
Oct 29, 2026
Merged

nvx-rs: serve clones of each processor count from one template #926

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c6f1e74
nvx-rs: bring a clone's processor target online in its repair
Oct 27, 2026
f87f1f1
nvx-rs: serve clones of each processor count from one template
Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 34 additions & 27 deletions nvx-rs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -574,15 +574,16 @@ template's capture, without I/O: the placeholder `distro`, which its
over a dormant scratch device, the sparse file `SCRATCH_NAME` in the capture's
transaction directory, with the dormant filesystem slot, and no NIC or one under
the template's policy, which allows neither ingress nor host-loopback access
both ways; `TemplateResources`, with every processor online at boot, and RAM and
a `--memory-capacity` in blocks of `MEMORY_BLOCK_SIZE`; the kernel command line
`nvx_sandbox=1`, so that the capture's whole command line holds tokens of
OpenVMM's platform allowlist alone, and no tenant data, which the tests check
with a copy of the allowlist's source; and the capture's options,
`--snapshot-tier platform`, a `--snapshot-destination`, `SNAPSHOT_NAME` in the
transaction directory, and `--snapshot-block-identity generation` with the
capture's bound-block `StorageGeneration`, so that the capture hashes no block
device.
both ways; `TemplateResources`, with the processors, 1, 2, 4, or 8, and those
online at boot, as many or fewer, and RAM and a `--memory-capacity` in blocks of
`MEMORY_BLOCK_SIZE`; the kernel command line `nvx_sandbox=1`, with `maxcpus=`
and the processors online at boot when they are fewer, so that the capture's
whole command line holds tokens of OpenVMM's platform allowlist alone, and no
tenant data, which the tests check with a copy of the allowlist's source; and
the capture's options, `--snapshot-tier platform`, a `--snapshot-destination`,
`SNAPSHOT_NAME` in the transaction directory, and
`--snapshot-block-identity generation` with the capture's bound-block
`StorageGeneration`, so that the capture hashes no block device.

`TemplateManifest::read` reads the `manifest.bin` that OpenVMM wrote for a
capture, at most `MAX_MANIFEST_LEN` bytes of a regular file that is not a link
Expand All @@ -593,8 +594,8 @@ identity is unbound, a fresh scratch, and the dormant filesystem slot.

`CaptureClass` holds the invariant inputs of a capture, known before it: the
runtime, the guest's build, the backend, the requested CPU profile, the
processors and RAM, the geometries of the disks, the network, and the adapter's
other `Invariants`.
processors, those online at boot, and RAM, the geometries of the disks, the
network, and the adapter's other `Invariants`.

`TemplateKey::new` checks that the capture made the machine that its class
requested, and derives the template's compatibility key, from metadata alone:
Expand Down Expand Up @@ -666,7 +667,9 @@ its key, which must speak the current contracts, the boot and control contracts
of the runtime that captured it included, and its snapshot; the template's
network exactly, as its key records it, or none for a template without a NIC, so
that a template serves the clones of its network policy, at its guest's address,
which OpenVMM restores from the snapshot, alone; a memory target in whole blocks
which OpenVMM restores from the snapshot, alone; a processor target of 1, 2, 4,
or 8 from the template's processors online at boot to its capacity; a memory
target in whole blocks
of `MEMORY_BLOCK_SIZE` from the template's RAM to its capacity; the clone's
image, which the check of its `DistroBinding` opens; its upper, either the
caller's scratch image, which the check of its `ScratchBinding` claims, and
Expand Down Expand Up @@ -703,9 +706,9 @@ or the scratch image's binding identity with the journal policy; the export, a
directory, with its plan's identity, and its mappings, the plan's binds, from 1
to 4,096, or no export and no mappings, which leaves the slot dormant; the
template NIC's policy identity, as its key records it, unpublished listeners,
and the guest's proxy URL; and the template's processors and the memory target.
The identity of a binding or of an export plan is the SHA-256 digest of a domain
and of its serde form, never of a file's content.
and the guest's proxy URL; and the processor and memory targets. The identity of
a binding or of an export plan is the SHA-256 digest of a domain and of its
serde form, never of a file's content.

The image must present the logical and physical block sizes of the template's
placeholder and be no longer than it, since a restore presents every clone's
Expand Down Expand Up @@ -734,7 +737,9 @@ key records it, unchanged, its export bound to the dormant slot, `--mount` with
the directory, and its hidden, re-exposed, and writable paths and its owner,
`--restore-snapshot` with the generation's snapshot in place, which no clone
copies or changes, so that clones of one generation restore it at once,
`--restore-memory` with the memory target, and `--restore-ready-path`, under the
`--restore-memory` with the memory target, `--restore-processors` with the
processor target, for a template whose processors were not all online at boot,
and `--restore-ready-path`, under the
template's backend, CPU profile, and processors, with a boot console that
discards its output, as cold boots have, and the control console, but no kernel,
initramfs, command line, or guest address, which the snapshot fixes.
Expand Down Expand Up @@ -1207,17 +1212,19 @@ may fail in a clone whose restore it could not repair. Each failure is a
`CaptureError`, with its `CaptureStage` and `CaptureCause`.

`repair`, or a `Repair` that the caller may retry, repairs the clone of a
`Restore` next: it checks that the online CPUs are a prefix that meets the
restore's processor target, since it brings no CPU online; with a memory target,
it checks, before it adds any memory block, that the kernel's block size is 128
MiB and that the restore's ranges continue the captured RAM in logical order,
with no gap, overlap, or range in the MMIO aperture, while the host bounds the
target by the snapshot's capacity, then adds and onlines the blocks through
sysfs, as `nvx-snapshot` does, and verifies them; it reseeds the kernel's random
number generator with the `Seed`, having drawn no randomness before, and drops
it; and it finishes the restore through `TimeAbi::restore_finish`, with the
acknowledgement when the host gates its input until then, which hands the
deferred time work to the time daemon that the caller supervises.
`Restore` next: it checks that the online CPUs are a prefix that the restore's
processor target does not undercut, brings the target's other CPUs online
through sysfs, and verifies them; with a memory target, it checks, before it
adds any memory block, that the kernel's block size is 128 MiB and that the
restore's ranges continue the captured RAM in logical order, with no gap,
overlap, or range in the MMIO aperture, while the host bounds the target by the
snapshot's capacity, then adds and onlines the blocks through sysfs, as
`nvx-snapshot` does, and verifies them; it reseeds the kernel's random number
generator with the `Seed`, having drawn no randomness before, and drops it; and
it finishes the restore through `TimeAbi::restore_finish`, with the CPUs that
the restore brought online and the acknowledgement when the host gates its input
until then, which hands the deferred time work to the time daemon that the
caller supervises.

It returns the `Repaired` clone, with its generation ID, its online CPUs and
memory, and when it acknowledged the restore, and never touches the fence. It
Expand Down
52 changes: 28 additions & 24 deletions nvx-rs/src/guest/clone.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,11 @@
//! runs one once. [`Repair::run`], in order:
//!
//! 1. checks that the restore holds its seed, and that its capture did not finish it;
//! 2. reads the online CPUs, which must be a prefix of the CPUs, and refuses a processor target
//! that they don't meet already: the repair brings no CPU online;
//! 2. reads the online CPUs, which must be a prefix of the CPUs, those that the clone captured
//! online. With a processor target, which may not be below them, it brings each CPU of the
//! target's prefix that is offline online, through sysfs, as `nvx-snapshot` does, and verifies
//! that the online CPUs are the target's prefix. Those are CPUs that the VM has had since its
//! boot, offline until then, so no device is plugged;
//! 3. with a memory target, checks it before it adds any memory block: the kernel's memory block
//! size must be 128 MiB, and the restore's ranges, aligned to it, must continue the RAM that
//! the clone captured, in logical order: the first starts where the blocks that the kernel
Expand All @@ -68,24 +71,25 @@
//! [`reseed::reseed`](crate::guest::reseed::reseed), then drops the seed, which zeroes its
//! bytes. Nothing before draws randomness, from `getrandom` or from `/dev/urandom`;
//! 5. finishes the restore through [`TimeAbi::restore_finish`], with the acknowledgement when
//! the host gates its input until then, which hands the deferred time work to the time
//! daemon: the restore checks, the stall detectors' grace period and release, and the
//! wall-clock discipline's fast cadence. The caller starts the daemon with
//! [`TimeAbi::start_daemon`] at boot, before the capture, and supervises it. A finish without
//! a running daemon fails with a violation of the time ABI's check C10, and in report-only
//! mode, it runs the work itself before it returns. The work may end after the clone is
//! ready, and its failures are fatal repair failures: the daemon reports them as violations
//! of the time ABI, which stop the VM, with the time ABI's status, unless the ABI is
//! report-only, since the daemon enforces it.
//! the host gates its input until then, and the CPUs that it brought online, which hands the
//! deferred time work to the time daemon: the restore checks, of the new CPUs too, the stall
//! detectors' grace period and release, and the wall-clock discipline's fast cadence. The
//! caller starts the daemon with [`TimeAbi::start_daemon`] at boot, before the capture, and
//! supervises it. A finish without a running daemon fails with a violation of the time ABI's
//! check C10, and in report-only mode, it runs the work itself before it returns. The work
//! may end after the clone is ready, and its failures are fatal repair failures: the daemon
//! reports them as violations of the time ABI, which stop the VM, with the time ABI's status,
//! unless the ABI is report-only, since the daemon enforces it.
//!
//! It returns the [`Repaired`] clone: its new VM generation ID, its number of online CPUs, the
//! size of its online memory blocks, and when it acknowledged the restore, which an activation
//! checks against the resources that it expects. The repair records how far it got, so that it
//! never adds a memory block twice, and runs the reseed and the finish at most once, whoever
//! retries it: a failure at or after the reseed is final. It never touches the fence. A failure is
//! a [`RepairError`], which names the [`RepairStage`] that failed, its [`RepairCause`], and the
//! repair's [`RepairProgress`], and selects the status with which the caller ends the VM,
//! [`RepairError::status`], which `nvx::protocol::status` documents.
//! never brings a CPU online twice nor adds a memory block twice, and runs the reseed and the
//! finish at most once, whoever retries it: a failure at or after the reseed is final. It never
//! touches the fence. A failure is a [`RepairError`], which names the [`RepairStage`] that
//! failed, its [`RepairCause`], and the repair's [`RepairProgress`], and selects the status with
//! which the caller ends the VM, [`RepairError::status`], which `nvx::protocol::status`
//! documents.
//!
//! # Strict mode
//!
Expand Down Expand Up @@ -256,14 +260,14 @@
//! reseed needs `CAP_SYS_ADMIN`. The capture reads the mount table of the caller's mount
//! namespace, which must be the one where the guest mounts its devices, such as its initial
//! namespace, and the repair reads the kernel's CPU and memory interfaces in sysfs, and writes
//! its memory interface. They mount nothing and start no thread, and they start no process and
//! collect no child's status, except a report-only finish without a running time daemon, which
//! runs the deferred time work itself, in a child that it waits for, as
//! [`TimeAbi::restore_finish`] tells. Apart from a violation of the time ABI in the capture's
//! preparation, which stops the VM unless the ABI is report-only, as [strict
//! mode](#strict-mode) tells, they never power the VM off, so a caller that is not PID 1 keeps
//! its own reaping, transport, and shutdown. The activation needs no privilege, and makes no
//! system call: of the clone, it changes only the caller's fence, which it opens. Its
//! the states of the CPUs that it brings online and its memory interface. They mount nothing and
//! start no thread, and they start no process and collect no child's status, except a
//! report-only finish without a running time daemon, which runs the deferred time work itself,
//! in a child that it waits for, as [`TimeAbi::restore_finish`] tells. Apart from a violation of
//! the time ABI in the capture's preparation, which stops the VM unless the ABI is report-only,
//! as [strict mode](#strict-mode) tells, they never power the VM off, so a caller that is not
//! PID 1 keeps its own reaping, transport, and shutdown. The activation needs no privilege, and
//! makes no system call: of the clone, it changes only the caller's fence, which it opens. Its
//! application, [`apply`](fn@apply), needs the guest's root user, in the guest's initial mount
//! namespace, as [`root::assemble`] does: for a GPT disk, it asks the kernel to reread the disk's
//! partition table, which needs `CAP_SYS_ADMIN`; it mounts the root, reads the account databases
Expand Down
Loading
Loading