You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nvx-rs: prepare a template's guest for a capture before root assembly - #877
A platform template's snapshot must hold no tenant's root, so that every clone that a restore starts attaches its own image, upper, and export. Today only the shell nvx-snapshot prepares a capture, after the layered init has assembled the root. This adds nvx::guest::clone, through which a privileged guest init or agent, whether or not it is PID 1, prepares its guest for a capture before its root's assembly, behind its own fence of workload admission, and requests the capture.
Part of #840: the first of its two pull requests. The second, #932, adds a template mode to the test adapter's guest, which uses this module; #841 and #842 build on it too.
The preparation (nvx-rs/src/guest/clone.rs). CapturePreparation::new(fence, devices) takes the caller's AdmissionFence and the template's Devices: the distro and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag. CapturePreparation::prepare, in order:
closes the fence, so that no execution starts. Closing acquires the fence: AdmissionFence::close fails while the fence is closed, such as by another preparation through another handle of the same admission state, and a preparation opens only a fence that it closed;
reads at most MOUNT_TABLE_LIMIT (1 MiB) of /proc/self/mountinfo, and refuses, with a Mounted that names the first such mount, anything mounted from the distro device or one of its partitions, the scratch device, an overlay, or the filesystem tag, bind mounts included;
waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as nvx-time pre-capture does,
and returns an owned CaptureBarrier. A preparation that fails opens the fence again, and leaves the detectors' settings as they were.
The capture.CaptureBarrier::capture reads the VM generation ID and calls TimeAbi::capture with the platform tier's request, which keeps no scratch. It returns:
Captured::Continued, once it opened the fence and restored the detectors' settings;
or, in a clone, Captured::Restored, with the fence still closed and the detectors still suppressed. Restored::into_parts hands the Restore that repair (nvx-rs: repair a restored clone in the guest, with an owned seed and one acknowledgment #842) takes next, and the closed fence, to the caller, which opens the fence once repair acknowledged the restore. The time daemon restores the detectors' settings when the restore finishes.
CaptureBarrier::cancel, a dropped barrier, and a failure to read the VM generation ID open the fence, then restore the detectors' settings. A capture request that fails restores the settings but leaves the fence closed. Each failure is a CaptureError, which names the CaptureStage that failed and its CaptureCause.
The mount table (nvx-rs/src/guest/clone/mounts.rs). A strict parser of the kernel's mountinfo lines, with its escapes, refuses any line of another form. A mount on a block device is from the distro or the scratch device when the device's canonical sysfs entry, /sys/dev/block/MAJ:MIN, is below the device window's sysfs directory, which covers a GPT disk's partitions, those of the extended major included. A virtiofs mount whose source is the tag is from the host filesystem device, which covers a host export and its bind mounts, and any overlay mount blocks a capture. A table longer than the limit, a malformed line, or a mount on a block device that sysfs doesn't show fails the check; a window whose device the kernel hasn't added has nothing mounted from it.
The time ABI (nvx-rs/src/guest/time/):
TimeAbi::pre_capture and TimeAbi::cancel_capture are crate-internal typed calls, through which nvx-time pre-capture and nvx-time cancel-capture now run, with unchanged diagnostics and statuses.
Like the other typed calls, pre_capture fails when the thread may still hold some of the permission for the time ABI's ports. A preparation that fails but still holds the detectors' suppression says so, and its caller restores their settings: nvx-time pre-capture before it fails, so it no longer leaves them suppressed after a port-release failure, and the barrier before it reports the failure.
The stall detectors' suppression now saves under the lock that its restore takes. A save that fails undoes only what it changed, so it never restores another capture's record, and keeps its record when it can't undo every change.
Supporting changes.nvx::guest::root shares its lookup of a virtio MMIO window's sysfs directories; root assembly finds its devices as before. The crate's README and documentation list the module.
Design notes
Restored clones. The issue specifies that Captured::Restored keeps the detectors suppressed, and leaves the fence's state open. Guide 5.5.7 runs nothing before the restore's acknowledgement, and 5.5.8 thaws the workload only after it, so the fence stays closed, and the caller opens it after repair.
Failed requests.TimeAbi::capture can fail after it found a restore packet, in a clone whose repair it couldn't complete, which the barrier can't tell from a source's failure. A failed request therefore leaves the fence closed, and only restores the detectors' settings; the caller ends the VM either way. A rejected or rolled-back capture is not a failure: it returns Captured::Continued, with both barriers released.
Release order. The barrier opens the fence first, then restores the detectors' settings, as guide 5.3.2 orders it. A failure to restore them is a CaptureStage::Release error, after which the caller ends the VM, unless the thread may still hold port permission, which takes precedence.
Entropy. A restore writes its entropy to /run/nvx/restore-entropy, as nvx-snapshot does today.
Validation
At 060eefc, the head, rebased onto #886 at 7c97c9c without conflicts or changes to its own lines, and with no VM: the module's tests use fakes for the mount table, the fence, and the time ABI, as the issue specifies, and temporary procfs, sysfs, and detector trees for the live lookups and the suppression.
Each of this PR's commits passes, on Linux x86_64, cargo fmt --check; cargo clippy with -D warnings with all features and targets, of the library with default and without features, and with the guest feature alone; the host modules' layering test; the macOS check; and the guest::{clone,time,root} tests; and on Windows x86_64, the same formatting, Clippy, and layering checks.
At the head, on Linux x86_64 (toolchain 1.95.0, MSRV 1.89), check-nvx-rs: cargo fmt --check; cargo clippy with --all-targets --all-features, --lib, --lib --no-default-features, and the guest feature alone, -D warnings; cargo test with all, default, and no features, 1,434 library tests passing with all features; the host modules' layering test; the guest::{clone,time,root} tests five more times; the bundled artifacts test; cargo doc with -D warnings; cargo +1.89 check; the macOS check; the guest binaries' lint, tests, build, and check; and the runtime adapter's protocol-only, host-only, and musl guest-only configurations, with their MSRV checks.
On Windows x86_64: nvx.py verify; check-nvx-rs's host steps, with 866 library tests passing with all features, and the layering test; and the runtime adapter's protocol-only and host-only configurations.
CI's pull-request run at the head, which started before this PR's base moved to #886's branch, passed. A run dispatched at #932's head, which holds this PR, passed every job but one, a runtime adapter network scenario on Linux with MSHV that neither PR changes; #932 describes it.
A final review of the change, and Copilot's review, found and fixed: a restored clone that opened its fence before repair, in the code and then in the module's example; a preparation that skipped the time ABI's port-release check; a suppression save that could restore another capture's record, or lose its own; cleanup errors that could hide a retained port permission, or, in a failed capture, be dropped; and a second preparation that could reopen a fence that another preparation's barrier held closed.
A final review of the rebase, and Copilot's review at the head, found no issues.
Adds typed guest-side preparation for tenant-free platform snapshots before root assembly, establishing the capture API needed by the upcoming template adapter and clone-repair work.
Changes:
Adds capture preparation, admission fencing, mount checks, and capture outcomes.
Shares time-ABI preparation calls and strengthens detector-suppression rollback.
Adds lifecycle and lookup tests, plus public API documentation.
File
Description
nvx-rs/src/lib.rs
Lists the new guest capture API.
nvx-rs/src/guest/time/system.rs
Serializes suppression saves and restores; adds rollback tests.
nvx-rs/src/guest/time/mod.rs
Exposes preparation errors internally.
nvx-rs/src/guest/time/commands.rs
Routes capture preparation commands through typed calls.
nvx-rs/src/guest/time/api.rs
Adds typed preparation, cancellation, and permission-error handling.
nvx-rs/src/guest/root/kernel.rs
Shares kernel interface paths internally.
nvx-rs/src/guest/root/block.rs
Extracts shared MMIO-window lookup.
nvx-rs/src/guest/root.rs
Re-exports shared lookup helpers internally.
nvx-rs/src/guest/mod.rs
Exposes the x86-64 clone module.
nvx-rs/src/guest/clone/tests.rs
Tests capture ordering, cleanup, outcomes, and device lookups.
nvx-rs/src/guest/clone/mounts.rs
Parses mount information and identifies capture-blocking mounts.
nvx-rs/src/guest/clone.rs
Implements preparation, barriers, outcomes, and typed errors.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Privileged snapshot transitions, admission fencing, and detector-state recovery warrant final human review, with cross-backend VM validation deferred to follow-up work.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Example opens fence before repair and restore acknowledgment
nvx-rs/src/guest/clone.rs:81
The example opens admission immediately after into_parts(), but neither that method nor the default capture request completes repair or acknowledges the restore. Following this example would admit workloads before reseeding/repair and leave the detectors suppressed because restore_finish was never called. Please show an explicit repair-and-acknowledgment step before fence.open(), or stop the example with the fence closed until that step is available.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Snapshot isolation, restore-time admission fencing, and shared detector-state cleanup warrant final human review, with VM integration validation deferred to follow-up work.
Merges the template mode of the runtime adapter's guest (#840 PR 2, on
#822's chain and #877) into the capture and publication of templates
(#887, on #839's stack).
The adapter host's launch keeps both sides: it reserves its launch with
a LaunchId, as the supervisor's launch now requires, and its OpenVMM
inherits the files that the prepared launch checked. The command's
closure is passed where the launch takes it, since the inherited handles
borrow the prepared launch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 32eea669-a52c-4a53-87fa-521c498537e5
A platform template's snapshot must hold no tenant root, so that each
clone attaches its own image, upper, and export. nvx::guest::clone gives
any guest init or agent the preparation that the shell nvx-snapshot
lacks: CapturePreparation closes the caller's AdmissionFence, refuses to
capture while the mount table shows anything mounted from the distro
device or a partition of it, the scratch device, the host filesystem
device's tag, or an overlay, bind mounts included, then waits for the
time ABI's boot checks, runs its capture checks, and suppresses the
stall detectors. The CaptureBarrier that it returns requests the capture
through TimeAbi::capture with the platform tier's request, and releases
the fence and the detectors when the capture continues, fails, or is
canceled, or the barrier is dropped. A restored clone returns with the
fence open and the detectors suppressed, for repair to finish.
The time module now offers the pre-capture and cancel-capture steps as
crate-internal typed calls, which nvx-time's commands run with unchanged
diagnostics, and the root module shares its MMIO-window lookup.
Part of #840
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
…ression
Review of the capture preparation found three gaps, which this fixes.
A restored clone runs nothing before repair acknowledges the restore, so
the barrier no longer opens the fence on Captured::Restored: Restored
holds the caller's fence, still closed, and Restored::into_parts hands
it over with the restore. A capture request that fails may fail in a
clone whose restore it could not repair, so it restores the stall
detectors' settings but leaves the fence closed too.
TimeAbi::pre_capture now fails, as the other calls do, when the thread
may still hold some of the permission for the time ABI's ports, once it
restored the detectors' settings that it saved. nvx-time pre-capture
still reports that once, but no longer leaves the detectors suppressed
after it.
The detectors' suppression now saves under the lock that its restore
takes, so that no restore runs between its reads and its writes, and a
save that fails undoes only what it changed: it never restores another
capture's record.
Part of #840
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
A save of the stall detectors' settings that failed to suppress one, then
to undo an earlier suppression, removed its record anyway: the detector
stayed suppressed, and its original setting was lost, so a later capture
could record the suppressed value as the original one.
The save now keeps its record unless it undid every change, and reports
which of the two happened. A preparation that kept the record, or that
suppressed the detectors before the thread failed to release the time
ABI's ports, now tells its caller that it holds their suppression, which
the caller releases as after a preparation that succeeded: nvx-time
pre-capture restores the settings before it fails, as it did before, and
the capture barrier restores them, or fails with a Release error.
The tests inject the failing writes instead of relying on a read-only
sysctl.
Part of #840
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
A preparation that failed because the thread may still hold some of the
permission for the time ABI's ports, after it suppressed the stall
detectors, reported a failure of the barrier to restore their settings
instead, which hid that the thread must start neither a thread nor a
process. The port release's failure now takes precedence, and the
barrier's restore stays best effort.
Part of #840
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
A failure to read the VM generation ID, or a capture request that failed,
left the barrier's release to its drop, which discarded a failure to
restore the stall detectors' settings. Both paths now release explicitly,
as a failed preparation does, and report that failure as a Release error,
unless the thread may still hold some of the permission for the time
ABI's ports, which takes precedence. A failed request still leaves the
fence closed.
Part of #840
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
The module's example opened a restored clone's admission fence as soon as
it took the fence from the restore, before the clone reseeded and finished,
and if needed acknowledged, its restore. Show those steps first, as the
time module's example does, so the example keeps nothing running before
the restore's acknowledgement.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
A caller may lend preparations several handles of one admission state.
With a close that succeeds on a closed fence, a second preparation, which
would fail with the first one's suppression pending, reopened the fence
while the first barrier still held it, and executions could start after
the first preparation's mount check.
AdmissionFence::close now fails while the fence is closed, whoever closed
it, and a failure leaves the fence as it was. Since a preparation never
opens a fence that failed to close, it opens only one that it closed. The
example's fence refuses a second close, and the tests' fence shares its
state between handles, refuses a close while closed, and checks that
nothing opens an open fence, with tests of two preparations behind one
gate and of a preparation behind a restored clone's closed fence.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da708b6-0bda-4804-b01b-936cdc5f054a
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The current fake-based tests need planned KVM/MSHV/WHP integration coverage to validate snapshot isolation’s reliance on live mountinfo/sysfs identities and admission-fence persistence across restore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A platform template's snapshot must hold no tenant's root, so that every clone that a restore starts attaches its own image, upper, and export. Today only the shell
nvx-snapshotprepares a capture, after the layered init has assembled the root. This addsnvx::guest::clone, through which a privileged guest init or agent, whether or not it is PID 1, prepares its guest for a capture before its root's assembly, behind its own fence of workload admission, and requests the capture.Part of #840: the first of its two pull requests. The second, #932, adds a template mode to the test adapter's guest, which uses this module; #841 and #842 build on it too.
Chain: #886 → this PR → #932 → #884 → #887 → #841 PR 3, on
dev; #879, #881 and #885 have merged.What changed
The preparation (
nvx-rs/src/guest/clone.rs).CapturePreparation::new(fence, devices)takes the caller'sAdmissionFenceand the template'sDevices: thedistroand scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag.CapturePreparation::prepare, in order:AdmissionFence::closefails while the fence is closed, such as by another preparation through another handle of the same admission state, and a preparation opens only a fence that it closed;MOUNT_TABLE_LIMIT(1 MiB) of/proc/self/mountinfo, and refuses, with aMountedthat names the first such mount, anything mounted from the distro device or one of its partitions, the scratch device, an overlay, or the filesystem tag, bind mounts included;nvx-time pre-capturedoes,and returns an owned
CaptureBarrier. A preparation that fails opens the fence again, and leaves the detectors' settings as they were.The capture.
CaptureBarrier::capturereads the VM generation ID and callsTimeAbi::capturewith the platform tier's request, which keeps no scratch. It returns:Captured::Continued, once it opened the fence and restored the detectors' settings;Captured::Restored, with the fence still closed and the detectors still suppressed.Restored::into_partshands theRestorethat repair (nvx-rs: repair a restored clone in the guest, with an owned seed and one acknowledgment #842) takes next, and the closed fence, to the caller, which opens the fence once repair acknowledged the restore. The time daemon restores the detectors' settings when the restore finishes.CaptureBarrier::cancel, a dropped barrier, and a failure to read the VM generation ID open the fence, then restore the detectors' settings. A capture request that fails restores the settings but leaves the fence closed. Each failure is aCaptureError, which names theCaptureStagethat failed and itsCaptureCause.The mount table (
nvx-rs/src/guest/clone/mounts.rs). A strict parser of the kernel'smountinfolines, with its escapes, refuses any line of another form. A mount on a block device is from the distro or the scratch device when the device's canonical sysfs entry,/sys/dev/block/MAJ:MIN, is below the device window's sysfs directory, which covers a GPT disk's partitions, those of the extended major included. Avirtiofsmount whose source is the tag is from the host filesystem device, which covers a host export and its bind mounts, and anyoverlaymount blocks a capture. A table longer than the limit, a malformed line, or a mount on a block device that sysfs doesn't show fails the check; a window whose device the kernel hasn't added has nothing mounted from it.The time ABI (
nvx-rs/src/guest/time/):TimeAbi::pre_captureandTimeAbi::cancel_captureare crate-internal typed calls, through whichnvx-time pre-captureandnvx-time cancel-capturenow run, with unchanged diagnostics and statuses.pre_capturefails when the thread may still hold some of the permission for the time ABI's ports. A preparation that fails but still holds the detectors' suppression says so, and its caller restores their settings:nvx-time pre-capturebefore it fails, so it no longer leaves them suppressed after a port-release failure, and the barrier before it reports the failure.Supporting changes.
nvx::guest::rootshares its lookup of a virtio MMIO window's sysfs directories; root assembly finds its devices as before. The crate's README and documentation list the module.Design notes
Captured::Restoredkeeps the detectors suppressed, and leaves the fence's state open. Guide 5.5.7 runs nothing before the restore's acknowledgement, and 5.5.8 thaws the workload only after it, so the fence stays closed, and the caller opens it after repair.TimeAbi::capturecan fail after it found a restore packet, in a clone whose repair it couldn't complete, which the barrier can't tell from a source's failure. A failed request therefore leaves the fence closed, and only restores the detectors' settings; the caller ends the VM either way. A rejected or rolled-back capture is not a failure: it returnsCaptured::Continued, with both barriers released.CaptureStage::Releaseerror, after which the caller ends the VM, unless the thread may still hold port permission, which takes precedence./run/nvx/restore-entropy, asnvx-snapshotdoes today.Validation
At
060eefc, the head, rebased onto #886 at7c97c9cwithout conflicts or changes to its own lines, and with no VM: the module's tests use fakes for the mount table, the fence, and the time ABI, as the issue specifies, and temporary procfs, sysfs, and detector trees for the live lookups and the suppression.Each of this PR's commits passes, on Linux x86_64,
cargo fmt --check;cargo clippywith-D warningswith all features and targets, of the library with default and without features, and with theguestfeature alone; the host modules' layering test; the macOS check; and theguest::{clone,time,root}tests; and on Windows x86_64, the same formatting, Clippy, and layering checks.At the head, on Linux x86_64 (toolchain 1.95.0, MSRV 1.89), check-nvx-rs:
cargo fmt --check;cargo clippywith--all-targets --all-features,--lib,--lib --no-default-features, and theguestfeature alone,-D warnings;cargo testwith all, default, and no features, 1,434 library tests passing with all features; the host modules' layering test; theguest::{clone,time,root}tests five more times; thebundledartifacts test;cargo docwith-D warnings;cargo +1.89 check; the macOS check; the guest binaries' lint, tests, build, and check; and the runtime adapter's protocol-only, host-only, and musl guest-only configurations, with their MSRV checks.On Windows x86_64:
nvx.py verify; check-nvx-rs's host steps, with 866 library tests passing with all features, and the layering test; and the runtime adapter's protocol-only and host-only configurations.CI's pull-request run at the head, which started before this PR's base moved to #886's branch, passed. A run dispatched at #932's head, which holds this PR, passed every job but one, a runtime adapter network scenario on Linux with MSHV that neither PR changes; #932 describes it.
A final review of the change, and Copilot's review, found and fixed: a restored clone that opened its fence before repair, in the code and then in the module's example; a preparation that skipped the time ABI's port-release check; a suppression save that could restore another capture's record, or lose its own; cleanup errors that could hide a retained port permission, or, in a failed capture, be dropped; and a second preparation that could reopen a fence that another preparation's barrier held closed.
A final review of the rebase, and Copilot's review at the head, found no issues.