Skip to content

nvx-rs: repair a clone's restore in the guest - #936

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
esaurez/842-pr2-repair-20261024
Oct 28, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
esaurez/842-pr2-repair-20261024

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #880; this PR retargets when its parent merges.
Chain: #880 → this PR → #937 → #939 → #940 → #941, on dev; #879, #881, #885, #886, #877, #932, #884, #887, and #933 have merged.

Part of #842 (PR 2 of 3: repair).

What and why

A clone that a restore of a template's snapshot starts must repair its restore before it admits any work: activate the restore's memory target, reseed the kernel's random number generator with the restore's fresh entropy, and acknowledge the restore exactly once. Only the guest's nvx-snapshot shell steps did that. This adds the repair to nvx::guest::clone, as typed calls in the caller's process, for any guest init or agent, and switches the clone's capture to the in-memory mode of #880, so that no file holds the restore's fresh entropy.

repair(restore), or a Repair that its caller may retry with Repair::run, takes the Restore that Restored::into_parts hands over, and returns the Repaired clone:

  1. It checks that the restore holds its seed and that its capture did not finish it, and that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online.
  2. With a memory target, it checks, before it adds any memory block, that the kernel's memory block size is 128 MiB, and that the restore's ranges continue the captured RAM in logical order, low RAM up to 3 GiB, then high RAM from 4 GiB, with no gap, overlap, or range in the MMIO aperture. It then adds each block that the kernel lacks and onlines each that is offline, through sysfs, as nvx-snapshot does, and verifies them.
  3. It reseeds the kernel's random number generator with the restore's owned Seed, through reseed::reseed, then drops the seed, which zeroes it. Nothing before it draws randomness.
  4. It finishes the restore through TimeAbi::restore_finish, with the acknowledgement when the host gates its input until then, which hands the deferred time work (the restore checks, the RCU grace period, the stall detectors' restoration, and the faster discipline cadence) to the time daemon that the caller started with TimeAbi::start_daemon and supervises.
  • Retries. The memory activation is idempotent: the repair derives the captured RAM's end once, before its first addition, and keeps it, so that a retry counts the blocks that it added as done and adds none twice. The reseed and the finish run at most once: a failure at or after the reseed, or a finish that never returned, is final, and every retry returns the same error, without a second reseed or acknowledgement. Once the repair succeeded, a retry returns the same Repaired.
  • Result. Repaired reports the clone's generation ID, its online CPUs, the size of its online memory blocks, and when it acknowledged the restore, for the clone's activation to check against what it expects. The repair takes that instant right before the finish, which acknowledges the restore before any deferred time work and may run that work before it returns, so the instant is no later than the acknowledgement, and a deadline that starts at it never ends late.
  • The fence. The repair never opens the caller's fence of workload admission: the caller keeps it closed through the repair, and opens it once the clone may admit executions, such as once it is activated.
  • Failures. Each is a RepairError, with its RepairStage and RepairCause, and a RangeFault for a refused range. nvx-rs: give a clone's failures documented statuses #937 gives each class a documented exit status.
  • Capture. CaptureBarrier::capture requests CaptureRequest::in_memory, so the Restore holds the seed, and Restored::entropy_path is gone.

Deviation from the issue: the memory capacity

The issue has the repair check the memory target against the captured capacity, with a test that refuses a target above it. The guest has no record of that capacity: version 4 of the restore packet carries none, and the guest's memory map shows only the RAM that it has. The host bounds the target instead: OpenVMM checks a restore's memory target against the snapshot's capacity before the clone runs. The guest checks what it can see, the block size and that the ranges are the canonical continuation of the captured RAM, and the module docs name the host as the owner of the capacity bound. So the tests replace "a target above the capacity, refused" with refusals, each before any block is added, of a gap, an overlap, a misaligned range, the wrong block size, a range past the end of low RAM or into the aperture, an extra low or high range, and ranges without a memory target, and with a retry after a failed addition that completes without adding a block twice.

Not in scope, as the issue says: CPU targets, so the repair refuses a restore whose processor target needs CPUs brought online; the machine ID; NVX's init.

API changes

  • New in nvx::guest::clone: repair, Repair, Repaired, RepairError, RepairStage, RepairCause, and RangeFault.
  • Removed: Restored::entropy_path, since the capture writes no entropy file.

Validation

This head is the previous one, 6baaf66, with its first commit, rebased without changes onto #880's rebased head, 9b6f035, which brings #933's rebase onto dev, OpenVMM at d2338ea2 and nvx-rs download among it; both commits' per-commit checks and the checks below passed on it, except the mutations and the merge into #941's head, which ran on 6baaf66, and the tests under all features grew only by the 55 that nvx-rs download adds to host::cli.

On this head, with the fix of the acknowledgement's instant:

Linux x86_64:

  • nvx-rs: cargo fmt --check; Clippy with -D warnings, --all-targets --all-features, --lib, and --lib --no-default-features, and with each feature alone; cargo test --all-features (1551 passed), cargo test (802 passed), and the --lib --no-default-features build and test (258 passed); tests/host_modules.rs (22 passed); the bundled-artifacts test; rustdoc with -D warnings; the MSRV (1.89) check; the aarch64-apple-darwin check; the guest's tests and MSRV check with only guest.
  • Guest binaries: nvx.py check-guest-binaries (musl Clippy and tests, 865 passed), build-guest-binaries, test-guest-binaries (33 passed), check-static-elf.
  • tests/runtime-adapter: fmt, Clippy, build, tests, and MSRV in its protocol-only, host-only, and musl guest configurations.
  • scripts/check_markdown_links.py.
  • As root: the privileged capture test, and the tests of guest::clone, the repair's included, and guest::reseed (48 passed).
  • On 6baaf66, seven mutations of the repair, each killed by a test: randomness drawn before the reseed, a finish that never returned run again, the captured RAM's end derived again on a retry, the seed kept after the reseed, the ranges' check skipped, the processor target left unchecked, and the acknowledgement's instant taken once the finish returned.
  • 6baaf66, merged locally, never pushed, into the head of runtime-adapter: activate a clone through Activate in the framing #941, the top of the stack above this PR, whose activation starts its deadline at the acknowledgement's instant: fmt, Clippy, the tests of guest::clone, the activation's included, guest::time, and guest::reseed (188 passed), and the runtime adapter's guest configuration.

Windows (x86_64-pc-windows-msvc), where the guest module doesn't build: nvx-rs fmt, Clippy (all features, --lib, no default features, and each feature alone), tests (all features 955 passed, default 817, no default features 258), tests/host_modules.rs, rustdoc, MSRV; tests/runtime-adapter in its protocol-only and host configurations; the Markdown links.

At the top of the chain, #937, which now holds both of this PR's commits: ci.yml run 40617461513, dispatched on its head, 5683164, since this chain's PRs don't target dev and get no PR CI, passed: 36 jobs succeeded, among them nvx-rs and the CLI on Linux and Windows, the guest artifacts, OpenVMM's unit tests and vmm-tests, the NVX microVM tests and the platform jobs on KVM, MSHV, and WHP; the 6 jobs that only pull requests or dev run were skipped. Before this rebase, on #937's 3d8aed5, which held this PR's first commit, with OpenVMM at e68871cc, the runtime adapter's VM suites passed 58 of 58 on WHP and on MSHV, whose template captures take this PR's in-memory capture, and ci.yml run 40588761279 passed; the fix of the acknowledgement's instant changes nothing that a VM shows, and ci.yml run 40608587945 passed on 6baaf66.

Tests added, with fakes for the time ABI, sysfs, and the reseed:

  • A clone is reseeded, then acknowledged once, and a re-entry returns the same clone; an ungated restore is finished without an acknowledgement; the acknowledgement's instant follows the reseed and precedes deferred work that the finish runs once it acknowledged.
  • No randomness before the reseed: the repair runs, in a process of its own, under a seccomp filter that ends the process at the first getrandom or open.
  • A memory target adds and onlines the blocks that continue the captured RAM; blocks that the kernel onlines itself are only added; the refusals above; a retry after a failed activation adds no block twice; a block that stays offline fails each activation.
  • The reseed and the finish run once, and their failures are final; a finish that never returned never runs again.
  • The online CPUs must be a prefix that meets the processor target; a restore without its seed, or that its capture finished, is refused; errors name their stage and their cause.
  • The live system reads and writes the kernel's CPU and memory interfaces, below a temporary directory.

A clone's repair in a VM is tested with #844.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged memory activation, entropy refresh, and restore acknowledgement need final human review alongside the pending integration validation.

No open issues 🎉

What changed in this PR

Adds in-process restore repair to nvx::guest::clone, building on the owned-seed support from #880 as the second part of #842.

Changes:

  • Adds retry-aware memory activation, kernel reseeding, and at-most-once restore completion while leaving workload admission fenced.
  • Switches clone captures to in-memory entropy and removes Restored::entropy_path.
  • Adds repair tests, syscall guards for randomness checks, and API documentation.
File Description
nvx-rs/​src/​guest/​time/​api.rs Adds restore construction helpers for tests.
nvx-rs/​src/​guest/​sys/​process.rs Extends seccomp helpers to forbid selected system calls in tests.
nvx-rs/​src/​guest/​sys/​mod.rs Exposes the new syscall guard to guest tests.
nvx-rs/​src/​guest/​reseed.rs Shares seed-drop observations with repair tests.
nvx-rs/​src/​guest/​mod.rs Documents guest clone repair capabilities.
nvx-rs/​src/​guest/​clone/​tests.rs Updates capture tests for owned, in-memory seeds.
nvx-rs/​src/​guest/​clone/​repair/​tests.rs Tests repair ordering, retries, failures, memory validation, and sysfs operations.
nvx-rs/​src/​guest/​clone/​repair.rs Implements repair, resource reporting, and typed errors.
nvx-rs/​src/​guest/​clone.rs Exports repair APIs and switches captures to in-memory entropy.
nvx-rs/​README.md Documents the repair API and lifecycle guarantees.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The privileged restore-repair path has only simulated coverage, leaving memory hotplug, acknowledgement gating, and deferred time work awaiting real restored-VM validation on KVM, MSHV, and WHP in #844.

No open issues 🎉

Review effort: Balanced

Enrique Saurez and others added 2 commits October 27, 2026 18:39
A clone that a restore of a template's snapshot starts now repairs its
restore through nvx::guest::clone::repair, or through a Repair that its
caller may retry, which returns the Repaired clone:

- It checks that the restore holds its seed and that its capture did
  not finish it, and that the online CPUs are a prefix that meets the
  restore's processor target, since it brings no CPU online.
- With a memory target, it checks, before it adds any memory block,
  that the kernel's memory block size is 128 MiB, and that the
  restore's ranges continue the captured RAM in logical order, low RAM
  up to 3 GiB, then high RAM from 4 GiB, with no gap, overlap, or range
  in the MMIO aperture. It derives the captured RAM's end once, before
  the first addition, and keeps it, so that a retry counts the blocks
  that it added as done. It then adds each block that the kernel lacks
  and onlines each that is offline, through sysfs, as nvx-snapshot
  does, and verifies them. The host bounds the target by the snapshot's
  capacity, of which the guest has no record.
- It reseeds the kernel's random number generator with the restore's
  owned seed, then drops the seed, which zeroes it. Nothing before it
  draws randomness.
- It finishes the restore through TimeAbi::restore_finish, with the
  acknowledgement when the host gates its input until then, which hands
  the deferred time work to the time daemon that the caller supervises.

The reseed and the finish run at most once: a failure at or after the
reseed, or a finish that never returned, is final, and every retry
returns the same error. Repaired reports the clone's generation ID, its
online CPUs, the size of its online memory blocks, and when it
acknowledged the restore. The repair never opens the caller's fence of
workload admission, which the caller opens once the clone may admit
executions.

The clone's capture now requests the in-memory mode, so that no file
holds the restore's fresh entropy, and Restored::entropy_path is gone.

Part of #842

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fbebd4aa-2ad7-427d-80ea-49358a645e9a
…nish

Repaired::acknowledged_at was taken once the restore's finish returned.
The finish acknowledges the restore before any of the deferred time
work, and without a time daemon it runs that work in line before it
returns, which may last up to the stall detectors' timeout. A repaired
clone's activation (#843) starts its deadline at the acknowledgement,
so the instant could start that deadline late, by as long as the
deferred work took, and let an activation run past its timeout.

The repair now takes the instant right before the finish, so it is no
later than the acknowledgement: a deadline that starts at it may end
early, by the finish's own steps before it acknowledges, but never
late. A test whose fake finish runs deferred work for 50 ms once it
acknowledged checks that the instant falls between the reseed and the
acknowledgement.

Part of #842

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fbebd4aa-2ad7-427d-80ea-49358a645e9a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants