From abb40cd678730ed020805743732ee1e40a146c27 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 14:21:21 -0700 Subject: [PATCH 1/2] nvx-rs: repair a clone's restore in the guest A clone that a restore of a template's snapshot starts now repairs its restore through nvx::guest::clone::repair, or through a Repair that its caller may retry, which returns the Repaired clone: - It checks that the restore holds its seed and that its capture did not finish it, and that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online. - With a memory target, it checks, before it adds any memory block, that the kernel's memory block size is 128 MiB, and that the restore's ranges continue the captured RAM in logical order, low RAM up to 3 GiB, then high RAM from 4 GiB, with no gap, overlap, or range in the MMIO aperture. It derives the captured RAM's end once, before the first addition, and keeps it, so that a retry counts the blocks that it added as done. It then adds each block that the kernel lacks and onlines each that is offline, through sysfs, as nvx-snapshot does, and verifies them. The host bounds the target by the snapshot's capacity, of which the guest has no record. - It reseeds the kernel's random number generator with the restore's owned seed, then drops the seed, which zeroes it. Nothing before it draws randomness. - It finishes the restore through TimeAbi::restore_finish, with the acknowledgement when the host gates its input until then, which hands the deferred time work to the time daemon that the caller supervises. The reseed and the finish run at most once: a failure at or after the reseed, or a finish that never returned, is final, and every retry returns the same error. Repaired reports the clone's generation ID, its online CPUs, the size of its online memory blocks, and when it acknowledged the restore. The repair never opens the caller's fence of workload admission, which the caller opens once the clone may admit executions. The clone's capture now requests the in-memory mode, so that no file holds the restore's fresh entropy, and Restored::entropy_path is gone. Part of #842 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbebd4aa-2ad7-427d-80ea-49358a645e9a --- nvx-rs/README.md | 6 +- nvx-rs/src/guest/clone.rs | 133 ++-- nvx-rs/src/guest/clone/repair.rs | 798 ++++++++++++++++++++++++ nvx-rs/src/guest/clone/repair/tests.rs | 800 +++++++++++++++++++++++++ nvx-rs/src/guest/clone/tests.rs | 32 +- nvx-rs/src/guest/mod.rs | 5 +- nvx-rs/src/guest/reseed.rs | 8 +- nvx-rs/src/guest/sys/mod.rs | 2 + nvx-rs/src/guest/sys/process.rs | 88 ++- nvx-rs/src/guest/time/api.rs | 41 ++ 10 files changed, 1837 insertions(+), 76 deletions(-) create mode 100644 nvx-rs/src/guest/clone/repair.rs create mode 100644 nvx-rs/src/guest/clone/repair/tests.rs diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 0b3dc9dd8..9ff1e576c 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -119,7 +119,7 @@ on Linux and Windows. | `nvx::guest::exec` | `Execution`: drives one program that the caller spawned with the program's ends of its `Pipes`, or that the caller started itself and adopts with its own ends of the program's pipes and a `Child` that names it, without blocking, from the caller's own loop or reactor, and reports each step as an `Event`: the launch report, which an adopted program has at once, the output of each stream, kill failures, and the `Outcome`. It reads a stream only as far as the caller's `grant` allows, so a stalled sink stalls its own execution alone, writes the program's standard input without `SIGPIPE`, and ends the program on a cancellation, a timeout, or a disconnection, with `SIGKILL` through the program's pidfd to the program and, if the caller asks, its process group, while no collector has taken its status, and, through the opened directory of a cgroup that the caller names, to that cgroup. It closes the program's input once the program has ended. It waits for its program alone, by a pidfd, or takes the status from the caller's collector with `observe`; with `Options::caller_reaps`, that collector is the program's only one. NVX's managed agent is one caller | | `nvx::guest::exec::binding` | `Binding`: an `nvx::exec::Runner` bound to an `Execution`, the glue that a guest adapter would otherwise write. The runner's read allowances become grants, its discards discards, the input that it accepted writes and then a close, its terminations a cancellation, a disconnection, or a termination, and its settlement a kill of the execution's cgroup and checks until no process occupies it; without a cgroup, nothing proves that the program's descendants are gone, so the execution is fenced. It feeds the execution's events back to the runner, statuses that a caller who reaps hands over with `observe` included, and never blocks: the caller polls its descriptors and its input writer until its deadline, and each call takes a bounded number of steps, so that no execution holds the caller's loop | | `nvx::guest::time` | x86_64: `TimeAbi`, the steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent: the cold-boot time step and, separately, the start of the time daemon in a child process that the caller supervises, the VM generation ID, a `CaptureRequest` that repairs a `Restore`, and a `RestoreFinish`. The request's mode selects how a restore hands over its fresh entropy: the file mode, which `nvx-time` keeps, writes it to a file, and the in-memory mode returns it as the `Restore`'s owned `Seed`, and writes no file. A call takes the permission for OpenVMM's ports on the calling thread for each port transaction alone, and fails with `TimeError::PortRelease` if the thread may still hold some of it, which a process that it started would inherit, and from which `nvx::guest::spawn` starts none. No call waits for a child that it did not start. `nvx-time` runs its steps through them | -| `nvx::guest::clone` | x86_64: the preparation of a template's guest for a capture before its root's assembly, for a guest init or agent, whether or not it is PID 1. `CapturePreparation`, of the caller's `AdmissionFence` and the `Devices` that the template's clones replace, the `distro` and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag: `prepare` closes the fence, so that no execution starts, refuses with a `Mounted` the first mount from those devices, a distro partition included, or from an overlay, bind mounts included, in a bounded read of `/proc/self/mountinfo` of at most `MOUNT_TABLE_LIMIT` bytes, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as `nvx-time pre-capture` does, and returns an owned `CaptureBarrier`. `CaptureBarrier::capture` requests the capture with the platform tier's request, through `TimeAbi::capture`, and returns `Captured::Continued` once it opened the fence and restored the detectors' settings, or, in a clone, `Captured::Restored`, with the fence still closed, since nothing else may run before repair acknowledges the restore, and the detectors suppressed until the time daemon restores their settings: `Restored::into_parts` hands the `Restore` that repair takes next and the closed fence to the caller, which opens it once repair acknowledged the restore. `CaptureBarrier::cancel`, a dropped barrier, and a failure to read the VM generation ID open the fence and restore the detectors' settings; a capture request that fails restores the settings but leaves the fence closed, since it may fail in a clone whose restore it could not repair. Each failure is a `CaptureError`, with its `CaptureStage` and `CaptureCause` | +| `nvx::guest::clone` | x86_64: the preparation of a template's guest for a capture before its root's assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init or agent, whether or not it is PID 1. `CapturePreparation`, of the caller's `AdmissionFence` and the `Devices` that the template's clones replace, the `distro` and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag: `prepare` closes the fence, so that no execution starts, refuses with a `Mounted` the first mount from those devices, a distro partition included, or from an overlay, bind mounts included, in a bounded read of `/proc/self/mountinfo` of at most `MOUNT_TABLE_LIMIT` bytes, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as `nvx-time pre-capture` does, and returns an owned `CaptureBarrier`. `CaptureBarrier::capture` requests the capture with the platform tier's request, in the in-memory mode, through `TimeAbi::capture`, and returns `Captured::Continued` once it opened the fence and restored the detectors' settings, or, in a clone, `Captured::Restored`, with the fence still closed, since nothing else may run before repair acknowledges the restore, and the detectors suppressed until the time daemon restores their settings: `Restored::into_parts` hands the `Restore` that repair takes next, which holds the restore's `Seed`, and the closed fence to the caller, which keeps it closed through the repair and opens it once the clone may admit executions, such as once it is activated. `CaptureBarrier::cancel`, a dropped barrier, and a failure to read the VM generation ID open the fence and restore the detectors' settings; a capture request that fails restores the settings but leaves the fence closed, since it may fail in a clone whose restore it could not repair. Each failure is a `CaptureError`, with its `CaptureStage` and `CaptureCause`. `repair`, or a `Repair` that the caller may retry, repairs the clone of a `Restore` next: it checks that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online; with a memory target, it checks, before it adds any memory block, that the kernel's block size is 128 MiB and that the restore's ranges continue the captured RAM in logical order, with no gap, overlap, or range in the MMIO aperture, while the host bounds the target by the snapshot's capacity, then adds and onlines the blocks through sysfs, as `nvx-snapshot` does, and verifies them; it reseeds the kernel's random number generator with the `Seed`, having drawn no randomness before, and drops it; and it finishes the restore through `TimeAbi::restore_finish`, with the acknowledgement when the host gates its input until then, which hands the deferred time work to the time daemon that the caller supervises. It returns the `Repaired` clone, with its generation ID, its online CPUs and memory, and when it acknowledged the restore, and never touches the fence. It adds no memory block twice, and reseeds and finishes at most once, so a failure at or after the reseed is final. Each failure is a `RepairError`, with its `RepairStage` and `RepairCause` | | `nvx::guest::reseed` | `Seed`, `Seed::read`, and `reseed`: a restore's fresh entropy, which starts with its `GenerationId`, and the reseed of the kernel's random number generator with it, in the caller's process. A `Seed` keeps its bytes in one allocation, which no move copies, zeroes them with volatile writes when it is dropped, can't be cloned, and formats none of them. `nvx-reseed` runs them | | `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their roles' access modes and their expected sizes, and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already, once it has checked everything: the read-only layers of the `Image` that the caller names, which it never probes: an EROFS image, whose expected `nvx::uuid::Uuid` it checks in its superblock, or a GPT disk whose partition table it reads and checks with `nvx::guest::gpt::read_disk`, against the caller's `nvx::gpt::Expected` values, whose first partition it reads, up to the caller's bound, as opaque bytes that it never mounts, and whose later partitions, at least one, hold ext4 layers that it mounts `ro,nosuid,nodev` with `errors=remount-ro`, so that an error that the kernel finds in a layer fails the operation that found it, rather than panic the guest, as the layer's superblock may ask, from the block devices that the kernel created when it found the disk, once their superblocks show that no journal needs a recovery, which their read-only device can't take, partition 2's at the overlay's base and the last partition's at its top, under an overlay mounted `nosuid,nodev` with the one option set of the caller's `OverlayMode`, `metacopy=on,xino=on` or, for lowers that may carry untrusted overlay xattrs, `metacopy=off,redirect_dir=nofollow`, with no fallback, whose upper and work directories are on the `Upper`: an ext4 scratch mounted `nosuid,nodev`, whose superblock it checks for a journal that the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` at the kernel's default size. It refuses overlay options longer than `MOUNT_OPTIONS_LIMIT`, the most that a mount passes, which, below `/run/nvx`, fit up to 195 layers of a GPT disk, and an overlay that the kernel's mount table shows read-only or without the mode's options in effect, since the kernel mounts one so, rather than fail, when its upper can't hold its work directory or its extended attributes. The `Root` reports the first partition's bytes, its `Lower` layers, its `UpperKind`, and its `OverlayMode`. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::mount_export`: the host export, the aggregate device of `protocol::export`, one per root, mounted `nosuid,nodev` with the access of its `ExportMode` at `hostfs/root` in the runtime directory, out of the root, below a directory that only root may enter, whose root it returns as the `SourceRoot` of the caller's mappings. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, the export, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then the mounts of an aggregate and of the export, the later first, the overlay, the layers from the top down, and the upper, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off | | `nvx::guest::gpt` | `read_disk`: the partition table of a GPT disk, read and checked with `nvx::gpt`'s `Disk::read` through the disk's block device, which the caller opened, at the logical block size and size that `BLKSSZGET` and `BLKGETSIZE64` report, with the caller's `Expected` values, such as the disk GUID that a boot parameter names. It only reads the device, at explicit offsets, so it never moves the file's position. It neither compares the partitions that the kernel created with the table nor asks the kernel to reread the table. Each failure is a typed `DeviceError` | @@ -1132,7 +1132,9 @@ and `nvx-reseed` in its own process, through the `nvx::guest::time` and `nvx::guest::reseed` calls, on which both programs run, and a template's guest prepares for a capture before its root's assembly through `nvx::guest::clone`, which runs the steps of `nvx-time pre-capture` behind the -caller's fence of workload admission. +caller's fence of workload admission, and through which each clone repairs its +restore: it activates the restore's memory target, reseeds the kernel's random +number generator, and finishes the restore, as `nvx-snapshot` does. Nor is the `gpt` library a program: it reads the partition table of a GPT disk's block device. diff --git a/nvx-rs/src/guest/clone.rs b/nvx-rs/src/guest/clone.rs index 22d1580d3..bc78dbce0 100644 --- a/nvx-rs/src/guest/clone.rs +++ b/nvx-rs/src/guest/clone.rs @@ -1,5 +1,6 @@ //! Templates and clones in the guest: the capture of a template's guest before its root's -//! assembly, for a guest init or agent, whether or not it is PID 1. +//! assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init +//! or agent, whether or not it is PID 1. //! //! A platform template is a VM whose snapshot holds no tenant's root, so that every clone that a //! restore of the snapshot starts can attach its own image, upper, and export. Its guest captures @@ -21,18 +22,20 @@ //! //! It returns an owned [`CaptureBarrier`], which holds the fence closed and the detectors //! suppressed. [`CaptureBarrier::capture`] reads the VM generation ID, and requests the capture -//! through [`TimeAbi::capture`] with the platform tier's request, which keeps no scratch. It +//! through [`TimeAbi::capture`] with the platform tier's request, which keeps no scratch, in the +//! in-memory mode, so that no file holds the fresh entropy that a restore hands each clone. It //! returns how the capture ended: //! //! - [`Captured::Continued`]: the VM continued without a restore, and the barrier opened the //! fence, then restored the detectors' settings. //! - [`Captured::Restored`]: a restore of the snapshot started this VM, a clone whose wall clock -//! the time ABI repaired. Nothing else may run before repair acknowledges the restore, so the -//! fence stays closed: [`Restored::into_parts`] hands it to the caller, which opens it once -//! repair acknowledged the restore, or later. The detectors stay suppressed until the -//! restore's finish, [`TimeAbi::restore_finish`], hands their release to the time daemon. The -//! caller repairs the clone next, from the [`Restored`] restore. OpenVMM ends a platform -//! template's VM once its capture commits, so only clones return with this outcome. +//! the time ABI repaired, and whose [`Restore`] holds that entropy as its seed. Nothing else may +//! run before the clone's repair acknowledges the restore, so the fence stays closed: +//! [`Restored::into_parts`] hands it to the caller, which keeps it closed through the repair, +//! and opens it once the clone may admit executions, such as once it is activated. The +//! detectors stay suppressed until the repair finishes the restore, which hands their release +//! to the time daemon. OpenVMM ends a platform template's VM once its capture commits, so only +//! clones return with this outcome. //! //! [`CaptureBarrier::cancel`] opens the fence, then restores the detectors' settings, and so does //! a barrier that is dropped, which ignores a failure. A capture that fails restores the @@ -41,13 +44,53 @@ //! fails releases what it took. A failure is a [`CaptureError`], which names the //! [`CaptureStage`] that failed and its [`CaptureCause`]. //! +//! # Repair +//! +//! A clone repairs its restore next, before anything else runs, with a [`Repair`] of the +//! [`Restore`] that [`Restored::into_parts`] hands over, or with [`repair`](fn@repair), which +//! runs one once. [`Repair::run`], in order: +//! +//! 1. checks that the restore holds its seed, and that its capture did not finish it; +//! 2. reads the online CPUs, which must be a prefix of the CPUs, and refuses a processor target +//! that they don't meet already: the repair brings no CPU online; +//! 3. with a memory target, checks it before it adds any memory block: the kernel's memory block +//! size must be 128 MiB, and the restore's ranges, aligned to it, must continue the RAM that +//! the clone captured, in logical order: the first starts where the blocks that the kernel +//! had before the activation end, and low RAM, which ends at 3 GiB, continues as high RAM at +//! 4 GiB, so that there is at most one range of each, and none enters the MMIO aperture +//! between them. The host bounds the target by the snapshot's RAM capacity, of which the +//! guest has no record: OpenVMM's restore refuses a target above it, in +//! `validate_restore_memory_target`, as the host's plan of the clone does before its launch. +//! The repair then adds each block of the ranges that the kernel lacks, through its memory +//! probe interface, onlines each that is not online, and verifies that it is, as +//! `nvx-snapshot` does; +//! 4. reseeds the kernel's random number generator with the seed, through +//! [`reseed::reseed`](crate::guest::reseed::reseed), then drops the seed, which zeroes its +//! bytes. Nothing before draws randomness, from `getrandom` or from `/dev/urandom`; +//! 5. finishes the restore through [`TimeAbi::restore_finish`], with the acknowledgement when +//! the host gates its input until then, which hands the deferred time work to the time +//! daemon: the restore checks, the stall detectors' grace period and release, and the +//! wall-clock discipline's fast cadence. The caller starts the daemon with +//! [`TimeAbi::start_daemon`] at boot, before the capture, and supervises it. In enforcing +//! mode, a finish without a running daemon stops the VM, and in report-only mode, it runs the +//! work itself before it returns. The work may end after the clone is ready, and its failures +//! are fatal repair failures: the daemon reports them as violations of the time ABI, which +//! stop the VM in enforcing mode. +//! +//! It returns the [`Repaired`] clone: its new VM generation ID, its number of online CPUs, the +//! size of its online memory blocks, and when it acknowledged the restore, which an activation +//! checks against the resources that it expects. The repair records how far it got, so that it +//! never adds a memory block twice, and runs the reseed and the finish at most once, whoever +//! retries it: a failure at or after the reseed is final. It never touches the fence. A failure is +//! a [`RepairError`], which names the [`RepairStage`] that failed and its [`RepairCause`]. +//! +//! # Example +//! //! ```no_run //! use std::io; //! -//! use nvx::guest::clone::{AdmissionFence, CapturePreparation, Captured, Devices}; -//! use nvx::guest::reseed::{self, Seed}; +//! use nvx::guest::clone::{self, AdmissionFence, CapturePreparation, Captured, Devices}; //! use nvx::guest::root::Device; -//! use nvx::guest::time::{RestoreFinish, TimeAbi}; //! use nvx::protocol::root::{DISTRO_MMIO_BASE, SCRATCH_MMIO_BASE}; //! //! /// The agent's admission of executions, which it checks before it starts one. @@ -71,6 +114,7 @@ //! } //! //! # fn main() -> Result<(), Box> { +//! // At boot, the agent started the time daemon with `TimeAbi::start_daemon`, and supervises it. //! let mut admission = Admission { open: true }; //! let devices = Devices { //! distro: Device { mmio: DISTRO_MMIO_BASE, sectors: None }, @@ -81,15 +125,12 @@ //! match barrier.capture()? { //! Captured::Continued => println!("the VM continued without a restore"), //! Captured::Restored(restored) => { -//! // A clone continues here, with its wall clock repaired. It finishes the restore, and -//! // acknowledges it if the restore needs that, before its fence opens. -//! reseed::reseed(&Seed::read(restored.entropy_path())?)?; +//! // A clone continues here, with its wall clock repaired. Its repair activates its +//! // memory, reseeds it, and acknowledges the restore, while its fence stays closed. //! let (restore, fence) = restored.into_parts(); -//! let mut finish = RestoreFinish::new(); -//! if restore.acknowledgement_required() { -//! finish = finish.acknowledge(); -//! } -//! TimeAbi::new().restore_finish(finish)?; +//! let repaired = clone::repair(restore)?; +//! println!("{} CPUs, {} bytes of memory", repaired.cpus(), repaired.memory()); +//! // The clone opens the fence once it may admit executions, such as once activated. //! fence.open(); //! } //! } @@ -97,14 +138,21 @@ //! # } //! ``` //! -//! The operations need the guest's root user and the privileges of the time ABI's calls. They -//! read the mount table of the caller's mount namespace, which must be the one where the guest -//! mounts its devices, such as its initial namespace. They mount nothing, start no process or -//! thread, and collect no child's status. Apart from a violation of the time ABI in enforcing -//! mode, which stops the VM as [`TimeAbi`] tells, they never power the VM off, so a caller that is -//! not PID 1 keeps its own reaping, transport, and shutdown. +//! # Privileges and effects +//! +//! The operations need the guest's root user and the privileges of the time ABI's calls, and the +//! reseed needs `CAP_SYS_ADMIN`. The capture reads the mount table of the caller's mount +//! namespace, which must be the one where the guest mounts its devices, such as its initial +//! namespace, and the repair reads the kernel's CPU and memory interfaces in sysfs, and writes +//! its memory interface. They mount nothing and start no thread, and they start no process and +//! collect no child's status, except a report-only finish without a running time daemon, which +//! runs the deferred time work itself, in a child that it waits for, as +//! [`TimeAbi::restore_finish`] tells. Apart from a violation of the time ABI in enforcing mode, +//! which stops the VM as [`TimeAbi`] tells, they never power the VM off, so a caller that is not +//! PID 1 keeps its own reaping, transport, and shutdown. mod mounts; +mod repair; #[cfg(test)] mod tests; @@ -123,6 +171,9 @@ use super::time::{ }; pub use self::mounts::{Mounted, Origin}; +pub use self::repair::{ + RangeFault, Repair, RepairCause, RepairError, RepairStage, Repaired, repair, +}; /// The most bytes of the mount table that a [`CapturePreparation`] reads: 1 MiB, the lines of /// thousands of mounts. It refuses a longer table, rather than check a part of it. @@ -133,8 +184,6 @@ const MOUNT_TABLE: &str = "self/mountinfo"; /// The directory of the block devices' sysfs entries, below sysfs, which names each by its /// device number, `MAJ:MIN`. const BLOCK_DEVICES: &str = "dev/block"; -/// The file to which a restore writes its fresh entropy, in the time ABI's runtime directory. -const ENTROPY_PATH: &str = "/run/nvx/restore-entropy"; /// The caller's fence of workload admission, which its agent checks before it starts each /// execution: while the fence is closed, no execution starts. @@ -304,8 +353,8 @@ pub struct CaptureBarrier { impl CaptureBarrier { /// Requests the capture: reads the VM generation ID, then requests a capture through - /// [`TimeAbi::capture`] with the platform tier's request, which keeps no scratch, and returns - /// how it ended. + /// [`TimeAbi::capture`] with the platform tier's request, which keeps no scratch, in the + /// in-memory mode, and returns how it ended. /// /// A VM that continues without a restore gets [`Captured::Continued`], once the barrier /// opened the fence and restored the detectors' settings; a failure to restore them is a @@ -329,11 +378,9 @@ impl CaptureBarrier { return Err(self.fail(error)); } }; - let request = CaptureRequest::new( - ScratchPolicy::Fresh, - generation_id, - PathBuf::from(ENTROPY_PATH), - ); + // The restore holds its fresh entropy as its seed, which no file holds, until repair + // takes it. + let request = CaptureRequest::in_memory(ScratchPolicy::Fresh, generation_id); let outcome = match self.system.capture(request) { Ok(outcome) => outcome, Err(error) => { @@ -350,7 +397,7 @@ impl CaptureBarrier { } CaptureOutcome::Restored(restore) => { // The time daemon restores the detectors' settings once repair finishes the - // restore, and the caller opens the fence once repair acknowledged it. + // restore, and the caller keeps the fence closed through repair. self.suppressed = false; let fence = self .fence @@ -443,8 +490,9 @@ impl fmt::Debug for Captured { /// A clone's restore, which [`CaptureBarrier::capture`] returns for repair to take next, with /// the caller's fence, which stays closed until the caller opens it. /// -/// No execution may start before repair acknowledged the restore, so the caller opens the fence -/// only then, or later. Dropping it leaves the fence closed. +/// No execution may start before the clone's repair acknowledged the restore, and repair never +/// opens the fence: the caller opens it once the clone may admit executions, such as once it is +/// activated. Dropping it leaves the fence closed. pub struct Restored { /// The restore that the time ABI repaired, as its restore packet describes it. restore: Restore, @@ -459,14 +507,9 @@ impl Restored { &self.restore } - /// Returns the file to which the restore wrote its fresh entropy, 64 bytes that start with the - /// new VM generation ID, and that only root may read: `/run/nvx/restore-entropy`. The caller - /// reseeds from it, then removes it. - pub fn entropy_path(&self) -> &'static Path { - Path::new(ENTROPY_PATH) - } - - /// Returns the restore that the time ABI repaired, and the caller's fence, still closed. + /// Returns the restore that the time ABI repaired, which holds the restore's fresh entropy as + /// its seed, for [`repair`](fn@repair) or a [`Repair`] to take, and the caller's fence, still + /// closed. pub fn into_parts(self) -> (Restore, F) { (self.restore, self.fence) } diff --git a/nvx-rs/src/guest/clone/repair.rs b/nvx-rs/src/guest/clone/repair.rs new file mode 100644 index 000000000..20f1c16e2 --- /dev/null +++ b/nvx-rs/src/guest/clone/repair.rs @@ -0,0 +1,798 @@ +//! The repair of a clone that a restore of a template's snapshot started, in the caller's +//! process: [`Repair`], and [`repair`], which runs one once, as the [module](super) describes. + +#[cfg(test)] +mod tests; + +use std::fmt; +use std::fs::{self, OpenOptions}; +use std::io::{self, Write}; +use std::os::unix::ffi::OsStrExt; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::Instant; + +use super::at; +use crate::guest::GenerationId; +use crate::guest::parse::{parse_decimal, parse_hexadecimal}; +use crate::guest::reseed::{self, ReseedError, Seed}; +use crate::guest::root::Paths; +use crate::guest::sys; +use crate::guest::time::{MemoryRange, Restore, RestoreFinish, TimeAbi, TimeError}; + +/// The memory block size of the snapshot ABI, 128 MiB, to which a restore's memory ranges are +/// aligned. +const BLOCK_SIZE: u64 = 128 << 20; +/// The end of low RAM, where the MMIO aperture starts: 3 GiB. +const LOW_RAM_END: u64 = 3 << 30; +/// The start of high RAM, where the MMIO aperture ends: 4 GiB. +const HIGH_RAM_START: u64 = 4 << 30; +/// The directory of the kernel's memory blocks, below sysfs. +const MEMORY_BLOCKS: &str = "devices/system/memory"; +/// The memory block size, which the kernel writes in hexadecimal, in the directory of the +/// memory blocks. +const BLOCK_SIZE_FILE: &str = "block_size_bytes"; +/// The file to which a physical address is written to add the memory block that starts at it, +/// in the directory of the memory blocks. +const PROBE_FILE: &str = "probe"; +/// The name of a memory block's directory, which the block's number follows, in the directory +/// of the memory blocks. +const BLOCK_PREFIX: &str = "memory"; +/// The state of a memory block, such as `online` or `offline`, in the block's directory. +const STATE_FILE: &str = "state"; +/// The CPUs that are online, as a CPU list, below sysfs. +const ONLINE_CPUS: &str = "devices/system/cpu/online"; +/// The state of a memory block that is online. +const ONLINE: &[u8] = b"online"; + +/// Repairs the clone that `restore` started, once, as [`Repair::run`] does, and returns the +/// repaired clone. +/// +/// The call takes the restore, so it can't be retried: a caller that retries a failed repair +/// keeps a [`Repair`] instead. +pub fn repair(restore: Restore) -> Result { + Repair::new(restore).run() +} + +/// The repair of a clone that a restore started, of the [`Restore`] that the clone's capture +/// returned, as the [module](super) describes: its memory target, its reseed, and its finish, +/// with the acknowledgement that the host awaits. +/// +/// [`run`](Self::run) records how far the repair got, so that the repair never adds a memory +/// block twice, never reseeds twice, and never finishes the restore twice, whoever retries it: +/// +/// - Until the memory target is active, a failed attempt may be retried, and the next one +/// repeats the checks and adds or onlines only the memory blocks that are not online yet. The +/// first attempt derives the end of the RAM that the clone captured, before it adds any block, +/// and every later one keeps it. +/// - The reseed and the finish run at most once. A failure of either is final: every later call +/// returns its error again. +/// - Once the repair completed, every later call returns the same [`Repaired`]. +/// +/// The repair never touches the caller's fence of workload admission, which stays closed: the +/// caller opens it when the clone may admit executions, such as once it is activated. +pub struct Repair { + /// The restore that the clone's capture returned, without its seed. + restore: Restore, + /// The restore's fresh entropy, until the reseed drops it. + seed: Option, + /// How far the repair got. + progress: Progress, + /// The kernel and the time ABI. + system: Box, +} + +impl Repair { + /// Returns the repair of the clone that `restore` started, which takes the restore's seed: + /// the clone's capture requests the in-memory mode, + /// [`CaptureRequest::in_memory`](crate::guest::time::CaptureRequest::in_memory), so that the + /// restore holds it. + pub fn new(restore: Restore) -> Self { + let system = LiveSystem { + paths: Paths::default(), + abi: TimeAbi::new(), + }; + Self::with_system(restore, Box::new(system)) + } + + /// Returns the repair of the clone that `restore` started, on `system`. + fn with_system(mut restore: Restore, system: Box) -> Self { + let seed = restore.take_seed(); + Self { + restore, + seed, + progress: Progress::Memory { captured: None }, + system, + } + } + + /// Runs the repair from where it got, as the [module](super) describes, and returns the + /// repaired clone. + /// + /// A failure names its [`RepairStage`] and its [`RepairCause`]. One before the reseed may be + /// retried, and changes nothing that a retry undoes: a refused restore or memory target adds + /// no memory block, and a failed activation leaves the blocks that it added and onlined, + /// which a retry counts as done. The reseed and the finish run at most once, so a failure of + /// either, or of a finish that never returned, such as after a panic, is final, and every + /// later call returns that error again. A failed finish may have acknowledged the restore + /// already. Once the repair completed, every later call returns the same [`Repaired`], + /// without another step. + pub fn run(&mut self) -> Result { + loop { + match &self.progress { + Progress::Memory { .. } => { + let resources = self.activate()?; + self.progress = Progress::Reseed(resources); + } + Progress::Reseed(resources) => { + let resources = *resources; + if let Err(error) = self.reseed() { + self.progress = Progress::Failed(error.clone()); + return Err(error); + } + self.progress = Progress::Finish(resources); + } + Progress::Finish(resources) => return self.finish(*resources), + Progress::Repaired(repaired) => return Ok(*repaired), + Progress::Failed(error) => return Err(error.clone()), + } + } + } + + /// Checks the restore and the online CPUs, and activates the memory target: the steps before + /// the reseed, which a failure leaves to a retry. + fn activate(&mut self) -> Result { + if self.restore.finished() { + return Err(RepairError::new( + RepairStage::Restore, + RepairCause::Finished, + )); + } + if self.seed.is_none() { + return Err(RepairError::new(RepairStage::Restore, RepairCause::NoSeed)); + } + let cpus = self + .check_processors() + .map_err(|cause| RepairError::new(RepairStage::Processors, cause))?; + let memory = self + .activate_memory() + .map_err(|cause| RepairError::new(RepairStage::Memory, cause))?; + Ok(Resources { cpus, memory }) + } + + /// Returns the number of online CPUs, which must be a prefix of the CPUs, and must meet the + /// restore's processor target, if it has one, since the repair brings no CPU online. + fn check_processors(&mut self) -> Result { + let text = self.system.online_cpus().map_err(RepairCause::Io)?; + let online = online_prefix(&text).ok_or_else(|| RepairCause::OnlineCpus(lossy(&text)))?; + if let Some(target) = self.restore.processor_target() + && u32::from(target) != online + { + return Err(RepairCause::ProcessorTarget { target, online }); + } + Ok(online) + } + + /// Activates the restore's memory target, if it has one, and returns the memory size of the + /// online memory blocks. + fn activate_memory(&mut self) -> Result { + let ranges = !self.restore.memory_ranges().is_empty(); + if !self.restore.memory_target() && ranges { + return Err(RepairCause::UntargetedRanges); + } + let text = self.system.block_size().map_err(RepairCause::Io)?; + let block_size = parse_hexadecimal(trimmed(&text)) + .filter(|&size| size != 0 && (!self.restore.memory_target() || size == BLOCK_SIZE)) + .ok_or_else(|| RepairCause::BlockSize(lossy(&text)))?; + if ranges { + self.add_ranges()?; + } + self.online_memory(block_size) + } + + /// Checks the restore's memory ranges against the RAM that the clone captured, before it adds + /// any memory block, then adds each block of the ranges that the kernel lacks, onlines each + /// that is not online, and verifies that it is. + fn add_ranges(&mut self) -> Result<(), RepairCause> { + let captured = match self.progress { + Progress::Memory { + captured: Some(captured), + } => captured, + _ => { + let blocks = self.system.memory_blocks().map_err(RepairCause::Io)?; + let captured = captured_end(&blocks)?; + self.progress = Progress::Memory { + captured: Some(captured), + }; + captured + } + }; + check_ranges(self.restore.memory_ranges(), captured)?; + for range in self.restore.memory_ranges() { + let mut address = range.start(); + while address < range.start() + range.length() { + activate_block(self.system.as_mut(), address)?; + address += BLOCK_SIZE; + } + } + Ok(()) + } + + /// Returns the memory size of the online memory blocks, of `block_size` bytes each. + fn online_memory(&mut self, block_size: u64) -> Result { + let mut online = 0_u64; + for block in self.system.memory_blocks().map_err(RepairCause::Io)? { + // A block that the kernel removed since it listed it is not online. + if let Some(state) = self.system.block_state(block).map_err(RepairCause::Io)? + && trimmed(&state) == ONLINE + { + online += 1; + } + } + Ok(online.saturating_mul(block_size)) + } + + /// Reseeds the kernel's random number generator with the restore's seed, then drops the + /// seed, which zeroes its bytes, whatever the reseed's outcome. + fn reseed(&mut self) -> Result<(), RepairError> { + let fail = |cause| RepairError::new(RepairStage::Reseed, cause); + // A reseed that never returned, such as after a panic, dropped the seed already. + let seed = self.seed.take().ok_or_else(|| fail(RepairCause::NoSeed))?; + let result = self.system.reseed(&seed); + drop(seed); + result.map_err(|error| fail(RepairCause::Reseed(error))) + } + + /// Finishes the restore, with the acknowledgement if the host gates its input until then, + /// once, and returns the repaired clone, which has `resources`. + fn finish(&mut self, resources: Resources) -> Result { + let acknowledge = self.restore.acknowledgement_required(); + let mut finish = RestoreFinish::new(); + if acknowledge { + finish = finish.acknowledge(); + } + // The finish runs at most once, even when it never returns, such as after a panic. + self.progress = Progress::Failed(RepairError::new( + RepairStage::Finish, + RepairCause::Incomplete, + )); + if let Err(error) = self.system.restore_finish(finish) { + let error = RepairError::new(RepairStage::Finish, RepairCause::Time(error)); + self.progress = Progress::Failed(error.clone()); + return Err(error); + } + let repaired = Repaired { + generation_id: self.restore.generation_id(), + cpus: resources.cpus, + memory: resources.memory, + acknowledged_at: acknowledge.then(Instant::now), + }; + self.progress = Progress::Repaired(repaired); + Ok(repaired) + } +} + +impl fmt::Debug for Repair { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("Repair") + .field("restore", &self.restore) + .field("progress", &self.progress) + .finish_non_exhaustive() + } +} + +/// How far a repair got, from which its next attempt resumes. +#[derive(Debug)] +enum Progress { + /// The memory target is not active yet, which the next attempt activates. + Memory { + /// The logical end of the RAM that the clone captured, which the first attempt that + /// checks the memory ranges derives, before it adds any memory block. + captured: Option, + }, + /// The memory target is active, with these resources: the reseed is next. + Reseed(Resources), + /// The kernel's random number generator is reseeded: the finish is next. + Finish(Resources), + /// The repair completed. + Repaired(Repaired), + /// The repair failed at or after its reseed, which never runs again, nor does the finish. + Failed(RepairError), +} + +/// The resources that a clone has once its memory target is active. +#[derive(Clone, Copy, Debug)] +struct Resources { + /// The number of online CPUs. + cpus: u32, + /// The memory size of the online memory blocks, in bytes. + memory: u64, +} + +/// A clone that its [`Repair`] repaired: its new VM generation ID, the resources that it has, +/// which an activation checks against the resources that it expects, and when its repair +/// acknowledged its restore. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Repaired { + /// The new VM generation ID. + generation_id: GenerationId, + /// The number of online CPUs. + cpus: u32, + /// The memory size of the online memory blocks, in bytes. + memory: u64, + /// When the repair acknowledged the restore, if the restore asked for it. + acknowledged_at: Option, +} + +impl Repaired { + /// Returns the clone's new VM generation ID, with which the restore's seed started. + pub fn generation_id(&self) -> GenerationId { + self.generation_id + } + + /// Returns the number of online CPUs, a prefix of the CPUs. + pub fn cpus(&self) -> u32 { + self.cpus + } + + /// Returns the memory size, in bytes: the size of the memory blocks that are online, which + /// is the restore's memory target when the restore has one. + pub fn memory(&self) -> u64 { + self.memory + } + + /// Returns when the repair acknowledged the restore, once its finish returned, or `None` if + /// the host did not gate its input until the acknowledgement. + pub fn acknowledged_at(&self) -> Option { + self.acknowledged_at + } +} + +/// A stage of a clone's repair, which a [`RepairError`] names. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum RepairStage { + /// Checking that the restore holds its seed, and that its capture did not finish it. + Restore, + /// Checking the online CPUs against the restore's processor target. + Processors, + /// Checking the restore's memory target, adding and onlining its memory blocks, and + /// verifying them. + Memory, + /// Reseeding the kernel's random number generator with the restore's seed. + Reseed, + /// Finishing the restore, with its acknowledgement, which hands the deferred time work to + /// the time daemon. + Finish, +} + +impl fmt::Display for RepairStage { + /// Writes what the stage does, such as `activate the memory target`. + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::Restore => "check the restore", + Self::Processors => "check the online CPUs", + Self::Memory => "activate the memory target", + Self::Reseed => "reseed the kernel's random number generator", + Self::Finish => "finish the restore", + }) + } +} + +/// What is wrong with a memory range of a restore, which a [`RepairCause::Range`] names. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum RangeFault { + /// The range is empty, ends past the address space, or is not aligned to 128 MiB memory + /// blocks. + Misaligned, + /// The range enters the MMIO aperture, between the end of low RAM at 3 GiB and the start of + /// high RAM at 4 GiB. + Aperture, + /// The range is a second range of low RAM or of high RAM, or one of low RAM after one of + /// high RAM. + Extra, + /// The range starts past the end of the RAM before it, in logical order, which leaves a gap. + Gap, + /// The range starts before the end of the RAM before it, in logical order, which it + /// overlaps. + Overlap, +} + +impl fmt::Display for RangeFault { + /// Writes what is wrong with the range, such as `overlaps the RAM before it`. + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::Misaligned => "is not a nonempty range of whole 128 MiB memory blocks", + Self::Aperture => "enters the MMIO aperture between 3 GiB and 4 GiB", + Self::Extra => "is another range of low or high RAM", + Self::Gap => "leaves a gap after the RAM before it", + Self::Overlap => "overlaps the RAM before it", + }) + } +} + +/// Why a stage of a clone's repair failed, which a [`RepairError`] holds. +#[derive(Debug)] +#[non_exhaustive] +pub enum RepairCause { + /// A read or write of sysfs failed with this error, which names the file. + Io(io::Error), + /// The restore holds no seed: its capture requested the file mode, or the seed was taken. + NoSeed, + /// The restore's capture finished it, so that the host may admit input before its repair. + Finished, + /// The online CPUs, as the kernel lists them, are not a prefix of the CPUs, such as `0-3`. + OnlineCpus(String), + /// The restore asks for another number of online CPUs, which needs CPUs brought online, + /// which the repair doesn't do. + ProcessorTarget { + /// The number of online CPUs that the restore asks for. + target: u8, + /// The number of online CPUs. + online: u32, + }, + /// The kernel's memory block size, which it writes in hexadecimal, is not 128 MiB, as the + /// restore's memory target needs, or is malformed. + BlockSize(String), + /// The memory blocks that the kernel had added before the activation are not the first ones + /// of RAM in logical order, which a captured RAM holds. + CapturedRam, + /// The restore lists memory ranges without a memory target. + UntargetedRanges, + /// A memory range of the restore does not continue the RAM that the clone captured. + Range { + /// The range's index in the restore's list. + index: usize, + /// What is wrong with it. + fault: RangeFault, + }, + /// The kernel lacks this memory block after its addition. + Missing(u64), + /// A memory block is not online after its onlining. + Offline { + /// The block's number. + block: u64, + /// The block's state, as the kernel writes it. + state: String, + }, + /// The kernel refused the reseed. + Reseed(ReseedError), + /// The time ABI's finish failed: a violation that a report-only call reports, or a failure of + /// its ports or its time state. + Time(TimeError), + /// An earlier attempt at the stage, which runs at most once, never returned, such as after a + /// panic. + Incomplete, +} + +impl fmt::Display for RepairCause { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Io(error) => formatter.write_str(&sys::describe(error)), + Self::NoSeed => formatter.write_str("the restore holds no seed"), + Self::Finished => formatter.write_str("the restore's capture finished it"), + Self::OnlineCpus(cpus) => { + write!( + formatter, + "the online CPUs {cpus} are not a prefix of the CPUs" + ) + } + Self::ProcessorTarget { target, online } => write!( + formatter, + "the restore asks for {target} online CPUs, and {online} are online" + ), + Self::BlockSize(size) => write!( + formatter, + "the memory block size is {size} in hexadecimal, not 128 MiB" + ), + Self::CapturedRam => { + formatter.write_str("the memory blocks before the activation are not a prefix") + } + Self::UntargetedRanges => { + formatter.write_str("the restore lists memory ranges without a memory target") + } + Self::Range { index, fault } => write!(formatter, "memory range {index} {fault}"), + Self::Missing(block) => write!(formatter, "memory block {block} was not added"), + Self::Offline { block, state } => { + write!(formatter, "memory block {block} is {state}, not online") + } + Self::Reseed(error) => error.fmt(formatter), + Self::Time(error) => error.fmt(formatter), + Self::Incomplete => formatter.write_str("an earlier attempt never returned"), + } + } +} + +/// Why a clone's repair failed: the stage that failed, and its cause. +/// +/// Its `Display` form names both, such as `failed to activate the memory target: memory range 0 +/// overlaps the RAM before it`. A clone keeps one error of a final failure, which every later +/// [`Repair::run`] returns again. +#[derive(Clone, Debug)] +pub struct RepairError { + /// The stage that failed. + stage: RepairStage, + /// Why it failed. + cause: Arc, +} + +impl RepairError { + /// Returns the failure of `stage`, because of `cause`. + fn new(stage: RepairStage, cause: RepairCause) -> Self { + Self { + stage, + cause: Arc::new(cause), + } + } + + /// Returns the stage that failed. + pub fn stage(&self) -> RepairStage { + self.stage + } + + /// Returns why it failed. + pub fn cause(&self) -> &RepairCause { + &self.cause + } +} + +impl fmt::Display for RepairError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "failed to {}: {}", self.stage, self.cause) + } +} + +impl std::error::Error for RepairError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self.cause.as_ref() { + RepairCause::Io(error) => Some(error), + RepairCause::Reseed(error) => Some(error), + RepairCause::Time(error) => Some(error), + _ => None, + } + } +} + +/// Returns the number of online CPUs that the CPU list `text`, as the kernel writes it, names, +/// if they are a prefix of the CPUs, `0` or `0-N`, or `None`. +fn online_prefix(text: &[u8]) -> Option { + let last = match trimmed(text) { + b"0" => 0, + range => parse_decimal(range.strip_prefix(b"0-")?)?, + }; + u32::try_from(last).ok()?.checked_add(1) +} + +/// Returns the logical end of the RAM that the memory blocks `blocks` hold, in ascending order, +/// which must be the first blocks of RAM in logical order: those of low RAM from address 0, then +/// those of high RAM from 4 GiB. +fn captured_end(blocks: &[u64]) -> Result { + for (logical, &block) in (0_u64..).zip(blocks) { + if block != physical_block(logical) { + return Err(RepairCause::CapturedRam); + } + } + Ok((blocks.len() as u64).saturating_mul(BLOCK_SIZE)) +} + +/// Returns the number of the memory block that holds the RAM's `logical`th block, in logical +/// order, which skips the MMIO aperture. +fn physical_block(logical: u64) -> u64 { + /// The blocks of low RAM. + const LOW_BLOCKS: u64 = LOW_RAM_END / BLOCK_SIZE; + /// The blocks of the MMIO aperture. + const APERTURE_BLOCKS: u64 = (HIGH_RAM_START - LOW_RAM_END) / BLOCK_SIZE; + if logical < LOW_BLOCKS { + logical + } else { + logical + APERTURE_BLOCKS + } +} + +/// Checks that `ranges` are the ranges that continue the RAM that the clone captured, whose +/// logical end is `captured`, as a restore's expansion of the RAM selects them: aligned to +/// memory blocks, in ascending order, with no gap or overlap in logical order, and at most one of +/// low RAM, which ends at 3 GiB, and one of high RAM, which starts at 4 GiB, so that none enters +/// the MMIO aperture between them. +fn check_ranges(ranges: &[MemoryRange], captured: u64) -> Result<(), RepairCause> { + let mut next = captured; + let (mut low, mut high) = (false, false); + for (index, range) in ranges.iter().enumerate() { + let fault = |fault| RepairCause::Range { index, fault }; + let (start, length) = (range.start(), range.length()); + let end = start + .checked_add(length) + .ok_or_else(|| fault(RangeFault::Misaligned))?; + if length == 0 || !start.is_multiple_of(BLOCK_SIZE) || !length.is_multiple_of(BLOCK_SIZE) { + return Err(fault(RangeFault::Misaligned)); + } + let logical = if end <= LOW_RAM_END { + if low || high { + return Err(fault(RangeFault::Extra)); + } + low = true; + start + } else if start >= HIGH_RAM_START { + if high { + return Err(fault(RangeFault::Extra)); + } + high = true; + start - (HIGH_RAM_START - LOW_RAM_END) + } else { + return Err(fault(RangeFault::Aperture)); + }; + if logical < next { + return Err(fault(RangeFault::Overlap)); + } + if logical > next { + return Err(fault(RangeFault::Gap)); + } + next = logical + length; + } + Ok(()) +} + +/// Adds the memory block that starts at `address` through `system`, unless the kernel has it +/// already, onlines it, unless it is online, and verifies that it is. +fn activate_block(system: &mut dyn RepairSystem, address: u64) -> Result<(), RepairCause> { + let block = address / BLOCK_SIZE; + let state = match system.block_state(block).map_err(RepairCause::Io)? { + Some(state) => state, + None => { + system.add_block(address).map_err(RepairCause::Io)?; + system + .block_state(block) + .map_err(RepairCause::Io)? + .ok_or(RepairCause::Missing(block))? + } + }; + if trimmed(&state) == ONLINE { + return Ok(()); + } + system.online_block(block).map_err(RepairCause::Io)?; + let state = system + .block_state(block) + .map_err(RepairCause::Io)? + .ok_or(RepairCause::Missing(block))?; + if trimmed(&state) == ONLINE { + Ok(()) + } else { + Err(RepairCause::Offline { + block, + state: lossy(&state), + }) + } +} + +/// Returns `text`, a value that the kernel writes in sysfs, without the newline that ends it. +fn trimmed(text: &[u8]) -> &[u8] { + text.strip_suffix(b"\n").unwrap_or(text) +} + +/// Returns `text`, a value that the kernel writes in sysfs, without its newline, as a string for +/// a diagnostic. +fn lossy(text: &[u8]) -> String { + String::from_utf8_lossy(trimmed(text)).into_owned() +} + +/// What a repair reads and writes of the kernel, and asks of the time ABI, which the tests' fake +/// replaces. +trait RepairSystem: Send { + /// Reads the CPUs that are online, as the kernel lists them. + fn online_cpus(&mut self) -> io::Result>; + + /// Reads the memory block size, which the kernel writes in hexadecimal. + fn block_size(&mut self) -> io::Result>; + + /// Returns the numbers of the memory blocks that the kernel added, in ascending order. + fn memory_blocks(&mut self) -> io::Result>; + + /// Reads the state of the memory block `block`, or returns `None` if the kernel lacks it. + fn block_state(&mut self, block: u64) -> io::Result>>; + + /// Adds the memory block that starts at the physical address `address`. + fn add_block(&mut self, address: u64) -> io::Result<()>; + + /// Onlines the memory block `block`. + fn online_block(&mut self, block: u64) -> io::Result<()>; + + /// Reseeds the kernel's random number generator with `seed`, as [`reseed::reseed`] does. + fn reseed(&mut self, seed: &Seed) -> Result<(), ReseedError>; + + /// Finishes the restore, as [`TimeAbi::restore_finish`] does. + fn restore_finish(&mut self, finish: RestoreFinish) -> Result<(), TimeError>; +} + +/// The guest's kernel, at the paths of its interfaces, and its time ABI. +struct LiveSystem { + /// The directories of the kernel's interfaces. + paths: Paths, + /// The time ABI, in the mode that the kernel command line selects. + abi: TimeAbi, +} + +impl LiveSystem { + /// Returns the path of `name` in the directory of the memory blocks. + fn memory(&self, name: &str) -> PathBuf { + self.paths.sys.join(MEMORY_BLOCKS).join(name) + } + + /// Returns the path of the state of the memory block `block`. + fn state(&self, block: u64) -> PathBuf { + self.memory(&format!("{BLOCK_PREFIX}{block}")) + .join(STATE_FILE) + } +} + +impl RepairSystem for LiveSystem { + fn online_cpus(&mut self) -> io::Result> { + read(&self.paths.sys.join(ONLINE_CPUS)) + } + + fn block_size(&mut self) -> io::Result> { + read(&self.memory(BLOCK_SIZE_FILE)) + } + + fn memory_blocks(&mut self) -> io::Result> { + let directory = self.paths.sys.join(MEMORY_BLOCKS); + let mut blocks = Vec::new(); + for entry in fs::read_dir(&directory).map_err(|error| at(&directory, &error))? { + let entry = entry.map_err(|error| at(&directory, &error))?; + if let Some(block) = entry + .file_name() + .as_bytes() + .strip_prefix(BLOCK_PREFIX.as_bytes()) + .and_then(parse_decimal) + { + blocks.push(block); + } + } + blocks.sort_unstable(); + Ok(blocks) + } + + fn block_state(&mut self, block: u64) -> io::Result>> { + let path = self.state(block); + match fs::read(&path) { + Ok(state) => Ok(Some(state)), + Err(error) if error.kind() == io::ErrorKind::NotFound => Ok(None), + Err(error) => Err(at(&path, &error)), + } + } + + fn add_block(&mut self, address: u64) -> io::Result<()> { + write( + &self.memory(PROBE_FILE), + format!("{address:#x}\n").as_bytes(), + ) + } + + fn online_block(&mut self, block: u64) -> io::Result<()> { + write(&self.state(block), b"online\n") + } + + fn reseed(&mut self, seed: &Seed) -> Result<(), ReseedError> { + reseed::reseed(seed) + } + + fn restore_finish(&mut self, finish: RestoreFinish) -> Result<(), TimeError> { + self.abi.restore_finish(finish) + } +} + +/// Reads the file at `path`, with errors that name it. +fn read(path: &Path) -> io::Result> { + fs::read(path).map_err(|error| at(path, &error)) +} + +/// Writes `bytes` to the file at `path`, which must exist, such as a sysfs attribute, with errors +/// that name it. +fn write(path: &Path, bytes: &[u8]) -> io::Result<()> { + OpenOptions::new() + .write(true) + .open(path) + .and_then(|mut file| file.write_all(bytes)) + .map_err(|error| at(path, &error)) +} diff --git a/nvx-rs/src/guest/clone/repair/tests.rs b/nvx-rs/src/guest/clone/repair/tests.rs new file mode 100644 index 000000000..ab0743d84 --- /dev/null +++ b/nvx-rs/src/guest/clone/repair/tests.rs @@ -0,0 +1,800 @@ +//! Tests of a clone's repair, with a fake system whose CPUs, memory blocks, reseed, and time ABI +//! each test chooses, and which records its calls in one log, and of the live system's reads and +//! writes of sysfs, in a temporary directory. + +use std::collections::BTreeMap; +use std::fs; +use std::path::Path; +use std::sync::{Arc, Mutex, PoisonError}; + +use super::*; +use crate::guest::reseed::tests::take_dropped; +use crate::guest::sys::Errno; + +/// The new VM generation ID of the clone. +const RESTORED: GenerationId = GenerationId::from_bytes([8; GenerationId::LENGTH]); +/// One mebibyte. +const MIB: u64 = 1 << 20; +/// One gibibyte. +const GIB: u64 = 1 << 30; +/// The state of a memory block that is online, as the kernel writes it. +const ONLINE_STATE: &[u8] = b"online\n"; +/// The state of a memory block that is offline, as the kernel writes it. +const OFFLINE_STATE: &[u8] = b"offline\n"; + +/// A call of a test's fake system. +#[derive(Clone, Debug, PartialEq, Eq)] +enum Call { + /// The online CPUs were read. + OnlineCpus, + /// The memory block size was read. + BlockSize, + /// The memory blocks were listed. + MemoryBlocks, + /// The memory block at this physical address was added. + Add(u64), + /// The memory block of this number was onlined. + Online(u64), + /// The kernel was reseeded with the seed of this generation ID. + Reseed(GenerationId), + /// The bytes of the seeds that the thread dropped since the repair started, as their drops + /// left them, which the finish records before it runs. + Dropped(Vec<[u8; Seed::LENGTH]>), + /// The restore was finished. + Finish(RestoreFinish), +} + +/// The log of a test's calls, which the test shares with its fake system. +#[derive(Clone, Default)] +struct Calls(Arc>>); + +impl Calls { + /// Records `call`. + fn push(&self, call: Call) { + self.0 + .lock() + .unwrap_or_else(PoisonError::into_inner) + .push(call); + } + + /// Returns the calls recorded since the last take, in order. + fn take(&self) -> Vec { + std::mem::take(&mut *self.0.lock().unwrap_or_else(PoisonError::into_inner)) + } +} + +/// A system that records its calls, except its reads of the blocks' states, with CPUs, memory +/// blocks, a reseed, and a time ABI that a test chooses: by default, two online CPUs, 128 MiB +/// memory blocks, the four online blocks of a clone that captured 512 MiB of RAM, and a reseed +/// and a finish that succeed. It opens no file and draws no randomness: it keeps its blocks in +/// ordered maps, which draw no random keys, as hashed maps would. +struct FakeSystem { + /// The log. + calls: Calls, + /// The online CPUs, as the kernel lists them. + cpus: &'static [u8], + /// The memory block size, as the kernel writes it. + block_size: &'static [u8], + /// The state of each memory block that the kernel has, by the block's number. + blocks: BTreeMap, + /// The error numbers with which the next addition of the block at each physical address + /// fails, once. + add_failures: BTreeMap, + /// The error numbers with which the next onlining of each block fails, once. + online_failures: BTreeMap, + /// Whether an added block comes online by itself, as the kernel onlines added memory when its + /// policy says so. + auto_online: bool, + /// A block that the kernel leaves offline when it is onlined. + stuck: Option, + /// The error number with which the reseed fails, if any. + reseed_failure: Option, + /// Whether the finish fails. + finish_fails: bool, + /// Whether the finish panics, as one that never returns would leave the repair. + finish_panics: bool, +} + +impl FakeSystem { + /// Returns the default system, which records its calls in `calls`. + fn new(calls: &Calls) -> Self { + Self { + calls: calls.clone(), + cpus: b"0-1\n", + block_size: b"8000000\n", + blocks: (0..4).map(|block| (block, ONLINE_STATE)).collect(), + add_failures: BTreeMap::new(), + online_failures: BTreeMap::new(), + auto_online: false, + stuck: None, + reseed_failure: None, + finish_fails: false, + finish_panics: false, + } + } + + /// Returns the system whose kernel has the first `count` memory blocks of RAM, in logical + /// order, online instead. + fn capturing(mut self, count: u64) -> Self { + self.blocks = (0..count) + .map(|logical| (physical_block(logical), ONLINE_STATE)) + .collect(); + self + } +} + +impl RepairSystem for FakeSystem { + fn online_cpus(&mut self) -> io::Result> { + self.calls.push(Call::OnlineCpus); + Ok(self.cpus.to_vec()) + } + + fn block_size(&mut self) -> io::Result> { + self.calls.push(Call::BlockSize); + Ok(self.block_size.to_vec()) + } + + fn memory_blocks(&mut self) -> io::Result> { + self.calls.push(Call::MemoryBlocks); + Ok(self.blocks.keys().copied().collect()) + } + + fn block_state(&mut self, block: u64) -> io::Result>> { + Ok(self.blocks.get(&block).map(|state| state.to_vec())) + } + + fn add_block(&mut self, address: u64) -> io::Result<()> { + self.calls.push(Call::Add(address)); + if let Some(errno) = self.add_failures.remove(&address) { + return Err(errno.into()); + } + let block = address / BLOCK_SIZE; + // The kernel refuses to add a block that it has. + if self.blocks.contains_key(&block) { + return Err(Errno::EEXIST.into()); + } + let state = if self.auto_online { + ONLINE_STATE + } else { + OFFLINE_STATE + }; + self.blocks.insert(block, state); + Ok(()) + } + + fn online_block(&mut self, block: u64) -> io::Result<()> { + self.calls.push(Call::Online(block)); + if let Some(errno) = self.online_failures.remove(&block) { + return Err(errno.into()); + } + let state = self.blocks.get_mut(&block).ok_or(Errno::ENOENT)?; + if self.stuck != Some(block) { + *state = ONLINE_STATE; + } + Ok(()) + } + + fn reseed(&mut self, seed: &Seed) -> Result<(), ReseedError> { + self.calls.push(Call::Reseed(seed.generation_id())); + match self.reseed_failure { + Some(errno) => Err(ReseedError::Reseed(errno.into())), + None => Ok(()), + } + } + + fn restore_finish(&mut self, finish: RestoreFinish) -> Result<(), TimeError> { + self.calls.push(Call::Dropped(take_dropped())); + self.calls.push(Call::Finish(finish)); + if self.finish_panics { + panic!("the finish never returns"); + } + if self.finish_fails { + Err(TimeError::StateUnavailable) + } else { + Ok(()) + } + } +} + +/// Returns the seed of the clone, whose bytes start with its new VM generation ID. +fn seed() -> Seed { + let mut bytes = [0x5a; Seed::LENGTH]; + bytes[..GenerationId::LENGTH].copy_from_slice(&RESTORED.to_bytes()); + Seed::from_bytes(bytes) +} + +/// Returns the restore of the clone, which holds the clone's seed, and whose host gates its input +/// until the clone acknowledges it, with nothing to activate. +fn restore() -> Restore { + Restore::to(RESTORED).holding(seed()).gated() +} + +/// Returns the repair of `restore` on `system`, once the test's thread forgot the seeds that it +/// dropped before. +fn repairing(restore: Restore, system: FakeSystem) -> Repair { + take_dropped(); + Repair::with_system(restore, Box::new(system)) +} + +/// Returns the calls of a repair once its memory target is active: the listing of the memory +/// blocks, whose online ones it counts, the reseed, the drop of the seed, which zeroed it before +/// the finish, and the finish, with the acknowledgement. +fn finishing() -> [Call; 4] { + [ + Call::MemoryBlocks, + Call::Reseed(RESTORED), + Call::Dropped(vec![[0; Seed::LENGTH]]), + Call::Finish(RestoreFinish::new().acknowledge()), + ] +} + +/// Returns the addition and the onlining of each of the memory blocks `blocks`, in order. +fn activations(blocks: impl IntoIterator) -> Vec { + blocks + .into_iter() + .flat_map(|block| [Call::Add(block * BLOCK_SIZE), Call::Online(block)]) + .collect() +} + +/// Returns `calls` without the reads of the CPUs and of the block size, which each attempt +/// repeats. +fn changes(calls: Vec) -> Vec { + calls + .into_iter() + .filter(|call| !matches!(call, Call::OnlineCpus | Call::BlockSize)) + .collect() +} + +#[test] +fn a_clone_is_reseeded_then_acknowledged_once_and_reentry_returns_the_same_clone() { + let calls = Calls::default(); + let mut repair = repairing(restore(), FakeSystem::new(&calls)); + let repaired = repair.run().unwrap(); + assert_eq!(repaired.generation_id(), RESTORED); + assert_eq!(repaired.cpus(), 2); + assert_eq!(repaired.memory(), 512 * MIB); + assert!(repaired.acknowledged_at().is_some()); + // The repair dropped the seed, which zeroed it, after the reseed and before the finish. + let mut expected = vec![Call::OnlineCpus, Call::BlockSize]; + expected.extend(finishing()); + assert_eq!(calls.take(), expected); + // Entered again, the repair returns the same clone, without another step. + assert_eq!(repair.run().unwrap(), repaired); + assert_eq!(repair.run().unwrap(), repaired); + assert_eq!(calls.take(), Vec::::new()); + assert_eq!(take_dropped(), Vec::<[u8; Seed::LENGTH]>::new()); +} + +#[test] +fn an_ungated_restore_is_finished_without_an_acknowledgement() { + let calls = Calls::default(); + let restore = Restore::to(RESTORED).holding(seed()); + let repaired = repairing(restore, FakeSystem::new(&calls)).run().unwrap(); + assert_eq!(repaired.acknowledged_at(), None); + assert_eq!( + calls.take().last(), + Some(&Call::Finish(RestoreFinish::new())) + ); +} + +/// Checks that the repair of a clone that captured the first `captured` memory blocks of RAM, in +/// logical order, and whose restore's memory target has the expansion ranges `ranges`, adds and +/// onlines the blocks `added`, in order, after it listed the captured ones, and leaves the clone +/// with `memory` bytes of memory. +fn expands(captured: u64, ranges: &[(u64, u64)], added: &[u64], memory: u64) { + let calls = Calls::default(); + let system = FakeSystem::new(&calls).capturing(captured); + let repaired = repairing(restore().with_memory_target(ranges), system) + .run() + .unwrap(); + assert_eq!(repaired.memory(), memory, "{ranges:?}"); + let mut expected = vec![Call::OnlineCpus, Call::BlockSize]; + if !ranges.is_empty() { + // The listing of the captured blocks, before any addition. + expected.push(Call::MemoryBlocks); + } + expected.extend(activations(added.iter().copied())); + expected.extend(finishing()); + assert_eq!(calls.take(), expected, "{ranges:?}"); +} + +#[test] +fn a_memory_target_adds_and_onlines_the_blocks_that_continue_the_captured_ram() { + // Low RAM, continued. + expands(4, &[(512 * MIB, 512 * MIB)], &[4, 5, 6, 7], GIB); + // Low RAM, continued to its end at 3 GiB, then high RAM from 4 GiB. + expands( + 20, + &[(2560 * MIB, 512 * MIB), (4 * GIB, 512 * MIB)], + &[20, 21, 22, 23, 32, 33, 34, 35], + 3584 * MIB, + ); + // High RAM, after all of low RAM. + expands(24, &[(4 * GIB, 256 * MIB)], &[32, 33], 3328 * MIB); + // High RAM, continued. + expands(28, &[(4608 * MIB, 128 * MIB)], &[36], 3712 * MIB); + // An explicit target without ranges, which the RAM meets. + expands(24, &[], &[], 3 * GIB); +} + +#[test] +fn blocks_that_the_kernel_onlines_itself_are_only_added() { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.auto_online = true; + let restore = restore().with_memory_target(&[(512 * MIB, 256 * MIB)]); + let repaired = repairing(restore, system).run().unwrap(); + assert_eq!(repaired.memory(), 768 * MIB); + let mut expected = vec![ + Call::MemoryBlocks, + Call::Add(512 * MIB), + Call::Add(640 * MIB), + ]; + expected.extend(finishing()); + assert_eq!(changes(calls.take()), expected); +} + +/// Checks that the repair of a clone that captured the first `captured` memory blocks of RAM, in +/// logical order, refuses the memory target with the expansion ranges `ranges`, of which the +/// one at `index` has `fault`, before it adds, onlines, or reseeds anything. +fn refuses(case: &str, captured: u64, ranges: &[(u64, u64)], index: usize, fault: RangeFault) { + let calls = Calls::default(); + let system = FakeSystem::new(&calls).capturing(captured); + let error = repairing(restore().with_memory_target(ranges), system) + .run() + .unwrap_err(); + assert_eq!(error.stage(), RepairStage::Memory, "{case}"); + assert!( + matches!( + error.cause(), + RepairCause::Range { index: faulty, fault: found } if *faulty == index && *found == fault + ), + "{case}: {error}" + ); + assert_eq!( + calls.take(), + [Call::OnlineCpus, Call::BlockSize, Call::MemoryBlocks], + "{case}" + ); +} + +#[test] +fn a_target_that_does_not_continue_the_captured_ram_is_refused_before_any_addition() { + use RangeFault::{Aperture, Extra, Gap, Misaligned, Overlap}; + + // The clone captured 512 MiB of RAM, 2.5 GiB of it, or all 3 GiB of low RAM. + refuses("a gap", 4, &[(640 * MIB, 128 * MIB)], 0, Gap); + refuses("an overlap", 4, &[(384 * MIB, 256 * MIB)], 0, Overlap); + refuses("captured RAM", 4, &[(0, 128 * MIB)], 0, Overlap); + let unaligned = [(512 * MIB + 4096, 128 * MIB)]; + refuses("a misaligned start", 4, &unaligned, 0, Misaligned); + let partial = [(512 * MIB, 64 * MIB)]; + refuses("a misaligned length", 4, &partial, 0, Misaligned); + refuses("an empty range", 4, &[(512 * MIB, 0)], 0, Misaligned); + refuses("an overflow", 4, &[(512 * MIB, u64::MAX)], 0, Misaligned); + refuses("past low RAM", 20, &[(2560 * MIB, GIB)], 0, Aperture); + refuses("the aperture", 24, &[(3 * GIB, 128 * MIB)], 0, Aperture); + let low = [(512 * MIB, 128 * MIB), (640 * MIB, 128 * MIB)]; + refuses("another range of low RAM", 4, &low, 1, Extra); + let high = [(4 * GIB, 128 * MIB), (4224 * MIB, 128 * MIB)]; + refuses("another range of high RAM", 24, &high, 1, Extra); + let reversed = [(4 * GIB, 128 * MIB), (2 * GIB, 128 * MIB)]; + refuses("low RAM after high RAM", 24, &reversed, 1, Extra); + let early = [(512 * MIB, 128 * MIB), (4 * GIB, 128 * MIB)]; + refuses("high RAM before low RAM's end", 4, &early, 1, Gap); +} + +#[test] +fn a_wrong_block_size_ranges_without_a_target_and_unordered_captured_ram_are_refused() { + let calls = Calls::default(); + // A kernel with 1 GiB memory blocks. + let mut system = FakeSystem::new(&calls); + system.block_size = b"40000000\n"; + let target = restore().with_memory_target(&[(512 * MIB, 512 * MIB)]); + let error = repairing(target, system).run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Memory); + assert!(matches!(error.cause(), RepairCause::BlockSize(size) if size == "40000000")); + assert_eq!(calls.take(), [Call::OnlineCpus, Call::BlockSize]); + // Without a target, the online memory counts in blocks of any size. + let mut system = FakeSystem::new(&calls); + system.block_size = b"40000000\n"; + let repaired = repairing(restore(), system).run().unwrap(); + assert_eq!(repaired.memory(), 4 * GIB); + calls.take(); + + // Ranges that the restore lists without a memory target. + let untargeted = restore().with_memory_ranges(&[(512 * MIB, 128 * MIB)]); + let error = repairing(untargeted, FakeSystem::new(&calls)) + .run() + .unwrap_err(); + assert_eq!(error.stage(), RepairStage::Memory); + assert!(matches!(error.cause(), RepairCause::UntargetedRanges)); + assert_eq!(calls.take(), [Call::OnlineCpus]); + + // Captured RAM whose blocks are not the first ones of RAM. + let mut system = FakeSystem::new(&calls); + system.blocks.remove(&2); + let target = restore().with_memory_target(&[(384 * MIB, 128 * MIB)]); + let error = repairing(target, system).run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Memory); + assert!(matches!(error.cause(), RepairCause::CapturedRam)); + assert_eq!( + calls.take(), + [Call::OnlineCpus, Call::BlockSize, Call::MemoryBlocks] + ); +} + +#[test] +fn a_retry_after_a_failed_activation_adds_no_block_twice() { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.add_failures.insert(768 * MIB, Errno::EIO); + system.online_failures.insert(7, Errno::EBUSY); + let target = restore().with_memory_target(&[(512 * MIB, 512 * MIB)]); + let mut repair = repairing(target, system); + let error = repair.run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Memory); + assert!(matches!( + error.cause(), + RepairCause::Io(error) if error.raw_os_error() == Some(Errno::EIO as i32) + )); + let mut expected = vec![Call::MemoryBlocks]; + expected.extend(activations([4, 5])); + expected.push(Call::Add(768 * MIB)); + assert_eq!(changes(calls.take()), expected); + + // The retry keeps the captured RAM's end, so that the blocks that the first attempt added + // count as done, not as captured RAM: it neither lists the blocks before it adds one, nor + // adds one twice. + let error = repair.run().unwrap_err(); + assert!(matches!( + error.cause(), + RepairCause::Io(error) if error.raw_os_error() == Some(Errno::EBUSY as i32) + )); + let mut expected = activations([6]); + expected.extend([Call::Add(896 * MIB), Call::Online(7)]); + assert_eq!(changes(calls.take()), expected); + + // The next retry onlines the block that failed to come online, without adding it again. + let repaired = repair.run().unwrap(); + assert_eq!(repaired.memory(), GIB); + let mut expected = vec![Call::Online(7)]; + expected.extend(finishing()); + assert_eq!(changes(calls.take()), expected); +} + +#[test] +fn a_block_that_stays_offline_fails_each_activation() { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.stuck = Some(5); + let target = restore().with_memory_target(&[(512 * MIB, 256 * MIB)]); + let mut repair = repairing(target, system); + for _ in 0..2 { + let error = repair.run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Memory); + assert!(matches!( + error.cause(), + RepairCause::Offline { block: 5, state } if state == "offline" + )); + } + let mut expected = vec![Call::MemoryBlocks]; + expected.extend(activations([4, 5])); + expected.push(Call::Online(5)); + assert_eq!(changes(calls.take()), expected); +} + +#[test] +fn the_reseed_runs_once_and_its_failure_is_final() { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.reseed_failure = Some(Errno::EPERM); + let mut repair = repairing(restore(), system); + let error = repair.run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Reseed); + assert!(matches!( + error.cause(), + RepairCause::Reseed(ReseedError::Reseed(_)) + )); + assert_eq!( + calls.take(), + [ + Call::OnlineCpus, + Call::BlockSize, + Call::MemoryBlocks, + Call::Reseed(RESTORED) + ] + ); + // The failed reseed dropped the seed all the same, which zeroed it. + assert_eq!(take_dropped(), [[0; Seed::LENGTH]]); + // Every retry returns the same error, without a second reseed or a finish. + for _ in 0..2 { + let again = repair.run().unwrap_err(); + assert!(Arc::ptr_eq(&again.cause, &error.cause)); + } + assert_eq!(calls.take(), Vec::::new()); +} + +#[test] +fn the_finish_runs_once_and_its_failure_is_final() { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.finish_fails = true; + let mut repair = repairing(restore(), system); + let error = repair.run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Finish); + assert!(matches!( + error.cause(), + RepairCause::Time(TimeError::StateUnavailable) + )); + let mut expected = vec![Call::OnlineCpus, Call::BlockSize]; + expected.extend(finishing()); + assert_eq!(calls.take(), expected); + // Every retry returns the same error, without a second acknowledgement. + for _ in 0..2 { + let again = repair.run().unwrap_err(); + assert!(Arc::ptr_eq(&again.cause, &error.cause)); + } + assert_eq!(calls.take(), Vec::::new()); +} + +#[test] +fn a_finish_that_never_returned_never_runs_again() { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.finish_panics = true; + let mut repair = repairing(restore(), system); + let unwound = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| repair.run())); + assert!(unwound.is_err()); + assert_eq!( + calls.take().last(), + Some(&Call::Finish(RestoreFinish::new().acknowledge())) + ); + for _ in 0..2 { + let error = repair.run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Finish); + assert!(matches!(error.cause(), RepairCause::Incomplete)); + } + assert_eq!(calls.take(), Vec::::new()); +} + +/// Checks that the repair of a clone whose kernel lists the online CPUs `cpus`, and whose restore +/// has the processor target `target`, if any, counts `online` CPUs. +fn counts(cpus: &'static [u8], target: Option, online: u32) { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.cpus = cpus; + let restore = match target { + Some(target) => restore().with_processor_target(target), + None => restore(), + }; + let repaired = repairing(restore, system).run().unwrap(); + assert_eq!(repaired.cpus(), online, "{cpus:?}"); +} + +/// Checks that the repair of a clone whose kernel lists the online CPUs `cpus`, and whose restore +/// has the processor target `target`, if any, fails with `cause`, before any other call. +fn refuses_cpus(cpus: &'static [u8], target: Option, cause: &RepairCause) { + let calls = Calls::default(); + let mut system = FakeSystem::new(&calls); + system.cpus = cpus; + let restore = match target { + Some(target) => restore().with_processor_target(target), + None => restore(), + }; + let error = repairing(restore, system).run().unwrap_err(); + assert_eq!(error.stage(), RepairStage::Processors, "{cpus:?}"); + assert_eq!(error.cause().to_string(), cause.to_string()); + assert_eq!(calls.take(), [Call::OnlineCpus], "{cpus:?}"); +} + +#[test] +fn the_online_cpus_must_be_a_prefix_that_meets_the_processor_target() { + counts(b"0\n", None, 1); + counts(b"0-3\n", None, 4); + counts(b"0-3\n", Some(4), 4); + // The repair brings no CPU online, nor takes one offline. + let cause = |target, online| RepairCause::ProcessorTarget { target, online }; + refuses_cpus(b"0-3\n", Some(8), &cause(8, 4)); + refuses_cpus(b"0-3\n", Some(2), &cause(2, 4)); + let lists: [&'static [u8]; 4] = [b"0-1,3\n", b"1-3\n", b"0-\n", b"\n"]; + for cpus in lists { + let listed = String::from_utf8_lossy(cpus.strip_suffix(b"\n").unwrap()).into_owned(); + refuses_cpus(cpus, None, &RepairCause::OnlineCpus(listed)); + } +} + +#[test] +fn a_restore_without_its_seed_or_that_its_capture_finished_is_refused() { + let calls = Calls::default(); + let error = repairing(Restore::to(RESTORED).gated(), FakeSystem::new(&calls)) + .run() + .unwrap_err(); + assert_eq!(error.stage(), RepairStage::Restore); + assert!(matches!(error.cause(), RepairCause::NoSeed)); + let finished = restore().finished_by_capture(); + let error = repairing(finished, FakeSystem::new(&calls)) + .run() + .unwrap_err(); + assert_eq!(error.stage(), RepairStage::Restore); + assert!(matches!(error.cause(), RepairCause::Finished)); + assert_eq!(calls.take(), Vec::::new()); +} + +#[test] +fn errors_name_their_stage_and_their_cause() { + use std::error::Error; + + let memory = |cause| RepairError::new(RepairStage::Memory, cause); + let cases = [ + ( + memory(RepairCause::Io(at( + Path::new("/sys/x"), + &io::Error::from(Errno::EIO), + ))), + "failed to activate the memory target: /sys/x: I/O error", + ), + ( + memory(RepairCause::Range { + index: 0, + fault: RangeFault::Overlap, + }), + "failed to activate the memory target: memory range 0 overlaps the RAM before it", + ), + ( + memory(RepairCause::Range { + index: 1, + fault: RangeFault::Aperture, + }), + "failed to activate the memory target: memory range 1 enters the MMIO aperture \ + between 3 GiB and 4 GiB", + ), + ( + memory(RepairCause::BlockSize("40000000".to_owned())), + "failed to activate the memory target: the memory block size is 40000000 in \ + hexadecimal, not 128 MiB", + ), + ( + memory(RepairCause::Offline { + block: 5, + state: "offline".to_owned(), + }), + "failed to activate the memory target: memory block 5 is offline, not online", + ), + ( + RepairError::new( + RepairStage::Processors, + RepairCause::ProcessorTarget { + target: 8, + online: 4, + }, + ), + "failed to check the online CPUs: the restore asks for 8 online CPUs, and 4 are \ + online", + ), + ( + RepairError::new(RepairStage::Restore, RepairCause::NoSeed), + "failed to check the restore: the restore holds no seed", + ), + ( + RepairError::new( + RepairStage::Reseed, + RepairCause::Reseed(ReseedError::Reseed(Errno::EPERM.into())), + ), + "failed to reseed the kernel's random number generator: random ioctl: Operation not \ + permitted", + ), + ( + RepairError::new( + RepairStage::Finish, + RepairCause::Time(TimeError::StateUnavailable), + ), + "failed to finish the restore: the time state is unavailable", + ), + ( + RepairError::new(RepairStage::Finish, RepairCause::Incomplete), + "failed to finish the restore: an earlier attempt never returned", + ), + ]; + for (error, text) in cases { + assert_eq!(error.to_string(), text); + } + assert!( + memory(RepairCause::Io(Errno::EIO.into())) + .source() + .is_some() + ); + let reseed = RepairError::new( + RepairStage::Reseed, + RepairCause::Reseed(ReseedError::Reseed(Errno::EPERM.into())), + ); + assert!(reseed.source().is_some()); + assert!(memory(RepairCause::CapturedRam).source().is_none()); +} + +#[test] +fn a_repair_draws_no_randomness_before_its_reseed() { + // A seccomp filter lasts as long as the thread that installs it, and passes to the processes + // that the thread starts, so the test runs in a process of its own. + if crate::guest::managed_agent::isolate_process_test( + "NVX_CLONE_REPAIR_RANDOMNESS_TEST_CHILD", + "guest::clone::repair::tests::a_repair_draws_no_randomness_before_its_reseed", + ) { + return; + } + let calls = Calls::default(); + let target = restore().with_memory_target(&[(512 * MIB, 512 * MIB)]); + let mut repair = repairing(target, FakeSystem::new(&calls)); + // From here on, `getrandom`, or an open, which a read of `/dev/urandom` or `/dev/random` + // needs, ends the process with `SIGSYS`. The fake system opens no file, so only randomness + // that the repair drew, such as the keys of a hashed map, would make either call. + sys::forbid_system_calls(&[ + nix::libc::SYS_getrandom, + nix::libc::SYS_open, + nix::libc::SYS_openat, + nix::libc::SYS_openat2, + ]); + let repaired = repair.run().unwrap(); + assert_eq!(repaired.memory(), GIB); + let mut expected = vec![Call::OnlineCpus, Call::BlockSize, Call::MemoryBlocks]; + expected.extend(activations(4..8)); + expected.extend(finishing()); + assert_eq!(calls.take(), expected); +} + +/// Writes `contents` to the file at `path`, below directories that it creates. +fn write_file(path: &Path, contents: &str) { + fs::create_dir_all(path.parent().unwrap()).unwrap(); + fs::write(path, contents).unwrap(); +} + +#[test] +fn the_live_system_reads_and_writes_the_kernel_s_cpu_and_memory_interfaces() { + let root = tempfile::tempdir().unwrap(); + let memory = root.path().join(MEMORY_BLOCKS); + write_file(&root.path().join(ONLINE_CPUS), "0-3\n"); + write_file(&memory.join(BLOCK_SIZE_FILE), "8000000\n"); + write_file(&memory.join(PROBE_FILE), ""); + write_file(&memory.join("memory0/state"), "online\n"); + write_file(&memory.join("memory12/state"), "offline\n"); + write_file(&memory.join("memory2/state"), "online\n"); + // None of these is a memory block. + write_file(&memory.join("auto_online_blocks"), "offline\n"); + fs::create_dir(memory.join("power")).unwrap(); + fs::create_dir(memory.join("memoryx")).unwrap(); + let mut system = LiveSystem { + paths: Paths { + proc: root.path().join("proc"), + sys: root.path().to_owned(), + dev: root.path().join("dev"), + }, + abi: TimeAbi::report_only(), + }; + assert_eq!(system.online_cpus().unwrap(), b"0-3\n"); + assert_eq!(system.block_size().unwrap(), b"8000000\n"); + assert_eq!(system.memory_blocks().unwrap(), [0, 2, 12]); + assert_eq!(system.block_state(12).unwrap().unwrap(), b"offline\n"); + assert_eq!(system.block_state(4).unwrap(), None); + // An addition writes the block's physical address to the probe file, and an onlining writes + // the block's new state, as `nvx-snapshot` does. sysfs takes each write whole, so the test + // empties the state file before the onlining writes it. + system.add_block(4 * BLOCK_SIZE).unwrap(); + assert_eq!(fs::read(memory.join(PROBE_FILE)).unwrap(), b"0x20000000\n"); + fs::write(memory.join("memory12/state"), "").unwrap(); + system.online_block(12).unwrap(); + assert_eq!( + fs::read(memory.join("memory12/state")).unwrap(), + b"online\n" + ); + // A write creates no file, and errors name the file. + let state = memory.join("memory4/state"); + let error = system.online_block(4).unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::NotFound); + assert!(!state.exists()); + let named = format!("{}: ", state.display()); + assert!(error.to_string().starts_with(&named), "{error}"); + fs::remove_file(memory.join(BLOCK_SIZE_FILE)).unwrap(); + let error = system.block_size().unwrap_err(); + let named = format!("{}: ", memory.join(BLOCK_SIZE_FILE).display()); + assert!(error.to_string().starts_with(&named), "{error}"); +} diff --git a/nvx-rs/src/guest/clone/tests.rs b/nvx-rs/src/guest/clone/tests.rs index 52b0ae1a3..8fc75afe9 100644 --- a/nvx-rs/src/guest/clone/tests.rs +++ b/nvx-rs/src/guest/clone/tests.rs @@ -12,6 +12,7 @@ use std::sync::{Arc, Mutex, PoisonError}; use super::mounts::{self, DeviceNumber}; use super::*; +use crate::guest::reseed::Seed; use crate::guest::sys::{Errno, PortError}; /// The base address of the distro device's MMIO window. @@ -299,7 +300,9 @@ impl System for FakeSystem { self.calls.push(Call::Capture(request)); match self.outcome { Outcome::Continued => Ok(CaptureOutcome::Continued), - Outcome::Restored => Ok(CaptureOutcome::Restored(Restore::to(RESTORED))), + Outcome::Restored => Ok(CaptureOutcome::Restored( + Restore::to(RESTORED).holding(seed()), + )), Outcome::Fails => Err(TimeError::StateUnavailable), Outcome::PortRelease => Err(PortError::Revoke(Errno::ENOMEM).into()), } @@ -363,13 +366,16 @@ fn preparation_calls() -> Vec { ] } -/// Returns the platform tier's request of a capture of the template. +/// Returns the platform tier's request of a capture of the template, in the in-memory mode. fn platform_request() -> CaptureRequest { - CaptureRequest::new( - ScratchPolicy::Fresh, - GENERATION, - PathBuf::from("/run/nvx/restore-entropy"), - ) + CaptureRequest::in_memory(ScratchPolicy::Fresh, GENERATION) +} + +/// Returns the seed of the clone, whose bytes start with its new VM generation ID. +fn seed() -> Seed { + let mut bytes = [0x5a; Seed::LENGTH]; + bytes[..GenerationId::LENGTH].copy_from_slice(&RESTORED.to_bytes()); + Seed::from_bytes(bytes) } #[test] @@ -800,18 +806,22 @@ fn a_restored_clone_gets_the_fence_closed_with_the_detectors_suppressed() { [Call::GenerationId, Call::Capture(platform_request())] ); assert_eq!(clone.restore().generation_id(), RESTORED); - assert_eq!(clone.entropy_path(), Path::new("/run/nvx/restore-entropy")); - let (restore, mut fence) = clone.into_parts(); + let (mut restore, mut fence) = clone.into_parts(); assert_eq!( format!("{restore:?}"), - format!("{:?}", Restore::to(RESTORED)) + format!("{:?}", Restore::to(RESTORED).holding(seed())) + ); + // The restore holds the clone's seed, for repair to take. + assert_eq!( + restore.take_seed().map(|seed| seed.generation_id()), + Some(RESTORED) ); assert_eq!(calls.take(), Vec::::new()); // A preparation behind another handle of the fence cannot close it, so it opens nothing. let error = prepare(fence.handle(), FakeSystem::new(&calls)).unwrap_err(); assert_eq!(error.stage(), CaptureStage::Fence); assert_eq!(calls.take(), Vec::::new()); - // The caller opens the fence once repair acknowledged the restore. + // The caller opens the fence once the clone may admit executions, after its repair. fence.open(); assert_eq!(calls.take(), [Call::Open]); diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 329c42cca..60374c047 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -13,7 +13,10 @@ //! [`gpt`], through which it reads the partition table of a GPT disk that the kernel attached as //! a block device; and, on x86-64, `clone`, through which a template's guest prepares for and //! requests a snapshot's capture before its root's assembly, behind the caller's fence of -//! workload admission, with nothing mounted from the devices that its clones replace. +//! workload admission, with nothing mounted from the devices that its clones replace, and +//! through which each clone that a restore of the snapshot starts repairs its restore: it +//! activates the restore's memory target, reseeds the kernel's random number generator, and +//! finishes the restore, with its acknowledgement. //! //! Each other module holds one program, and the crate's binary of the same name runs its `main`. //! Two of them also offer their steps as typed calls, which any guest init or agent runs in its diff --git a/nvx-rs/src/guest/reseed.rs b/nvx-rs/src/guest/reseed.rs index 1f7654fd3..f30c7ff95 100644 --- a/nvx-rs/src/guest/reseed.rs +++ b/nvx-rs/src/guest/reseed.rs @@ -412,7 +412,7 @@ fn print_random_sample(out: &mut dyn Write, err: &mut dyn Write) -> Result<(), C } #[cfg(test)] -mod tests { +pub(crate) mod tests { use std::cell::RefCell; use std::fs; @@ -435,6 +435,12 @@ mod tests { DROPPED.with_borrow_mut(|dropped| dropped.push(*seed.0)); } + /// Returns the bytes of each seed that the test's thread dropped since the last call, in + /// order, as their drops left them. + pub(crate) fn take_dropped() -> Vec<[u8; Seed::LENGTH]> { + DROPPED.take() + } + struct Fixture { directory: tempfile::TempDir, } diff --git a/nvx-rs/src/guest/sys/mod.rs b/nvx-rs/src/guest/sys/mod.rs index de638151f..01494157b 100644 --- a/nvx-rs/src/guest/sys/mod.rs +++ b/nvx-rs/src/guest/sys/mod.rs @@ -115,6 +115,8 @@ pub(crate) use port::PortBackend; pub(crate) use port::{PORT_DEVICE, Port}; #[cfg(target_arch = "x86_64")] pub(crate) use port::{PortAccess, PortError, check_released, power_off, probe_port}; +#[cfg(all(test, target_arch = "x86_64"))] +pub(crate) use process::forbid_system_calls; pub(crate) use process::set_command_name; #[cfg(target_arch = "x86_64")] pub(crate) use process::syscall_process_id; diff --git a/nvx-rs/src/guest/sys/process.rs b/nvx-rs/src/guest/sys/process.rs index 53e226abb..80d333f9e 100644 --- a/nvx-rs/src/guest/sys/process.rs +++ b/nvx-rs/src/guest/sys/process.rs @@ -473,40 +473,96 @@ pub(crate) fn continue_traced(process: ProcessId) { /// that it starts from now on, through a seccomp filter that allows every other call, for a test /// that runs in a process of its own, since no filter can be removed. It first sets the thread's /// `no_new_privs`, without which a filter needs privileges. -/// -/// libc's `prctl` is used because `nix` does not wrap seccomp. #[cfg(test)] pub(crate) fn deny_system_call(number: nix::libc::c_long) { - use nix::libc::{self, c_ulong, sock_filter}; + use nix::libc; - let statement = |code: u32, if_equal: u8, otherwise: u8, k: u32| sock_filter { - // Every BPF code fits 16 bits. - code: code as u16, - jt: if_equal, - jf: otherwise, - k, - }; // The filter loads the call's number, the first word of the kernel's `seccomp_data`, and // returns `EPERM` for `number`, or allows the call. let mut filter = [ - statement(libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, 0, 0, 0), + filter_statement(libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, 0, 0, 0), // A system call number fits 32 bits. - statement( + filter_statement( libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 0, 1, number as u32, ), - statement( + filter_statement( libc::BPF_RET | libc::BPF_K, 0, 0, libc::SECCOMP_RET_ERRNO | libc::EPERM as u32, ), - statement(libc::BPF_RET | libc::BPF_K, 0, 0, libc::SECCOMP_RET_ALLOW), + filter_statement(libc::BPF_RET | libc::BPF_K, 0, 0, libc::SECCOMP_RET_ALLOW), ]; + install_seccomp_filter(&mut filter); +} + +/// Ends the calling process with `SIGSYS` at the first of the system calls `numbers` that the +/// calling thread, or a process that it starts from now on, makes, through a seccomp filter that +/// allows every other call, for a test that runs in a process of its own, since no filter can be +/// removed. It first sets the thread's `no_new_privs`, without which a filter needs privileges. +#[cfg(all(test, target_arch = "x86_64"))] +pub(crate) fn forbid_system_calls(numbers: &[nix::libc::c_long]) { + use nix::libc; + + let count = u8::try_from(numbers.len()).expect("a filter's jump spans at most 255 statements"); + // The filter loads the call's number, then compares it with each of `numbers`: a match jumps + // past the other comparisons and the allowance, to the end of the process. + let mut filter = vec![filter_statement( + libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, + 0, + 0, + 0, + )]; + for (compared, &number) in (0..count).zip(numbers) { + // A system call number fits 32 bits. + filter.push(filter_statement( + libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, + count - compared, + 0, + number as u32, + )); + } + filter.push(filter_statement( + libc::BPF_RET | libc::BPF_K, + 0, + 0, + libc::SECCOMP_RET_ALLOW, + )); + filter.push(filter_statement( + libc::BPF_RET | libc::BPF_K, + 0, + 0, + libc::SECCOMP_RET_KILL_PROCESS, + )); + install_seccomp_filter(&mut filter); +} + +/// Returns the classic BPF statement of a seccomp filter with the operation `code`, the jumps +/// `if_equal` and `otherwise` of a comparison, and the operand `k`. +#[cfg(test)] +fn filter_statement(code: u32, if_equal: u8, otherwise: u8, k: u32) -> nix::libc::sock_filter { + nix::libc::sock_filter { + // Every BPF code fits 16 bits. + code: code as u16, + jt: if_equal, + jf: otherwise, + k, + } +} + +/// Sets the calling thread's `no_new_privs`, then installs `filter` as a seccomp filter of the +/// thread, which passes to the processes that it starts. +/// +/// libc's `prctl` is used because `nix` does not wrap seccomp. +#[cfg(test)] +fn install_seccomp_filter(filter: &mut [nix::libc::sock_filter]) { + use nix::libc::{self, c_ulong}; + let program = libc::sock_fprog { - len: filter.len() as u16, + len: u16::try_from(filter.len()).expect("a filter has fewer than 65,536 statements"), filter: filter.as_mut_ptr(), }; // SAFETY: `PR_SET_NO_NEW_PRIVS` reads only its integer arguments. @@ -520,7 +576,7 @@ pub(crate) fn deny_system_call(number: nix::libc::c_long) { ) }; assert_eq!(result, 0, "PR_SET_NO_NEW_PRIVS: {}", Errno::last()); - // SAFETY: the program names its four statements, which the kernel copies before it returns. + // SAFETY: the program names its statements, which the kernel copies before it returns. let result = unsafe { libc::prctl( libc::PR_SET_SECCOMP, diff --git a/nvx-rs/src/guest/time/api.rs b/nvx-rs/src/guest/time/api.rs index 0fc72ea83..e425a3e8d 100644 --- a/nvx-rs/src/guest/time/api.rs +++ b/nvx-rs/src/guest/time/api.rs @@ -596,6 +596,47 @@ impl Restore { seed: None, } } + + /// Returns the restore with `seed`, as a capture in the in-memory mode returns it. + pub(crate) fn holding(mut self, seed: Seed) -> Self { + self.seed = Some(seed); + self + } + + /// Returns the restore with the host's input gated until the guest acknowledges it. + pub(crate) fn gated(mut self) -> Self { + self.flags |= decoders::PACKET_ACK_REQUIRED; + self + } + + /// Returns the restore that asks for `count` online processors. + pub(crate) fn with_processor_target(mut self, count: u8) -> Self { + self.online_vp_count = count; + self + } + + /// Returns the restore with a memory target, whose expansion ranges are `ranges`, each a + /// start and a length. + pub(crate) fn with_memory_target(mut self, ranges: &[(u64, u64)]) -> Self { + self.flags |= decoders::PACKET_MEMORY_TARGET; + self.with_memory_ranges(ranges) + } + + /// Returns the restore that lists `ranges`, each a start and a length, whatever its flags + /// select. + pub(crate) fn with_memory_ranges(mut self, ranges: &[(u64, u64)]) -> Self { + self.memory_ranges = ranges + .iter() + .map(|&(start, length)| MemoryRange { start, length }) + .collect(); + self + } + + /// Returns the restore as one that its capture finished. + pub(crate) fn finished_by_capture(mut self) -> Self { + self.finished = true; + self + } } /// A range of guest physical memory that a restore adds. From 9831e19d95ecd0cdd9c0272ba25412675c26c529 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Tue, 27 Oct 2026 17:09:58 -0700 Subject: [PATCH 2/2] nvx-rs: take a repaired clone's acknowledgement instant before its finish Repaired::acknowledged_at was taken once the restore's finish returned. The finish acknowledges the restore before any of the deferred time work, and without a time daemon it runs that work in line before it returns, which may last up to the stall detectors' timeout. A repaired clone's activation (#843) starts its deadline at the acknowledgement, so the instant could start that deadline late, by as long as the deferred work took, and let an activation run past its timeout. The repair now takes the instant right before the finish, so it is no later than the acknowledgement: a deadline that starts at it may end early, by the finish's own steps before it acknowledges, but never late. A test whose fake finish runs deferred work for 50 ms once it acknowledged checks that the instant falls between the reseed and the acknowledgement. Part of #842 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbebd4aa-2ad7-427d-80ea-49358a645e9a --- nvx-rs/src/guest/clone/repair.rs | 13 +++- nvx-rs/src/guest/clone/repair/tests.rs | 97 +++++++++++++++++++++++++- 2 files changed, 106 insertions(+), 4 deletions(-) diff --git a/nvx-rs/src/guest/clone/repair.rs b/nvx-rs/src/guest/clone/repair.rs index 20f1c16e2..006071d5f 100644 --- a/nvx-rs/src/guest/clone/repair.rs +++ b/nvx-rs/src/guest/clone/repair.rs @@ -256,6 +256,8 @@ impl Repair { RepairStage::Finish, RepairCause::Incomplete, )); + // The finish acknowledges before any deferred work, which it may run in line. + let acknowledged_at = acknowledge.then(Instant::now); if let Err(error) = self.system.restore_finish(finish) { let error = RepairError::new(RepairStage::Finish, RepairCause::Time(error)); self.progress = Progress::Failed(error.clone()); @@ -265,7 +267,7 @@ impl Repair { generation_id: self.restore.generation_id(), cpus: resources.cpus, memory: resources.memory, - acknowledged_at: acknowledge.then(Instant::now), + acknowledged_at, }; self.progress = Progress::Repaired(repaired); Ok(repaired) @@ -321,7 +323,8 @@ pub struct Repaired { cpus: u32, /// The memory size of the online memory blocks, in bytes. memory: u64, - /// When the repair acknowledged the restore, if the restore asked for it. + /// An instant no later than the repair's acknowledgement of the restore, taken right before + /// the finish, if the restore asked for the acknowledgement. acknowledged_at: Option, } @@ -342,8 +345,12 @@ impl Repaired { self.memory } - /// Returns when the repair acknowledged the restore, once its finish returned, or `None` if + /// Returns an instant no later than the repair's acknowledgement of the restore, or `None` if /// the host did not gate its input until the acknowledgement. + /// + /// The repair takes the instant right before its finish, which acknowledges the restore + /// before any of the deferred time work, and may run that work before it returns, so a + /// deadline that starts at the acknowledgement may count from the instant. pub fn acknowledged_at(&self) -> Option { self.acknowledged_at } diff --git a/nvx-rs/src/guest/clone/repair/tests.rs b/nvx-rs/src/guest/clone/repair/tests.rs index ab0743d84..ee3217fba 100644 --- a/nvx-rs/src/guest/clone/repair/tests.rs +++ b/nvx-rs/src/guest/clone/repair/tests.rs @@ -5,7 +5,9 @@ use std::collections::BTreeMap; use std::fs; use std::path::Path; -use std::sync::{Arc, Mutex, PoisonError}; +use std::sync::{Arc, Mutex, OnceLock, PoisonError}; +use std::thread; +use std::time::Duration; use super::*; use crate::guest::reseed::tests::take_dropped; @@ -277,6 +279,99 @@ fn an_ungated_restore_is_finished_without_an_acknowledgement() { ); } +/// How long the finish of a [`Deferring`] system runs the restore's deferred work after its +/// acknowledgement: long enough that an instant taken once the finish returned would follow the +/// acknowledgement by far. +const DEFERRED_WORK: Duration = Duration::from_millis(50); + +/// A system that makes its calls through a [`FakeSystem`], and whose finish acknowledges the +/// restore, then runs the restore's deferred work for [`DEFERRED_WORK`] before it returns, as the +/// time ABI's finish does without a time daemon. It records when its reseed, the step before the +/// finish, returned, and when its finish acknowledged the restore. +struct Deferring { + /// The system through which it makes its calls. + system: FakeSystem, + /// When the reseed returned. + reseeded: Arc>, + /// When the finish acknowledged the restore. + acknowledged: Arc>, +} + +impl RepairSystem for Deferring { + fn online_cpus(&mut self) -> io::Result> { + self.system.online_cpus() + } + + fn block_size(&mut self) -> io::Result> { + self.system.block_size() + } + + fn memory_blocks(&mut self) -> io::Result> { + self.system.memory_blocks() + } + + fn block_state(&mut self, block: u64) -> io::Result>> { + self.system.block_state(block) + } + + fn add_block(&mut self, address: u64) -> io::Result<()> { + self.system.add_block(address) + } + + fn online_block(&mut self, block: u64) -> io::Result<()> { + self.system.online_block(block) + } + + fn reseed(&mut self, seed: &Seed) -> Result<(), ReseedError> { + let result = self.system.reseed(seed); + self.reseeded.set(Instant::now()).unwrap(); + result + } + + fn restore_finish(&mut self, finish: RestoreFinish) -> Result<(), TimeError> { + self.system.restore_finish(finish)?; + self.acknowledged.set(Instant::now()).unwrap(); + thread::sleep(DEFERRED_WORK); + Ok(()) + } +} + +#[test] +fn the_instant_of_the_acknowledgement_precedes_the_deferred_work_of_the_finish() { + let calls = Calls::default(); + let reseeded = Arc::new(OnceLock::new()); + let acknowledged = Arc::new(OnceLock::new()); + let system = Deferring { + system: FakeSystem::new(&calls), + reseeded: Arc::clone(&reseeded), + acknowledged: Arc::clone(&acknowledged), + }; + let restore = restore(); + take_dropped(); + let repaired = Repair::with_system(restore, Box::new(system)) + .run() + .unwrap(); + let returned = Instant::now(); + let reseeded = *reseeded.get().unwrap(); + let acknowledged = *acknowledged.get().unwrap(); + // The finish returned long after it acknowledged the restore, but the repair took its instant + // between the reseed and the acknowledgement, so that a deadline that starts at the instant + // starts no later than at the acknowledgement. + assert!(returned.duration_since(acknowledged) >= DEFERRED_WORK); + let instant = repaired.acknowledged_at().unwrap(); + assert!( + reseeded <= instant, + "{instant:?} precedes the reseed, {reseeded:?}" + ); + assert!( + instant <= acknowledged, + "{instant:?} follows the acknowledgement, {acknowledged:?}" + ); + let mut expected = vec![Call::OnlineCpus, Call::BlockSize]; + expected.extend(finishing()); + assert_eq!(calls.take(), expected); +} + /// Checks that the repair of a clone that captured the first `captured` memory blocks of RAM, in /// logical order, and whose restore's memory target has the expansion ranges `ranges`, adds and /// onlines the blocks `added`, in order, after it listed the captured ones, and leaves the clone