diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 1a6ef325b..5d5d42286 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -120,7 +120,7 @@ on Linux and Windows. | `nvx::guest::exec` | `Execution`: drives one program that the caller spawned with the program's ends of its `Pipes`, or that the caller started itself and adopts with its own ends of the program's pipes and a `Child` that names it, without blocking, from the caller's own loop or reactor, and reports each step as an `Event`: the launch report, which an adopted program has at once, the output of each stream, kill failures, and the `Outcome`. It reads a stream only as far as the caller's `grant` allows, so a stalled sink stalls its own execution alone, writes the program's standard input without `SIGPIPE`, and ends the program on a cancellation, a timeout, or a disconnection, with `SIGKILL` through the program's pidfd to the program and, if the caller asks, its process group, while no collector has taken its status, and, through the opened directory of a cgroup that the caller names, to that cgroup. It closes the program's input once the program has ended. It waits for its program alone, by a pidfd, or takes the status from the caller's collector with `observe`; with `Options::caller_reaps`, that collector is the program's only one. NVX's managed agent is one caller | | `nvx::guest::exec::binding` | `Binding`: an `nvx::exec::Runner` bound to an `Execution`, the glue that a guest adapter would otherwise write. The runner's read allowances become grants, its discards discards, the input that it accepted writes and then a close, its terminations a cancellation, a disconnection, or a termination, and its settlement a kill of the execution's cgroup and checks until no process occupies it; without a cgroup, nothing proves that the program's descendants are gone, so the execution is fenced. It feeds the execution's events back to the runner, statuses that a caller who reaps hands over with `observe` included, and never blocks: the caller polls its descriptors and its input writer until its deadline, and each call takes a bounded number of steps, so that no execution holds the caller's loop | | `nvx::guest::time` | x86_64: `TimeAbi`, the steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent: the cold-boot time step and, separately, the start of the time daemon in a child process that the caller supervises, the VM generation ID, a `CaptureRequest` that repairs a `Restore`, and a `RestoreFinish`. The request's mode selects how a restore hands over its fresh entropy: the file mode, which `nvx-time` keeps, writes it to a file, and the in-memory mode returns it as the `Restore`'s owned `Seed`, and writes no file. A violation of the ABI stops the VM in enforcing mode, with the time ABI's status, as `nvx-time` does; in strict mode, `TimeAbi::strict`, which a caller that owns the VM's exit status and shutdown selects, the call fails with the `Violation`, whose `status` is the one with which enforcing mode stops the VM, and the time daemon and a capture's preparation enforce the ABI all the same. A call takes the permission for OpenVMM's ports on the calling thread for each port transaction alone, and fails with `TimeError::PortRelease` if the thread may still hold some of it, which a process that it started would inherit, and from which `nvx::guest::spawn` starts none. No call waits for a child that it did not start. `nvx-time` runs its steps through them | -| `nvx::guest::clone` | x86_64: the preparation of a template's guest for a capture before its root's assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init or agent, whether or not it is PID 1. `CapturePreparation`, of the caller's `AdmissionFence` and the `Devices` that the template's clones replace, the `distro` and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag: `prepare` closes the fence, so that no execution starts, refuses with a `Mounted` the first mount from those devices, a distro partition included, or from an overlay, bind mounts included, in a bounded read of `/proc/self/mountinfo` of at most `MOUNT_TABLE_LIMIT` bytes, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as `nvx-time pre-capture` does, and returns an owned `CaptureBarrier`. `CaptureBarrier::capture` requests the capture with the platform tier's request, in the in-memory mode, through `TimeAbi::capture`, and returns `Captured::Continued` once it opened the fence and restored the detectors' settings, or, in a clone, `Captured::Restored`, with the fence still closed, since nothing else may run before repair acknowledges the restore, and the detectors suppressed until the time daemon restores their settings: `Restored::into_parts` hands the `Restore` that repair takes next, which holds the restore's `Seed`, and the closed fence to the caller, which keeps it closed through the repair and opens it once the clone may admit executions, such as once it is activated. `CaptureBarrier::cancel`, a dropped barrier, and a failure to read the VM generation ID open the fence and restore the detectors' settings; a capture request that fails restores the settings but leaves the fence closed, since it may fail in a clone whose restore it could not repair. Each failure is a `CaptureError`, with its `CaptureStage` and `CaptureCause`. `repair`, or a `Repair` that the caller may retry, repairs the clone of a `Restore` next: it checks that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online; with a memory target, it checks, before it adds any memory block, that the kernel's block size is 128 MiB and that the restore's ranges continue the captured RAM in logical order, with no gap, overlap, or range in the MMIO aperture, while the host bounds the target by the snapshot's capacity, then adds and onlines the blocks through sysfs, as `nvx-snapshot` does, and verifies them; it reseeds the kernel's random number generator with the `Seed`, having drawn no randomness before, and drops it; and it finishes the restore through `TimeAbi::restore_finish`, with the acknowledgement when the host gates its input until then, which hands the deferred time work to the time daemon that the caller supervises. It returns the `Repaired` clone, with its generation ID, its online CPUs and memory, and when it acknowledged the restore, and never touches the fence. It adds no memory block twice, and reseeds and finishes at most once, so a failure at or after the reseed is final. Each failure is a `RepairError`, with its `RepairStage`, its `RepairCause`, and the repair's `RepairProgress`, and `RepairError::status` selects the status, of `nvx::protocol::status`, with which the caller ends the VM. The capture and the repair take the time ABI in strict mode, so that a violation of it fails them, with its status, rather than stopping the VM, and only the capture's preparation and the time daemon enforce it | +| `nvx::guest::clone` | x86_64: the preparation of a template's guest for a capture before its root's assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init or agent, whether or not it is PID 1. `CapturePreparation`, of the caller's `AdmissionFence` and the `Devices` that the template's clones replace, the `distro` and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag: `prepare` closes the fence, so that no execution starts, refuses with a `Mounted` the first mount from those devices, a distro partition included, or from an overlay, bind mounts included, in a bounded read of `/proc/self/mountinfo` of at most `MOUNT_TABLE_LIMIT` bytes, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as `nvx-time pre-capture` does, and returns an owned `CaptureBarrier`. `CaptureBarrier::capture` requests the capture with the platform tier's request, in the in-memory mode, through `TimeAbi::capture`, and returns `Captured::Continued` once it opened the fence and restored the detectors' settings, or, in a clone, `Captured::Restored`, with the fence still closed, since nothing else may run before repair acknowledges the restore, and the detectors suppressed until the time daemon restores their settings: `Restored::into_parts` hands the `Restore` that repair takes next, which holds the restore's `Seed`, and the closed fence to the caller, which keeps it closed through the repair and opens it once the clone may admit executions, such as once it is activated. `CaptureBarrier::cancel`, a dropped barrier, and a failure to read the VM generation ID open the fence and restore the detectors' settings; a capture request that fails restores the settings but leaves the fence closed, since it may fail in a clone whose restore it could not repair. Each failure is a `CaptureError`, with its `CaptureStage` and `CaptureCause`. `repair`, or a `Repair` that the caller may retry, repairs the clone of a `Restore` next: it checks that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online; with a memory target, it checks, before it adds any memory block, that the kernel's block size is 128 MiB and that the restore's ranges continue the captured RAM in logical order, with no gap, overlap, or range in the MMIO aperture, while the host bounds the target by the snapshot's capacity, then adds and onlines the blocks through sysfs, as `nvx-snapshot` does, and verifies them; it reseeds the kernel's random number generator with the `Seed`, having drawn no randomness before, and drops it; and it finishes the restore through `TimeAbi::restore_finish`, with the acknowledgement when the host gates its input until then, which hands the deferred time work to the time daemon that the caller supervises. It returns the `Repaired` clone, with its generation ID, its online CPUs and memory, and when it acknowledged the restore, and never touches the fence. It adds no memory block twice, and reseeds and finishes at most once, so a failure at or after the reseed is final. Each failure is a `RepairError`, with its `RepairStage`, its `RepairCause`, and the repair's `RepairProgress`, and `RepairError::status` selects the status, of `nvx::protocol::status`, with which the caller ends the VM. The capture and the repair take the time ABI in strict mode, so that a violation of it fails them, with its status, rather than stopping the VM, and only the capture's preparation and the time daemon enforce it. An `Activation` of the `Repaired` clone and of the caller's closed `AdmissionFence` then takes the clone's own configuration from the host, in one Activate, on a restricted control session: a state machine without I/O, independent of any framing, to which the adapter passes what its session receives, as an `ActivationInput`, and which returns the `ActivationAction` that the adapter takes. From its transport bootstrap, which takes the reset that the restore queued for the NVXS session, the restricted session takes only flow control and one Activate, and anything else, a request or another reset, fails it; the activation decodes the Activate's record with `ActivationRecord::decode`, and accepts only a clone's, with a restore section, that expects the CPU count and the memory size that the repair reports, before any side effect; the adapter applies the record, and `Activation::complete` then opens the fence and returns the `ReadyActivation` that Ready reports, that the activation applied, with the record's transaction ID, after which the host's mapping table and start follow in the adapter's order, as after a cold boot, and `Activation::start` records the start. A retry with the same transaction ID and a byte-equal record joins the running activation, or gets its cached result, also in a later session once the clone is ready; a changed record, another transaction, another reset before Ready, or a failed activation is final. Its deadline, which the caller chooses, and which neither input nor retries extend, starts at the restore's acknowledgement, and the adapter ticks it however its input trickles and its output blocks. Each failure is an `ActivationFailure`, with its `ActivationState` and `ActivationCause`, and never powers the VM off: the adapter ends it | | `nvx::guest::reseed` | `Seed`, `Seed::read`, and `reseed`: a restore's fresh entropy, which starts with its `GenerationId`, and the reseed of the kernel's random number generator with it, in the caller's process. A `Seed` keeps its bytes in one allocation, which no move copies, zeroes them with volatile writes when it is dropped, can't be cloned, and formats none of them. `nvx-reseed` runs them | | `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their roles' access modes and their expected sizes, and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already, once it has checked everything: the read-only layers of the `Image` that the caller names, which it never probes: an EROFS image, whose expected `nvx::uuid::Uuid` it checks in its superblock, or a GPT disk whose partition table it reads and checks with `nvx::guest::gpt::read_disk`, against the caller's `nvx::gpt::Expected` values, whose first partition it reads, up to the caller's bound, as opaque bytes that it never mounts, and whose later partitions, at least one, hold ext4 layers that it mounts `ro,nosuid,nodev` with `errors=remount-ro`, so that an error that the kernel finds in a layer fails the operation that found it, rather than panic the guest, as the layer's superblock may ask, from the block devices that the kernel created when it found the disk, once their superblocks show that no journal needs a recovery, which their read-only device can't take, partition 2's at the overlay's base and the last partition's at its top, under an overlay mounted `nosuid,nodev` with the one option set of the caller's `OverlayMode`, `metacopy=on,xino=on` or, for lowers that may carry untrusted overlay xattrs, `metacopy=off,redirect_dir=nofollow`, with no fallback, whose upper and work directories are on the `Upper`: an ext4 scratch mounted `nosuid,nodev`, whose superblock it checks for a journal that the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` at the kernel's default size. It refuses overlay options longer than `MOUNT_OPTIONS_LIMIT`, the most that a mount passes, which, below `/run/nvx`, fit up to 195 layers of a GPT disk, and an overlay that the kernel's mount table shows read-only or without the mode's options in effect, since the kernel mounts one so, rather than fail, when its upper can't hold its work directory or its extended attributes. The `Root` reports the first partition's bytes, its `Lower` layers, its `UpperKind`, and its `OverlayMode`. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::mount_export`: the host export, the aggregate device of `protocol::export`, one per root, mounted `nosuid,nodev` with the access of its `ExportMode` at `hostfs/root` in the runtime directory, out of the root, below a directory that only root may enter, whose root it returns as the `SourceRoot` of the caller's mappings. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, the export, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then the mounts of an aggregate and of the export, the later first, the overlay, the layers from the top down, and the upper, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off | | `nvx::guest::gpt` | `read_disk`: the partition table of a GPT disk, read and checked with `nvx::gpt`'s `Disk::read` through the disk's block device, which the caller opened, at the logical block size and size that `BLKSSZGET` and `BLKGETSIZE64` report, with the caller's `Expected` values, such as the disk GUID that a boot parameter names. It only reads the device, at explicit offsets, so it never moves the file's position. It neither compares the partitions that the kernel created with the table nor asks the kernel to reread the table. Each failure is a typed `DeviceError` | diff --git a/nvx-rs/src/guest/clone.rs b/nvx-rs/src/guest/clone.rs index ea27f73a8..37a8d7986 100644 --- a/nvx-rs/src/guest/clone.rs +++ b/nvx-rs/src/guest/clone.rs @@ -1,6 +1,6 @@ //! Templates and clones in the guest: the capture of a template's guest before its root's -//! assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init -//! or agent, whether or not it is PID 1. +//! assembly, and the repair and the activation of each clone that a restore of its snapshot +//! starts, for a guest init or agent, whether or not it is PID 1. //! //! A platform template is a VM whose snapshot holds no tenant's root, so that every clone that a //! restore of the snapshot starts can attach its own image, upper, and export. Its guest captures @@ -100,12 +100,53 @@ //! `nvx-time pre-capture` does, and so does the time daemon: a violation of either stops the VM, //! unless the ABI is report-only. //! +//! # Activation +//! +//! A repaired clone then takes its own configuration from the host, in one Activate, on a +//! restricted control session, through an [`Activation`] of the [`Repaired`] clone and of the +//! caller's fence, which stays closed until the activation is ready, so that no execution starts +//! before. The activation is a state machine without I/O, independent of any framing: the adapter +//! passes it what it receives on the clone's NVXS session, each an [`ActivationInput`], and does +//! what each [`ActivationAction`] that it returns says. Its [`ActivationState`]s, in order, are: +//! +//! 1. the transport bootstrap: the clone awaits the reset that the restore queued for the session +//! that the guest attached before the capture, which the adapter acknowledges; +//! 2. restricted: the session takes only flow control and one Activate, and anything else, a +//! request or another reset, fails the activation. The activation decodes the Activate's +//! record with [`ActivationRecord::decode`](crate::activation::ActivationRecord::decode), and +//! accepts it only as a clone's, with a restore section, that expects the CPU count and the +//! memory size that the repair reports, before anything has a side effect; +//! 3. activating: the adapter applies the accepted record, activating the root and the workload +//! with its settings, then reports the result to [`Activation::complete`]; +//! 4. ready, with the mappings pending: the activation completed, opened the fence, and returned +//! what Ready reports, a [`ReadyActivation`](crate::activation::ReadyActivation): that the +//! activation applied, with the record's transaction ID. The host's mapping table, which the +//! record announced, and the start of the sandbox follow in the adapter's order, as after a +//! cold boot; +//! 5. started: the adapter's order started the sandbox, which [`Activation::start`] records. +//! +//! The host retries an Activate with the same transaction ID and a byte-equal record, and such a +//! retry joins the activation while it runs, or gets its cached result afterwards, without +//! activating again, also in a later session, such as once the host's attachment ended, whose +//! reset a ready clone takes. A changed record, another transaction, another reset before the +//! clone is ready, and an activation that failed, wholly or partway, are final. The activation's +//! deadline, which the caller chooses, starts at the restore's acknowledgement, or, for a restore +//! that the host did not gate until one, when the caller makes the activation, and neither input +//! nor retries extend it: the adapter wakes by it, and [ticks](Activation::tick) the activation +//! then, however slowly its input trickles in, and whether or not its output drains. A failure +//! is an [`ActivationFailure`], which names the [`ActivationState`] in which the activation failed +//! and its [`ActivationCause`], and which every later call returns again. The activation never +//! powers the VM off: the adapter reports the failure, and ends the VM. +//! //! # Example //! //! ```no_run //! use std::io; +//! use std::time::{Duration, Instant}; //! -//! use nvx::guest::clone::{self, AdmissionFence, CapturePreparation, Captured, Devices}; +//! use nvx::guest::clone::{ +//! self, Activation, AdmissionFence, CapturePreparation, Captured, Devices, +//! }; //! use nvx::guest::root::Device; //! use nvx::protocol::root::{DISTRO_MMIO_BASE, SCRATCH_MMIO_BASE}; //! @@ -153,8 +194,11 @@ //! } //! }; //! println!("{} CPUs, {} bytes of memory", repaired.cpus(), repaired.memory()); -//! // The clone opens the fence once it may admit executions, such as once activated. -//! fence.open(); +//! // Its activation takes the host's Activate, and opens the fence once the clone is +//! // ready: the agent drives it with what its control session receives. +//! let timeout = Duration::from_secs(30); +//! let activation = Activation::new(&repaired, fence, timeout, Instant::now()); +//! println!("activate by {:?}", activation.deadline()); //! } //! } //! # Ok(()) @@ -173,8 +217,10 @@ //! [`TimeAbi::restore_finish`] tells. Apart from a violation of the time ABI in the capture's //! preparation, which stops the VM unless the ABI is report-only, as [strict //! mode](#strict-mode) tells, they never power the VM off, so a caller that is not PID 1 keeps -//! its own reaping, transport, and shutdown. +//! its own reaping, transport, and shutdown. The activation needs no privilege, and makes no +//! system call: of the clone, it changes only the caller's fence, which it opens. +mod activation; mod mounts; mod repair; #[cfg(test)] @@ -194,6 +240,10 @@ use super::time::{ CaptureOutcome, CaptureRequest, PreCaptureError, Restore, ScratchPolicy, TimeAbi, TimeError, }; +pub use self::activation::{ + Activation, ActivationAction, ActivationCause, ActivationFailure, ActivationInput, + ActivationState, +}; pub use self::mounts::{Mounted, Origin}; pub use self::repair::{ RangeFault, Repair, RepairCause, RepairError, RepairProgress, RepairStage, Repaired, repair, diff --git a/nvx-rs/src/guest/clone/activation.rs b/nvx-rs/src/guest/clone/activation.rs new file mode 100644 index 000000000..bc1c9547e --- /dev/null +++ b/nvx-rs/src/guest/clone/activation.rs @@ -0,0 +1,662 @@ +//! The activation of a repaired clone, in the caller's process: [`Activation`], the state machine +//! of the clone's restricted control session, through which the clone takes its configuration from +//! the host in one Activate, as the [module](super#activation) describes. + +#[cfg(test)] +mod tests; + +use std::error::Error; +use std::fmt; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use super::{AdmissionFence, Repaired}; +use crate::activation::{ + ActivationError, ActivationRecord, ExpectedResources, ReadyActivation, TransactionId, +}; + +/// The activation of a repaired clone: the state machine, without I/O and independent of any +/// framing, of the clone's control session, which takes the clone's configuration from the host +/// in one Activate, behind the caller's fence of workload admission, which it opens once the +/// activation is ready; see the [module](super#activation). +/// +/// The adapter passes it what it receives on the session, each an [`ActivationInput`], through +/// [`input`](Self::input), and does what the [`ActivationAction`] that it returns says. The +/// adapter applies the record of the Activate that the activation accepted, and reports the +/// result through [`complete`](Self::complete), and it reports the start of the sandbox, which +/// its own order takes once the activation is ready, through [`start`](Self::start). The +/// adapter's loop wakes by the [`deadline`](Self::deadline), and [ticks](Self::tick) the +/// activation then, whether input arrives or not, and whether its output drains or not. +/// +/// It goes through the [`ActivationState`]s in order, and each takes these inputs: +/// +/// | State | Reset | Flow | Activate | Request | +/// | --- | --- | --- | --- | --- | +/// | [`Bootstrap`](ActivationState::Bootstrap) | [`Acknowledge`](ActivationAction::Acknowledge), restricted next | [`Continue`](ActivationAction::Continue) | Fails | Fails | +/// | [`Restricted`](ActivationState::Restricted) | Fails | [`Continue`](ActivationAction::Continue) | [`Apply`](ActivationAction::Apply), activating next, once its record is valid | Fails | +/// | [`Activating`](ActivationState::Activating) | Fails | [`Continue`](ActivationAction::Continue) | [`Join`](ActivationAction::Join) if byte-equal | Fails | +/// | [`Ready`](ActivationState::Ready), [`Started`](ActivationState::Started) | [`Acknowledge`](ActivationAction::Acknowledge), a later session | [`Continue`](ActivationAction::Continue) | [`Ready`](ActivationAction::Ready) if byte-equal | [`Serve`](ActivationAction::Serve) | +/// +/// Every failure is final, and leaves the activation [failed](ActivationState::Failed): every +/// later call returns the same [`ActivationFailure`]. A failure before the activation is ready +/// leaves the fence closed, and one after it leaves the fence open. The activation never powers +/// the VM off: the adapter reports the failure and ends the VM. +/// +/// # Example +/// +/// ```no_run +/// use std::error::Error; +/// use std::time::Instant; +/// +/// use nvx::activation::{ActivationRecord, ReadyActivation}; +/// use nvx::guest::clone::{ +/// Activation, ActivationAction, ActivationFailure, ActivationInput, AdmissionFence, +/// }; +/// +/// /// What the agent's control session received from the host, in the agent's own framing. +/// enum Received { +/// /// The session's NVXS RESET. +/// Reset, +/// /// Bytes that complete no message, such as a part of a DATA record's payload. +/// Partial, +/// /// The host's Activate, with its record's bytes. +/// Activate(Vec), +/// /// Any other message of the host, such as a part of the mapping table, or the start. +/// Other(Vec), +/// } +/// +/// /// The agent of a clone, which serves its control session in its own framing. +/// trait Agent { +/// /// Waits until the host sends something, or until `deadline`, and returns what it sent. +/// fn receive(&mut self, deadline: Option) -> Option; +/// /// Acknowledges the session's reset, and abandons the earlier epoch's work. +/// fn acknowledge(&mut self); +/// /// Activates the root and the workload with the settings of `record`. +/// fn apply(&mut self, record: &ActivationRecord) -> Result<(), Box>; +/// /// Sends Ready, which reports `activation`. +/// fn ready(&mut self, activation: ReadyActivation); +/// /// Serves `message` in the agent's order, as after a cold boot, and returns whether the +/// /// host started the sandbox with it. +/// fn serve(&mut self, message: Vec) -> bool; +/// } +/// +/// /// Serves the control session of the clone whose activation is `activation`, until it fails. +/// fn serve( +/// agent: &mut impl Agent, +/// mut activation: Activation, +/// ) -> Result<(), ActivationFailure> { +/// loop { +/// let Some(received) = agent.receive(activation.deadline()) else { +/// // The deadline passed, unless the clone is ready, which ends the activation. +/// activation.tick(Instant::now())?; +/// continue; +/// }; +/// let input = match &received { +/// Received::Reset => ActivationInput::Reset, +/// Received::Partial => ActivationInput::Flow, +/// Received::Activate(record) => ActivationInput::Activate(record), +/// Received::Other(_) => ActivationInput::Request, +/// }; +/// match activation.input(Instant::now(), input)? { +/// ActivationAction::Acknowledge => agent.acknowledge(), +/// ActivationAction::Continue | ActivationAction::Join => {} +/// ActivationAction::Apply(record) => { +/// // The record is valid, and nothing happened before: the activation's side +/// // effects start here, and the fence opens once they all succeeded. +/// let result = agent.apply(&record); +/// let ready = activation.complete(Instant::now(), result)?; +/// agent.ready(ready); +/// } +/// ActivationAction::Ready(ready) => agent.ready(ready), +/// // The activation serves only another message, once it is ready. +/// ActivationAction::Serve => { +/// if let Received::Other(message) = received +/// && agent.serve(message) +/// { +/// activation.start()?; +/// } +/// } +/// } +/// } +/// } +/// ``` +pub struct Activation { + /// The caller's fence of workload admission, closed until the activation is ready. + fence: F, + /// The resources that the clone has, as its repair reports them, which the record of an + /// Activate must expect. + resources: ExpectedResources, + /// When the activation fails unless it is ready, or `None` if no instant is that late. + deadline: Option, + /// Where the activation is. + state: State, +} + +impl Activation { + /// Returns the activation of the clone that `repaired` describes, in its + /// [transport bootstrap](ActivationState::Bootstrap), behind the caller's `fence`, which + /// [`Restored::into_parts`](super::Restored::into_parts) handed over closed, and which the + /// activation opens once it is ready. + /// + /// The activation must be ready by its deadline: `timeout`, which the caller chooses, after + /// the repair acknowledged the restore, as [`Repaired::acknowledged_at`] tells, or after `now` + /// for a restore that the host did not gate until an acknowledgement, whose repair + /// acknowledged nothing. No input or retry extends it. A deadline that no instant can hold + /// never passes. + pub fn new(repaired: &Repaired, fence: F, timeout: Duration, now: Instant) -> Self { + let resources = ExpectedResources { + cpus: repaired.cpus(), + memory: repaired.memory(), + }; + Self::of(resources, repaired.acknowledged_at(), fence, timeout, now) + } + + /// Returns the activation of a clone that has `resources`, and whose repair acknowledged its + /// restore at `acknowledged_at`, if it did, as [`new`](Self::new) does. + fn of( + resources: ExpectedResources, + acknowledged_at: Option, + fence: F, + timeout: Duration, + now: Instant, + ) -> Self { + let start = acknowledged_at.unwrap_or(now); + Self { + fence, + resources, + deadline: start.checked_add(timeout), + state: State::Bootstrap, + } + } + + /// Takes `input`, which the adapter received at `now` on the clone's control session, and + /// returns what the adapter does next, as the [type](Self) tabulates it, or fails the + /// activation. + /// + /// Until the activation is ready, it fails once `now` reaches its deadline, whatever the + /// input, with [`ActivationCause::Expired`]. In order, it then takes: + /// + /// 1. In the transport bootstrap, the session's reset, which the restore queued: the adapter + /// acknowledges it, and the session is restricted next. Before it, the host can send + /// nothing, so an Activate or another request fails with [`ActivationCause::Request`]. + /// 2. While restricted, flow control, and one Activate. The activation decodes its record with + /// [`ActivationRecord::decode`], which refuses it with [`ActivationCause::Record`] unless it + /// is canonical and keeps every limit, checks that it is a clone's, with a restore + /// section, or refuses it with [`ActivationCause::Cold`], and checks that it expects the + /// resources that the clone has, the CPU count and the memory size that its repair + /// reports, or refuses it with [`ActivationCause::Resources`]. Only then does it accept it, + /// and hand the record to the adapter with [`ActivationAction::Apply`]: nothing has a side + /// effect before. Any other request fails with [`ActivationCause::Request`], and another + /// reset with [`ActivationCause::Reset`]. + /// 3. While activating, flow control, and retries of the Activate: one of the same + /// transaction, with a byte-equal record, joins the activation that runs. Any other + /// Activate fails: one whose record does not decode, or is a cold boot's, with the cause + /// with which the first would have failed, one of another transaction with + /// [`ActivationCause::Transaction`], and any other of the same transaction with + /// [`ActivationCause::Changed`]. A request or a reset fails as while restricted. + /// 4. Once ready, or started, everything: a reset starts a later session, which the adapter + /// acknowledges, a byte-equal retry of the Activate gets the cached result, what Ready + /// reports, without activating again, and any other Activate fails as while activating. + /// Every other request goes to the adapter's order, as after a cold boot. + pub fn input( + &mut self, + now: Instant, + input: ActivationInput<'_>, + ) -> Result { + self.check(now)?; + match (self.state.name(), input) { + (_, ActivationInput::Flow) => Ok(ActivationAction::Continue), + (ActivationState::Bootstrap, ActivationInput::Reset) => { + self.state = State::Restricted; + Ok(ActivationAction::Acknowledge) + } + (ActivationState::Ready | ActivationState::Started, ActivationInput::Reset) => { + Ok(ActivationAction::Acknowledge) + } + (_, ActivationInput::Reset) => Err(self.fail(ActivationCause::Reset)), + (ActivationState::Restricted, ActivationInput::Activate(record)) => self.accept(record), + ( + ActivationState::Activating | ActivationState::Ready | ActivationState::Started, + ActivationInput::Activate(record), + ) => self.retry(record), + (ActivationState::Ready | ActivationState::Started, ActivationInput::Request) => { + Ok(ActivationAction::Serve) + } + (_, ActivationInput::Activate(_) | ActivationInput::Request) => { + Err(self.fail(ActivationCause::Request)) + } + } + } + + /// Completes the activation of the accepted Activate's record at `now`, with the `result` of + /// the adapter's application of the record, and returns what Ready reports: that the + /// activation applied, with the record's transaction ID. + /// + /// A result that succeeded by the deadline makes the activation ready: it opens the fence, + /// which no input before opened, and the adapter reports Ready. A failure, which may have + /// applied part of the record, is final, with [`ActivationCause::Failed`], so that no retry + /// activates the record again, and the fence stays closed. Past the deadline, the activation + /// fails with [`ActivationCause::Expired`] instead, whatever the result. Without an activation + /// that runs, it fails with [`ActivationCause::Completion`]. + pub fn complete( + &mut self, + now: Instant, + result: Result<(), Box>, + ) -> Result { + self.check(now)?; + let accepted = match &self.state { + State::Activating(accepted) => accepted.clone(), + _ => return Err(self.fail(ActivationCause::Completion)), + }; + if let Err(error) = result { + return Err(self.fail(ActivationCause::Failed(error))); + } + let ready = accepted.ready; + self.state = State::Ready(accepted); + self.fence.open(); + Ok(ready) + } + + /// Records that the adapter's order started the sandbox, once the activation is ready: the + /// host's start, after the mapping table that the record announced, as after a cold boot. + /// + /// A start before the activation is ready, or a second one, fails with + /// [`ActivationCause::Start`]. + pub fn start(&mut self) -> Result<(), ActivationFailure> { + let accepted = match &self.state { + State::Failed(failure) => return Err(failure.clone()), + State::Ready(accepted) => accepted.clone(), + _ => return Err(self.fail(ActivationCause::Start)), + }; + self.state = State::Started(accepted); + Ok(()) + } + + /// Fails the activation with [`ActivationCause::Expired`] once `now` reaches its deadline, + /// unless it is ready, as each input does, so that the adapter wakes by the + /// [deadline](Self::deadline) and ticks the activation then, whether input arrives or not. + pub fn tick(&mut self, now: Instant) -> Result<(), ActivationFailure> { + self.check(now) + } + + /// Returns when the activation fails unless it is ready, for the adapter to + /// [tick](Self::tick) it then, or `None` once it is ready or failed, or if no instant is that + /// late. + pub fn deadline(&self) -> Option { + match self.state { + State::Bootstrap | State::Restricted | State::Activating(_) => self.deadline, + State::Ready(_) | State::Started(_) | State::Failed(_) => None, + } + } + + /// Returns where the activation is. + pub fn state(&self) -> ActivationState { + self.state.name() + } + + /// Returns what Ready reports about the established activation, once it is ready: that the + /// activation applied, with the record's transaction ID, which a later session gets without + /// activating again. + pub fn established(&self) -> Option { + match &self.state { + State::Ready(accepted) | State::Started(accepted) => Some(accepted.ready), + _ => None, + } + } + + /// Accepts the host's first Activate, whose record's bytes are `record`, once it decoded and + /// validated it: a clone's record, with a restore section, that expects the clone's resources. + fn accept(&mut self, record: &[u8]) -> Result { + let decoded = match ActivationRecord::decode(record) { + Ok(decoded) => decoded, + Err(error) => return Err(self.fail(ActivationCause::Record(error))), + }; + let Some(restore) = decoded.restore() else { + return Err(self.fail(ActivationCause::Cold)); + }; + let expected = decoded.configuration().resources; + if expected != self.resources { + let actual = self.resources; + return Err(self.fail(ActivationCause::Resources { expected, actual })); + } + self.state = State::Activating(Accepted { + record: record.into(), + transaction: restore.transaction, + ready: ReadyActivation::of(&decoded), + }); + Ok(ActivationAction::Apply(Box::new(decoded))) + } + + /// Takes a retry of the accepted Activate, whose record's bytes are `record`: a byte-equal + /// record joins the activation that runs, or gets its cached result once it is ready, and + /// any other fails the activation. + fn retry(&mut self, record: &[u8]) -> Result { + let (accepted, ready) = match &self.state { + State::Activating(accepted) => (accepted, false), + State::Ready(accepted) | State::Started(accepted) => (accepted, true), + _ => return Err(self.fail(ActivationCause::Request)), + }; + if *accepted.record == *record { + return Ok(if ready { + ActivationAction::Ready(accepted.ready) + } else { + ActivationAction::Join + }); + } + let cause = accepted.refusal(record); + Err(self.fail(cause)) + } + + /// Fails the activation with the failure that it already had, or with + /// [`ActivationCause::Expired`] once `now` reaches its deadline, unless it is ready. + fn check(&mut self, now: Instant) -> Result<(), ActivationFailure> { + if let State::Failed(failure) = &self.state { + return Err(failure.clone()); + } + if self.deadline().is_some_and(|deadline| now >= deadline) { + return Err(self.fail(ActivationCause::Expired)); + } + Ok(()) + } + + /// Fails the activation in its state, because of `cause`, and returns the failure, unless it + /// failed already, which it then returns. + fn fail(&mut self, cause: ActivationCause) -> ActivationFailure { + if let State::Failed(failure) = &self.state { + return failure.clone(); + } + let failure = ActivationFailure { + state: self.state.name(), + cause: Arc::new(cause), + }; + self.state = State::Failed(failure.clone()); + failure + } +} + +impl fmt::Debug for Activation { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("Activation") + .field("state", &self.state.name()) + .field("resources", &self.resources) + .field("deadline", &self.deadline) + .finish_non_exhaustive() + } +} + +/// What the adapter received on a clone's control session, in the terms of its [`Activation`], +/// whatever its framing: the session's reset, flow control, the host's Activate, or another +/// request of the host. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ActivationInput<'a> { + /// The NVXS session's reset, a [`GuestEvent::Reset`](crate::protocol::nvxs::GuestEvent::Reset), + /// which starts a new epoch: the one that the restore queued for the session that the guest + /// attached before the capture, or a later one, such as once the host's attachment ended. + Reset, + /// Flow control, which carries no request: bytes of a message that they do not complete, such + /// as a DATA record's payload, or a credit that the adapter returned to its receive window. + Flow, + /// The host's Activate, with the bytes of its activation record, the canonical encoding of an + /// [`ActivationRecord`]. + Activate(&'a [u8]), + /// Any other request of the host, such as a part of the mapping table, the start of the + /// sandbox, an execution, or a probe of the adapter's framing. + Request, +} + +/// What the adapter does next, which [`Activation::input`] returns. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ActivationAction { + /// The activation took the session's reset: the adapter acknowledges it with its receive + /// window, unless its transport acknowledged it already, and abandons the work of the earlier + /// epoch. + Acknowledge, + /// The activation took the input, which needs nothing more of the adapter. + Continue, + /// The activation accepted the host's Activate, whose record decoded and validated, and + /// which nothing before had any side effect of: the adapter applies the record, activating + /// the root and the workload with its settings, then reports the result to + /// [`Activation::complete`]. + Apply(Box), + /// A retry of the accepted Activate joined the activation that runs: what + /// [`Activation::complete`] returns answers it too, as the adapter's framing answers each + /// request. + Join, + /// A retry of the accepted Activate, once the activation is ready: the adapter answers it with + /// Ready, which reports this cached result, without activating again. + Ready(ReadyActivation), + /// A request once the activation is ready: the adapter serves it in its order, as after a + /// cold boot. + Serve, +} + +/// The state of an [`Activation`], in the order in which it goes through them, which an +/// [`ActivationFailure`] names. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum ActivationState { + /// The transport bootstrap: the clone awaits the reset of its NVXS session that the restore + /// queued, which the adapter acknowledges. + Bootstrap, + /// The restricted session: it takes only flow control and one Activate. + Restricted, + /// The adapter applies the accepted Activate's record. + Activating, + /// Ready, with the mappings pending: the activation completed, which opened the fence, and + /// returned what Ready reports. The host's mapping table, which the record announced, and the + /// start of the sandbox follow in the adapter's order, as after a cold boot. + Ready, + /// The adapter's order started the sandbox. + Started, + /// The activation failed, finally. + Failed, +} + +impl fmt::Display for ActivationState { + /// Writes the state's name, such as `restricted`. + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::Bootstrap => "transport bootstrap", + Self::Restricted => "restricted", + Self::Activating => "activating", + Self::Ready => "ready", + Self::Started => "started", + Self::Failed => "failed", + }) + } +} + +/// Why an [`Activation`] failed, which an [`ActivationFailure`] holds. +#[derive(Debug)] +#[non_exhaustive] +pub enum ActivationCause { + /// The deadline passed before the activation was ready. + Expired, + /// The session was reset after its bootstrap, before the activation was ready. + Reset, + /// The host sent a request that the state does not take, before the activation was ready: + /// any before the session's bootstrap, and any but one Activate, and its retries, after it. + Request, + /// The Activate's record does not decode, or breaks a limit of `nvx::activation`, which + /// refused it with this error. + Record(ActivationError), + /// The Activate's record is a cold boot's, without the restore section of a clone's. + Cold, + /// The Activate's record expects other resources than the clone has. + Resources { + /// The CPU count and the memory size that the record expects. + expected: ExpectedResources, + /// The CPU count and the memory size that the clone has, as its repair reports them: the + /// number of its online CPUs, and the size of its online memory blocks. + actual: ExpectedResources, + }, + /// The Activate's record names another transaction than the accepted Activate's. + Transaction { + /// The transaction of the accepted Activate. + accepted: TransactionId, + /// The transaction that the retry names. + offered: TransactionId, + }, + /// The Activate's record names the accepted Activate's transaction, but is not byte-equal to + /// its record. + Changed, + /// The adapter's application of the accepted record failed, wholly or partway, with this + /// error. + Failed(Box), + /// The adapter completed an activation that did not run: before the activation accepted an + /// Activate, or once it completed one. + Completion, + /// The adapter's order started the sandbox before the activation was ready, or again. + Start, +} + +impl fmt::Display for ActivationCause { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Expired => { + formatter.write_str("the deadline passed before the activation was ready") + } + Self::Reset => { + formatter.write_str("the session was reset again before the activation was ready") + } + Self::Request => { + formatter.write_str("the host sent a request before the activation was ready") + } + Self::Record(error) => write!(formatter, "the Activate's record is refused: {error}"), + Self::Cold => formatter + .write_str("the Activate's record is a cold boot's, without a restore section"), + Self::Resources { expected, actual } => write!( + formatter, + "the Activate's record expects {} CPUs and {} bytes of memory, and the clone has \ + {} CPUs and {} bytes", + expected.cpus, expected.memory, actual.cpus, actual.memory + ), + Self::Transaction { accepted, offered } => write!( + formatter, + "the Activate names the transaction {offered}, after an Activate of the \ + transaction {accepted}" + ), + Self::Changed => formatter.write_str( + "the Activate changes the record of the accepted Activate's transaction", + ), + Self::Failed(error) => write!(formatter, "the record's application failed: {error}"), + Self::Completion => formatter.write_str("an activation that did not run completed"), + Self::Start => { + formatter.write_str("the sandbox started before the activation was ready, or again") + } + } + } +} + +/// Why an [`Activation`] failed, finally: the state in which it failed, and its cause. +/// +/// Its `Display` form names both, such as `the activation failed in its restricted state: the +/// host sent a request before the activation was ready`. An activation keeps its failure, which +/// every later call returns again. The failure has no exit status: the adapter that ends the VM +/// documents its own. +#[derive(Clone, Debug)] +pub struct ActivationFailure { + /// The state in which the activation failed. + state: ActivationState, + /// Why it failed. + cause: Arc, +} + +impl ActivationFailure { + /// Returns the state in which the activation failed, which is never + /// [`ActivationState::Failed`]. + pub fn state(&self) -> ActivationState { + self.state + } + + /// Returns why it failed. + pub fn cause(&self) -> &ActivationCause { + &self.cause + } +} + +impl fmt::Display for ActivationFailure { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + formatter, + "the activation failed in its {} state: {}", + self.state, self.cause + ) + } +} + +impl Error for ActivationFailure { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self.cause.as_ref() { + ActivationCause::Record(error) => Some(error), + ActivationCause::Failed(error) => Some(error.as_ref()), + _ => None, + } + } +} + +/// Where an [`Activation`] is, with the Activate that it accepted, from the activating state on. +#[derive(Clone, Debug)] +enum State { + /// The transport bootstrap. + Bootstrap, + /// The restricted session. + Restricted, + /// The adapter applies the accepted Activate's record. + Activating(Accepted), + /// Ready, with the mappings pending. + Ready(Accepted), + /// The adapter's order started the sandbox. + Started(Accepted), + /// The activation failed, finally, with this failure. + Failed(ActivationFailure), +} + +impl State { + /// Returns the state's name. + fn name(&self) -> ActivationState { + match self { + Self::Bootstrap => ActivationState::Bootstrap, + Self::Restricted => ActivationState::Restricted, + Self::Activating(_) => ActivationState::Activating, + Self::Ready(_) => ActivationState::Ready, + Self::Started(_) => ActivationState::Started, + Self::Failed(_) => ActivationState::Failed, + } + } +} + +/// The Activate that an [`Activation`] accepted, against which it checks each retry. +#[derive(Clone, Debug)] +struct Accepted { + /// The bytes of its record, to which a retry's must be equal. + record: Box<[u8]>, + /// The transaction that its record names. + transaction: TransactionId, + /// What Ready reports about its activation. + ready: ReadyActivation, +} + +impl Accepted { + /// Returns why a retry whose record's bytes are `record`, which are not the accepted record's, + /// is refused: its record does not decode, or is a cold boot's, as the first Activate's would + /// have been refused, or it names another transaction, or it changes the record of the same + /// one. + fn refusal(&self, record: &[u8]) -> ActivationCause { + let decoded = match ActivationRecord::decode(record) { + Ok(decoded) => decoded, + Err(error) => return ActivationCause::Record(error), + }; + match decoded.restore() { + None => ActivationCause::Cold, + Some(restore) if restore.transaction != self.transaction => { + ActivationCause::Transaction { + accepted: self.transaction, + offered: restore.transaction, + } + } + Some(_) => ActivationCause::Changed, + } + } +} diff --git a/nvx-rs/src/guest/clone/activation/tests.rs b/nvx-rs/src/guest/clone/activation/tests.rs new file mode 100644 index 000000000..af51896bd --- /dev/null +++ b/nvx-rs/src/guest/clone/activation/tests.rs @@ -0,0 +1,977 @@ +//! Tests of a clone's activation, with a fake fence that counts how often it opened, records that +//! each test builds with `nvx::activation`, a fake clock, whose instants each test takes after a +//! base instant, and a fake adapter of a framing of the tests' own, whose transport delivers the +//! host's bytes when and as a test chooses, and whose output takes nothing. + +use std::cell::Cell; +use std::io; +use std::num::NonZeroU64; +use std::rc::Rc; + +use super::*; +use crate::activation::{ + Configuration, Identity, Image, MAX_RECORD_BYTES, MachineIdMode, OverlayMode, RestoreSection, + StoreGeneration, TemplateReference, Upper, WorkloadView, +}; + +/// The number of online CPUs of the clone. +const CPUS: u32 = 2; +/// The memory size of the clone: eight online memory blocks of 128 MiB. +const MEMORY: u64 = 8 << 27; +/// How long the activation may take after the restore's acknowledgement. +const TIMEOUT: Duration = Duration::from_secs(10); +/// The transaction of the host's Activate. +const TRANSACTION: TransactionId = TransactionId::from_bytes([0x22; TransactionId::LEN]); +/// The transaction of another restore. +const OTHER_TRANSACTION: TransactionId = TransactionId::from_bytes([0x33; TransactionId::LEN]); +/// The kind of the host's Activate in the tests' framing. +const ACTIVATE: u8 = 1; +/// The kind of any other request of the host in the tests' framing. +const PING: u8 = 2; + +/// Returns the activation, behind a fence, in a state that each such function chooses, that it +/// reached at an instant. +type Setup = fn(&FakeFence, Instant) -> Activation; + +/// Returns whether a failure's cause is the one that a test expects. +type Expected = fn(&ActivationCause) -> bool; + +/// A fence that a restore handed over closed, which counts how often it opened, and which a test +/// shares with the activation that holds it. +#[derive(Clone, Default)] +struct FakeFence { + /// How often the fence opened. + opened: Rc>, +} + +impl FakeFence { + /// Returns how often the fence opened. + fn opened(&self) -> u32 { + self.opened.get() + } +} + +impl AdmissionFence for FakeFence { + fn close(&mut self) -> io::Result<()> { + unreachable!("an activation never closes the fence"); + } + + fn open(&mut self) { + self.opened.set(self.opened.get() + 1); + } +} + +/// Returns the instant `seconds` seconds after `base`. +fn after(base: Instant, seconds: u64) -> Instant { + base + Duration::from_secs(seconds) +} + +/// Returns the resources that the clone has, as its repair reports them. +fn resources() -> ExpectedResources { + ExpectedResources { + cpus: CPUS, + memory: MEMORY, + } +} + +/// Returns the configuration of a root over a tmpfs upper, named `hostname`, that expects +/// `resources`. +fn configuration(hostname: &[u8], resources: ExpectedResources) -> Configuration { + Configuration { + image: Image::Erofs { + binding: Identity::from_bytes([0x11; Identity::LEN]), + uuid: None, + }, + upper: Upper::Tmpfs, + overlay: OverlayMode::UntrustedLowers, + export: None, + mappings: 0, + network: None, + workload: WorkloadView { + hostname: hostname.to_vec(), + machine_id: MachineIdMode::Image, + identity: None, + }, + resources, + } +} + +/// Returns the record of a clone's restore of `transaction`, of a root named `hostname` that +/// expects `resources`. +fn clone_record( + transaction: TransactionId, + hostname: &[u8], + resources: ExpectedResources, +) -> ActivationRecord { + let restore = RestoreSection { + transaction, + template: TemplateReference { + generation: StoreGeneration::new(NonZeroU64::MIN), + key: Identity::from_bytes([0x44; Identity::LEN]), + }, + }; + ActivationRecord::new(Some(restore), configuration(hostname, resources)).unwrap() +} + +/// Returns the record of the host's Activate: the clone's restore of [`TRANSACTION`], which +/// expects the clone's resources. +fn accepted() -> ActivationRecord { + clone_record(TRANSACTION, b"clone", resources()) +} + +/// Returns the bytes of the record of the host's Activate. +fn record() -> Vec { + accepted().encode() +} + +/// Returns what Ready reports about the activation of the host's Activate. +fn reported() -> ReadyActivation { + ReadyActivation::of(&accepted()) +} + +/// Returns the activation, in its transport bootstrap, of the clone behind `fence`, whose repair +/// acknowledged its restore at `acknowledged`. +fn bootstrap(fence: &FakeFence, acknowledged: Instant) -> Activation { + Activation::of( + resources(), + Some(acknowledged), + fence.clone(), + TIMEOUT, + acknowledged, + ) +} + +/// Returns the activation of [`bootstrap`] once the session's reset at `at` restricted it. +fn restricted(fence: &FakeFence, at: Instant) -> Activation { + let mut activation = bootstrap(fence, at); + let action = activation.input(at, ActivationInput::Reset).unwrap(); + assert_eq!(action, ActivationAction::Acknowledge); + assert_eq!(activation.state(), ActivationState::Restricted); + activation +} + +/// Returns the activation of [`restricted`] once it accepted the host's Activate at `at`. +fn activating(fence: &FakeFence, at: Instant) -> Activation { + let mut activation = restricted(fence, at); + let action = activation + .input(at, ActivationInput::Activate(&record())) + .unwrap(); + assert_eq!(action, ActivationAction::Apply(Box::new(accepted()))); + assert_eq!(activation.state(), ActivationState::Activating); + activation +} + +/// Returns the activation of [`activating`] once its application of the record completed at +/// `at`. +fn ready(fence: &FakeFence, at: Instant) -> Activation { + let mut activation = activating(fence, at); + assert_eq!(activation.complete(at, Ok(())).unwrap(), reported()); + assert_eq!(activation.state(), ActivationState::Ready); + activation +} + +/// Returns the activation of [`ready`] once the adapter's order started the sandbox. +fn started(fence: &FakeFence, at: Instant) -> Activation { + let mut activation = ready(fence, at); + activation.start().unwrap(); + assert_eq!(activation.state(), ActivationState::Started); + activation +} + +/// Checks that `activation`, behind `fence`, failed finally with `failure` at `now`: it has no +/// deadline and no established activation, every later call returns the same failure, and the +/// fence opened `opened` times, as it had before the failure. +fn assert_final( + activation: &mut Activation, + fence: &FakeFence, + failure: &ActivationFailure, + now: Instant, + opened: u32, +) { + assert_ne!(failure.state(), ActivationState::Failed); + assert_eq!(activation.state(), ActivationState::Failed); + assert_eq!(activation.deadline(), None); + assert_eq!(activation.established(), None); + let record = record(); + let later = [ + activation.input(now, ActivationInput::Flow).unwrap_err(), + activation.input(now, ActivationInput::Reset).unwrap_err(), + activation + .input(now, ActivationInput::Activate(&record)) + .unwrap_err(), + activation.input(now, ActivationInput::Request).unwrap_err(), + activation.complete(now, Ok(())).unwrap_err(), + activation.start().unwrap_err(), + activation.tick(now).unwrap_err(), + ]; + for later in later { + assert_eq!(later.state(), failure.state()); + assert!(Arc::ptr_eq(&later.cause, &failure.cause)); + } + assert_eq!(fence.opened(), opened); +} + +#[test] +fn a_clone_bootstraps_takes_one_activate_and_opens_the_fence_once_ready() { + let t0 = Instant::now(); + let fence = FakeFence::default(); + let mut activation = bootstrap(&fence, t0); + assert_eq!(activation.state(), ActivationState::Bootstrap); + assert_eq!(activation.deadline(), Some(after(t0, 10))); + // Bytes of the session's reset arrive before it completes, and then the reset. + let flow = activation.input(after(t0, 1), ActivationInput::Flow); + assert_eq!(flow.unwrap(), ActivationAction::Continue); + let reset = activation.input(after(t0, 1), ActivationInput::Reset); + assert_eq!(reset.unwrap(), ActivationAction::Acknowledge); + assert_eq!(activation.state(), ActivationState::Restricted); + let flow = activation.input(after(t0, 2), ActivationInput::Flow); + assert_eq!(flow.unwrap(), ActivationAction::Continue); + // The Activate's record, which the activation decoded, goes to the adapter, which applies it. + let action = activation.input(after(t0, 2), ActivationInput::Activate(&record())); + assert_eq!( + action.unwrap(), + ActivationAction::Apply(Box::new(accepted())) + ); + assert_eq!(activation.state(), ActivationState::Activating); + assert_eq!(activation.deadline(), Some(after(t0, 10))); + assert_eq!(activation.established(), None); + assert_eq!(fence.opened(), 0); + let flow = activation.input(after(t0, 3), ActivationInput::Flow); + assert_eq!(flow.unwrap(), ActivationAction::Continue); + // Once the application succeeded, the fence opens, and Ready reports the activation, with the + // record's transaction. + let ready = activation.complete(after(t0, 4), Ok(())).unwrap(); + assert_eq!(ready, reported()); + assert_eq!(ready.transaction, Some(TRANSACTION)); + assert_eq!(fence.opened(), 1); + assert_eq!(activation.state(), ActivationState::Ready); + assert_eq!(activation.established(), Some(reported())); + // No deadline holds a ready clone: the host's mapping table and its start go to the adapter's + // order, whenever they come. + assert_eq!(activation.deadline(), None); + activation.tick(after(t0, 3600)).unwrap(); + let request = activation.input(after(t0, 3600), ActivationInput::Request); + assert_eq!(request.unwrap(), ActivationAction::Serve); + activation.start().unwrap(); + assert_eq!(activation.state(), ActivationState::Started); + assert_eq!(activation.established(), Some(reported())); + let request = activation.input(after(t0, 7200), ActivationInput::Request); + assert_eq!(request.unwrap(), ActivationAction::Serve); + let flow = activation.input(after(t0, 7200), ActivationInput::Flow); + assert_eq!(flow.unwrap(), ActivationAction::Continue); + assert_eq!(fence.opened(), 1); +} + +#[test] +fn no_request_is_accepted_before_activate() { + let t0 = Instant::now(); + let record = record(); + // Before the session's bootstrap, the host can send nothing, Activate included; while the + // session is restricted, and while the activation runs, nothing but the Activate. + let cases: [(Setup, ActivationInput<'_>); 4] = [ + (bootstrap, ActivationInput::Request), + (bootstrap, ActivationInput::Activate(&record)), + (restricted, ActivationInput::Request), + (activating, ActivationInput::Request), + ]; + for (activation, input) in cases { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + let failure = activation.input(after(t0, 1), input).unwrap_err(); + assert_eq!(failure.state(), state, "{input:?}"); + assert!(matches!(failure.cause(), ActivationCause::Request)); + assert_final(&mut activation, &fence, &failure, after(t0, 2), 0); + } +} + +#[test] +fn a_reset_before_ready_is_final_and_one_after_it_starts_a_later_session() { + let t0 = Instant::now(); + // Only the reset that the restore queued bootstraps the session: another, such as once the + // host's attachment ended, fails the activation before it is ready. + for activation in [restricted, activating] { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + let failure = activation + .input(after(t0, 1), ActivationInput::Reset) + .unwrap_err(); + assert_eq!(failure.state(), state); + assert!(matches!(failure.cause(), ActivationCause::Reset)); + assert_final(&mut activation, &fence, &failure, after(t0, 1), 0); + } + // Once the clone is ready, or started, a reset starts a later session, which gets the + // established activation and transaction without activating again. + for activation in [ready, started] { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + for session in 1..=3 { + let now = after(t0, 60 * session); + let reset = activation.input(now, ActivationInput::Reset); + assert_eq!(reset.unwrap(), ActivationAction::Acknowledge); + assert_eq!(activation.state(), state); + assert_eq!(activation.established(), Some(reported())); + let retry = activation.input(now, ActivationInput::Activate(&record())); + assert_eq!(retry.unwrap(), ActivationAction::Ready(reported())); + let request = activation.input(now, ActivationInput::Request); + assert_eq!(request.unwrap(), ActivationAction::Serve); + } + assert_eq!(fence.opened(), 1); + } +} + +#[test] +fn retries_join_the_running_activation_and_get_its_cached_result() { + let t0 = Instant::now(); + let fence = FakeFence::default(); + let mut activation = activating(&fence, t0); + // A retry of the same transaction, with a byte-equal record, joins the activation that runs, + // whose deadline stays where it was. + for second in 1..=3 { + let retry = activation.input(after(t0, second), ActivationInput::Activate(&record())); + assert_eq!(retry.unwrap(), ActivationAction::Join); + assert_eq!(activation.state(), ActivationState::Activating); + assert_eq!(activation.deadline(), Some(after(t0, 10))); + } + assert_eq!(fence.opened(), 0); + assert_eq!( + activation.complete(after(t0, 4), Ok(())).unwrap(), + reported() + ); + // Afterwards, one gets the cached result, before and after the start, without applying the + // record again or opening the fence again. + let retry = activation.input(after(t0, 5), ActivationInput::Activate(&record())); + assert_eq!(retry.unwrap(), ActivationAction::Ready(reported())); + activation.start().unwrap(); + let retry = activation.input(after(t0, 6), ActivationInput::Activate(&record())); + assert_eq!(retry.unwrap(), ActivationAction::Ready(reported())); + assert_eq!(activation.state(), ActivationState::Started); + assert_eq!(fence.opened(), 1); +} + +#[test] +fn a_changed_record_or_a_new_transaction_is_final() { + let t0 = Instant::now(); + let renamed = clone_record(TRANSACTION, b"renamed", resources()).encode(); + let other = clone_record(OTHER_TRANSACTION, b"clone", resources()).encode(); + let cold = ActivationRecord::new(None, configuration(b"clone", resources())) + .unwrap() + .encode(); + let whole = record(); + let truncated = &whole[..whole.len() - 1]; + let retries: [(&[u8], Expected); 4] = [ + (&renamed, |cause| matches!(cause, ActivationCause::Changed)), + (&other, |cause| { + matches!( + cause, + ActivationCause::Transaction { + accepted: TRANSACTION, + offered: OTHER_TRANSACTION, + } + ) + }), + (&cold, |cause| matches!(cause, ActivationCause::Cold)), + (truncated, |cause| { + matches!(cause, ActivationCause::Record(ActivationError::Truncated)) + }), + ]; + // While the activation runs, and once it is ready, or started, whose fence stays open. + let states: [(Setup, u32); 3] = [(activating, 0), (ready, 1), (started, 1)]; + for (activation, opened) in states { + for (retry, cause) in retries { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + let failure = activation + .input(after(t0, 1), ActivationInput::Activate(retry)) + .unwrap_err(); + assert_eq!(failure.state(), state); + assert!(cause(failure.cause()), "{state}: {failure}"); + assert_final(&mut activation, &fence, &failure, after(t0, 2), opened); + } + } +} + +#[test] +fn a_failed_application_is_final_and_no_retry_applies_the_record_again() { + let t0 = Instant::now(); + let fence = FakeFence::default(); + let mut activation = activating(&fence, t0); + let failure = activation + .complete(after(t0, 1), Err("the root's assembly failed".into())) + .unwrap_err(); + assert_eq!(failure.state(), ActivationState::Activating); + let ActivationCause::Failed(error) = failure.cause() else { + panic!("unexpected cause: {failure}"); + }; + assert_eq!(error.to_string(), "the root's assembly failed"); + assert_eq!( + failure.source().unwrap().to_string(), + "the root's assembly failed" + ); + // A byte-equal retry neither joins nor applies the record again: it gets the failure. + assert_final(&mut activation, &fence, &failure, after(t0, 2), 0); +} + +#[test] +fn an_activate_is_decoded_and_validated_before_any_side_effect() { + let t0 = Instant::now(); + let mut long = record(); + long.resize(MAX_RECORD_BYTES + 1, 0); + let mut version = record(); + version[0] = 2; + let cold = ActivationRecord::new(None, configuration(b"clone", resources())) + .unwrap() + .encode(); + let more_cpus = ExpectedResources { + cpus: CPUS + 2, + memory: MEMORY, + }; + let less_memory = ExpectedResources { + cpus: CPUS, + memory: MEMORY - (128 << 20), + }; + // The clone's memory counts whole blocks of 128 MiB, so it never has a part of one. + let partial_block = ExpectedResources { + cpus: CPUS, + memory: MEMORY + (64 << 20), + }; + let cases: [(Vec, Expected); 8] = [ + (Vec::new(), |cause| { + matches!(cause, ActivationCause::Record(ActivationError::Truncated)) + }), + (long, |cause| { + matches!( + cause, + ActivationCause::Record(ActivationError::TooLong(1025)) + ) + }), + (version, |cause| { + matches!(cause, ActivationCause::Record(ActivationError::Version(2))) + }), + (cold, |cause| matches!(cause, ActivationCause::Cold)), + ( + clone_record(TRANSACTION, b"clone", more_cpus).encode(), + |cause| { + matches!( + cause, + ActivationCause::Resources { + expected: ExpectedResources { cpus: 4, .. }, + actual: ExpectedResources { + cpus: CPUS, + memory: MEMORY + }, + } + ) + }, + ), + ( + clone_record(TRANSACTION, b"clone", less_memory).encode(), + |cause| matches!(cause, ActivationCause::Resources { expected, .. } if expected.memory == MEMORY - (128 << 20)), + ), + ( + clone_record(TRANSACTION, b"clone", partial_block).encode(), + |cause| matches!(cause, ActivationCause::Resources { expected, .. } if expected.memory == MEMORY + (64 << 20)), + ), + ( + clone_record(OTHER_TRANSACTION, b"clone", more_cpus).encode(), + |cause| matches!(cause, ActivationCause::Resources { .. }), + ), + ]; + for (bytes, cause) in cases { + let fence = FakeFence::default(); + let mut activation = restricted(&fence, t0); + let failure = activation + .input(after(t0, 1), ActivationInput::Activate(&bytes)) + .unwrap_err(); + assert_eq!(failure.state(), ActivationState::Restricted); + assert!(cause(failure.cause()), "{failure}"); + assert_final(&mut activation, &fence, &failure, after(t0, 2), 0); + } + // A valid record goes to the adapter exactly as it decodes, and opens nothing yet. + let fence = FakeFence::default(); + let mut activation = restricted(&fence, t0); + let action = activation.input(after(t0, 1), ActivationInput::Activate(&record())); + assert_eq!( + action.unwrap(), + ActivationAction::Apply(Box::new(ActivationRecord::decode(&record()).unwrap())) + ); + assert_eq!(fence.opened(), 0); +} + +#[test] +fn the_deadline_starts_at_the_acknowledgement_and_retries_never_extend_it() { + let t0 = Instant::now(); + let fence = FakeFence::default(); + // The repair acknowledged the restore at `t0`, and the adapter made the activation later. + let mut activation = + Activation::of(resources(), Some(t0), fence.clone(), TIMEOUT, after(t0, 3)); + assert_eq!(activation.deadline(), Some(after(t0, 10))); + activation + .input(after(t0, 8), ActivationInput::Reset) + .unwrap(); + let action = activation.input(after(t0, 9), ActivationInput::Activate(&record())); + assert_eq!( + action.unwrap(), + ActivationAction::Apply(Box::new(accepted())) + ); + let late = after(t0, 9) + Duration::from_millis(999); + let retry = activation.input(late, ActivationInput::Activate(&record())); + assert_eq!(retry.unwrap(), ActivationAction::Join); + activation.tick(late).unwrap(); + assert_eq!(activation.deadline(), Some(after(t0, 10))); + // At the deadline, the activation fails, however its application ends after it. + let failure = activation.tick(after(t0, 10)).unwrap_err(); + assert_eq!(failure.state(), ActivationState::Activating); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert_final(&mut activation, &fence, &failure, after(t0, 11), 0); + // An application that completes at the deadline completes too late, as does any input. + for complete in [true, false] { + let fence = FakeFence::default(); + let mut activation = activating(&fence, t0); + let failure = if complete { + activation.complete(after(t0, 10), Ok(())).unwrap_err() + } else { + let retry = ActivationInput::Activate(&record()); + activation.input(after(t0, 10), retry).unwrap_err() + }; + assert_eq!(failure.state(), ActivationState::Activating); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert_final(&mut activation, &fence, &failure, after(t0, 11), 0); + } +} + +#[test] +fn an_ungated_restore_s_deadline_starts_at_the_activation() { + let t0 = Instant::now(); + let fence = FakeFence::default(); + // Without an acknowledgement, the deadline starts when the adapter makes the activation. + let mut activation = Activation::of(resources(), None, fence.clone(), TIMEOUT, after(t0, 5)); + assert_eq!(activation.deadline(), Some(after(t0, 15))); + activation.tick(after(t0, 14)).unwrap(); + let failure = activation + .input(after(t0, 15), ActivationInput::Reset) + .unwrap_err(); + assert_eq!(failure.state(), ActivationState::Bootstrap); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert_final(&mut activation, &fence, &failure, after(t0, 16), 0); + // A deadline that no instant can hold never passes. + let mut activation = Activation::of(resources(), Some(t0), fence.clone(), Duration::MAX, t0); + assert_eq!(activation.deadline(), None); + activation.tick(after(t0, 1 << 30)).unwrap(); + let reset = activation.input(after(t0, 1 << 30), ActivationInput::Reset); + assert_eq!(reset.unwrap(), ActivationAction::Acknowledge); +} + +#[test] +fn out_of_order_completions_and_starts_are_final() { + let t0 = Instant::now(); + // An activation completes only while it runs, and the sandbox starts only once it is ready, + // and once: the fence opens no sooner, and no more than once. + let completions: [(Setup, u32); 4] = + [(bootstrap, 0), (restricted, 0), (ready, 1), (started, 1)]; + for (activation, opened) in completions { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + let failure = activation.complete(after(t0, 1), Ok(())).unwrap_err(); + assert_eq!(failure.state(), state); + assert!(matches!(failure.cause(), ActivationCause::Completion)); + assert_final(&mut activation, &fence, &failure, after(t0, 2), opened); + } + let starts: [(Setup, u32); 4] = [ + (bootstrap, 0), + (restricted, 0), + (activating, 0), + (started, 1), + ]; + for (activation, opened) in starts { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + let failure = activation.start().unwrap_err(); + assert_eq!(failure.state(), state); + assert!(matches!(failure.cause(), ActivationCause::Start)); + assert_final(&mut activation, &fence, &failure, after(t0, 2), opened); + } +} + +/// Returns `body` as a message of the tests' own framing, which no adapter of NVX's speaks: its +/// `kind`, its length, a little-endian `u16`, and the body. +fn message(kind: u8, body: &[u8]) -> Vec { + let length = u16::try_from(body.len()).unwrap(); + let mut message = vec![kind]; + message.extend_from_slice(&length.to_le_bytes()); + message.extend_from_slice(body); + message +} + +/// A reply of the tests' adapter to the host. +#[derive(Debug, PartialEq, Eq)] +enum Reply { + /// The acknowledgement of the session's reset. + Acknowledgement, + /// Ready, which reports the activation. + Ready(ReadyActivation), +} + +/// An adapter of the tests' framing, which drives an activation as an adapter's loop does: it +/// takes the host's bytes as its transport delivers them, passes the activation each message +/// that they complete, and flow control for each byte that completes none, keeps the records +/// that the activation accepts, whose application the test completes when it chooses, and queues +/// its replies for an output that takes nothing, as a host that reads nothing leaves it, and +/// which it never waits for. +struct Adapter { + /// The activation. + activation: Activation, + /// The bytes of the host's message in progress. + message: Vec, + /// The replies that wait for the output. + output: Vec, + /// The records whose application the activation handed over. + applied: Vec, +} + +impl Adapter { + /// Returns the adapter that drives `activation`. + fn new(activation: Activation) -> Self { + Self { + activation, + message: Vec::new(), + output: Vec::new(), + applied: Vec::new(), + } + } + + /// Takes the session's reset at `now`. + fn reset(&mut self, now: Instant) -> Result<(), ActivationFailure> { + let action = self.activation.input(now, ActivationInput::Reset)?; + self.act(action); + Ok(()) + } + + /// Takes `bytes` of the host's messages, which the transport delivered at `now`, one at a + /// time. + fn receive(&mut self, now: Instant, bytes: &[u8]) -> Result<(), ActivationFailure> { + for &byte in bytes { + self.message.push(byte); + let complete = match self.message.as_slice() { + [_, low, high, body @ ..] => { + body.len() == usize::from(u16::from_le_bytes([*low, *high])) + } + _ => false, + }; + let action = if complete { + let message = std::mem::take(&mut self.message); + let input = match message.as_slice() { + [ACTIVATE, _, _, record @ ..] => ActivationInput::Activate(record), + _ => ActivationInput::Request, + }; + self.activation.input(now, input)? + } else { + self.activation.input(now, ActivationInput::Flow)? + }; + self.act(action); + } + Ok(()) + } + + /// Does what `action` says, except that it leaves an accepted record's application to the + /// test. + fn act(&mut self, action: ActivationAction) { + match action { + ActivationAction::Acknowledge => self.output.push(Reply::Acknowledgement), + ActivationAction::Continue | ActivationAction::Join | ActivationAction::Serve => {} + ActivationAction::Apply(record) => self.applied.push(*record), + ActivationAction::Ready(ready) => self.output.push(Reply::Ready(ready)), + } + } +} + +#[test] +fn the_deadline_stays_live_under_trickled_input_and_blocked_output() { + let t0 = Instant::now(); + let activate = message(ACTIVATE, &record()); + assert!(activate.len() > 10); + // A host that trickles its Activate, a byte a second, never completes it by the deadline: + // the byte at the deadline fails the activation, while the output took nothing. + let fence = FakeFence::default(); + let mut adapter = Adapter::new(bootstrap(&fence, t0)); + adapter.reset(t0).unwrap(); + let mut failure = None; + for (second, byte) in (1..).zip(&activate) { + if let Err(error) = adapter.receive(after(t0, second), &[*byte]) { + failure = Some((second, error)); + break; + } + } + let (second, failure) = failure.unwrap(); + assert_eq!(second, 10); + assert_eq!(failure.state(), ActivationState::Restricted); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert!(adapter.applied.is_empty()); + assert_eq!(adapter.output, [Reply::Acknowledgement]); + assert_final(&mut adapter.activation, &fence, &failure, after(t0, 11), 0); + // A host that sends nothing leaves the adapter only its wake at the deadline, which fails the + // activation as well. + let fence = FakeFence::default(); + let mut adapter = Adapter::new(bootstrap(&fence, t0)); + adapter.reset(after(t0, 1)).unwrap(); + let deadline = adapter.activation.deadline().unwrap(); + assert_eq!(deadline, after(t0, 10)); + adapter + .activation + .tick(deadline - Duration::from_millis(1)) + .unwrap(); + let failure = adapter.activation.tick(deadline).unwrap_err(); + assert_eq!(failure.state(), ActivationState::Restricted); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert_final(&mut adapter.activation, &fence, &failure, after(t0, 11), 0); + // A host whose Activate arrives at once, and that trickles a retry while the record's + // application runs, gets the retry joined, without another application or a later deadline, + // and an application that completes after the deadline fails. + let fence = FakeFence::default(); + let mut adapter = Adapter::new(bootstrap(&fence, t0)); + adapter.reset(t0).unwrap(); + adapter.receive(after(t0, 1), &activate).unwrap(); + assert_eq!(adapter.applied, [accepted()]); + for (second, byte) in (2..9).zip(&activate) { + adapter.receive(after(t0, second), &[*byte]).unwrap(); + } + adapter.receive(after(t0, 9), &activate[7..]).unwrap(); + assert_eq!(adapter.applied, [accepted()]); + assert_eq!(adapter.activation.deadline(), Some(after(t0, 10))); + let failure = adapter + .activation + .complete(after(t0, 11), Ok(())) + .unwrap_err(); + assert_eq!(failure.state(), ActivationState::Activating); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert_eq!(adapter.output, [Reply::Acknowledgement]); + assert_final(&mut adapter.activation, &fence, &failure, after(t0, 12), 0); + // Another message of the framing, whole, while the record's application runs fails the + // activation, as it would before the Activate. + let fence = FakeFence::default(); + let mut adapter = Adapter::new(activating(&fence, t0)); + let failure = adapter + .receive(after(t0, 1), &message(PING, b"")) + .unwrap_err(); + assert_eq!(failure.state(), ActivationState::Activating); + assert!(matches!(failure.cause(), ActivationCause::Request)); + assert_final(&mut adapter.activation, &fence, &failure, after(t0, 2), 0); +} + +/// A call that fails an activation. +type Failing = Box) -> Result<(), ActivationFailure>>; + +#[test] +fn strict_failures_are_returned_and_never_power_off() { + let t0 = Instant::now(); + let renamed = clone_record(TRANSACTION, b"renamed", resources()).encode(); + let other = clone_record(OTHER_TRANSACTION, b"clone", resources()).encode(); + let cold = ActivationRecord::new(None, configuration(b"clone", resources())) + .unwrap() + .encode(); + let more_cpus = ExpectedResources { + cpus: CPUS + 1, + memory: MEMORY, + }; + let unexpected = clone_record(TRANSACTION, b"clone", more_cpus).encode(); + let activate = move |record: Vec| -> Failing { + Box::new(move |activation| { + let input = ActivationInput::Activate(&record); + activation.input(after(t0, 1), input).map(drop) + }) + }; + // Each failure, in a state in which it comes: the activation returns it, keeps it, and changes + // nothing else. It has no system through which it could power the VM off: the adapter reports + // the failure, and ends the VM. + let failures: [(Setup, Failing); 11] = [ + ( + restricted, + Box::new(move |activation| activation.tick(after(t0, 10))), + ), + ( + activating, + Box::new(move |activation| { + activation + .input(after(t0, 1), ActivationInput::Reset) + .map(drop) + }), + ), + ( + restricted, + Box::new(move |activation| { + activation + .input(after(t0, 1), ActivationInput::Request) + .map(drop) + }), + ), + (restricted, activate(Vec::new())), + (restricted, activate(cold)), + (restricted, activate(unexpected)), + (ready, activate(other)), + (started, activate(renamed)), + ( + activating, + Box::new(move |activation| { + activation + .complete(after(t0, 1), Err("failed".into())) + .map(drop) + }), + ), + ( + bootstrap, + Box::new(move |activation| activation.complete(after(t0, 1), Ok(())).map(drop)), + ), + (activating, Box::new(|activation| activation.start())), + ]; + let mut causes = Vec::new(); + for (activation, fail) in failures { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let opened = fence.opened(); + let failure = fail(&mut activation).unwrap_err(); + let cause = std::mem::discriminant(failure.cause()); + assert!(!causes.contains(&cause), "{failure}"); + causes.push(cause); + assert_final(&mut activation, &fence, &failure, after(t0, 11), opened); + } +} + +#[test] +fn failures_name_their_state_and_their_cause() { + let states = [ + (ActivationState::Bootstrap, "transport bootstrap"), + (ActivationState::Restricted, "restricted"), + (ActivationState::Activating, "activating"), + (ActivationState::Ready, "ready"), + (ActivationState::Started, "started"), + (ActivationState::Failed, "failed"), + ]; + for (state, text) in states { + assert_eq!(state.to_string(), text); + } + let failure = |state, cause| ActivationFailure { + state, + cause: Arc::new(cause), + }; + let cases = [ + ( + failure(ActivationState::Bootstrap, ActivationCause::Expired), + "the activation failed in its transport bootstrap state: the deadline passed before \ + the activation was ready", + ), + ( + failure(ActivationState::Activating, ActivationCause::Reset), + "the activation failed in its activating state: the session was reset again before \ + the activation was ready", + ), + ( + failure(ActivationState::Restricted, ActivationCause::Request), + "the activation failed in its restricted state: the host sent a request before the \ + activation was ready", + ), + ( + failure( + ActivationState::Restricted, + ActivationCause::Record(ActivationError::Truncated), + ), + "the activation failed in its restricted state: the Activate's record is refused: \ + the record ends inside its version or one of its fields", + ), + ( + failure(ActivationState::Restricted, ActivationCause::Cold), + "the activation failed in its restricted state: the Activate's record is a cold \ + boot's, without a restore section", + ), + ( + failure( + ActivationState::Restricted, + ActivationCause::Resources { + expected: ExpectedResources { + cpus: 4, + memory: 2 << 30, + }, + actual: resources(), + }, + ), + "the activation failed in its restricted state: the Activate's record expects 4 CPUs \ + and 2147483648 bytes of memory, and the clone has 2 CPUs and 1073741824 bytes", + ), + ( + failure( + ActivationState::Ready, + ActivationCause::Transaction { + accepted: TRANSACTION, + offered: OTHER_TRANSACTION, + }, + ), + "the activation failed in its ready state: the Activate names the transaction \ + 33333333333333333333333333333333, after an Activate of the transaction \ + 22222222222222222222222222222222", + ), + ( + failure(ActivationState::Started, ActivationCause::Changed), + "the activation failed in its started state: the Activate changes the record of the \ + accepted Activate's transaction", + ), + ( + failure( + ActivationState::Activating, + ActivationCause::Failed("the root's assembly failed".into()), + ), + "the activation failed in its activating state: the record's application failed: \ + the root's assembly failed", + ), + ( + failure(ActivationState::Restricted, ActivationCause::Completion), + "the activation failed in its restricted state: an activation that did not run \ + completed", + ), + ( + failure(ActivationState::Activating, ActivationCause::Start), + "the activation failed in its activating state: the sandbox started before the \ + activation was ready, or again", + ), + ]; + for (failure, text) in cases { + assert_eq!(failure.to_string(), text); + let source = match failure.cause() { + ActivationCause::Record(_) => { + Some("the record ends inside its version or one of its fields") + } + ActivationCause::Failed(_) => Some("the root's assembly failed"), + _ => None, + }; + assert_eq!( + failure.source().map(|source| source.to_string()), + source.map(str::to_owned) + ); + } +} + +#[test] +fn an_activation_debugs_its_state_resources_and_deadline_without_its_fence() { + let t0 = Instant::now(); + let activation = Activation::of( + ExpectedResources { + cpus: 4, + memory: 2 << 30, + }, + Some(t0), + FakeFence::default(), + TIMEOUT, + after(t0, 1), + ); + assert_eq!( + format!("{activation:?}"), + format!( + "Activation {{ state: Bootstrap, resources: ExpectedResources {{ cpus: 4, memory: \ + 2147483648 }}, deadline: {:?}, .. }}", + Some(after(t0, 10)) + ) + ); +} diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 60374c047..c275b46f8 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -16,7 +16,8 @@ //! workload admission, with nothing mounted from the devices that its clones replace, and //! through which each clone that a restore of the snapshot starts repairs its restore: it //! activates the restore's memory target, reseeds the kernel's random number generator, and -//! finishes the restore, with its acknowledgement. +//! finishes the restore, with its acknowledgement, and then takes its own configuration from the +//! host, in one Activate on a restricted control session, before it admits any execution. //! //! Each other module holds one program, and the crate's binary of the same name runs its `main`. //! Two of them also offer their steps as typed calls, which any guest init or agent runs in its