diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 5413b1ae1..18df974e7 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -120,7 +120,7 @@ on Linux and Windows. | `nvx::guest::exec` | `Execution`: drives one program that the caller spawned with the program's ends of its `Pipes`, or that the caller started itself and adopts with its own ends of the program's pipes and a `Child` that names it, without blocking, from the caller's own loop or reactor, and reports each step as an `Event`: the launch report, which an adopted program has at once, the output of each stream, kill failures, and the `Outcome`. It reads a stream only as far as the caller's `grant` allows, so a stalled sink stalls its own execution alone, writes the program's standard input without `SIGPIPE`, and ends the program on a cancellation, a timeout, or a disconnection, with `SIGKILL` through the program's pidfd to the program and, if the caller asks, its process group, while no collector has taken its status, and, through the opened directory of a cgroup that the caller names, to that cgroup. It closes the program's input once the program has ended. It waits for its program alone, by a pidfd, or takes the status from the caller's collector with `observe`; with `Options::caller_reaps`, that collector is the program's only one. NVX's managed agent is one caller | | `nvx::guest::exec::binding` | `Binding`: an `nvx::exec::Runner` bound to an `Execution`, the glue that a guest adapter would otherwise write. The runner's read allowances become grants, its discards discards, the input that it accepted writes and then a close, its terminations a cancellation, a disconnection, or a termination, and its settlement a kill of the execution's cgroup and checks until no process occupies it; without a cgroup, nothing proves that the program's descendants are gone, so the execution is fenced. It feeds the execution's events back to the runner, statuses that a caller who reaps hands over with `observe` included, and never blocks: the caller polls its descriptors and its input writer until its deadline, and each call takes a bounded number of steps, so that no execution holds the caller's loop | | `nvx::guest::time` | x86_64: `TimeAbi`, the steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent: the cold-boot time step and, separately, the start of the time daemon in a child process that the caller supervises, the VM generation ID, a `CaptureRequest` that repairs a `Restore`, and a `RestoreFinish`. The request's mode selects how a restore hands over its fresh entropy: the file mode, which `nvx-time` keeps, writes it to a file, and the in-memory mode returns it as the `Restore`'s owned `Seed`, and writes no file. A violation of the ABI stops the VM in enforcing mode, with the time ABI's status, as `nvx-time` does; in strict mode, `TimeAbi::strict`, which a caller that owns the VM's exit status and shutdown selects, the call fails with the `Violation`, whose `status` is the one with which enforcing mode stops the VM, and the time daemon and a capture's preparation enforce the ABI all the same. A call takes the permission for OpenVMM's ports on the calling thread for each port transaction alone, and fails with `TimeError::PortRelease` if the thread may still hold some of it, which a process that it started would inherit, and from which `nvx::guest::spawn` starts none. No call waits for a child that it did not start. `nvx-time` runs its steps through them | -| `nvx::guest::clone` | x86_64: the preparation of a template's guest for a capture before its root's assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init or agent, whether or not it is PID 1. `CapturePreparation`, of the caller's `AdmissionFence` and the `Devices` that the template's clones replace, the `distro` and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag: `prepare` closes the fence, so that no execution starts, refuses with a `Mounted` the first mount from those devices, a distro partition included, or from an overlay, bind mounts included, in a bounded read of `/proc/self/mountinfo` of at most `MOUNT_TABLE_LIMIT` bytes, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as `nvx-time pre-capture` does, and returns an owned `CaptureBarrier`. `CaptureBarrier::capture` requests the capture with the platform tier's request, in the in-memory mode, through `TimeAbi::capture`, and returns `Captured::Continued` once it opened the fence and restored the detectors' settings, or, in a clone, `Captured::Restored`, with the fence still closed, since nothing else may run before repair acknowledges the restore, and the detectors suppressed until the time daemon restores their settings: `Restored::into_parts` hands the `Restore` that repair takes next, which holds the restore's `Seed`, and the closed fence to the caller, which keeps it closed through the repair and opens it once the clone may admit executions, such as once it is activated. `CaptureBarrier::cancel`, a dropped barrier, and a failure to read the VM generation ID open the fence and restore the detectors' settings; a capture request that fails restores the settings but leaves the fence closed, since it may fail in a clone whose restore it could not repair. Each failure is a `CaptureError`, with its `CaptureStage` and `CaptureCause`. `repair`, or a `Repair` that the caller may retry, repairs the clone of a `Restore` next: it checks that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online; with a memory target, it checks, before it adds any memory block, that the kernel's block size is 128 MiB and that the restore's ranges continue the captured RAM in logical order, with no gap, overlap, or range in the MMIO aperture, while the host bounds the target by the snapshot's capacity, then adds and onlines the blocks through sysfs, as `nvx-snapshot` does, and verifies them; it reseeds the kernel's random number generator with the `Seed`, having drawn no randomness before, and drops it; and it finishes the restore through `TimeAbi::restore_finish`, with the acknowledgement when the host gates its input until then, which hands the deferred time work to the time daemon that the caller supervises. It returns the `Repaired` clone, with its generation ID, its online CPUs and memory, and when it acknowledged the restore, and never touches the fence. It adds no memory block twice, and reseeds and finishes at most once, so a failure at or after the reseed is final. Each failure is a `RepairError`, with its `RepairStage`, its `RepairCause`, and the repair's `RepairProgress`, and `RepairError::status` selects the status, of `nvx::protocol::status`, with which the caller ends the VM. The capture and the repair take the time ABI in strict mode, so that a violation of it fails them, with its status, rather than stopping the VM, and only the capture's preparation and the time daemon enforce it. An `Activation` of the `Repaired` clone and of the caller's closed `AdmissionFence` then takes the clone's own configuration from the host, in one Activate, on a restricted control session: a state machine without I/O, independent of any framing, to which the adapter passes what its session receives, as an `ActivationInput`, and which returns the `ActivationAction` that the adapter takes. From its transport bootstrap, which takes the reset that the restore queued for the NVXS session, the restricted session takes only flow control and one Activate, and anything else, a request or another reset, fails it; the activation decodes the Activate's record with `ActivationRecord::decode`, and accepts only a clone's, with a restore section, that expects the CPU count and the memory size that the repair reports, before any side effect; the adapter applies the record with `apply`, as at a cold boot: it assembles the root with `root::assemble`, with the record's image, an EROFS image whose UUID it checks if the record expects one, its upper, a tmpfs or the scratch with the record's journal policy, and its overlay mode, and with the `RootTemplate` that the guest's own configuration holds, which its template captured: the runtime directory, the block devices, and the timeout. It refuses a GPT disk before any side effect, since the kernel created the block devices of its partitions for the template's placeholder, and, with the record's workload identity, checks its account in the assembled root before Ready, as NVX's init checks it at boot: `entry::Identity::new(uid, gid)`, without supplementary groups or capabilities, in existing-account mode, as which every execution runs, while without one each execution names its own identity, which `Entry::check` checks. It returns the `Applied` root and identity, or an `ApplyError`, after which nothing that it mounted stays mounted unless the error reports the unmounts that failed, and applies none of the record's export, network, hostname, or machine ID. `Activation::complete` then opens the fence and returns the `ReadyActivation` that Ready reports beside the adapter's own identities, that the activation applied, with the record's transaction ID, after which the host's mapping table and start follow in the adapter's order, as after a cold boot, and `Activation::start` records the start. A retry with the same transaction ID and a byte-equal record joins the running activation, or gets its cached result, also in a later session once the clone is ready; a changed record, another transaction, another reset before Ready, or a failed activation is final. Its deadline, which the caller chooses, and which neither input nor retries extend, starts at the restore's acknowledgement, and the adapter ticks it however its input trickles and its output blocks. Each failure is an `ActivationFailure`, with its `ActivationState` and `ActivationCause`, and never powers the VM off: the adapter ends it | +| `nvx::guest::clone` | x86_64: the preparation of a template's guest for a capture before its root's assembly, and the repair of each clone that a restore of its snapshot starts, for a guest init or agent, whether or not it is PID 1. `CapturePreparation`, of the caller's `AdmissionFence` and the `Devices` that the template's clones replace, the `distro` and scratch block devices, which their MMIO windows name, and the host filesystem device's virtio-fs tag: `prepare` closes the fence, so that no execution starts, refuses with a `Mounted` the first mount from those devices, a distro partition included, or from an overlay, bind mounts included, in a bounded read of `/proc/self/mountinfo` of at most `MOUNT_TABLE_LIMIT` bytes, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors, as `nvx-time pre-capture` does, and returns an owned `CaptureBarrier`. `CaptureBarrier::capture` requests the capture with the platform tier's request, in the in-memory mode, through `TimeAbi::capture`, and returns `Captured::Continued` once it opened the fence and restored the detectors' settings, or, in a clone, `Captured::Restored`, with the fence still closed, since nothing else may run before repair acknowledges the restore, and the detectors suppressed until the time daemon restores their settings: `Restored::into_parts` hands the `Restore` that repair takes next, which holds the restore's `Seed`, and the closed fence to the caller, which keeps it closed through the repair and opens it once the clone may admit executions, such as once it is activated. `CaptureBarrier::cancel`, a dropped barrier, and a failure to read the VM generation ID open the fence and restore the detectors' settings; a capture request that fails restores the settings but leaves the fence closed, since it may fail in a clone whose restore it could not repair. Each failure is a `CaptureError`, with its `CaptureStage` and `CaptureCause`. `repair`, or a `Repair` that the caller may retry, repairs the clone of a `Restore` next: it checks that the online CPUs are a prefix that meets the restore's processor target, since it brings no CPU online; with a memory target, it checks, before it adds any memory block, that the kernel's block size is 128 MiB and that the restore's ranges continue the captured RAM in logical order, with no gap, overlap, or range in the MMIO aperture, while the host bounds the target by the snapshot's capacity, then adds and onlines the blocks through sysfs, as `nvx-snapshot` does, and verifies them; it reseeds the kernel's random number generator with the `Seed`, having drawn no randomness before, and drops it; and it finishes the restore through `TimeAbi::restore_finish`, with the acknowledgement when the host gates its input until then, which hands the deferred time work to the time daemon that the caller supervises. It returns the `Repaired` clone, with its generation ID, its online CPUs and memory, and when it acknowledged the restore, and never touches the fence. It adds no memory block twice, and reseeds and finishes at most once, so a failure at or after the reseed is final. Each failure is a `RepairError`, with its `RepairStage`, its `RepairCause`, and the repair's `RepairProgress`, and `RepairError::status` selects the status, of `nvx::protocol::status`, with which the caller ends the VM. The capture and the repair take the time ABI in strict mode, so that a violation of it fails them, with its status, rather than stopping the VM, and only the capture's preparation and the time daemon enforce it. An `Activation` of the `Repaired` clone, or, with `Activation::with_resources`, of the resources and the instant of the acknowledgement that a repair reports, as in an adapter's tests, and of the caller's closed `AdmissionFence` then takes the clone's own configuration from the host, in one Activate, on a restricted control session: a state machine without I/O, independent of any framing, to which the adapter passes what its session receives, as an `ActivationInput`, and which returns the `ActivationAction` that the adapter takes. From its transport bootstrap, which takes the reset that the restore queued for the NVXS session, the restricted session takes only flow control and one Activate, and anything else, a request or another reset, fails it; the activation decodes the Activate's record with `ActivationRecord::decode`, and accepts only a clone's, with a restore section, that expects the CPU count and the memory size that the repair reports, before any side effect, while an Activate whose record the adapter's framing refused unread, as longer than a record can be, fails as a record that long does; the adapter applies the record with `apply`, as at a cold boot: it assembles the root as `root::assemble` does, with the record's image, an EROFS image whose UUID it checks if the record expects one, its upper, a tmpfs or the scratch with the record's journal policy, and its overlay mode, and with the `RootTemplate` that the guest's own configuration holds, which its template captured: the runtime directory, the block devices, and the timeout, but waits for no device past the activation's deadline, which the adapter passes, so that the application ends by then. It refuses a GPT disk before any side effect, since the kernel created the block devices of its partitions for the template's placeholder, and, with the record's workload identity, checks its account in the assembled root before Ready, as NVX's init checks it at boot: `entry::Identity::new(uid, gid)`, without supplementary groups or capabilities, in existing-account mode, as which every execution runs, while without one each execution names its own identity, which `Entry::check` checks. It returns the `Applied` root and identity, or an `ApplyError`, after which nothing that it mounted stays mounted unless the error reports the unmounts that failed, and applies none of the record's export, network, hostname, or machine ID. Once the adapter passed the activation what the session received while the application ran, which came before Ready, `Activation::complete` opens the fence and returns the `ReadyActivation` that Ready reports beside the adapter's own identities, that the activation applied, with the record's transaction ID, after which the host's mapping table and start follow in the adapter's order, as after a cold boot, and `Activation::start` records the start. A retry with the same transaction ID and a byte-equal record joins the running activation, or gets its cached result, also in a later session once the clone is ready; a changed record, another transaction, another reset before Ready, or a failed activation is final. Its deadline, which the caller chooses, and which neither input nor retries extend, starts at the restore's acknowledgement, and the adapter ticks it however its input trickles and its output blocks. Each failure is an `ActivationFailure`, with its `ActivationState` and `ActivationCause`, and never powers the VM off: the adapter ends it | | `nvx::guest::reseed` | `Seed`, `Seed::read`, and `reseed`: a restore's fresh entropy, which starts with its `GenerationId`, and the reseed of the kernel's random number generator with it, in the caller's process. A `Seed` keeps its bytes in one allocation, which no move copies, zeroes them with volatile writes when it is dropped, can't be cloned, and formats none of them. `nvx-reseed` runs them | | `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their roles' access modes and their expected sizes, and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already, once it has checked everything: the read-only layers of the `Image` that the caller names, which it never probes: an EROFS image, whose expected `nvx::uuid::Uuid` it checks in its superblock, or a GPT disk whose partition table it reads and checks with `nvx::guest::gpt::read_disk`, against the caller's `nvx::gpt::Expected` values, whose first partition it reads, up to the caller's bound, as opaque bytes that it never mounts, and whose later partitions, at least one, hold ext4 layers that it mounts `ro,nosuid,nodev` with `errors=remount-ro`, so that an error that the kernel finds in a layer fails the operation that found it, rather than panic the guest, as the layer's superblock may ask, from the block devices that the kernel created when it found the disk, once their superblocks show that no journal needs a recovery, which their read-only device can't take, partition 2's at the overlay's base and the last partition's at its top, under an overlay mounted `nosuid,nodev` with the one option set of the caller's `OverlayMode`, `metacopy=on,xino=on` or, for lowers that may carry untrusted overlay xattrs, `metacopy=off,redirect_dir=nofollow`, with no fallback, whose upper and work directories are on the `Upper`: an ext4 scratch mounted `nosuid,nodev`, whose superblock it checks for a journal that the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` at the kernel's default size. It refuses overlay options longer than `MOUNT_OPTIONS_LIMIT`, the most that a mount passes, which, below `/run/nvx`, fit up to 195 layers of a GPT disk, and an overlay that the kernel's mount table shows read-only or without the mode's options in effect, since the kernel mounts one so, rather than fail, when its upper can't hold its work directory or its extended attributes. The `Root` reports the first partition's bytes, its `Lower` layers, its `UpperKind`, and its `OverlayMode`. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::mount_export`: the host export, the aggregate device of `protocol::export`, one per root, mounted `nosuid,nodev` with the access of its `ExportMode` at `hostfs/root` in the runtime directory, out of the root, below a directory that only root may enter, whose root it returns as the `SourceRoot` of the caller's mappings. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, the export, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then the mounts of an aggregate and of the export, the later first, the overlay, the layers from the top down, and the upper, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off | | `nvx::guest::gpt` | `read_disk`: the partition table of a GPT disk, read and checked with `nvx::gpt`'s `Disk::read` through the disk's block device, which the caller opened, at the logical block size and size that `BLKSSZGET` and `BLKGETSIZE64` report, with the caller's `Expected` values, such as the disk GUID that a boot parameter names. It only reads the device, at explicit offsets, so it never moves the file's position. It neither compares the partitions that the kernel created with the table nor asks the kernel to reread the table. Each failure is a typed `DeviceError` | diff --git a/nvx-rs/src/guest/clone.rs b/nvx-rs/src/guest/clone.rs index e0270c55e..4062548b9 100644 --- a/nvx-rs/src/guest/clone.rs +++ b/nvx-rs/src/guest/clone.rs @@ -115,10 +115,13 @@ //! request or another reset, fails the activation. The activation decodes the Activate's //! record with [`ActivationRecord::decode`](crate::activation::ActivationRecord::decode), and //! accepts it only as a clone's, with a restore section, that expects the CPU count and the -//! memory size that the repair reports, before anything has a side effect; +//! memory size that the repair reports, before anything has a side effect. An Activate whose +//! record the adapter's framing refused unread, as longer than a record can be, fails as a +//! record that long does; //! 3. activating: the adapter applies the accepted record with [`apply`](fn@apply), which -//! activates the root and the workload with its settings, as below, then reports the result -//! to [`Activation::complete`]; +//! activates the root and the workload with its settings, as below, passes the activation +//! what the session received meanwhile, which came before Ready, then reports the result to +//! [`Activation::complete`]; //! 4. ready, with the mappings pending: the activation completed, opened the fence, and returned //! what Ready reports beside the adapter's own identities, such as its build, a //! [`ReadyActivation`](crate::activation::ReadyActivation): that the activation applied, with @@ -139,15 +142,16 @@ //! and its [`ActivationCause`], and which every later call returns again. The activation never //! powers the VM off: the adapter reports the failure, and ends the VM. //! -//! [`apply`](fn@apply) activates the clone as at a cold boot. It assembles the root with -//! [`root::assemble`], with the record's image, an EROFS image whose UUID it checks if the record -//! expects one, its upper, a tmpfs or the scratch, with the record's journal policy, and its -//! overlay mode, and with the inputs that the guest takes from its own configuration, which its -//! template captured, a [`RootTemplate`]: the runtime directory, the block devices, and how long -//! the assembly waits for them. It refuses a GPT disk before anything has a side effect: the -//! kernel created the block devices of a disk's partitions when it found the template's -//! placeholder. If the record has a workload identity, it then checks its account in the -//! assembled root, before Ready, as NVX's init checks it at boot: the identity is +//! [`apply`](fn@apply) activates the clone as at a cold boot. It assembles the root as +//! [`root::assemble`] does, with the record's image, an EROFS image whose UUID it checks if the +//! record expects one, its upper, a tmpfs or the scratch, with the record's journal policy, and +//! its overlay mode, and with the inputs that the guest takes from its own configuration, which +//! its template captured, a [`RootTemplate`]: the runtime directory, the block devices, and how +//! long the assembly waits for each, but never past the activation's deadline, which the adapter +//! passes, so that the application ends by then. It refuses a GPT disk before anything has a +//! side effect: the kernel created the block devices of a disk's partitions when it found the +//! template's placeholder. If the record has a workload identity, it then checks its account in +//! the assembled root, before Ready, as NVX's init checks it at boot: the identity is //! [`Identity::new`](super::entry::Identity::new) of the record's user and group IDs, without //! supplementary groups or capabilities, in existing-account mode, and every execution runs as //! it. Without one, the adapter gives each execution its own identity, which diff --git a/nvx-rs/src/guest/clone/activation.rs b/nvx-rs/src/guest/clone/activation.rs index 063efe4e6..dbab2363d 100644 --- a/nvx-rs/src/guest/clone/activation.rs +++ b/nvx-rs/src/guest/clone/activation.rs @@ -27,11 +27,13 @@ pub use self::apply::{Applied, ApplyError, RootTemplate, apply}; /// The adapter passes it what it receives on the session, each an [`ActivationInput`], through /// [`input`](Self::input), and does what the [`ActivationAction`] that it returns says. The /// adapter applies the record of the Activate that the activation accepted, with -/// [`apply`](fn@apply), and reports the result through [`complete`](Self::complete), and it -/// reports the start of the sandbox, which its own order takes once the activation is ready, -/// through [`start`](Self::start). The adapter's loop wakes by the [`deadline`](Self::deadline), -/// and [ticks](Self::tick) the activation then, whether input arrives or not, and whether its -/// output drains or not. +/// [`apply`](fn@apply), whose waits end by the activation's [`deadline`](Self::deadline), passes +/// the activation what the session received while the application ran, which came before Ready, +/// then reports the result through [`complete`](Self::complete), whose Ready answers the Activate +/// and each retry that [joined](ActivationAction::Join) it, and it reports the start of the +/// sandbox, which its own order takes once the activation is ready, through +/// [`start`](Self::start). The adapter's loop wakes by the deadline, and [ticks](Self::tick) the +/// activation then, whether input arrives or not, and whether its output drains or not. /// /// It goes through the [`ActivationState`]s in order, and each takes these inputs: /// @@ -66,20 +68,27 @@ pub use self::apply::{Applied, ApplyError, RootTemplate, apply}; /// Partial, /// /// The host's Activate, with its record's bytes. /// Activate(Vec), +/// /// The host's Activate, whose header announced this many bytes, more than a record has, +/// /// which the framing refused before it read any of them. +/// TooLong(usize), /// /// Any other message of the host, such as a part of the mapping table, or the start. /// Other(Vec), /// } /// -/// /// The agent of a clone, which serves its control session in its own framing. +/// /// The agent of a clone, which serves its control session in its own framing, and queues what +/// /// it sends, so that a host that reads nothing holds up neither the agent nor the deadline. /// trait Agent { -/// /// Waits until the host sends something, or until `deadline`, and returns what it sent. +/// /// Waits until the host sends something, or until `deadline`, while it writes what the +/// /// agent queued, and returns what the host sent, or `None` once the deadline passed. A +/// /// deadline that passed already returns only what the host sent already. /// fn receive(&mut self, deadline: Option) -> Option; -/// /// Acknowledges the session's reset, and abandons the earlier epoch's work. +/// /// Queues the acknowledgement of the session's reset, and abandons the earlier epoch's +/// /// work. /// fn acknowledge(&mut self); /// /// Keeps the clone's activated root, and the identity, if any, in which it runs every /// /// execution. /// fn keep(&mut self, applied: Applied); -/// /// Sends Ready, which reports `activation` beside the agent's own identities, such as its +/// /// Queues Ready, which reports `activation` beside the agent's own identities, such as its /// /// build. /// fn ready(&mut self, activation: ReadyActivation); /// /// Serves `message` in the agent's order, as after a cold boot, and returns whether the @@ -100,19 +109,26 @@ pub use self::apply::{Applied, ApplyError, RootTemplate, apply}; /// activation.tick(Instant::now())?; /// continue; /// }; -/// let input = match &received { -/// Received::Reset => ActivationInput::Reset, -/// Received::Partial => ActivationInput::Flow, -/// Received::Activate(record) => ActivationInput::Activate(record), -/// Received::Other(_) => ActivationInput::Request, -/// }; -/// match activation.input(Instant::now(), input)? { +/// match activation.input(Instant::now(), input(&received))? { /// ActivationAction::Acknowledge => agent.acknowledge(), /// ActivationAction::Continue | ActivationAction::Join => {} /// ActivationAction::Apply(record) => { /// // The record is valid, and nothing happened before: the activation's side -/// // effects start here, and the fence opens once they all succeeded. -/// let ready = match clone::apply(&record, template) { +/// // effects start here, and the fence opens once they all succeeded. The +/// // application waits for the devices until the activation's deadline at the +/// // latest, so that it ends by then. +/// let applied = clone::apply(&record, template, activation.deadline()); +/// // What the host sent meanwhile came before Ready: a retry of the Activate, +/// // byte for byte, joins the application, and anything else, a reset included, +/// // fails the activation, which drops what the application assembled. +/// let mut joined = 0; +/// while let Some(received) = agent.receive(Some(Instant::now())) { +/// let action = activation.input(Instant::now(), input(&received))?; +/// if action == ActivationAction::Join { +/// joined += 1; +/// } +/// } +/// let ready = match applied { /// Ok(applied) => { /// agent.keep(applied); /// activation.complete(Instant::now(), Ok(()))? @@ -120,7 +136,10 @@ pub use self::apply::{Applied, ApplyError, RootTemplate, apply}; /// // A failed application fails the activation: the fence stays closed. /// Err(error) => activation.complete(Instant::now(), Err(error.into()))?, /// }; -/// agent.ready(ready); +/// // Ready answers the Activate, and each retry that joined its application. +/// for _ in 0..=joined { +/// agent.ready(ready); +/// } /// } /// ActivationAction::Ready(ready) => agent.ready(ready), /// // The activation serves only another message, once it is ready. @@ -134,6 +153,17 @@ pub use self::apply::{Applied, ApplyError, RootTemplate, apply}; /// } /// } /// } +/// +/// /// Returns the activation's input of `received`. +/// fn input(received: &Received) -> ActivationInput<'_> { +/// match received { +/// Received::Reset => ActivationInput::Reset, +/// Received::Partial => ActivationInput::Flow, +/// Received::Activate(record) => ActivationInput::Activate(record), +/// Received::TooLong(length) => ActivationInput::TooLong(*length), +/// Received::Other(_) => ActivationInput::Request, +/// } +/// } /// ``` pub struct Activation { /// The caller's fence of workload admission, closed until the activation is ready. @@ -163,12 +193,14 @@ impl Activation { cpus: repaired.cpus(), memory: repaired.memory(), }; - Self::of(resources, repaired.acknowledged_at(), fence, timeout, now) + Self::with_resources(resources, repaired.acknowledged_at(), fence, timeout, now) } /// Returns the activation of a clone that has `resources`, and whose repair acknowledged its - /// restore at `acknowledged_at`, if it did, as [`new`](Self::new) does. - fn of( + /// restore at `acknowledged_at`, if it did, as [`new`](Self::new) does with the resources and + /// the acknowledgement of a [`Repaired`] clone: for a caller that has them otherwise, such as + /// the tests of an adapter's framing, which run its session without a restore. + pub fn with_resources( resources: ExpectedResources, acknowledged_at: Option, fence: F, @@ -213,6 +245,11 @@ impl Activation { /// acknowledges, a byte-equal retry of the Activate gets the cached result, what Ready /// reports, without activating again, and any other Activate fails as while activating. /// Every other request goes to the adapter's order, as after a cold boot. + /// + /// An Activate whose record the adapter's framing refused unread, as longer than a record can + /// be, [`ActivationInput::TooLong`], fails as one whose record has that many bytes: in the + /// transport bootstrap with [`ActivationCause::Request`], and from then on with + /// [`ActivationCause::Record`]. pub fn input( &mut self, now: Instant, @@ -234,12 +271,24 @@ impl Activation { ActivationState::Activating | ActivationState::Ready | ActivationState::Started, ActivationInput::Activate(record), ) => self.retry(record), + // A record that long never decodes, so its Activate fails as one whose bytes the + // adapter read would, whether the activation accepted another or not. + ( + ActivationState::Restricted + | ActivationState::Activating + | ActivationState::Ready + | ActivationState::Started, + ActivationInput::TooLong(length), + ) => Err(self.fail(ActivationCause::Record(ActivationError::TooLong(length)))), (ActivationState::Ready | ActivationState::Started, ActivationInput::Request) => { Ok(ActivationAction::Serve) } - (_, ActivationInput::Activate(_) | ActivationInput::Request) => { - Err(self.fail(ActivationCause::Request)) - } + ( + _, + ActivationInput::Activate(_) + | ActivationInput::TooLong(_) + | ActivationInput::Request, + ) => Err(self.fail(ActivationCause::Request)), } } @@ -415,6 +464,12 @@ pub enum ActivationInput<'a> { /// The host's Activate, with the bytes of its activation record, the canonical encoding of an /// [`ActivationRecord`]. Activate(&'a [u8]), + /// The host's Activate, whose record the adapter's framing refused before it took any byte of + /// it, since its header announced this many bytes, more than + /// [`MAX_RECORD_BYTES`](crate::activation::MAX_RECORD_BYTES): the activation takes it as an + /// Activate whose record's bytes are that long, which [`ActivationRecord::decode`] refuses + /// with [`ActivationError::TooLong`]. + TooLong(usize), /// Any other request of the host, such as a part of the mapping table, the start of the /// sandbox, an execution, or a probe of the adapter's framing. Request, @@ -431,8 +486,9 @@ pub enum ActivationAction { Continue, /// The activation accepted the host's Activate, whose record decoded and validated, and /// which nothing before had any side effect of: the adapter applies the record with - /// [`apply`](fn@apply), which activates the root and the workload with its settings, then - /// reports the result to [`Activation::complete`]. + /// [`apply`](fn@apply), which activates the root and the workload with its settings, passes + /// the activation what the session received meanwhile, which came before Ready, then reports + /// the result to [`Activation::complete`]. Apply(Box), /// A retry of the accepted Activate joined the activation that runs: what /// [`Activation::complete`] returns answers it too, as the adapter's framing answers each diff --git a/nvx-rs/src/guest/clone/activation/apply.rs b/nvx-rs/src/guest/clone/activation/apply.rs index 431b69b98..a19d8241a 100644 --- a/nvx-rs/src/guest/clone/activation/apply.rs +++ b/nvx-rs/src/guest/clone/activation/apply.rs @@ -7,7 +7,7 @@ mod tests; use std::fmt; use std::path::{Path, PathBuf}; -use std::time::Duration; +use std::time::{Duration, Instant}; #[cfg(doc)] use super::Activation; @@ -31,7 +31,8 @@ pub struct RootTemplate { /// The scratch block device, which holds the upper of a record whose upper is the scratch, /// and which a record whose upper is a tmpfs leaves alone. pub scratch: Device, - /// How long the assembly waits for each device to appear. + /// How long the assembly waits for each device to appear, until the deadline that + /// [`apply`](fn@apply) takes at the latest. pub timeout: Duration, } @@ -76,8 +77,9 @@ impl Applied { } /// Applies `record`, the record of the Activate that a clone's [`Activation`] accepted, to the -/// clone, as at a cold boot, and returns the root and the workload that it activated, which the -/// adapter keeps once it reported the result to [`Activation::complete`]. +/// clone, as at a cold boot, by `deadline`, the activation's [deadline](Activation::deadline), if +/// any, and returns the root and the workload that it activated, which the adapter keeps once it +/// reported the result to [`Activation::complete`]. /// /// In order, it: /// @@ -86,9 +88,12 @@ impl Applied { /// with the record's journal policy; and its overlay mode; and, from `template`, the runtime /// directory, the devices, and the timeout. It refuses a GPT disk, with /// [`ApplyError::Gpt`], before anything has a side effect; -/// 2. assembles the root with [`root::assemble`], which checks the devices, and the image as the -/// record expects it, and mounts the root, or fails with [`ApplyError::Assembly`], once it -/// unmounted what it mounted; +/// 2. assembles the root as [`root::assemble`] does, which checks the devices, and the image as +/// the record expects it, and mounts the root, or fails with [`ApplyError::Assembly`], once it +/// unmounted what it mounted. It waits for each device as long as the template's timeout +/// allows, but never past `deadline`: a device that has not appeared by then fails the +/// assembly, as at its timeout, so that the application ends by the activation's deadline, +/// past which [`Activation::complete`] fails the activation as expired, whatever the result; /// 3. if the record has a workload identity, checks its account in the assembled root, before /// Ready, as NVX's init checks it at boot: the identity is [`Identity::new`] of the record's /// user and group IDs, without supplementary groups or capabilities, in @@ -103,8 +108,12 @@ impl Applied { /// /// It needs the guest's root user, in the guest's initial mount namespace, as /// [`root::assemble`] does. -pub fn apply(record: &ActivationRecord, template: &RootTemplate) -> Result { - let (root, identity) = apply_with(&mut LiveSystem, record, template)?; +pub fn apply( + record: &ActivationRecord, + template: &RootTemplate, + deadline: Option, +) -> Result { + let (root, identity) = apply_with(&mut LiveSystem, record, template, deadline)?; Ok(Applied { root, identity }) } @@ -167,8 +176,13 @@ trait System { /// What an assembly mounted: a [`Root`], in a VM. type Root; - /// Assembles the root of `assembly`, as [`root::assemble`] does. - fn assemble(&mut self, assembly: &Assembly) -> Result; + /// Assembles the root of `assembly`, as [`root::assemble`] does, but waits for each device + /// until `deadline` at the latest, if any. + fn assemble( + &mut self, + assembly: &Assembly, + deadline: Option, + ) -> Result; /// Reads and checks the account of `identity` in `root`, as [`check_account`] does. fn account(&mut self, root: &Self::Root, identity: &Identity) -> Result<(), entry::Error>; @@ -183,8 +197,12 @@ struct LiveSystem; impl System for LiveSystem { type Root = Root; - fn assemble(&mut self, assembly: &Assembly) -> Result { - root::assemble(assembly) + fn assemble( + &mut self, + assembly: &Assembly, + deadline: Option, + ) -> Result { + root::assemble_until(assembly, deadline) } fn account(&mut self, root: &Root, identity: &Identity) -> Result<(), entry::Error> { @@ -196,17 +214,21 @@ impl System for LiveSystem { } } -/// Applies `record` with `template` through `system`, as [`apply`](fn@apply) does, and returns -/// the root that it assembled, and the identity in which every execution runs, if any. +/// Applies `record` with `template` by `deadline`, if any, through `system`, as +/// [`apply`](fn@apply) does, and returns the root that it assembled, and the identity in which +/// every execution runs, if any. fn apply_with( system: &mut S, record: &ActivationRecord, template: &RootTemplate, + deadline: Option, ) -> Result<(S::Root, Option), ApplyError> { let configuration = record.configuration(); let assembly = assembly(template, configuration)?; let identity = configuration.workload.identity.map(workload_identity); - let root = system.assemble(&assembly).map_err(ApplyError::Assembly)?; + let root = system + .assemble(&assembly, deadline) + .map_err(ApplyError::Assembly)?; if let Some(identity) = &identity && let Err(error) = system.account(&root, identity) { diff --git a/nvx-rs/src/guest/clone/activation/apply/tests.rs b/nvx-rs/src/guest/clone/activation/apply/tests.rs index 02e78650d..e832d7ae9 100644 --- a/nvx-rs/src/guest/clone/activation/apply/tests.rs +++ b/nvx-rs/src/guest/clone/activation/apply/tests.rs @@ -84,6 +84,8 @@ struct FakeSystem { failing: bool, /// What the clone did, in order. effects: Vec, + /// The deadline of each assembly, in order, by which it waits for its devices. + deadlines: Vec>, } impl FakeSystem { @@ -98,6 +100,7 @@ impl FakeSystem { image, failing: false, effects: Vec::new(), + deadlines: Vec::new(), } } @@ -124,8 +127,13 @@ impl FakeSystem { impl System for FakeSystem { type Root = PathBuf; - fn assemble(&mut self, assembly: &Assembly) -> Result { + fn assemble( + &mut self, + assembly: &Assembly, + deadline: Option, + ) -> Result { self.effects.push(Effect::Assemble(assembly.clone())); + self.deadlines.push(deadline); if self.failing { // An assembly refuses a relative runtime directory before it does anything else, // which gives the fake a real failure. @@ -202,7 +210,7 @@ impl Adapter { cpus: CPUS, memory: MEMORY, }; - let activation = Activation::of( + let activation = Activation::with_resources( resources, Some(acknowledged), fence.clone(), @@ -234,7 +242,9 @@ impl Adapter { ActivationAction::Acknowledge | ActivationAction::Continue | ActivationAction::Join => { } ActivationAction::Apply(record) => { - let result = match apply_with(&mut self.system, &record, &template()) { + let deadline = self.activation.deadline(); + let applied = apply_with(&mut self.system, &record, &template(), deadline); + let result = match applied { Ok(applied) => { self.applied = Some(applied); Ok(()) @@ -543,6 +553,8 @@ fn a_valid_activate_activates_the_root_and_the_workload_once_before_ready() { Effect::Spawn, ] ); + // The assembly waited for the devices until the activation's deadline at the latest. + assert_eq!(adapter.system.deadlines, [Some(base + TIMEOUT)]); } #[test] @@ -585,7 +597,8 @@ fn assembly_and_identity_inputs_are_taken_exactly_from_the_record() { for identity in [None, Some(WORKLOAD)] { let record = clone_record(configuration(image, upper, overlay, identity)); let mut system = FakeSystem::new(); - let (root, applied) = apply_with(&mut system, &record, &template).unwrap(); + let (root, applied) = + apply_with(&mut system, &record, &template, None).unwrap(); assert_eq!(root, system.root()); let assembly = Assembly { runtime: template.runtime.clone(), @@ -631,8 +644,10 @@ fn assembly_and_identity_inputs_are_taken_exactly_from_the_record() { }; let other = ActivationRecord::new(None, other).unwrap(); let (mut first, mut second) = (FakeSystem::new(), FakeSystem::new()); - let first_identity = apply_with(&mut first, &accepted(), &template).unwrap().1; - let second_identity = apply_with(&mut second, &other, &template).unwrap().1; + let first_identity = apply_with(&mut first, &accepted(), &template, None) + .unwrap() + .1; + let second_identity = apply_with(&mut second, &other, &template, None).unwrap().1; assert_eq!(first.effects, second.effects); assert_eq!(first_identity, second_identity); } @@ -713,7 +728,7 @@ fn a_workload_identity_without_an_account_in_the_image_is_refused_before_ready() None, )); let mut system = FakeSystem::with(b"", b""); - let (_, identity) = apply_with(&mut system, &anyone, &template()).unwrap(); + let (_, identity) = apply_with(&mut system, &anyone, &template(), None).unwrap(); assert_eq!(identity, None); assert_eq!(system.effects, [Effect::Assemble(accepted_assembly())]); } @@ -759,12 +774,15 @@ fn the_application_refuses_without_privileges_what_it_cannot_assemble() { activation::OverlayMode::TrustedLowers, Some(WORKLOAD), )); - assert!(matches!(apply(&disk, &template()), Err(ApplyError::Gpt))); + assert!(matches!( + apply(&disk, &template(), None), + Err(ApplyError::Gpt) + )); let relative = RootTemplate { runtime: PathBuf::from("run/nvx"), ..template() }; - let Err(ApplyError::Assembly(error)) = apply(&accepted(), &relative) else { + let Err(ApplyError::Assembly(error)) = apply(&accepted(), &relative, None) else { panic!("a relative runtime directory was assembled"); }; assert!( diff --git a/nvx-rs/src/guest/clone/activation/tests.rs b/nvx-rs/src/guest/clone/activation/tests.rs index af51896bd..7554724aa 100644 --- a/nvx-rs/src/guest/clone/activation/tests.rs +++ b/nvx-rs/src/guest/clone/activation/tests.rs @@ -132,7 +132,7 @@ fn reported() -> ReadyActivation { /// Returns the activation, in its transport bootstrap, of the clone behind `fence`, whose repair /// acknowledged its restore at `acknowledged`. fn bootstrap(fence: &FakeFence, acknowledged: Instant) -> Activation { - Activation::of( + Activation::with_resources( resources(), Some(acknowledged), fence.clone(), @@ -394,6 +394,54 @@ fn a_changed_record_or_a_new_transaction_is_final() { } } +#[test] +fn an_activate_too_long_to_read_fails_as_a_record_that_long() { + let t0 = Instant::now(); + // In the transport bootstrap, as any Activate. + let fence = FakeFence::default(); + let mut activation = bootstrap(&fence, t0); + let failure = activation + .input(after(t0, 1), ActivationInput::TooLong(MAX_RECORD_BYTES + 1)) + .unwrap_err(); + assert_eq!(failure.state(), ActivationState::Bootstrap); + assert!(matches!(failure.cause(), ActivationCause::Request)); + assert_final(&mut activation, &fence, &failure, after(t0, 2), 0); + // From then on, as an Activate whose record has that many bytes, which no record has: while + // restricted, while the activation runs, and once it is ready, or started, whose fence stays + // open. + let states: [(Setup, u32); 4] = [(restricted, 0), (activating, 0), (ready, 1), (started, 1)]; + for (activation, opened) in states { + for length in [MAX_RECORD_BYTES + 1, usize::MAX] { + let fence = FakeFence::default(); + let mut activation = activation(&fence, t0); + let state = activation.state(); + let failure = activation + .input(after(t0, 1), ActivationInput::TooLong(length)) + .unwrap_err(); + assert_eq!(failure.state(), state); + assert!( + matches!( + failure.cause(), + ActivationCause::Record(ActivationError::TooLong(long)) if *long == length + ), + "{state}: {failure}" + ); + assert_final(&mut activation, &fence, &failure, after(t0, 2), opened); + } + } + // Past the deadline, before the clone is ready, it fails as every input does. + let fence = FakeFence::default(); + let mut activation = restricted(&fence, t0); + let failure = activation + .input( + after(t0, 10), + ActivationInput::TooLong(MAX_RECORD_BYTES + 1), + ) + .unwrap_err(); + assert!(matches!(failure.cause(), ActivationCause::Expired)); + assert_final(&mut activation, &fence, &failure, after(t0, 11), 0); +} + #[test] fn a_failed_application_is_final_and_no_retry_applies_the_record_again() { let t0 = Instant::now(); @@ -507,7 +555,7 @@ fn the_deadline_starts_at_the_acknowledgement_and_retries_never_extend_it() { let fence = FakeFence::default(); // The repair acknowledged the restore at `t0`, and the adapter made the activation later. let mut activation = - Activation::of(resources(), Some(t0), fence.clone(), TIMEOUT, after(t0, 3)); + Activation::with_resources(resources(), Some(t0), fence.clone(), TIMEOUT, after(t0, 3)); assert_eq!(activation.deadline(), Some(after(t0, 10))); activation .input(after(t0, 8), ActivationInput::Reset) @@ -548,7 +596,8 @@ fn an_ungated_restore_s_deadline_starts_at_the_activation() { let t0 = Instant::now(); let fence = FakeFence::default(); // Without an acknowledgement, the deadline starts when the adapter makes the activation. - let mut activation = Activation::of(resources(), None, fence.clone(), TIMEOUT, after(t0, 5)); + let mut activation = + Activation::with_resources(resources(), None, fence.clone(), TIMEOUT, after(t0, 5)); assert_eq!(activation.deadline(), Some(after(t0, 15))); activation.tick(after(t0, 14)).unwrap(); let failure = activation @@ -558,7 +607,8 @@ fn an_ungated_restore_s_deadline_starts_at_the_activation() { assert!(matches!(failure.cause(), ActivationCause::Expired)); assert_final(&mut activation, &fence, &failure, after(t0, 16), 0); // A deadline that no instant can hold never passes. - let mut activation = Activation::of(resources(), Some(t0), fence.clone(), Duration::MAX, t0); + let mut activation = + Activation::with_resources(resources(), Some(t0), fence.clone(), Duration::MAX, t0); assert_eq!(activation.deadline(), None); activation.tick(after(t0, 1 << 30)).unwrap(); let reset = activation.input(after(t0, 1 << 30), ActivationInput::Reset); @@ -956,7 +1006,7 @@ fn failures_name_their_state_and_their_cause() { #[test] fn an_activation_debugs_its_state_resources_and_deadline_without_its_fence() { let t0 = Instant::now(); - let activation = Activation::of( + let activation = Activation::with_resources( ExpectedResources { cpus: 4, memory: 2 << 30, diff --git a/nvx-rs/src/guest/root.rs b/nvx-rs/src/guest/root.rs index 5b506c4b7..f6a8bfff2 100644 --- a/nvx-rs/src/guest/root.rs +++ b/nvx-rs/src/guest/root.rs @@ -170,7 +170,7 @@ use std::io; use std::os::unix::ffi::OsStrExt; use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; -use std::time::Duration; +use std::time::{Duration, Instant}; use self::kernel::{Kernel, SystemKernel}; use super::gpt::DeviceError; @@ -484,14 +484,30 @@ impl OverlayMode { /// /// A failure unmounts what the assembly mounted, as [`Root::teardown`] does, and reports both. pub fn assemble(assembly: &Assembly) -> Result { - assemble_with(Box::new(SystemKernel), &Paths::default(), assembly) + assemble_until(assembly, None) } -/// Implements [`assemble`] with `kernel`, and with the kernel's interfaces at `paths`. +/// Assembles the layered root of `assembly`, as [`assemble`] does, but waits for each device until +/// `deadline` at the latest, if any, however long `assembly`'s timeout allows: a device that has +/// not appeared by then is missing, as one that has not appeared by the timeout is. +pub(crate) fn assemble_until( + assembly: &Assembly, + deadline: Option, +) -> Result { + assemble_with( + Box::new(SystemKernel), + &Paths::default(), + assembly, + deadline, + ) +} + +/// Implements [`assemble_until`] with `kernel`, and with the kernel's interfaces at `paths`. fn assemble_with( kernel: Box, paths: &Paths, assembly: &Assembly, + deadline: Option, ) -> Result { let runtime = assembly.runtime.as_os_str().as_bytes(); if below_its_root(&assembly.runtime, true).is_none() @@ -520,7 +536,7 @@ fn assemble_with( latched: false, reservation: None, }; - match root.mount_layers(paths, assembly) { + match root.mount_layers(paths, assembly, deadline) { Ok(()) => Ok(root), Err(error) => Err(AssemblyError { error, @@ -664,8 +680,14 @@ impl Root { self.unmount() } - /// Implements [`assemble`]'s steps after the first. - fn mount_layers(&mut self, paths: &Paths, assembly: &Assembly) -> Result<(), Error> { + /// Implements [`assemble`]'s steps after the first, waiting for each device until `deadline` + /// at the latest, if any. + fn mount_layers( + &mut self, + paths: &Paths, + assembly: &Assembly, + deadline: Option, + ) -> Result<(), Error> { let runtime = self.runtime.clone(); fs::create_dir_all(&runtime).map_err(|error| Error::MountPoint { path: runtime.clone(), @@ -736,6 +758,7 @@ impl Root { Role::Distro, assembly.distro, assembly.timeout, + deadline, )?; let lowers = match &assembly.image { Image::Erofs { uuid } => { @@ -760,6 +783,7 @@ impl Root { Role::Scratch, device, assembly.timeout, + deadline, )?; if journal == Journal::Forbidden { check_no_journal(&device)?; diff --git a/nvx-rs/src/guest/root/block.rs b/nvx-rs/src/guest/root/block.rs index f79a37900..413b3e170 100644 --- a/nvx-rs/src/guest/root/block.rs +++ b/nvx-rs/src/guest/root/block.rs @@ -10,7 +10,7 @@ use std::io; use std::os::unix::ffi::OsStrExt; use std::os::unix::fs::FileExt; use std::path::{Path, PathBuf}; -use std::time::Duration; +use std::time::{Duration, Instant}; use super::kernel::{Kernel, Paths}; use super::{Device, DeviceStep, Error, Role}; @@ -58,18 +58,19 @@ pub(super) struct Ext4Features { } /// Finds the block device of `role` that the VMM attached at `device`'s MMIO window, polling -/// until it appears for as long as `timeout` allows, checks that its access mode is the role's, -/// read-only for the distro and writable for the scratch, checks its size if the caller expects -/// one, and flushes its buffers, so that later reads come from the device. Returns the path of -/// its node. +/// until it appears for as long as `timeout` allows, but never past `deadline`, if any, checks +/// that its access mode is the role's, read-only for the distro and writable for the scratch, +/// checks its size if the caller expects one, and flushes its buffers, so that later reads come +/// from the device. Returns the path of its node. pub(super) fn prepare( kernel: &mut dyn Kernel, paths: &Paths, role: Role, device: Device, timeout: Duration, + deadline: Option, ) -> Result { - let (name, path) = resolve(kernel, paths, role, device.mmio, timeout)?; + let (name, path) = resolve(kernel, paths, role, device.mmio, timeout, deadline)?; let attributes = paths.sys.join("class/block").join(&name); let read_only = fs::read(attributes.join("ro")) .and_then(|text| match parse_count(&text)? { @@ -119,18 +120,31 @@ pub(super) fn prepare( } /// Returns the name and the node of the block device that the VMM attached at `mmio`, polling -/// for it every [`POLL_INTERVAL`] while `timeout` allows another poll. +/// for it every [`POLL_INTERVAL`] while `timeout` allows another poll, or, if `deadline` comes +/// first, until the deadline, at which it polls once more, after a shorter sleep: it sleeps no +/// longer than until the deadline, and polls no more once the deadline passed, however long a poll +/// or a sleep took. fn resolve( kernel: &mut dyn Kernel, paths: &Paths, role: Role, mmio: u64, timeout: Duration, + deadline: Option, ) -> Result<(String, PathBuf), Error> { - let polls = timeout.as_millis() / POLL_INTERVAL.as_millis() + 1; + let left = |deadline: Instant| deadline.saturating_duration_since(Instant::now()); + let polls = match deadline.map(left) { + Some(left) if left < timeout => left.as_nanos().div_ceil(POLL_INTERVAL.as_nanos()) + 1, + _ => timeout.as_millis() / POLL_INTERVAL.as_millis() + 1, + }; for poll in 0..polls { if poll > 0 { - kernel.sleep(POLL_INTERVAL); + let interval = match deadline.map(left) { + Some(left) if left.is_zero() => break, + Some(left) => POLL_INTERVAL.min(left), + None => POLL_INTERVAL, + }; + kernel.sleep(interval); } let found = find(paths, mmio).map_err(|error| Error::Lookup { role, mmio, error })?; if let Some(name) = found { diff --git a/nvx-rs/src/guest/root/tests.rs b/nvx-rs/src/guest/root/tests.rs index 03f70ebb8..6ba8c4670 100644 --- a/nvx-rs/src/guest/root/tests.rs +++ b/nvx-rs/src/guest/root/tests.rs @@ -7,7 +7,7 @@ use std::io; use std::os::unix::fs::{MetadataExt, PermissionsExt}; use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex}; -use std::time::Duration; +use std::time::{Duration, Instant}; use super::kernel::{Call, FakeKernel, Operation, Paths, super_options}; use super::*; @@ -151,7 +151,18 @@ impl Guest { /// Assembles `assembly` with a clone of the guest's kernel. fn assemble(&self, assembly: &Assembly) -> Result { - assemble_with(Box::new(self.kernel.clone()), &self.paths, assembly) + assemble_with(Box::new(self.kernel.clone()), &self.paths, assembly, None) + } + + /// Assembles `assembly` with a clone of the guest's kernel, waiting for each device until + /// `deadline` at the latest. + fn assemble_until( + &self, + assembly: &Assembly, + deadline: Instant, + ) -> Result { + let kernel = Box::new(self.kernel.clone()); + assemble_with(kernel, &self.paths, assembly, Some(deadline)) } /// Sets the overlay module's parameter `name`, the default of one of the overlay's options: @@ -520,6 +531,106 @@ fn devices_are_found_when_they_appear_and_missing_ones_time_out() { assert!(guest.kernel.unmounts().is_empty()); } +#[test] +fn an_assembly_waits_for_its_devices_until_its_deadline_at_the_latest() { + // However long its timeout, an assembly whose deadline passed looks for each device once, + // without sleeping, and finds a missing one missing, before it mounts anything. + let guest = Guest::bare(); + attach(&guest.base, 0, "vda", 4096); + let mut assembly = guest.assembly(); + assembly.timeout = Duration::from_secs(600); + let error = guest.assemble_until(&assembly, Instant::now()).unwrap_err(); + assert!( + matches!( + error.error(), + Error::Missing { + role: Role::Scratch, + mmio: SCRATCH, + } + ), + "{error:?}" + ); + assert_eq!(guest.kernel.sleeps(), 0); + assert!(guest.kernel.mounts().is_empty()); + + // One whose deadline is a second away polls, 50 ms apart, until the deadline rather than + // until its timeout's end. + let guest = Guest::bare(); + attach(&guest.base, 0, "vda", 4096); + let deadline = Instant::now() + Duration::from_secs(1); + let error = guest.assemble_until(&assembly, deadline).unwrap_err(); + assert!( + matches!( + error.error(), + Error::Missing { + role: Role::Scratch, + .. + } + ), + "{error:?}" + ); + let sleeps = guest.kernel.sleeps(); + assert!((1..=20).contains(&sleeps), "{sleeps}"); + + // The deadline bounds the polls in time too, whatever a poll or a sleep takes: a sleep that + // overruns it is the last. + let guest = Guest::bare(); + attach(&guest.base, 0, "vda", 4096); + guest + .kernel + .on_sleep(|_| std::thread::sleep(Duration::from_millis(600))); + let deadline = Instant::now() + Duration::from_millis(500); + let error = guest.assemble_until(&assembly, deadline).unwrap_err(); + assert!( + matches!( + error.error(), + Error::Missing { + role: Role::Scratch, + .. + } + ), + "{error:?}" + ); + let sleeps = guest.kernel.sleeps(); + assert!(sleeps <= 1, "{sleeps}"); + + // A device that appears before the deadline is found. + let guest = Guest::bare(); + attach(&guest.base, 0, "vda", 4096); + let root_directory = guest.base.clone(); + guest.kernel.on_sleep(move |sleeps| { + if sleeps == 1 { + attach(&root_directory, 1, "vdb", 8192); + } + }); + let deadline = Instant::now() + Duration::from_secs(1); + let root = guest.assemble_until(&assembly, deadline).unwrap(); + assert_eq!(guest.kernel.sleeps(), 1); + root.teardown().unwrap(); + + // So is one that appears while the assembly sleeps less than an interval, until a deadline + // that comes before the interval ends, at which it polls once more. + let guest = Guest::bare(); + attach(&guest.base, 0, "vda", 4096); + let root_directory = guest.base.clone(); + guest.kernel.on_sleep(move |sleeps| { + if sleeps == 1 { + attach(&root_directory, 1, "vdb", 8192); + } + }); + let wait = Duration::from_millis(40); + let started = Instant::now(); + match guest.assemble_until(&assembly, started + wait) { + Ok(root) => { + assert_eq!(guest.kernel.sleeps(), 1); + root.teardown().unwrap(); + } + // Unless the test ran so late that the deadline passed before the assembly looked for + // the scratch. + Err(error) => assert!(started.elapsed() >= wait, "{error:?}"), + } +} + #[test] fn a_device_of_another_size_is_refused_before_it_is_flushed() { let guest = Guest::new(); @@ -862,7 +973,7 @@ fn a_runtime_directory_that_another_root_holds_is_refused() { if sleeps == 1 { attach(&directory, 0, "vda", 4096); let calls = kernel.calls().len(); - let result = assemble_with(Box::new(kernel.clone()), &paths, &assembly); + let result = assemble_with(Box::new(kernel.clone()), &paths, &assembly, None); *outcome.lock().unwrap() = Some((result.map(drop), kernel.calls().len() - calls)); } }); @@ -1087,7 +1198,8 @@ fn a_runtime_directory_where_a_directory_is_bound_is_refused() { runtime, ..guest.assembly() }; - let error = assemble_with(Box::new(SystemKernel), &guest.paths, &assembly).unwrap_err(); + let error = + assemble_with(Box::new(SystemKernel), &guest.paths, &assembly, None).unwrap_err(); assert!( matches!(error.error(), Error::Occupied { path } if *path == distro), "{error:?}" diff --git a/nvx-rs/src/protocol/status.rs b/nvx-rs/src/protocol/status.rs index 9e2c4a25e..b73444b1e 100644 --- a/nvx-rs/src/protocol/status.rs +++ b/nvx-rs/src/protocol/status.rs @@ -55,9 +55,9 @@ pub const REPAIR_FINISH: u8 = 45; /// The clone was not activated before its deadline, which starts at its repair's /// acknowledgement. pub const ACTIVATION_DEADLINE: u8 = 50; -/// The host broke the restricted session of a repaired clone before the clone was ready: it -/// reset the session again, sent a request before the activation, or sent an Activate of -/// another transaction, or one that changes the accepted record. +/// The host broke the session of a repaired clone's activation: before the clone was ready, it +/// reset the session again, or sent a request before the activation, and, before or after, it +/// sent an Activate of another transaction, or one that changes the accepted record. pub const ACTIVATION_PROTOCOL: u8 = 51; /// The Activate's record is refused: it does not decode, breaks a limit, or is a cold boot's, /// without the restore section of a clone's. diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 3d878d95e..d5479aa6b 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -55,7 +55,7 @@ length, a little-endian `u32`, then the message's kind: | 4 | `Stop`: asks the guest to stop serving | none | | 5 | `Stopped`: the guest's confirmation of `Stop`, its last message | none | | 6 | `RootConfig`: the host's [root configuration](#root-configurations) of a layered root, which the guest answers with `Ready` or `RootFailed` | the configuration, at most 4 KiB | -| 7 | `Ready`: the guest assembled the root | the guest agent's [build](#ready-mapping-tables-and-start), at most 4 KiB | +| 7 | `Ready`: the guest assembled the root, or activated a clone | the guest agent's [build](#ready-mapping-tables-and-start), and a clone's activation, at most 4 KiB | | 8 | `RootFailed`: a stage of the root failed: the guest's answer to `RootConfig`, or its report of a failed teardown before it confirms a `Stop` | the stage, then the problem, at most 4 KiB | | 9 | `Probe`: asks the guest to run a [probe](#probes) on its root | the probe, at most 4 KiB | | 10 | `Report`: the guest's answer to `Probe`: the facts that the probe found | at most 256 KiB | @@ -66,22 +66,27 @@ length, a little-endian `u32`, then the message's kind: | 24 | `MapFailed`: the guest refused the part, or failed to apply one of its mappings | how many mappings it applied, then the problem, at most 4 KiB | | 25 | `Start`: asks the guest to start the sandbox's workloads once it applied the mapping table, which it confirms with `Started` | none | | 26 | `Started`: the guest's confirmation of `Start` | none | +| 27 | `Activate`: the host's activation of a clone, which a restore of a [template's](#template-mode) snapshot started, and which the guest answers with `Ready` | a clone's activation record, as `nvx::activation::ActivationRecord::encode` writes it, at most 1 KiB | The session keeps no message boundaries, so a message may span several of its records, and one record may end a message and start the next. `Decoder` reassembles the messages. It checks each header before it takes any byte of the body, and holds at most one partial message. A malformed message ends the -stream, because the framing cannot find the next message after it. +stream, because the framing cannot find the next message after it. A clone's +`Activate` whose header announces more than 1 KiB fails the clone's +activation instead, as a record that long does. The framing carries each body as bytes, within its kind's limit. The bodies of kinds 6 to 22 and 24 have encodings of their own, which the `root`, `probe`, -and `exec` modules write and read. A body that does not decode leaves the -stream intact, at the start of the next message. The guest answers a root -configuration that does not decode with `RootFailed`, as a configuration that -it refuses, and a probe with `ProbeFailed`, while a message about an execution -that does not decode ends the worker's session, as one that breaks the -execution's rules does. The host program's client ends on an answer that does -not decode, since it cannot tell what the guest did. +and `exec` modules write and read, and that of kind 27 is `nvx::activation`'s. +A body that does not decode leaves the stream intact, at the start of the next +message. The guest answers a root configuration that does not decode with +`RootFailed`, as a configuration that it refuses, and a probe with +`ProbeFailed`, while a message about an execution that does not decode ends the +worker's session, as one that breaks the execution's rules does, and an +activation record that does not decode fails the clone's activation. The host +program's client ends on an answer that does not decode, since it cannot tell +what the guest did. ### Root configurations @@ -163,6 +168,13 @@ also the build that the host program ships unless it names another; see [The runtime identity](#the-runtime-identity). The host sends a guest agent of another build nothing more; see [The host's order](#the-hosts-order). +A clone's `Ready`, its answer to the host's `Activate`, also reports its +activation after the build, as `nvx::activation::ReadyActivation::encode` +writes it: 1, then the transaction ID of the record's restore, 16 bytes. +`root::Ready::with_activation` adds it, and `root::Ready::activation` returns +it. A guest that boots cold reports none, and `root::Ready::decode` refuses +bytes after the build that are not an activation. + The host's mapping table, of as many mappings as the root configuration announces, travels in parts, each in a `Maps`, whose body is a `MAPS` payload of `nvx::protocol`, as its `encode_maps_payloads` writes it: the index in the @@ -490,7 +502,10 @@ unmounts a root that it still holds, as `nvx::guest::root::Root` does when it drops, ignoring failures. A teardown that failed leaves the root mounted until the VM ends. The worker runs the root's operations through `worker::RootOperations`, and enters its executions through -`worker::EntryOperations`, so that tests run it against a fake root. +`worker::EntryOperations`, so that tests run it against a fake root. In a +template's VM, the worker captures the VM instead, and then activates and serves +each clone that a restore of the snapshot starts, as +[Template mode](#template-mode) describes. #### The mapping table and the start @@ -706,18 +721,30 @@ through `worker::capture::capture_on_terminal`: 1. opens the control tty, in raw mode, as the worker of a VM that boots cold does; -2. attaches the control session with `nvx::guest::nvxs::GuestConnection::attach`, - and waits until the tty has sent the attachment, so that the snapshot holds - the guest attached. It reads nothing from the session before the capture, - and serves no application session; -3. prepares the guest for the capture with +2. bootstraps the guest for the root of every clone, with the template's own + root configuration, `worker::capture::template_config`, the default one, + since the host sends a template's VM none: it mounts the cgroup v2 + hierarchy unless it is mounted, and prepares the agent's and the workload + cgroups, with their resource values, as the worker of a VM that boots cold + does before it assembles its root. The snapshot holds them, so that a + clone assembles its root without them; +3. attaches the control session with `nvx::guest::nvxs::GuestConnection::attach`, + takes the reset with which the broker answers the attachment at once, + acknowledges it with the guest agent's receive window, and waits until the + tty has sent the acknowledgement, so that the snapshot holds the guest's + session active, in the broker's first epoch. A clone then reads no reset but + the one that the restore queues, of another instance: the restore's broker + would finish sending a reset that the template's broker began to send, and + the clone would read one that the template did not. The worker reads nothing + else from the session before the capture, and serves no application session; +4. prepares the guest for the capture with `nvx::guest::clone::CapturePreparation`, which closes the worker's fence of workload admission, checks that nothing is mounted from the `distro` or the scratch block device, at the MMIO windows from which the worker assembles a root, from an overlay, or from OpenVMM's host filesystem device, whose tag is `microvm`, then waits for the time ABI's boot checks, runs its capture checks, and suppresses the stall detectors; -4. requests the capture with `CaptureBarrier::capture`, with the platform +5. requests the capture with `CaptureBarrier::capture`, with the platform tier's request, which keeps no scratch. Closing the worker's fence, `worker::capture::Admission`, acquires it: a close @@ -735,15 +762,70 @@ started the daemon in the template, with the time ABI's boot step, before the guest agent, so every clone holds it, and the guest agent's PID 1 collects its status. The repair takes the time ABI in strict mode, so that a violation of the ABI fails the repair, rather than stopping the VM, and the worker ends the -VM with the failure's status. The worker does not activate a clone yet: a -clone that it repaired ends at once, with its fence still closed, which may cut -the deferred time work short. A failure of that work stops the VM with the time -ABI's own status, since the daemon enforces the ABI. The worker, and with it -the VM, ends with one of these statuses, after a diagnostic on standard error: +VM with the failure's status. A failure of the deferred time work stops the VM +with the time ABI's own status, since the daemon enforces the ABI. + +Once it repaired a clone, the worker activates it, on the same tty, with +`worker::work_clone`, through the clone's `nvx::guest::clone::Activation`, +whose deadline ends `worker::activation::ACTIVATION_TIMEOUT`, 120 seconds, +after the repair's acknowledgement of the restore: + +1. It resumes the session that the template attached, and whose first epoch it + acknowledged, which the snapshot holds, and acknowledges the reset that the + restore queued, the only one that the clone reads, with its receive window: + it attaches no session. +2. It takes the host's `Activate`. The activation decodes its record, which + must be a clone's, with a restore section, and checks that it expects the + CPU count and the memory size that the repair found, before anything has a + side effect. +3. It applies the record with `nvx::guest::clone::apply`, with + `worker::capture::root_template` of the template's configuration: its + runtime directory, the `distro` and scratch block devices, and its device + timeout. The application assembles the root, an EROFS image over a tmpfs + or the scratch, in the record's overlay mode, waiting for each device until + the activation's deadline at the latest, so that it ends by then, and checks + the account of the record's workload identity, if it has one, in the + assembled root, in existing-account mode. It applies none of the record's + other settings: its export, its network, its hostname, and its machine ID. +4. Once it took what the session received while the application ran, which + came before `Ready`, it completes the activation, which opens the fence, + and answers `Ready`, which reports the activation after the guest agent's + build, to the `Activate` and to each retry of it, byte for byte, that came + meanwhile, which joins the application. Anything else that came meanwhile, + a reset included, fails the activation, as before the application, so the + host waits for `Ready` before it sends the mapping table or the start. +5. It then serves the host's order, as after a cold boot: the mapping table + that the record announces, the start, the executions, the probes, and the + stop. The record's workload identity, if it has one, is the whole + sandbox's, as a root configuration's is: every execution runs as it, and a + start that names an identity is refused. Without an export, the clone + refuses each part of a mapping table with `MapFailed`. + +Until the clone is ready, its session takes nothing but that `Activate`: +another reset, any other message, a `Stop` and a `Ping` included, an +`Activate` that the activation refuses, and the passing of the deadline fail +the activation finally, as does an application that failed, which nothing +retries. Once the clone is ready, a retry of the same `Activate`, byte for +byte, gets the same `Ready` without a second application, and any other +`Activate` fails the activation. A failed activation ends the clone with the +status of its class, which `worker::activation::failure_status` selects, and a +broker that closes the stream before the clone is ready ends the clone with +status 1, once the worker dropped its executions, and waited for its stream, +and then the tty, to take what it wrote, for at most +`worker::UNACTIVATED_DRAIN`, a second each, whatever the stream does, so that a +stream that takes nothing, as once the broker failed, does not keep the clone +from ending; the fence stays closed if the clone was not ready. Once the clone +is ready, a broker that closes the stream ends it as it ends the worker of a VM +that boots cold. The worker of a VM that boots cold refuses an `Activate` as +any message that it does not take in its state. The worker, and with it the VM, +ends with one of these statuses, after a diagnostic on standard error: | Status | When | | --- | --- | -| 1 | The control tty cannot be opened or put in raw mode, or the attachment cannot be sent | +| 0 | In a clone, the host stopped the worker once the clone was ready, after a teardown that succeeded | +| 1 | The control tty cannot be opened or put in raw mode, the attachment or the acknowledgement of the broker's reset cannot be sent, or the broker closes the session or sends anything but a reset first, or, in a clone, its session failed or the broker closed it | +| 13 | The bootstrap of the guest for the template's root failed, before the attachment | +| 15 | In a clone, the root's teardown failed once the host stopped the worker | | 20 | The kernel command line names the platform tier, but holds another parameter, a double quote, or `--`, or the worker cannot read it | | 21 | The fence of workload admission cannot be closed | | 22 | Something is mounted from the template's devices or from an overlay, or the mount table cannot be checked | @@ -755,13 +837,19 @@ the VM, ends with one of these statuses, after a diagnostic on standard error: | 28 | The stall detectors' settings cannot be restored once the capture ended without a snapshot | | 29 | A stage of the capture failed that this build of the worker does not know | | 30 | The capture ended without a snapshot, as without a destination, or once OpenVMM rejected it, and the template continued | -| 31 | A restore of the template's snapshot started this VM, a clone, which the worker repaired, and does not activate yet | | 40 | The clone's restore cannot be repaired: it holds no fresh entropy, or its capture finished it | | 41 | The clone's online CPUs cannot be read, are not a prefix of the CPUs, or do not meet the restore's processor target | | 42 | The restore's memory target is refused, before any memory block is added | | 43 | The activation of the restore's memory target failed | | 44 | The reseed of the kernel's random number generator failed | | 45 | The restore's finish failed, other than by a violation of the time ABI | +| 50 | The clone was not ready before its activation's deadline | +| 51 | The host broke the clone's session: it reset the session again, or sent another request, before the clone was ready, or an `Activate` of another transaction, or one that changes the accepted record | +| 52 | The `Activate`'s record does not decode, breaks a limit, or is a cold boot's | +| 53 | The record expects other resources than the clone has | +| 54 | The root's assembly failed, or the record's image is a GPT disk, which a clone does not assemble | +| 55 | The image refuses the record's workload identity: it holds no account of the identity, or not its group, or its account databases cannot be read | +| 58 | The worker's own order misused the activation: it completed an activation that did not run, or started the sandbox before the clone was ready, or again | | 193 | A conformance check of the time ABI failed in the clone's repair, such as the finish's, which needs a running time daemon | | 195 | The time ABI's part of the clone's repair failed, such as the validation of the restore packet, or the record of the restore | @@ -773,7 +861,9 @@ the time ABI could not repair, ends with the violation's status instead, as `worker::capture::capture_status` tells. `nvx::protocol::status` documents the statuses of a clone's repair, from 40 to 45, which `nvx::guest::clone::RepairError::status` selects, the time ABI's own, from 193 -to 195, and those that it reserves for a clone's activation, from 50 to 58. A +to 195, and those of a clone's activation, from 50 to 58, of which the worker +leaves 56 and 57, the export's and the network's, to the activation of those +settings. A stage whose failure may leave the worker some of the permission for the time ABI's ports fails with the time ABI's port error, and the worker then starts no thread or process before it ends. A violation of the time ABI in enforcing @@ -1404,12 +1494,13 @@ independent adapter's does: The unit tests of `frame`, `root`, `probe`, `exec`, and `template` run in every configuration. They cover each kind's encoding, messages in pieces of any size, a message of each kind split at every byte, the headers that the framing refuses -before their body arrives, and the root configurations, readinesses, parts of -mapping tables, failures, probes, facts, and messages of executions that decode, -and those that don't, and the host's side of an execution, which still sends the -cancellation of one that it gives up on in the step that ends it, without grace -or settlement time, and which cancels one whose stream outgrows the 16 MiB that -it keeps. Those of `template` check the platform tier's allowlist, token by +before their body arrives, and the root configurations, readinesses, with or +without a clone's activation, parts of mapping tables, failures, probes, facts, +and messages of executions that decode, and those that don't, and the host's +side of an execution, which still sends the cancellation of one that it gives up +on in the step that ends it, without grace or settlement time, and which cancels +one whose stream outgrows the 16 MiB that it keeps. Those of `template` check +the platform tier's allowlist, token by token, against the tokens that OpenVMM's capture of a platform snapshot allows and refuses, through NVX's copy of OpenVMM's check, which refuses a token that is not UTF-8 text, and admits by its prefix one that the kernel splits in two at @@ -1439,10 +1530,16 @@ Those of `worker::entries` cover the names of capabilities, the fields of a process's status, the outcomes of attempts, the entry's refusals, and the probe's arguments. Those of `worker::capture` check that closing the worker's fence acquires it, the status of each stage of a capture, which no other -outcome of the worker shares, a template's devices, and that the worker -attaches its session, and the stream sent the attachment, before the capture -runs, without reading anything of the session, and that no capture runs once -the attachment cannot be written or sent. +outcome of the worker shares, a template's devices, the template's own root +configuration, and the inputs of a clone's root that it gives, and that the +worker attaches its session, takes the broker's reset, and acknowledges it, +which the stream sent, before the capture runs, without reading anything more +of the session, and keeps the session, active in the broker's first epoch, and +that no capture runs once the attachment cannot be written, the broker closes +the stream or sends anything but a reset first, or the acknowledgement cannot +be sent. Those of `worker::activation` check the status of +each class of a clone's activation's failures, and that the activation's +timeout covers the wait for both block devices. `tests/guest_session.rs` checks, in the guest-only configuration, that the guest agent serves the control session with the adapter's own framing. A test @@ -1538,8 +1635,9 @@ the worker reports before it ends with the stage's status, without the operations of the later stages; the account check of a sandbox's workload identity, once the root is assembled, with the default hostname; a teardown that fails; before the root, the configuration and the `Stop` that it accepts, -and the messages that it refuses; a second configuration and a broker that -closes the stream, which leave the root that it assembled to its operations; +and the messages that it refuses, a clone's `Activate` among them; once the +root is ready, a second configuration, an `Activate`, and a broker that closes +the stream, which leave the root that it assembled to its operations; and a broker that closes only its end of the stream while the worker probes, which still receives the replies that the worker queued. Through the binary, it checks that `runtime-adapter-guest init TTY layered`, @@ -1646,6 +1744,67 @@ while its broker pauses, ends the worker's session as it drops its host, so that the worker drops its executions, and their programs end, which a test checks too. +On x86_64, `tests/guest_session.rs` also runs the worker of a clone, which a +restore of a template's snapshot started, with `worker::work_clone`, behind its +test broker, with the session that the template's VM attached, active in the +epoch of its broker's first reset, and the clone's activation, +`nvx::guest::clone::Activation::with_resources`, behind a closed fence. The fake +root applies each record as `nvx::guest::clone::apply` does, without a root: it +refuses a GPT disk, takes as long as a test asks, but no longer than until the +activation's deadline, as the waits for the devices do, and checks the account +of the record's workload identity in the tests' directory of account databases. +The tests cover: + +- the reset that the restore queued, of another instance, which the worker + acknowledges without an attachment, and the `Activate`, whose record it + applies once, with the template's inputs, and answers with the `Ready` that + reports the activation; +- retries of the `Activate`, in records of 7 bytes, after a later reset, and + once the sandbox started, which get the same `Ready` without a second + application, and the order that the worker serves once the clone is ready, + as after a cold boot; +- the messages before the `Activate`, a stop and a ping among them, and another + reset, which fail the activation with the status of the host's session, with + nothing answered or applied; +- a record that does not decode, a cold boot's, and one that expects other + resources, which fail the activation with nothing applied, and a GPT disk and + a workload identity whose account or group the image lacks, which fail its + application, each with the status of its class, and no `Ready`; +- an `Activate` whose header announces more than 1 KiB, which the framing + refuses before it takes any byte of its body, and which fails the activation + with the record's status, before the clone is ready and once it is; +- the activation's deadline, whether the restore's reset came or not, or part + of an `Activate`, which the broker sends before the worker runs, so that the + worker takes it at once, however late it runs: the deadline fails the + activation with nothing applied, and ends nothing once the clone is ready; +- an application that would outlast the activation's deadline, which gets that + deadline, and which ends the clone with the deadline's status once the + deadline passed, with no `Ready`; +- a reset, a request, and a retry of the `Activate` that the host sends while + the application runs, which the worker takes before it completes the + activation: the reset and the request fail it, with the status of the host's + session and no `Ready`, and the retry joins it, so that both get `Ready`, + after one application; +- a stream that takes nothing more from the worker, as once the broker failed, + after which the worker still ends once its activation's deadline passed, + with the deadline's status, or once the broker closed its end of the stream + before the clone was ready, with status 1, once it waited for the stream for + `worker::UNACTIVATED_DRAIN`; +- a broker that closes the stream before the clone is ready, whether the + restore's reset came or not, or part of an `Activate`, which ends the clone + with status 1, with nothing applied, and once the clone is ready, which ends + it as it ends the worker of a VM that boots cold; +- a broker that sent the restore's reset, and a ping or nothing more, and is + gone before the worker runs, so that the worker cannot write its + acknowledgement: the worker still ends with the activation's status, or with + status 1, rather than with the failure of its stream; +- an `Activate` of another transaction, or of the same transaction with + another record, once the clone is ready, or its sandbox started, which fails + the activation with the status of the host's session; +- the record's workload identity, as which every execution runs, while a start + that names an identity is refused with `EINVAL`, as is a start that names + none once the record has none. + `tests/initramfs.rs` and `tests/runtime.rs` check, in the host-only configuration, on Linux and Windows: diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs b/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs index 205748826..c6e49517e 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs @@ -22,9 +22,11 @@ //! session failed or the broker closed the tty, each but the first after a diagnostic on //! standard error. In a template's VM, whose kernel command line names the platform snapshot //! tier, the worker captures the VM before it assembles any root instead, on x86_64, as -//! `runtime_adapter::worker::capture` describes: it exits with 20 to 30, or, in a clone, which -//! it repairs, with 31 once it repaired the clone, with 40 to 45, 193, or 195 once the repair -//! failed, as `nvx::protocol::status` documents, or with 1 once its session failed, each after a +//! `runtime_adapter::worker::capture` describes: it exits with 13 once the template's bootstrap +//! failed, or with 20 to 30. In a clone, which it repairs, then activates and serves, it exits +//! with 40 to 45, 193, or 195 once the repair failed, with 50 to 58 once the clone's activation +//! failed, as `nvx::protocol::status` documents, and, once the clone was ready, as the worker of +//! a VM that boots cold does. It exits with 1 once its session failed. Each comes after a //! diagnostic on standard error, unless a committed capture ended the VM; see //! `runtime_adapter::template`. //! - `runtime-adapter-guest idle` sleeps until a signal ends it. @@ -158,9 +160,13 @@ fn worker(tty: &std::path::Path, sibling: Option) -> ExitCode { match &worked { Worked::Stopped => {} Worked::Failed(failure) => eprintln!("runtime-adapter-guest: {failure}"), - Worked::Closed => { + Worked::Closed | Worked::Abandoned => { eprintln!("runtime-adapter-guest: the broker closed {}", tty.display()); } + // Only a clone's worker activates, in `capture_template`. + Worked::Unactivated { problem, .. } => { + eprintln!("runtime-adapter-guest: {problem}"); + } } worked.status() } @@ -176,10 +182,10 @@ fn worker(tty: &std::path::Path, sibling: Option) -> ExitCode { } /// Reads the kernel command line through `operations`, before anything else, and captures a -/// template's VM on the tty `tty`, as `runtime_adapter::worker::capture` describes, then repairs -/// a clone that a restore of its snapshot started. Returns the status with which the worker ends -/// then, after a diagnostic on standard error, or `None` in a VM that boots cold, whose worker -/// serves the host. +/// template's VM on the tty `tty`, with `operations`, as `runtime_adapter::worker::capture` +/// describes, then repairs a clone that a restore of its snapshot started, and serves it through +/// its activation. Returns the status with which the worker ends then, after a diagnostic on +/// standard error, or `None` in a VM that boots cold, whose worker serves the host. /// /// A command line that the worker cannot read, or that `runtime_adapter::template::boot` refuses, /// ends the worker with `COMMAND_LINE_STATUS`. @@ -203,7 +209,7 @@ fn capture_template( return Some(COMMAND_LINE_STATUS); } } - Some(match capture::capture_on_terminal(tty) { + Some(match capture::capture_on_terminal(tty, operations) { Ok(templated) => { eprintln!("runtime-adapter-guest: {templated}"); templated.status() diff --git a/tests/runtime-adapter/src/frame.rs b/tests/runtime-adapter/src/frame.rs index aafd7bd97..cfddb0941 100644 --- a/tests/runtime-adapter/src/frame.rs +++ b/tests/runtime-adapter/src/frame.rs @@ -14,11 +14,13 @@ //! The framing carries the body of each kind as bytes, within the kind's limit. The bodies of the //! messages of a layered root, of its probes, and of executions have encodings of their own, which //! the [`root`](crate::root), [`probe`](crate::probe), and [`exec`](crate::exec) modules write and -//! read: a body that does not decode is a [`BodyError`], which the receiver answers or refuses, -//! while the stream goes on. +//! read, and that of a clone's activation is the canonical encoding of an activation record, which +//! `nvx::activation` writes and reads: a body that does not decode is a [`BodyError`], which the +//! receiver answers or refuses, while the stream goes on. use std::fmt; +use nvx::activation::MAX_RECORD_BYTES; use nvx::protocol::OUTER_MAX_PAYLOAD; /// Bytes of a message's header: the body's length, then the message's kind. @@ -37,6 +39,10 @@ pub const MAX_TEXT_BODY: usize = 4096; /// fields; see [`exec`](crate::exec). pub const MAX_CONTROL_BODY: usize = 64; +/// Most bytes of the body of an `Activate`: the canonical encoding of an activation record, which +/// holds at most `nvx::activation::MAX_RECORD_BYTES`. +pub const MAX_ACTIVATE_BODY: usize = MAX_RECORD_BYTES; + /// A message of the adapter's framing. #[derive(Clone, Debug, PartialEq, Eq)] pub enum Message { @@ -54,8 +60,9 @@ pub enum Message { /// encoded as a [`RootConfig`](crate::root::RootConfig). The guest decodes and checks it /// itself, so that it answers one that it refuses with [`Message::RootFailed`]. RootConfig(Vec), - /// The guest's answer to [`Message::RootConfig`] once it has assembled the root, with the - /// guest agent's build, encoded as a [`Ready`](crate::root::Ready). + /// The guest's answer to [`Message::RootConfig`] once it has assembled the root, or to + /// [`Message::Activate`] once it activated a clone, with the guest agent's build, and what it + /// reports about a clone's activation, encoded as a [`Ready`](crate::root::Ready). Ready(Vec), /// A stage of the guest's layered root failed, encoded as a /// [`RootFailure`](crate::root::RootFailure): the guest's answer to [`Message::RootConfig`], @@ -107,6 +114,11 @@ pub enum Message { Start, /// The guest's confirmation of [`Message::Start`]. Started, + /// The host's activation of a clone, which a restore of a template's snapshot started: the + /// clone's activation record, in its canonical encoding, which + /// `nvx::activation::ActivationRecord::encode` writes, and which the guest decodes and checks + /// itself. The guest answers it with [`Message::Ready`] once it activated the clone. + Activate(Vec), } impl Message { @@ -119,7 +131,8 @@ impl Message { /// its kind allows: [`MAX_BODY`] for an `Echo`, a `Report`, an execution's start, or its input /// or output, or a part of the mapping table, [`MAX_TEXT_BODY`] for a root configuration, the /// guest's readiness, a root's failure, a failure of the mapping table, a probe, or a probe's - /// failure, [`MAX_CONTROL_BODY`] for an execution's other messages, and none for the others. + /// failure, [`MAX_CONTROL_BODY`] for an execution's other messages, [`MAX_ACTIVATE_BODY`] for + /// an `Activate`, and none for the others. pub fn encode(&self) -> Result, FrameError> { let (kind, body) = self.parts(); if body.len() > kind.max_body() { @@ -166,6 +179,7 @@ impl Message { Self::MapFailed(body) => (Kind::MapFailed, body), Self::Start => (Kind::Start, &[]), Self::Started => (Kind::Started, &[]), + Self::Activate(body) => (Kind::Activate, body), } } } @@ -222,6 +236,8 @@ const KIND_MAP_FAILED: u8 = 24; const KIND_START: u8 = 25; /// Kind of a [`Message::Started`]. const KIND_STARTED: u8 = 26; +/// Kind of a [`Message::Activate`]. +const KIND_ACTIVATE: u8 = 27; /// Kind of a [`Message`]. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -279,6 +295,8 @@ enum Kind { Start = KIND_START, /// [`Message::Started`]. Started = KIND_STARTED, + /// [`Message::Activate`]. + Activate = KIND_ACTIVATE, } impl Kind { @@ -311,6 +329,7 @@ impl Kind { KIND_MAP_FAILED => Some(Self::MapFailed), KIND_START => Some(Self::Start), KIND_STARTED => Some(Self::Started), + KIND_ACTIVATE => Some(Self::Activate), _ => None, } } @@ -342,6 +361,7 @@ impl Kind { | Self::ExecEnd | Self::ExecInputClosed | Self::ExecTerminal => MAX_CONTROL_BODY, + Self::Activate => MAX_ACTIVATE_BODY, Self::Ping | Self::Pong | Self::Stop @@ -382,6 +402,7 @@ impl Kind { Self::MapFailed => Message::MapFailed(body), Self::Start => Message::Start, Self::Started => Message::Started, + Self::Activate => Message::Activate(body), } } } @@ -393,7 +414,7 @@ pub enum FrameError { /// A kind that the framing does not define. Kind(u8), /// A body longer than its kind allows: [`MAX_BODY`] bytes, [`MAX_TEXT_BODY`], - /// [`MAX_CONTROL_BODY`], or none. + /// [`MAX_CONTROL_BODY`], [`MAX_ACTIVATE_BODY`], or none. Length { /// Kind of the message. kind: u8, @@ -536,7 +557,8 @@ impl Decoder { #[cfg(test)] mod tests { use super::{ - Decoder, FrameError, HEADER_LEN, Kind, MAX_BODY, MAX_CONTROL_BODY, MAX_TEXT_BODY, Message, + Decoder, FrameError, HEADER_LEN, Kind, MAX_ACTIVATE_BODY, MAX_BODY, MAX_CONTROL_BODY, + MAX_TEXT_BODY, Message, }; /// Returns one message of each kind, with bodies of no, few, and the most bytes. @@ -577,6 +599,9 @@ mod tests { Message::MapFailed(vec![b'm'; MAX_TEXT_BODY]), Message::Start, Message::Started, + Message::Activate(Vec::new()), + Message::Activate(b"\x01\x00record".to_vec()), + Message::Activate(vec![b'a'; MAX_ACTIVATE_BODY]), Message::Stop, Message::Stopped, ] @@ -613,6 +638,7 @@ mod tests { Message::MapFailed(body(24)), Message::Start, Message::Started, + Message::Activate(body(27)), ] } @@ -702,6 +728,10 @@ mod tests { ); assert_eq!(Message::Start.encode().unwrap(), [0, 0, 0, 0, 25]); assert_eq!(Message::Started.encode().unwrap(), [0, 0, 0, 0, 26]); + assert_eq!( + Message::Activate(b"x".to_vec()).encode().unwrap(), + [1, 0, 0, 0, 27, b'x'] + ); // The kinds are numbered from 1, without gaps. for (kind, message) in (1..).zip(one_of_each()) { assert_eq!(message.kind(), kind, "{message:?}"); @@ -714,7 +744,7 @@ mod tests { #[test] fn each_kind_decodes_from_its_code_alone() { for code in 0..=u8::MAX { - let defined = (1..=26).contains(&code).then_some(code); + let defined = (1..=27).contains(&code).then_some(code); assert_eq!(Kind::from_code(code).map(Kind::code), defined, "{code}"); } } @@ -769,11 +799,12 @@ mod tests { let too_long = u32::try_from(MAX_BODY + 1).unwrap().to_le_bytes(); let too_long_text = u32::try_from(MAX_TEXT_BODY + 1).unwrap().to_le_bytes(); let too_long_control = u32::try_from(MAX_CONTROL_BODY + 1).unwrap().to_le_bytes(); + let too_long_record = u32::try_from(MAX_ACTIVATE_BODY + 1).unwrap().to_le_bytes(); let header = |length: [u8; 4], kind: u8| [length[0], length[1], length[2], length[3], kind]; let text = |kind: u8| header(too_long_text, kind); let mut cases = vec![ ([0, 0, 0, 0, 0], FrameError::Kind(0)), - ([0, 0, 0, 0, 27], FrameError::Kind(27)), + ([0, 0, 0, 0, 28], FrameError::Kind(28)), ([9, 0, 0, 0, 0xff], FrameError::Kind(0xff)), ( [0xff, 0xff, 0xff, 0xff, 3], @@ -782,6 +813,13 @@ mod tests { length: u32::MAX as usize, }, ), + ( + header(too_long_record, 27), + FrameError::Length { + kind: 27, + length: MAX_ACTIVATE_BODY + 1, + }, + ), ]; // The kinds without a body. for kind in [1, 2, 4, 5, 23, 25, 26] { @@ -835,6 +873,12 @@ mod tests { assert_eq!(maps.len(), HEADER_LEN + MAX_BODY); let ready = Message::Ready(vec![0; MAX_TEXT_BODY]).encode().unwrap(); assert_eq!(ready.len(), HEADER_LEN + MAX_TEXT_BODY); + // An Activate carries the largest record, whose limit is the activation record's own. + assert_eq!(MAX_ACTIVATE_BODY, nvx::activation::MAX_RECORD_BYTES); + let activate = Message::Activate(vec![0; MAX_ACTIVATE_BODY]) + .encode() + .unwrap(); + assert_eq!(activate.len(), HEADER_LEN + MAX_ACTIVATE_BODY); let cases = [ (Message::Echo(vec![0; MAX_BODY + 1]), 3, MAX_BODY + 1), (Message::Report(vec![0; MAX_BODY + 1]), 10, MAX_BODY + 1), @@ -882,6 +926,11 @@ mod tests { 24, MAX_TEXT_BODY + 1, ), + ( + Message::Activate(vec![0; MAX_ACTIVATE_BODY + 1]), + 27, + MAX_ACTIVATE_BODY + 1, + ), ]; for (message, kind, length) in cases { assert_eq!( diff --git a/tests/runtime-adapter/src/guest.rs b/tests/runtime-adapter/src/guest.rs index 1e006ae95..ae67eb049 100644 --- a/tests/runtime-adapter/src/guest.rs +++ b/tests/runtime-adapter/src/guest.rs @@ -9,6 +9,9 @@ use std::fmt; use std::io::{self, Read, Write}; use std::path::Path; +use std::sync::mpsc; +use std::thread; +use std::time::Duration; use nvx::guest::nvxs::{GuestConnection, GuestConnectionError}; use nvx::guest::terminal::Terminal; @@ -210,6 +213,20 @@ pub fn serve(stream: S) -> Result { pub fn on_terminal( path: &Path, run: impl FnOnce(&mut Terminal) -> Result, +) -> Result { + on_terminal_within(path, |terminal| run(terminal).map(|ran| (ran, None))) +} + +/// Opens the tty at `path` as [`on_terminal`] does, and returns what `run` returns with it, but +/// waits until the tty has sent what `run` wrote for at most the time that `run` returns beside +/// its result, if it returns one: a thread of its own then waits for the tty, and holds it until +/// the tty has sent everything, or the process ends, so that a tty that sends nothing does not +/// keep the caller from ending. +/// +/// It fails as [`on_terminal`] does. +pub fn on_terminal_within( + path: &Path, + run: impl FnOnce(&mut Terminal) -> Result<(T, Option), ServeError>, ) -> Result { let failed = |action: String| { move |error: io::Error| { @@ -221,9 +238,24 @@ pub fn on_terminal( terminal .configure_raw() .map_err(failed(format!("cannot put {} in raw mode", path.display())))?; - let ran = run(&mut terminal)?; + let (ran, limit) = run(&mut terminal)?; // As NVX's managed agent does, a drain that fails is no failure of serving, which ended. - let _ = terminal.drain(); + match limit { + None => { + let _ = terminal.drain(); + } + Some(limit) => { + let (drained, done) = mpsc::channel(); + let drain = move || { + let _ = terminal.drain(); + let _ = drained.send(()); + }; + // Without a thread for it, the tty is not drained, which could take longer. + if thread::Builder::new().spawn(drain).is_ok() { + let _ = done.recv_timeout(limit); + } + } + } Ok(ran) } diff --git a/tests/runtime-adapter/src/lib.rs b/tests/runtime-adapter/src/lib.rs index def09cf7a..93a08542b 100644 --- a/tests/runtime-adapter/src/lib.rs +++ b/tests/runtime-adapter/src/lib.rs @@ -49,7 +49,9 @@ //! worker captures before it assembles any root, so that every clone of the snapshot attaches a //! root of its own, in [`template`] mode. The host program's `runtime-adapter-host create` //! captures such a VM through NVX's template store, which publishes the snapshot as a generation -//! of a template, in the `capture` module. +//! of a template, in the `capture` module. In each clone, the worker repairs the restore, then +//! activates the clone with the host's `Activate`, whose record the guest library applies, answers +//! `Ready` with the activation, and serves the host's order as after a cold boot. #[cfg(feature = "host")] pub mod capture; diff --git a/tests/runtime-adapter/src/root.rs b/tests/runtime-adapter/src/root.rs index 7c6f96232..3ee94af3a 100644 --- a/tests/runtime-adapter/src/root.rs +++ b/tests/runtime-adapter/src/root.rs @@ -50,7 +50,12 @@ //! //! Once the guest assembled the root, it answers the configuration with its [`Ready`]: the length //! of the guest agent's build, one byte, then the build, 1 to [`MAX_BUILD_LEN`] printable ASCII -//! characters without a space, such as [`GUEST_BUILD`](crate::GUEST_BUILD). +//! characters without a space, such as [`GUEST_BUILD`](crate::GUEST_BUILD). A clone, which a +//! restore of a template's snapshot started, answers the host's Activate with its `Ready` once it +//! activated, which reports its activation after the build: the canonical encoding of +//! `nvx::activation::ReadyActivation`, `1`, then the transaction ID of the restore whose record it +//! applied. A cold boot's `Ready` reports no activation, since a root configuration, not an +//! activation record, configures its root. //! //! The host's mapping table travels in parts, each a `MAPS` payload of `nvx::protocol`, as its //! `encode_maps_payloads` writes it: the index in the table of the part's first mapping, then the @@ -67,6 +72,7 @@ use std::collections::BTreeMap; use std::fmt; use std::time::Duration; +use nvx::activation::ReadyActivation; use nvx::protocol::{ GuestMapsPayload, HostMap, MAX_HOST_MAPPINGS, decode_guest_maps_payload, encode_maps_payloads, }; @@ -734,17 +740,22 @@ pub enum RootReply { /// The most bytes of the guest agent's build in a [`Ready`]: as many as the byte before it counts. pub const MAX_BUILD_LEN: usize = u8::MAX as usize; -/// The guest's readiness, with which it answers a root configuration once it assembled the root: -/// the guest agent's build, which the host can check against the build that it ships. +/// The guest's readiness, with which it answers a root configuration once it assembled the root, +/// or a clone's Activate once it activated the clone: the guest agent's build, which the host can +/// check against the build that it ships, and, for a clone, what the guest reports about its +/// activation, which the host can check against the record that it sent. #[derive(Clone, Debug, PartialEq, Eq)] pub struct Ready { /// The guest agent's build. build: String, + /// What the guest reports about its activation, if anything: a clone's, but not a cold + /// boot's, whose root a root configuration configures. + activation: Option, } impl Ready { /// Returns the readiness of the guest agent of the build `build`, such as - /// [`GUEST_BUILD`](crate::GUEST_BUILD). + /// [`GUEST_BUILD`](crate::GUEST_BUILD), which reports no activation. /// /// Fails with a [`BodyError`] unless the build holds 1 to [`MAX_BUILD_LEN`] printable ASCII /// characters, without a space. @@ -756,7 +767,20 @@ impl Ready { ASCII characters without a space" ))); } - Ok(Self { build }) + Ok(Self { + build, + activation: None, + }) + } + + /// Returns the readiness, which reports `activation`: what `nvx::guest::clone::Activation` + /// returns once it completed a clone's activation, that the activation applied, with the + /// restore's transaction ID. + pub fn with_activation(self, activation: ReadyActivation) -> Self { + Self { + activation: Some(activation), + ..self + } } /// Returns the guest agent's build. @@ -764,32 +788,39 @@ impl Ready { &self.build } - /// Returns the readiness's encoding: the build's length, one byte, then the build. + /// Returns what the guest reports about its activation, if anything. + pub fn activation(&self) -> Option { + self.activation + } + + /// Returns the readiness's encoding: the build's length, one byte, then the build, then the + /// canonical encoding of the activation that it reports, if any, which + /// `ReadyActivation::encode` writes. pub fn encode(&self) -> Vec { - let mut body = Vec::with_capacity(1 + self.build.len()); + let activation = self.activation.as_ref().map(ReadyActivation::encode); + let reported = activation.as_ref().map_or(0, Vec::len); + let mut body = Vec::with_capacity(1 + self.build.len() + reported); // The build has at most `MAX_BUILD_LEN` bytes, so its length fits the byte. body.push(self.build.len() as u8); body.extend_from_slice(self.build.as_bytes()); + body.extend(activation.into_iter().flatten()); body } - /// Reads a readiness from its encoding. + /// Reads a readiness from its encoding, which reports an activation if it has bytes after its + /// build. /// - /// Fails with a [`BodyError`] for a body that ends before its build does, or that has bytes - /// after it, and for a build that [`new`](Self::new) refuses. + /// Fails with a [`BodyError`] for a body that ends before its build does, for a build that + /// [`new`](Self::new) refuses, and for bytes after the build that `ReadyActivation::decode` + /// refuses: anything but `0`, or `1` followed by a transaction ID. pub fn decode(body: &[u8]) -> Result { let refused = |problem: String| BodyError::new(format!("the guest's readiness {problem}")); let Some((&length, rest)) = body.split_first() else { return Err(refused("names no build".to_owned())); }; - let Some((build, extra)) = rest.split_at_checked(usize::from(length)) else { + let Some((build, reported)) = rest.split_at_checked(usize::from(length)) else { return Err(refused("ends early".to_owned())); }; - match extra.len() { - 0 => {} - 1 => return Err(refused("has 1 byte too many".to_owned())), - extra => return Err(refused(format!("has {extra} bytes too many"))), - } if !valid_build(build) { return Err(refused(format!( "names a build that does not hold 1 to {MAX_BUILD_LEN} printable ASCII \ @@ -797,8 +828,23 @@ impl Ready { String::from_utf8_lossy(build) ))); } + let activation = match reported.len() { + 0 => None, + count => Some(ReadyActivation::decode(reported).map_err(|error| { + let (bytes, verb) = if count == 1 { + ("byte", "is") + } else { + ("bytes", "are") + }; + refused(format!( + "has {count} {bytes} after its build, which {verb} not an activation: \ + {error}" + )) + })?), + }; Ok(Self { build: build.iter().copied().map(char::from).collect(), + activation, }) } } @@ -928,6 +974,7 @@ impl std::error::Error for MapFailure {} #[cfg(test)] mod tests { + use nvx::activation::TransactionId; use nvx::protocol::{GuestHostMap, MAP_READ_ONLY, MAX_MAP_PATH_BYTES}; use super::*; @@ -1301,6 +1348,7 @@ mod tests { fn a_readiness_is_the_length_of_its_build_then_the_build() { let ready = Ready::new("0.1.0").unwrap(); assert_eq!(ready.build(), "0.1.0"); + assert_eq!(ready.activation(), None); assert_eq!(ready.encode(), b"\x050.1.0"); assert_eq!(Ready::decode(&ready.encode()), Ok(ready)); @@ -1325,6 +1373,44 @@ mod tests { } } + #[test] + fn a_clone_s_readiness_reports_its_activation_after_the_build() { + let clone = ReadyActivation { + transaction: Some(TransactionId::from_bytes([0x7e; TransactionId::LEN])), + }; + // A cold boot's activation, which the encoding takes too, though the adapter's cold boots + // report none. + let cold = ReadyActivation { transaction: None }; + let longest = "b".repeat(MAX_BUILD_LEN); + for build in ["0.1.0", longest.as_str()] { + for activation in [clone, cold] { + let ready = Ready::new(build).unwrap().with_activation(activation); + assert_eq!(ready.build(), build); + assert_eq!(ready.activation(), Some(activation)); + let body = ready.encode(); + let mut expected = vec![u8::try_from(build.len()).unwrap()]; + expected.extend_from_slice(build.as_bytes()); + expected.extend(activation.encode()); + assert_eq!(body, expected, "{build}: {activation:?}"); + assert!(body.len() <= MAX_TEXT_BODY, "{build}"); + assert_eq!(Ready::decode(&body), Ok(ready), "{build}: {activation:?}"); + } + } + // The clone's: `1`, and the transaction ID, after the build; a cold boot's: `0`. + let ready = Ready::new("0.1.0").unwrap().with_activation(clone); + let mut expected = b"\x050.1.0".to_vec(); + expected.push(1); + expected.extend([0x7e; 16]); + assert_eq!(ready.encode(), expected); + assert_eq!( + ready.encode().len(), + 6 + ReadyActivation::CLONE_LEN, + "{ready:?}" + ); + let ready = Ready::new("0.1.0").unwrap().with_activation(cold); + assert_eq!(ready.encode(), b"\x050.1.0\x00"); + } + #[test] fn a_readiness_that_breaks_the_rules_is_refused() { let too_long = "b".repeat(MAX_BUILD_LEN + 1); @@ -1346,7 +1432,7 @@ mod tests { "{build:?}: {error}" ); } - let cases: [(&[u8], &str); 10] = [ + let cases: [(&[u8], &str); 11] = [ (b"", "names no build"), ( b"\x00", @@ -1354,9 +1440,22 @@ mod tests { ), (b"\x060.1.0", "ends early"), (b"\xff0.1.0", "ends early"), - (b"\x050.1.0\x00", "has 1 byte too many"), - (b"\x040.1.0", "has 1 byte too many"), - (b"\x010.1.0", "has 4 bytes too many"), + ( + b"\x050.1.0\x01", + "has 1 byte after its build, which is not an activation", + ), + ( + b"\x050.1.0\x02", + "has 1 byte after its build, which is not an activation", + ), + ( + b"\x040.1.0", + "has 1 byte after its build, which is not an activation", + ), + ( + b"\x010.1.0", + "has 4 bytes after its build, which are not an activation", + ), (b"\x050.1 0", "without a space: \"0.1 0\""), (b"\x050.1\xff0", "without a space: \"0.1\u{fffd}0\""), (b"\x02\xc3\xa9", "without a space: \"\u{e9}\""), @@ -1369,6 +1468,32 @@ mod tests { "{body:?}: {error}" ); } + // Bytes after the build that are neither `0`, nor `1` followed by a transaction ID, which + // `ReadyActivation::decode` refuses: a cold boot's or a clone's with bytes after it, a + // clone's whose transaction ID ends early, and another first byte. + let reported = |rest: &[u8]| { + let mut body = b"\x050.1.0".to_vec(); + body.extend_from_slice(rest); + body + }; + for (body, count) in [ + (reported(&[0, 0]), 2), + (reported(&[0; 17]), 17), + (reported(&[1; 18]), 18), + (reported(&[1; 16]), 16), + (reported(&[2; 17]), 17), + ] { + let error = Ready::decode(&body).unwrap_err(); + assert_eq!( + error.problem(), + format!( + "the guest's readiness has {count} bytes after its build, which are not an \ + activation: what Ready reports is neither 0, nor 1 followed by a transaction \ + ID" + ), + "{body:?}" + ); + } } /// Returns the host's mapping of `source` at `target`, read-only if `read_only`. diff --git a/tests/runtime-adapter/src/template.rs b/tests/runtime-adapter/src/template.rs index 86931dc9e..625ca1a0f 100644 --- a/tests/runtime-adapter/src/template.rs +++ b/tests/runtime-adapter/src/template.rs @@ -18,12 +18,14 @@ //! with a control console. //! //! The worker of a template's VM, and with it the VM, ends with [`COMMAND_LINE_STATUS`] once -//! `boot` refused its command line, with [`CAPTURE_STATUS_BASE`] plus the number of the stage of -//! the capture that failed, with [`CONTINUED_STATUS`] once the capture ended without a snapshot, -//! and, in a clone, which it repairs, with [`RESTORED_STATUS`] once it repaired the clone, which it -//! does not activate yet, or with the status of the repair's failure, which -//! `nvx::protocol::status` documents. A committed capture ends the template's VM before the worker -//! returns. +//! `boot` refused its command line, with the bootstrap's status, 13, once the bootstrap of the +//! guest for the template's root failed, with [`CAPTURE_STATUS_BASE`] plus the number of the stage +//! of the capture that failed, and with [`CONTINUED_STATUS`] once the capture ended without a +//! snapshot. In a clone, which it repairs, it ends with the status of the repair's failure, which +//! `nvx::protocol::status` documents, and once it repaired the clone, as the clone's session ends: +//! with the status of the class of the clone's activation's failure, which `nvx::protocol::status` +//! documents too, or, once the clone was ready, as the worker of a VM that boots cold does. A +//! committed capture ends the template's VM before the worker returns. use std::{fmt, str}; @@ -88,11 +90,6 @@ pub const CAPTURE_STATUS_BASE: u8 = 20; /// snapshot, and the template continued. pub const CONTINUED_STATUS: u8 = 30; -/// The status with which the worker of a template's VM ends in a clone, which a restore of the -/// template's snapshot started, once the worker repaired it: the worker does not activate a clone -/// yet, so the clone ends at once, with its fence of workload admission still closed. -pub const RESTORED_STATUS: u8 = 31; - /// How a VM boots, as its kernel command line tells the guest agent. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum Boot { @@ -528,9 +525,9 @@ mod tests { #[test] fn a_template_s_statuses_are_its_own() { - let mut statuses = vec![COMMAND_LINE_STATUS, CONTINUED_STATUS, RESTORED_STATUS]; + let mut statuses = vec![COMMAND_LINE_STATUS, CONTINUED_STATUS]; statuses.extend((1..=9).map(|stage| CAPTURE_STATUS_BASE + stage)); - assert_eq!(statuses, [20, 30, 31, 21, 22, 23, 24, 25, 26, 27, 28, 29]); + assert_eq!(statuses, [20, 30, 21, 22, 23, 24, 25, 26, 27, 28, 29]); // None is a root stage's, the worker's after its session failed, or the usage error's. for stage in RootStage::ALL { assert!(!statuses.contains(&stage.exit_status()), "{stage}"); diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index dfeefb382..02ca0240a 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -51,15 +51,74 @@ //! //! The worker, and with it the VM, ends with 0 once the host stopped it, after a teardown that //! succeeded, with the stage's [exit status](RootStage::exit_status) once a stage failed, from 11 -//! for the configuration to 16 for the identity, and with [`SESSION_STATUS`] once its session -//! failed or the broker closed it. Once the broker closed the stream at a record boundary, which -//! may close only the broker's end, the worker drops its executions, and writes what it queued, -//! as once the host stopped it. +//! for the configuration to 16 for the identity, with [`SESSION_STATUS`] once its session failed +//! or the broker closed it, and, in a clone, with the status of its activation's failure, from 50 +//! to 58, as the [section on clones](#clones) describes. Once the broker closed the stream at a +//! record boundary, which may close only the broker's end, the worker drops its executions, and +//! writes what it queued, as once the host stopped it. //! //! In a template's VM, whose kernel command line names the platform snapshot tier, the worker //! assembles no root: on x86_64, the guest agent reads the kernel command line before anything //! else with `crate::template::boot`, and the worker captures the VM instead, as the `capture` -//! module describes. +//! module describes, and then serves each clone that a restore of the snapshot starts, as the +//! [next section](#clones) describes. +//! +//! # Clones +//! +//! A restore of a template's snapshot starts a clone, on x86_64, whose worker, the template's, +//! repairs the restore, as the `capture` module describes, behind its fence of workload admission, +//! which stays closed. The worker then serves the clone's session with `work_clone`, through the +//! clone's activation, `nvx::guest::clone::Activation`, which the `activation` module drives: +//! +//! 1. The session that the template attached before its capture, and whose first epoch it +//! acknowledged, resumes on the same tty: the worker attaches none, and acknowledges the reset +//! that the restore queued, the only one that the clone reads, which ends the activation's +//! transport bootstrap and restricts the session. +//! 2. The host sends one `Activate`, whose body is the canonical encoding of a clone's +//! `nvx::activation::ActivationRecord`. The activation decodes and checks it, and checks the +//! resources that it expects against those that the repair found, before anything has a side +//! effect. +//! 3. The worker applies the record through `RootOperations::apply`, which runs +//! `nvx::guest::clone::apply`: it assembles the root from the template's devices, with the +//! record's image, upper, and overlay mode, waiting for each device until the activation's +//! deadline at the latest, so that the application ends by then, and checks the account of the +//! record's workload identity, if it has one, in the assembled root, as the worker of a VM that +//! boots cold does for a root configuration. The template bootstrapped the guest for its root +//! with its own configuration before its capture, so the clone has its cgroups. +//! 4. Once it took what the session received while the application ran, which came before +//! `Ready`, it completes the activation, which opens the fence, and answers `Ready`, which +//! reports that the activation applied, with the record's transaction, after the guest agent's +//! build, to the `Activate` and to each retry of it, byte for byte, that came meanwhile, which +//! joins the application. Anything else that came meanwhile, a reset included, fails the +//! activation, as before the application, so the host waits for `Ready` before it sends the +//! mapping table or the start. +//! 5. It then serves the host's order as after a cold boot: the mapping table that the record +//! announces, the start, the executions, the probes, and the stop. The record's workload +//! identity, if it has one, is the whole sandbox's, as a root configuration's is, so that +//! executions name none, and a clone's executions run in the cgroups of the template's +//! bootstrap. +//! +//! The application applies none of the record's other settings: its export, its network, its +//! hostname, and its machine ID. Without an export, a clone refuses each part of a mapping table +//! with `MapFailed`, as the worker of a VM that boots cold does without one. +//! +//! Until the clone is ready, its session takes nothing but that `Activate`: another reset, any +//! other message, a `Stop` and a `Ping` included, an `Activate` that the activation refuses, one +//! whose header announces more bytes than a record can have, which the framing refuses before it +//! takes any byte of its body, and the passing of the activation's deadline, which ends +//! `activation::ACTIVATION_TIMEOUT` after the repair's acknowledgement of the restore, fail the +//! activation finally, as does an application that failed, which nothing retries. Once the clone +//! is ready, the worker answers a retry of the same `Activate`, byte for byte, with the same +//! `Ready`, without applying it again, and fails the activation on another. A failed activation +//! ends the worker, and with it the VM, with the status of `nvx::protocol::status` of the +//! failure's class, from 50 to 58, as `activation::failure_status` selects it, once the worker +//! dropped its executions and wrote what it queued, for at most [`UNACTIVATED_DRAIN`], so that a +//! stream that takes nothing, as once the broker failed, does not keep the clone from ending, and +//! whatever the stream does then. A broker that closes the stream before the clone is ready ends +//! it the same way, with [`SESSION_STATUS`], as [`Worked::Abandoned`] tells, and once the clone is +//! ready, as the worker of a VM that boots cold ends. The worker of a VM that boots cold, which +//! has no activation, refuses an `Activate` as it refuses any message that it does not take in +//! its state. //! //! # The mapping table and the start //! @@ -195,6 +254,8 @@ //! in all for those programs to end and those cgroups to empty, removing each cgroup that //! emptied once it collected the statuses of those programs. +#[cfg(target_arch = "x86_64")] +pub mod activation; #[cfg(target_arch = "x86_64")] pub mod capture; mod connection; @@ -219,8 +280,12 @@ use nix::poll::{self, PollFd, PollFlags, PollTimeout}; use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, Id, WaitPidFlag, WaitStatus}; use nix::unistd::Pid; +#[cfg(target_arch = "x86_64")] +use nvx::activation::ActivationRecord; use nvx::gpt::Expected; use nvx::guest::cgroup::Layout; +#[cfg(target_arch = "x86_64")] +use nvx::guest::clone::{self, Activation, ApplyError, RootTemplate}; use nvx::guest::entry::{self, Entry, Identity, Prepared, Publication}; use nvx::guest::mappings::{Mapping, SourceRoot}; use nvx::guest::nvxs::GuestConnectionError; @@ -230,6 +295,8 @@ use nvx::guest::root::{ }; use nvx::guest::spawn::{self, Spawned}; use nvx::protocol::export::HostExport; +#[cfg(target_arch = "x86_64")] +use nvx::protocol::nvxs::GuestSession; use nvx::protocol::root::{ DISTRO_MMIO_BASE, ImageFormat, RootLayout, RootUpper, SCRATCH_MMIO_BASE, }; @@ -243,6 +310,10 @@ use crate::root::{ DEFAULT_FIRST_PARTITION_BOUND, JournalPolicy, Listeners, OverlayMode, Ready, RootConfig, RootFailure, RootStage, }; +#[cfg(target_arch = "x86_64")] +use activation::CloneState; +#[cfg(target_arch = "x86_64")] +use capture::Admission; use connection::{Connection, Received}; use executions::{Executions, Sandbox}; use order::Order; @@ -251,6 +322,11 @@ use order::Order; /// with which PID 1 ends the VM if a child cannot start or the worker's status cannot be collected. pub const SESSION_STATUS: u8 = 1; +/// How long the worker waits for its stream, and then its tty, to take what it wrote once a +/// clone's activation failed, or the broker closed the stream before the clone was ready: the +/// clone then ends, and with it the VM, even if they take nothing, as once the broker failed. +pub const UNACTIVATED_DRAIN: Duration = Duration::from_secs(1); + /// The problem with which the worker answers a probe while it runs executions. pub const EXECUTING: &str = "the worker runs executions, and probes nothing until they end"; @@ -350,6 +426,24 @@ pub trait RootOperations: EntryOperations { /// Tears the assembled root down. fn teardown(&mut self) -> Result<(), String>; + + /// Applies `record`, the record of the Activate that a clone's activation accepted, to the + /// clone, as `nvx::guest::clone::apply` does, with `template`, the inputs of the root's + /// assembly that the clone's template captured, by `deadline`, the activation's, if any, until + /// which it waits for each device at the latest, and keeps the root that it assembled: in + /// place of a root configuration's [layout](Self::command_line), assembly, and account check. + /// The template's VM [bootstrapped](Self::bootstrap) the guest before its capture. + /// + /// Returns the identity in which every execution runs, that of the record's workload identity, + /// whose account the application checked in the assembled root, if the record has one. Fails + /// as `nvx::guest::clone::apply` does, with an [`ApplyError`]. + #[cfg(target_arch = "x86_64")] + fn apply( + &mut self, + record: &ActivationRecord, + template: &RootTemplate, + deadline: Option, + ) -> Result, ApplyError>; } /// A mapping that [`RootOperations::map`] refused or failed to apply, once it applied those before @@ -372,17 +466,33 @@ pub enum Worked { Failed(RootFailure), /// The broker closed the stream at a record boundary. Closed, + /// The broker closed the stream at a record boundary before a clone was ready, whose + /// activation can complete no more, which left the worker's fence of workload admission + /// closed. + Abandoned, + /// A clone's activation failed, finally: before the clone was ready, which left the worker's + /// fence of workload admission closed, or once it was ready. + Unactivated { + /// The status of `nvx::protocol::status` of the failure's class, with which the worker + /// ends. + status: u8, + /// The failure, as `nvx::guest::clone::ActivationFailure` describes it: the state in which + /// the activation failed, and its cause. + problem: String, + }, } impl Worked { /// Returns the status with which the worker ends: 0 once the host stopped it, the stage's - /// [exit status](RootStage::exit_status) once a stage failed, and [`SESSION_STATUS`] once the - /// broker closed the stream. + /// [exit status](RootStage::exit_status) once a stage failed, [`SESSION_STATUS`] once the + /// broker closed the stream, before a clone was ready too, and the status of the failure's + /// class once a clone's activation failed, from 50 to 58. pub fn status(&self) -> u8 { match self { Self::Stopped => 0, Self::Failed(failure) => failure.stage().exit_status(), - Self::Closed => SESSION_STATUS, + Self::Closed | Self::Abandoned => SESSION_STATUS, + Self::Unactivated { status, .. } => *status, } } } @@ -402,26 +512,202 @@ pub fn work( stream: S, operations: &mut dyn RootOperations, ) -> Result { - let mut connection = Connection::new(stream)?; + serve(Connection::new(stream)?, operations, Phase::Configuring) +} + +/// Runs the session of a clone, which a restore of a template's snapshot started, on `stream`, the +/// guest's end of the control session, which it makes nonblocking, with the root operations +/// `operations`, as [`work`] does for a VM that boots cold, but through the clone's `activation`, +/// as the [module documentation](self#clones) describes, until the host stops the worker, the +/// activation fails, or the broker closes the stream. +/// +/// `activation` is `nvx::guest::clone::Activation::new` of the clone that the worker repaired, +/// behind the fence that its capture's barrier handed over closed, in its transport bootstrap. +/// `session` is the guest's role of the session that the template's VM attached on the same tty +/// before its capture, and whose first epoch it acknowledged, which the reset that the restore +/// queued starts again: the worker attaches no session. `template` holds the inputs of the root's +/// assembly that the template captured, with which the worker [applies](RootOperations::apply) +/// the record of the host's `Activate`. +/// +/// It ends as [`work`] does, and with [`Worked::Unactivated`] once the activation failed, or +/// [`Worked::Abandoned`] once the broker closed the stream before the clone was ready, after which +/// the worker drops its executions, which ends their programs, and writes what it queued, waiting +/// while the stream takes none, for at most [`UNACTIVATED_DRAIN`], and whatever the stream does. +#[cfg(target_arch = "x86_64")] +pub fn work_clone( + stream: S, + session: GuestSession, + operations: &mut dyn RootOperations, + activation: Activation, + template: RootTemplate, +) -> Result { + let state = CloneState::new(activation, template); + serve( + Connection::resume(stream, session)?, + operations, + Phase::Clone(Box::new(state)), + ) +} + +/// Where the worker is in its order: before its root is ready, through a root configuration or a +/// clone's activation, and once it is, as the [module documentation](self) describes. +enum Phase { + /// The worker of a VM that boots cold awaits its root configuration. + Configuring, + /// The root is ready: the worker's order. + Ready(Order), + /// The worker of a clone takes the host's messages through the clone's activation, before the + /// clone is ready, and once it is, in its order then. + #[cfg(target_arch = "x86_64")] + Clone(Box), +} + +impl Phase { + /// Takes the session's reset, which the connection acknowledged already, and returns how the + /// worker ends once the reset ended the session: once it failed a clone's activation. + fn reset(&mut self) -> Option { + match self { + Self::Configuring | Self::Ready(_) => None, + #[cfg(target_arch = "x86_64")] + Self::Clone(state) => state.reset(), + } + } + + /// Returns how the worker ends once the broker closed the stream at a record boundary: + /// [`Worked::Abandoned`] before a clone was ready, and [`Worked::Closed`] otherwise. + fn closed(&self) -> Worked { + match self { + Self::Configuring | Self::Ready(_) => Worked::Closed, + #[cfg(target_arch = "x86_64")] + Self::Clone(state) if !state.is_ready() => Worked::Abandoned, + #[cfg(target_arch = "x86_64")] + Self::Clone(_) => Worked::Closed, + } + } + + /// Returns how the worker ends once the session's receipt failed with `error`, if its phase + /// takes the failure: a clone's activation takes an `Activate` that the framing refused + /// unread, as longer than a record can be, which fails it. Otherwise the error ends the + /// session. + #[cfg_attr(not(target_arch = "x86_64"), allow(unused_variables))] + fn refused(&mut self, error: &ServeError) -> Option { + match self { + Self::Configuring | Self::Ready(_) => None, + #[cfg(target_arch = "x86_64")] + Self::Clone(state) => state.refused(error), + } + } + + /// Completes what the phase defers until the session took everything that it received, and + /// returns the worker's step then, if it deferred anything: a clone's activation, with the + /// application that ran meanwhile, through `operations`, after which `executions` run in the + /// clone's sandbox. + #[cfg_attr(not(target_arch = "x86_64"), allow(unused_variables))] + fn settle( + &mut self, + operations: &mut dyn RootOperations, + executions: &mut Executions, + ) -> Option> { + match self { + Self::Configuring | Self::Ready(_) => None, + #[cfg(target_arch = "x86_64")] + Self::Clone(state) => state.settle(operations, executions), + } + } + + /// Handles `message` at `now`, through `operations`, and hands a message about an execution to + /// `executions`, which queue what they answer on `connection`, and returns the worker's step. + fn handle( + &mut self, + operations: &mut dyn RootOperations, + executions: &mut Executions, + connection: &mut Connection, + now: Instant, + message: Message, + ) -> Result, ServeError> { + match self { + Self::Configuring => { + let (step, ready) = configure_root(operations, executions, message)?; + if let Some(order) = ready { + *self = Self::Ready(order); + } + Ok(step) + } + Self::Ready(order) => { + serve_root(operations, order, executions, connection, now, message) + } + #[cfg(target_arch = "x86_64")] + Self::Clone(state) => state.handle(operations, executions, connection, now, message), + } + } + + /// Returns when the worker's session ends unless something happens, if it may: a clone's + /// activation's deadline, until the clone is ready. + fn deadline(&self) -> Option { + match self { + Self::Configuring | Self::Ready(_) => None, + #[cfg(target_arch = "x86_64")] + Self::Clone(state) => state.deadline(), + } + } + + /// Returns how the worker ends once its session's deadline passed: a clone's activation's, + /// before the clone was ready. + fn tick(&mut self) -> Option { + match self { + Self::Configuring | Self::Ready(_) => None, + #[cfg(target_arch = "x86_64")] + Self::Clone(state) => state.tick(), + } + } +} + +/// Runs the worker's session on `connection`, from `phase` on, with the root operations +/// `operations`, as [`work`] and [`work_clone`] describe. +fn serve( + mut connection: Connection, + operations: &mut dyn RootOperations, + mut phase: Phase, +) -> Result { let mut executions = Executions::new(); - // The worker's order, once its root is ready. - let mut order = None; loop { connection.flush()?; while !connection.is_overloaded() { - let message = match connection.receive()? { - None => break, + let received = match connection.receive() { + Ok(received) => received, + Err(error) => match phase.refused(&error) { + Some(worked) => { + executions.clear(); + return end(&mut connection, worked); + } + None => return Err(error), + }, + }; + let message = match received { + // The session took everything that it received: a clone's activation completes + // with the application that ran meanwhile, if any. + None => { + if let Some(step) = phase.settle(operations, &mut executions) + && let Some(worked) = replied(&mut connection, step)? + { + return end(&mut connection, worked); + } + break; + } Some(Received::Closed) => { // The broker may have closed only its end of the stream, and read on: what // the worker queued reaches it, as once the host stopped the worker, after // the worker dropped its executions, which a stalled reader then holds back // no more. executions.clear(); - connection.finish()?; - return Ok(Worked::Closed); + return end(&mut connection, phase.closed()); } Some(Received::Reset) => { executions.reset(Instant::now()); + if let Some(worked) = phase.reset() { + executions.clear(); + return end(&mut connection, worked); + } continue; } Some(Received::Message(message)) => message, @@ -430,35 +716,19 @@ pub fn work( // launch that came before it takes nothing from the timers of an execution that it // starts. let now = Instant::now(); - let step = match order.as_mut() { - Some(order) => serve_root( - operations, - order, - &mut executions, - &mut connection, - now, - message, - )?, - None => { - let (step, ready) = configure_root(operations, &mut executions, message)?; - order = ready; - step - } - }; - let (replies, ended) = match step { - Step::Continue(replies) => (replies, None), - Step::End(replies, worked) => (replies, Some(worked)), - }; - for reply in &replies { - connection.send(reply)?; - } - if let Some(worked) = ended { - connection.finish()?; - return Ok(worked); + let step = phase.handle(operations, &mut executions, &mut connection, now, message)?; + if let Some(worked) = replied(&mut connection, step)? { + return end(&mut connection, worked); } } executions.run(Instant::now(), &mut connection)?; connection.credit()?; + // A clone's activation fails once its deadline passed before the clone was ready, whether + // the host sent anything or not. + if let Some(worked) = phase.tick() { + executions.clear(); + return end(&mut connection, worked); + } // While records back up on the stream, the executions read no output, and may keep what // they read in their runners, which no descriptor tells: once a flush ends the backlog, // the next turn comes at once. @@ -467,8 +737,43 @@ pub fn work( if backlogged && !connection.is_backlogged() { continue; } - wait(&connection, &executions)?; + wait(&connection, &executions, phase.deadline())?; + } +} + +/// Queues the replies of `step` on `connection`, and returns how the worker ends, if the step ends +/// it. +fn replied( + connection: &mut Connection, + step: Step, +) -> Result, ServeError> { + let (replies, ended) = match step { + Step::Continue(replies) => (replies, None), + Step::End(replies, worked) => (replies, Some(worked)), + }; + for reply in &replies { + connection.send(reply)?; } + Ok(ended) +} + +/// Ends the session on `connection` as `worked` tells, once the worker dropped its executions: +/// writes what the worker queued for the stream, waiting while the stream takes none, and returns +/// `worked`. A clone that ends before it is ready, or whose activation failed, ends with its own +/// status whatever its stream does: the worker waits for the stream for at most +/// [`UNACTIVATED_DRAIN`], and a failure of the stream changes nothing. +fn end( + connection: &mut Connection, + worked: Worked, +) -> Result { + match worked { + Worked::Abandoned | Worked::Unactivated { .. } => { + let now = Instant::now(); + let _ = connection.finish(Some(now.checked_add(UNACTIVATED_DRAIN).unwrap_or(now))); + } + _ => connection.finish(None)?, + } + Ok(worked) } /// Runs the worker's session on the tty at `tty`, as [`work`] does, through @@ -482,27 +787,34 @@ pub fn work_on_terminal( tty: &Path, operations: &mut dyn RootOperations, ) -> Result { - guest::on_terminal(tty, |_| { - let file = OpenOptions::new() - .read(true) - .write(true) - .custom_flags(OFlag::O_NOCTTY.bits()) - .open(tty) - .map_err(|error| { - let problem = format!("cannot open {}: {error}", tty.display()); - ServeError::Terminal(io::Error::new(error.kind(), problem)) - })?; - work(file, operations) - }) + guest::on_terminal(tty, |_| work(open_tty(tty)?, operations)) +} + +/// Opens the tty at `tty` again, for reading and writing, without making it the controlling +/// terminal, so that the open file that the worker makes nonblocking is its own, while +/// [`guest::on_terminal`] holds the tty open in raw mode. +/// +/// Fails with [`ServeError::Terminal`] if the tty cannot be opened. +fn open_tty(tty: &Path) -> Result { + OpenOptions::new() + .read(true) + .write(true) + .custom_flags(OFlag::O_NOCTTY.bits()) + .open(tty) + .map_err(|error| { + let problem = format!("cannot open {}: {error}", tty.display()); + ServeError::Terminal(io::Error::new(error.kind(), problem)) + }) } -/// Waits until the session's stream or an execution's descriptor polls ready, or the earliest -/// deadline of the executions passes; at once if the connection holds bytes that the worker -/// takes next. The worker waits for the host's messages unless the connection is overloaded, -/// and for the executions' output unless it is backlogged. +/// Waits until the session's stream or an execution's descriptor polls ready, the earliest +/// deadline of the executions passes, or `deadline`, the session's own, if it has one; at once if +/// the connection holds bytes that the worker takes next. The worker waits for the host's messages +/// unless the connection is overloaded, and for the executions' output unless it is backlogged. fn wait( connection: &Connection, executions: &Executions, + deadline: Option, ) -> Result<(), ServeError> { let overloaded = connection.is_overloaded(); let mut events = PollFlags::empty(); @@ -514,10 +826,14 @@ fn wait( } let mut descriptors = vec![PollFd::new(connection.descriptor(), events)]; executions.poll_descriptors(&mut descriptors, connection.is_backlogged()); + let deadline = [executions.deadline(), deadline] + .into_iter() + .flatten() + .min(); let timeout = if connection.has_input() && !overloaded { PollTimeout::ZERO } else { - match executions.deadline() { + match deadline { None => PollTimeout::NONE, Some(deadline) => { let timeout = deadline.saturating_duration_since(Instant::now()); @@ -935,6 +1251,19 @@ impl RootOperations for LibraryRoot { None => Ok(()), } } + + /// Runs `nvx::guest::clone::apply`, and keeps the root that it assembled. + #[cfg(target_arch = "x86_64")] + fn apply( + &mut self, + record: &ActivationRecord, + template: &RootTemplate, + deadline: Option, + ) -> Result, ApplyError> { + let (root, identity) = clone::apply(record, template, deadline)?.into_parts(); + self.root = Some(root); + Ok(identity) + } } impl EntryOperations for LibraryRoot { diff --git a/tests/runtime-adapter/src/worker/activation.rs b/tests/runtime-adapter/src/worker/activation.rs new file mode 100644 index 000000000..8105462dc --- /dev/null +++ b/tests/runtime-adapter/src/worker/activation.rs @@ -0,0 +1,492 @@ +//! The activation of a clone, which a restore of a template's snapshot started, and which the +//! worker repaired, on x86_64, whose time ABI a clone needs: the worker takes the clone's own +//! configuration from the host in one `Activate`, through the clone's +//! `nvx::guest::clone::Activation`, and then serves the host's order as after a cold boot, as the +//! [worker's documentation](super#clones) describes. +//! +//! The activation's failures are final, and end the worker, and with it the VM, with the status of +//! `nvx::protocol::status` that [`failure_status`] selects by the failure's class. + +use std::io::{Read, Write}; +use std::os::fd::AsFd; +use std::time::{Duration, Instant}; + +use nvx::activation::{ActivationRecord, ReadyActivation}; +use nvx::guest::clone::{ + Activation, ActivationAction, ActivationCause, ActivationFailure, ActivationInput, ApplyError, + RootTemplate, +}; +use nvx::guest::entry::{Identity, Publication}; +use nvx::protocol::status; + +use super::capture::Admission; +use super::connection::Connection; +use super::executions::{Executions, Sandbox}; +use super::order::Order; +use super::{RootOperations, Worked, serve_root}; +use crate::GUEST_BUILD; +use crate::frame::{FrameError, Message}; +use crate::guest::{ServeError, Step}; +use crate::root::Ready; + +/// How long a clone's activation may take, from the repair's acknowledgement of the restore until +/// the clone is ready: long enough for the host to connect once the restore is ready, and for the +/// application of the record to wait for each block device as long as the template's default +/// configuration allows. The application's waits end by then at the latest, and a clone that the +/// host did not activate by then ends. +pub const ACTIVATION_TIMEOUT: Duration = Duration::from_secs(120); + +/// Returns the status with which the worker of a clone, and with it the VM, ends once the clone's +/// activation failed with `failure`: the status of `nvx::protocol::status` of the failure's +/// class, by its cause: +/// +/// - [`ACTIVATION_DEADLINE`](status::ACTIVATION_DEADLINE): the deadline passed before the clone +/// was ready; +/// - [`ACTIVATION_PROTOCOL`](status::ACTIVATION_PROTOCOL): the host broke the restricted session: +/// it reset the session again, or sent a request before the clone was ready, or an Activate of +/// another transaction, or one that changes the accepted record; +/// - [`ACTIVATION_RECORD`](status::ACTIVATION_RECORD): the Activate's record does not decode, +/// breaks a limit, or is a cold boot's; +/// - [`ACTIVATION_RESOURCES`](status::ACTIVATION_RESOURCES): the record expects other resources +/// than the repaired clone has; +/// - [`ACTIVATION_ROOT`](status::ACTIVATION_ROOT): the root's assembly failed, or the record's +/// image is a GPT disk, which a clone does not assemble yet, as +/// [`ApplyError::Assembly`] and [`ApplyError::Gpt`] tell, or the application failed otherwise; +/// - [`ACTIVATION_WORKLOAD`](status::ACTIVATION_WORKLOAD): the image refuses the record's +/// workload identity, as [`ApplyError::Account`] tells; +/// - [`ACTIVATION_ORDER`](status::ACTIVATION_ORDER): the worker's own order completed an +/// activation that did not run, or started the sandbox before the clone was ready, or again. +/// +/// A cause that this build of the adapter does not know ends it with `ACTIVATION_PROTOCOL`. +pub fn failure_status(failure: &ActivationFailure) -> u8 { + match failure.cause() { + ActivationCause::Expired => status::ACTIVATION_DEADLINE, + ActivationCause::Reset + | ActivationCause::Request + | ActivationCause::Transaction { .. } + | ActivationCause::Changed => status::ACTIVATION_PROTOCOL, + ActivationCause::Record(_) | ActivationCause::Cold => status::ACTIVATION_RECORD, + ActivationCause::Resources { .. } => status::ACTIVATION_RESOURCES, + ActivationCause::Failed(error) => match error.downcast_ref::() { + Some(ApplyError::Account { .. }) => status::ACTIVATION_WORKLOAD, + // The root's assembly, a GPT image, and any other failure of the application. + _ => status::ACTIVATION_ROOT, + }, + ActivationCause::Completion | ActivationCause::Start => status::ACTIVATION_ORDER, + _ => status::ACTIVATION_PROTOCOL, + } +} + +/// A clone's activation, as the worker's session drives it, with the inputs of the root's +/// assembly that the clone's template captured, the application that awaits the activation's +/// completion, if any, and the worker's order once the clone is ready. +pub(super) struct CloneState { + /// The clone's activation, behind the worker's fence of workload admission. + activation: Activation, + /// The inputs of the root's assembly that the clone's template captured. + template: RootTemplate, + /// The application of the accepted record, once it ran, until the worker + /// [settles](Self::settle) the activation with its result. + applied: Option, + /// The worker's order, once the clone is ready. + order: Option, +} + +/// The application of the record of the Activate that a clone's activation accepted, which ran, +/// and which awaits the activation's completion. +struct Applied { + /// The identity in which every execution runs, if the record has a workload identity, or why + /// the application failed. + result: Result, ApplyError>, + /// The number of mappings that the record announces. + mappings: u32, + /// The number of retries of the Activate that joined the application, which its completion + /// answers too. + joined: usize, +} + +impl CloneState { + /// Returns the state of the clone whose activation is `activation`, in its transport + /// bootstrap, and whose template captured `template`. + pub(super) fn new(activation: Activation, template: RootTemplate) -> Self { + Self { + activation, + template, + applied: None, + order: None, + } + } + + /// Takes the session's reset, which the connection acknowledged already: the one that the + /// restore queued, which ends the activation's transport bootstrap, or a later one once the + /// clone is ready, as when the host attaches again. Returns how the worker ends once the reset + /// failed the activation: another reset before the clone is ready. + pub(super) fn reset(&mut self) -> Option { + let action = self + .activation + .input(Instant::now(), ActivationInput::Reset); + action.err().map(|failure| unactivated(&failure)) + } + + /// Returns when the activation fails unless the clone is ready, if it is not. + pub(super) fn deadline(&self) -> Option { + self.activation.deadline() + } + + /// Returns whether the clone is ready: whether the activation completed, which gave the worker + /// its order. + pub(super) fn is_ready(&self) -> bool { + self.order.is_some() + } + + /// Returns how the worker ends once the activation's deadline passed before the clone was + /// ready, as it may whether the host sent anything or not. + pub(super) fn tick(&mut self) -> Option { + let ticked = self.activation.tick(Instant::now()); + ticked.err().map(|failure| unactivated(&failure)) + } + + /// Returns how the worker ends once the session's receipt failed with `error`, if the + /// activation takes the failure: an `Activate` whose header announced more than + /// [`MAX_ACTIVATE_BODY`](crate::frame::MAX_ACTIVATE_BODY) bytes, the most that a record has, + /// which the framing refused before it took any byte of its body. The activation takes it as + /// an `Activate` whose record has that many bytes, which fails it, with + /// [`ACTIVATION_RECORD`](status::ACTIVATION_RECORD) once the session is restricted, whether + /// the clone is ready or not. Any other failure ends the session as it ends that of a VM that + /// boots cold. + pub(super) fn refused(&mut self, error: &ServeError) -> Option { + let ServeError::Frame(FrameError::Length { kind, length }) = *error else { + return None; + }; + if kind != Message::Activate(Vec::new()).kind() { + return None; + } + let input = ActivationInput::TooLong(length); + let failure = self.activation.input(Instant::now(), input).err()?; + Some(unactivated(&failure)) + } + + /// Handles `message` at `now` through the activation, and, once the clone is ready, in the + /// worker's order, through `operations`, with `executions`, which queue what they answer on + /// `connection`, as [`serve_root`] does once the root of a VM that boots cold is ready. + /// + /// Before the clone is ready, the activation takes an `Activate`, whose record it hands back + /// once it accepted it, and which the worker applies then, through `operations`, and keeps the + /// result of until it [settles](Self::settle) the activation. Until then, the activation takes + /// what the session received meanwhile, as while it activates: a retry of the same + /// `Activate`, byte for byte, joins the application, and any other message fails the + /// activation. The worker answers an `Activate` that the activation completed, and each retry + /// that joined its application, with `Ready`, which reports the activation, as it answers a + /// retry once the clone is ready. Once the activation failed, the worker drops its executions, + /// and ends. + pub(super) fn handle( + &mut self, + operations: &mut dyn RootOperations, + executions: &mut Executions, + connection: &mut Connection, + now: Instant, + message: Message, + ) -> Result, ServeError> { + let input = match &message { + Message::Activate(record) => ActivationInput::Activate(record), + _ => ActivationInput::Request, + }; + let action = match self.activation.input(now, input) { + Ok(action) => action, + Err(failure) => return Ok(self.end(executions, &failure)), + }; + Ok(match action { + ActivationAction::Apply(record) => { + self.apply(operations, &record); + Step::Continue(Vec::new()) + } + ActivationAction::Join => { + if let Some(applied) = &mut self.applied { + applied.joined += 1; + } + Step::Continue(Vec::new()) + } + ActivationAction::Ready(activation) => Step::Continue(vec![ready(activation)]), + ActivationAction::Serve => { + let order = self + .order + .as_mut() + .expect("a ready activation has the worker's order"); + let started = order.is_started(); + let step = serve_root(operations, order, executions, connection, now, message)?; + if !started + && order.is_started() + && let Err(failure) = self.activation.start() + { + return Ok(self.end(executions, &failure)); + } + step + } + ActivationAction::Continue | ActivationAction::Acknowledge => { + Step::Continue(Vec::new()) + } + }) + } + + /// Applies `record`, the record of the Activate that the activation accepted, through + /// `operations`, with the template's inputs, by the activation's deadline, and keeps the + /// application's result until the worker [settles](Self::settle) the activation with it. + fn apply(&mut self, operations: &mut dyn RootOperations, record: &ActivationRecord) { + // The application's waits for the devices end by the activation's deadline, past which + // the completion fails the activation, whatever the application's result. + let deadline = self.activation.deadline(); + let result = operations.apply(record, &self.template, deadline); + // A record announces at most `nvx::activation::MAX_MAPPINGS` mappings, which fit a `u32`. + let mappings = record.configuration().mappings as u32; + self.applied = Some(Applied { + result, + mappings, + joined: 0, + }); + } + + /// Completes the activation with the result of the application that ran, if any, once the + /// worker took what the session received while it ran, which comes before `Ready`, and + /// returns the worker's step: `Ready`, which reports the activation, for the `Activate` and + /// for each retry that joined its application, once the clone is ready, after which + /// `executions` run in the cgroups that the template's bootstrap prepared, as the record's + /// workload identity, if it has one, and the worker's order awaits the record's mappings; or + /// the end, once the activation failed. + pub(super) fn settle( + &mut self, + operations: &mut dyn RootOperations, + executions: &mut Executions, + ) -> Option> { + let applied = self.applied.take()?; + let (result, identity) = match applied.result { + Ok(identity) => (Ok(()), identity), + Err(error) => (Err(error.into()), None), + }; + Some(match self.activation.complete(Instant::now(), result) { + Ok(activation) => { + let identity = identity.map(|identity| (identity.uid(), identity.gid())); + // The executions' listeners are published, the entry's default, which refuses + // the most, as a root configuration that names no publication has them. + let sandbox = Sandbox::new(identity, Publication::Published); + executions.run_in(operations.cgroups(), sandbox); + self.order = Some(Order::new(applied.mappings)); + Step::Continue(vec![ready(activation); 1 + applied.joined]) + } + Err(failure) => self.end(executions, &failure), + }) + } + + /// Drops the worker's executions, which ends their programs, once the activation failed with + /// `failure`, and returns the end of the worker. + fn end(&mut self, executions: &mut Executions, failure: &ActivationFailure) -> Step { + executions.clear(); + Step::End(Vec::new(), unactivated(failure)) + } +} + +/// Returns the worker's `Ready` once the activation completed, which reports `activation` after +/// the guest agent's build. +fn ready(activation: ReadyActivation) -> Message { + let ready = Ready::new(GUEST_BUILD).expect("this crate's version is a build"); + Message::Ready(ready.with_activation(activation).encode()) +} + +/// Returns how the worker ends once the activation failed with `failure`: with its +/// [status](failure_status), and its description. +fn unactivated(failure: &ActivationFailure) -> Worked { + Worked::Unactivated { + status: failure_status(failure), + problem: failure.to_string(), + } +} + +#[cfg(test)] +mod tests { + use std::io; + use std::num::NonZeroU64; + + use nvx::activation::{ + Configuration, ExpectedResources, Identity, Image, MachineIdMode, OverlayMode, + RestoreSection, StoreGeneration, TemplateReference, TransactionId, Upper, WorkloadIdentity, + WorkloadView, + }; + use nvx::guest::clone::AdmissionFence; + use nvx::guest::entry; + + use super::*; + + /// The resources of the clones of the tests. + const RESOURCES: ExpectedResources = ExpectedResources { + cpus: 2, + memory: 1 << 30, + }; + + /// Returns a clone's record of the transaction `transaction`, with the workload identity of + /// `uid`. + fn record(transaction: u8, uid: u32) -> ActivationRecord { + let configuration = Configuration { + image: Image::Erofs { + binding: Identity::from_bytes([0x11; 32]), + uuid: None, + }, + upper: Upper::Tmpfs, + overlay: OverlayMode::UntrustedLowers, + export: None, + mappings: 0, + network: None, + workload: WorkloadView { + hostname: b"clone".to_vec(), + machine_id: MachineIdMode::Image, + identity: Some(WorkloadIdentity { uid, gid: uid }), + }, + resources: RESOURCES, + }; + let restore = RestoreSection { + transaction: TransactionId::from_bytes([transaction; 16]), + template: TemplateReference { + generation: StoreGeneration::new(NonZeroU64::MIN), + key: Identity::from_bytes([0x44; 32]), + }, + }; + ActivationRecord::new(Some(restore), configuration).unwrap() + } + + /// Returns a clone's activation, in its transport bootstrap, behind a closed fence, with + /// `resources`, which the deadline ends `timeout` after now. + fn activation(resources: ExpectedResources, timeout: Duration) -> Activation { + let mut fence = Admission::default(); + fence.close().unwrap(); + let now = Instant::now(); + Activation::with_resources(resources, Some(now), fence, timeout, now) + } + + /// Returns an activation that took the session's reset, and is restricted. + fn restricted() -> Activation { + let mut activation = activation(RESOURCES, ACTIVATION_TIMEOUT); + let action = activation.input(Instant::now(), ActivationInput::Reset); + assert_eq!(action.unwrap(), ActivationAction::Acknowledge); + activation + } + + /// Returns an activation that accepted the record of the transaction 1. + fn activating() -> Activation { + let mut activation = restricted(); + let bytes = record(1, 1000).encode(); + let action = activation.input(Instant::now(), ActivationInput::Activate(&bytes)); + assert!( + matches!(action, Ok(ActivationAction::Apply(_))), + "{action:?}" + ); + activation + } + + /// Returns the failure of the activation with which `result` fails. + fn failed(result: Result) -> ActivationFailure { + result.unwrap_err() + } + + #[test] + fn each_class_of_failure_ends_the_clone_with_its_own_status() { + let now = Instant::now; + let bytes = record(1, 1000).encode(); + let mut cases = Vec::new(); + + // The deadline. + let mut expired = activation(RESOURCES, Duration::ZERO); + cases.push((failed(expired.tick(now())), status::ACTIVATION_DEADLINE)); + // The host's session: another reset, a request, another transaction, a changed record. + let mut reset = restricted(); + let failure = failed(reset.input(now(), ActivationInput::Reset)); + cases.push((failure, status::ACTIVATION_PROTOCOL)); + let mut request = restricted(); + let failure = failed(request.input(now(), ActivationInput::Request)); + cases.push((failure, status::ACTIVATION_PROTOCOL)); + let early = + activation(RESOURCES, ACTIVATION_TIMEOUT).input(now(), ActivationInput::Request); + cases.push((failed(early), status::ACTIVATION_PROTOCOL)); + let other = record(2, 1000).encode(); + let failure = failed(activating().input(now(), ActivationInput::Activate(&other))); + assert!(matches!( + failure.cause(), + ActivationCause::Transaction { .. } + )); + cases.push((failure, status::ACTIVATION_PROTOCOL)); + let changed = record(1, 1001).encode(); + let failure = failed(activating().input(now(), ActivationInput::Activate(&changed))); + assert!(matches!(failure.cause(), ActivationCause::Changed)); + cases.push((failure, status::ACTIVATION_PROTOCOL)); + // The record: one that does not decode, and a cold boot's. + let failure = failed(restricted().input(now(), ActivationInput::Activate(b"\x01"))); + cases.push((failure, status::ACTIVATION_RECORD)); + let cold = ActivationRecord::new(None, record(1, 1000).configuration().clone()).unwrap(); + let cold = cold.encode(); + let failure = failed(restricted().input(now(), ActivationInput::Activate(&cold))); + assert!(matches!(failure.cause(), ActivationCause::Cold)); + cases.push((failure, status::ACTIVATION_RECORD)); + // The resources. + let mut more = activation( + ExpectedResources { + cpus: 4, + ..RESOURCES + }, + ACTIVATION_TIMEOUT, + ); + more.input(now(), ActivationInput::Reset).unwrap(); + let failure = failed(more.input(now(), ActivationInput::Activate(&bytes))); + cases.push((failure, status::ACTIVATION_RESOURCES)); + // The application: of the root, of the workload, and any other. + let root = activating().complete(now(), Err(ApplyError::Gpt.into())); + cases.push((failed(root), status::ACTIVATION_ROOT)); + let refused = ApplyError::Account { + error: entry::Error::Root(io::Error::other("no account")), + teardown: None, + }; + let workload = activating().complete(now(), Err(refused.into())); + cases.push((failed(workload), status::ACTIVATION_WORKLOAD)); + let other = activating().complete(now(), Err("another failure of the application".into())); + cases.push((failed(other), status::ACTIVATION_ROOT)); + // The worker's own order. + let failure = failed(restricted().complete(now(), Ok(()))); + assert!(matches!(failure.cause(), ActivationCause::Completion)); + cases.push((failure, status::ACTIVATION_ORDER)); + let failure = failed(restricted().start()); + assert!(matches!(failure.cause(), ActivationCause::Start)); + cases.push((failure, status::ACTIVATION_ORDER)); + + for (failure, expected) in &cases { + assert_eq!(failure_status(failure), *expected, "{failure}"); + let ended = unactivated(failure); + assert_eq!(ended.status(), *expected, "{failure}"); + assert_eq!( + ended, + Worked::Unactivated { + status: *expected, + problem: failure.to_string(), + } + ); + } + // Every class that the adapter's clones reach has its status; the export's and the + // network's are left to the activation of those settings. + let mut statuses = cases + .iter() + .map(|(failure, _)| failure_status(failure)) + .collect::>(); + statuses.sort_unstable(); + statuses.dedup(); + assert_eq!(statuses, [50, 51, 52, 53, 54, 55, 58]); + assert!( + statuses + .iter() + .all(|status| status::ACTIVATION_STATUSES.contains(status)) + ); + } + + #[test] + fn the_activation_s_timeout_covers_the_wait_for_both_devices() { + use super::super::capture::{root_template, template_config}; + + let template = template_config(); + assert!(ACTIVATION_TIMEOUT >= 2 * template.device_timeout); + assert_eq!(root_template(&template).timeout, template.device_timeout); + } +} diff --git a/tests/runtime-adapter/src/worker/capture.rs b/tests/runtime-adapter/src/worker/capture.rs index 3b528c1f5..2cebab22e 100644 --- a/tests/runtime-adapter/src/worker/capture.rs +++ b/tests/runtime-adapter/src/worker/capture.rs @@ -1,65 +1,117 @@ //! The worker of a template's VM, which captures the VM before it assembles any root, through -//! `nvx::guest::clone`, on x86_64, whose time ABI a capture needs. The -//! [`template`](crate::template) module tells such a VM by its kernel command line, and holds the -//! statuses with which its worker ends. +//! `nvx::guest::clone`, on x86_64, whose time ABI a capture needs, and then serves each clone that +//! a restore of the snapshot starts. The [`template`](crate::template) module tells such a VM by +//! its kernel command line, and holds the statuses with which its worker ends. //! //! [`capture_on_terminal`] opens the VM's control tty, in raw mode, as the worker of a VM that //! boots cold does, and then, in order: //! -//! 1. attaches the control session with `nvx::guest::nvxs::GuestConnection::attach`, and waits -//! until the tty has sent the attachment, so that the snapshot holds the guest attached. It -//! reads nothing from the session before the capture, and serves no application session: the -//! host opens none with a template's VM; -//! 2. prepares the guest for the capture with `nvx::guest::clone::CapturePreparation`, behind the +//! 1. bootstraps the guest for the root of every clone, with the template's own root +//! configuration, [`template_config`], since the host sends a template's VM none: its cgroups +//! and their resource values, through [`RootOperations::bootstrap`], as the worker of a VM that +//! boots cold does before it assembles its root. The snapshot holds them, so that a clone +//! assembles its root without them; +//! 2. attaches the control session with `nvx::guest::nvxs::GuestConnection::attach`, takes the +//! reset with which the broker answers the attachment at once, acknowledges it with the guest +//! agent's receive window, and waits until the tty has sent the acknowledgement, so that the +//! snapshot holds the guest's session active, in the broker's first epoch. A clone then reads +//! no reset but the one that the restore queues, of another instance: the restore's broker +//! would finish sending a reset that the template's broker began to send, and the clone would +//! read one that the template did not. The worker reads nothing else from the session before +//! the capture, and serves no application session: the host opens none with a template's VM; +//! 3. prepares the guest for the capture with `nvx::guest::clone::CapturePreparation`, behind the //! worker's [`Admission`], its fence of workload admission, and from whose [`devices`] nothing //! may be mounted: the preparation closes the fence, checks the mount table, waits for the time //! ABI's boot checks, runs its capture checks, and suppresses the stall detectors; -//! 3. requests the capture with `CaptureBarrier::capture`, with the platform tier's request, +//! 4. requests the capture with `CaptureBarrier::capture`, with the platform tier's request, //! which keeps no scratch. //! -//! A committed capture ends the template's VM. The worker returns only once the capture ended -//! without a snapshot, or failed, or in a clone, which a restore of the snapshot started, as a -//! [`Templated`] tells, and with which the worker ends. In a clone, the worker repairs the restore -//! next, with `nvx::guest::clone::repair`, behind its fence, which stays closed: the repair -//! activates the restore's memory target, reseeds the kernel's random number generator, and -//! finishes the restore, with the acknowledgement that the host awaits, which hands the deferred -//! time work to the time daemon. NVX's `/init` started the daemon in the template, with the time -//! ABI's boot step, before the worker, so the snapshot holds it, and PID 1 collects its status. The -//! worker does not activate a clone yet: a clone that it repaired ends at once, with -//! [`RESTORED_STATUS`] and its fence still closed, which may cut the deferred time work short. A -//! repair that failed ends the clone with the [status](nvx::guest::clone::RepairError::status) of -//! its failure, which `nvx::protocol::status` documents, and a capture request that failed in a -//! clone, since the time ABI could not repair its restore, with the status of the time ABI's -//! violation, as [`capture_status`] tells. +//! A committed capture ends the template's VM. The worker returns only once the bootstrap failed, +//! or the capture ended without a snapshot, or failed, or in a clone, which a restore of the +//! snapshot started, as a [`Templated`] tells, and with which the worker ends. In a clone, the +//! worker repairs the restore next, with `nvx::guest::clone::repair`, behind its fence, which +//! stays closed: the repair activates the restore's memory target, reseeds the kernel's random +//! number generator, and finishes the restore, with the acknowledgement that the host awaits, +//! which hands the deferred time work to the time daemon. NVX's `/init` started the daemon in the +//! template, with the time ABI's boot step, before the worker, so the snapshot holds it, and PID 1 +//! collects its status. A repair that failed ends the clone with the +//! [status](nvx::guest::clone::RepairError::status) of its failure, which `nvx::protocol::status` +//! documents, and a capture request that failed in a clone, since the time ABI could not repair +//! its restore, with the status of the time ABI's violation, as [`capture_status`] tells. +//! +//! Once the worker repaired a clone, it serves the clone's session on the same tty, with +//! [`work_clone`]: through the clone's activation, `nvx::guest::clone::Activation` of the repaired +//! clone and of the fence, whose deadline ends [`ACTIVATION_TIMEOUT`] after the repair's +//! acknowledgement, it answers the reset that the restore queued for the session that the snapshot +//! holds, takes the host's `Activate`, and applies its record with [`root_template`] of the +//! template's configuration, then serves the host's order as after a cold boot, as the +//! [worker's documentation](super#clones) describes. The clone ends as that session does. Once its +//! activation failed, or the broker closed the stream before the clone was ready, the worker waits +//! for the tty to send what it wrote for at most [`UNACTIVATED_DRAIN`], so that a tty that sends +//! nothing does not keep the clone from ending. use std::fmt; use std::io::{self, Read, Write}; -use std::path::Path; +use std::path::{Path, PathBuf}; +use std::time::Instant; use nvx::guest::clone::{ - self, AdmissionFence, CaptureBarrier, CaptureCause, CaptureError, CapturePreparation, - CaptureStage, Captured, Devices, RepairError, Repaired, + self, Activation, AdmissionFence, CaptureBarrier, CaptureCause, CaptureError, + CapturePreparation, CaptureStage, Captured, Devices, RepairError, Repaired, RootTemplate, }; use nvx::guest::nvxs::GuestConnection; use nvx::guest::root::Device; use nvx::guest::terminal::Terminal; use nvx::guest::time::TimeError; +use nvx::protocol::nvxs::GuestSession; use nvx::protocol::root::{DISTRO_MMIO_BASE, SCRATCH_MMIO_BASE}; -use crate::guest::{self, ServeError}; -use crate::template::{CAPTURE_STATUS_BASE, CONTINUED_STATUS, RESTORED_STATUS}; +use super::activation::ACTIVATION_TIMEOUT; +use super::{RootOperations, UNACTIVATED_DRAIN, Worked, open_tty, work_clone}; +use crate::guest::{self, RECEIVE_WINDOW, ServeError}; +use crate::root::{RootConfig, RootFailure, RootStage}; +use crate::template::{CAPTURE_STATUS_BASE, CONTINUED_STATUS}; /// The virtio-fs tag of the VM's host filesystem device, OpenVMM's one filesystem slot: dormant in /// a template's VM, and the device of the host export that the worker of a VM that boots cold /// mounts. pub const FILESYSTEM_TAG: &str = "microvm"; +/// Returns the root configuration that a template's VM holds as its own, since the host sends it +/// none: [`RootConfig::default`], the configuration that the host program sends a VM that boots +/// cold unless its caller names another. Its cgroups and their resource values, with which the +/// template bootstraps the guest before its capture, and the inputs of the root's assembly that +/// [`root_template`] takes from it, are the template's, which every clone of it shares. +pub fn template_config() -> RootConfig { + RootConfig::default() +} + +/// Returns the inputs of a clone's root's assembly that its template captured, from `config`, the +/// template's own root configuration: its runtime directory; the distro and the scratch block +/// devices, at the MMIO windows from which the worker of a VM that boots cold assembles its root, +/// with the sizes that the configuration names, if any; and how long the assembly waits for each. +/// The record of the clone's `Activate` holds the rest. +pub fn root_template(config: &RootConfig) -> RootTemplate { + RootTemplate { + runtime: PathBuf::from(&config.runtime), + distro: Device { + mmio: DISTRO_MMIO_BASE, + sectors: config.distro_sectors, + }, + scratch: Device { + mmio: SCRATCH_MMIO_BASE, + sectors: config.scratch_sectors, + }, + timeout: config.device_timeout, + } +} + /// The worker's fence of workload admission in a template's VM, which it hands the capture's /// preparation. /// /// The worker of a template's VM admits no execution before the capture, nor in a clone, which it -/// repairs behind the fence, and whose activation would open it, which the worker does not run -/// yet. Closing the fence acquires it, as `nvx::guest::clone` requires: +/// repairs behind the fence, before the clone's activation opens it, once the clone is ready. +/// Closing the fence acquires it, as `nvx::guest::clone` requires: /// [`close`](AdmissionFence::close) fails while the fence is closed, whoever closed it, and leaves /// it closed. #[derive(Debug, Default)] @@ -149,21 +201,21 @@ fn failure_status(stage: CaptureStage, violation: Option) -> u8 { } } -/// How the capture of a template's VM ended, once the worker attached its session. +/// How the worker of a template's VM ended, once it opened its tty. #[derive(Debug)] pub enum Templated { + /// The bootstrap of the guest for the template's root, with the template's own root + /// configuration, failed, before the capture. + Unbootstrapped(RootFailure), /// The capture ended without a snapshot, as without a destination or once OpenVMM rejected /// it, and the template continued: the barrier opened the fence, and restored the stall /// detectors' settings. Continued, /// A restore of the template's snapshot started this VM, a clone, which the worker repaired, - /// and does not activate yet. - Repaired { - /// The repaired clone, with its resources. - repaired: Repaired, - /// The worker's fence of workload admission, still closed: opening it is the activation's. - fence: Admission, - }, + /// then served, through its activation, and once the clone was ready, in its order, until the + /// session ended as this tells: once the host stopped the worker, a teardown failed, the + /// broker closed the stream, before the clone was ready too, or the activation failed. + Cloned(Worked), /// A restore of the template's snapshot started this VM, a clone, whose repair failed. The /// fence stays closed. Unrepaired(RepairError), @@ -173,14 +225,16 @@ pub enum Templated { } impl Templated { - /// Returns the status with which the worker, and with it the VM, ends: [`CONTINUED_STATUS`], - /// [`RESTORED_STATUS`] once it repaired a clone, the + /// Returns the status with which the worker, and with it the VM, ends: the + /// [status](RootStage::exit_status) of the bootstrap's stage once it failed, + /// [`CONTINUED_STATUS`], the [status](Worked::status) with which a clone's session ended, the /// [status](nvx::guest::clone::RepairError::status) of a repair that failed, or the /// [status](capture_status) of a capture that failed. pub fn status(&self) -> u8 { match self { + Self::Unbootstrapped(failure) => failure.stage().exit_status(), Self::Continued => CONTINUED_STATUS, - Self::Repaired { .. } => RESTORED_STATUS, + Self::Cloned(worked) => worked.status(), Self::Unrepaired(error) => error.status(), Self::Failed(error) => capture_status(error), } @@ -189,17 +243,38 @@ impl Templated { impl fmt::Display for Templated { fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + let clone = "a restore of the template's snapshot started this clone, which the worker \ + repaired"; match self { + Self::Unbootstrapped(failure) => { + write!(formatter, "the template could not capture: {failure}") + } Self::Continued => formatter.write_str( "the template's capture ended without a snapshot, and the template continued", ), - Self::Repaired { repaired, .. } => write!( + Self::Cloned(Worked::Stopped) => { + write!( + formatter, + "{clone}, activated, and served until the host stopped it" + ) + } + Self::Cloned(Worked::Failed(failure)) => write!( + formatter, + "{clone}, activated, and served until the host stopped it: {failure}" + ), + Self::Cloned(Worked::Closed) => { + write!( + formatter, + "{clone}, activated, and whose session the broker closed" + ) + } + Self::Cloned(Worked::Abandoned) => write!( formatter, - "a restore of the template's snapshot started this clone, which the worker \ - repaired, with {} online CPUs and {} bytes of memory, and does not activate", - repaired.cpus(), - repaired.memory() + "{clone}, and whose session the broker closed before the clone was activated" ), + Self::Cloned(Worked::Unactivated { problem, .. }) => { + write!(formatter, "{clone}, but not activated: {problem}") + } Self::Unrepaired(error) => write!( formatter, "the repair of this clone, which a restore of the template's snapshot started, \ @@ -211,67 +286,121 @@ impl fmt::Display for Templated { } } -/// Captures the template's VM on its control tty at `tty`, as the [module documentation](self) -/// describes, and returns how the capture ended, once the worker repaired a clone that a restore -/// of the snapshot started. +/// How a template's capture ended, once the worker repaired a clone that a restore of the snapshot +/// started. +enum Capture { + /// The capture ended as this tells, without a clone to serve. + Ended(Templated), + /// A restore of the template's snapshot started this VM, a clone, which the worker repaired. + Repaired { + /// The repaired clone, with its resources. + repaired: Repaired, + /// The worker's fence of workload admission, still closed: opening it is the activation's. + fence: Admission, + }, +} + +/// Captures the template's VM on its control tty at `tty`, with the root operations `operations`, +/// as the [module documentation](self) describes, and returns how the worker ended: once the +/// bootstrap failed, or the capture ended without a snapshot, or failed, or in a clone, which a +/// restore of the snapshot started, once its repair failed, or the session that the worker served +/// once it repaired the clone ended. /// -/// It opens the tty through [`guest::on_terminal`], as the worker of a VM that boots cold does, -/// and fails as that does if the tty cannot be opened or put in raw mode. It fails with -/// [`ServeError::Session`] if the attachment cannot be written, and with [`ServeError::Terminal`] -/// if the tty does not send it, before the preparation. -pub fn capture_on_terminal(tty: &Path) -> Result { - guest::on_terminal(tty, |terminal| { +/// It opens the tty through [`guest::on_terminal_within`], as the worker of a VM that boots cold +/// does, and fails as that does if the tty cannot be opened or put in raw mode. It fails with +/// [`ServeError::Session`] if the attachment or the acknowledgement of the broker's reset cannot +/// be written, or if the broker closes the stream or sends anything but a reset first, and with +/// [`ServeError::Terminal`] if the tty does not send the acknowledgement, before the preparation. +/// A clone's session fails as [`work_clone`] does, on the tty, which the worker opens again for +/// itself, as [`work_on_terminal`](super::work_on_terminal) does. +pub fn capture_on_terminal( + tty: &Path, + operations: &mut dyn RootOperations, +) -> Result { + guest::on_terminal_within(tty, |terminal| { + let config = template_config(); + if let Err(problem) = operations.bootstrap(&config) { + let failure = RootFailure::new(RootStage::Bootstrap, problem); + return Ok((Templated::Unbootstrapped(failure), None)); + } let send = |terminal: &&mut Terminal| { terminal.drain().map_err(|error| { - let problem = format!("cannot send the attachment on {}: {error}", tty.display()); + let problem = format!( + "cannot send the acknowledgement of the broker's reset on {}: {error}", + tty.display() + ); io::Error::new(error.kind(), problem) }) }; - attached(terminal, send, || { + let (captured, session) = attached(terminal, send, || { let preparation = CapturePreparation::new(Admission::default(), devices()); match preparation.prepare().and_then(CaptureBarrier::capture) { - Ok(Captured::Continued) => Templated::Continued, + Ok(Captured::Continued) => Capture::Ended(Templated::Continued), Ok(Captured::Restored(restored)) => { // The repair runs behind the fence, which stays closed. let (restore, fence) = restored.into_parts(); match clone::repair(restore) { - Ok(repaired) => Templated::Repaired { repaired, fence }, - Err(error) => Templated::Unrepaired(error), + Ok(repaired) => Capture::Repaired { repaired, fence }, + Err(error) => Capture::Ended(Templated::Unrepaired(error)), } } - Err(error) => Templated::Failed(error), + Err(error) => Capture::Ended(Templated::Failed(error)), } - }) + })?; + let (repaired, fence) = match captured { + Capture::Ended(templated) => return Ok((templated, None)), + Capture::Repaired { repaired, fence } => (repaired, fence), + }; + // The activation's deadline starts at the repair's acknowledgement of the restore. + let activation = Activation::new(&repaired, fence, ACTIVATION_TIMEOUT, Instant::now()); + let template = root_template(&config); + let worked = work_clone(open_tty(tty)?, session, operations, activation, template)?; + // A clone that ends before it is ready, or whose activation failed, ends even if the tty + // sends nothing. + let unready = matches!(worked, Worked::Abandoned | Worked::Unactivated { .. }); + Ok(( + Templated::Cloned(worked), + unready.then_some(UNACTIVATED_DRAIN), + )) }) } -/// Attaches the guest's end of the control session on `stream`, waits until `send` has sent the -/// attachment, then returns what `capture` returns, which runs without reading the stream. +/// Attaches the guest's end of the control session on `stream`, takes the reset with which the +/// broker answers the attachment, acknowledges it with the guest agent's [`RECEIVE_WINDOW`], +/// waits until `send` has sent the acknowledgement, then returns what `capture` returns, which +/// runs without reading the stream any further, with the guest's role of the session, which the +/// acknowledgement activated in the broker's first epoch. /// -/// Fails with [`ServeError::Session`] if the attachment cannot be written, and with +/// Fails with [`ServeError::Session`] if the attachment or the acknowledgement cannot be written, +/// or if the broker closes the stream or sends anything but a reset first, and with /// [`ServeError::Terminal`] if `send` fails, before `capture` runs. fn attached( stream: S, send: impl FnOnce(&S) -> io::Result<()>, capture: impl FnOnce() -> T, -) -> Result { +) -> Result<(T, GuestSession), ServeError> { let mut connection = GuestConnection::new(stream); connection.attach()?; + // A session takes nothing but a reset before its first epoch, so the event is the broker's + // reset, which the acknowledgement requires. + connection.next_event()?; + connection.acknowledge(RECEIVE_WINDOW)?; send(connection.get_ref()).map_err(ServeError::Terminal)?; - Ok(capture()) + let captured = capture(); + Ok((captured, connection.session().clone())) } #[cfg(test)] mod tests { use std::cell::Cell; + use std::net::Shutdown; use std::os::unix::net::UnixStream; use nvx::guest::nvxs::GuestConnectionError; - use nvx::protocol::nvxs::GuestSession; + use nvx::protocol::nvxs::{GuestState, Identity, RecordType, SessionError, encode_record}; use nvx::protocol::status; use super::*; - use crate::root::RootStage; use crate::template::COMMAND_LINE_STATUS; use crate::worker::SESSION_STATUS; @@ -319,13 +448,7 @@ mod tests { // No two outcomes of the worker share a status, nor one of a clone's repair or // activation. let mut statuses = stages.map(|(_, status)| status).to_vec(); - statuses.extend([ - COMMAND_LINE_STATUS, - CONTINUED_STATUS, - RESTORED_STATUS, - SESSION_STATUS, - 0, - ]); + statuses.extend([COMMAND_LINE_STATUS, CONTINUED_STATUS, SESSION_STATUS, 0]); statuses.extend(RootStage::ALL.map(RootStage::exit_status)); statuses.extend(status::TIME_STATUSES); statuses.extend(status::REPAIR_STATUSES); @@ -372,39 +495,96 @@ mod tests { } #[test] - fn the_worker_attaches_its_session_before_the_capture_and_reads_nothing() { + fn a_clone_assembles_its_root_with_the_template_s_configuration() { + // The configuration that the host program sends a VM that boots cold unless its caller + // names another. + let config = template_config(); + assert_eq!(config, RootConfig::default()); + let template = root_template(&config); + assert_eq!(template.runtime, Path::new(&config.runtime)); + assert_eq!(template.timeout, config.device_timeout); + // The clone's root comes from the devices whose mounts the template's capture checks. + let devices = devices(); + assert_eq!(template.distro, devices.distro); + assert_eq!(template.scratch, devices.scratch); + // A configuration that sizes the devices sizes the clone's. + let sized = RootConfig { + distro_sectors: Some(8), + scratch_sectors: Some(16), + ..config + }; + let template = root_template(&sized); + assert_eq!( + (template.distro.sectors, template.scratch.sectors), + (Some(8), Some(16)) + ); + assert_eq!( + (template.distro.mmio, template.scratch.mmio), + (DISTRO_MMIO_BASE, SCRATCH_MMIO_BASE) + ); + } + + /// Identity of the epoch that the broker of the tests' templates starts, with the reset with + /// which it answers the attachment. + const TEMPLATE: Identity = Identity { + instance_id: [0x7e; 16], + epoch: 1, + }; + + /// Returns the record of `record_type`, with `payload`, that starts or acknowledges the + /// template's epoch, as the first of its side. + fn first(record_type: RecordType, payload: &[u8]) -> Vec { + let mut record = Vec::new(); + let (instance, epoch) = (TEMPLATE.instance_id, TEMPLATE.epoch); + encode_record(record_type, &instance, epoch, 0, payload, &mut record).unwrap(); + record + } + + #[test] + fn the_worker_acknowledges_the_broker_s_reset_before_the_capture_and_reads_nothing_more() { let (guest, broker) = UnixStream::pair().unwrap(); - // The broker's record waits on the stream, as a reset would once a host attached. + // The broker answers the attachment with a reset, and a record waits on the stream behind + // it, as one of the restore's broker would. + (&broker).write_all(&first(RecordType::Reset, &[])).unwrap(); (&broker).write_all(b"waiting").unwrap(); let mut attachment = Vec::new(); GuestSession::new().attach(&mut attachment).unwrap(); + let acknowledgement = first(RecordType::Ack, &RECEIVE_WINDOW.to_le_bytes()); let sent = Cell::new(false); - let captured = attached( + let (captured, session) = attached( &guest, |_| { sent.set(true); Ok(()) }, || { - // The capture runs once the attachment was sent, which the broker holds. + // The capture runs once the acknowledgement was sent, after the attachment, which + // the broker holds. assert!(sent.get()); - let mut received = vec![0; attachment.len()]; + let mut received = vec![0; attachment.len() + acknowledgement.len()]; (&broker).read_exact(&mut received).unwrap(); - assert_eq!(received, attachment); + assert_eq!(received, [attachment, acknowledgement].concat()); "captured" }, ) .unwrap(); assert_eq!(captured, "captured"); - // The worker read nothing of the session. + // The session that a clone resumes is active in the broker's first epoch, so that the + // restore's reset, of another instance, is the only one that the clone reads. + assert_eq!(session.state(), GuestState::Active); + assert_eq!(session.identity(), Some(TEMPLATE)); + assert_eq!(session.window(), Some(RECEIVE_WINDOW)); + // The worker read nothing past the reset. let mut waiting = [0; 7]; (&guest).read_exact(&mut waiting).unwrap(); assert_eq!(&waiting, b"waiting"); } #[test] - fn no_capture_runs_without_the_attachment() { - let (guest, _broker) = UnixStream::pair().unwrap(); + fn no_capture_runs_without_the_session_s_handshake() { + // The tty does not send the acknowledgement. + let (guest, broker) = UnixStream::pair().unwrap(); + (&broker).write_all(&first(RecordType::Reset, &[])).unwrap(); let unsent = io::Error::new(io::ErrorKind::TimedOut, "the tty sent nothing"); let error = attached::<_, ()>(&guest, |_| Err(unsent), || panic!("captured")).unwrap_err(); assert!( @@ -412,6 +592,7 @@ mod tests { "{error}" ); + // The attachment cannot be written. let (guest, broker) = UnixStream::pair().unwrap(); drop(broker); let error = @@ -420,5 +601,30 @@ mod tests { matches!(error, ServeError::Session(GuestConnectionError::Io(_))), "{error}" ); + + // The broker closes the stream, or sends anything but a reset first. + let (guest, broker) = UnixStream::pair().unwrap(); + broker.shutdown(Shutdown::Write).unwrap(); + let error = + attached::<_, ()>(&guest, |_| panic!("sent"), || panic!("captured")).unwrap_err(); + assert!( + matches!(error, ServeError::Session(GuestConnectionError::Closed)), + "{error}" + ); + let (guest, broker) = UnixStream::pair().unwrap(); + (&broker) + .write_all(&first(RecordType::Data, b"early")) + .unwrap(); + let error = + attached::<_, ()>(&guest, |_| panic!("sent"), || panic!("captured")).unwrap_err(); + assert!( + matches!( + error, + ServeError::Session(GuestConnectionError::Protocol(SessionError::Unexpected( + RecordType::Data + ))) + ), + "{error}" + ); } } diff --git a/tests/runtime-adapter/src/worker/connection.rs b/tests/runtime-adapter/src/worker/connection.rs index aa4f351b0..cae5a6ff4 100644 --- a/tests/runtime-adapter/src/worker/connection.rs +++ b/tests/runtime-adapter/src/worker/connection.rs @@ -7,11 +7,14 @@ //! earlier epoch's partial message, and credits each DATA payload back, once it decoded the //! payload's messages. A reset also drops the records that the stream has not begun to take, //! which belong to the earlier epoch, and ends the one that it began, so that the stream stays -//! whole. +//! whole. In a clone, the connection resumes the session that the template's VM attached before +//! its capture, and whose first epoch it acknowledged, which the snapshot holds, rather than +//! attaching: the restore's broker resets it, which the connection acknowledges as any reset. use std::collections::VecDeque; use std::io::{self, Read, Write}; use std::os::fd::{AsFd, BorrowedFd}; +use std::time::{Duration, Instant}; use nix::errno::Errno; use nix::fcntl::{self, FcntlArg, OFlag}; @@ -81,12 +84,33 @@ impl Connection { /// Makes `stream` nonblocking, and returns the connection over it, with the guest's /// attachment queued. pub(super) fn new(stream: S) -> Result { + let mut connection = Self::over(stream, GuestSession::new())?; + connection.record(GuestSession::attach)?; + Ok(connection) + } + + /// Makes `stream` nonblocking, and returns the connection over it, with `session`, the guest's + /// role of a session that the guest attached already, on the same stream: that of a template's + /// VM, which attached before its capture, and acknowledged its broker's reset, in a clone that + /// a restore of the snapshot started, whose broker resets the session that the snapshot holds + /// again, with another instance. The connection queues no attachment. + /// + /// A session that has not attached takes no record, so that the connection fails as the + /// guest agent's blocking `session` does once the broker sends one. + #[cfg(target_arch = "x86_64")] + pub(super) fn resume(stream: S, session: GuestSession) -> Result { + Self::over(stream, session) + } + + /// Makes `stream` nonblocking, and returns the connection over it, with `session`, the guest's + /// role of the session, and nothing queued. + fn over(stream: S, session: GuestSession) -> Result { let flags = fcntl::fcntl(&stream, FcntlArg::F_GETFL).map_err(failed)?; let flags = OFlag::from_bits_retain(flags) | OFlag::O_NONBLOCK; fcntl::fcntl(&stream, FcntlArg::F_SETFL(flags)).map_err(failed)?; - let mut connection = Self { + Ok(Self { stream, - session: GuestSession::new(), + session, input: vec![0; READ_CHUNK].into_boxed_slice(), start: 0, end: 0, @@ -97,9 +121,7 @@ impl Connection { output: VecDeque::new(), written: 0, queued: 0, - }; - connection.record(GuestSession::attach)?; - Ok(connection) + }) } /// Returns what the connection received next, or `None` until the stream has more. @@ -198,15 +220,29 @@ impl Connection { Ok(()) } - /// Writes every queued record, waiting while the stream takes none. - pub(super) fn finish(&mut self) -> Result<(), ServeError> { + /// Writes every queued record, waiting while the stream takes none, until `deadline` passes, + /// if there is one, after which it leaves the rest queued. + pub(super) fn finish(&mut self, deadline: Option) -> Result<(), ServeError> { loop { self.flush()?; if self.output.is_empty() { return Ok(()); } + let timeout = match deadline { + None => PollTimeout::NONE, + Some(deadline) => { + let left = deadline.saturating_duration_since(Instant::now()); + if left.is_zero() { + return Ok(()); + } + // Poll counts whole milliseconds, so a shorter wait would return before the + // deadline. + PollTimeout::try_from(left.max(Duration::from_millis(1))) + .unwrap_or(PollTimeout::MAX) + } + }; let mut descriptors = [PollFd::new(self.stream.as_fd(), PollFlags::POLLOUT)]; - match poll::poll(&mut descriptors, PollTimeout::NONE) { + match poll::poll(&mut descriptors, timeout) { Ok(_) | Err(Errno::EINTR) => {} Err(error) => return Err(failed(error)), } diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index e7cd2ac44..31e414656 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -21,6 +21,16 @@ //! kernel command line names no layered root, where PID 1 also collects the statuses that real-time //! signals leave, whatever the names of its children. //! +//! On x86_64, the worker of a clone, which a restore of a template's snapshot started, resumes the +//! session that the template's VM attached before its capture, active in the epoch of its broker's +//! first reset, with the same fake root operations, whose application of an activation record +//! mirrors the guest's: the tests cover the reset that the restore queued, the one `Activate` that +//! the clone takes before it is ready, the application of its record, the `Ready` that reports the +//! activation, retries of the `Activate`, the order that the worker serves once the clone is +//! ready, the status of each class of the activation's failures, which the worker answers with +//! nothing, before any side effect where the class allows, its deadline, also once the stream +//! takes nothing more, and the record's workload identity, as which every execution runs. +//! //! The worker's executions run behind the test broker too, whose host side drives the engine's //! peers through the host program's own `exec::HostExecution`: the tests cover several executions //! at once, byte-exact, a stalled sink and a full stream to the host, which stall only what waits @@ -755,7 +765,7 @@ fn the_guest_agent_refuses_messages_that_its_framing_does_not_allow() { }; let cases = [ (header([0; 4], 0), FrameError::Kind(0)), - (header([0; 4], 27), FrameError::Kind(27)), + (header([0; 4], 28), FrameError::Kind(28)), ( header(too_long, 3), FrameError::Length { @@ -776,8 +786,8 @@ fn the_guest_agent_refuses_messages_that_its_framing_does_not_allow() { // The guest agent answers requests, and refuses the messages that only it sends, and those // about a layered root, which its VM does not have: its configuration, a part of its mapping - // table, and the start of its workloads, and executions, which only the worker of a layered - // root runs. + // table, and the start of its workloads, a clone's activation, and executions, which only the + // worker of a layered root runs. let refused_messages = [ Message::Pong, Message::Stopped, @@ -791,6 +801,7 @@ fn the_guest_agent_refuses_messages_that_its_framing_does_not_allow() { Message::RootConfig(RootConfig::default().encode()), Message::Maps(maps_part()), Message::Start, + Message::Activate(Vec::new()), Message::ExecStart(Vec::new()), Message::ExecCredit(Vec::new()), Message::ExecTerminal(Vec::new()), @@ -990,13 +1001,13 @@ fn the_guest_binary_serves_the_session_on_its_standard_streams() { // The host sends a malformed message. let (mut broker, child) = start_binary(); broker.attach(FIRST); - broker.send_data(&[0, 0, 0, 0, 27], OUTER_MAX_PAYLOAD); + broker.send_data(&[0, 0, 0, 0, 28], OUTER_MAX_PAYLOAD); let output = child.finished(); assert_eq!(output.status.code(), Some(1), "{output:?}"); let diagnostic = String::from_utf8_lossy(&output.stderr); assert_eq!( diagnostic, - "runtime-adapter-guest: unknown message kind 27\n" + "runtime-adapter-guest: unknown message kind 28\n" ); } @@ -1137,6 +1148,14 @@ enum Call { Probe(Probe), /// The teardown ran. Teardown, + /// A clone's activation record was applied with the inputs that its template captured. + #[cfg(target_arch = "x86_64")] + Apply( + Box<( + nvx::activation::ActivationRecord, + nvx::guest::clone::RootTemplate, + )>, + ), } /// Root operations that record the worker's calls, and fail as a test asks. @@ -1154,11 +1173,15 @@ struct FakeRoot { latched: bool, /// How long the assembly takes. assembly: Duration, - /// The gate that each probe waits on, once it recorded its call, until the test opens it, if - /// any. + /// The gate that each probe, and each of a clone's applications, waits on, once it recorded + /// its call, until the test opens it, if any. gate: Option>, /// The worker's calls, in order. calls: Vec, + /// The deadline of each of a clone's applications, in order, until which it waits for the + /// devices at the latest. + #[cfg(target_arch = "x86_64")] + deadlines: Vec>, /// The workload cgroup of the test's own, below which the worker runs its executions in /// cgroups of their own, where the test can create cgroups. workload: Option, @@ -1197,6 +1220,8 @@ impl FakeRoot { assembly: Duration::ZERO, gate: None, calls: Vec::new(), + #[cfg(target_arch = "x86_64")] + deadlines: Vec::new(), workload, } } @@ -1223,8 +1248,8 @@ impl FakeRoot { self } - /// Makes each probe wait until the test opens the gate, which the returned sender opens for - /// one probe at a time. + /// Makes each probe, and each of a clone's applications, wait until the test opens the gate, + /// which the returned sender opens for one at a time. fn gated(mut self) -> (Self, Sender<()>) { let (open, gate) = mpsc::channel(); self.gate = Some(gate); @@ -1307,6 +1332,51 @@ impl RootOperations for FakeRoot { self.calls.push(Call::Teardown); self.result(Operation::Teardown) } + + /// Applies the record as `nvx::guest::clone::apply` does, without a root, once the test opened + /// the gate, if any: it refuses a GPT disk before anything else, takes as long as the + /// assembly, but no longer than until `deadline`, as the waits for the devices do, and checks + /// the account of the record's workload identity, if it has one, in existing-account mode, in + /// the [accounts' root](accounts), which stands for the root that it assembled. + #[cfg(target_arch = "x86_64")] + fn apply( + &mut self, + record: &nvx::activation::ActivationRecord, + template: &nvx::guest::clone::RootTemplate, + deadline: Option, + ) -> Result, nvx::guest::clone::ApplyError> { + use nvx::guest::clone::ApplyError; + + let applied = (record.clone(), template.clone()); + self.calls.push(Call::Apply(Box::new(applied))); + self.deadlines.push(deadline); + if let Some(gate) = &self.gate { + gate.recv_timeout(TIMEOUT) + .expect("the test opens the gate in time"); + } + let configuration = record.configuration(); + if matches!(configuration.image, nvx::activation::Image::Gpt { .. }) { + return Err(ApplyError::Gpt); + } + let assembly = match deadline { + Some(deadline) => self + .assembly + .min(deadline.saturating_duration_since(Instant::now())), + None => self.assembly, + }; + thread::sleep(assembly); + let Some(workload) = configuration.workload.identity else { + return Ok(None); + }; + let identity = entry::Identity::new(workload.uid, workload.gid); + match entry::Entry::new(accounts(), identity.clone()).prepare() { + Ok(_) => Ok(Some(identity)), + Err(error) => Err(ApplyError::Account { + error, + teardown: None, + }), + } + } } impl EntryOperations for FakeRoot { @@ -1933,13 +2003,15 @@ fn the_worker_accepts_only_its_configuration_or_a_stop_before_its_root() { assert_eq!(worked.unwrap(), Worked::Stopped); assert_eq!(root.calls, []); - // Any other message ends it with an error, as do the messages that only the guest sends. + // Any other message ends it with an error, as do the messages that only the guest sends, and + // a clone's activation, which a VM that boots cold does not take. let refused = [ Message::Ping, Message::Echo(b"early".to_vec()), Message::Probe(b"mounts".to_vec()), Message::Maps(maps_part()), Message::Start, + Message::Activate(Vec::new()), Message::Pong, ready(), Message::Report(Vec::new()), @@ -1958,17 +2030,20 @@ fn the_worker_accepts_only_its_configuration_or_a_stop_before_its_root() { assert_eq!(root.calls, [], "{message:?}"); } - // Once the root is ready, a second configuration ends it with an error too, and the worker - // leaves the root that it assembled to its operations. - let (mut broker, working) = start_worker(FakeRoot::booting(&layout())); - broker.attach(FIRST); - let replies = broker.exchange(&[configuration()], OUTER_MAX_PAYLOAD, 1); - assert_eq!(replies, [ready()]); - broker.send_data(&configuration().encode().unwrap(), OUTER_MAX_PAYLOAD); - let (root, worked) = joined(working); - let failed = matches!(&worked, Err(ServeError::Unexpected(6))); - assert!(failed, "{worked:?}"); - assert!(!root.calls.contains(&Call::Teardown), "{:?}", root.calls); + // Once the root is ready, a second configuration ends it with an error too, as does a clone's + // activation, and the worker leaves the root that it assembled to its operations. + for refused in [configuration(), Message::Activate(Vec::new())] { + let (mut broker, working) = start_worker(FakeRoot::booting(&layout())); + broker.attach(FIRST); + let replies = broker.exchange(&[configuration()], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [ready()]); + broker.send_data(&refused.encode().unwrap(), OUTER_MAX_PAYLOAD); + let (root, worked) = joined(working); + let failed = + matches!(&worked, Err(ServeError::Unexpected(kind)) if *kind == refused.kind()); + assert!(failed, "{refused:?}: {worked:?}"); + assert!(!root.calls.contains(&Call::Teardown), "{:?}", root.calls); + } // So does a broker that closes the stream. let (mut broker, working) = start_worker(FakeRoot::booting(&layout())); @@ -3889,3 +3964,729 @@ fn the_worker_runs_executions_on_a_tty() { let (_, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); } + +/// The worker of a clone, which a restore of a template's snapshot started, on x86_64, whose time +/// ABI a clone needs. It resumes the session that the template's VM attached before its capture, +/// answers the reset that the restore queued, takes nothing but the host's `Activate` until the +/// clone is ready, applies its record, and then serves the host's order, as after a cold boot. +#[cfg(target_arch = "x86_64")] +mod clones { + use std::num::NonZeroU64; + + use nvx::activation::{ + self, ActivationRecord, Configuration, ExpectedResources, MachineIdMode, ReadyActivation, + RestoreSection, StoreGeneration, TemplateReference, TransactionId, WorkloadIdentity, + WorkloadView, + }; + use nvx::guest::clone::{Activation, AdmissionFence, RootTemplate}; + use nvx::protocol::nvxs::{GuestEvent, GuestSession}; + use nvx::protocol::status; + use runtime_adapter::frame::MAX_ACTIVATE_BODY; + use runtime_adapter::worker::UNACTIVATED_DRAIN; + use runtime_adapter::worker::activation::ACTIVATION_TIMEOUT; + use runtime_adapter::worker::capture::{self, Admission}; + + use super::*; + + /// Identity of the epoch that the broker of the clones' template started with the reset with + /// which it answered the template's attachment, before the capture. + const TEMPLATE: Identity = Identity { + instance_id: [0x7e; 16], + epoch: 1, + }; + + /// Identity of the epoch that the broker of a clone starts with the reset that the restore + /// queued, of another instance. + const RESTORED: Identity = Identity { + instance_id: [0xc1; 16], + epoch: 1, + }; + + /// The CPU count and the memory size of the tests' clones, as their repair found them. + const RESOURCES: ExpectedResources = ExpectedResources { + cpus: 2, + memory: 1 << 30, + }; + + /// Returns the record of a clone's Activate of the transaction `transaction`, whose workload + /// runs as `identity`, if any: an EROFS image over a scratch, without an export, a mapping, or + /// a network, which expects [`RESOURCES`]. + fn record(transaction: u8, identity: Option<(u32, u32)>) -> ActivationRecord { + let configuration = Configuration { + image: activation::Image::Erofs { + binding: activation::Identity::from_bytes([0x11; 32]), + uuid: None, + }, + upper: activation::Upper::Scratch { + binding: activation::Identity::from_bytes([0x22; 32]), + journal: activation::Journal::Forbidden, + }, + overlay: activation::OverlayMode::UntrustedLowers, + export: None, + mappings: 0, + network: None, + workload: WorkloadView { + hostname: b"clone".to_vec(), + machine_id: MachineIdMode::Image, + identity: identity.map(|(uid, gid)| WorkloadIdentity { uid, gid }), + }, + resources: RESOURCES, + }; + let restore = RestoreSection { + transaction: TransactionId::from_bytes([transaction; 16]), + template: TemplateReference { + generation: StoreGeneration::new(NonZeroU64::MIN), + key: activation::Identity::from_bytes([0x44; 32]), + }, + }; + ActivationRecord::new(Some(restore), configuration).unwrap() + } + + /// Returns the record of `record`'s transaction whose configuration `change` changed. + fn changed( + record: &ActivationRecord, + change: impl FnOnce(&mut Configuration), + ) -> ActivationRecord { + let mut configuration = record.configuration().clone(); + change(&mut configuration); + ActivationRecord::new(record.restore().cloned(), configuration).unwrap() + } + + /// Returns the activation of a clone that the worker repaired, with [`RESOURCES`], behind the + /// worker's fence, which the capture's barrier handed over closed, and whose deadline ends + /// `timeout` from now, as from the repair's acknowledgement of the restore. + fn repaired(timeout: Duration) -> Activation { + let mut fence = Admission::default(); + fence.close().unwrap(); + let now = Instant::now(); + Activation::with_resources(RESOURCES, Some(now), fence, timeout, now) + } + + /// Returns the inputs of the assembly of the clones' roots that their template captured: + /// those of the template's own configuration. + fn template() -> RootTemplate { + capture::root_template(&capture::template_config()) + } + + /// Returns the guest's role of the session that the template's VM attached before its + /// capture, as the snapshot holds it: active in the epoch of [`TEMPLATE`], which the + /// template's broker started with the reset with which it answered the attachment, and which + /// the template acknowledged. + fn captured_session() -> GuestSession { + let mut session = GuestSession::new(); + let mut output = Vec::new(); + session.attach(&mut output).unwrap(); + let reset = super::record(RecordType::Reset, TEMPLATE, 0, &[]); + let (consumed, event) = session.receive(&reset).unwrap(); + assert_eq!( + (consumed, event), + (reset.len(), Some(GuestEvent::Reset(TEMPLATE))) + ); + session.acknowledge(RECEIVE_WINDOW, &mut output).unwrap(); + session + } + + /// Starts a clone's worker in a thread, with the root operations `root`, on one end of a + /// socket pair, and returns the broker on the other end with the thread. The worker resumes + /// the session that the template's VM attached, so it sends no attachment, and its + /// activation's deadline ends `timeout` from now. + fn start_clone(root: FakeRoot, timeout: Duration) -> (Broker, Working) { + start_clone_after(root, timeout, |_| {}) + } + + /// Starts a clone's worker as [`start_clone`] does, once `ahead` sent what the broker sends + /// before the worker runs, which the worker then takes at once, however late its thread runs. + fn start_clone_after( + root: FakeRoot, + timeout: Duration, + ahead: impl FnOnce(&mut Broker), + ) -> (Broker, Working) { + let (broker, guest) = UnixStream::pair().unwrap(); + let mut broker = Broker::new(broker); + ahead(&mut broker); + (broker, run_clone(guest, root, timeout)) + } + + /// Runs a clone's worker in a thread, with the root operations `root`, on `guest`, the guest's + /// end of the stream, and returns the thread, as [`start_clone`] does. + fn run_clone(guest: UnixStream, root: FakeRoot, timeout: Duration) -> Working { + let session = captured_session(); + let activation = repaired(timeout); + thread::spawn(move || { + let mut root = root; + let worked = worker::work_clone(guest, session, &mut root, activation, template()); + (root, worked) + }) + } + + /// Returns the broker's end and the guest's end of a socket pair whose guest direction is + /// full, as once the broker failed: the stream takes nothing more from the guest. + fn stalled() -> (UnixStream, UnixStream) { + let (broker, guest) = UnixStream::pair().unwrap(); + let filler = guest.try_clone().unwrap(); + filler.set_nonblocking(true).unwrap(); + loop { + match (&filler).write(&[0; 4096]) { + Ok(_) => {} + Err(error) if error.kind() == io::ErrorKind::WouldBlock => break, + Err(error) => panic!("cannot fill the stream: {error}"), + } + } + (broker, guest) + } + + impl Broker { + /// Starts the epoch of `identity` with a RESET, as [`reset`](Self::reset) does, but takes + /// the guest's acknowledgement later, with [`acknowledged`](Self::acknowledged), as from a + /// guest that may not run yet: meanwhile, the broker takes the guest's receive window as + /// the guest agent's, which the acknowledgement grants. + fn reset_ahead(&mut self, identity: Identity) { + self.identity = identity; + self.send_sequence = 0; + self.receive_sequence = 0; + self.send(RecordType::Reset, &[]); + self.room = RECEIVE_WINDOW as usize; + } + + /// Takes the guest's acknowledgement of the reset that [`reset_ahead`](Self::reset_ahead) + /// sent. + fn acknowledged(&mut self) { + let ack = self.expect(); + assert_eq!(ack.record_type(), RecordType::Ack); + assert_eq!(ack.window(), Some(RECEIVE_WINDOW)); + } + } + + /// Returns the host's Activate of `record`. + fn activate(record: &ActivationRecord) -> Message { + Message::Activate(record.encode()) + } + + /// Returns the worker's answer to the Activate of `record` once it activated the clone: its + /// readiness, with the guest agent's build, and the activation, which reports that it applied, + /// with the record's transaction. + fn activated(record: &ActivationRecord) -> Message { + let ready = Ready::new(GUEST_BUILD).unwrap(); + Message::Ready(ready.with_activation(ReadyActivation::of(record)).encode()) + } + + /// Returns the call of a fake root that applied `record` with the clones' [`template`]. + fn applied(record: &ActivationRecord) -> Call { + Call::Apply(Box::new((record.clone(), template()))) + } + + /// Returns the root operations of the worker that runs in `working`, once it ended because the + /// clone's activation failed with the status `expected`, and checks that it answered nothing, + /// but credited what it took, before it ended its stream. + fn unactivated(broker: &mut Broker, working: Working, expected: u8) -> FakeRoot { + let (root, worked) = joined(working); + let worked = worked.unwrap(); + assert!(matches!(worked, Worked::Unactivated { .. }), "{worked:?}"); + assert_eq!(worked.status(), expected, "{worked:?}"); + while let Some(record) = broker.read() { + broker.check(&record); + assert_eq!(broker.accept(&record), [], "{worked:?}"); + } + root + } + + impl Host { + /// Starts a clone's worker with the root operations `root`, and returns the host, once the + /// worker activated the clone with `record`, which announces no mapping, and started the + /// sandbox. The host waits for Ready before it starts the sandbox. + fn activated(root: FakeRoot, record: &ActivationRecord) -> Self { + let (mut broker, working) = start_clone(root, ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + let replies = broker.exchange(&[activate(record)], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [activated(record)]); + let replies = broker.exchange(&[Message::Start], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [Message::Started]); + Self { + broker, + working: Some(working), + executions: BTreeMap::new(), + replies: Vec::new(), + stopping: false, + } + } + } + + #[test] + fn a_clone_answers_the_restore_s_reset_and_activates_with_the_record_of_its_activate() { + let (mut broker, working) = start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + // The worker sends no attachment: it acknowledges the reset that the restore queued, with + // its receive window, first. + broker.reset(RESTORED); + let record = record(1, Some(WORKLOAD)); + let replies = broker.exchange(&[activate(&record)], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [activated(&record)]); + let Message::Ready(body) = &replies[0] else { + panic!("{replies:?}"); + }; + let ready = Ready::decode(body).unwrap(); + assert_eq!(ready.build(), GUEST_BUILD); + assert_eq!(ready.activation(), Some(ReadyActivation::of(&record))); + + // A retry of the Activate, byte for byte, gets the same Ready, without a second + // application, in records of any size, and after a later reset, as once the host attached + // again. + let replies = broker.exchange(&[activate(&record)], 7, 1); + assert_eq!(replies, [activated(&record)]); + broker.settle(); + broker.reset(Identity { + epoch: 2, + ..RESTORED + }); + let replies = broker.exchange(&[activate(&record)], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [activated(&record)]); + + // The worker then serves the host's order, as after a cold boot: the record announces no + // mapping, so the sandbox starts at once, and a retry still gets the same Ready. + let messages = [ + Message::Ping, + Message::Start, + Message::Probe(b"mounts".to_vec()), + activate(&record), + Message::Stop, + ]; + let expected = [ + Message::Pong, + Message::Started, + Message::Report(found(&probe("mounts")).encode()), + activated(&record), + Message::Stopped, + ]; + let replies = broker.exchange(&messages, OUTER_MAX_PAYLOAD, expected.len()); + assert_eq!(replies, expected); + let (root, worked) = joined(working); + let worked = worked.unwrap(); + assert_eq!((&worked, worked.status()), (&Worked::Stopped, 0)); + assert!(broker.read().is_none(), "the worker ended its stream"); + // It applied the record once, with the template's inputs, and read no command line, + // bootstrapped nothing, and assembled no root of a configuration: the template + // bootstrapped the guest before its capture. + let expected = [ + applied(&record), + Call::Probe(probe("mounts")), + Call::Teardown, + ]; + assert_eq!(root.calls, expected); + } + + #[test] + fn a_clone_takes_nothing_but_its_activate_before_it_is_ready() { + // Any other message fails the activation, finally, with the status of the host's session, + // a stop and a ping included, as do the messages that only the guest sends. The worker + // answers nothing, and applies, mounts, and spawns nothing. + let refused = [ + Message::Ping, + Message::Stop, + Message::Echo(b"early".to_vec()), + Message::Probe(b"mounts".to_vec()), + configuration(), + Message::Maps(maps_part()), + Message::Start, + HostMessage::Start(start_of(1, shell("exit 0"))).encode(), + Message::Pong, + ready(), + Message::Started, + ]; + for message in refused { + let (mut broker, working) = + start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + broker.send_data(&message.encode().unwrap(), OUTER_MAX_PAYLOAD); + let root = unactivated(&mut broker, working, status::ACTIVATION_PROTOCOL); + assert_eq!(root.calls, [], "{message:?}"); + } + + // So does another reset, which the worker acknowledges first. + let (mut broker, working) = start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + broker.reset(Identity { + epoch: 2, + ..RESTORED + }); + let root = unactivated(&mut broker, working, status::ACTIVATION_PROTOCOL); + assert_eq!(root.calls, []); + } + + #[test] + fn a_clone_s_activation_fails_finally_with_the_status_of_the_failure_s_class() { + // The activation refuses a record that does not decode, a cold boot's, and one that + // expects other resources than the repair found, before anything has a side effect. + let clone = record(1, None); + let cold = ActivationRecord::new(None, clone.configuration().clone()).unwrap(); + let more = changed(&clone, |configuration| configuration.resources.memory *= 2); + let cases = [ + (Message::Activate(Vec::new()), status::ACTIVATION_RECORD), + ( + Message::Activate(b"\x01".to_vec()), + status::ACTIVATION_RECORD, + ), + (activate(&cold), status::ACTIVATION_RECORD), + (activate(&more), status::ACTIVATION_RESOURCES), + ]; + for (message, expected) in cases { + let (mut broker, working) = + start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + broker.send_data(&message.encode().unwrap(), OUTER_MAX_PAYLOAD); + let root = unactivated(&mut broker, working, expected); + assert_eq!(root.calls, [], "{message:?}"); + } + + // The application refuses a GPT disk, which a clone does not assemble, and a workload + // identity whose account the image lacks, or whose primary group it lacks, before Ready. + let gpt = changed(&clone, |configuration| { + configuration.image = activation::Image::Gpt { + binding: activation::Identity::from_bytes([0x11; 32]), + disk_guid: None, + first_partition_bound: 1 << 20, + }; + }); + let cases = [ + (gpt, status::ACTIVATION_ROOT), + (record(1, Some((4242, 4242))), status::ACTIVATION_WORKLOAD), + (record(1, Some(ORPHAN)), status::ACTIVATION_WORKLOAD), + ]; + for (record, expected) in cases { + let (mut broker, working) = + start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + broker.send_data(&activate(&record).encode().unwrap(), OUTER_MAX_PAYLOAD); + let root = unactivated(&mut broker, working, expected); + assert_eq!(root.calls, [applied(&record)]); + } + } + + #[test] + fn a_clone_fails_on_an_activate_longer_than_a_record_before_it_takes_its_body() { + // The framing refuses the header of an Activate that announces more bytes than a record + // can have before it takes any byte of its body, and the activation takes it as a record + // that long: it fails with the record's status, whether the clone is ready or not, and the + // worker answers nothing more. + let accepted = record(1, Some(WORKLOAD)); + let kind = activate(&accepted).kind(); + for length in [MAX_ACTIVATE_BODY + 1, usize::try_from(u32::MAX).unwrap()] { + let mut header = u32::try_from(length).unwrap().to_le_bytes().to_vec(); + header.push(kind); + for ready in [false, true] { + let (mut broker, working) = + start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + let mut calls = Vec::new(); + if ready { + let replies = broker.exchange(&[activate(&accepted)], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [activated(&accepted)]); + calls.push(applied(&accepted)); + } + broker.send_data(&header, OUTER_MAX_PAYLOAD); + let (root, worked) = joined(working); + let worked = worked.unwrap(); + let Worked::Unactivated { + status: code, + problem, + } = &worked + else { + panic!("{length}, ready {ready}: {worked:?}"); + }; + assert_eq!(*code, status::ACTIVATION_RECORD, "{problem}"); + assert!(problem.contains(&format!("{length} bytes")), "{problem}"); + while let Some(record) = broker.read() { + broker.check(&record); + assert_eq!(broker.accept(&record), [], "{problem}"); + } + assert_eq!(root.calls, calls, "{problem}"); + } + } + } + + #[test] + fn a_clone_that_the_host_does_not_activate_in_time_ends_with_the_deadline_s_status() { + // Whether the restore's reset came or not, or part of an Activate, the deadline ends the + // activation, before anything has a side effect. The broker sends what it sends before the + // worker runs, which then takes it at once: a worker that runs only once the deadline + // passed still acknowledges the reset, and fails with the deadline's status. + let record = record(1, Some(WORKLOAD)); + let bytes = activate(&record).encode().unwrap(); + let cases: [Option<&[u8]>; 3] = [None, Some(&[]), Some(&bytes[..bytes.len() / 2])]; + for sent in cases { + let deadline = Duration::from_millis(200); + let root = FakeRoot::booting(&layout()); + let (mut broker, working) = start_clone_after(root, deadline, |broker| { + if let Some(sent) = sent { + broker.reset_ahead(RESTORED); + broker.send_data(sent, OUTER_MAX_PAYLOAD); + } + }); + if sent.is_some() { + broker.acknowledged(); + } + let root = unactivated(&mut broker, working, status::ACTIVATION_DEADLINE); + assert_eq!(root.calls, [], "{sent:?}"); + } + + // Once the clone is ready, the deadline ends nothing. + let deadline = Duration::from_secs(2); + let root = FakeRoot::booting(&layout()); + let (mut broker, working) = start_clone_after(root, deadline, |broker| { + broker.reset_ahead(RESTORED); + broker.send_data(&bytes, OUTER_MAX_PAYLOAD); + }); + broker.acknowledged(); + let mut replies = Vec::new(); + while replies.is_empty() { + replies.extend(broker.take()); + } + assert_eq!(replies, [activated(&record)]); + thread::sleep(deadline + Duration::from_millis(500)); + let replies = broker.exchange(&[Message::Ping, Message::Stop], OUTER_MAX_PAYLOAD, 2); + assert_eq!(replies, [Message::Pong, Message::Stopped]); + let (_, worked) = joined(working); + assert_eq!(worked.unwrap(), Worked::Stopped); + } + + #[test] + fn a_clone_s_application_ends_by_the_activation_s_deadline() { + // The worker hands the application the activation's deadline, until which it waits for + // the devices at the latest: an Activate whose application would outlast the deadline, + // such as one whose device never appears, ends the clone with the deadline's status once + // the deadline passed, rather than once the application's waits would have ended, with + // no Ready. + let timeout = Duration::from_secs(2); + let slow = Duration::from_secs(60); + let root = FakeRoot::booting(&layout()).assembling_for(slow); + let started = Instant::now(); + let (mut broker, working) = start_clone(root, timeout); + broker.reset(RESTORED); + let record = record(1, Some(WORKLOAD)); + broker.send_data(&activate(&record).encode().unwrap(), OUTER_MAX_PAYLOAD); + let root = unactivated(&mut broker, working, status::ACTIVATION_DEADLINE); + let ended = Instant::now(); + assert!(ended - started < slow, "{:?}", ended - started); + assert_eq!(root.calls, [applied(&record)]); + // It is the activation's deadline: `timeout` after the repair's acknowledgement of the + // restore, which the test's clone made once the test started. + let [Some(deadline)] = root.deadlines[..] else { + panic!("{:?}", root.deadlines); + }; + assert!(deadline >= started + timeout, "{deadline:?}"); + assert!(deadline <= ended + timeout, "{deadline:?}"); + } + + #[test] + fn a_clone_takes_what_came_during_its_application_before_it_is_ready() { + // The application runs until the test opens the gate, by when the host sent more after + // its Activate, before Ready, which the worker takes before it completes the activation: + // another reset, which the worker acknowledges, or a request fails the activation, with + // the status of the host's session, after one application, and with no Ready. + let record = record(1, Some(WORKLOAD)); + let bytes = activate(&record).encode().unwrap(); + for reset in [true, false] { + let (root, gate) = FakeRoot::booting(&layout()).gated(); + let (mut broker, working) = start_clone(root, ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + broker.send_data(&bytes, OUTER_MAX_PAYLOAD); + if reset { + broker.reset_ahead(Identity { + epoch: 2, + ..RESTORED + }); + } else { + broker.send_data(&Message::Ping.encode().unwrap(), OUTER_MAX_PAYLOAD); + } + gate.send(()).unwrap(); + if reset { + broker.acknowledged(); + } + let root = unactivated(&mut broker, working, status::ACTIVATION_PROTOCOL); + assert_eq!(root.calls, [applied(&record)], "reset {reset}"); + } + + // A retry of the Activate, byte for byte, joins the application: the worker answers both + // with Ready, after one application, then serves the host's order. + let (root, gate) = FakeRoot::booting(&layout()).gated(); + let (mut broker, working) = start_clone(root, ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + broker.send_data(&bytes, OUTER_MAX_PAYLOAD); + broker.send_data(&bytes, OUTER_MAX_PAYLOAD); + gate.send(()).unwrap(); + let mut replies = Vec::new(); + while replies.len() < 2 { + replies.extend(broker.take()); + } + assert_eq!(replies, [activated(&record), activated(&record)]); + let replies = broker.exchange(&[Message::Stop], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [Message::Stopped]); + let (root, worked) = joined(working); + assert_eq!(worked.unwrap(), Worked::Stopped); + assert_eq!(root.calls, [applied(&record), Call::Teardown]); + } + + #[test] + fn a_clone_whose_stream_takes_nothing_still_ends_once_its_activation_failed() { + // The guest's direction of the stream is full, as once the broker failed: the worker + // cannot send its acknowledgement of the restore's reset, and still ends once the + // deadline passed, with the deadline's status, once it waited for the stream for + // `UNACTIVATED_DRAIN`. + let (broker, guest) = stalled(); + let reset = super::record(RecordType::Reset, RESTORED, 0, &[]); + (&broker).write_all(&reset).unwrap(); + let deadline = Duration::from_millis(200); + let started = Instant::now(); + let working = run_clone(guest, FakeRoot::booting(&layout()), deadline); + let (root, worked) = joined(working); + let worked = worked.unwrap(); + assert_eq!(worked.status(), status::ACTIVATION_DEADLINE, "{worked:?}"); + assert!(started.elapsed() >= deadline + UNACTIVATED_DRAIN); + assert_eq!(root.calls, []); + + // So does a worker whose broker closes its end of the stream before the clone is ready, + // as an abandoned clone, long before the deadline passes. + let (broker, guest) = stalled(); + (&broker).write_all(&reset).unwrap(); + broker.shutdown(Shutdown::Write).unwrap(); + let started = Instant::now(); + let working = run_clone(guest, FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + let (root, worked) = joined(working); + assert_eq!(worked.unwrap(), Worked::Abandoned); + assert!(started.elapsed() >= UNACTIVATED_DRAIN); + assert_eq!(root.calls, []); + } + + #[test] + fn a_clone_whose_broker_closes_the_stream_before_it_is_ready_is_abandoned() { + // Whether the restore's reset came or not, or part of an Activate, a broker that closes + // the stream before the clone is ready ends the clone with the status of the host's + // session, with nothing applied, and its fence closed. + let record = record(1, Some(WORKLOAD)); + let bytes = activate(&record).encode().unwrap(); + let cases: [Option<&[u8]>; 3] = [None, Some(&[]), Some(&bytes[..bytes.len() / 2])]; + for sent in cases { + let (mut broker, working) = + start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + if let Some(sent) = sent { + broker.reset(RESTORED); + broker.send_data(sent, OUTER_MAX_PAYLOAD); + } + broker.close(); + let (root, worked) = joined(working); + let worked = worked.unwrap(); + assert_eq!( + (&worked, worked.status()), + (&Worked::Abandoned, 1), + "{sent:?}" + ); + assert_eq!(root.calls, [], "{sent:?}"); + while let Some(record) = broker.read() { + broker.check(&record); + assert_eq!(broker.accept(&record), [], "{sent:?}"); + } + } + + // Once the clone is ready, it ends as the worker of a VM that boots cold does, and leaves + // the root that it assembled to its operations. + let (mut broker, working) = start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + let replies = broker.exchange(&[activate(&record)], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [activated(&record)]); + broker.close(); + let (root, worked) = joined(working); + let worked = worked.unwrap(); + assert_eq!((&worked, worked.status()), (&Worked::Closed, 1)); + assert_eq!(root.calls, [applied(&record)]); + } + + #[test] + fn a_clone_whose_broker_is_gone_still_ends_as_its_activation_tells() { + // The broker sends the restore's reset, and a ping, which fails the activation, or + // nothing more, and closes its socket before the worker runs: the worker cannot write + // its acknowledgement, and still ends with the activation's status, or as an abandoned + // clone, rather than with the failure of its stream. + let reset = super::record(RecordType::Reset, RESTORED, 0, &[]); + let ping = Message::Ping.encode().unwrap(); + let ping = super::record(RecordType::Data, RESTORED, 1, &ping); + let cases = [ + ( + [reset.as_slice(), &ping].concat(), + Some(status::ACTIVATION_PROTOCOL), + ), + (reset, None), + ]; + for (sent, expected) in cases { + let (broker, guest) = UnixStream::pair().unwrap(); + (&broker).write_all(&sent).unwrap(); + drop(broker); + let working = run_clone(guest, FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + let (root, worked) = joined(working); + let worked = worked.unwrap(); + match expected { + Some(expected) => { + assert!(matches!(worked, Worked::Unactivated { .. }), "{worked:?}"); + assert_eq!(worked.status(), expected, "{worked:?}"); + } + None => assert_eq!(worked, Worked::Abandoned), + } + assert_eq!(root.calls, [], "{worked:?}"); + } + } + + #[test] + fn a_ready_clone_fails_on_an_activate_that_is_not_a_retry() { + // An Activate of another transaction, or of the same transaction with another record, once + // the clone is ready or its sandbox started, fails the activation with the status of the + // host's session. The worker leaves the root that it assembled to its operations. + let accepted = record(1, Some(WORKLOAD)); + let cases = [ + (record(2, Some(WORKLOAD)), false), + (record(1, None), false), + (record(1, None), true), + ]; + for (other, started) in cases { + let (mut broker, working) = + start_clone(FakeRoot::booting(&layout()), ACTIVATION_TIMEOUT); + broker.reset(RESTORED); + let replies = broker.exchange(&[activate(&accepted)], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [activated(&accepted)]); + if started { + let replies = broker.exchange(&[Message::Start], OUTER_MAX_PAYLOAD, 1); + assert_eq!(replies, [Message::Started]); + } + broker.send_data(&activate(&other).encode().unwrap(), OUTER_MAX_PAYLOAD); + let root = unactivated(&mut broker, working, status::ACTIVATION_PROTOCOL); + assert_eq!(root.calls, [applied(&accepted)], "{other:?}"); + } + } + + #[test] + fn a_clone_s_executions_run_as_the_workload_identity_of_its_record() { + // The record's workload identity is the whole sandbox's, as a root configuration's is: + // every execution runs as it, and a start that names an identity is refused. + let refused = LaunchFailure::new(0, Errno::EINVAL as i32); + let workload = exec::Identity::new(WORKLOAD.0, WORKLOAD.1); + let mut host = Host::activated(FakeRoot::booting(&layout()), &record(1, Some(WORKLOAD))); + host.start(start_of(1, shell("exit 0")), Vec::new()); + let named = start_of(2, shell("exit 0")).with_identity(workload.clone()); + host.start(named, Vec::new()); + host.run_until(|host| (1..=2).all(|id| host.outcome(id).is_some())); + check_lossless(&host, 1, Status::Exited(0)); + assert_eq!(host.outcome(2).unwrap().launch_failure(), Some(refused)); + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + let identity = entry::Identity::new(WORKLOAD.0, WORKLOAD.1); + assert_eq!(entrances(&root), [entrance(identity.clone())]); + + // Without one, each start names its own. + let mut host = Host::activated(FakeRoot::booting(&layout()), &record(1, None)); + host.start(start_of(1, shell("exit 0")), Vec::new()); + let named = start_of(2, shell("exit 0")).with_identity(workload); + host.start(named, Vec::new()); + host.run_until(|host| (1..=2).all(|id| host.outcome(id).is_some())); + assert_eq!(host.outcome(1).unwrap().launch_failure(), Some(refused)); + check_lossless(&host, 2, Status::Exited(0)); + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + assert_eq!(entrances(&root), [entrance(identity)]); + } +} diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index a9a7424c0..a74ed462c 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -1143,7 +1143,7 @@ fn the_client_refuses_messages_that_its_framing_does_not_allow() { let too_long_text = u32::try_from(MAX_TEXT_BODY + 1).unwrap().to_le_bytes(); let cases = [ (header([0; 4], 0), FrameError::Kind(0)), - (header([0; 4], 27), FrameError::Kind(27)), + (header([0; 4], 28), FrameError::Kind(28)), ( header(too_long, 3), FrameError::Length {