From 73d640d1e050aa5d155b53b90ad9b9119f241991 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 14:52:19 -0700 Subject: [PATCH 1/2] nvx-rs: reserve the launch of a leased clone, and document the lock order A clone's lease now reserves the launch of its clone. The acquisition draws the launch's identity, records it in the lease's record and in the clone binding before any VMM of the clone exists, and returns it as Lease::launch. The supervisor takes the launch's identity from its caller: supervisor::launch and LockedStore::begin_launch take a LaunchId, which callers that launch no clone draw with LaunchId::generate, now public. A lease allows one launch of its clone, under the lock that acquired it: begin_launch refuses, with sandbox-leased, any other launch of a sandbox that holds a lease, a launch under another lock, and a second launch, until a release ends the lease. After a crash before the launch, a release ends the lease at once, and another acquisition checks the generation's eligibility again. A stop or a termination clears the runtime record only if it is still the record of its own VM, and not another VMM's under a launch identity that its caller used again. The supervisor's documentation describes the launch of a clone, its crashes, and one lock order: the template store's lock, then the sandbox's lifecycle lock, which guards its clone binding and launch journal, then the scratch image's claim. No lock lasts for a VM's lifetime. Tests crash a clone's orchestrator, in a process of its own, at each step of an acquisition and of a release, and around the spawn of the clone's VMM, and check what each evidence of a VM's exit means for its lease: an earlier boot, a reused process ID, a process that cannot be examined, and a forced teardown that fails. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc335726-838c-4538-a3c4-d6791aba5101 --- doc/usage.md | 2 +- nvx-rs/README.md | 6 +- nvx-rs/src/error.rs | 4 +- nvx-rs/src/host/cli/sandbox/lifecycle.rs | 5 +- nvx-rs/src/host/instance.rs | 25 +- nvx-rs/src/host/platform/mod.rs | 14 + nvx-rs/src/host/store/lease.rs | 416 ++++++++++++ nvx-rs/src/host/store/mod.rs | 73 +- nvx-rs/src/host/store/record.rs | 13 +- .../{template/files.rs => store/regular.rs} | 10 +- nvx-rs/src/host/store/tests.rs | 66 +- nvx-rs/src/host/supervisor/mod.rs | 107 ++- nvx-rs/src/host/supervisor/tests.rs | 217 +++++- nvx-rs/src/host/template/key.rs | 15 +- nvx-rs/src/host/template/manifest.rs | 6 +- nvx-rs/src/host/template/mod.rs | 1 - nvx-rs/src/host/template/store.rs | 158 +---- nvx-rs/src/host/template/store/leases.rs | 284 ++------ .../src/host/template/store/leases/tests.rs | 635 +++++++++++++++++- nvx-rs/tests/store.rs | 32 +- nvx-rs/tests/supervisor.rs | 48 +- .../src/bin/runtime-adapter-host.rs | 14 +- 22 files changed, 1644 insertions(+), 507 deletions(-) create mode 100644 nvx-rs/src/host/store/lease.rs rename nvx-rs/src/host/{template/files.rs => store/regular.rs} (89%) diff --git a/doc/usage.md b/doc/usage.md index 2a4ee581a..2d075e3fa 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -109,7 +109,7 @@ each code for callers of the `nvx` library. | `foreign-files` | 1 | `deprovision` found files that NVX does not own in the state directory, which the message names, and kept them with the directory. | | `outcome-report-exists` | 1 | A file exists at the `--outcome-report` path. | | `scratch-claimed` | 1 | Another sandbox claims the scratch image. | -| `sandbox-leased` | 1 | The sandbox holds a lease of a template's generation already, which a release ends once its VM has exited. | +| `sandbox-leased` | 1 | The sandbox holds a lease of a template's generation, which allows no other lease, and one launch of its clone, until a release ends it once its VM has exited. | | `template-ineligible` | 1 | The restore window of a template's generation, which the time contract of its capture sets, excludes the host's time, so no clone may restore it now. | | `template-tombstoned` | 1 | A tombstone quarantines the template's generation, which takes no new clones. | | `exited` | 1 | OpenVMM exited, as when a start fails, whose message names the category and status of OpenVMM's outcome report when it has one. | diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 2300007b0..c0129d510 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -95,8 +95,8 @@ nvx-rs/ | `nvx::host::instance` | `Instance`: a VM in a sandbox store of its own, which `nvx::host::store`, `nvx::host::supervisor`, and `nvx::host::session` drive, and in which the caller keeps the files that it declares as `CallerFiles`. Under the store's lifecycle lock of an existing sandbox, `LockedInstance::find` reattaches to the VM, ends a start that was interrupted, and reports a VM of another runtime identity as `Mismatched` rather than driving it; `LockedInstance::start` launches OpenVMM through the supervisor with a fresh capability, under the caller's `RuntimeIdentity`, waits until the guest is ready and has mounted the mapped paths, and commits the launch; `LockedInstance::stop` stops the VM with a forced fallback; `LockedInstance::terminate` stops a `Mismatched` VM by identity-verified forced termination, without a control session or its outcome report; and `LockedInstance::remove` removes a stopped instance, with the caller's files and its directory, through the store. `Instance::exec` runs workloads without the lock and pumps their output. `write_json` and `read_json` keep the configuration records that callers store beside an instance | | `nvx::host::nvxs` | `HostConnection`: the host's role of the NVXS session over a platform transport, on which `nvx::host::session` and other host applications run their protocols. It attaches with the launch capability, waiting while the broker asks it to, sends application bytes in as many DATA records as they take, and returns each DATA payload from the guest, with deadlines; a receive that times out keeps a partial record | | `nvx::host::process` | Verified process references. A `ProcessRef` names one process by its ID, its start time, and the `BootId` of the host boot during which it started. `Process::open` checks a reference and holds its process through a pidfd on Linux or a process handle on Windows, whether or not the caller started it, so that waits sleep until the process exits and termination never reaches a process that reuses its ID. A reference from an earlier boot names an exited process, and a process whose identity this user cannot check, such as another user's, makes `Process::open` fail rather than report an exit. Linux hosts without pidfds poll the reference instead, and cannot terminate the process | -| `nvx::host::supervisor` | The supervisor of a sandbox's VMM, beneath the CLI, `Instance`, and external orchestrators, which keep the configuration, the authorization, and the application protocol. `launch` runs the store's launch steps under the sandbox's lock: it writes the journal's marker, spawns OpenVMM with the claimed log, records its `ProcessRef` right after, and returns a `Starting` launch that the caller commits once the VMM serves its control endpoint, or abandons. A launch that fails never leaves an unrecorded VMM running. `VmmCommand::inherit` lists the descriptors (Linux) or handles (Windows) that the VMM inherits under their numbers, besides its standard streams, and it inherits nothing else of the caller's: on Linux, they must be close-on-exec, a flag that only the VMM's own process clears; on Windows, they join the VMM's explicit handle list. `Vmm::stop` runs one graceful step of the caller's and waits for the VMM's exit event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. `Vmm::terminate` takes that forced step alone, within the caller's bound, for a VM whose guest the caller must not ask anything, such as one of another runtime identity. `reattach` recovers an interrupted launch, and holds the recorded VMM again after the orchestrator restarts within the same host boot: `Live`, `Mismatched` with another runtime identity, `Exited`, or `Indeterminate` | -| `nvx::host::template` | The metadata of templates, captures of a microVM from which independent clones are restored. `TemplateManifest::read` reads the `manifest.bin` that OpenVMM wrote for a capture, at most `MAX_MANIFEST_LEN` bytes of a regular file that is not a link or other reparse point, and refuses another snapshot format version, an unknown, mistyped, or repeated field, truncated bytes, and a capture that is not a template's: a platform-tier capture for clones, with a read-only `distro` whose identity is unbound, a fresh scratch, and the dormant filesystem slot. `CaptureClass` holds the invariant inputs of a capture, known before it: the runtime, the guest's build, the backend, the requested CPU profile, the processors and RAM, the geometries of the disks, the network, and the adapter's other `Invariants`. `TemplateKey::new` checks that the capture made the machine that its class requested, and derives the template's compatibility key, from metadata alone: the runtime and contract versions through which its clones are restored, and the backend, CPU profile and effective CPUID, topology, rates, resources, kernel command line, disks, and devices of its machine. A capture's own metadata, such as its time anchor and host, stays out of the key. A key and a class each have one canonical, versioned encoding, whose domain-separated SHA-256 digest is its identity, a `KeyIdentity`, which converts to and from the `nvx::activation::Identity` by which a clone's activation record names its template's key, or a `ClassIdentity`; the [conformance vectors](tests/vectors/README.md) of version 1, `tests/vectors/template-key-v1.txt`, pin the key's encoding and identity. `Store` keeps the generations of templates that a host published, apart from every sandbox's store, at a location that the adapter chooses: a private directory with an identity of its own, a `StoreId`, the immutable payload of each generation, its key's canonical encoding beside the snapshot that OpenVMM published, and the versioned `Index` of them, which lists each generation with the identities of its key and capture class and its own metadata, such as its time contract. `Store::create` creates a store, or opens the one that its directory holds, crash-safely and once between concurrent creations, and refuses a directory with foreign entries; `Store::open` opens one. Its records are JSON in one format, replaced atomically, and records of another format fail rather than being migrated. Its directories must be private to the current user, and it refuses links and, on Windows, other reparse points among its directories and files. Four namespaces stay apart: the `nvx::activation::StoreGeneration`, the host's publication ID, which the index assigns in the order of publication and never twice, and by which a clone's activation record names its template; the bound-block `StorageGeneration` of a capture, which belongs to the storage owner; the time lineage counter of the time contract; and the VM generation ID, which the store never records. `TimeContract::eligibility` decides, apart from compatibility, whether a clone may restore a capture at a host's time, a `HostTime`, which `HostTime::now` samples as OpenVMM does: within the boot of the capture, by the host's monotonic clock, and otherwise by UTC, the downtime must not be negative, and the interval by which a restore advances the clone's VM time at most `MAX_DOWNTIME_NS`, 30 days. `Store::select` selects the latest eligible generation of a key that no tombstone quarantines. `Store::acquire` leases a generation to a clone in one transaction, under the store's lock and then the sandbox's: it checks the generation's eligibility, writes the lease's record in the store and the clone binding, `clone.json`, beside the sandbox's launch journal, and returns the lease with the sandbox's lock, under which the caller launches the clone. A lease never expires: `Store::release` ends it once the sandbox records no VM and no launch. `Store::tombstone` quarantines a generation, which takes no new lease while its leases keep working, and `Store::retire` deletes it once its leases have drained, crash-safely, removing the records of leases that no clone binding names any more | +| `nvx::host::supervisor` | The supervisor of a sandbox's VMM, beneath the CLI, `Instance`, and external orchestrators, which keep the configuration, the authorization, and the application protocol. `launch` runs the store's launch steps under the sandbox's lock, as a launch whose `LaunchId` the caller reserved beforehand, a fresh one for each launch, or that a clone's lease reserves: it writes the journal's marker, which names the launch, spawns OpenVMM with the claimed log, records its `ProcessRef` right after, and returns a `Starting` launch that the caller commits once the VMM serves its control endpoint, or abandons. A launch that fails never leaves an unrecorded VMM running. `VmmCommand::inherit` lists the descriptors (Linux) or handles (Windows) that the VMM inherits under their numbers, besides its standard streams, and it inherits nothing else of the caller's: on Linux, they must be close-on-exec, a flag that only the VMM's own process clears; on Windows, they join the VMM's explicit handle list. `Vmm::stop` runs one graceful step of the caller's and waits for the VMM's exit event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. `Vmm::terminate` takes that forced step alone, within the caller's bound, for a VM whose guest the caller must not ask anything, such as one of another runtime identity. `reattach` recovers an interrupted launch, and holds the recorded VMM again after the orchestrator restarts within the same host boot: `Live`, `Mismatched` with another runtime identity, `Exited`, or `Indeterminate`. A clone's launch takes its locks in one order, the template store's, then the sandbox's, which guards its clone binding and launch journal, then the scratch image's claim, and holds none for the VM's lifetime; its lease lasts through crashes and reattaches, until a release ends it once a stop, a termination, or a reattach has verified the VM's exit | +| `nvx::host::template` | The metadata of templates, captures of a microVM from which independent clones are restored. `TemplateManifest::read` reads the `manifest.bin` that OpenVMM wrote for a capture, at most `MAX_MANIFEST_LEN` bytes of a regular file that is not a link or other reparse point, and refuses another snapshot format version, an unknown, mistyped, or repeated field, truncated bytes, and a capture that is not a template's: a platform-tier capture for clones, with a read-only `distro` whose identity is unbound, a fresh scratch, and the dormant filesystem slot. `CaptureClass` holds the invariant inputs of a capture, known before it: the runtime, the guest's build, the backend, the requested CPU profile, the processors and RAM, the geometries of the disks, the network, and the adapter's other `Invariants`. `TemplateKey::new` checks that the capture made the machine that its class requested, and derives the template's compatibility key, from metadata alone: the runtime and contract versions through which its clones are restored, and the backend, CPU profile and effective CPUID, topology, rates, resources, kernel command line, disks, and devices of its machine. A capture's own metadata, such as its time anchor and host, stays out of the key. A key and a class each have one canonical, versioned encoding, whose domain-separated SHA-256 digest is its identity, a `KeyIdentity`, which converts to and from the `nvx::activation::Identity` by which a clone's activation record names its template's key, or a `ClassIdentity`; the [conformance vectors](tests/vectors/README.md) of version 1, `tests/vectors/template-key-v1.txt`, pin the key's encoding and identity. `Store` keeps the generations of templates that a host published, apart from every sandbox's store, at a location that the adapter chooses: a private directory with an identity of its own, a `StoreId`, the immutable payload of each generation, its key's canonical encoding beside the snapshot that OpenVMM published, and the versioned `Index` of them, which lists each generation with the identities of its key and capture class and its own metadata, such as its time contract. `Store::create` creates a store, or opens the one that its directory holds, crash-safely and once between concurrent creations, and refuses a directory with foreign entries; `Store::open` opens one. Its records are JSON in one format, replaced atomically, and records of another format fail rather than being migrated. Its directories must be private to the current user, and it refuses links and, on Windows, other reparse points among its directories and files. Four namespaces stay apart: the `nvx::activation::StoreGeneration`, the host's publication ID, which the index assigns in the order of publication and never twice, and by which a clone's activation record names its template; the bound-block `StorageGeneration` of a capture, which belongs to the storage owner; the time lineage counter of the time contract; and the VM generation ID, which the store never records. `TimeContract::eligibility` decides, apart from compatibility, whether a clone may restore a capture at a host's time, a `HostTime`, which `HostTime::now` samples as OpenVMM does: within the boot of the capture, by the host's monotonic clock, and otherwise by UTC, the downtime must not be negative, and the interval by which a restore advances the clone's VM time at most `MAX_DOWNTIME_NS`, 30 days. `Store::select` selects the latest eligible generation of a key that no tombstone quarantines. `Store::acquire` leases a generation to a clone in one transaction, under the store's lock and then the sandbox's: it checks the generation's eligibility, writes the lease's record in the store and the clone binding, `clone.json`, beside the sandbox's launch journal, and returns the lease with the sandbox's lock, the only one under which the caller launches the clone, once, as the launch that the lease reserves, `Lease::launch`. A lease never expires: `Store::release` ends it once the sandbox records no VM and no launch. `Store::tombstone` quarantines a generation, which takes no new lease while its leases keep working, and `Store::retire` deletes it once its leases have drained, crash-safely, removing the records of leases that no clone binding names any more | | `nvx::host::seal` | `FileSeal`: what the host reports about an open file from its metadata alone, without reading it: the platform scheme, the device or volume serial number, the file ID, the kind, the length, the modification time, and on Linux the change time, with a versioned serde form. A seal detects accidental change, not tampering; see [Distro bindings](#distro-bindings) | | `nvx::host::session`, `platform`, `report` | The authenticated control session, which carries NVX's control protocol, `NVXC`, one frame per DATA record of a `HostConnection`; Linux and Windows primitives; and OpenVMM's outcome report | | `nvx::host::sandbox` | The launch profile of `nvx-rs sandbox`, with native input validation: `SandboxLaunch`, with the distro `SandboxLayer`, an EROFS layer or a GPT disk, the one layer that a sandbox takes, the ext4 scratch disk, the entrypoint, arguments, hostname, workload identity, and `SandboxSettings` resource limits (`Option` for memory and `Option` for processes), and the `SandboxPlan` that it makes of them. `plan`: `SandboxPlan`, the typed plan of a cold layered launch, which is persistable in a versioned serde form: one read-only `distro` block device, an EROFS filesystem or a GPT disk with the identity that the guest expects of it, if any, an upper, the ext4 scratch disk or RAM, the launch settings, the `ExportPlan` of its live shares, and optionally a `SandboxNetwork`, the guest network and its typed `NetworkPolicy`, whose proxy's URL as the guest sees it the plan reports. The plan of an adapter that boots its own guest, `SandboxPlan::adapter`, holds no launch settings and no export plan. A plan knows only the `distro` and `scratch` `BlockRole`s, and gives, for every combination, its `--microvm-sandbox-block` arguments, with `--microvm-workload-identity` when it has launch settings, with no scratch block device for a RAM upper, and those of its network, which `nvx::host::network` lowers, and its kernel command line, with the boot tokens of `nvx::protocol::root`, within the 1024-byte budget that OpenVMM leaves it, refusing one whose root tokens the guest's parser refuses. Both deliver the live shares as the managed lifecycle does: the arguments export them at `nvx::host::filesystem::GUEST_EXPORT`, the command line announces the number of their mappings with `nvx_maps=`, and `mapping_table` gives the ordered table that `start` sends once the guest agent is ready; the CLI's one-shot `run` still renders them as boot tokens. An adapter's plan renders the caller's tokens before those of its root, and refuses tokens that its root or OpenVMM's shares own. `mount`: `SandboxMount` requests of live shares of host directories and files, their guest targets and the reserved ones, their access policies and owner, and the checks of their host paths; `plan_mounts`, which plans them through `nvx::host::filesystem` for the container root, and `boot_shares`, which renders a plan as the OpenVMM arguments and boot tokens of one share or of the aggregate of several, as the one-shot lifecycle delivers them. `options`: the checks that `nvx-rs sandbox` and `nvx-rs run` apply to their options: the attribution of `--mount-deny`, `--mount-allow`, and `--mount-write` to shares, the systemd entrypoint check and the explicit verification of the distro layer, the targets of `nvx-rs run` shares, and the managed exec environment; see [Sandbox input contract](#sandbox-input-contract). `distro`: the `DistroBinding` of a distro layer, an EROFS layer or a GPT disk, which explicit verification makes or a staging caller imports, and its check from metadata alone, which returns the layer open; see [Distro bindings](#distro-bindings). `scratch`: the `ScratchBinding` of a scratch image, its identity, kind, and length, and its check from metadata alone, which claims the image for one sandbox at a time and returns it open. `prepared`: `PreparedLaunch`, the launch of a plan with the checked files of its block devices, which renders the plan's OpenVMM arguments and kernel command line, naming the checked files as `/proc/self/fd/N` on Linux and by their final paths on Windows, and lists the descriptors that the VMM inherits through `VmmCommand::inherit`; see [Checked handoff](#checked-handoff) | @@ -1009,7 +1009,7 @@ usage errors, which no library error has. | `endpoint-closed` | `Protocol` | OpenVMM closed the control endpoint, as it does when the VM shuts down | | `template-ineligible` | `Unavailable` | The restore window of a template's generation, which the time contract of its capture sets, excludes the host's time, so no clone may restore it now: its capture is too old, or the host's clock is behind the capture's | | `template-tombstoned` | `Unavailable` | A tombstone quarantines the template's generation, which takes no new clones, while the clones that lease it keep their leases | -| `sandbox-leased` | `Other` | The sandbox holds a lease of a template's generation already, which a release ends once its VM has exited | +| `sandbox-leased` | `Other` | The sandbox holds a lease of a template's generation, which allows no other lease, and one launch of its clone, until a release ends it once its VM has exited | ## Bundled artifacts diff --git a/nvx-rs/src/error.rs b/nvx-rs/src/error.rs index 01d52b41e..a8db44e7b 100644 --- a/nvx-rs/src/error.rs +++ b/nvx-rs/src/error.rs @@ -154,8 +154,8 @@ pub enum ErrorCode { /// `template-tombstoned`: a tombstone quarantines the template's generation, which takes no /// new clones, while the clones that lease it keep their leases. TemplateTombstoned, - /// `sandbox-leased`: the sandbox holds a lease of a template's generation already, which a - /// release ends once its VM has exited. + /// `sandbox-leased`: the sandbox holds a lease of a template's generation, which allows no + /// other lease, and one launch of its clone, until a release ends it once its VM has exited. SandboxLeased, } diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle.rs b/nvx-rs/src/host/cli/sandbox/lifecycle.rs index 0c98e73dc..f94b736d9 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle.rs @@ -75,7 +75,7 @@ use crate::host::sandbox::distro::DISTRO_POLICY_REVISION; use crate::host::sandbox::scratch::CLAIM_RELEASE_WAIT; use crate::host::sandbox::{PreparedLaunch, SandboxLaunch, require_file}; use crate::host::store::{ - IdentityPart, LockedStore, OUTCOME_NAME, RUNTIME_NAME, Recovery, SandboxStore, + IdentityPart, LaunchId, LockedStore, OUTCOME_NAME, RUNTIME_NAME, Recovery, SandboxStore, }; use crate::host::supervisor::{self, Launched, Reattached, StopOutcome, Vmm, VmmCommand}; use crate::protocol::{CAPABILITY_LEN, GuestFeatures}; @@ -581,7 +581,8 @@ fn start_locked( ]; let directory = std::env::current_dir().map_err(Exit::from_os_error)?; let mut command = Vec::new(); - let mut starting = supervisor::launch(locked, &runtime, &capability, |endpoint| { + let launch = LaunchId::generate().map_err(error_with_cause)?; + let mut starting = supervisor::launch(locked, &launch, &runtime, &capability, |endpoint| { command = head; command.push(format!("listen={endpoint}")); command.extend(tail); diff --git a/nvx-rs/src/host/instance.rs b/nvx-rs/src/host/instance.rs index 4b99a44d6..5a6ed6792 100644 --- a/nvx-rs/src/host/instance.rs +++ b/nvx-rs/src/host/instance.rs @@ -68,8 +68,8 @@ use super::process::KILL_TIMEOUT; use super::report::{self, Report}; use super::session::{self, ControlSession, ExecEvent, SessionError}; use super::store::{ - CallerFiles, IdentityPart, LockedStore, OUTCOME_NAME, Recovery, RemovalError, RuntimeIdentity, - RuntimeRecord, SOCKET_NAME, SandboxStore, + CallerFiles, IdentityPart, LaunchId, LockedStore, OUTCOME_NAME, Recovery, RemovalError, + RuntimeIdentity, RuntimeRecord, SOCKET_NAME, SandboxStore, }; use super::supervisor::{self, Reattached, Starting, StopOutcome, Vmm, VmmCommand}; use crate::error::{Error, ErrorCode, ErrorKind, Result}; @@ -706,14 +706,21 @@ impl LockedInstance { let endpoint = control_endpoint(&instance.store.socket_path())?; spec.fit_process_command_line(&spec.openvmm_arguments(&endpoint, &report)?, 0)?; let capability = generate_capability()?; + let launch = LaunchId::generate()?; let started = Instant::now(); - let starting = supervisor::launch(&mut self.locked, runtime, &capability, |endpoint| { - let arguments = spec.openvmm_arguments(endpoint, &report)?; - spec.fit_process_command_line(&arguments, 0)?; - let mut command = VmmCommand::new(&spec.openvmm, arguments, instance.dir()); - command.breakaway_from_job = spec.breakaway_from_job; - Ok(command) - })?; + let starting = supervisor::launch( + &mut self.locked, + &launch, + runtime, + &capability, + |endpoint| { + let arguments = spec.openvmm_arguments(endpoint, &report)?; + spec.fit_process_command_line(&arguments, 0)?; + let mut command = VmmCommand::new(&spec.openvmm, arguments, instance.dir()); + command.breakaway_from_job = spec.breakaway_from_job; + Ok(command) + }, + )?; let (pid, start_time) = { let reference = starting.process().reference(); (reference.pid, reference.start_time) diff --git a/nvx-rs/src/host/platform/mod.rs b/nvx-rs/src/host/platform/mod.rs index c9d91826a..3243f86ca 100644 --- a/nvx-rs/src/host/platform/mod.rs +++ b/nvx-rs/src/host/platform/mod.rs @@ -100,6 +100,20 @@ pub(crate) fn parse_hex_id(digits: &str) -> Option<[u8; 16]> { Some(id) } +/// Parses 64 hexadecimal digits, in either case, into the 32 bytes that they spell. +pub(crate) fn parse_hex32(text: &str) -> Option<[u8; 32]> { + let digits = text.as_bytes(); + if digits.len() != 64 { + return None; + } + let mut bytes = [0; 32]; + for (byte, pair) in bytes.iter_mut().zip(digits.chunks_exact(2)) { + let nibble = |digit: u8| char::from(digit).to_digit(16); + *byte = u8::try_from(nibble(pair[0])? * 16 + nibble(pair[1])?).ok()?; + } + Some(bytes) +} + /// Parses the identity of a host as OpenVMM reads it: 32 hexadecimal digits, in either case, /// between optional braces and surrounding whitespace, ignoring dashes, as in `/etc/machine-id` /// and the Windows machine GUID. diff --git a/nvx-rs/src/host/store/lease.rs b/nvx-rs/src/host/store/lease.rs new file mode 100644 index 000000000..1ce7ac867 --- /dev/null +++ b/nvx-rs/src/host/store/lease.rs @@ -0,0 +1,416 @@ +//! The lease of a template's generation that the clone of a sandbox holds, as the sandbox's +//! [clone binding](CLONE_BINDING_NAME) records it, and the reading of the records of the template +//! store, in whose format the binding is. +//! +//! The sandbox's store reads the binding before it begins a launch. The template store, +//! [`template::Store`](crate::host::template::Store), grants and ends leases, and reads its own +//! records, with these, and [`template`](crate::host::template) re-exports what is public here. + +use std::fmt; +use std::io; +use std::num::NonZeroU64; +use std::path::{Path, PathBuf}; +use std::str::FromStr; + +use serde::de::DeserializeOwned; +use serde::{Deserialize, Serialize}; + +use super::regular::{open_regular, read_at_most, was_removed}; +use super::{CLONE_BINDING_NAME, LaunchId, SandboxStore}; +use crate::activation::{Identity, StoreGeneration, TemplateReference}; +use crate::error::{Error, ErrorCode, ErrorKind, Result}; +use crate::host::platform::{self, parse_hex_id, parse_hex32}; + +/// Version of the store's records: its identity record and its index. +pub const STORE_FORMAT: u32 = 1; + +/// Bytes of a [`StoreId`] and of a [`StorageGeneration`](crate::host::template::StorageGeneration). +pub(crate) const ID_LEN: usize = 16; + +/// Longest record of a lease, of a clone binding, or of a tombstone, in bytes: more than a record +/// that names the longest path of a sandbox takes. +pub(crate) const MAX_LEASE_RECORD_LEN: u64 = 256 * 1024; + +/// The binding of a clone in its sandbox, in messages. +pub(crate) const BINDING_DESCRIPTION: &str = "clone binding"; + +/// Returns `bytes` as lowercase hexadecimal digits. +pub(crate) fn hex(bytes: &[u8]) -> String { + bytes.iter().map(|byte| format!("{byte:02x}")).collect() +} + +/// The identity of a store: 16 random bytes that its creation draws. +/// +/// Its text form, which `Display` writes and `FromStr` parses, is 32 lowercase hexadecimal +/// digits; parsing also accepts uppercase ones. +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +pub struct StoreId([u8; ID_LEN]); + +impl StoreId { + /// Returns the identity whose bytes are `bytes`. + pub const fn from_bytes(bytes: [u8; ID_LEN]) -> Self { + Self(bytes) + } + + /// Returns the bytes of the identity. + pub const fn as_bytes(&self) -> &[u8; ID_LEN] { + &self.0 + } +} + +impl fmt::Display for StoreId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&hex(&self.0)) + } +} + +impl fmt::Debug for StoreId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "StoreId({self})") + } +} + +impl FromStr for StoreId { + type Err = Error; + + /// Parses the text form of a store's identity, failing with + /// [`ErrorKind::Unsupported`] for anything else. + fn from_str(text: &str) -> Result { + parse_hex_id(text).map(Self).ok_or_else(|| { + Error::unsupported(format!( + "{text:?} is not a template store's identity, which is 32 hexadecimal digits" + )) + }) + } +} + +/// The identity of a lease: 16 random bytes that its acquisition draws. +/// +/// Its text form, which `Display` writes and `FromStr` parses, is 32 lowercase hexadecimal +/// digits; parsing also accepts uppercase ones. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct LeaseId([u8; ID_LEN]); + +impl LeaseId { + /// Returns the identity whose bytes are `bytes`. + pub const fn from_bytes(bytes: [u8; ID_LEN]) -> Self { + Self(bytes) + } + + /// Returns the bytes of the identity. + pub const fn as_bytes(&self) -> &[u8; ID_LEN] { + &self.0 + } +} + +impl fmt::Display for LeaseId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&hex(&self.0)) + } +} + +impl fmt::Debug for LeaseId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "LeaseId({self})") + } +} + +impl FromStr for LeaseId { + type Err = Error; + + /// Parses the text form of a lease's identity, failing with [`ErrorKind::Unsupported`] for + /// anything else. + fn from_str(text: &str) -> Result { + parse_hex_id(text).map(Self).ok_or_else(|| { + Error::unsupported(format!( + "{text:?} is not the identity of a lease, which is 32 hexadecimal digits" + )) + }) + } +} + +/// A lease of a generation of a template, which keeps the generation in its store for the clone +/// of one sandbox, whose VM restores it, and reserves the clone's launch. +/// +/// The store keeps the record of each lease that it granted, and the sandbox keeps the same +/// record as its clone binding, [`CLONE_BINDING_NAME`], beside its launch journal. A lease names +/// its sandbox by the path of its directory with its links resolved, so that no removal or change +/// of a link, or of a directory that a `..` component passes through, hides the sandbox from a +/// [retirement](crate::host::template::Store::retire). A sandbox that holds a lease must not move, +/// and goes only through its store's [removal](super::LockedStore::remove), which makes the +/// binding's absence durable before the sandbox goes. The lease allows one launch of the clone, as +/// the identity that it reserves, [`launch`](Self::launch), under the lock that acquired it, and no +/// other launch of the sandbox until a release ends it, as +/// [`LockedStore::begin_launch`](super::LockedStore::begin_launch) requires. +#[derive(Clone, PartialEq, Eq)] +pub struct Lease { + /// The lease's identity. + id: LeaseId, + /// The identity of the store that granted it. + store: StoreId, + /// The store generation of the leased generation. + generation: StoreGeneration, + /// The identity of the leased generation's key. + key: Identity, + /// The directory of the sandbox that holds the lease, as an absolute path with its links + /// resolved, which is valid Unicode. + sandbox: PathBuf, + /// The launch of the clone that the lease reserves. + launch: LaunchId, +} + +impl Lease { + /// Returns the lease `id`, which the store `store` granted, of its generation `generation`, + /// whose key's identity is `key`, to the sandbox in `sandbox`, an absolute path with its links + /// resolved, which is valid Unicode, for the launch `launch` of its clone. + pub(crate) fn new( + id: LeaseId, + store: StoreId, + generation: StoreGeneration, + key: Identity, + sandbox: PathBuf, + launch: LaunchId, + ) -> Self { + Self { + id, + store, + generation, + key, + sandbox, + launch, + } + } + + /// Reads the clone binding of the sandbox of `sandbox`: the lease that it holds, if it holds + /// one. + /// + /// A read that holds no lock sees one whole binding, as the latest transition left it. Fails + /// with [`ErrorKind::Io`] for a binding that cannot be read, or is malformed, and with + /// [`ErrorCode::FormatUnsupported`] for one of another format. + pub fn of(sandbox: &SandboxStore) -> Result> { + read_binding(sandbox.dir()) + } + + /// Returns the lease's identity. + pub fn id(&self) -> LeaseId { + self.id + } + + /// Returns the identity of the store that granted the lease. + pub fn store(&self) -> StoreId { + self.store + } + + /// Returns the store generation of the leased generation. + pub fn generation(&self) -> StoreGeneration { + self.generation + } + + /// Returns the directory of the sandbox that holds the lease, as an absolute path with its + /// links resolved, which may differ from the path through which the sandbox was leased, and + /// through which a [release](crate::host::template::Store::release) takes the sandbox's lock. + pub fn sandbox(&self) -> &Path { + &self.sandbox + } + + /// Returns the launch of the clone that the lease reserves: the identity under which the + /// caller launches the clone, with [`supervisor::launch`](crate::host::supervisor::launch), + /// once, under the lock that acquired the lease, and the only launch of the sandbox that + /// [`LockedStore::begin_launch`](super::LockedStore::begin_launch) begins while the sandbox + /// holds the lease. + pub fn launch(&self) -> &LaunchId { + &self.launch + } + + /// Returns the reference by which the activation record of the clone names the template that + /// it restores: the leased generation's store generation, and its key's identity. + pub fn template(&self) -> TemplateReference { + TemplateReference { + generation: self.generation, + key: self.key, + } + } +} + +impl fmt::Debug for Lease { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("Lease") + .field("id", &self.id) + .field("store", &self.store) + .field("generation", &self.generation) + // The key writes itself as the `KeyIdentity` that `Lease::key` returns. + .field("key", &format_args!("KeyIdentity({})", self.key)) + .field("sandbox", &self.sandbox) + .field("launch", &self.launch) + .finish() + } +} + +/// Reads the clone binding of the sandbox in `dir`, or returns `None` if it holds none, or if the +/// directory does not exist. +pub(crate) fn read_binding(dir: &Path) -> Result> { + read_lease(&dir.join(CLONE_BINDING_NAME), BINDING_DESCRIPTION) +} + +/// Reads the record of a lease at `path`, its `description`, or returns `None` if it does not +/// exist. +pub(crate) fn read_lease(path: &Path, description: &str) -> Result> { + let Some(record) = read_record::(path, description, MAX_LEASE_RECORD_LEN)? else { + return Ok(None); + }; + record.into_lease().map(Some).map_err(|problem| { + Error::io(format!( + "the {description} {} is malformed: {problem}", + path.display() + )) + }) +} + +/// The record of a lease, as JSON: in its store, and as the clone binding of its sandbox. +#[derive(Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub(crate) struct LeaseRecord { + /// [`STORE_FORMAT`]. + format: u32, + /// The identity of the store that granted the lease. + store: String, + /// The identity of the lease. + lease: String, + /// The store generation of the leased generation. + generation: u64, + /// The identity of the leased generation's key. + key: String, + /// The directory of the sandbox that holds the lease, as an absolute path. + sandbox: String, + /// The launch of the clone that the lease reserves. + launch: String, +} + +impl LeaseRecord { + /// Returns the record of `lease`. + pub(crate) fn new(lease: &Lease) -> Self { + Self { + format: STORE_FORMAT, + store: lease.store.to_string(), + lease: lease.id.to_string(), + generation: lease.generation.get().get(), + key: lease.key.to_string(), + sandbox: lease.sandbox.to_string_lossy().into_owned(), + launch: lease.launch.to_string(), + } + } + + /// Checks the record, and returns its lease, or says why it is malformed. + fn into_lease(self) -> std::result::Result { + let field = + |what: &str, error: Error| format!("its {what} is invalid: {}", error.message()); + let sandbox = PathBuf::from(self.sandbox); + if !sandbox.is_absolute() { + return Err(format!( + "its sandbox, {}, is not an absolute path", + sandbox.display() + )); + } + Ok(Lease { + id: self + .lease + .parse() + .map_err(|error| field("identity", error))?, + store: self.store.parse().map_err(|error| field("store", error))?, + generation: NonZeroU64::new(self.generation) + .map(StoreGeneration::new) + .ok_or("it leases store generation 0, which no generation has")?, + key: parse_key(&self.key).map_err(|error| field("key", error))?, + sandbox, + launch: LaunchId::try_from(self.launch) + .map_err(|problem| format!("its launch is invalid: {problem}"))?, + }) + } +} + +/// Parses the text form of the identity of a template's key, 64 hexadecimal digits, as +/// [`KeyIdentity`](crate::host::template::KeyIdentity) parses it, failing with +/// [`ErrorKind::Unsupported`] for anything else. +fn parse_key(text: &str) -> Result { + parse_hex32(text).map(Identity::from_bytes).ok_or_else(|| { + Error::unsupported(format!( + "{text:?} is not a template key's identity, which is 64 hexadecimal digits" + )) + }) +} + +/// Builds an error mapper that keeps an I/O source under `context`, as an error of +/// [`ErrorKind::NotFound`] if the source says that a path does not exist, and of +/// [`ErrorKind::Io`] otherwise. +pub(crate) fn store_error(context: String) -> impl FnOnce(io::Error) -> Error { + move |error| { + let kind = match error.kind() { + io::ErrorKind::NotFound => ErrorKind::NotFound, + _ => ErrorKind::Io, + }; + Error::new(kind, context).with_source(error) + } +} + +/// Reads the private regular file at `path`, the store's `description`, of at most `limit` bytes, +/// or returns `None` if it does not exist, or was removed as this opened it, as [`was_removed`] +/// tells, which a transaction in another process may do. +/// +/// The file is opened without following a link, or on Windows any other reparse point, at its +/// own name, and refused unless it is a regular file, before anything is read from it. A file +/// that is not private to the current user is refused, since others could have written it: on +/// Windows, a file's own access control list decides who may open it, whatever the store's +/// directory allows. +pub(crate) fn read_private(path: &Path, description: &str, limit: u64) -> Result>> { + let failed = || store_error(format!("cannot read the {description} {}", path.display())); + let file = match open_regular(path) { + Ok(file) => file, + Err(error) if error.kind() == io::ErrorKind::NotFound || was_removed(path, &error) => { + return Ok(None); + } + Err(error) => return Err(failed()(error)), + }; + if let Some(problem) = platform::private_file_problem(&file).map_err(failed())? { + return Err(Error::io(format!( + "the {description} {} {problem}", + path.display() + ))); + } + read_at_most(file, limit, &format!("the {description}")) + .map(Some) + .map_err(failed()) +} + +/// Reads the store's record at `path`, its `description`, of at most `limit` bytes, in format +/// [`STORE_FORMAT`], or returns `None` if it does not exist. +pub(crate) fn read_record( + path: &Path, + description: &str, + limit: u64, +) -> Result> { + let Some(text) = read_private(path, description, limit)? else { + return Ok(None); + }; + let malformed = || format!("the {description} {} is malformed", path.display()); + let value: serde_json::Value = + serde_json::from_slice(&text).map_err(|error| Error::io(malformed()).with_source(error))?; + match value.get("format") { + Some(format) if format.as_u64() == Some(STORE_FORMAT.into()) => {} + Some(format) => { + return Err(Error::new( + ErrorCode::FormatUnsupported, + format!( + "the {description} {} has the unsupported format {format}; this build reads \ + format {STORE_FORMAT} only and migrates no other, so create a new template \ + store, and capture its templates again", + path.display() + ), + )); + } + None => return Err(Error::io(format!("{}: it has no format", malformed()))), + } + serde_json::from_value(value) + .map(Some) + .map_err(|error| Error::io(malformed()).with_source(error)) +} diff --git a/nvx-rs/src/host/store/mod.rs b/nvx-rs/src/host/store/mod.rs index ea7a25930..114f987ae 100644 --- a/nvx-rs/src/host/store/mod.rs +++ b/nvx-rs/src/host/store/mod.rs @@ -49,10 +49,13 @@ //! //! # Launches //! -//! A launch holds the sandbox's lock throughout: +//! A launch holds the sandbox's lock throughout, under an identity that the caller reserved +//! before it began, with [`LaunchId::generate`], a fresh one for each launch, or that a clone's +//! lease reserved, which records it durably before any VMM of the clone exists, for the one +//! launch of the clone that the lease allows: //! //! 1. [`LockedStore::begin_launch`] stores the capability, creates a fresh log and claims it, and -//! writes the journal's marker, before anything is spawned. +//! writes the journal's marker, which names the launch, before anything is spawned. //! 2. The caller spawns the VMM with [`Launch::take_log`] as its standard output and error, and //! [`Launch::record_process`] records the VMM's [`ProcessRef`] right after the spawn. //! 3. Once the VM is ready, [`Launch::commit`] replaces the journal with the runtime record. A @@ -65,7 +68,7 @@ //! use std::time::{Duration, Instant}; //! //! use nvx::host::launch::LaunchSpec; -//! use nvx::host::store::{RuntimeIdentity, SandboxStore}; +//! use nvx::host::store::{LaunchId, RuntimeIdentity, SandboxStore}; //! use nvx::host::supervisor::{self, VmmCommand}; //! use nvx::protocol::CAPABILITY_LEN; //! @@ -77,7 +80,7 @@ //! let store = SandboxStore::create("/var/lib/example/sandbox")?; //! let mut locked = store.lock(Some(Instant::now() + Duration::from_secs(10)))?; //! locked.recover()?; -//! let mut launch = locked.begin_launch(runtime, capability)?; +//! let mut launch = locked.begin_launch(&LaunchId::generate()?, runtime, capability)?; //! let arguments = spec.openvmm_arguments(launch.endpoint(), &store.outcome_path())?; //! let command = VmmCommand::new(&spec.openvmm, arguments, store.dir()); //! let log = launch.take_log()?; @@ -94,7 +97,7 @@ //! transition of the sandbox overlaps it, not even through the same guard: //! //! ```compile_fail -//! # use nvx::host::store::{RuntimeIdentity, SandboxStore}; +//! # use nvx::host::store::{LaunchId, RuntimeIdentity, SandboxStore}; //! # use nvx::protocol::CAPABILITY_LEN; //! # fn overlap( //! # store: &SandboxStore, @@ -102,7 +105,7 @@ //! # capability: &[u8; CAPABILITY_LEN], //! # ) -> nvx::Result<()> { //! let mut locked = store.lock(None)?; -//! let launch = locked.begin_launch(runtime, capability)?; +//! let launch = locked.begin_launch(&LaunchId::generate()?, runtime, capability)?; //! locked.recover()?; // `locked` is still borrowed by `launch`. //! launch.commit()?; //! # Ok(()) @@ -219,8 +222,10 @@ mod caller; pub(crate) mod files; mod identity; +pub(crate) mod lease; mod lifecycle; mod record; +pub(crate) mod regular; #[cfg(test)] mod tests; mod vmm; @@ -238,6 +243,7 @@ use self::files::{io_error, remove_if_present}; pub use self::identity::{ ArtifactIdentity, IdentityCheck, IdentityPart, MAX_IDENTITY_LEN, RuntimeIdentity, }; +use self::lease::Lease; pub use self::lifecycle::{Lingering, RemovalError}; pub use self::record::{ LaunchId, LaunchJournal, LaunchState, LogClaim, MAX_ENDPOINT_LEN, RuntimeRecord, @@ -623,6 +629,9 @@ pub struct LockedStore { files: CallerFiles, /// Whether this holder removed its lock file, after which the path may name another's. lock_removed: bool, + /// The launch of the sandbox's clone that the lease acquired under this lock reserves, until + /// [`begin_launch`](Self::begin_launch) begins it, or a release ends the lease. + reserved: Option, } impl Deref for LockedStore { @@ -642,29 +651,44 @@ impl LockedStore { lock, files, lock_removed: false, + reserved: None, } } + /// Lets [`begin_launch`](Self::begin_launch) begin `launch`, once, as the launch of the + /// sandbox's clone that the lease acquired under this lock reserves, or, given `None`, no + /// launch of a clone, once a release ends the lease. + pub(crate) fn reserve(&mut self, launch: Option) { + self.reserved = launch; + } + /// Returns the caller's files that the lock knows: those that a [lifecycle /// lock](self#lifecycle-locks) was taken with, and none for [`SandboxStore::lock`]. pub fn files(&self) -> &CallerFiles { &self.files } - /// Begins a launch of the runtime `runtime`, whose VMM receives `capability`. + /// Begins the launch `launch`, which the caller reserved with [`LaunchId::generate`], of the + /// runtime `runtime`, whose VMM receives `capability`. /// /// It stores the capability, replaces the log of the previous launch with a fresh one, which /// it claims, removes the outcome report, console log, and control socket of the previous /// launch, and any outcome report that its VMM staged without publishing it, chooses a control - /// endpoint, and writes the journal's marker. It keeps the console log of a caller that keeps - /// the console of every launch, as [`CallerFiles::keeping_console`] declares. Nothing is - /// spawned yet: the caller spawns the VMM with [`Launch::take_log`]. + /// endpoint, and writes the journal's marker, which names the launch. It keeps the console log + /// of a caller that keeps the console of every launch, as [`CallerFiles::keeping_console`] + /// declares. Nothing is spawned yet: the caller spawns the VMM with [`Launch::take_log`]. /// /// Fails if the sandbox's VM runs, or a launch was interrupted and is not recovered yet, and - /// with [`ErrorKind::Unsupported`] for an all-zero capability - /// or an invalid runtime identity. + /// with [`ErrorKind::Unsupported`] for an all-zero capability or an invalid runtime identity. + /// A sandbox whose clone holds a lease of a template's generation, which its + /// [clone binding](CLONE_BINDING_NAME) records, launches once, as the launch that the lease + /// reserves, under the lock that acquired the lease, as + /// [`template::Store::acquire`](crate::host::template::Store::acquire) returns it: any other + /// launch, and any launch under another lock or after that one began, fails with + /// [`ErrorCode::SandboxLeased`] until a release ends the lease. pub fn begin_launch( &mut self, + launch: &LaunchId, runtime: &RuntimeIdentity, capability: &[u8; CAPABILITY_LEN], ) -> Result> { @@ -686,6 +710,26 @@ impl LockedStore { self.dir.display() ))); } + if let Some(lease) = Lease::of(self)? + && (lease.launch() != launch || self.reserved.as_ref() != Some(launch)) + { + let reservation = if lease.launch() == launch { + format!("whose launch {launch} begins once, under the lock that acquired the lease") + } else { + format!("which reserves the launch {}, not {launch}", lease.launch()) + }; + return Err(Error::new( + ErrorCode::SandboxLeased, + format!( + "the sandbox in {} holds the lease {} of the generation {} of the template \ + store {}, {reservation}", + self.dir.display(), + lease.id(), + lease.generation(), + lease.store() + ), + )); + } let socket = self.socket_path(); for path in [&socket, &self.outcome_path()] { remove_if_present(path)?; @@ -698,12 +742,13 @@ impl LockedStore { .map_err(|error| Error::io("cannot choose a control endpoint").with_source(error))?; let (log, claim) = self.create_log()?; files::write_private(&self.capability_path(), capability)?; - let journal = - LaunchJournal::prepared(LaunchId::generate()?, endpoint, runtime.clone(), claim); + let journal = LaunchJournal::prepared(launch.clone(), endpoint, runtime.clone(), claim); if let Err(error) = record::write(&self.path(JOURNAL_NAME), &journal) { let _ = remove_if_present(&self.capability_path()); return Err(error); } + // A lease reserves one launch, which has begun. + self.reserved = None; Ok(Launch { locked: self, journal, diff --git a/nvx-rs/src/host/store/record.rs b/nvx-rs/src/host/store/record.rs index e9465c18b..748c3e3b7 100644 --- a/nvx-rs/src/host/store/record.rs +++ b/nvx-rs/src/host/store/record.rs @@ -36,15 +36,20 @@ fn is_hex_id(text: &str) -> bool { .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) } -/// Identity of one launch: 32 lowercase hexadecimal digits, the 16 random bytes that the store -/// draws for every launch, and which its journal and runtime record share. +/// Identity of one launch: 32 lowercase hexadecimal digits, the 16 random bytes that +/// [`generate`](Self::generate) draws for every launch, and which its journal and runtime record +/// share. #[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] #[serde(try_from = "String", into = "String")] pub struct LaunchId(String); impl LaunchId { - /// Draws a fresh launch identity. - pub(super) fn generate() -> Result { + /// Draws a fresh launch identity, which a caller reserves for one launch before it begins it + /// with [`LockedStore::begin_launch`](super::LockedStore::begin_launch), or + /// [`supervisor::launch`](crate::host::supervisor::launch), and never uses for another. A + /// clone's lease reserves one, as + /// [`template::Store::acquire`](crate::host::template::Store::acquire) draws it. + pub fn generate() -> Result { let mut bytes = [0u8; LAUNCH_ID_BYTES]; getrandom::fill(&mut bytes) .map_err(|error| Error::other("cannot draw a launch identity").with_source(error))?; diff --git a/nvx-rs/src/host/template/files.rs b/nvx-rs/src/host/store/regular.rs similarity index 89% rename from nvx-rs/src/host/template/files.rs rename to nvx-rs/src/host/store/regular.rs index e8a229307..7a5404b00 100644 --- a/nvx-rs/src/host/template/files.rs +++ b/nvx-rs/src/host/store/regular.rs @@ -1,5 +1,5 @@ -//! The files that the template modules read: regular files that are not links, which they read -//! within a bound. +//! The files that the template modules read, and that a sandbox's store reads as its clone +//! binding: regular files that are not links, which they read within a bound. use std::fs::{File, OpenOptions}; use std::io::{self, Read}; @@ -20,7 +20,7 @@ fn not_regular() -> io::Error { /// pending, as [`platform::is_deleted`] tells, which a file whose access control list refuses the /// access is not. No open reaches a removed file again, so it is gone for its readers, and for any /// other removal of it. -pub(super) fn was_removed(path: &Path, error: &io::Error) -> bool { +pub(crate) fn was_removed(path: &Path, error: &io::Error) -> bool { cfg!(windows) && error.kind() == io::ErrorKind::PermissionDenied && platform::is_deleted(path) } @@ -29,7 +29,7 @@ pub(super) fn was_removed(path: &Path, error: &io::Error) -> bool { /// /// The open does not wait, as it would on Unix for a FIFO without a writer: a file that is not a /// regular file is refused before anything is read from it. -pub(super) fn open_regular(path: &Path) -> io::Result { +pub(crate) fn open_regular(path: &Path) -> io::Result { let mut options = OpenOptions::new(); options.read(true); #[cfg(unix)] @@ -70,7 +70,7 @@ pub(super) fn open_regular(path: &Path) -> io::Result { /// Reads `file` to its end, failing if it holds more than `limit` bytes, the most that `what` may /// hold, without reading more than one byte past the limit. -pub(super) fn read_at_most(file: File, limit: u64, what: &str) -> io::Result> { +pub(crate) fn read_at_most(file: File, limit: u64, what: &str) -> io::Result> { let mut bytes = Vec::new(); file.take(limit + 1).read_to_end(&mut bytes)?; if bytes.len() as u64 > limit { diff --git a/nvx-rs/src/host/store/tests.rs b/nvx-rs/src/host/store/tests.rs index 70b6e9954..44300b42d 100644 --- a/nvx-rs/src/host/store/tests.rs +++ b/nvx-rs/src/host/store/tests.rs @@ -511,7 +511,9 @@ fn the_store_is_private_to_the_current_user() { assert_eq!(SandboxStore::open(store.dir()).unwrap(), store); assert_eq!(SandboxStore::create(store.dir()).unwrap(), store); let mut locked = store.lock(None).unwrap(); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); drop(launch); for name in [LOCK_NAME, JOURNAL_NAME, CAPABILITY_NAME, LOG_NAME] { assert_eq!(mode(&store.path(name)), 0o600, "{name}"); @@ -597,7 +599,9 @@ fn the_store_is_private_to_the_current_user() { assert_eq!(SandboxStore::open(store.dir()).unwrap(), store); assert_eq!(SandboxStore::create(store.dir()).unwrap(), store); let mut locked = store.lock(None).unwrap(); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); drop(launch); let problem = |name: &str| { let file = fs::File::open(store.path(name)).unwrap(); @@ -641,7 +645,9 @@ fn a_caller_that_keeps_the_console_keeps_it_across_launches() { let (console, outcome) = (locked.console_path(), locked.outcome_path()); fs::write(&console, b"earlier launches\n").unwrap(); fs::write(&outcome, b"{}").unwrap(); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); // The launch leaves the console of the earlier launches for the caller to append to, and // removes the previous launch's outcome report, as for any caller. assert_eq!(fs::read(&console).unwrap(), b"earlier launches\n"); @@ -693,7 +699,9 @@ fn a_launch_writes_its_marker_before_its_vmm_runs() { // Held open, the previous log keeps its inode, which the fresh log cannot then reuse. let previous = fs::File::open(store.log_path()).unwrap(); let mut locked = store.lock(None).unwrap(); - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let journal = store.journal().unwrap().unwrap(); assert_eq!(&journal, launch.journal()); assert_eq!(journal.state(), LaunchState::Prepared); @@ -729,7 +737,9 @@ fn a_launch_writes_its_marker_before_its_vmm_runs() { // A second launch waits until the first is recovered. drop(launch); - let error = locked.begin_launch(&runtime(), &CAPABILITY).unwrap_err(); + let error = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap_err(); assert!( error.message().contains("must be recovered first"), "{error}" @@ -748,14 +758,18 @@ fn launches_check_their_inputs() { let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); let error = locked - .begin_launch(&runtime(), &[0; CAPABILITY_LEN]) + .begin_launch( + &LaunchId::generate().unwrap(), + &runtime(), + &[0; CAPABILITY_LEN], + ) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Unsupported); let mut invalid = runtime(); invalid.control_contract = String::new(); assert_eq!( locked - .begin_launch(&invalid, &CAPABILITY) + .begin_launch(&LaunchId::generate().unwrap(), &invalid, &CAPABILITY) .unwrap_err() .kind(), ErrorKind::Unsupported @@ -770,11 +784,15 @@ fn launches_check_their_inputs() { fn an_abandoned_launch_that_spawned_nothing_has_exited() { let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); // A launch commits only once it records its VMM. let error = launch.commit().unwrap_err(); assert!(error.message().contains("before it records"), "{error}"); - let error = locked.begin_launch(&runtime(), &CAPABILITY).unwrap_err(); + let error = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap_err(); assert!( error.message().contains("must be recovered first"), "{error}" @@ -782,7 +800,9 @@ fn an_abandoned_launch_that_spawned_nothing_has_exited() { // A process that a concurrent test spawned may still hold the log that the launch closed, // until it executes its program, and a later recovery then ends the launch. assert!(released(|| locked.recover().unwrap() == Recovery::Exited)); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); match launch.abandon().unwrap() { Recovery::Exited => {} Recovery::Indeterminate(_) => { @@ -798,7 +818,9 @@ fn an_abandoned_launch_that_spawned_nothing_has_exited() { fn a_recorded_vmm_that_cannot_be_held_has_exited_once_its_log_is_released() { let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let log = launch.take_log().unwrap(); let reference = ProcessRef { boot: BootId::current().unwrap(), @@ -835,7 +857,9 @@ fn a_recorded_vmm_that_cannot_be_held_has_exited_once_its_log_is_released() { fn a_journal_that_its_runtime_record_supersedes_is_removed() { let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let journal = launch.journal().spawned(reference(1)); drop(launch); // The launch wrote its runtime record and crashed before it removed its journal. @@ -858,12 +882,18 @@ fn a_journal_that_its_runtime_record_supersedes_is_removed() { assert!(error.message().contains("names launch"), "{error}"); // A runtime record keeps a launch from beginning until it is cleared. fs::remove_file(store.path(JOURNAL_NAME)).unwrap(); - let error = locked.begin_launch(&runtime(), &CAPABILITY).unwrap_err(); + let error = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap_err(); assert!(error.message().contains("already running"), "{error}"); locked.clear_runtime().unwrap(); assert!(store.runtime().unwrap().is_none()); assert!(!store.capability_path().exists()); - drop(locked.begin_launch(&runtime(), &CAPABILITY).unwrap()); + drop( + locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(), + ); } #[test] @@ -879,7 +909,9 @@ fn report_staging_files_go_once_nothing_of_their_launch_runs() { fs::write(&staging, b"{").unwrap(); let mut locked = store.lock(None).unwrap(); // A launch begins without the staging file of an earlier one. - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); assert!(!staging.exists()); // While a process of the launch may still hold its log, as its VMM does, the file stays. let log = launch.take_log().unwrap(); @@ -896,7 +928,9 @@ fn report_staging_files_go_once_nothing_of_their_launch_runs() { assert!(released(|| locked.recover().unwrap() == Recovery::Exited)); assert!(!staging.exists()); // An abandoned launch removes it, and so does clearing the runtime of a VM that exited. - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); fs::write(&staging, b"{").unwrap(); match launch.abandon().unwrap() { Recovery::Exited => {} diff --git a/nvx-rs/src/host/supervisor/mod.rs b/nvx-rs/src/host/supervisor/mod.rs index ce80c7b2a..b40c284a9 100644 --- a/nvx-rs/src/host/supervisor/mod.rs +++ b/nvx-rs/src/host/supervisor/mod.rs @@ -10,10 +10,13 @@ //! //! # Launches //! -//! [`launch`] runs the store's launch steps under the sandbox's lock: +//! [`launch`] runs the store's launch steps under the sandbox's lock, as a launch whose identity +//! the caller reserved beforehand, with [`LaunchId::generate`], or through a clone's lease, as +//! [Clones](#clones) describes: //! -//! 1. [`LockedStore::begin_launch`] writes the journal's marker, and the caller's `command` builds -//! the VMM's [`VmmCommand`] with the control endpoint that the launch chose. +//! 1. [`LockedStore::begin_launch`] writes the journal's marker, which names the launch, and the +//! caller's `command` builds the VMM's [`VmmCommand`] with the control endpoint that the +//! launch chose. //! 2. [`spawn`] starts the VMM with the launch's claimed log as its standard output and error, and //! [`Launch::record_process`] records its identity right after. //! 3. The caller checks, through its own transport, that the VMM serves its control endpoint, and @@ -27,7 +30,7 @@ //! use std::time::{Duration, Instant}; //! //! use nvx::host::launch::LaunchSpec; -//! use nvx::host::store::{RuntimeIdentity, SandboxStore}; +//! use nvx::host::store::{LaunchId, RuntimeIdentity, SandboxStore}; //! use nvx::host::supervisor::{self, Reattached, StopOutcome, Vmm, VmmCommand}; //! use nvx::protocol::CAPABILITY_LEN; //! @@ -51,7 +54,8 @@ //! } //! Reattached::Indeterminate(reason) => return Err(nvx::Error::other(reason)), //! } -//! let starting = supervisor::launch(&mut locked, runtime, capability, |endpoint| { +//! let launch = LaunchId::generate()?; +//! let starting = supervisor::launch(&mut locked, &launch, runtime, capability, |endpoint| { //! let arguments = spec.openvmm_arguments(endpoint, &store.outcome_path())?; //! Ok(VmmCommand::new(&spec.openvmm, arguments, store.dir())) //! })?; @@ -95,6 +99,60 @@ //! that it has not released, a [`Process`] that matched its recorded reference, or, during the //! recovery of a launch that did not record its VMM, the process that recovery verified to be that //! VMM. It never signals a process that reuses a recorded ID. +//! +//! # Clones +//! +//! A clone restores a generation of a template that it leases from a +//! [template store](crate::host::template::Store), and its lease reserves its launch: +//! +//! 1. [`Store::acquire`](crate::host::template::Store::acquire) leases the generation, and writes +//! the lease's record in the store and the clone binding in the sandbox, durably, with the +//! identity of the launch that the lease reserves, +//! [`Lease::launch`](crate::host::template::Lease::launch), before any VMM of the clone +//! exists. It returns the sandbox's lock. +//! 2. Under that lock, the caller claims the clone's scratch image, if it has one, as +//! [`ScratchBinding::check`](crate::host::sandbox::scratch::ScratchBinding::check) does, and +//! runs [`launch`] as the reserved launch, which [`LockedStore::begin_launch`] requires of a +//! sandbox that holds a lease, and begins once, under that lock alone, whatever its outcome. +//! 3. Once the clone's VM has exited, as a stop, a termination, or a reattach verifies, +//! [`Store::release`](crate::host::template::Store::release) ends the lease, under the +//! sandbox's lock, taken through the lease's path, +//! [`Lease::sandbox`](crate::host::template::Lease::sandbox), as the acquisition took it. +//! +//! ## The lock order +//! +//! These steps, and every other transition of a template store or a sandbox, take their locks in +//! one order: the template store's lock, then the sandbox's +//! [lifecycle lock](super::store#lifecycle-locks), which guards its clone binding and its launch +//! journal, then the scratch image's claim. None waits for a lock while it holds a later one: an +//! acquisition releases the store's lock before it returns; a release, a launch, a stop, a +//! reattach, and a removal of the sandbox take no lock of the store; and a retirement takes no +//! lock of a sandbox. No lock lasts for a VM's lifetime: the caller releases the sandbox's lock +//! once the launch commits or ends, and the scratch image's claim belongs to the VMM, which holds +//! it until it exits. +//! +//! ## Crashes +//! +//! A crash at any step leaves records from which a later transition, under the same locks, decides +//! what becomes of the lease, which a release ends only once the sandbox records no VM and no +//! launch: +//! +//! - After the acquisition, the binding reserves a launch that never began, and that no later lock +//! begins: a release ends the lease at once, and another acquisition, which checks the +//! generation's eligibility again, reserves another launch. +//! - After the launch's journal, before or after the spawn, [`reattach`] ends the launch, as +//! [`LockedStore::recover`] does, and the lease lasts until it has: a launch that cannot be +//! ended yet is [`Reattached::Indeterminate`], and keeps it. +//! - After the commit, the runtime record names the VM, which [`reattach`] holds again: the lease +//! lasts as long as the VM runs, until a stop or a termination verifies its exit. +//! - During a release, the binding goes first, durably, and leaves the record of the lease, which +//! a retirement removes. +//! +//! The evidence of the VM's exit decides each outcome. A record of an earlier boot, or of a process +//! ID that another process now has, names a VM that has exited: reattach clears it, and the lease +//! may be released. A VM that cannot be examined is indeterminate, and keeps its records, and so +//! the lease. A forced teardown that fails, or that does not end the VM, keeps its runtime record, +//! and the lease. #[cfg(test)] mod tests; @@ -107,14 +165,16 @@ pub use super::platform::inherit::Inherited; use super::process::{Process, ProcessRef}; pub use super::spawn::{Launched, VmmCommand, spawn}; use super::store::{ - IdentityCheck, IdentityPart, Launch, LaunchJournal, LockedStore, Recovery, RuntimeIdentity, - RuntimeRecord, SandboxStore, + IdentityCheck, IdentityPart, Launch, LaunchId, LaunchJournal, LockedStore, Recovery, + RuntimeIdentity, RuntimeRecord, SandboxStore, }; use crate::error::{Error, Result}; use crate::protocol::CAPABILITY_LEN; -/// Launches a VMM in the sandbox of `locked`, through the store's launch journal, under the -/// runtime identity `runtime`, with `capability` on its standard input. +/// Launches a VMM in the sandbox of `locked`, through the store's launch journal, as the launch +/// `reserved`, which the caller reserved with [`LaunchId::generate`], or which the sandbox's lease +/// reserves, [`Lease::launch`](crate::host::template::Lease::launch), under the runtime identity +/// `runtime`, with `capability` on its standard input. /// /// It begins the launch with [`LockedStore::begin_launch`], asks `command` for the VMM's command, /// given the control endpoint that the launch chose, [`spawn`]s the VMM with the launch's claimed @@ -126,15 +186,18 @@ use crate::protocol::CAPABILITY_LEN; /// identity it could not record through its handle, and returns the step's error. A launch that /// cannot be ended then, because its VMM did not exit, keeps its journal for /// [`LockedStore::recover`], and the error says so. Fails as [`LockedStore::begin_launch`] does if -/// the sandbox's VM runs, or an interrupted launch has not been recovered. +/// the sandbox's VM runs, an interrupted launch has not been recovered, or the sandbox holds a +/// lease that does not let this lock begin `reserved`: one that reserves another launch, or that +/// another lock acquired, or whose launch has begun. pub fn launch<'s, 'h>( locked: &'s mut LockedStore, + reserved: &LaunchId, runtime: &RuntimeIdentity, capability: &[u8; CAPABILITY_LEN], command: impl FnOnce(&str) -> Result>, ) -> Result> { let store = SandboxStore::clone(locked); - let mut launch = locked.begin_launch(runtime, capability)?; + let mut launch = locked.begin_launch(reserved, runtime, capability)?; let command = match command(launch.endpoint()) { Ok(command) => command, Err(error) => return Err(abandon_after(launch, error)), @@ -380,8 +443,8 @@ impl Vmm { /// /// The result says whether the VMM exited without the forced step, the forced step ended it, /// or it still runs; see [`StopOutcome`]. Once the VMM has exited, the stop clears the runtime - /// record of its launch, as [`LockedStore::clear_runtime`] does, unless the record names - /// another launch by then. A VMM that still runs keeps its record. + /// record of its launch, as [`LockedStore::clear_runtime`] does, unless the record is another + /// VM's by then. A VMM that still runs keeps its record. /// /// `locked` must be the locked store of the VM's sandbox. Fails with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if `forced` is zero or longer @@ -433,9 +496,20 @@ impl Vmm { /// determined, or when it cannot be terminated, as on Linux hosts without pidfds, which cannot /// hold it. pub fn terminate(&self, locked: &mut LockedStore, bound: Duration) -> Result { + self.terminate_with(locked, bound, &self.process) + } + + /// Ends the VM as [`terminate`](Self::terminate) does, through `process`: its VMM, or, in + /// tests, a stand-in for it. + fn terminate_with( + &self, + locked: &mut LockedStore, + bound: Duration, + process: &impl Stoppable, + ) -> Result { self.check_store(locked)?; check_bound(bound, "the termination")?; - let outcome = terminate_process(&self.process, self.record.process().pid, bound)?; + let outcome = terminate_process(process, self.record.process().pid, bound)?; self.clear_if_exited(locked, outcome)?; Ok(outcome) } @@ -453,12 +527,13 @@ impl Vmm { } /// Clears the runtime record of the VM's launch from `locked` once its VMM has exited, which - /// `outcome` tells, unless the record names another launch by then. + /// `outcome` tells, unless the record is another VM's by then: another launch's, or that of + /// another VMM under a launch identity that its caller used again. fn clear_if_exited(&self, locked: &mut LockedStore, outcome: StopOutcome) -> Result<()> { if outcome != StopOutcome::TimedOut && locked .runtime()? - .is_some_and(|record| record.launch() == self.record.launch()) + .is_some_and(|record| record == self.record) { locked.clear_runtime()?; } diff --git a/nvx-rs/src/host/supervisor/tests.rs b/nvx-rs/src/host/supervisor/tests.rs index 7d9ca6218..4a71a08a1 100644 --- a/nvx-rs/src/host/supervisor/tests.rs +++ b/nvx-rs/src/host/supervisor/tests.rs @@ -7,14 +7,19 @@ use std::fs; use serde_json::json; use super::*; -use crate::ErrorKind; use crate::host::launch::MAX_TIMEOUT; use crate::host::platform; use crate::host::process::BootId; -use crate::host::store::{ArtifactIdentity, FORMAT, JOURNAL_NAME, RUNTIME_NAME}; +use crate::host::store::{ArtifactIdentity, CallerFiles, FORMAT, JOURNAL_NAME, RUNTIME_NAME}; +use crate::host::template::{self, Acquired, HostTime, Lease}; +use crate::{ErrorCode, ErrorKind}; /// A launch capability. const CAPABILITY: [u8; CAPABILITY_LEN] = [7; CAPABILITY_LEN]; +/// The launch of the runtime records that [`record`] writes. +const RECORDED_LAUNCH: &str = "5d41402abc4b2a76b9719d911017c592"; +/// The configuration record of these tests' clones. +const CONFIG: &str = "config.json"; /// Returns the identity `sha256:` and 64 copies of `digit`. fn artifact(digit: char) -> ArtifactIdentity { @@ -47,9 +52,20 @@ fn this_process() -> ProcessRef { /// Writes the runtime record of a VM whose VMM is `process`, launched with `runtime`, as another /// orchestrator would have committed it. fn record(store: &SandboxStore, process: &ProcessRef, runtime: &RuntimeIdentity) { + record_launch(store, RECORDED_LAUNCH, process, runtime); +} + +/// Writes the runtime record of the launch `launch` of a VM whose VMM is `process`, launched with +/// `runtime`, as another orchestrator would have committed it. +fn record_launch( + store: &SandboxStore, + launch: &str, + process: &ProcessRef, + runtime: &RuntimeIdentity, +) { let record = json!({ "format": FORMAT, - "launch": "5d41402abc4b2a76b9719d911017c592", + "launch": launch, "process": process, "endpoint": "/var/lib/sandboxes/example/control.sock", "runtime": runtime, @@ -61,6 +77,52 @@ fn record(store: &SandboxStore, process: &ProcessRef, runtime: &RuntimeIdentity) .unwrap(); } +/// A clone, in a fresh temporary directory, that leases a generation of a template from the +/// template store that it returns, and whose launch, the one that its lease reserved, committed a +/// VM whose VMM is `process`; and the lease, with the lock of the clone's sandbox that the +/// acquisition returned. +fn committed_clone( + process: &ProcessRef, +) -> (tempfile::TempDir, template::Store, Lease, LockedStore) { + let directory = tempfile::tempdir().unwrap(); + let templates = template::Store::create(directory.path().join("templates")).unwrap(); + let generation = templates.publish_for_tests(); + let sandbox = directory.path().join("sandbox"); + let files = CallerFiles::new(CONFIG).unwrap(); + let provisioned = SandboxStore::lock_for_provision(&sandbox, &files, None).unwrap(); + fs::write(sandbox.join(CONFIG), b"{}").unwrap(); + drop(provisioned); + // A second after the capture, on the host that captured the generation. + let time = generation.time(); + let now = HostTime { + host_id: time.host_id, + boot_id: time.boot_id, + clock: time.host_clock, + monotonic_ns: time.capture_monotonic_ns + 1_000_000_000, + utc_ns: time.capture_utc_ns + 1_000_000_000, + }; + let Acquired { lease, mut locked } = templates + .acquire(generation.generation(), &sandbox, &files, &now, None) + .unwrap(); + // The reserved launch begins, and its commit replaces its journal with its runtime record. + drop( + locked + .begin_launch(lease.launch(), &runtime(), &CAPABILITY) + .unwrap(), + ); + record_launch(&locked, lease.launch().as_str(), process, &runtime()); + fs::remove_file(sandbox.join(JOURNAL_NAME)).unwrap(); + (directory, templates, lease, locked) +} + +/// Checks that `templates` refuses to release the lease of the sandbox of `locked`, whose VM may +/// still run, and that the sandbox keeps it. +fn refuses_release(templates: &template::Store, locked: &mut LockedStore, lease: &Lease) { + let error = templates.release(locked).unwrap_err(); + assert_eq!(error.code(), ErrorCode::SandboxRunning, "{error}"); + assert_eq!(Lease::of(locked).unwrap().as_ref(), Some(lease)); +} + /// A VMM whose state drives a stop's decisions without a process: it exits once the graceful /// step runs, if it heeds that step, or once it is killed, if killing ends it. #[derive(Default)] @@ -473,10 +535,16 @@ fn stops_check_their_store_and_forced_bound_before_anything_else() { fn a_launch_whose_command_fails_ends_before_anything_runs() { let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let error = launch(&mut locked, &runtime(), &CAPABILITY, |endpoint| { - assert!(!endpoint.is_empty()); - Err(Error::unsupported("no command")) - }) + let error = launch( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |endpoint| { + assert!(!endpoint.is_empty()); + Err(Error::unsupported("no command")) + }, + ) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Unsupported); assert_eq!(error.message(), "no command"); @@ -492,9 +560,13 @@ fn a_launch_whose_vmm_cannot_start_ends() { let (directory, store) = store(); let mut locked = store.lock(None).unwrap(); let missing = directory.path().join("missing-openvmm"); - let error = launch(&mut locked, &runtime(), &CAPABILITY, |_| { - Ok(VmmCommand::new(&missing, Vec::new(), directory.path())) - }) + let error = launch( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |_| Ok(VmmCommand::new(&missing, Vec::new(), directory.path())), + ) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Io); assert_eq!( @@ -503,8 +575,127 @@ fn a_launch_whose_vmm_cannot_start_ends() { ); assert!(store.journal().unwrap().is_none()); // The next launch begins at once. - let next = launch(&mut locked, &runtime(), &CAPABILITY, |_| { - Err(Error::other("the next launch began")) - }); + let next = launch( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |_| Err(Error::other("the next launch began")), + ); assert_eq!(next.unwrap_err().message(), "the next launch began"); } + +#[test] +fn a_clone_whose_record_names_a_vmm_that_exited_may_release_its_lease() { + let live = this_process(); + let mut earlier_boot = *live.boot.as_bytes(); + earlier_boot[0] ^= 1; + for exited in [ + // A record of an earlier boot names a VMM that has exited. + ProcessRef { + boot: BootId::from_bytes(earlier_boot), + ..live.clone() + }, + // The recorded ID now names another process: this one. + ProcessRef { + start_time: live.start_time + 1, + ..live.clone() + }, + ] { + let (_directory, templates, lease, mut locked) = committed_clone(&exited); + // While the record names the clone's VM, the lease stays. + refuses_release(&templates, &mut locked, &lease); + let Reattached::Exited { record, recovery } = reattach(&mut locked, &runtime()).unwrap() + else { + panic!("{exited:?} names a live VMM"); + }; + assert_eq!(record.unwrap().launch(), lease.launch()); + assert_eq!(recovery, Recovery::Clean); + // Reattach cleared the record of the VM that exited, so the release ends the lease. + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); + assert_eq!(Lease::of(&locked).unwrap(), None); + } +} + +#[test] +fn a_clone_whose_vmm_cannot_be_examined_keeps_its_lease() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + let hidden = |_: &ProcessRef| -> io::Result> { + Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) + }; + let Reattached::Indeterminate(reason) = reattach_with(&mut locked, &runtime(), hidden).unwrap() + else { + panic!("a VMM that cannot be examined was classified"); + }; + assert!(reason.ends_with("cannot be examined: hidden"), "{reason}"); + assert!(locked.runtime().unwrap().is_some()); + refuses_release(&templates, &mut locked, &lease); +} + +#[test] +fn a_clone_whose_forced_teardown_fails_keeps_its_lease_until_one_ends_its_vmm() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + let Reattached::Live(vmm) = reattach(&mut locked, &runtime()).unwrap() else { + panic!("the clone's VMM was not reattached"); + }; + // This test process stands for the VMM, which these teardowns reach only through stand-ins. + let bound = Duration::from_secs(10); + // A teardown that cannot terminate the VMM fails, and keeps its record, and so the lease. + let unkillable = Scripted { + unkillable: true, + ..Scripted::default() + }; + let error = vmm + .terminate_with(&mut locked, bound, &unkillable) + .unwrap_err(); + assert_eq!( + error.message(), + format!("cannot terminate VMM process {}", std::process::id()) + ); + refuses_release(&templates, &mut locked, &lease); + // One that does not end the VMM within its bound keeps them too. + let survivor = Scripted::default(); + assert_eq!( + vmm.terminate_with(&mut locked, bound, &survivor).unwrap(), + StopOutcome::TimedOut + ); + refuses_release(&templates, &mut locked, &lease); + // One that ends it clears the record, and the release ends the lease. + let dying = Scripted { + dies_when_killed: true, + ..Scripted::default() + }; + assert_eq!( + vmm.terminate_with(&mut locked, bound, &dying).unwrap(), + StopOutcome::Forced + ); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + +#[test] +fn a_stop_keeps_the_record_of_another_vmm_under_the_same_launch_identity() { + let (_directory, store) = store(); + let process = this_process(); + record(&store, &process, &runtime()); + let mut locked = store.lock(None).unwrap(); + let Reattached::Live(vmm) = reattach(&mut locked, &runtime()).unwrap() else { + panic!("the recorded VMM was not reattached"); + }; + // Another VMM's record replaced it, under the launch identity that its caller used again. + let other = ProcessRef { + start_time: process.start_time + 1, + ..process.clone() + }; + record(&store, &other, &runtime()); + let exited = Scripted::default(); + exited.exited.set(true); + assert_eq!( + vmm.terminate_with(&mut locked, Duration::from_secs(10), &exited) + .unwrap(), + StopOutcome::Graceful + ); + let kept = store.runtime().unwrap().unwrap(); + assert_eq!(kept.launch().as_str(), RECORDED_LAUNCH); + assert_eq!(kept.process(), &other); +} diff --git a/nvx-rs/src/host/template/key.rs b/nvx-rs/src/host/template/key.rs index fcef996f5..a28c0d68a 100644 --- a/nvx-rs/src/host/template/key.rs +++ b/nvx-rs/src/host/template/key.rs @@ -18,6 +18,7 @@ use super::manifest::{ use crate::activation; use crate::contract; use crate::error::{Error, Result}; +use crate::host::platform::parse_hex32; use crate::host::sandbox::SandboxNetwork; use crate::host::sandbox::mount::MOUNT_TAG; use crate::host::store::{ArtifactIdentity, MAX_IDENTITY_LEN, RuntimeIdentity}; @@ -151,20 +152,6 @@ fn write_hex(formatter: &mut fmt::Formatter<'_>, bytes: &[u8]) -> fmt::Result { .try_for_each(|byte| write!(formatter, "{byte:02x}")) } -/// Parses 64 hexadecimal digits, in either case, into the 32 bytes that they spell. -fn parse_hex32(text: &str) -> Option<[u8; 32]> { - let digits = text.as_bytes(); - if digits.len() != 64 { - return None; - } - let mut bytes = [0; 32]; - for (byte, pair) in bytes.iter_mut().zip(digits.chunks_exact(2)) { - let nibble = |digit: u8| char::from(digit).to_digit(16); - *byte = u8::try_from(nibble(pair[0])? * 16 + nibble(pair[1])?).ok()?; - } - Some(bytes) -} - /// Declares a 32-byte identity: the domain-separated SHA-256 digest of a canonical encoding. macro_rules! identity { ($(#[$attribute:meta])* $name:ident, $what:literal) => { diff --git a/nvx-rs/src/host/template/manifest.rs b/nvx-rs/src/host/template/manifest.rs index 28c1c99fa..cd84c5b54 100644 --- a/nvx-rs/src/host/template/manifest.rs +++ b/nvx-rs/src/host/template/manifest.rs @@ -14,11 +14,11 @@ use std::time::{Duration, SystemTime}; use sha2::{Digest, Sha256}; -use super::files; use super::wire::{Field, Message, WireError}; use crate::contract; use crate::error::{Error, ErrorCode, Result}; use crate::host::process::BootId; +use crate::host::store::regular; use crate::hypervisor::Hypervisor; /// Name of the manifest in a snapshot's directory. @@ -706,8 +706,8 @@ impl TemplateManifest { pub fn read(snapshot: &Path) -> Result { let path = snapshot.join(MANIFEST_NAME); let description = format!("the snapshot manifest {}", path.display()); - let bytes = files::open_regular(&path) - .and_then(|file| files::read_at_most(file, MAX_MANIFEST_LEN, "a manifest")) + let bytes = regular::open_regular(&path) + .and_then(|file| regular::read_at_most(file, MAX_MANIFEST_LEN, "a manifest")) .map_err(|error| { let code = match error.kind() { io::ErrorKind::NotFound => ErrorCode::NotFound, diff --git a/nvx-rs/src/host/template/mod.rs b/nvx-rs/src/host/template/mod.rs index c8f6661b3..f249b05fd 100644 --- a/nvx-rs/src/host/template/mod.rs +++ b/nvx-rs/src/host/template/mod.rs @@ -49,7 +49,6 @@ mod canonical; mod eligibility; -mod files; mod key; mod manifest; mod store; diff --git a/nvx-rs/src/host/template/store.rs b/nvx-rs/src/host/template/store.rs index 6996afb28..4401f2196 100644 --- a/nvx-rs/src/host/template/store.rs +++ b/nvx-rs/src/host/template/store.rs @@ -14,10 +14,8 @@ use std::str::FromStr; use std::thread; use std::time::{Duration, Instant, SystemTime}; -use serde::de::DeserializeOwned; use serde::{Deserialize, Serialize}; -use super::files::{open_regular, read_at_most, was_removed}; use super::key::{ClassIdentity, KeyIdentity, TemplateKey}; use super::manifest::{HostClock, TimeContract}; use crate::activation::StoreGeneration; @@ -25,11 +23,11 @@ use crate::error::{Error, ErrorCode, ErrorKind, Result}; use crate::host::platform::{self, parse_hex_id}; use crate::host::process::BootId; use crate::host::store::files::{open_lock, write_private}; +use crate::host::store::lease::{ID_LEN, hex, read_private, read_record, store_error}; -pub use self::leases::{Acquired, LEASES_NAME, Lease, LeaseId, Retirement, TOMBSTONES_NAME}; +pub use self::leases::{Acquired, LEASES_NAME, Retirement, TOMBSTONES_NAME}; +pub use crate::host::store::lease::{Lease, LeaseId, STORE_FORMAT, StoreId}; -/// Version of the store's records: its identity record and its index. -pub const STORE_FORMAT: u32 = 1; /// Name of the store's identity record. pub const STORE_RECORD_NAME: &str = "store.json"; /// Name of the store's index. @@ -58,14 +56,6 @@ const MAX_INDEX_LEN: u64 = 4 * 1024 * 1024; /// until a deadline. const LOCK_RETRY_INTERVAL: Duration = Duration::from_millis(25); -/// Bytes of a [`StoreId`] and of a [`StorageGeneration`]. -const ID_LEN: usize = 16; - -/// Returns `bytes` as lowercase hexadecimal digits. -fn hex(bytes: &[u8]) -> String { - bytes.iter().map(|byte| format!("{byte:02x}")).collect() -} - /// Draws 16 random bytes, for `what`. fn random_id(what: &str) -> Result<[u8; ID_LEN]> { let mut bytes = [0; ID_LEN]; @@ -74,51 +64,6 @@ fn random_id(what: &str) -> Result<[u8; ID_LEN]> { Ok(bytes) } -/// The identity of a store: 16 random bytes that its creation draws. -/// -/// Its text form, which `Display` writes and `FromStr` parses, is 32 lowercase hexadecimal -/// digits; parsing also accepts uppercase ones. -#[derive(Clone, Copy, PartialEq, Eq, Hash)] -pub struct StoreId([u8; ID_LEN]); - -impl StoreId { - /// Returns the identity whose bytes are `bytes`. - pub const fn from_bytes(bytes: [u8; ID_LEN]) -> Self { - Self(bytes) - } - - /// Returns the bytes of the identity. - pub const fn as_bytes(&self) -> &[u8; ID_LEN] { - &self.0 - } -} - -impl fmt::Display for StoreId { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - formatter.write_str(&hex(&self.0)) - } -} - -impl fmt::Debug for StoreId { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(formatter, "StoreId({self})") - } -} - -impl FromStr for StoreId { - type Err = Error; - - /// Parses the text form of a store's identity, failing with - /// [`ErrorKind::Unsupported`] for anything else. - fn from_str(text: &str) -> Result { - parse_hex_id(text).map(Self).ok_or_else(|| { - Error::unsupported(format!( - "{text:?} is not a template store's identity, which is 32 hexadecimal digits" - )) - }) - } -} - /// The bound-block storage generation of a capture: 16 bytes, not all zero, that a capture gives /// OpenVMM for every block that it binds, which the storage owner keeps immutable for those /// blocks, so that neither the capture nor a restore reads a block to identify it. @@ -360,7 +305,12 @@ impl Index { /// [`CLONE_BINDING_NAME`](crate::host::store::CLONE_BINDING_NAME), beside the sandbox's launch /// journal, before any VMM of the clone is spawned. It returns the lease with the sandbox's lock, /// under which the caller launches the clone, and holds no lock of the store for the VM's -/// lifetime. A lease never expires: it lasts until [`Store::release`] ends it, which it does only +/// lifetime. The lease reserves one launch of the clone, [`Lease::launch`], which only that lock +/// begins, with +/// [`LockedStore::begin_launch`](crate::host::store::LockedStore::begin_launch), and the sandbox +/// launches nothing else while the lease lasts; [the supervisor](crate::host::supervisor#clones) +/// describes the launch of a clone, its crashes, and the order of the locks that it takes. A lease +/// never expires: it lasts until [`Store::release`] ends it, which it does only /// once the sandbox records no VM and no launch, and so once its VM has exited, as its store /// verifies through the VM's recorded process, also after a reattach. A caller that reads a /// generation's payload, as a restore does, holds a lease of it. @@ -425,6 +375,19 @@ pub struct Store { id: StoreId, } +#[cfg(test)] +impl Store { + /// Publishes a generation of a template, whose fixed time contract names a host and boot of + /// its own, for the tests of other modules, and returns it. + pub(crate) fn publish_for_tests(&self) -> Generation { + tests::publish( + self, + &super::test_support::ManifestSpec::template(), + tests::storage(1), + ) + } +} + impl Store { /// Creates the directory `dir` if needed, whose parent must exist, private to the current /// user, and a store in it, or opens the store that it holds. @@ -480,7 +443,7 @@ impl Store { )?; let store = Self { dir, - id: StoreId(random_id("a template store's identity")?), + id: StoreId::from_bytes(random_id("a template store's identity")?), }; store.write_index(&Index { next: FIRST_GENERATION, @@ -661,19 +624,6 @@ fn absolute(dir: PathBuf) -> Result { }) } -/// Builds an error mapper that keeps an I/O source under `context`, as an error of -/// [`ErrorKind::NotFound`] if the source says that a path does not exist, and of -/// [`ErrorKind::Io`] otherwise. -fn store_error(context: String) -> impl FnOnce(io::Error) -> Error { - move |error| { - let kind = match error.kind() { - io::ErrorKind::NotFound => ErrorKind::NotFound, - _ => ErrorKind::Io, - }; - Error::new(kind, context).with_source(error) - } -} - /// Fails unless `problem`, why the directory `path`, which `what` names, is not private to the /// current user, is `None`. fn refuse_problem(path: &Path, what: &str, problem: Option) -> Result<()> { @@ -813,68 +763,6 @@ fn take_lock(dir: &Path, deadline: Option) -> Result { } } -/// Reads the private regular file at `path`, the store's `description`, of at most `limit` bytes, -/// or returns `None` if it does not exist, or was removed as this opened it, as [`was_removed`] -/// tells, which a transaction in another process may do. -/// -/// The file is opened without following a link, or on Windows any other reparse point, at its -/// own name, and refused unless it is a regular file, before anything is read from it. A file -/// that is not private to the current user is refused, since others could have written it: on -/// Windows, a file's own access control list decides who may open it, whatever the store's -/// directory allows. -fn read_private(path: &Path, description: &str, limit: u64) -> Result>> { - let failed = || store_error(format!("cannot read the {description} {}", path.display())); - let file = match open_regular(path) { - Ok(file) => file, - Err(error) if error.kind() == io::ErrorKind::NotFound || was_removed(path, &error) => { - return Ok(None); - } - Err(error) => return Err(failed()(error)), - }; - if let Some(problem) = platform::private_file_problem(&file).map_err(failed())? { - return Err(Error::io(format!( - "the {description} {} {problem}", - path.display() - ))); - } - read_at_most(file, limit, &format!("the {description}")) - .map(Some) - .map_err(failed()) -} - -/// Reads the store's record at `path`, its `description`, of at most `limit` bytes, in format -/// [`STORE_FORMAT`], or returns `None` if it does not exist. -fn read_record( - path: &Path, - description: &str, - limit: u64, -) -> Result> { - let Some(text) = read_private(path, description, limit)? else { - return Ok(None); - }; - let malformed = || format!("the {description} {} is malformed", path.display()); - let value: serde_json::Value = - serde_json::from_slice(&text).map_err(|error| Error::io(malformed()).with_source(error))?; - match value.get("format") { - Some(format) if format.as_u64() == Some(STORE_FORMAT.into()) => {} - Some(format) => { - return Err(Error::new( - ErrorCode::FormatUnsupported, - format!( - "the {description} {} has the unsupported format {format}; this build reads \ - format {STORE_FORMAT} only and migrates no other, so create a new template \ - store, and capture its templates again", - path.display() - ), - )); - } - None => return Err(Error::io(format!("{}: it has no format", malformed()))), - } - serde_json::from_value(value) - .map(Some) - .map_err(|error| Error::io(malformed()).with_source(error)) -} - /// Atomically replaces the store's record at `path`, its `description`, with `record`, as pretty /// JSON and a newline. fn write_record(path: &Path, record: &T, description: &str) -> Result<()> { diff --git a/nvx-rs/src/host/template/store/leases.rs b/nvx-rs/src/host/template/store/leases.rs index f928b6305..64059689d 100644 --- a/nvx-rs/src/host/template/store/leases.rs +++ b/nvx-rs/src/host/template/store/leases.rs @@ -5,29 +5,30 @@ #[cfg(test)] mod tests; -use std::fmt; use std::fs; use std::io; use std::num::NonZeroU64; use std::path::{Path, PathBuf}; -use std::str::FromStr; use std::sync::{Mutex, MutexGuard, PoisonError}; use std::time::Instant; use serde::{Deserialize, Serialize}; use super::{ - GENERATIONS_NAME, Generation, ID_LEN, STORE_FORMAT, Store, StoreId, entries, hex, random_id, - read_record, refuse_problem, store_error, take_lock, write_record, + GENERATIONS_NAME, Generation, STORE_FORMAT, Store, StoreId, entries, random_id, read_record, + refuse_problem, store_error, take_lock, write_record, }; -use crate::activation::{Identity, StoreGeneration, TemplateReference}; +use crate::activation::{Identity, StoreGeneration}; use crate::error::{Error, ErrorCode, ErrorKind, Result}; use crate::host::filesystem; -use crate::host::platform::{self, parse_hex_id}; +use crate::host::platform; use crate::host::store::files::{remove_if_present, sync_dir}; -use crate::host::store::{CLONE_BINDING_NAME, CallerFiles, LockedStore, SandboxStore}; +use crate::host::store::lease::{ + Lease, LeaseId, LeaseRecord, MAX_LEASE_RECORD_LEN, read_binding, read_lease, +}; +use crate::host::store::regular::was_removed; +use crate::host::store::{CLONE_BINDING_NAME, CallerFiles, LaunchId, LockedStore, SandboxStore}; use crate::host::template::eligibility::HostTime; -use crate::host::template::files::was_removed; use crate::host::template::key::KeyIdentity; /// Name of the directory of the records of the leases that a store granted. @@ -57,136 +58,22 @@ fn durable_record_dirs() -> MutexGuard<'static, Vec> { .unwrap_or_else(PoisonError::into_inner) } -/// Longest record of a lease, of a clone binding, or of a tombstone, in bytes: more than a record -/// that names the longest path of a sandbox takes. -const MAX_LEASE_RECORD_LEN: u64 = 256 * 1024; - /// The directory of a store's leases, in messages. const LEASES_WHAT: &str = "the directory of the leases of the template store"; /// The directory of a store's tombstones, in messages. const TOMBSTONES_WHAT: &str = "the directory of the tombstones of the template store"; /// The record of a lease in its store, in messages. const LEASE_DESCRIPTION: &str = "lease of the template store"; -/// The binding of a clone in its sandbox, in messages. -const BINDING_DESCRIPTION: &str = "clone binding"; /// A tombstone, in messages. const TOMBSTONE_DESCRIPTION: &str = "tombstone of the template store"; -/// The identity of a lease: 16 random bytes that its acquisition draws. -/// -/// Its text form, which `Display` writes and `FromStr` parses, is 32 lowercase hexadecimal -/// digits; parsing also accepts uppercase ones. -#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct LeaseId([u8; ID_LEN]); - -impl LeaseId { - /// Returns the identity whose bytes are `bytes`. - pub const fn from_bytes(bytes: [u8; ID_LEN]) -> Self { - Self(bytes) - } - - /// Returns the bytes of the identity. - pub const fn as_bytes(&self) -> &[u8; ID_LEN] { - &self.0 - } -} - -impl fmt::Display for LeaseId { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - formatter.write_str(&hex(&self.0)) - } -} - -impl fmt::Debug for LeaseId { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(formatter, "LeaseId({self})") - } -} - -impl FromStr for LeaseId { - type Err = Error; - - /// Parses the text form of a lease's identity, failing with [`ErrorKind::Unsupported`] for - /// anything else. - fn from_str(text: &str) -> Result { - parse_hex_id(text).map(Self).ok_or_else(|| { - Error::unsupported(format!( - "{text:?} is not the identity of a lease, which is 32 hexadecimal digits" - )) - }) - } -} - -/// A lease of a generation of a template, which keeps the generation in its store for the clone -/// of one sandbox, whose VM restores it. -/// -/// The store keeps the record of each lease that it granted, and the sandbox keeps the same -/// record as its clone binding, [`CLONE_BINDING_NAME`], beside its launch journal. A lease names -/// its sandbox by the path of its directory with its links resolved, so that no removal or change -/// of a link, or of a directory that a `..` component passes through, hides the sandbox from a -/// [retirement](Store::retire). A sandbox that holds a lease must not move, and goes only through -/// its store's [removal](LockedStore::remove), which makes the binding's absence durable before -/// the sandbox goes. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct Lease { - /// The lease's identity. - id: LeaseId, - /// The identity of the store that granted it. - store: StoreId, - /// The store generation of the leased generation. - generation: StoreGeneration, - /// The identity of the leased generation's key. - key: KeyIdentity, - /// The directory of the sandbox that holds the lease, as an absolute path with its links - /// resolved, which is valid Unicode. - sandbox: PathBuf, -} - +// A lease is the sandbox's store's, which reads it from the clone binding, and which holds its key +// as the identity by which the clone's activation record names it: the key's identity is a type of +// this module. impl Lease { - /// Reads the clone binding of the sandbox of `sandbox`: the lease that it holds, if it holds - /// one. - /// - /// A read that holds no lock sees one whole binding, as the latest transition left it. Fails - /// with [`ErrorKind::Io`] for a binding that cannot be read, or is malformed, and with - /// [`ErrorCode::FormatUnsupported`] for one of another format. - pub fn of(sandbox: &SandboxStore) -> Result> { - read_binding(sandbox.dir()) - } - - /// Returns the lease's identity. - pub fn id(&self) -> LeaseId { - self.id - } - - /// Returns the identity of the store that granted the lease. - pub fn store(&self) -> StoreId { - self.store - } - - /// Returns the store generation of the leased generation. - pub fn generation(&self) -> StoreGeneration { - self.generation - } - /// Returns the identity of the leased generation's key. pub fn key(&self) -> KeyIdentity { - self.key - } - - /// Returns the directory of the sandbox that holds the lease, as an absolute path with its - /// links resolved, which may differ from the path through which the sandbox was leased, and - /// through which a [release](Store::release) takes the sandbox's lock. - pub fn sandbox(&self) -> &Path { - &self.sandbox - } - - /// Returns the reference by which the activation record of the clone names the template that - /// it restores: the leased generation's store generation, and its key's identity. - pub fn template(&self) -> TemplateReference { - TemplateReference { - generation: self.generation, - key: Identity::from(self.key), - } + KeyIdentity::from(self.template().key) } } @@ -195,8 +82,10 @@ impl Lease { pub struct Acquired { /// The lease. pub lease: Lease, - /// The lock of the sandbox, which names it by the lease's path, under which the caller launches - /// its clone. Dropping it releases the lock, and keeps the lease. + /// The lock of the sandbox, which names it by the lease's path, the only one under which the + /// caller launches its clone, as the launch that the lease reserves. Dropping it releases the + /// lock, and keeps the lease, whose launch no later lock begins: a release ends the lease, and + /// another acquisition reserves another launch. pub locked: LockedStore, } @@ -250,8 +139,9 @@ impl Store { /// that keeps its generation in the store. It resolves the links of the sandbox's path before /// it takes the sandbox's lock, and the lease, the binding, and the lock that it returns all /// name the sandbox by that path, as [`Lease::sandbox`] returns it, which no later change of a - /// link redirects to another directory. It returns the lease, with the sandbox's lock, under - /// which the caller launches the clone, and releases the store's lock. + /// link redirects to another directory. It returns the lease, with the sandbox's lock, the only + /// one under which the caller launches the clone, once, as the launch that the lease reserves, + /// [`Lease::launch`], and releases the store's lock. /// /// Fails, and leases nothing: /// @@ -344,7 +234,7 @@ impl Store { }); } }; - let locked = match SandboxStore::lock_existing(&resolved, files, deadline) { + let mut locked = match SandboxStore::lock_existing(&resolved, files, deadline) { Ok(locked) => locked, Err(error) if error.kind() == ErrorKind::NotFound => { return Err(not_provisioned().with_source(error)); @@ -375,9 +265,9 @@ impl Store { "the sandbox in {} holds the lease {} of the generation {} of the template \ store {} already", locked.dir().display(), - held.id, - held.generation, - held.store + held.id(), + held.generation(), + held.store() ), )); } @@ -388,16 +278,17 @@ impl Store { locked.dir().display() ))); } - let lease = Lease { - id: LeaseId(random_id("a lease's identity")?), - store: self.id, + let lease = Lease::new( + LeaseId::from_bytes(random_id("a lease's identity")?), + self.id, generation, - key: indexed.key, - sandbox: locked.dir().to_owned(), - }; + Identity::from(indexed.key), + locked.dir().to_owned(), + LaunchId::generate()?, + ); let record = LeaseRecord::new(&lease); self.create_records_dir(LEASES_NAME, LEASES_WHAT, sync)?; - let path = self.lease_path(lease.id); + let path = self.lease_path(lease.id()); write_record(&path, &record, "the lease of the template store")?; // A retirement deletes a generation that no record of the store leases, so the record // must be durable before a binding names it. @@ -410,12 +301,13 @@ impl Store { // the generation for a binding that may outlast a crash, and a retirement removes it // once no binding names it. if unbind(locked.dir(), sync).is_err() { - return Err(held_still(error, locked.dir(), lease.id)); + return Err(held_still(error, locked.dir(), lease.id())); } let _ = remove_if_present(&path); return Err(error); } drop(lock); + locked.reserve(Some(lease.launch().clone())); Ok(Acquired { lease, locked }) } @@ -452,26 +344,27 @@ impl Store { let Some(lease) = read_binding(locked.dir())? else { return Ok(None); }; - if lease.store != self.id { + if lease.store() != self.id { return Err(Error::other(format!( "the sandbox in {} holds a lease of the template store {}, not of this one, {}", locked.dir().display(), - lease.store, + lease.store(), self.id ))); } - if locked.dir() != lease.sandbox { + if locked.dir() != lease.sandbox() { return Err(Error::other(format!( "the lock of the sandbox in {} does not name it by the path of its lease, {}, \ through which a release takes it", locked.dir().display(), - lease.sandbox.display() + lease.sandbox().display() ))); } let recorded = self.check_records_dir(LEASES_NAME, LEASES_WHAT)?; unbind(locked.dir(), sync)?; + locked.reserve(None); if recorded { - remove_lease_record(&self.lease_path(lease.id))?; + remove_lease_record(&self.lease_path(lease.id()))?; } Ok(Some(lease)) } @@ -487,12 +380,12 @@ impl Store { for id in self.records(LEASES_NAME, LEASES_WHAT, lease_name)?.records { // A release may remove the record after the listing. if let Some(lease) = self.read_lease(id)? - && lease.generation == generation + && lease.generation() == generation { leases.push(lease); } } - leases.sort_by_key(|lease| lease.id); + leases.sort_by_key(|lease| lease.id()); Ok(leases) } @@ -603,7 +496,7 @@ impl Store { let Some(lease) = self.read_lease(id)? else { continue; }; - if lease.generation != generation { + if lease.generation() != generation { continue; } if is_bound(&lease)? { @@ -611,14 +504,14 @@ impl Store { } else { // A release may have removed the binding without making that durable yet, so the // binding's absence must be durable before its record goes. - for dir in binding_dirs(&lease.sandbox)? { + for dir in binding_dirs(lease.sandbox())? { sync(dir)?; } remove_lease_record(&self.lease_path(id))?; } } if !live.is_empty() { - live.sort_by_key(|lease| lease.id); + live.sort_by_key(|lease| lease.id()); return Ok(Retirement::Draining(live)); } self.remove_payload(generation)?; @@ -749,13 +642,13 @@ impl Store { let Some(lease) = read_lease(&path, LEASE_DESCRIPTION)? else { return Ok(None); }; - if lease.id != id || lease.store != self.id { + if lease.id() != id || lease.store() != self.id { return Err(Error::io(format!( "the {LEASE_DESCRIPTION} {} is malformed: it is the lease {} of the template \ store {}", path.display(), - lease.id, - lease.store + lease.id(), + lease.store() ))); } Ok(Some(lease)) @@ -861,8 +754,8 @@ fn may_run(dir: &Path) -> Error { /// Returns whether the clone binding of the sandbox of `lease` names it still: unless the sandbox /// is gone, holds no clone binding, or holds one of another lease. fn is_bound(lease: &Lease) -> Result { - Ok(read_binding(&lease.sandbox)? - .is_some_and(|binding| binding.id == lease.id && binding.store == lease.store)) + Ok(read_binding(lease.sandbox())? + .is_some_and(|binding| binding.id() == lease.id() && binding.store() == lease.store())) } /// Returns the directories whose entries changed as the clone binding of the sandbox in `sandbox` @@ -953,26 +846,6 @@ fn held_still(error: Error, dir: &Path, id: LeaseId) -> Error { } } -/// Reads the clone binding of the sandbox in `dir`, or returns `None` if it holds none, or if the -/// directory does not exist. -fn read_binding(dir: &Path) -> Result> { - read_lease(&dir.join(CLONE_BINDING_NAME), BINDING_DESCRIPTION) -} - -/// Reads the record of a lease at `path`, its `description`, or returns `None` if it does not -/// exist. -fn read_lease(path: &Path, description: &str) -> Result> { - let Some(record) = read_record::(path, description, MAX_LEASE_RECORD_LEN)? else { - return Ok(None); - }; - record.into_lease().map(Some).map_err(|problem| { - Error::io(format!( - "the {description} {} is malformed: {problem}", - path.display() - )) - }) -} - /// Parses the name of the record of a lease, `ID.json`, in the form in which the store writes /// it. fn lease_name(name: &str) -> Option { @@ -1007,63 +880,6 @@ struct Records { temporaries: Vec, } -/// The record of a lease, as JSON: in its store, and as the clone binding of its sandbox. -#[derive(Debug, Serialize, Deserialize)] -#[serde(rename_all = "camelCase", deny_unknown_fields)] -struct LeaseRecord { - /// [`STORE_FORMAT`]. - format: u32, - /// The identity of the store that granted the lease. - store: String, - /// The identity of the lease. - lease: String, - /// The store generation of the leased generation. - generation: u64, - /// The identity of the leased generation's key. - key: String, - /// The directory of the sandbox that holds the lease, as an absolute path. - sandbox: String, -} - -impl LeaseRecord { - /// Returns the record of `lease`. - fn new(lease: &Lease) -> Self { - Self { - format: STORE_FORMAT, - store: lease.store.to_string(), - lease: lease.id.to_string(), - generation: lease.generation.get().get(), - key: lease.key.to_string(), - sandbox: lease.sandbox.to_string_lossy().into_owned(), - } - } - - /// Checks the record, and returns its lease, or says why it is malformed. - fn into_lease(self) -> std::result::Result { - let field = - |what: &str, error: Error| format!("its {what} is invalid: {}", error.message()); - let sandbox = PathBuf::from(self.sandbox); - if !sandbox.is_absolute() { - return Err(format!( - "its sandbox, {}, is not an absolute path", - sandbox.display() - )); - } - Ok(Lease { - id: self - .lease - .parse() - .map_err(|error| field("identity", error))?, - store: self.store.parse().map_err(|error| field("store", error))?, - generation: NonZeroU64::new(self.generation) - .map(StoreGeneration::new) - .ok_or("it leases store generation 0, which no generation has")?, - key: self.key.parse().map_err(|error| field("key", error))?, - sandbox, - }) - } -} - /// A tombstone, as JSON. #[derive(Debug, Serialize, Deserialize)] #[serde(rename_all = "camelCase", deny_unknown_fields)] diff --git a/nvx-rs/src/host/template/store/leases/tests.rs b/nvx-rs/src/host/template/store/leases/tests.rs index 56fb937cb..2c90bad64 100644 --- a/nvx-rs/src/host/template/store/leases/tests.rs +++ b/nvx-rs/src/host/template/store/leases/tests.rs @@ -1,17 +1,25 @@ //! Tests of leases, clone bindings, tombstones, and retirement, by one process, and by -//! transactions in processes of their own, which [`transaction_role`] runs. +//! transactions in processes of their own, which [`transaction_role`] runs; and of the crashes of +//! a clone's orchestrator, which [`clone_role`] runs, with a [`vmm_role`] as the clone's VMM. use std::cell::{Cell, RefCell}; +use std::convert::Infallible; use std::env; +use std::ffi::OsString; use std::io::{BufRead, BufReader, Read, Write}; use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio}; +use std::thread; use std::time::Duration; use serde_json::{Value, json}; use super::*; +use crate::activation::TemplateReference; +use crate::host::process::{Process, ProcessRef}; use crate::host::store::files::write_private; -use crate::host::store::{FORMAT, RUNTIME_NAME, Recovery, RuntimeIdentity}; +use crate::host::store::lease::BINDING_DESCRIPTION; +use crate::host::store::{FORMAT, LaunchState, RUNTIME_NAME, Recovery, RuntimeIdentity}; +use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; use crate::host::template::eligibility::MAX_DOWNTIME_NS; use crate::host::template::store::tests::{ link_dir, link_file, publish, publish_with, remove, storage, store as new_store, unlink_dir, @@ -43,6 +51,30 @@ const ROLE_SANDBOX: &str = "NVX_TEMPLATE_LEASE_TEST_SANDBOX"; const READY: &str = "nvx-template-lease-test-ready"; /// The start of the line in which [`transaction_role`] prints how its transaction ended. const OUTCOME: &str = "nvx-template-lease-test-outcome "; +/// The step after which [`clone_role`] crashes: `after-record`, `after-binding`, or `acquired`, of +/// its acquisition; `before-spawn`, `after-spawn`, `after-identity`, or `committed`, of its +/// clone's launch; or `releasing`, of a release. +const ROLE_CRASH: &str = "NVX_TEMPLATE_LEASE_TEST_CRASH"; +/// The working directory of the VMM that [`clone_role`] launches. +const ROLE_VMM: &str = "NVX_TEMPLATE_LEASE_TEST_VMM"; +/// The start of the line in which [`clone_role`] prints the reference of the VMM that it spawned, +/// as JSON. +const REFERENCE: &str = "nvx-template-lease-test-vmm "; +/// The start of the line that [`clone_role`] prints as it crashes, before the step after which it +/// crashes. +const CRASHED: &str = "nvx-template-lease-test-crashed "; +/// File whose presence in its working directory marks a [`vmm_role`]. Without it, the role +/// returns at once, as it does when `cargo test -- --ignored` runs it. +const VMM_MARKER: &str = "vmm-role"; +/// File whose appearance in its working directory ends a [`vmm_role`]. +const EXIT_MARKER: &str = "exit"; +/// The capability of the VMMs of these tests' clones. +const CAPABILITY: [u8; CAPABILITY_LEN] = [7; CAPABILITY_LEN]; +/// Longest wait for a VMM that is expected to exit, for a sandbox's lock, or for a reattach to +/// become determinate. +const TIMEOUT: Duration = Duration::from_secs(60); +/// Longest life of a [`vmm_role`] that nothing ends, so that a failed test leaves no process. +const VMM_LIFETIME: Duration = Duration::from_secs(180); /// Returns the files that these tests' callers keep in their sandboxes. fn files() -> CallerFiles { @@ -148,6 +180,28 @@ fn ids(leases: &[Lease]) -> String { ids.join(" ") } +/// Returns the arguments that run only the role `name` of this test executable, on libtest's main +/// thread, so that libtest prints the role's name before the role's own output. +fn role(name: &str) -> Vec { + [ + "--exact", + &format!("host::template::store::leases::tests::{name}"), + "--ignored", + "--nocapture", + "--test-threads=1", + ] + .map(OsString::from) + .to_vec() +} + +/// Returns the error with which `locked` refuses to begin the launch `launch`. +fn launch_refusal(locked: &mut LockedStore, launch: &LaunchId) -> Error { + match locked.begin_launch(launch, &runtime(), &CAPABILITY) { + Ok(_) => panic!("the launch {launch} began"), + Err(error) => error, + } +} + #[test] fn leases_bind_clones_to_the_generations_that_they_restore() { let (directory, store) = new_store(); @@ -238,7 +292,11 @@ fn acquisitions_refuse_what_they_cannot_lease_and_lease_nothing() { let mut locked = SandboxStore::lock_existing(&sandbox, &files(), None).unwrap(); drop( locked - .begin_launch(&runtime(), &[7; CAPABILITY_LEN]) + .begin_launch( + &LaunchId::generate().unwrap(), + &runtime(), + &[7; CAPABILITY_LEN], + ) .unwrap(), ); drop(locked); @@ -277,7 +335,7 @@ fn releases_end_leases_once_their_clones_have_exited() { drop( acquired .locked - .begin_launch(&runtime(), &[7; CAPABILITY_LEN]) + .begin_launch(lease.launch(), &runtime(), &[7; CAPABILITY_LEN]) .unwrap(), ); let error = store.release(&mut acquired.locked).unwrap_err(); @@ -312,6 +370,79 @@ fn releases_end_leases_once_their_clones_have_exited() { assert_eq!(store.leases(generation).unwrap(), [again]); } +#[test] +fn a_lease_allows_one_launch_of_its_clone_under_the_lock_that_acquired_it() { + let (directory, store) = new_store(); + let published = publish(&store, &ManifestSpec::template(), storage(1)); + let sandbox = provision(directory.path(), "sandbox"); + let mut acquired = store + .acquire( + published.generation(), + &sandbox, + &files(), + &after(&published, 1), + None, + ) + .unwrap(); + let reserved = acquired.lease.launch().clone(); + let once = + format!("whose launch {reserved} begins once, under the lock that acquired the lease"); + // No launch but the reserved one. + let other = LaunchId::generate().unwrap(); + let error = launch_refusal(&mut acquired.locked, &other); + assert_eq!(error.code(), ErrorCode::SandboxLeased, "{error}"); + assert!( + error.message().ends_with(&format!( + "which reserves the launch {reserved}, not {other}" + )), + "{error}" + ); + // The reserved launch, which its journal names, once. + drop( + acquired + .locked + .begin_launch(&reserved, &runtime(), &CAPABILITY) + .unwrap(), + ); + assert_eq!( + acquired.locked.journal().unwrap().unwrap().launch(), + &reserved + ); + assert_eq!(acquired.locked.recover().unwrap(), Recovery::Exited); + for launch in [&reserved, &other] { + let error = launch_refusal(&mut acquired.locked, launch); + assert_eq!(error.code(), ErrorCode::SandboxLeased, "{error}"); + } + let error = launch_refusal(&mut acquired.locked, &reserved); + assert!(error.message().ends_with(&once), "{error}"); + // Once a release ends the lease, the sandbox launches as any launch. + assert_eq!( + store.release(&mut acquired.locked).unwrap(), + Some(acquired.lease.clone()) + ); + drop( + acquired + .locked + .begin_launch(&other, &runtime(), &CAPABILITY) + .unwrap(), + ); + assert_eq!(acquired.locked.recover().unwrap(), Recovery::Exited); + drop(acquired); + // The launch of a lease whose acquisition's lock is gone, as a crash leaves it, begins under + // no other lock. + let lease = leased(&store, &published, &sandbox); + let mut locked = SandboxStore::lock_existing(&sandbox, &files(), None).unwrap(); + let error = launch_refusal(&mut locked, lease.launch()); + assert_eq!(error.code(), ErrorCode::SandboxLeased, "{error}"); + assert!( + error.message().ends_with(&format!( + "whose launch {} begins once, under the lock that acquired the lease", + lease.launch() + )), + "{error}" + ); +} + #[test] fn tombstones_quarantine_generations_and_keep_their_leases() { let (directory, store) = new_store(); @@ -1082,6 +1213,7 @@ fn records_keep_their_json_form() { "generation": generation.get().get(), "key": published.key().to_string(), "sandbox": sandbox.to_str().unwrap(), + "launch": lease.launch().as_str(), }); let record = store .dir() @@ -1150,7 +1282,7 @@ fn malformed_records_and_foreign_entries_fail_clearly() { let error = store.leases(generation).unwrap_err(); assert_eq!(error.code(), ErrorCode::FormatUnsupported, "{error}"); // Records that are not leases. - let malformed: [(Change, &str); 5] = [ + let malformed: [(Change, &str); 7] = [ ( |value| value["generation"] = json!(0), "it leases store generation 0", @@ -1164,6 +1296,17 @@ fn malformed_records_and_foreign_entries_fail_clearly() { |value| value["lease"] = json!("00"), "its identity is invalid", ), + ( + |value| value["launch"] = json!("00"), + "its launch is invalid", + ), + // A lease reserves its clone's launch. + ( + |value| { + value.as_object_mut().unwrap().remove("launch"); + }, + "is malformed", + ), (|value| value["extra"] = json!(1), "is malformed"), ]; for (change, problem) in malformed { @@ -1173,6 +1316,16 @@ fn malformed_records_and_foreign_entries_fail_clearly() { assert!(error.message().contains("is malformed"), "{error}"); assert!(error.message().contains(problem), "{error}"); } + // A key that is not a key's identity fails as its parse does. + rewrite(&bound, |value| value["key"] = json!("00")); + let error = Lease::of(&SandboxStore::open(&sandbox).unwrap()).unwrap_err(); + let parse = "00".parse::().unwrap_err(); + assert!( + error + .message() + .ends_with(&format!("its key is invalid: {}", parse.message())), + "{error}" + ); fs::write(&bound, valid.to_string()).unwrap(); // A record of another lease than its name names, or of another store. let other_lease: Change = |value| value["lease"] = json!("0".repeat(32)); @@ -1306,13 +1459,7 @@ impl Transaction { ) -> Self { let mut command = Command::new(env::current_exe().unwrap()); command - .args([ - "--exact", - "host::template::store::leases::tests::transaction_role", - "--ignored", - "--nocapture", - "--test-threads=1", - ]) + .args(role("transaction_role")) .env(ROLE_STORE, store) .env(ROLE_ACTION, action) .env(ROLE_GENERATION, generation.to_string()) @@ -1574,6 +1721,470 @@ fn a_retirement_that_races_acquisitions_and_releases_in_other_processes_waits_fo assert!(store.tombstones().unwrap().is_empty()); } +/// A clone's orchestrator in a process of its own, which [`run_clone_role`] runs: it leases the +/// store generation [`ROLE_GENERATION`] of the store in [`ROLE_STORE`] to the sandbox in +/// [`ROLE_SANDBOX`], and launches the clone with a [`vmm_role`] in [`ROLE_VMM`] as its VMM, or it +/// releases the sandbox's lease; and it exits without unwinding after the step that [`ROLE_CRASH`] +/// names, as a crash would, once it has printed the reference of the VMM that it spawned, if any. +#[test] +#[ignore = "a clone's orchestrator that the lease tests run in a process of its own"] +fn clone_role() { + let (Some(dir), Ok(generation), Some(sandbox), Ok(step), Some(vmm)) = ( + env::var_os(ROLE_STORE), + env::var(ROLE_GENERATION), + env::var_os(ROLE_SANDBOX).map(PathBuf::from), + env::var(ROLE_CRASH), + env::var_os(ROLE_VMM).map(PathBuf::from), + ) else { + return; + }; + let store = Store::open(dir).unwrap(); + let generation = StoreGeneration::new(generation.parse().unwrap()); + // The step's transaction crashes as it makes its write in `target` durable. + let target = match step.as_str() { + "after-record" => Some(store.dir().join(LEASES_NAME)), + "after-binding" | "releasing" => Some(sandbox.clone()), + _ => None, + }; + let sync = |dir: &Path| -> Result<()> { + if target.as_deref() == Some(dir) { + crash(&step); + } + sync_dir(dir) + }; + if step == "releasing" { + let mut locked = SandboxStore::lock_existing(&sandbox, &files(), None).unwrap(); + store.release_with(&mut locked, &sync).unwrap(); + panic!("the release did not make the removal of its binding durable"); + } + let now = after(store.index().unwrap().get(generation).unwrap(), 1); + let mut acquired = store + .acquire_with(generation, sandbox.clone(), &files(), &now, None, &sync) + .unwrap(); + let reserved = acquired.lease.launch().clone(); + let command = |_: &str| -> Result> { + Ok(VmmCommand::new( + env::current_exe().unwrap(), + role("vmm_role"), + &vmm, + )) + }; + match step.as_str() { + "acquired" => crash(&step), + "before-spawn" => { + let _ = supervisor::launch( + &mut acquired.locked, + &reserved, + &runtime(), + &CAPABILITY, + |_| crash(&step), + ); + } + "after-spawn" => { + // The steps of `supervisor::launch`, up to the spawn. + let mut launch = acquired + .locked + .begin_launch(&reserved, &runtime(), &CAPABILITY) + .unwrap(); + let command = command(launch.endpoint()).unwrap(); + let log = launch.take_log().unwrap(); + let launched = supervisor::spawn(&command, &CAPABILITY, log).unwrap(); + print_reference(launched.process().unwrap().reference()); + crash(&step); + } + "after-identity" => { + let starting = supervisor::launch( + &mut acquired.locked, + &reserved, + &runtime(), + &CAPABILITY, + command, + ) + .unwrap(); + print_reference(starting.process().reference()); + crash(&step); + } + "committed" => { + let starting = supervisor::launch( + &mut acquired.locked, + &reserved, + &runtime(), + &CAPABILITY, + command, + ) + .unwrap(); + let running = starting.commit().unwrap(); + print_reference(running.record().process()); + crash(&step); + } + _ => {} + } + panic!("the clone's orchestrator did not crash after {step}"); +} + +/// The VMM of a clone of [`clone_role`]: reads its capability as OpenVMM does, then runs until a +/// file named [`EXIT_MARKER`] appears in its working directory, which [`VMM_MARKER`] marks, or for +/// [`VMM_LIFETIME`] at most. +#[test] +#[ignore = "a VMM that the lease tests run in a process of its own"] +fn vmm_role() { + let directory = env::current_dir().unwrap(); + if !directory.join(VMM_MARKER).exists() { + return; + } + let mut capability = [0; CAPABILITY_LEN]; + std::io::stdin().read_exact(&mut capability).unwrap(); + assert_eq!(capability, CAPABILITY); + let started = Instant::now(); + while !directory.join(EXIT_MARKER).exists() && started.elapsed() < VMM_LIFETIME { + thread::sleep(Duration::from_millis(10)); + } +} + +/// Prints `reference` as JSON, on a line of its own, after [`REFERENCE`]. +fn print_reference(reference: &ProcessRef) { + println!("\n{REFERENCE}{}", serde_json::to_string(reference).unwrap()); +} + +/// Ends this process at once, once it has printed that it crashes after `step`, without unwinding +/// or running destructors, as a crash would. +fn crash(step: &str) -> ! { + println!("\n{CRASHED}{step}"); + std::io::stdout().flush().unwrap(); + std::process::exit(0) +} + +/// Runs [`clone_role`] for the generation `generation` of `store`, the sandbox in `sandbox`, and a +/// VMM in `vmm`, until it crashes after `step`, and returns the reference of the VMM that it +/// spawned, if it spawned one. +fn run_clone_role( + step: &str, + store: &Store, + generation: StoreGeneration, + sandbox: &Path, + vmm: &Path, +) -> Option { + let output = Command::new(env::current_exe().unwrap()) + .args(role("clone_role")) + .env(ROLE_STORE, store.dir()) + .env(ROLE_GENERATION, generation.to_string()) + .env(ROLE_SANDBOX, sandbox) + .env(ROLE_CRASH, step) + .env(ROLE_VMM, vmm) + .stdin(Stdio::null()) + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&output.stdout); + let crashed = format!("{CRASHED}{step}"); + assert!( + output.status.success() && stdout.lines().any(|line| line == crashed), + "{}: {stdout}{}", + output.status, + String::from_utf8_lossy(&output.stderr) + ); + stdout + .lines() + .find_map(|line| line.strip_prefix(REFERENCE)) + .map(|reference| serde_json::from_str(reference).unwrap()) +} + +/// The working directory of a [`vmm_role`], which it marks. Dropping it ends the VMM, and waits +/// for it to exit, so that a failed test leaves no process behind. +struct VmmDir { + /// The reference of the VMM, once the test knows it. + reference: Option, + /// The directory, which is removed after the VMM exits. + dir: tempfile::TempDir, +} + +impl VmmDir { + /// Creates the directory, and marks it. + fn new() -> Self { + let dir = tempfile::tempdir().unwrap(); + fs::write(dir.path().join(VMM_MARKER), b"").unwrap(); + Self { + reference: None, + dir, + } + } + + /// Returns the path of the directory. + fn path(&self) -> &Path { + self.dir.path() + } + + /// Asks the VMM to exit. + fn end(&self) { + fs::write(self.path().join(EXIT_MARKER), b"").unwrap(); + } +} + +impl Drop for VmmDir { + fn drop(&mut self) { + let _ = fs::write(self.path().join(EXIT_MARKER), b""); + if let Some(reference) = &self.reference + && let Ok(Some(process)) = Process::open(reference) + && !process.wait(Instant::now() + TIMEOUT).unwrap_or(false) + { + let _ = process.kill(); + let _ = process.wait(Instant::now() + TIMEOUT); + } + } +} + +/// A clone whose orchestrator, [`clone_role`], crashed after a step of the clone's launch, in a +/// fresh temporary directory: its VMM's working directory, its template store and sandbox, and +/// the lease that the sandbox holds. +struct Crashed { + /// The working directory of the clone's VMM, which ends the VMM as it drops, before the rest. + vmm: VmmDir, + /// The template store. + store: Store, + /// The sandbox's directory. + sandbox: PathBuf, + /// The lease that the sandbox holds. + lease: Lease, + /// The temporary directory of the store and the sandbox. + _directory: tempfile::TempDir, +} + +impl Crashed { + /// Publishes a generation in a new store, and leases it to a new sandbox, whose clone + /// [`clone_role`] launches until it crashes after `step`. + fn after(step: &str) -> Self { + let (directory, store) = new_store(); + let published = publish(&store, &ManifestSpec::template(), storage(1)); + let sandbox = provision(directory.path(), "sandbox"); + let mut vmm = VmmDir::new(); + vmm.reference = run_clone_role(step, &store, published.generation(), &sandbox, vmm.path()); + let lease = binding(&sandbox).expect("the crash kept the clone's lease"); + Self { + vmm, + store, + sandbox, + lease, + _directory: directory, + } + } + + /// Takes the lock of the clone's sandbox, as its restarted orchestrator does. + fn lock(&self) -> LockedStore { + SandboxStore::lock_existing(&self.sandbox, &files(), Some(Instant::now() + TIMEOUT)) + .unwrap() + } + + /// Checks that the store refuses to release the lease of the sandbox of `locked`, whose VM + /// may still run, and that the sandbox keeps it. + fn refuses_release(&self, locked: &mut LockedStore) { + let error = self.store.release(locked).unwrap_err(); + assert_eq!(error.code(), ErrorCode::SandboxRunning, "{error}"); + assert_eq!(binding(&self.sandbox), Some(self.lease.clone())); + } + + /// Checks that the store releases the lease of the sandbox of `locked`, and that the lease is + /// gone. + fn releases(&self, locked: &mut LockedStore) { + assert_eq!( + self.store.release(locked).unwrap(), + Some(self.lease.clone()) + ); + assert_eq!(binding(&self.sandbox), None); + assert!( + self.store + .leases(self.lease.generation()) + .unwrap() + .is_empty() + ); + } +} + +/// Reattaches to the VM of the sandbox of `locked` until the result is determinate, as a caller +/// retries an indeterminate reattach, and returns it. +/// +/// On Windows, a scanner that briefly opens the log of a launch can keep its claim held for a +/// moment. +fn reattach_until_determinate(locked: &mut LockedStore) -> Reattached { + let deadline = Instant::now() + TIMEOUT; + loop { + match supervisor::reattach(locked, &runtime()).unwrap() { + Reattached::Indeterminate(reason) if Instant::now() < deadline => { + eprintln!("reattach is indeterminate: {reason}"); + thread::sleep(Duration::from_millis(100)); + } + reattached => return reattached, + } + } +} + +#[test] +fn a_crash_during_an_acquisition_leaves_no_lease_or_one_that_launches_nothing() { + let (directory, store) = new_store(); + let published = publish(&store, &ManifestSpec::template(), storage(1)); + let generation = published.generation(); + // No VMM runs: the orchestrator crashes before any launch. + let vmm = directory.path(); + // After the record, before the binding: the sandbox holds no lease, and leases the generation + // again, while the record, which no binding names, keeps the generation. + let unbound = provision(directory.path(), "unbound"); + assert_eq!( + run_clone_role("after-record", &store, generation, &unbound, vmm), + None + ); + assert_eq!(binding(&unbound), None); + let [orphan]: [Lease; 1] = store.leases(generation).unwrap().try_into().unwrap(); + assert_eq!(orphan.sandbox(), unbound); + let mut leases = vec![leased(&store, &published, &unbound)]; + // After the binding, before or after the acquisition returned: the sandbox holds the lease, + // whose launch no later lock begins, and which a release ends. Another acquisition then + // reserves another launch. + for step in ["after-binding", "acquired"] { + let sandbox = provision(directory.path(), step); + assert_eq!( + run_clone_role(step, &store, generation, &sandbox, vmm), + None + ); + let lease = binding(&sandbox).unwrap(); + assert!(store.leases(generation).unwrap().contains(&lease), "{step}"); + let mut locked = SandboxStore::lock_existing(&sandbox, &files(), None).unwrap(); + let error = launch_refusal(&mut locked, lease.launch()); + assert_eq!(error.code(), ErrorCode::SandboxLeased, "{step}: {error}"); + assert_eq!(store.release(&mut locked).unwrap(), Some(lease.clone())); + drop(locked); + let again = leased(&store, &published, &sandbox); + assert_ne!(again.launch(), lease.launch()); + leases.push(again); + } + // A retirement removes the record that no binding names, and waits for the leases. + leases.sort_by_key(Lease::id); + assert_eq!( + store.retire(generation, None).unwrap(), + Retirement::Draining(leases.clone()) + ); + assert_eq!(store.leases(generation).unwrap(), leases); +} + +#[test] +fn a_clone_that_crashes_before_its_spawn_keeps_its_lease_until_its_launch_ends() { + let crashed = Crashed::after("before-spawn"); + assert_eq!(crashed.vmm.reference, None); + let mut locked = crashed.lock(); + // The journal names the launch that the lease reserved, which spawned nothing. + let journal = locked.journal().unwrap().unwrap(); + assert_eq!(journal.launch(), crashed.lease.launch()); + assert_eq!(journal.state(), LaunchState::Prepared); + crashed.refuses_release(&mut locked); + match reattach_until_determinate(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + crashed.releases(&mut locked); +} + +#[test] +fn a_clone_that_crashes_after_its_spawn_keeps_its_lease_until_its_unrecorded_vmm_exits() { + let crashed = Crashed::after("after-spawn"); + let reference = crashed.vmm.reference.clone().unwrap(); + let process = Process::open(&reference).unwrap().unwrap(); + let mut locked = crashed.lock(); + assert_eq!( + locked.journal().unwrap().unwrap().state(), + LaunchState::Prepared + ); + // Reattach can neither prove that the VMM, which no record names, exited, nor verify it: it + // signals nothing, and the lease stays. + let Reattached::Indeterminate(_) = supervisor::reattach(&mut locked, &runtime()).unwrap() + else { + panic!("reattach ended a launch whose VMM runs unverified"); + }; + assert!(!process.has_exited().unwrap()); + crashed.refuses_release(&mut locked); + // Once the VMM exits, the claim that it released proves it. + crashed.vmm.end(); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + match reattach_until_determinate(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + crashed.releases(&mut locked); +} + +#[test] +fn a_clone_that_crashes_after_recording_its_vmm_keeps_its_lease_until_reattach_ends_it() { + let crashed = Crashed::after("after-identity"); + let reference = crashed.vmm.reference.clone().unwrap(); + let process = Process::open(&reference).unwrap().unwrap(); + let mut locked = crashed.lock(); + let journal = locked.journal().unwrap().unwrap(); + assert_eq!(journal.launch(), crashed.lease.launch()); + assert_eq!(journal.state(), LaunchState::Spawned); + assert_eq!(journal.process(), Some(&reference)); + crashed.refuses_release(&mut locked); + // The launch never committed, so reattach ends its VMM through its verified reference. + match supervisor::reattach(&mut locked, &runtime()).unwrap() { + Reattached::Exited { + record: None, + recovery: Recovery::Terminated(terminated), + } => assert_eq!(terminated, reference), + other => panic!("unexpected reattach: {other:?}"), + } + assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + crashed.releases(&mut locked); +} + +#[test] +fn a_committed_clone_keeps_its_lease_through_a_crash_until_its_vm_stops() { + let crashed = Crashed::after("committed"); + let reference = crashed.vmm.reference.clone().unwrap(); + let mut locked = crashed.lock(); + crashed.refuses_release(&mut locked); + // The restarted orchestrator holds the clone's VM again, and its lease lasts while it runs. + let Reattached::Live(running) = supervisor::reattach(&mut locked, &runtime()).unwrap() else { + panic!("the committed clone was not reattached"); + }; + assert_eq!(running.record().launch(), crashed.lease.launch()); + assert_eq!(running.record().process(), &reference); + crashed.refuses_release(&mut locked); + let stopped = running + .stop( + &mut locked, + Instant::now() + TIMEOUT, + None, + |_running, _deadline| { + crashed.vmm.end(); + Ok::<(), Infallible>(()) + }, + ) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + crashed.releases(&mut locked); +} + +#[test] +fn a_crash_during_a_release_leaves_a_record_that_a_retirement_removes() { + let (directory, store) = new_store(); + let published = publish(&store, &ManifestSpec::template(), storage(1)); + let generation = published.generation(); + let sandbox = provision(directory.path(), "sandbox"); + let lease = leased(&store, &published, &sandbox); + assert_eq!( + run_clone_role("releasing", &store, generation, &sandbox, directory.path()), + None + ); + // The binding went first: the sandbox holds no lease, while the record, which no binding + // names, keeps the generation until a retirement removes it. + assert_eq!(binding(&sandbox), None); + assert_eq!(store.leases(generation).unwrap(), [lease]); + assert_eq!(release(&store, &sandbox).unwrap(), None); + assert_eq!(store.retire(generation, None).unwrap(), Retirement::Retired); + assert!(store.leases(generation).unwrap().is_empty()); +} + /// Tests of the records that a Windows file system without POSIX deletion keeps under their names /// while other handles hold them, after a transaction in another process removed them, and of the /// records whose access control lists refuse access, which are not removed ones. diff --git a/nvx-rs/tests/store.rs b/nvx-rs/tests/store.rs index 877480e49..8bb3b1cde 100644 --- a/nvx-rs/tests/store.rs +++ b/nvx-rs/tests/store.rs @@ -22,8 +22,8 @@ use nvx::ErrorKind; use nvx::host::platform; use nvx::host::process::{BootId, Process, ProcessRef}; use nvx::host::store::{ - ArtifactIdentity, IdentityCheck, IdentityPart, JOURNAL_NAME, Launch, LaunchState, LockedStore, - RUNTIME_NAME, Recovery, RuntimeIdentity, SandboxStore, + ArtifactIdentity, IdentityCheck, IdentityPart, JOURNAL_NAME, Launch, LaunchId, LaunchState, + LockedStore, RUNTIME_NAME, Recovery, RuntimeIdentity, SandboxStore, }; use nvx::host::supervisor::{self, Launched, VmmCommand}; use nvx::protocol::CAPABILITY_LEN; @@ -285,7 +285,9 @@ fn orchestrator_role() { // another working directory, and that recovery examines, is the same. assert!(store.dir().is_absolute()); let mut locked = store.lock(None).unwrap(); - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); if crash == "before-spawn" { crash_now(); } @@ -345,7 +347,9 @@ fn restrictive_umask_role() { // The second launch opens the lock file again and replaces the first launch's files. for _ in 0..2 { let mut locked = store.lock(None).unwrap(); - let launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); assert_eq!(store.journal().unwrap().as_ref(), Some(launch.journal())); assert_eq!(store.read_capability().unwrap(), CAPABILITY); for path in [ @@ -448,7 +452,9 @@ fn recovery_never_signals_a_process_that_the_journal_does_not_name() { let _spawns = serialized(); let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let vmm = Vmm(Some(spawn_vmm(&mut launch, &store))); let recorded = launch .record_process(vmm.0.as_ref().unwrap()) @@ -492,7 +498,9 @@ fn a_committed_launch_names_its_vm_for_another_build() { let _spawns = serialized(); let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let endpoint = launch.endpoint().to_owned(); let vmm = Vmm(Some(spawn_vmm(&mut launch, &store))); let process = launch.record_process(vmm.0.as_ref().unwrap()).unwrap(); @@ -535,7 +543,9 @@ fn an_abandoned_launch_ends_its_recorded_vmm() { let _spawns = serialized(); let (_directory, store) = store(); let mut locked = store.lock(None).unwrap(); - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let mut vmm = Vmm(Some(spawn_vmm(&mut launch, &store))); let held = launch.record_process(vmm.0.as_ref().unwrap()).unwrap(); let process = held.reference().clone(); @@ -576,7 +586,9 @@ fn an_endpoint_served_without_the_claimed_log_is_not_signaled() { // A VM of another sandbox serves an endpoint. let (_other_directory, other) = store(); let mut other_locked = other.lock(None).unwrap(); - let mut other_launch = other_locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut other_launch = other_locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let served = other_launch.endpoint().to_owned(); let fake = Vmm(Some(spawn_fake(&mut other_launch, &other, 0))); let fake_process = other_launch @@ -743,7 +755,9 @@ fn unsupported_and_corrupt_records_fail_clearly() { assert!(error.message().contains(expected), "{name}: {error}"); // Nothing reads past the record or rewrites it: a launch and a recovery fail the same // way, and the record stays as it was. - let launch = locked.begin_launch(&runtime(), &CAPABILITY).map(drop); + let launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .map(drop); assert!(launch.unwrap_err().message().contains(expected), "{name}"); if as_name == JOURNAL_NAME { assert!( diff --git a/nvx-rs/tests/supervisor.rs b/nvx-rs/tests/supervisor.rs index c51226bda..280a173fb 100644 --- a/nvx-rs/tests/supervisor.rs +++ b/nvx-rs/tests/supervisor.rs @@ -34,7 +34,7 @@ use nvx::host::process::{BootId, Process, ProcessRef, WaitMethod}; #[cfg(windows)] use nvx::host::store::FORMAT; use nvx::host::store::{ - ArtifactIdentity, IdentityPart, LaunchState, LockedStore, RUNTIME_NAME, Recovery, + ArtifactIdentity, IdentityPart, LaunchId, LaunchState, LockedStore, RUNTIME_NAME, Recovery, RuntimeIdentity, SandboxStore, }; use nvx::host::supervisor::{self, Inherited, Launched, Reattached, StopOutcome, Vmm, VmmCommand}; @@ -210,12 +210,20 @@ fn orchestrator_fixture() { }; match step.as_str() { "before-spawn" => { - let _ = supervisor::launch(&mut locked, &runtime(), &CAPABILITY, |_| crash()); + let _ = supervisor::launch( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |_| crash(), + ); unreachable!("the orchestrator crashed before its spawn"); } "after-spawn" => { // The steps of `supervisor::launch`, up to the spawn. - let mut launch = locked.begin_launch(&runtime(), &CAPABILITY).unwrap(); + let mut launch = locked + .begin_launch(&LaunchId::generate().unwrap(), &runtime(), &CAPABILITY) + .unwrap(); let command = command(launch.endpoint()).unwrap(); let log = launch.take_log().unwrap(); let launched = supervisor::spawn(&command, &CAPABILITY, log).unwrap(); @@ -223,14 +231,26 @@ fn orchestrator_fixture() { crash(); } "after-identity" => { - let starting = - supervisor::launch(&mut locked, &runtime(), &CAPABILITY, command).unwrap(); + let starting = supervisor::launch( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + command, + ) + .unwrap(); print_reference(starting.process().reference()); crash(); } "committed" => { - let starting = - supervisor::launch(&mut locked, &runtime(), &CAPABILITY, command).unwrap(); + let starting = supervisor::launch( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + command, + ) + .unwrap(); let vmm = starting.commit().unwrap(); drop(locked); print_reference(vmm.record().process()); @@ -333,10 +353,16 @@ fn spawn_vmm(directory: &Path, breakaway: bool) -> Launched { /// Launches a VMM fixture in `vmm` through the supervisor in `locked`, under [`runtime`], and /// commits the launch, as an orchestrator does once the VMM serves its control endpoint. fn launch_vmm(locked: &mut LockedStore, vmm: &mut VmmDirectory) -> Vmm { - let starting = supervisor::launch(locked, &runtime(), &CAPABILITY, |endpoint| { - assert!(!endpoint.is_empty()); - Ok(vmm_command(vmm.path(), false)) - }) + let starting = supervisor::launch( + locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |endpoint| { + assert!(!endpoint.is_empty()); + Ok(vmm_command(vmm.path(), false)) + }, + ) .unwrap(); vmm.record(starting.process().reference()); // The launch recorded its VMM before it returned. diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 26bde76f4..8dba4d3f3 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -170,7 +170,7 @@ use nvx::host::instance::generate_capability; use nvx::host::launch::LaunchSpec; use nvx::host::report; use nvx::host::session; -use nvx::host::store::{IdentityPart, LockedStore, SandboxStore}; +use nvx::host::store::{IdentityPart, LaunchId, LockedStore, SandboxStore}; use nvx::host::supervisor::{self, Reattached, Starting, StopOutcome, Vmm, VmmCommand}; use nvx::protocol::CAPABILITY_LEN; use nvx::protocol::root::ImageFormat; @@ -970,7 +970,8 @@ fn launch( hypervisor, ); let mut deadline = Instant::now() + START_TIMEOUT; - let starting = supervisor::launch(&mut locked, &runtime.identity, &capability, |endpoint| { + let launch_id = LaunchId::generate().map_err(|error| describe(&error))?; + let command = |endpoint: &str| -> nvx::Result> { // The launch has checked that no VM of the sandbox runs, which could still read the // initramfs of an earlier launch. initramfs::write(&runtime.artifacts.initrd, guest_agent, &initramfs).map_err(|error| { @@ -1005,7 +1006,14 @@ fn launch( command.inherit = prepared.inherited(); } Ok(command) - }) + }; + let starting = supervisor::launch( + &mut locked, + &launch_id, + &runtime.identity, + &capability, + command, + ) .map_err(|error| describe(&error))?; let process = starting.process().reference().clone(); println!("pid={}", process.pid); From a101e0509f1f9eb904cdc6f0a4bc2bbe9f27ebc9 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 17:29:09 -0700 Subject: [PATCH 2/2] nvx-rs: retry the recoveries of closed launches in the lease tests Four lease tests began a launch, dropped it, and asserted that the next recovery ends it. On Linux, a process that a concurrent test forks holds a copy of the closed log until it executes its program. If that outlasts recovery's own wait, the recovery is indeterminate and the test fails although nothing is wrong. The tests now retry the recovery until it ends the launch, through test_support::released, as the sandbox store's tests do. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc335726-838c-4538-a3c4-d6791aba5101 --- nvx-rs/src/host/template/store/leases/tests.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nvx-rs/src/host/template/store/leases/tests.rs b/nvx-rs/src/host/template/store/leases/tests.rs index 2c90bad64..f4074f923 100644 --- a/nvx-rs/src/host/template/store/leases/tests.rs +++ b/nvx-rs/src/host/template/store/leases/tests.rs @@ -408,7 +408,7 @@ fn a_lease_allows_one_launch_of_its_clone_under_the_lock_that_acquired_it() { acquired.locked.journal().unwrap().unwrap().launch(), &reserved ); - assert_eq!(acquired.locked.recover().unwrap(), Recovery::Exited); + recover_unspawned(&mut acquired.locked); for launch in [&reserved, &other] { let error = launch_refusal(&mut acquired.locked, launch); assert_eq!(error.code(), ErrorCode::SandboxLeased, "{error}"); @@ -426,7 +426,7 @@ fn a_lease_allows_one_launch_of_its_clone_under_the_lock_that_acquired_it() { .begin_launch(&other, &runtime(), &CAPABILITY) .unwrap(), ); - assert_eq!(acquired.locked.recover().unwrap(), Recovery::Exited); + recover_unspawned(&mut acquired.locked); drop(acquired); // The launch of a lease whose acquisition's lock is gone, as a crash leaves it, begins under // no other lock.