diff --git a/nvx-rs/README.md b/nvx-rs/README.md index b0c5f5d08..a461affcd 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -59,11 +59,11 @@ nvx-rs/ │ │ the typed calls through which it runs the time ABI's steps and the reseed │ └── bin/ nvx-rs, nvx-fake-openvmm, and the guest binaries ├── tests/ fake-OpenVMM, CLI, NVXS session, process-supervision, checked-handoff, store, -│ and guest-binary tests, the CLI's scenario tests and their harness, in cli/, -│ the store's record fixtures, the hand-written sandbox configurations of every -│ format, the cases of the export and root boot tokens, and the template -│ manifests that OpenVMM wrote, in fixtures/, and the conformance vectors of -│ the wire formats and of a template's key, in vectors/ +│ template-capture, and guest-binary tests, the CLI's scenario tests and their +│ harness, in cli/, the store's record fixtures, the hand-written sandbox +│ configurations of every format, the cases of the export and root boot tokens, +│ and the template manifests that OpenVMM wrote, in fixtures/, and the +│ conformance vectors of the wire formats and of a template's key, in vectors/ ├── build.rs bundled-release staging └── artifacts.json the NVX release that the `bundled` feature stages ``` @@ -93,11 +93,11 @@ nvx-rs/ | `nvx::host::nvxs` | `HostConnection`: the host's role of the NVXS session over a platform transport, on which `nvx::host::session` and other host applications run their protocols. It attaches with the launch capability, waiting while the broker asks it to, sends application bytes in as many DATA records as they take, and returns each DATA payload from the guest, with deadlines; a receive that times out keeps a partial record | | `nvx::host::process` | Verified process references. A `ProcessRef` names one process by its ID, its start time, and the `BootId` of the host boot during which it started. `Process::open` checks a reference and holds its process through a pidfd on Linux or a process handle on Windows, whether or not the caller started it, so that waits sleep until the process exits and termination never reaches a process that reuses its ID. A reference from an earlier boot names an exited process, and a process whose identity this user cannot check, such as another user's, makes `Process::open` fail rather than report an exit. Linux hosts without pidfds poll the reference instead, and cannot terminate the process | | `nvx::host::supervisor` | The supervisor of a sandbox's VMM, beneath the CLI, `Instance`, and external orchestrators, which keep the configuration, the authorization, and the application protocol. `launch` runs the store's launch steps under the sandbox's lock, as a launch whose `LaunchId` the caller reserved beforehand, a fresh one for each launch, or that a clone's lease reserves: it writes the journal's marker, which names the launch, spawns OpenVMM with the claimed log, records its `ProcessRef` right after, and returns a `Starting` launch that the caller commits once the VMM serves its control endpoint, or abandons. A launch that fails never leaves an unrecorded VMM running. `VmmCommand::inherit` lists the descriptors (Linux) or handles (Windows) that the VMM inherits under their numbers, besides its standard streams, and it inherits nothing else of the caller's: on Linux, they must be close-on-exec, a flag that only the VMM's own process clears; on Windows, they join the VMM's explicit handle list. `Vmm::stop` runs one graceful step of the caller's and waits for the VMM's exit event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. `Vmm::terminate` takes that forced step alone, within the caller's bound, for a VM whose guest the caller must not ask anything, such as one of another runtime identity. `reattach` recovers an interrupted launch, and holds the recorded VMM again after the orchestrator restarts within the same host boot: `Live`, `Mismatched` with another runtime identity, `Exited`, or `Indeterminate`. A clone's launch takes its locks in one order, the template store's, then the sandbox's, which guards its clone binding and launch journal, then the scratch image's claim, and holds none for the VM's lifetime; its lease lasts through crashes and reattaches, until a release ends it once a stop, a termination, or a reattach has verified the VM's exit | -| `nvx::host::template` | The metadata of templates, captures of a microVM from which independent clones are restored. `TemplatePlan::new` lowers an adapter's `SandboxPlan` into the plan of a template's capture, without I/O: the placeholder `distro`, which its `DistroBinding` binds, and whose length is the capacity of every clone's image, over a dormant scratch device, the sparse file `SCRATCH_NAME` in the capture's transaction directory, with the dormant filesystem slot, and no NIC or one under the template's policy, which allows neither ingress nor host-loopback access both ways; `TemplateResources`, with every processor online at boot, and RAM and a `--memory-capacity` in blocks of `MEMORY_BLOCK_SIZE`; the kernel command line `nvx_sandbox=1`, so that the capture's whole command line holds tokens of OpenVMM's platform allowlist alone, and no tenant data, which the tests check with a copy of the allowlist's source; and the capture's options, `--snapshot-tier platform`, a `--snapshot-destination`, `SNAPSHOT_NAME` in the transaction directory, and `--snapshot-block-identity generation` with the capture's bound-block `StorageGeneration`, so that the capture hashes no block device. `TemplateManifest::read` reads the `manifest.bin` that OpenVMM wrote for a capture, at most `MAX_MANIFEST_LEN` bytes of a regular file that is not a link or other reparse point, and refuses another snapshot format version, an unknown, mistyped, or repeated field, truncated bytes, and a capture that is not a template's: a platform-tier capture for clones, with a read-only `distro` whose identity is unbound, a fresh scratch, and the dormant filesystem slot. `CaptureClass` holds the invariant inputs of a capture, known before it: the runtime, the guest's build, the backend, the requested CPU profile, the processors and RAM, the geometries of the disks, the network, and the adapter's other `Invariants`. `TemplateKey::new` checks that the capture made the machine that its class requested, and derives the template's compatibility key, from metadata alone: the runtime and contract versions through which its clones are restored, and the backend, CPU profile and effective CPUID, topology, rates, resources, kernel command line, disks, and devices of its machine. A capture's own metadata, such as its time anchor and host, stays out of the key. A key and a class each have one canonical, versioned encoding, whose domain-separated SHA-256 digest is its identity, a `KeyIdentity`, which converts to and from the `nvx::activation::Identity` by which a clone's activation record names its template's key, or a `ClassIdentity`; the [conformance vectors](tests/vectors/README.md) of version 1, `tests/vectors/template-key-v1.txt`, pin the key's encoding and identity. `Store` keeps the generations of templates that a host published, apart from every sandbox's store, at a location that the adapter chooses: a private directory with an identity of its own, a `StoreId`, the immutable payload of each generation, its key's canonical encoding beside the snapshot that OpenVMM published, and the versioned `Index` of them, which lists each generation with the identities of its key and capture class and its own metadata, such as its time contract. `Store::create` creates a store, or opens the one that its directory holds, crash-safely and once between concurrent creations, and refuses a directory with foreign entries; `Store::open` opens one. Its records are JSON in one format, replaced atomically, and records of another format fail rather than being migrated. Its directories must be private to the current user, and it refuses links and, on Windows, other reparse points among its directories and files. Four namespaces stay apart: the `nvx::activation::StoreGeneration`, the host's publication ID, which the index assigns in the order of publication and never twice, and by which a clone's activation record names its template; the bound-block `StorageGeneration` of a capture, which belongs to the storage owner; the time lineage counter of the time contract; and the VM generation ID, which the store never records. `TimeContract::eligibility` decides, apart from compatibility, whether a clone may restore a capture at a host's time, a `HostTime`, which `HostTime::now` samples as OpenVMM does: within the boot of the capture, by the host's monotonic clock, and otherwise by UTC, the downtime must not be negative, and the interval by which a restore advances the clone's VM time at most `MAX_DOWNTIME_NS`, 30 days. `Store::select` selects the latest eligible generation of a key that no tombstone quarantines. `Store::acquire` leases a generation to a clone in one transaction, under the store's lock and then the sandbox's: it checks the generation's eligibility, writes the lease's record in the store and the clone binding, `clone.json`, beside the sandbox's launch journal, and returns the lease with the sandbox's lock, the only one under which the caller launches the clone, once, as the launch that the lease reserves, `Lease::launch`. A lease never expires: `Store::release` ends it once the sandbox records no VM and no launch. `Store::tombstone` quarantines a generation, which takes no new lease while its leases keep working, and `Store::retire` deletes it once its leases have drained, crash-safely, removing the records of leases that no clone binding names any more | +| `nvx::host::template` | The metadata of templates, captures of a microVM from which independent clones are restored. `TemplatePlan::new` lowers an adapter's `SandboxPlan` into the plan of a template's capture, without I/O: the placeholder `distro`, which its `DistroBinding` binds, and whose length is the capacity of every clone's image, over a dormant scratch device, the sparse file `SCRATCH_NAME` in the capture's transaction directory, with the dormant filesystem slot, and no NIC or one under the template's policy, which allows neither ingress nor host-loopback access both ways; `TemplateResources`, with every processor online at boot, and RAM and a `--memory-capacity` in blocks of `MEMORY_BLOCK_SIZE`; the kernel command line `nvx_sandbox=1`, so that the capture's whole command line holds tokens of OpenVMM's platform allowlist alone, and no tenant data, which the tests check with a copy of the allowlist's source; and the capture's options, `--snapshot-tier platform`, a `--snapshot-destination`, `SNAPSHOT_NAME` in the transaction directory, and `--snapshot-block-identity generation` with the capture's bound-block `StorageGeneration`, so that the capture hashes no block device. `TemplateManifest::read` reads the `manifest.bin` that OpenVMM wrote for a capture, at most `MAX_MANIFEST_LEN` bytes of a regular file that is not a link or other reparse point, and refuses another snapshot format version, an unknown, mistyped, or repeated field, truncated bytes, and a capture that is not a template's: a platform-tier capture for clones, with a read-only `distro` whose identity is unbound, a fresh scratch, and the dormant filesystem slot. `CaptureClass` holds the invariant inputs of a capture, known before it: the runtime, the guest's build, the backend, the requested CPU profile, the processors and RAM, the geometries of the disks, the network, and the adapter's other `Invariants`. `TemplateKey::new` checks that the capture made the machine that its class requested, and derives the template's compatibility key, from metadata alone: the runtime and contract versions through which its clones are restored, and the backend, CPU profile and effective CPUID, topology, rates, resources, kernel command line, disks, and devices of its machine. A capture's own metadata, such as its time anchor and host, stays out of the key. A key and a class each have one canonical, versioned encoding, whose domain-separated SHA-256 digest is its identity, a `KeyIdentity`, which converts to and from the `nvx::activation::Identity` by which a clone's activation record names its template's key, or a `ClassIdentity`; the [conformance vectors](tests/vectors/README.md) of version 1, `tests/vectors/template-key-v1.txt`, pin the key's encoding and identity. `Store` keeps the generations of templates that a host published, apart from every sandbox's store, at a location that the adapter chooses: a private directory with an identity of its own, a `StoreId`, the immutable payload of each generation, its key's canonical encoding beside the snapshot that OpenVMM published, and the versioned `Index` of them, which lists each generation with the identities of its key and capture class and its own metadata, such as its time contract. `Store::create` creates a store, or opens the one that its directory holds, crash-safely and once between concurrent creations, and refuses a directory with foreign entries; `Store::open` opens one. Its records are JSON in one format, replaced atomically, and records of another format fail rather than being migrated. Its directories must be private to the current user, and it refuses links and, on Windows, other reparse points among its directories and files. Four namespaces stay apart: the `nvx::activation::StoreGeneration`, the host's publication ID, which the index assigns in the order of publication and never twice, and by which a clone's activation record names its template; the bound-block `StorageGeneration` of a capture, which belongs to the storage owner; the time lineage counter of the time contract; and the VM generation ID, which the store never records. `TimeContract::eligibility` decides, apart from compatibility, whether a clone may restore a capture at a host's time, a `HostTime`, which `HostTime::now` samples as OpenVMM does: within the boot of the capture, by the host's monotonic clock, and otherwise by UTC, the downtime must not be negative, and the interval by which a restore advances the clone's VM time at most `MAX_DOWNTIME_NS`, 30 days. `Store::select` selects the latest eligible generation of a key that no tombstone quarantines. `Store::acquire` leases a generation to a clone in one transaction, under the store's lock and then the sandbox's: it checks the generation's eligibility, writes the lease's record in the store and the clone binding, `clone.json`, beside the sandbox's launch journal, and returns the lease with the sandbox's lock, the only one under which the caller launches the clone, once, as the launch that the lease reserves, `Lease::launch`. A lease never expires: `Store::release` ends it once the sandbox records no VM and no launch. `Store::tombstone` quarantines a generation, which takes no new lease while its leases keep working, and `Store::retire` deletes it once its leases have drained, crash-safely, removing the records of leases that no clone binding names any more. `Store::create_template` serves a `Capture`, an adapter's plan for a capture class with the placeholder's binding and the runtime's artifacts, with a generation of its class that the store holds and that is eligible at the host's time, which it reads once it holds the class's slot, `Created::Existing`, or captures one, `Created::Captured`, while `Store::refresh_template` always captures, and leaves older generations and their leases untouched. A capture runs in its class's slot under `captures/`, a sandbox store whose lock serializes the class's captures, which comes before the store's lock, and which keeps the log and outcome report of the VMM of the class's latest capture: in a private transaction on the store's volume, which is also the sandbox store of its VM, since OpenVMM requires the control socket of a VM that captures a snapshot, on Linux, beside the snapshot's destination, it creates the dormant scratch, checks the placeholder through its binding, and both block devices at their paths, regular files that are not links or other reparse points, that belong to the user or an administrator, and that no one else may change, and launches the template's VM through `supervisor::launch` from the transaction, holding no handle that could write either file, since OpenVMM opens them itself by path, as it does a restore's. Once the VMM has exited, it checks the outcome report, the snapshot's manifest and artifacts, and the key, and publishes the transaction in two commits, a rename into `generations/` that replaces nothing, then the index. A capture that fails publishes nothing, and `Store::recover_captures` ends the captures that a crash interrupted, publishing those that were checked, and retiring the others | | `nvx::host::seal` | `FileSeal`: what the host reports about an open file from its metadata alone, without reading it: the platform scheme, the device or volume serial number, the file ID, the kind, the length, the modification time, and on Linux the change time, with a versioned serde form. A seal detects accidental change, not tampering; see [Distro bindings](#distro-bindings) | | `nvx::host::session`, `platform`, `report` | The authenticated control session, which carries NVX's control protocol, `NVXC`, one frame per DATA record of a `HostConnection`; Linux and Windows primitives; and OpenVMM's outcome report | | `nvx::host::sandbox` | The launch profile of `nvx-rs sandbox`, with native input validation: `SandboxLaunch`, with the distro `SandboxLayer`, an EROFS layer or a GPT disk, the one layer that a sandbox takes, the ext4 scratch disk, the entrypoint, arguments, hostname, workload identity, and `SandboxSettings` resource limits (`Option` for memory and `Option` for processes), and the `SandboxPlan` that it makes of them. `plan`: `SandboxPlan`, the typed plan of a cold layered launch, which is persistable in a versioned serde form: one read-only `distro` block device, an EROFS filesystem or a GPT disk with the identity that the guest expects of it, if any, an upper, the ext4 scratch disk or RAM, the launch settings, the `ExportPlan` of its live shares, and optionally a `SandboxNetwork`, the guest network and its typed `NetworkPolicy`, whose proxy's URL as the guest sees it the plan reports. The plan of an adapter that boots its own guest, `SandboxPlan::adapter`, holds no launch settings and no export plan. A plan knows only the `distro` and `scratch` `BlockRole`s, and gives, for every combination, its `--microvm-sandbox-block` arguments, with `--microvm-workload-identity` when it has launch settings, with no scratch block device for a RAM upper, and those of its network, which `nvx::host::network` lowers, and its kernel command line, with the boot tokens of `nvx::protocol::root`, within the 1024-byte budget that OpenVMM leaves it, refusing one whose root tokens the guest's parser refuses. Both deliver the live shares as the managed lifecycle does: the arguments export them at `nvx::host::filesystem::GUEST_EXPORT`, the command line announces the number of their mappings with `nvx_maps=`, and `mapping_table` gives the ordered table that `start` sends once the guest agent is ready; the CLI's one-shot `run` still renders them as boot tokens. An adapter's plan renders the caller's tokens before those of its root, and refuses tokens that its root or OpenVMM's shares own. `mount`: `SandboxMount` requests of live shares of host directories and files, their guest targets and the reserved ones, their access policies and owner, and the checks of their host paths; `plan_mounts`, which plans them through `nvx::host::filesystem` for the container root, and `boot_shares`, which renders a plan as the OpenVMM arguments and boot tokens of one share or of the aggregate of several, as the one-shot lifecycle delivers them. `options`: the checks that `nvx-rs sandbox` and `nvx-rs run` apply to their options: the attribution of `--mount-deny`, `--mount-allow`, and `--mount-write` to shares, the systemd entrypoint check and the explicit verification of the distro layer, the targets of `nvx-rs run` shares, and the managed exec environment; see [Sandbox input contract](#sandbox-input-contract). `distro`: the `DistroBinding` of a distro layer, an EROFS layer or a GPT disk, which explicit verification makes or a staging caller imports, and its check from metadata alone, which returns the layer open; see [Distro bindings](#distro-bindings). `scratch`: the `ScratchBinding` of a scratch image, its identity, kind, and length, and its check from metadata alone, which claims the image for one sandbox at a time and returns it open. `prepared`: `PreparedLaunch`, the launch of a plan with the checked files of its block devices, which renders the plan's OpenVMM arguments and kernel command line, naming the checked files as `/proc/self/fd/N` on Linux and by their final paths on Windows, and lists the descriptors that the VMM inherits through `VmmCommand::inherit`; see [Checked handoff](#checked-handoff) | -| `nvx::testing::fake_openvmm` | A fake OpenVMM that serves the control console and runs scripted workloads without a VM, for direct launches and for the managed and one-shot sandboxes of `nvx-rs sandbox` | +| `nvx::testing::fake_openvmm` | A fake OpenVMM that serves the control console and runs scripted workloads without a VM, for direct launches and for the managed and one-shot sandboxes of `nvx-rs sandbox`, and runs the VMs of templates' captures, which publish a snapshot, or fail, as a scenario asks | | `nvx::host::cli` | Feature `cli`: the `nvx-rs` executable; see [The nvx-rs CLI](#the-nvx-rs-cli) | | `nvx::guest` | The programs that run in Linux guests; see [Guest binaries](#guest-binaries) | | `nvx::guest::mappings` | `Mapper`: applies host mappings for any guest init or agent beneath the descriptor of a `TargetRoot`, the root of a mount of a whole file system, which `statmount` tells from a bind of a directory below it, such as a layered root's overlay or a guest's `/`, from a `SourceRoot`, such as the guest's mount of the host's export. A `Mapping` names a relative source, an absolute target, its `Access`, and optionally the `Kind` that its source must have. The mapper opens each component with `openat2`, through no symbolic link or magic link, creates what a target lacks only on the root's own mounts, which `statx` tells apart by their unique mount IDs, never inside another mapping, and checks kinds, overlaps, and the targets that the root profile's `Reserved` set holds, with the `/run/nvx/time` tree and its ancestors always. It attaches each mapping as a detached copy of its source's mount, `open_tree`, restricted `nosuid`, `nodev`, and read-only when asked, `mount_setattr`, before `move_mount` puts it at its target, then checks that it is there. Zero mappings do no work; the first that fails is reported with its index as an `ApplyError`, and latches the mapper. `Mapper::teardown` detaches a mapping that a rename moved and that the attach failed to detach, then unmounts the mappings in reverse order, and reports each failure. `bind` applies one runtime bind through the same walk and attach. Linux 6.8 or later. NVX's managed agent applies its host mapping table through it, beneath `/` in direct mode and beneath the container's assembled root in sandbox mode, and a layered `Root` its live shares and mappings | @@ -825,7 +825,12 @@ command-line tokens make it fail: `fake_exit_on_start=STATUS`, OpenVMM, it opens the sandbox's block devices at the paths that its options name and holds them until it exits, after `fake_block_delay_ms=MS` if asked, and records what each held in `fake-openvmm-TOKEN-blocks.json` next to the -kernel. A managed `start` also reads the root's +kernel. It also runs the VMs of templates' captures, whose block devices it +opens as OpenVMM does, without following a link and, on Windows, sharing only +reads and deletion, recording in that file whether a handle could write them; +it then publishes the snapshot of the manifest that +`fake-openvmm-capture.json` next to the kernel names, or fails, never ends, or +crashes, as that scenario asks. A managed `start` also reads the root's [runtime identity](#runtime-identity), so a test root holds build provenance, as the roots of the CLI tests do. @@ -1106,7 +1111,9 @@ NVX_BUNDLE=skip cargo test --all-features # unit, conformance-vector, property ``` The tests need no hypervisor. The fake-OpenVMM test drives `Instance` end to end -against `nvx-fake-openvmm`. The CLI test runs the +against `nvx-fake-openvmm`, and the template-capture test runs templates' +captures through the supervisor against it, up to a crash of the adapter whose +VMM a recovery ends. The CLI test runs the [scenario tests](#scenario-tests) of `nvx-rs`, and checks the guest-feature requirement and, on Linux, Ctrl-C. The [conformance vectors](tests/vectors/README.md) of the wire formats cover both diff --git a/nvx-rs/src/host/platform/linux.rs b/nvx-rs/src/host/platform/linux.rs index 1b3c4c41d..a65e2b2ec 100644 --- a/nvx-rs/src/host/platform/linux.rs +++ b/nvx-rs/src/host/platform/linux.rs @@ -519,6 +519,54 @@ pub(crate) fn private_file_problem(_file: &File) -> io::Result> { Ok(None) } +/// Returns why the directory `path`, a snapshot that OpenVMM published, is not private to the +/// current user, if it is not, as [`check_owner_only_dir`] does: OpenVMM creates it with mode 0700, +/// and its files with mode 0600. +pub(crate) fn check_snapshot_dir(path: &Path) -> io::Result> { + check_owner_only_dir(path) +} + +/// Returns why another user than the current one, or root, may change the file that `file` holds +/// open, if one may: unless it belongs to the current user or to root, and grants neither its +/// group nor others the permission to write it. +pub(crate) fn foreign_writer_problem(file: &File) -> io::Result> { + let metadata = file.metadata()?; + Ok(foreign_writer( + metadata.uid(), + metadata.mode(), + sys::effective_ids().0, + )) +} + +/// Returns why another user than `euid`, or root, may change a file that the user `uid` owns with +/// the mode `mode`, if one may. +pub(crate) fn foreign_writer(uid: u32, mode: u32, euid: u32) -> Option { + let mode = mode & PERMISSION_BITS; + if uid != euid && uid != 0 { + Some(format!( + "belongs to user {uid}, neither the current user {euid} nor root" + )) + } else if mode & 0o022 != 0 { + Some(format!("is writable by other users (mode {mode:03o})")) + } else { + None + } +} + +/// Lets the regular file that `file` holds open for writing take no space for the blocks that +/// nobody writes: nothing to do on Linux, whose filesystems allocate no block that a file's +/// extension leaves unwritten. +pub(crate) fn make_sparse(_file: &File) -> io::Result<()> { + Ok(()) +} + +/// Renames `from` to `to` unless `to` exists, which it leaves as it is, failing with +/// [`io::ErrorKind::AlreadyExists`]: `renameat2` with `RENAME_NOREPLACE`, which a filesystem that +/// cannot rename without replacing refuses. +pub(crate) fn rename_no_replace(from: &Path, to: &Path) -> io::Result<()> { + sys::rename_no_replace(from, to) +} + /// Gives the regular file `path` the mode `mode` if it belongs to the current user, and leaves /// anything else, which includes a symbolic link or a missing file, unchanged. /// diff --git a/nvx-rs/src/host/platform/other.rs b/nvx-rs/src/host/platform/other.rs index ad7b1215d..b90eb0115 100644 --- a/nvx-rs/src/host/platform/other.rs +++ b/nvx-rs/src/host/platform/other.rs @@ -157,6 +157,27 @@ pub(crate) fn private_file_problem(_file: &File) -> io::Result> { Err(unsupported()) } +/// Checks the directory `path`, a snapshot that OpenVMM published, as [`check_owner_only_dir`] +/// does. +pub(crate) fn check_snapshot_dir(path: &Path) -> io::Result> { + check_owner_only_dir(path) +} + +/// Fails: this host cannot check who may change a file. +pub(crate) fn foreign_writer_problem(_file: &File) -> io::Result> { + Err(unsupported()) +} + +/// Fails: this host makes no template's files. +pub(crate) fn make_sparse(_file: &File) -> io::Result<()> { + Err(unsupported()) +} + +/// Fails: this host makes no template's files. +pub(crate) fn rename_no_replace(_from: &Path, _to: &Path) -> io::Result<()> { + Err(unsupported()) +} + /// Leaves the command as it is, unless it should inherit descriptors: this host cannot restrict /// what the command inherits, and fails with [`io::ErrorKind::Unsupported`] instead. pub(crate) fn detach(_command: &mut Command, inherit: &[Inherited<'_>]) -> io::Result<()> { diff --git a/nvx-rs/src/host/platform/windows.rs b/nvx-rs/src/host/platform/windows.rs index 708c5a34c..1058cd821 100644 --- a/nvx-rs/src/host/platform/windows.rs +++ b/nvx-rs/src/host/platform/windows.rs @@ -16,9 +16,9 @@ use windows_sys::Win32::Foundation::{ ERROR_ACCESS_DENIED, ERROR_ALREADY_EXISTS, ERROR_BROKEN_PIPE, ERROR_FILE_NOT_FOUND, ERROR_INVALID_PARAMETER, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_NO_DATA, ERROR_OPERATION_ABORTED, ERROR_PIPE_BUSY, ERROR_PIPE_NOT_CONNECTED, ERROR_SHARING_VIOLATION, - ERROR_SUCCESS, FILETIME, FreeLibrary, GENERIC_READ, GENERIC_WRITE, GetHandleInformation, - GetLastError, HANDLE, HANDLE_FLAG_INHERIT, INVALID_HANDLE_VALUE, SetHandleInformation, - WAIT_OBJECT_0, WAIT_TIMEOUT, + ERROR_SUCCESS, FILETIME, FreeLibrary, GENERIC_ALL, GENERIC_READ, GENERIC_WRITE, + GetHandleInformation, GetLastError, HANDLE, HANDLE_FLAG_INHERIT, INVALID_HANDLE_VALUE, + SetHandleInformation, WAIT_OBJECT_0, WAIT_TIMEOUT, }; use windows_sys::Win32::Security::{ ACCESS_ALLOWED_ACE, ACE_HEADER, ACL, ACL_REVISION, ACL_SIZE_INFORMATION, AclSizeInformation, @@ -31,17 +31,21 @@ use windows_sys::Win32::Security::{ WinBuiltinAdministratorsSid, WinCreatorOwnerRightsSid, WinCreatorOwnerSid, WinLocalSystemSid, }; use windows_sys::Win32::Storage::FileSystem::{ - BY_HANDLE_FILE_INFORMATION, CreateDirectoryW, CreateFileW, FILE_ALL_ACCESS, - FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT, FILE_FLAG_BACKUP_SEMANTICS, - FILE_FLAG_OPEN_REPARSE_POINT, FILE_FLAG_OVERLAPPED, GetFileInformationByHandle, OPEN_EXISTING, - READ_CONTROL, ReadFile, SECURITY_IDENTIFICATION, SECURITY_SQOS_PRESENT, WriteFile, + BY_HANDLE_FILE_INFORMATION, CreateDirectoryW, CreateFileW, DELETE, FILE_ALL_ACCESS, + FILE_APPEND_DATA, FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT, FILE_DELETE_CHILD, + FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_FLAG_OVERLAPPED, + FILE_WRITE_ATTRIBUTES, FILE_WRITE_DATA, FILE_WRITE_EA, GetFileInformationByHandle, MoveFileExW, + OPEN_EXISTING, READ_CONTROL, ReadFile, SECURITY_IDENTIFICATION, SECURITY_SQOS_PRESENT, + WRITE_DAC, WRITE_OWNER, WriteFile, }; #[cfg(feature = "cli")] use windows_sys::Win32::System::Console::{ CTRL_C_EVENT, GetStdHandle, STD_ERROR_HANDLE, STD_HANDLE, STD_INPUT_HANDLE, STD_OUTPUT_HANDLE, SetConsoleCtrlHandler, }; -use windows_sys::Win32::System::IO::{CancelIoEx, GetOverlappedResult, OVERLAPPED}; +use windows_sys::Win32::System::IO::{ + CancelIoEx, DeviceIoControl, GetOverlappedResult, OVERLAPPED, +}; use windows_sys::Win32::System::LibraryLoader::{ GetProcAddress, LOAD_LIBRARY_SEARCH_SYSTEM32, LoadLibraryExW, }; @@ -1079,6 +1083,16 @@ pub(crate) fn check_owner_only_dir(path: &Path) -> io::Result> { /// Returns why the directory `path` cannot hold a sandbox store of `user`, if it cannot. fn store_dir_problem(path: &Path, user: &CurrentUser) -> io::Result> { + plain_dir_problem(path, user, Required::PrivateDirectory) +} + +/// Returns why the directory `path` is not a plain directory, not a link or any other reparse +/// point, whose owner `user` trusts, and whose access control list is as `required`, if it is not. +fn plain_dir_problem( + path: &Path, + user: &CurrentUser, + required: Required, +) -> io::Result> { let directory = OpenOptions::new() .access_mode(READ_CONTROL) .custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT) @@ -1088,7 +1102,21 @@ fn store_dir_problem(path: &Path, user: &CurrentUser) -> io::Result io::Result> { + plain_dir_problem(path, &CurrentUser::get()?, Required::Private) } /// Returns why the file that `file` holds open, the lock, a record, or the capability of a @@ -1097,16 +1125,51 @@ fn store_dir_problem(path: &Path, user: &CurrentUser) -> io::Result io::Result> { - privacy_problem(file, &CurrentUser::get()?, false) + privacy_problem(file, &CurrentUser::get()?, Required::Private) +} + +/// Returns why another user than the current one, SYSTEM, or the Administrators group may change +/// the file that `file` holds open, if one may: unless it belongs to the user, or to SYSTEM or the +/// Administrators group, and its access control list allows no one else to write or extend it, to +/// change its attributes, its list, or its owner, or to delete it. Others may read it. +pub(crate) fn foreign_writer_problem(file: &fs::File) -> io::Result> { + privacy_problem(file, &CurrentUser::get()?, Required::Unwritable) +} + +/// The rights of an access control entry that change a file or directory: write or extend it, or +/// its extended attributes, delete it or its children, change its attributes, its list, or its +/// owner, and the generic rights that include those. +const CHANGE_RIGHTS: u32 = FILE_WRITE_DATA + | FILE_APPEND_DATA + | FILE_WRITE_EA + | FILE_DELETE_CHILD + | FILE_WRITE_ATTRIBUTES + | DELETE + | WRITE_DAC + | WRITE_OWNER + | GENERIC_WRITE + | GENERIC_ALL; + +/// What [`privacy_problem`] requires of the access control list of a file or directory, whose +/// owner the user must trust in any case. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Required { + /// That it allow no one else any access. + Private, + /// That it allow no one else any access, and pass entries on to the files created in it, as + /// a directory of a store must. + PrivateDirectory, + /// That it allow no one else any of the [`CHANGE_RIGHTS`]. + Unwritable, } /// Returns why the file or directory that `file` holds is not private to `user`, if it is not: it /// must belong to the user, or to SYSTEM or the Administrators group, and its access control list -/// must allow no one else. A `directory` must also pass entries on to the files created in it. +/// must be as `required`. fn privacy_problem( file: &fs::File, user: &CurrentUser, - directory: bool, + required: Required, ) -> io::Result> { let handle = file.as_raw_handle() as HANDLE; let information = OWNER_SECURITY_INFORMATION | DACL_SECURITY_INFORMATION; @@ -1168,9 +1231,14 @@ fn privacy_problem( match header.AceType { ACCESS_DENIED_ACE_TYPE => {} ACCESS_ALLOWED_ACE_TYPE => { + let entry = ace.cast::(); + // SAFETY: an entry that allows access starts as an ACCESS_ALLOWED_ACE. + let mask = unsafe { (*entry).Mask }; + if required == Required::Unwritable && mask & CHANGE_RIGHTS == 0 { + continue; + } // SAFETY: an entry that allows access holds its identifier from `SidStart` on. - let sid: PSID = - unsafe { (&raw mut (*ace.cast::()).SidStart).cast() }; + let sid: PSID = unsafe { (&raw mut (*entry).SidStart).cast() }; // The creator owner and the owner rights stand for an owner: the owner, or the // creator of a file in the directory, which only those that it allows may be. // SAFETY: the identifier is valid. @@ -1179,7 +1247,13 @@ fn privacy_problem( || IsWellKnownSid(sid, WinCreatorOwnerRightsSid) != 0 }; if !owner_alias && !user.trusts(sid) { - return Ok(Some("allows other users".to_owned())); + return Ok(Some( + match required { + Required::Unwritable => "allows other users to change it", + Required::Private | Required::PrivateDirectory => "allows other users", + } + .to_owned(), + )); } } other => { @@ -1191,7 +1265,7 @@ fn privacy_problem( } // A file created without entries to inherit takes its creator's default list instead, which // may allow anyone. - if directory && !inherited_by_files { + if required == Required::PrivateDirectory && !inherited_by_files { return Ok(Some( "passes no access control entries on to the files created in it".to_owned(), )); @@ -1199,6 +1273,43 @@ fn privacy_problem( Ok(None) } +/// Lets the regular file that `file` holds open for writing take no space for the ranges that +/// nobody writes: marks it sparse, which NTFS otherwise allocates whole as the file is extended. +pub(crate) fn make_sparse(file: &fs::File) -> io::Result<()> { + /// `FSCTL_SET_SPARSE`, in `winioctl.h`, which marks a file sparse without a buffer. + const FSCTL_SET_SPARSE: u32 = 0x0009_00c4; + let mut returned = 0; + // SAFETY: the handle is open for as long as `file` is borrowed, the control takes no input or + // output buffer, and the count's pointer is valid. + let set = unsafe { + DeviceIoControl( + file.as_raw_handle() as HANDLE, + FSCTL_SET_SPARSE, + ptr::null(), + 0, + ptr::null_mut(), + 0, + &mut returned, + ptr::null_mut(), + ) + }; + if set == 0 { + return Err(io::Error::last_os_error()); + } + Ok(()) +} + +/// Renames `from` to `to` unless `to` exists, which it leaves as it is, failing with +/// [`io::ErrorKind::AlreadyExists`]: `MoveFileExW` without `MOVEFILE_REPLACE_EXISTING`. +pub(crate) fn rename_no_replace(from: &Path, to: &Path) -> io::Result<()> { + let (from, to) = (wide(from), wide(to)); + // SAFETY: both names are NUL-terminated. + if unsafe { MoveFileExW(from.as_ptr(), to.as_ptr(), 0) } == 0 { + return Err(io::Error::last_os_error()); + } + Ok(()) +} + /// Replaces the access control list of the file or directory `path` with the protected list of /// the descriptor `sddl`, in the security descriptor definition language. #[cfg(test)] diff --git a/nvx-rs/src/host/store/files.rs b/nvx-rs/src/host/store/files.rs index 31ab4ce2a..63fde8e50 100644 --- a/nvx-rs/src/host/store/files.rs +++ b/nvx-rs/src/host/store/files.rs @@ -45,7 +45,7 @@ pub(super) fn private_options() -> OpenOptions { /// /// On Unix, the umask may clear bits of the mode that the file is created with, even the /// current user's permission to read it, so the mode is set again through the new file. -pub(super) fn create_new(path: &Path) -> io::Result { +pub(crate) fn create_new(path: &Path) -> io::Result { let file = private_options().create_new(true).open(path)?; #[cfg(unix)] { diff --git a/nvx-rs/src/host/supervisor/mod.rs b/nvx-rs/src/host/supervisor/mod.rs index acce0434d..888f84e7e 100644 --- a/nvx-rs/src/host/supervisor/mod.rs +++ b/nvx-rs/src/host/supervisor/mod.rs @@ -122,14 +122,19 @@ //! ## The lock order //! //! These steps, and every other transition of a template store or a sandbox, take their locks in -//! one order: the template store's lock, then the sandbox's -//! [lifecycle lock](super::store#lifecycle-locks), which guards its clone binding and its launch -//! journal, then the scratch image's claim. None waits for a lock while it holds a later one: an -//! acquisition releases the store's lock before it returns; a release, a launch, a stop, a -//! reattach, and a removal of the sandbox take no lock of the store; and a retirement takes no -//! lock of a sandbox. No lock lasts for a VM's lifetime: the caller releases the sandbox's lock -//! once the launch commits or ends, and the scratch image's claim belongs to the VMM, which holds -//! it until it exits. +//! one order: the lifecycle lock of a capture class's slot, which serializes the class's captures, +//! as [`Store::refresh_template`](crate::host::template::Store::refresh_template) describes, then +//! the template store's lock, then the [lifecycle lock](super::store#lifecycle-locks) of a sandbox, +//! which guards its clone binding and its launch journal, or of a capture's transaction, the +//! sandbox store of the capture's VM, then the scratch image's claim. None waits for a lock while +//! it holds a later one: under its slot's lock, a capture, or its recovery, takes its +//! transaction's lock and then, once it has released it, the store's lock, to publish, and no +//! other; an acquisition releases the store's lock before it returns; a release, a launch, a stop, +//! a reattach, and a removal of the sandbox take no lock of the store; and a retirement takes no +//! lock of a sandbox. No lock but those of a capture lasts for a VM's lifetime: a capture holds +//! its slot's lock and its transaction's until its VM has ended, while the caller of a clone's +//! launch releases the sandbox's lock once the launch commits or ends, and the scratch image's +//! claim belongs to the VMM, which holds it until it exits. //! //! ## Crashes //! diff --git a/nvx-rs/src/host/sys/fs.rs b/nvx-rs/src/host/sys/fs.rs new file mode 100644 index 000000000..ff8c76e42 --- /dev/null +++ b/nvx-rs/src/host/sys/fs.rs @@ -0,0 +1,36 @@ +//! Filesystem operations of Linux hosts that the standard library lacks. + +use std::ffi::CString; +use std::io; +use std::os::unix::ffi::OsStrExt; +use std::path::Path; + +use nix::errno::Errno; + +/// `renameat2`'s `RENAME_NOREPLACE`, in `linux/fs.h`. +const RENAME_NOREPLACE: nix::libc::c_uint = 1; + +/// Renames `from` to `to` unless `to` exists, which it leaves as it is, with `renameat2`'s +/// `RENAME_NOREPLACE`, through the system call itself, which `nix` wraps only for glibc. Fails +/// with [`io::ErrorKind::AlreadyExists`] if `to` exists, and as `rename` fails otherwise; a +/// filesystem that cannot rename without replacing fails it with `EINVAL`. +pub(crate) fn rename_no_replace(from: &Path, to: &Path) -> io::Result<()> { + let text = |path: &Path| { + CString::new(path.as_os_str().as_bytes()) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "a path holds a NUL byte")) + }; + let (from, to) = (text(from)?, text(to)?); + // SAFETY: `renameat2` reads the two NUL-terminated paths, which outlive the call. + Errno::result(unsafe { + nix::libc::syscall( + nix::libc::SYS_renameat2, + nix::libc::AT_FDCWD, + from.as_ptr(), + nix::libc::AT_FDCWD, + to.as_ptr(), + RENAME_NOREPLACE, + ) + }) + .map(drop) + .map_err(io::Error::from) +} diff --git a/nvx-rs/src/host/sys/mod.rs b/nvx-rs/src/host/sys/mod.rs index a791dea6a..8212248b7 100644 --- a/nvx-rs/src/host/sys/mod.rs +++ b/nvx-rs/src/host/sys/mod.rs @@ -3,10 +3,13 @@ //! The other host modules deny unsafe code. On Unix, they reach the operating system through the //! standard library or this module, which makes its system calls through `nix` and confines the //! unsafe ones behind wrappers that establish their ownership and async-signal-safety invariants. -//! `nix` has no wrapper for `pidfd_open`, `pidfd_send_signal`, `close_range`, or `getdents64`, so -//! those go through `nix`'s re-export of libc, as does the query of the `SIGINT` disposition that -//! precedes the CLI's handler. The Windows platform module still calls `windows-sys` directly. +//! `nix` has no wrapper for `pidfd_open`, `pidfd_send_signal`, `close_range`, or `getdents64`, and +//! wraps `renameat2` only for glibc, so those go through `nix`'s re-export of libc, as does the +//! query of the `SIGINT` disposition that precedes the CLI's handler. The Windows platform module +//! still calls `windows-sys` directly. +#[cfg(target_os = "linux")] +mod fs; #[cfg(feature = "cli")] mod interrupt; #[cfg(target_os = "linux")] @@ -18,6 +21,8 @@ mod socket; pub(crate) use nix::errno::Errno; use nix::unistd::{getegid, geteuid}; +#[cfg(target_os = "linux")] +pub(crate) use fs::rename_no_replace; #[cfg(feature = "cli")] pub(crate) use interrupt::{interrupted, watch_interrupts, write_interruptibly}; #[cfg(all(feature = "cli", target_os = "linux"))] diff --git a/nvx-rs/src/host/template/mod.rs b/nvx-rs/src/host/template/mod.rs index 373fdb804..8f7c5822d 100644 --- a/nvx-rs/src/host/template/mod.rs +++ b/nvx-rs/src/host/template/mod.rs @@ -20,6 +20,10 @@ //! sandbox's store: their payloads, which never change, and the versioned [`Index`] of them, //! which names each by its [`StoreGeneration`](crate::activation::StoreGeneration), at a //! location that the adapter chooses. +//! - [`Store::create_template`] and [`Store::refresh_template`] capture a template of a class, in +//! the VM that its [`TemplatePlan`] describes, which the supervisor launches, and publish it as +//! a generation of the store; [`Store::recover_captures`] ends the captures that a crash +//! interrupted. //! - [`TimeContract::eligibility`] decides whether a clone may restore a capture at a host's //! time, [`HostTime`], by the restore window of its time contract, apart from its //! compatibility. @@ -65,7 +69,8 @@ pub use self::manifest::{ }; pub use self::plan::{MEMORY_BLOCK_SIZE, SCRATCH_NAME, TemplatePlan, TemplateResources}; pub use self::store::{ - Acquired, GENERATIONS_NAME, Generation, INDEX_LOCK_NAME, INDEX_NAME, Index, KEY_NAME, - LEASES_NAME, Lease, LeaseId, MAX_GENERATIONS, Retirement, SNAPSHOT_NAME, STORE_FORMAT, - STORE_RECORD_NAME, StorageGeneration, Store, StoreId, TOMBSTONES_NAME, + Acquired, CAPTURE_RECORD_NAME, CAPTURES_NAME, Capture, Created, GENERATIONS_NAME, Generation, + INDEX_LOCK_NAME, INDEX_NAME, Index, KEY_NAME, LEASES_NAME, Lease, LeaseId, MAX_GENERATIONS, + Retirement, SNAPSHOT_NAME, STORE_FORMAT, STORE_RECORD_NAME, StorageGeneration, Store, StoreId, + TOMBSTONES_NAME, TRANSACTION_NAME, }; diff --git a/nvx-rs/src/host/template/store.rs b/nvx-rs/src/host/template/store.rs index 88f6d2a43..1c8e364d2 100644 --- a/nvx-rs/src/host/template/store.rs +++ b/nvx-rs/src/host/template/store.rs @@ -1,6 +1,7 @@ //! The store of templates, [`Store`]: its directory, its records, the namespaces of its -//! generations, and their leases, tombstones, and retirement. +//! generations, the captures that publish them, and their leases, tombstones, and retirement. +mod capture; mod leases; #[cfg(test)] mod tests; @@ -26,6 +27,7 @@ use crate::host::platform::{self, parse_hex_id}; use crate::host::process::BootId; use crate::host::store::files::{open_lock, write_private}; +pub use self::capture::{CAPTURE_RECORD_NAME, CAPTURES_NAME, Capture, Created, TRANSACTION_NAME}; pub use self::leases::{Acquired, LEASES_NAME, Lease, LeaseId, Retirement, TOMBSTONES_NAME}; /// Version of the store's records: its identity record and its index. @@ -298,12 +300,24 @@ impl Index { /// index.lock the lock that serializes the transactions of the index /// generations/ the payloads of the generations, which never change once indexed /// / the payload of the store generation G +/// capture.json the record of the capture that made it /// key.bin the canonical encoding of its compatibility key /// snapshot/ its snapshot, as OpenVMM published it /// leases/ the records of the leases that the store granted /// .json the record of the lease L /// tombstones/ the tombstones of the generations that take no new leases /// .json the tombstone of the store generation G +/// captures/ the slots of the capture classes +/// / the slot of the classes whose identities start with the digits C, +/// a sandbox store whose lock serializes the classes' captures +/// openvmm.log the log of the VMM of the slot's latest capture +/// outcome.json the outcome report of the VMM of the slot's latest capture +/// transaction/ the capture in progress: the sandbox store of its VM until the VM has +/// ended, and then the payload of a generation +/// capture.json the record of the capture +/// scratch.raw the dormant scratch device, which the capture removes once checked +/// snapshot/ the snapshot, once OpenVMM has published it +/// key.bin the template's key, once the capture has checked the snapshot /// ``` /// /// # Identity @@ -319,10 +333,10 @@ impl Index { /// of each that restores and their eligibility check without reading its payload: the identities /// of its key and of its capture class, its storage generation, and what its manifest records of /// the capture itself, its creation time, its time contract, and the lengths of its artifacts. -/// The payload of a generation never changes once the index lists it: the canonical encoding of -/// its key, which [`Store::read_key`] reads, and its snapshot. The index changes as generations -/// are published and retired, and the records of leases and tombstones as clones lease -/// generations, and as generations are quarantined and retired. +/// The payload of a generation never changes once the index lists it: the record of its capture, +/// the canonical encoding of its key, which [`Store::read_key`] reads, and its snapshot. The index +/// changes as generations are published and retired, and the records of leases and tombstones as +/// clones lease generations, and as generations are quarantined and retired. /// /// The store's records, its identity record, its index, and the records of its leases and /// tombstones, are JSON in format [`STORE_FORMAT`]. @@ -349,6 +363,99 @@ impl Index { /// - The VM generation ID is 16 random bytes that each VM process draws, which tell the clones of /// one generation apart. Their guests read it, and the store records none. /// +/// # Captures and publication +/// +/// [`Store::create_template`] and [`Store::refresh_template`] capture templates of a +/// [`CaptureClass`](crate::host::template::CaptureClass), and publish them as the store's +/// generations: a creation serves the class with a generation that the store holds, if one is +/// eligible, and captures one otherwise, while a refresh always captures. Each class has a slot in +/// `captures/`, a sandbox store named by the first 16 hexadecimal digits of the class's identity, +/// whose lock serializes the captures of the class, and which keeps the log and the outcome report +/// of the VMM of the class's latest capture. A capture holds its slot's lock until its VM has +/// ended, and takes the store's lock under it to publish, as the supervisor's +/// [lock order](crate::host::supervisor#the-lock-order) allows. +/// +/// A capture runs in a transaction, a directory private to the current user, in its slot, on the +/// store's volume. It writes the record of the capture there, `capture.json`, which names the +/// store, the class, and the [`StorageGeneration`] that it drew, and creates the template's +/// dormant scratch device, a sparse file, then lowers the adapter's plan to a +/// [`TemplatePlan`](crate::host::template::TemplatePlan), whose snapshot destination is in the +/// transaction. The transaction is also the sandbox store of the capture's VM, whose launch +/// journal records its VMM: OpenVMM requires the control socket of a VM that captures a snapshot, +/// on Linux, beside the snapshot's destination, and a sandbox store's VM listens in the store's +/// directory. The capture holds the transaction's lock, under its slot's, until the VM has ended, +/// and releases it before it publishes. It cold-boots the template's VM through +/// [`supervisor::launch`](crate::host::supervisor::launch), from the transaction, as a launch of +/// its own, and waits for the VMM to exit: the guest captures itself, and OpenVMM publishes the +/// snapshot at its destination, and exits. The capture then moves the VMM's log and outcome +/// report to its slot. +/// +/// ## Block devices by path +/// +/// OpenVMM opens the block devices of a VM whose snapshot it captures, or restores, itself, at the +/// paths that its command line names: without following a link, nor on Windows any other reparse +/// point, and sharing only reads and deletion, so that its open fails while another handle may +/// write the file. The host therefore hands OpenVMM the paths of these files, never handles, and +/// checks the file at each path just before the VM starts: +/// +/// - The directory that holds the VM's files, the capture's transaction, must be private to the +/// current user. +/// - The placeholder must pass its binding's check, +/// [`DistroBinding::check`](crate::host::sandbox::distro::DistroBinding::check), of its +/// identity and length, and then, at its path, be a regular file, not a link nor any other +/// reparse point, that is the file that the check found, that belongs to the current user or to +/// an administrator, and that no one else may change. +/// - The scratch must be, at its path, the regular file of the template's length that the capture +/// created, which no one else may change. +/// +/// The host tells these files apart by the identity that their seals record, +/// [`FileSeal`](crate::host::seal::FileSeal): their device or volume and their file ID, on Windows +/// the 128-bit ID of `FILE_ID_INFO`, which a file system such as ReFS keeps unique where the +/// 64-bit file index is not. +/// +/// As the VM starts, the host holds no handle through which it could write either file: it closes +/// the scratch once it has checked it, and holds only the placeholder that its binding's check +/// opened for reading, which on Windows lets no one write it, until the VM has ended. The +/// directories above the files must be ones that only the current user, or an administrator, may +/// change, since OpenVMM resolves each path again. A restore of a clone, whose block devices +/// OpenVMM also opens by their paths, takes the same rule. +/// +/// ## What the VM leaves +/// +/// Once the VMM has exited, and before anything is published, the capture checks that the VM's +/// outcome report records the capture's success, and that the snapshot is a directory private to +/// the current user, as OpenVMM creates it, whose manifest +/// [`TemplateManifest::read`](crate::host::template::TemplateManifest::read) reads, whose artifacts +/// are exactly its manifest, RAM, and saved state, each a regular file private to the current +/// user, of the length that the manifest records, and whose key, [`TemplateKey::new`], is its +/// class's. It then closes the lock of the transaction's sandbox store, removes the lock's file, +/// the scratch, and what OpenVMM left beside the snapshot, and writes the key, `key.bin`, which +/// marks the transaction as checked, durably. +/// +/// ## Two commits +/// +/// A capture publishes its checked transaction under the store's lock, in two commits: it renames +/// the transaction to the payload of the index's next store generation, which no rename +/// replaces, and makes the rename durable; then it writes the index, which lists the generation, +/// and makes it durable, with the store's own entry in its parent, which the store's creation +/// does not sync. A payload at the index's next store generation is one whose publication a crash +/// interrupted, which the next publication, or recovery, indexes first, once it has made the +/// rename durable as the publication would have. Each capture completes these publications, and +/// first drops from the index the generations whose payloads a crash lost, which then take none +/// of the store's capacity, before it looks for a generation or counts the store's. A creation +/// that serves a generation that the store holds makes the index, and the store's own entry, +/// durable before it returns it, since the commit that listed it may have been interrupted. +/// Publications leave the older generations, and their leases, untouched. +/// +/// A capture that fails before it checks its transaction publishes nothing, and retires the +/// transaction once its VMM has ended. One that a crash of the host interrupts is ended by the +/// next capture of its class, or by [`Store::recover_captures`], which an adapter calls as it +/// starts: each ends the launch that the transaction's sandbox store records, as +/// [`LockedStore::recover`](crate::host::store::LockedStore::recover) ends a sandbox's, and then +/// publishes the transaction if the capture had checked it, once it has synced the transaction +/// again, and retires it otherwise, once it has moved the VMM's log and outcome report to the +/// slot. +/// /// # Leases, tombstones, and retirement /// /// A clone that restores a generation leases it first, so that the generation stays in the store @@ -397,7 +504,10 @@ impl Index { /// store creates have mode 0700. On Windows, a directory that the store creates allows the user /// alone, through a protected access control list that its files inherit; one that it did not /// create may also allow SYSTEM and the Administrators group, as one under the user's profile -/// does, and must pass entries on to the files created in it. The store refuses a record whose +/// does, and must pass entries on to the files created in it. A snapshot's directory, which +/// OpenVMM creates, need not: OpenVMM gives it, and each file that it publishes in it, a protected +/// list of their own, which allows SYSTEM and the owner, and the directory's list already stops +/// anyone else from creating a file in it. The store refuses a record whose /// own list allows anyone else. A directory or a file of the store that is a link, or on Windows /// any other reparse point, is refused, and so is a record that is not a regular file. The store /// names its files by their paths, so the directories above it must be ones that only the user, @@ -591,16 +701,26 @@ impl Store { /// /// The directories of the generations, of the payload, and of the snapshot must each be /// private to the current user, and none of them a link, or on Windows any other reparse - /// point. Fails with [`ErrorKind::NotFound`] if the payload or its snapshot is missing. + /// point. The snapshot's directory is the one that OpenVMM published, which need not pass + /// access control entries on to the files created in it; see [Privacy](Store#privacy). Fails + /// with [`ErrorKind::NotFound`] if the payload or its snapshot is missing. pub fn snapshot_dir(&self, generation: &Generation) -> Result { let payload = self.payload_dir(generation.generation); let snapshot = payload.join(SNAPSHOT_NAME); self.check_generations_dir()?; - for (path, what) in [ - (&payload, "the payload of the generation"), - (&snapshot, "the snapshot of the generation"), + for (path, what, check) in [ + ( + &payload, + "the payload of the generation", + platform::check_owner_only_dir as fn(&Path) -> io::Result>, + ), + ( + &snapshot, + "the snapshot of the generation", + platform::check_snapshot_dir, + ), ] { - let problem = platform::check_owner_only_dir(path).map_err(store_error(format!( + let problem = check(path).map_err(store_error(format!( "cannot open {what} {}", path.display() )))?; diff --git a/nvx-rs/src/host/template/store/capture.rs b/nvx-rs/src/host/template/store/capture.rs new file mode 100644 index 000000000..f2c1d68e8 --- /dev/null +++ b/nvx-rs/src/host/template/store/capture.rs @@ -0,0 +1,1473 @@ +//! The captures of templates, and their publication in a store: [`Store::create_template`], +//! [`Store::refresh_template`], and [`Store::recover_captures`]; see +//! [the store's documentation](Store#captures-and-publication). + +#[cfg(test)] +mod tests; + +use std::ffi::{OsStr, OsString}; +use std::fs; +use std::io; +use std::path::{Path, PathBuf}; +use std::thread; +use std::time::{Duration, Instant, SystemTime}; + +use serde::{Deserialize, Serialize}; + +use super::{ + GENERATIONS_NAME, Generation, Index, KEY_NAME, MAX_GENERATIONS, SNAPSHOT_NAME, STORE_FORMAT, + StorageGeneration, Store, entries, read_private, read_record, refuse_problem, store_error, + take_lock, write_record, +}; +use crate::activation::StoreGeneration; +use crate::error::{Error, ErrorCode, ErrorKind, Result}; +use crate::host::artifacts::{self, Artifacts}; +use crate::host::instance::generate_capability; +use crate::host::platform; +use crate::host::sandbox::distro::{CheckedDistro, DistroBinding, DistroFormat}; +use crate::host::sandbox::{LayerRole, SandboxLayer, SandboxPlan}; +use crate::host::seal::FileSeal; +use crate::host::store::files::{create_new, remove_if_present, sync_dir, write_private}; +use crate::host::store::{ + JOURNAL_NAME, LOCK_NAME, LOG_NAME, LaunchId, LockedStore, OUTCOME_NAME, Recovery, SOCKET_NAME, + SandboxStore, +}; +use crate::host::supervisor::{self, VmmCommand}; +use crate::host::template::eligibility::HostTime; +use crate::host::template::files::{open_regular, read_at_most}; +use crate::host::template::key::{CaptureClass, ClassIdentity, KeyIdentity, TemplateKey}; +use crate::host::template::manifest::{MANIFEST_NAME, TemplateManifest, TimeContract}; +use crate::host::template::plan::{SCRATCH_NAME, TemplatePlan, TemplateResources}; + +/// Name of the directory of a store's captures, in which each capture class has a slot. +pub const CAPTURES_NAME: &str = "captures"; +/// Name of the transaction directory of a capture, in the slot of its class, which is also the +/// sandbox store of the capture's VM until the VM has ended. +pub const TRANSACTION_NAME: &str = "transaction"; +/// Name of the record of a capture, in its transaction and then in its generation's payload. +pub const CAPTURE_RECORD_NAME: &str = "capture.json"; + +/// Name of the saved state of a snapshot. +const STATE_NAME: &str = "state.bin"; +/// Name of the RAM of a snapshot. +const MEMORY_NAME: &str = "memory.bin"; +/// The artifacts of a platform-tier snapshot, in the order of their names: its manifest, its RAM, +/// and its saved state. +const SNAPSHOT_ARTIFACTS: [&str; 3] = [MANIFEST_NAME, MEMORY_NAME, STATE_NAME]; + +/// Prefix of the name of the file that backs the RAM of a capture's VM, which OpenVMM creates +/// beside the snapshot's destination, and removes as the VM ends. +const MEMORY_FILE_PREFIX: &str = ".openvmm-microvm-memory-"; +/// Prefix of the names of the directories in which OpenVMM stages the snapshot, beside its +/// destination, and which it renames to the destination once the snapshot is complete: +/// `.snapshot.staging-`. +const STAGING_PREFIX: &str = ".snapshot.staging-"; + +/// Hexadecimal digits of the identity of a capture class that name its slot: few enough that the +/// path of the control socket of a capture's VMM fits a Unix socket's. +const SLOT_NAME_LEN: usize = 16; + +/// Most bytes of a Unix socket's path on Linux, `sun_path`, with its terminating NUL. +const SOCKET_PATH_LIMIT: usize = 108; + +/// Longest record of a capture, in bytes. +const MAX_CAPTURE_RECORD_LEN: u64 = 4 * 1024; + +/// Longest outcome report of a capture's VM that the capture reads, in bytes. +const MAX_REPORT_LEN: u64 = 1024 * 1024; + +/// Interval at which a rename that Windows refuses, while another process holds a file below the +/// renamed directory open, is tried again, until the capture's deadline. +const RENAME_RETRY_INTERVAL: Duration = Duration::from_millis(25); + +/// The record of a capture, in messages. +const RECORD_DESCRIPTION: &str = "record of the capture"; +/// A capture's transaction, in messages. +const TRANSACTION_WHAT: &str = "the transaction of the capture"; +/// The directory of a store's captures, in messages. +const CAPTURES_WHAT: &str = "the captures of the template store"; + +/// What a template's capture takes: its class, the adapter's plan of its placeholder and its +/// network, the binding of the placeholder, and the artifacts that run its VM, which +/// [`Store::create_template`] and [`Store::refresh_template`] capture. +/// +/// The adapter's plan is lowered as [`TemplatePlan::new`] lowers it, with the processors and RAM +/// of the class, and a scratch of the class's scratch length; its upper is no part of the +/// template. The class and the plan must describe one VM: a class whose processors are all online +/// at boot, a placeholder whose binding records the class's `distro` length, and a network of the +/// plan that lowers to the OpenVMM arguments of the class's network. +#[derive(Debug, Clone)] +pub struct Capture { + /// The class of the capture, which names its slot, and from which the capture takes its + /// backend, its CPU profile, its processors, its RAM, its scratch length, and the runtime + /// identity of its launch. + pub class: CaptureClass, + /// The adapter's plan, which [`SandboxPlan::adapter`] makes, of the placeholder `distro` and + /// the template's network. + pub adapter: SandboxPlan, + /// The binding of the placeholder, which the capture checks before its VM starts. + pub binding: DistroBinding, + /// The revision of the distro policy that the caller accepts, such as + /// [`DISTRO_POLICY_REVISION`](crate::host::sandbox::distro::DISTRO_POLICY_REVISION), which + /// [`DistroBinding::check`] requires of the binding's verdict. + pub policy_revision: u32, + /// The OpenVMM executable, the guest kernel, and the initramfs, whose identities the class's + /// runtime holds, which relative paths name from the current directory. + pub artifacts: Artifacts, +} + +/// How [`Store::create_template`] served a capture class: with a generation that the store held +/// already, or with one that it captured and published. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Created { + /// A generation of the class that the store held already. + Existing(Generation), + /// A generation that a capture made, and that the store published. + Captured(Generation), +} + +impl Created { + /// Returns the generation. + pub fn generation(&self) -> &Generation { + match self { + Self::Existing(generation) | Self::Captured(generation) => generation, + } + } + + /// Returns the generation, consuming this. + pub fn into_generation(self) -> Generation { + match self { + Self::Existing(generation) | Self::Captured(generation) => generation, + } + } +} + +/// Syncs a directory, as [`sync_dir`] does. Captures take it as a parameter, so that tests +/// observe the order of their syncs, and fail them. +type SyncDir<'a> = &'a dyn Fn(&Path) -> Result<()>; + +/// Runs the VM of a capture in the slot whose lock the capture holds, as [`run_supervised`] does, +/// and returns once the VM has ended, and nothing of its launch runs. Tests run it in their own +/// process instead. +type RunVm<'a> = &'a dyn Fn(&mut LockedStore, &Vm<'_>) -> Result<()>; + +/// A point of a capture at which a test fails it, before the step that follows the point. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Point { + /// The transaction holds the record of the capture and its scratch, and the VM has not + /// started. + Prepared, + /// The VM has ended, and nothing that it left is checked yet. + Exited, + /// The transaction is checked: it holds the template's key, durably. + Checked, + /// The envelope is renamed into the store, and the index does not list it yet. + Renamed, + /// The index lists the new generation. + Indexed, +} + +/// The seams of a capture, through which tests observe and interrupt it. +struct Hooks<'a> { + /// Syncs a directory. + sync: SyncDir<'a>, + /// Runs the capture's VM. + run: RunVm<'a>, + /// Fails the capture at a point, as a test asks. + interrupt: &'a dyn Fn(Point) -> Result<()>, + /// Whether a failure leaves the capture's files as they are, as a crash of the host would, + /// rather than retiring its transaction: in tests that crash captures. + crash: bool, +} + +impl Hooks<'static> { + /// Returns the seams of every capture outside tests. + fn supervised() -> Self { + Self { + sync: &sync_dir, + run: &run_supervised, + interrupt: &uninterrupted, + crash: false, + } + } +} + +impl Hooks<'_> { + /// Fails the capture at `point` if a test asks. + fn at(&self, point: Point) -> Result<()> { + (self.interrupt)(point) + } +} + +/// Lets a capture pass every point, as captures outside tests do. +fn uninterrupted(_: Point) -> Result<()> { + Ok(()) +} + +/// What a capture takes of its inputs once it has checked them, before it opens anything. +struct Inputs { + /// The identity of the class. + class: ClassIdentity, + /// The template's processors and RAM, from the class. + resources: TemplateResources, + /// The artifacts, as absolute paths. + artifacts: Artifacts, +} + +impl Inputs { + /// Checks the inputs of `capture`, without reading or opening anything, and returns what the + /// capture takes of them. + /// + /// Fails with [`ErrorKind::Unsupported`] as [`CaptureClass::identity`] fails, and for a class + /// whose processors are not all online at boot, a placeholder whose binding records another + /// length than the class's `distro`, and a plan whose network lowers to other arguments than + /// the class's; and with [`ErrorKind::Unavailable`] for an artifact whose path cannot be + /// resolved. + fn check(capture: &Capture) -> Result { + let class = &capture.class; + let identity = class.identity()?; + if class.boot_processors != class.processors { + return Err(Error::unsupported(format!( + "a template boots with all of its processors online, but its class has {} \ + processors, {} online at boot", + class.processors, class.boot_processors + ))); + } + let length = capture.binding.image().length(); + if length != class.distro.length { + return Err(Error::unsupported(format!( + "the placeholder {} is {length} bytes long, as its binding records it, but the \ + class's distro is {} bytes long", + capture.adapter.distro().path().display(), + class.distro.length + ))); + } + let planned = match capture.adapter.network() { + None => Vec::new(), + Some(network) => network.openvmm_arguments()?, + }; + let requested = match &class.network { + None => Vec::new(), + Some(network) => network.openvmm_arguments()?, + }; + if planned != requested { + return Err(Error::unsupported(format!( + "the adapter's plan attaches the network {planned:?}, but the class requests \ + {requested:?}" + ))); + } + let given = &capture.artifacts; + Ok(Self { + class: identity, + resources: TemplateResources { + processors: class.processors, + memory: class.memory, + memory_capacity: class.memory_capacity, + }, + artifacts: Artifacts::new( + artifacts::absolute(&given.openvmm)?, + artifacts::absolute(&given.kernel)?, + artifacts::absolute(&given.initrd)?, + ), + }) + } +} + +/// The VM of a capture, as the capture's source runs it. +struct Vm<'a> { + /// The inputs of the capture. + capture: &'a Capture, + /// The plan of the template. + template: &'a TemplatePlan, + /// The OpenVMM executable, the guest kernel, and the initramfs, as absolute paths. + artifacts: &'a Artifacts, + /// When the capture must have ended. + deadline: Instant, +} + +impl Vm<'_> { + /// Returns the command of the capture's VMM, run in the capture's transaction `dir`, whose + /// control console listens on `endpoint`, and whose outcome report goes to `report`. + /// + /// Its arguments are the template's, as [`TemplatePlan::openvmm_arguments`] renders them, + /// then the backend, the CPU profile, the kernel and the initramfs, the template's kernel + /// command line, a boot console that discards its output, the control console, and the + /// outcome report. + fn command(&self, endpoint: &str, report: &Path, dir: &Path) -> Result> { + let class = &self.capture.class; + let mut arguments: Vec = self + .template + .openvmm_arguments()? + .into_iter() + .map(OsString::from) + .collect(); + let listen = format!("listen={endpoint}"); + for argument in [ + OsStr::new("--single-process"), + OsStr::new("--hypervisor"), + OsStr::new(class.hypervisor.as_str()), + OsStr::new("--cpu-profile"), + OsStr::new(&class.cpu_profile), + OsStr::new("--kernel"), + self.artifacts.kernel.as_os_str(), + OsStr::new("--initrd"), + self.artifacts.initrd.as_os_str(), + OsStr::new("--cmdline"), + OsStr::new(self.template.kernel_command_line()), + OsStr::new("--virtio-console"), + OsStr::new("none"), + OsStr::new("--microvm-control-console"), + OsStr::new(&listen), + OsStr::new("--microvm-control-auth-stdin"), + OsStr::new("--microvm-report"), + report.as_os_str(), + ] { + arguments.push(argument.to_owned()); + } + Ok(VmmCommand::new(&self.artifacts.openvmm, arguments, dir)) + } +} + +/// Runs the VM of a capture through the supervisor, from the sandbox store of `locked`, the +/// capture's transaction, as a launch of its own, until its VMM exits, or until the capture's +/// deadline, at which it ends the VMM, and then ends the launch. +/// +/// The capture opens no control session, and waits for no Ready: its guest captures itself, and +/// OpenVMM exits once it has published the snapshot, or once the capture failed. Fails with +/// [`ErrorKind::TimedOut`] if the deadline passes first, and with +/// [`ErrorCode::VmmIndeterminate`] if the launch cannot be ended, whose journal then remains for +/// the recovery of the slot; and as [`supervisor::launch`] fails. +fn run_supervised(locked: &mut LockedStore, vm: &Vm<'_>) -> Result<()> { + let report = locked.outcome_path(); + let dir = locked.dir().to_owned(); + let capability = generate_capability()?; + // A slot holds no lease, so each capture reserves its launch itself. + let launch = LaunchId::generate()?; + let starting = supervisor::launch( + locked, + &launch, + &vm.capture.class.runtime, + &capability, + |endpoint| vm.command(endpoint, &report, &dir), + )?; + let exited = starting.process().wait(vm.deadline).map_err(|error| { + Error::io(format!( + "cannot wait for the VMM of the capture in {}", + dir.display() + )) + .with_source(error) + }); + if let Recovery::Indeterminate(reason) = starting.abandon()? { + return Err(indeterminate(&dir, &reason)); + } + if !exited? { + return Err(Error::timed_out(format!( + "the VM of the capture in {} did not end before the capture's deadline, and was torn \ + down", + dir.display() + ))); + } + Ok(()) +} + +/// Returns the error of a capture whose VMM, in the slot `dir`, may still run, for `reason`, +/// [`ErrorCode::VmmIndeterminate`]. +fn indeterminate(dir: &Path, reason: &str) -> Error { + Error::new( + ErrorCode::VmmIndeterminate, + format!( + "the VMM of the capture in {} may still run, and its launch remains for recovery: \ + {reason}", + dir.display() + ), + ) +} + +impl Store { + /// Returns a generation of the class of `capture` that the store holds, or captures one and + /// publishes it, waiting for the capture's slot and the store's lock until `deadline`, which + /// bounds the capture too. + /// + /// Captures of one class are serialized by the lock of the class's slot, under which a + /// creation first ends a capture of the class that a crash interrupted, and completes the + /// store's publications that a crash interrupted, as + /// [`recover_captures`](Self::recover_captures) does. It then calls `now` for the host's time, + /// once, and looks for a generation: the latest that the store indexes of the class, of the + /// key of the class's latest generation, that no tombstone quarantines, and that is eligible + /// at that time, as [`Generation::eligibility`] decides it, as [`select`](Self::select) + /// chooses among the generations of a key. Without one, it captures one, as + /// [`refresh_template`](Self::refresh_template) does. Since `now` is called once the creation + /// holds the slot, a generation that a racing creation captured while this one waited is + /// eligible at that time, so creations that race for one class make one generation between + /// them. [`HostTime::now`] samples this host's time; a caller that keeps a margin, as before + /// the window of a generation closes, returns a later time. + /// + /// Fails as `now` fails, and otherwise as [`refresh_template`](Self::refresh_template) does. + pub fn create_template( + &self, + capture: &Capture, + now: impl Fn() -> Result, + deadline: Instant, + ) -> Result { + self.capture(capture, Some(&now), deadline, &Hooks::supervised()) + } + + /// Captures a template of the class of `capture`, and publishes it as the store's next + /// generation, which later selections of its key take before the older ones, waiting for the + /// capture's slot and the store's lock until `deadline`, which bounds the capture too. + /// + /// The capture cold-boots the template's VM through [`supervisor::launch`], from its + /// transaction, in the slot of its class, and waits for it to exit: its guest captures itself, + /// and OpenVMM publishes the snapshot into the capture's transaction and exits. The capture + /// checks the files before the VM starts, and what the VM left once it has exited, publishes + /// the transaction in two commits, and leaves the older generations, and their leases, + /// untouched; see [the store's documentation](Store#captures-and-publication). + /// + /// Fails, and publishes nothing, but a transaction that it checked, which the next recovery + /// of the slot publishes: + /// + /// - with [`ErrorKind::Unsupported`] for inputs that [`Capture`] does not admit, as + /// [`TemplatePlan::new`] refuses them, for a store whose path is too long for the control + /// socket of a capture's VMM, on Linux, and for a store that holds + /// [`MAX_GENERATIONS`] generations; + /// - with [`ErrorKind::Unavailable`] for a placeholder that its binding's check refuses, as + /// [`DistroBinding::check`] fails, that is a link, or on Windows any other reparse point, + /// or that is not a regular file, and for one that belongs to another user than the current + /// one or an administrator, or that anyone else may change; + /// - with [`ErrorKind::Exited`] if the VM ended without a capture, as when its guest failed, + /// or OpenVMM did, even after it published the snapshot; + /// - with [`ErrorKind::TimedOut`] if the deadline passes before the VM ends, which the + /// capture then tears down, or before the capture takes its slot's lock or the store's; + /// - with [`ErrorCode::VmmIndeterminate`] if the capture cannot prove that the VMM of a + /// capture of the class, its own or an interrupted one, no longer runs: the slot then keeps + /// the launch and its transaction for a later recovery; + /// - with [`ErrorKind::Io`] for files of the slot, of the transaction, or of the store that + /// are links, that are not private to the current user, or that cannot be created, read, + /// synced, or renamed, and for a capture that did not leave what its class requests: a + /// snapshot whose manifest [`TemplateManifest::read`] refuses, that is not the class's, as + /// [`TemplateKey::new`] checks it, or whose artifacts are not exactly its manifest, RAM, and + /// saved state, each a regular file of the length that the manifest records. + pub fn refresh_template(&self, capture: &Capture, deadline: Instant) -> Result { + self.capture(capture, None, deadline, &Hooks::supervised()) + .map(Created::into_generation) + } + + /// Ends the captures that a crash interrupted, waiting for the lock of each slot and for the + /// store's until `deadline`, and returns the generations that it published, with those whose + /// publications it completed, in the order of their store generations. + /// + /// In each slot, it ends the launch of the capture's VMM, as [`LockedStore::recover`] ends + /// it, then publishes the capture's transaction if the capture had checked it, and retires it + /// otherwise. It then completes a publication that a crash interrupted once it had renamed + /// its envelope into the store, and drops from the index the generations whose payloads a + /// crash lost. A capture recovers the slot of its class, and the store's publications, the + /// same way, so an adapter recovers the other slots once, as it starts. + /// + /// Fails with [`ErrorCode::ForeignFiles`] if the directory of the captures holds entries that + /// are not slots, and otherwise as [`refresh_template`](Self::refresh_template) fails. + pub fn recover_captures(&self, deadline: Instant) -> Result> { + self.recover_with(deadline, &Hooks::supervised()) + } + + /// Serves `capture` as [`create_template`](Self::create_template) does with the host's time + /// that `now` returns, or, without it, as [`refresh_template`](Self::refresh_template) does, + /// through `hooks`. + fn capture( + &self, + capture: &Capture, + now: Option<&dyn Fn() -> Result>, + deadline: Instant, + hooks: &Hooks<'_>, + ) -> Result { + let inputs = Inputs::check(capture)?; + let slot = self.open_slot(inputs.class)?; + let locked = lock_slot(&slot, deadline)?; + self.recover_slot(&locked, deadline, hooks)?; + // The index then lists every generation that a publication renamed into the store, and + // none whose payload a crash lost, for the lookup and the store's capacity alike. + self.recover_publications(deadline, hooks)?; + if let Some(now) = now + && let Some(existing) = self.existing(inputs.class, &now()?)? + { + // The commit that listed it may have been interrupted before it made the index, or + // the store's own entry in its parent, durable. + self.sync_store(hooks)?; + return Ok(Created::Existing(existing)); + } + if self.index()?.generations.len() >= MAX_GENERATIONS { + return Err(self.full()); + } + let transaction = locked.dir().join(TRANSACTION_NAME); + let template = TemplatePlan::new( + &capture.adapter, + &capture.binding, + capture.class.scratch.length, + inputs.resources, + &transaction, + StorageGeneration::generate()?, + )?; + let envelope = + match self.check_capture(capture, &inputs, &template, &locked, deadline, hooks) { + Ok(envelope) => envelope, + Err(error) => { + // A transaction stays while its sandbox store records a launch, whose VMM may + // still write it, for the slot's recovery to retire once the VMM has ended; + // and so does one that a retirement interrupted, as a crash would leave it. + if !hooks.crash && !records_launch(&transaction) { + let _ = keep_diagnostics(&transaction, locked.dir()); + let _ = retire(&transaction); + } + return Err(error); + } + }; + hooks.at(Point::Checked)?; + self.publish(&transaction, &envelope, deadline, hooks) + .map(|(published, _)| Created::Captured(published)) + } + + /// Runs the capture of `template`, with the inputs `capture`, checked as `inputs`, in the slot + /// whose lock `slot` holds, through `hooks`, until the capture checks its transaction, or until + /// `deadline`, and returns its envelope. + /// + /// The transaction is the sandbox store of the capture's VM: OpenVMM requires the control + /// socket of a VM that captures a snapshot, on Linux, beside the snapshot's destination, and a + /// sandbox store's VM listens in the store's directory. + fn check_capture( + &self, + capture: &Capture, + inputs: &Inputs, + template: &TemplatePlan, + slot: &LockedStore, + deadline: Instant, + hooks: &Hooks<'_>, + ) -> Result { + let transaction = template.transaction(); + // The slot keeps the log and outcome report of its class's latest capture, this one, so + // that the capture never reads an earlier VM's report for its own. + for name in [LOG_NAME, OUTCOME_NAME] { + remove_if_present(&slot.dir().join(name))?; + } + let mut launch = SandboxStore::create(transaction)?.lock(Some(deadline))?; + write_record( + &transaction.join(CAPTURE_RECORD_NAME), + &CaptureRecord::new(self, inputs.class, template.generation()), + &format!("the {RECORD_DESCRIPTION}"), + )?; + let scratch = create_scratch(template.scratch_path(), template.scratch_length())?; + hooks.at(Point::Prepared)?; + let placeholder = check_files(capture, template, scratch)?; + let vm = Vm { + capture, + template, + artifacts: &inputs.artifacts, + deadline, + }; + (hooks.run)(&mut launch, &vm)?; + drop(placeholder); + keep_diagnostics(transaction, slot.dir())?; + hooks.at(Point::Exited)?; + check_exit(&slot.outcome_path())?; + let snapshot = template.destination(); + let manifest = check_snapshot(snapshot)?; + let key = TemplateKey::new(&capture.class, &manifest)?; + // The host closes its handles before it publishes: the transaction's lock goes, and its + // file with what else the capture leaves. + drop(launch); + clean_transaction(transaction)?; + write_private(&transaction.join(KEY_NAME), &key.to_bytes())?; + // The checked transaction must survive a crash, for the slot's recovery to publish it, + // and so must the store, whose own entry in its parent its creation does not sync. + for dir in [transaction, slot.dir(), &self.captures_dir()] { + (hooks.sync)(dir)?; + } + self.sync_store(hooks)?; + Ok(Envelope::new( + &key, + inputs.class, + template.generation(), + &manifest, + )) + } + + /// Publishes the checked transaction `transaction`, whose envelope is `envelope`, as the + /// store's next generation, under the store's lock, which it waits for until `deadline`, + /// through `hooks`, and returns the generation, with those of the publications that it + /// completed first. + /// + /// It first recovers the index, as [`recover_index`](Self::recover_index) does. Then it + /// renames the transaction to the payload of the next store generation, which it never + /// replaces, makes the rename durable, and commits the index, which lists the generation with + /// its full key, as [`commit_index`](Self::commit_index) does. + fn publish( + &self, + transaction: &Path, + envelope: &Envelope, + deadline: Instant, + hooks: &Hooks<'_>, + ) -> Result<(Generation, Vec)> { + let _lock = take_lock(&self.dir, Some(deadline))?; + let mut index = self.index()?; + let (completed, changed) = self.recover_index(&mut index, hooks)?; + if index.generations.len() >= MAX_GENERATIONS { + if changed { + self.commit_index(&index, hooks)?; + } + return Err(self.full()); + } + let generation = index.next; + rename_until(transaction, &self.payload_dir(generation), deadline)?; + (hooks.sync)(&self.dir.join(GENERATIONS_NAME))?; + if let Some(slot) = transaction.parent() { + (hooks.sync)(slot)?; + } + hooks.at(Point::Renamed)?; + let published = envelope.generation(generation); + index.generations.push(published.clone()); + index.next = following(generation)?; + self.commit_index(&index, hooks)?; + hooks.at(Point::Indexed)?; + Ok((published, completed)) + } + + /// Writes `index`, which the store's lock holds, and makes it durable through `hooks`, as + /// [`sync_store`](Self::sync_store) does, so that a crash of the host loses neither a + /// generation that it lists nor the store. + fn commit_index(&self, index: &Index, hooks: &Hooks<'_>) -> Result<()> { + self.write_index(index)?; + self.sync_store(hooks) + } + + /// Syncs the store's directory, and then its parent, through `hooks`: [`Store::create`] + /// syncs neither, so the store's own entry in its parent may not be durable yet. + fn sync_store(&self, hooks: &Hooks<'_>) -> Result<()> { + (hooks.sync)(&self.dir)?; + if let Some(parent) = self.dir.parent() { + (hooks.sync)(parent)?; + } + Ok(()) + } + + /// Completes the store's publications that a crash interrupted, and drops from the index the + /// generations whose payloads a crash lost, as [`recover_index`](Self::recover_index) does, + /// under the store's lock, which it waits for until `deadline`, through `hooks`. Commits the + /// index if that changed it, and returns the generations that it completed. + fn recover_publications( + &self, + deadline: Instant, + hooks: &Hooks<'_>, + ) -> Result> { + let _lock = take_lock(&self.dir, Some(deadline))?; + let mut index = self.index()?; + let (completed, changed) = self.recover_index(&mut index, hooks)?; + if changed { + self.commit_index(&index, hooks)?; + } + Ok(completed) + } + + /// Recovers `index`, which the store's lock holds, from the publications and retirements that + /// a crash interrupted, through `hooks`, and returns the generations that it indexed, and + /// whether it changed the index, which the caller writes. + /// + /// A generation whose payload is gone was lost in a crash, as one that a publication renamed + /// on a host that cannot flush a directory, or whose retirement removed its payload before it + /// wrote the index: the index first drops it, so that it takes none of the store's capacity. + /// Its tombstone, if any, stays for its retirement to remove. A publication renames its + /// envelope to the payload of the index's next store generation before it writes the index, + /// so a payload there is one whose publication was interrupted: the index then takes it, as + /// the publication would have, once the recovery has made the rename durable, as the + /// publication would have too: it syncs the payload, the directory of the payloads, and the + /// slot that the envelope left. + fn recover_index( + &self, + index: &mut Index, + hooks: &Hooks<'_>, + ) -> Result<(Vec, bool)> { + self.check_generations_dir()?; + let listed = index.generations.len(); + let mut kept = Vec::with_capacity(listed); + for generation in index.generations.drain(..) { + match fs::symlink_metadata(self.payload_dir(generation.generation)) { + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + _ => kept.push(generation), + } + } + index.generations = kept; + let dropped = index.generations.len() != listed; + let mut completed = Vec::new(); + loop { + let payload = self.payload_dir(index.next); + match fs::symlink_metadata(&payload) { + Err(error) if error.kind() == io::ErrorKind::NotFound => break, + Err(error) => { + return Err(store_error(format!( + "cannot open the payload of the generation {}", + payload.display() + ))(error)); + } + Ok(_) => {} + } + if index.generations.len() >= MAX_GENERATIONS { + return Err(self.full()); + } + let generation = self.read_envelope(&payload)?.generation(index.next); + (hooks.sync)(&payload)?; + index.generations.push(generation.clone()); + index.next = following(index.next)?; + completed.push(generation); + } + if !completed.is_empty() { + (hooks.sync)(&self.dir.join(GENERATIONS_NAME))?; + let mut slots: Vec = completed + .iter() + .map(|generation| slot_name(generation.class)) + .collect(); + slots.sort(); + slots.dedup(); + for name in slots { + let slot = self.captures_dir().join(name); + match fs::symlink_metadata(&slot) { + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + Err(error) => { + return Err(store_error(format!( + "cannot open the slot {}", + slot.display() + ))(error)); + } + Ok(metadata) if metadata.is_dir() => (hooks.sync)(&slot)?, + Ok(_) => {} + } + } + } + let changed = dropped || !completed.is_empty(); + Ok((completed, changed)) + } + + /// Reads the envelope of a checked capture in the directory `dir`, its transaction or its + /// payload, and returns what the index lists of it: the record of the capture, the template's + /// key, and the manifest of its snapshot, whose artifacts it checks again by their kinds and + /// lengths. + fn read_envelope(&self, dir: &Path) -> Result { + let what = "the envelope of the capture"; + let problem = platform::check_owner_only_dir(dir) + .map_err(store_error(format!("cannot open {what} {}", dir.display())))?; + refuse_problem(dir, what, problem)?; + let mut names = entries(dir)?; + names.sort(); + if names != [CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME] { + return Err(Error::io(format!( + "{what} {} holds {}, not exactly the {RECORD_DESCRIPTION}, its key, and its \ + snapshot", + dir.display(), + names.join(", ") + ))); + } + let path = dir.join(CAPTURE_RECORD_NAME); + let record = + read_record::(&path, RECORD_DESCRIPTION, MAX_CAPTURE_RECORD_LEN)? + .ok_or_else(|| { + Error::io(format!( + "the {RECORD_DESCRIPTION} {} is missing", + path.display() + )) + })?; + let malformed = |problem: &str| { + Error::io(format!( + "the {RECORD_DESCRIPTION} {} is malformed: {problem}", + path.display() + )) + }; + if record.store != self.id.to_string() { + return Err(malformed(&format!( + "it is a capture of the template store {}", + record.store + ))); + } + let class: ClassIdentity = record + .class + .parse() + .map_err(|error: Error| malformed(error.message()))?; + let storage: StorageGeneration = record + .storage_generation + .parse() + .map_err(|error: Error| malformed(error.message()))?; + let path = dir.join(KEY_NAME); + let description = "template key"; + let bytes = read_private(&path, description, TemplateKey::MAX_LEN as u64)? + .ok_or_else(|| Error::io(format!("the {description} {} is missing", path.display())))?; + let key = TemplateKey::from_bytes(&bytes).map_err(|error| { + Error::new( + error.code(), + format!("cannot read {}: {}", path.display(), error.message()), + ) + })?; + let manifest = check_snapshot(&dir.join(SNAPSHOT_NAME))?; + Ok(Envelope::new(&key, class, storage, &manifest)) + } + + /// Ends the capture that a crash interrupted in the slot whose lock `slot` holds, if any, + /// through `hooks`: ends its VMM, which the transaction's sandbox store records, as + /// [`LockedStore::recover`] does, and then publishes its transaction if the capture checked + /// it, waiting for the store's lock until `deadline`, and retires it otherwise, once it has + /// moved the VMM's log and outcome report to the slot. Returns the generations that it + /// published: those of the publications that it completed first, as + /// [`publish`](Self::publish) does, and then the transaction's. + fn recover_slot( + &self, + slot: &LockedStore, + deadline: Instant, + hooks: &Hooks<'_>, + ) -> Result> { + let transaction = slot.dir().join(TRANSACTION_NAME); + match platform::check_owner_only_dir(&transaction) { + Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(Vec::new()), + checked => { + let problem = checked.map_err(store_error(format!( + "cannot open {TRANSACTION_WHAT} {}", + transaction.display() + )))?; + refuse_problem(&transaction, TRANSACTION_WHAT, problem)?; + } + } + end_launch(&transaction, deadline)?; + let key = transaction.join(KEY_NAME); + match fs::symlink_metadata(&key) { + Err(error) if error.kind() == io::ErrorKind::NotFound => { + keep_diagnostics(&transaction, slot.dir())?; + retire(&transaction)?; + Ok(Vec::new()) + } + Err(error) => Err(store_error(format!("cannot open {}", key.display()))(error)), + Ok(_) => { + let envelope = self.read_envelope(&transaction)?; + // The crash may have come before the capture synced the transaction that it + // checked, whose entries must be durable before the rename publishes them. + (hooks.sync)(&transaction)?; + let (published, mut completed) = + self.publish(&transaction, &envelope, deadline, hooks)?; + completed.push(published); + Ok(completed) + } + } + } + + /// Recovers the store's captures as [`recover_captures`](Self::recover_captures) does, + /// through `hooks`. + fn recover_with(&self, deadline: Instant, hooks: &Hooks<'_>) -> Result> { + let captures = self.captures_dir(); + let mut published = Vec::new(); + let exists = match platform::check_owner_only_dir(&captures) { + Err(error) if error.kind() == io::ErrorKind::NotFound => false, + checked => { + let problem = checked.map_err(store_error(format!( + "cannot open {CAPTURES_WHAT} {}", + captures.display() + )))?; + refuse_problem(&captures, CAPTURES_WHAT, problem)?; + true + } + }; + if exists { + let mut slots = entries(&captures)?; + slots.sort(); + let foreign: Vec<&str> = slots + .iter() + .map(String::as_str) + .filter(|name| !is_slot_name(name)) + .collect(); + if !foreign.is_empty() { + return Err(Error::new( + ErrorCode::ForeignFiles, + format!( + "{CAPTURES_WHAT} {} holds entries that are not the store's: {}", + captures.display(), + foreign.join(", ") + ), + )); + } + for name in slots { + let slot = SandboxStore::open(captures.join(name))?; + let locked = lock_slot(&slot, deadline)?; + published.extend(self.recover_slot(&locked, deadline, hooks)?); + } + } + published.extend(self.recover_publications(deadline, hooks)?); + published.sort_by_key(Generation::generation); + Ok(published) + } + + /// Returns the generation with which a creation serves the class `class` at the host's time + /// `now`, if the store holds one: the latest of the class, of the key of the class's latest + /// generation, that no tombstone quarantines, and that is eligible at `now`. + fn existing(&self, class: ClassIdentity, now: &HostTime) -> Result> { + let index = self.index()?; + let Some(latest) = index + .generations + .iter() + .rev() + .find(|generation| generation.class == class) + else { + return Ok(None); + }; + let key = latest.key; + let tombstoned = self.tombstones()?; + Ok(index + .generations + .iter() + .rev() + .find(|generation| { + generation.class == class + && generation.key == key + && !tombstoned.contains(&generation.generation) + && generation.eligibility(now).is_eligible() + }) + .cloned()) + } + + /// Creates the directory of the store's captures if needed, and the slot of the class + /// `class` in it, and returns the slot's store. + fn open_slot(&self, class: ClassIdentity) -> Result { + let captures = self.captures_dir(); + create_private_dir(&captures, CAPTURES_WHAT)?; + let slot = SandboxStore::create(captures.join(slot_name(class)))?; + // A capture's VM listens in its transaction, the sandbox store from which it launches. + let socket = slot.dir().join(TRANSACTION_NAME).join(SOCKET_NAME); + if cfg!(target_os = "linux") && socket.as_os_str().len() >= SOCKET_PATH_LIMIT { + return Err(Error::unsupported(format!( + "the template store {} is too deep for the control socket of a capture's VMM, \ + {}, whose path must be shorter than {SOCKET_PATH_LIMIT} bytes", + self.dir.display(), + socket.display() + ))); + } + Ok(slot) + } + + /// Returns the path of the directory of the store's captures. + fn captures_dir(&self) -> PathBuf { + self.dir.join(CAPTURES_NAME) + } + + /// Returns the error of a store that holds [`MAX_GENERATIONS`] generations, which takes no + /// other. + fn full(&self) -> Error { + Error::unsupported(format!( + "the template store {} holds {MAX_GENERATIONS} generations, the most that a store \ + holds: retire some before a capture publishes another", + self.dir.display() + )) + } +} + +/// Takes the lock of the slot `slot`, waiting for it until `deadline`, as another capture of its +/// class may hold it. +fn lock_slot(slot: &SandboxStore, deadline: Instant) -> Result { + slot.lock(Some(deadline)).map_err(|error| { + if error.code() == ErrorCode::SandboxBusy { + Error::new( + ErrorKind::TimedOut, + format!( + "another capture of the class of the slot {} is still in progress", + slot.dir().display() + ), + ) + } else { + error + } + }) +} + +/// Returns the name of the slot of the class `class`: the first [`SLOT_NAME_LEN`] digits of its +/// identity. +fn slot_name(class: ClassIdentity) -> String { + class.to_string()[..SLOT_NAME_LEN].to_owned() +} + +/// Returns whether `name` is the name of a slot. +fn is_slot_name(name: &str) -> bool { + name.len() == SLOT_NAME_LEN + && name + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +/// Creates the directory `path`, which `what` names, private to the current user, unless it +/// exists, which must be private already. +fn create_private_dir(path: &Path, what: &str) -> Result<()> { + let problem = platform::create_owner_only_dir(path).map_err(store_error(format!( + "cannot create {what} {}", + path.display() + )))?; + refuse_problem(path, what, problem) +} + +/// Creates the dormant scratch device's file at `path`, private to the current user: a sparse +/// file of `length` bytes, and returns its identity, as [`identity_of`] returns it. +fn create_scratch(path: &Path, length: u64) -> Result<(u64, u128)> { + let failed = |error| { + Error::io(format!( + "cannot create the scratch of the capture {}", + path.display() + )) + .with_source(error) + }; + let file = create_new(path).map_err(failed)?; + platform::make_sparse(&file).map_err(failed)?; + file.set_len(length).map_err(failed)?; + identity_of(&file).map_err(failed) +} + +/// Returns the identity of the file that `file` holds open, as its [`FileSeal`] records it: its +/// device or volume, and its file ID, the 128-bit one of `FILE_ID_INFO` on Windows, which a file +/// system such as ReFS keeps unique where the 64-bit file index is not. +fn identity_of(file: &fs::File) -> io::Result<(u64, u128)> { + let seal = FileSeal::of(file)?; + Ok((seal.device(), seal.file_id())) +} + +/// Checks the files of the capture of `template`, with the inputs `capture`, before its VM +/// starts: its transaction directory, its placeholder, and its scratch, which the capture +/// created with the identity `scratch`. Returns the placeholder that the check of its binding +/// opened, which the capture holds, for reading alone, until its VM has ended. +/// +/// OpenVMM opens the block devices of a capture itself, at their paths, so the capture holds no +/// other handle of them as its VM starts, through which anything could write them. +fn check_files( + capture: &Capture, + template: &TemplatePlan, + scratch: (u64, u128), +) -> Result { + let transaction = template.transaction(); + let problem = platform::check_owner_only_dir(transaction).map_err(store_error(format!( + "cannot open {TRANSACTION_WHAT} {}", + transaction.display() + )))?; + refuse_problem(transaction, TRANSACTION_WHAT, problem)?; + let placeholder = check_placeholder(capture, template)?; + check_scratch(template, scratch)?; + Ok(placeholder) +} + +/// Checks the placeholder of `template` against the binding of `capture`, as +/// [`DistroBinding::check`] does, and then at its path: a regular file, not a link, nor on +/// Windows any other reparse point, that is the file that the check found, that belongs to the +/// current user or to an administrator, and that no one else may change. Returns the file that +/// the binding's check opened. +fn check_placeholder(capture: &Capture, template: &TemplatePlan) -> Result { + let path = template.plan().distro().path(); + let refused = |problem: String| { + Error::unavailable(format!("the placeholder {} {problem}", path.display())) + }; + let text = path + .to_str() + .ok_or_else(|| refused("has a path that is not valid Unicode".to_owned()))?; + let binding = &capture.binding; + let layer = match binding.format() { + DistroFormat::Erofs => SandboxLayer::new(LayerRole::Distro, text, binding.identity()), + DistroFormat::Gpt => SandboxLayer::gpt(LayerRole::Distro, text, binding.identity()), + }; + let checked = binding.check(&layer, capture.policy_revision)?; + let file = open_regular(path).map_err(|error| refused(format!("cannot be used: {error}")))?; + let identity = + |file| identity_of(file).map_err(|error| refused(format!("cannot be identified: {error}"))); + if identity(&file)? != identity(checked.file())? { + return Err(refused( + "changed while its binding was checked: its path names another file".to_owned(), + )); + } + if let Some(problem) = platform::foreign_writer_problem(&file) + .map_err(|error| refused(format!("cannot be checked: {error}")))? + { + return Err(refused(problem)); + } + Ok(checked) +} + +/// Checks the scratch of `template` at its path, which the capture created with the identity +/// `created`: a regular file of the template's scratch length, not a link, nor on Windows any +/// other reparse point, that is the file that the capture created, and that no one else may +/// change. The check's file is closed as it returns. +fn check_scratch(template: &TemplatePlan, created: (u64, u128)) -> Result<()> { + let path = template.scratch_path(); + let refused = |problem: String| { + Error::io(format!( + "the scratch of the capture {} {problem}", + path.display() + )) + }; + let file = open_regular(path).map_err(|error| refused(format!("cannot be used: {error}")))?; + let identity = + identity_of(&file).map_err(|error| refused(format!("cannot be identified: {error}")))?; + if identity != created { + return Err(refused( + "is not the file that the capture created".to_owned(), + )); + } + let length = file + .metadata() + .map_err(|error| refused(format!("cannot be examined: {error}")))? + .len(); + if length != template.scratch_length() { + return Err(refused(format!( + "is {length} bytes long, not the {} bytes of the template's scratch", + template.scratch_length() + ))); + } + if let Some(problem) = platform::foreign_writer_problem(&file) + .map_err(|error| refused(format!("cannot be checked: {error}")))? + { + return Err(refused(problem)); + } + Ok(()) +} + +/// The outcome report of a VM, as OpenVMM writes it, of which a capture reads the outcome. +#[derive(Deserialize)] +struct OutcomeReport { + /// The outcome of the VM. + outcome: Outcome, +} + +/// The outcome of a VM, as its outcome report records it. +#[derive(Deserialize)] +struct Outcome { + /// The operation of the VM: `capture` for a VM that OpenVMM captures. + operation: String, + /// The category of the outcome: `success` for a capture that OpenVMM published. + category: String, + /// The status code of the outcome: 0 for a capture that OpenVMM published. + status_code: i64, +} + +/// Checks the source's exit through the outcome report of the capture's VM at `path`: that +/// OpenVMM wrote one, for a capture that succeeded, as it does once it has published the snapshot +/// and torn the VM down, and before it exits. The launch removed the report of the slot's previous +/// VM, as [`LockedStore::begin_launch`] does, and OpenVMM publishes its report without replacing +/// a file, so the report is the VM's own. +fn check_exit(path: &Path) -> Result<()> { + let description = format!("the outcome report {}", path.display()); + let file = match open_regular(path) { + Ok(file) => file, + Err(error) if error.kind() == io::ErrorKind::NotFound => { + return Err(Error::exited(format!( + "the VM of the capture ended without {description}" + ))); + } + Err(error) => { + return Err(Error::io(format!("cannot read {description}")).with_source(error)); + } + }; + let bytes = read_at_most(file, MAX_REPORT_LEN, "an outcome report") + .map_err(|error| Error::io(format!("cannot read {description}")).with_source(error))?; + let report: OutcomeReport = serde_json::from_slice(&bytes) + .map_err(|error| Error::io(format!("{description} is malformed")).with_source(error))?; + let Outcome { + operation, + category, + status_code, + } = report.outcome; + if operation != "capture" || category != "success" || status_code != 0 { + return Err(Error::exited(format!( + "the VM of the capture ended without a capture: {description} records the outcome \ + {category} with status {status_code} of the operation {operation}" + ))); + } + Ok(()) +} + +/// Checks the snapshot at `snapshot` that OpenVMM published: a directory private to the current +/// user, as [`platform::check_snapshot_dir`] checks it, whose manifest +/// [`TemplateManifest::read`] reads, and whose artifacts are exactly its manifest, RAM, and saved +/// state, each a regular file private to the current user, of the length that the manifest +/// records. Reads no artifact but the manifest. Returns the manifest. +fn check_snapshot(snapshot: &Path) -> Result { + let what = "the snapshot of the capture"; + // A snapshot that is missing once its capture reported it is lost, not merely absent. + let problem = platform::check_snapshot_dir(snapshot).map_err(|error| { + Error::io(format!("cannot open {what} {}", snapshot.display())).with_source(error) + })?; + refuse_problem(snapshot, what, problem)?; + // So is a manifest that is missing from the snapshot. + let manifest = TemplateManifest::read(snapshot).map_err(|error| { + if error.kind() != ErrorKind::NotFound { + return error; + } + let (_, message, source) = error.into_parts(); + let error = Error::io(message); + match source { + Some(source) => error.with_source(source), + None => error, + } + })?; + let mut names = entries(snapshot)?; + names.sort(); + if names != SNAPSHOT_ARTIFACTS { + return Err(Error::io(format!( + "{what} {} holds {}, not exactly the manifest, RAM, and saved state of a platform-tier \ + snapshot", + snapshot.display(), + names.join(", ") + ))); + } + for (name, length) in [ + (MANIFEST_NAME, None), + (MEMORY_NAME, Some(manifest.memory_len())), + (STATE_NAME, Some(manifest.state_len())), + ] { + let path = snapshot.join(name); + let refused = |problem: String| { + Error::io(format!( + "the artifact {} of {what} {problem}", + path.display() + )) + }; + let file = + open_regular(&path).map_err(|error| refused(format!("cannot be used: {error}")))?; + if let Some(problem) = platform::private_file_problem(&file) + .map_err(|error| refused(format!("cannot be checked: {error}")))? + { + return Err(refused(problem)); + } + let found = file + .metadata() + .map_err(|error| refused(format!("cannot be examined: {error}")))? + .len(); + if let Some(length) = length + && found != length + { + return Err(refused(format!( + "is {found} bytes long, not the {length} bytes that its manifest records" + ))); + } + } + Ok(manifest) +} + +/// Removes from the capture's transaction `transaction` what the template does not keep: the lock +/// file of its sandbox store, which the capture closed once its VM had ended, the dormant scratch +/// device's file, the file that backed the VM's RAM, which OpenVMM removes as the VM ends unless +/// it ends abruptly, and directories in which OpenVMM staged a snapshot that it did not publish. +/// Fails unless the transaction then holds the record of the capture and its snapshot alone. +fn clean_transaction(transaction: &Path) -> Result<()> { + for name in entries(transaction)? { + let path = transaction.join(&name); + if name == LOCK_NAME || name == SCRATCH_NAME || name.starts_with(MEMORY_FILE_PREFIX) { + remove_if_present(&path)?; + } else if name.starts_with(STAGING_PREFIX) { + match fs::remove_dir_all(&path) { + Err(error) if error.kind() != io::ErrorKind::NotFound => { + return Err(store_error(format!("cannot remove {}", path.display()))( + error, + )); + } + _ => {} + } + } + } + let mut names = entries(transaction)?; + names.sort(); + if names != [CAPTURE_RECORD_NAME, SNAPSHOT_NAME] { + return Err(Error::io(format!( + "{TRANSACTION_WHAT} {} holds {}, not exactly the {RECORD_DESCRIPTION} and its \ + snapshot", + transaction.display(), + names.join(", ") + ))); + } + Ok(()) +} + +/// Returns whether the sandbox store of the capture's transaction `transaction` records a launch, +/// whose VMM may still run, or whether that cannot be told: whether the journal of a launch is +/// there, unless nothing is. +fn records_launch(transaction: &Path) -> bool { + !matches!( + fs::symlink_metadata(transaction.join(JOURNAL_NAME)), + Err(error) if error.kind() == io::ErrorKind::NotFound + ) +} + +/// Ends the launch that the sandbox store of the capture's transaction `transaction` records, if +/// any, as [`LockedStore::recover`] ends it, waiting for the store's lock until `deadline`. +/// +/// A transaction without the store's lock file records no launch: a capture creates the lock file +/// before it launches, and removes it once its VM has ended, before it checks the transaction. +/// Fails with [`ErrorCode::VmmIndeterminate`] if the recovery cannot prove that the VMM no longer +/// runs, and the store then keeps the launch. +fn end_launch(transaction: &Path, deadline: Instant) -> Result<()> { + let lock = transaction.join(LOCK_NAME); + match fs::symlink_metadata(&lock) { + Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(()), + Err(error) => { + return Err(store_error(format!("cannot open {}", lock.display()))( + error, + )); + } + Ok(_) => {} + } + let mut launch = SandboxStore::open(transaction)?.lock(Some(deadline))?; + match launch.recover()? { + Recovery::Indeterminate(reason) => Err(indeterminate(transaction, &reason)), + _ => Ok(()), + } +} + +/// Moves the VMM's log and outcome report of the capture whose transaction is `transaction` to +/// its slot, `slot`, which keeps those of its class's latest capture, once the capture's launch +/// has ended. A file that the transaction lacks, such as one that this moved already, stays as it +/// is. +fn keep_diagnostics(transaction: &Path, slot: &Path) -> Result<()> { + for name in [LOG_NAME, OUTCOME_NAME] { + let from = transaction.join(name); + match fs::rename(&from, slot.join(name)) { + Err(error) if error.kind() != io::ErrorKind::NotFound => { + return Err(store_error(format!( + "cannot move {} to the slot {}", + from.display(), + slot.display() + ))(error)); + } + _ => {} + } + } + Ok(()) +} + +/// Removes the transaction `transaction` of a capture that did not check it, with everything in +/// it, once nothing of the capture runs, if it exists. One that is a link, or that is not private +/// to the current user, is refused, and kept. +fn retire(transaction: &Path) -> Result<()> { + match platform::check_owner_only_dir(transaction) { + Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(()), + checked => { + let problem = checked.map_err(store_error(format!( + "cannot open {TRANSACTION_WHAT} {}", + transaction.display() + )))?; + refuse_problem(transaction, TRANSACTION_WHAT, problem)?; + } + } + match fs::remove_dir_all(transaction) { + Err(error) if error.kind() != io::ErrorKind::NotFound => Err(store_error(format!( + "cannot retire {TRANSACTION_WHAT} {}", + transaction.display() + ))(error)), + _ => Ok(()), + } +} + +/// Renames the directory `from` to `to` unless `to` exists, which it leaves as it is, as +/// [`platform::rename_no_replace`] does. On Windows, which renames no directory while a file below +/// it is open, as a scanner may hold one, a refused rename is tried again until `deadline`. +fn rename_until(from: &Path, to: &Path, deadline: Instant) -> Result<()> { + loop { + match platform::rename_no_replace(from, to) { + Ok(()) => return Ok(()), + Err(error) + if cfg!(windows) + && error.kind() == io::ErrorKind::PermissionDenied + && Instant::now() < deadline => + { + thread::sleep(RENAME_RETRY_INTERVAL); + } + Err(error) => { + return Err(store_error(format!( + "cannot publish {} as {}", + from.display(), + to.display() + ))(error)); + } + } + } +} + +/// Returns the store generation that follows `generation`. +fn following(generation: StoreGeneration) -> Result { + generation + .get() + .checked_add(1) + .map(StoreGeneration::new) + .ok_or_else(|| Error::unsupported("the template store has assigned every store generation")) +} + +/// What a checked capture records of itself, which the index lists of its generation once a +/// publication gives it a store generation. +struct Envelope { + /// The identity of the template's key. + key: KeyIdentity, + /// The identity of the capture's class. + class: ClassIdentity, + /// The capture's storage generation. + storage: StorageGeneration, + /// When OpenVMM created the snapshot. + created_at: SystemTime, + /// The time contract of the capture. + time: TimeContract, + /// The length of the saved state, in bytes. + state_len: u64, + /// The length of the RAM, in bytes. + memory_len: u64, +} + +impl Envelope { + /// Returns the envelope of a capture of the class `class`, with the storage generation + /// `storage`, whose key is `key`, and whose snapshot's manifest is `manifest`. + fn new( + key: &TemplateKey, + class: ClassIdentity, + storage: StorageGeneration, + manifest: &TemplateManifest, + ) -> Self { + Self { + key: key.identity(), + class, + storage, + created_at: manifest.created_at(), + time: manifest.time().clone(), + state_len: manifest.state_len(), + memory_len: manifest.memory_len(), + } + } + + /// Returns the generation of the envelope, as the store generation `generation`. + fn generation(&self, generation: StoreGeneration) -> Generation { + Generation { + generation, + key: self.key, + class: self.class, + storage: self.storage, + created_at: self.created_at, + time: self.time.clone(), + state_len: self.state_len, + memory_len: self.memory_len, + } + } +} + +/// The record of a capture, as JSON, which the capture writes in its transaction before its VM +/// starts: the store, the class, and the storage generation of the capture. +#[derive(Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct CaptureRecord { + /// [`STORE_FORMAT`]. + format: u32, + /// The identity of the store. + store: String, + /// The identity of the capture's class. + class: String, + /// The capture's storage generation. + storage_generation: String, +} + +impl CaptureRecord { + /// Returns the record of a capture of the class `class` in `store`, with the storage + /// generation `storage`. + fn new(store: &Store, class: ClassIdentity, storage: StorageGeneration) -> Self { + Self { + format: STORE_FORMAT, + store: store.id.to_string(), + class: class.to_string(), + storage_generation: storage.to_string(), + } + } +} diff --git a/nvx-rs/src/host/template/store/capture/tests.rs b/nvx-rs/src/host/template/store/capture/tests.rs new file mode 100644 index 000000000..9f224c341 --- /dev/null +++ b/nvx-rs/src/host/template/store/capture/tests.rs @@ -0,0 +1,1368 @@ +//! Tests of the captures of templates and of their publication, whose VMs a source in this +//! process emulates as OpenVMM runs them: captures of a class and the generations that they +//! publish, the checks of their files before their VMs start and of what their VMs leave, their +//! failures and crashes around each commit, and their recovery. The tests of +//! `tests/template_capture.rs` run captures through the supervisor, with the fake OpenVMM. + +use std::cell::{Cell, RefCell}; +use std::fs::OpenOptions; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Barrier}; + +use serde_json::{Value, json}; + +use super::*; +use crate::host::sandbox::distro::{DISTRO_POLICY_REVISION, PolicyVerdict}; +use crate::host::sandbox::{SandboxDistro, SandboxUpper}; +use crate::host::store::{CallerFiles, JOURNAL_NAME, LaunchId}; +use crate::host::template::eligibility::MAX_DOWNTIME_NS; +use crate::host::template::store::Lease; +use crate::host::template::store::tests::{ + cause, link_dir, link_file, private_dir, publish_with, storage, store, +}; +use crate::host::template::test_support::{MIB, ManifestSpec, class, network_class}; +use crate::protocol::root::RootImage; +use crate::uuid::Uuid; + +/// Bytes in a GiB. +const GIB: u64 = 1024 * MIB; + +/// Nanoseconds in a second. +const SECOND: u64 = 1_000_000_000; + +/// The identity of the placeholder. +const IDENTITY: &str = "11111111-1111-1111-1111-111111111111"; + +/// The access control list that OpenVMM gives a snapshot's directory and each of its files on +/// Windows: SYSTEM and the owner alone, and nothing that files inherit. +#[cfg(windows)] +const OPENVMM_ACCESS: &str = "D:P(A;;FA;;;SY)(A;;FA;;;OW)"; + +/// Returns a deadline that no capture of these tests reaches. +fn deadline() -> Instant { + Instant::now() + Duration::from_secs(60) +} + +/// Returns the host's time `seconds` after the captures of [`ManifestSpec::template`], on the +/// host and in the boot that captured them. +fn after(seconds: u64) -> HostTime { + let manifest = TemplateManifest::parse(&ManifestSpec::template().encode()).unwrap(); + let time = manifest.time(); + HostTime { + host_id: time.host_id, + boot_id: time.boot_id, + clock: time.host_clock, + monotonic_ns: time.capture_monotonic_ns + seconds * SECOND, + utc_ns: time.capture_utc_ns + seconds * SECOND, + } +} + +/// Returns the host's time `seconds` before the captures of [`ManifestSpec::template`], at +/// which none of them is eligible yet. +fn before(seconds: u64) -> HostTime { + let anchor = after(0); + HostTime { + monotonic_ns: anchor.monotonic_ns - seconds * SECOND, + utc_ns: anchor.utc_ns - seconds * SECOND, + ..anchor + } +} + +/// Creates the sparse file `path` of `length` bytes, which no one else may write, whatever the +/// umask. +fn sparse(path: &Path, length: u64) { + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, 0o600); + let file = options.open(path).unwrap(); + platform::make_sparse(&file).unwrap(); + file.set_len(length).unwrap(); +} + +/// Lets anyone change the file or directory `path`. +fn let_anyone_write(path: &Path) { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + + let mode = if path.is_dir() { 0o777 } else { 0o666 }; + fs::set_permissions(path, fs::Permissions::from_mode(mode)).unwrap(); + } + #[cfg(windows)] + platform::set_access(path, "D:P(A;OICI;FA;;;OW)(A;OICI;FA;;;SY)(A;OICI;FA;;;WD)").unwrap(); +} + +/// Returns the names of the entries of `dir`, sorted. +fn sorted(dir: &Path) -> Vec { + let mut names = entries(dir).unwrap(); + names.sort(); + names +} + +/// Returns the file at `path` as JSON. +fn json_of(path: &Path) -> Value { + serde_json::from_slice(&fs::read(path).unwrap()).unwrap() +} + +/// What the emulated VM of a capture does, as OpenVMM would: it stages a snapshot of the +/// manifest that it publishes beside the destination, with the RAM as a link to the file that +/// backed it, renames it to the destination, and writes the outcome report. +#[derive(Clone)] +struct Emulation { + /// The manifest of the snapshot that it publishes, if it publishes one. + manifest: Option, + /// The operation, category, and status code that its outcome report records, if it writes + /// one. + outcome: Option<(&'static str, &'static str, i64)>, + /// An entry that it publishes in the snapshot besides its artifacts. + extra: Option<&'static str>, + /// An artifact that it leaves out of the snapshot. + missing: Option<&'static str>, + /// The length of the saved state that it publishes, if not the manifest's. + state_len: Option, + /// Whether it leaves the file that backed the RAM, and a staging directory, as OpenVMM leaves + /// them when it ends abruptly. + leftovers: bool, +} + +impl Emulation { + /// Returns the emulation of a capture of [`ManifestSpec::template`] that succeeds. + fn capture() -> Self { + Self { + manifest: Some(ManifestSpec::template()), + outcome: Some(("capture", "success", 0)), + extra: None, + missing: None, + state_len: None, + leftovers: false, + } + } + + /// Returns the emulation of a VM whose guest failed with 3 before it captured. + fn guest_failure() -> Self { + Self { + manifest: None, + outcome: Some(("capture", "guest-exit", 3)), + ..Self::capture() + } + } + + /// Runs the VM of `vm` from the sandbox store of `locked`, the capture's transaction, after + /// removing the outcome report of the store's previous VM, as a launch does. + fn run(&self, locked: &mut LockedStore, vm: &Vm<'_>) -> Result<()> { + // OpenVMM requires the control socket of a VM that captures a snapshot, on Linux, beside + // the snapshot's destination: the VM launches from its transaction. + assert_eq!(locked.dir(), vm.template.transaction()); + assert_eq!( + locked.socket_path().parent(), + vm.template.destination().parent() + ); + remove_if_present(&locked.outcome_path())?; + let transaction = vm.template.transaction(); + let ram = transaction.join(format!("{MEMORY_FILE_PREFIX}emulated")); + sparse(&ram, vm.capture.class.memory); + if let Some(spec) = &self.manifest { + let staging = transaction.join(format!("{STAGING_PREFIX}1-0-0")); + private_dir(&staging); + fs::write(staging.join(MANIFEST_NAME), spec.encode()).unwrap(); + sparse( + &staging.join(STATE_NAME), + self.state_len.unwrap_or(spec.state_size), + ); + fs::hard_link(&ram, staging.join(MEMORY_NAME)).unwrap(); + if let Some(extra) = self.extra { + fs::write(staging.join(extra), b"").unwrap(); + } + if let Some(missing) = self.missing { + fs::remove_file(staging.join(missing)).unwrap(); + } + #[cfg(windows)] + for name in entries(&staging).unwrap() { + platform::set_access(&staging.join(name), OPENVMM_ACCESS).unwrap(); + } + #[cfg(windows)] + platform::set_access(&staging, OPENVMM_ACCESS).unwrap(); + fs::rename(&staging, vm.template.destination()).unwrap(); + } + if self.leftovers { + private_dir(&transaction.join(format!("{STAGING_PREFIX}1-0-1"))); + } else { + fs::remove_file(&ram).unwrap(); + } + if let Some((operation, category, status_code)) = self.outcome { + let report = json!({ + "schema_version": 1, + "backend": vm.capture.class.hypervisor.as_str(), + "outcome": { + "operation": operation, + "category": category, + "status_code": status_code, + }, + "teardown": { "vm_stopped": true }, + }); + fs::write(locked.outcome_path(), report.to_string()).unwrap(); + } + Ok(()) + } +} + +/// A store, and the inputs of captures into it, of [`class`], whose VMs [`Emulation`]s run. +struct Bench { + /// The temporary directory that holds the store and the platform's files. + _directory: tempfile::TempDir, + /// The store. + store: Store, + /// The inputs of a capture. + capture: Capture, + /// The number of VMs that captures ran. + runs: Cell, +} + +impl Bench { + /// Returns a bench whose placeholder is a regular file, in a directory private to the current + /// user. + fn new() -> Self { + Self::with(Path::to_path_buf) + } + + /// Returns a bench whose placeholder is at the path that `prepare` returns, given a sparse + /// file of 1 GiB in a directory private to the current user, which it may change before the + /// placeholder's binding is made. + fn with(prepare: impl FnOnce(&Path) -> PathBuf) -> Self { + let (directory, store) = store(); + let platform = directory.path().join("platform"); + private_dir(&platform); + let image = platform.join("placeholder.erofs"); + sparse(&image, GIB); + let placeholder = prepare(&image); + let path = placeholder.to_str().unwrap(); + let layer = SandboxLayer::new(LayerRole::Distro, path, IDENTITY); + let binding = DistroBinding::import(&layer, PolicyVerdict::ACCEPTED).unwrap(); + let image = RootImage::erofs(Some(Uuid::parse(IDENTITY).unwrap())); + let upper = SandboxUpper::scratch(platform.join("clone.ext4").to_str().unwrap()); + let capture = Capture { + class: class(), + adapter: SandboxPlan::adapter(SandboxDistro::new(path, image), upper), + binding, + policy_revision: DISTRO_POLICY_REVISION, + artifacts: Artifacts::new( + platform.join("openvmm"), + platform.join("vmlinux"), + platform.join("initramfs.cpio.gz"), + ), + }; + Self { + _directory: directory, + store, + capture, + runs: Cell::new(0), + } + } + + /// Returns the slot of the class of the captures. + fn slot(&self) -> PathBuf { + self.store + .captures_dir() + .join(slot_name(self.capture.class.identity().unwrap())) + } + + /// Returns the transaction of the captures. + fn transaction(&self) -> PathBuf { + self.slot().join(TRANSACTION_NAME) + } + + /// Serves the capture as [`Store::create_template`] does with the host's time that `now` + /// returns, or, without it, as [`Store::refresh_template`] does, with `emulation` as its VM, + /// failing it at the points at which `interrupt` fails, and crashing it if `crash`. + fn serve( + &self, + now: Option<&dyn Fn() -> Result>, + emulation: &Emulation, + interrupt: &dyn Fn(Point) -> Result<()>, + crash: bool, + ) -> Result { + self.serve_with( + now, + &|locked, vm| emulation.run(locked, vm), + interrupt, + crash, + ) + } + + /// Serves the capture as [`serve`](Self::serve) does, with `run` as its source. + fn serve_with( + &self, + now: Option<&dyn Fn() -> Result>, + run: &dyn Fn(&mut LockedStore, &Vm<'_>) -> Result<()>, + interrupt: &dyn Fn(Point) -> Result<()>, + crash: bool, + ) -> Result { + let counted = |locked: &mut LockedStore, vm: &Vm<'_>| { + self.runs.set(self.runs.get() + 1); + run(locked, vm) + }; + let hooks = Hooks { + sync: &sync_dir, + run: &counted, + interrupt, + crash, + }; + self.store.capture(&self.capture, now, deadline(), &hooks) + } + + /// Serves the capture as [`Store::create_template`] does, a second after the captures. + fn create(&self, emulation: &Emulation) -> Result { + self.serve(Some(&|| Ok(after(1))), emulation, &uninterrupted, false) + } + + /// Serves the capture as [`Store::refresh_template`] does. + fn refresh(&self, emulation: &Emulation) -> Result { + self.serve(None, emulation, &uninterrupted, false) + .map(Created::into_generation) + } + + /// Refreshes the capture, failing it at `point`, or crashing it there if `crash`, and returns + /// the error. + fn interrupted(&self, point: Point, crash: bool) -> Error { + let interrupt = |at: Point| { + if at == point { + Err(Error::other(format!("failed at {at:?}"))) + } else { + Ok(()) + } + }; + self.serve(None, &Emulation::capture(), &interrupt, crash) + .unwrap_err() + } + + /// Recovers the store's captures. + fn recover(&self) -> Result> { + self.recover_syncing(&sync_dir) + } + + /// Recovers the store's captures, syncing directories through `sync`. + fn recover_syncing(&self, sync: SyncDir<'_>) -> Result> { + let run = |_: &mut LockedStore, _: &Vm<'_>| -> Result<()> { + unreachable!("a recovery runs no VM") + }; + let hooks = Hooks { + sync, + run: &run, + interrupt: &uninterrupted, + crash: false, + }; + self.store.recover_with(deadline(), &hooks) + } + + /// Returns the store generations that the index lists. + fn indexed(&self) -> Vec { + numbers(self.store.index().unwrap().generations()) + } + + /// Drops the index's last generation and keeps its payload, as a crash leaves a publication + /// that renamed its envelope into the store before it committed the index. Captures complete + /// such publications before they run, so tests that need one beside a checked transaction + /// make it this way. + fn unindex_last(&self) { + let mut index = self.store.index().unwrap(); + let last = index.generations.pop().unwrap(); + index.next = last.generation(); + self.store.write_index(&index).unwrap(); + } +} + +/// Returns the store generation `generation`, which must not be 0. +fn store_generation(generation: u64) -> StoreGeneration { + StoreGeneration::new(std::num::NonZeroU64::new(generation).unwrap()) +} + +/// Returns the store generations of `generations`, as numbers. +fn numbers(generations: &[Generation]) -> Vec { + generations + .iter() + .map(|generation| generation.generation().get().get()) + .collect() +} + +/// Returns the files that the sandboxes of these tests keep: a configuration record. +fn files() -> CallerFiles { + CallerFiles::new("config.json").unwrap() +} + +/// Provisions the sandbox `name` in `parent`, and returns its directory. +fn provision(parent: &Path, name: &str) -> PathBuf { + let dir = parent.join(name); + let locked = SandboxStore::lock_for_provision(&dir, &files(), None).unwrap(); + fs::write(dir.join("config.json"), b"{}").unwrap(); + drop(locked); + dir +} + +/// Asserts that the slot of `bench` holds no transaction, and records no launch. +fn assert_clean(bench: &Bench) { + assert!(!bench.transaction().exists()); + assert!(!bench.slot().join(JOURNAL_NAME).exists()); +} + +#[test] +fn captures_publish_generations_that_creations_then_serve() { + let bench = Bench::new(); + let Created::Captured(first) = bench.create(&Emulation::capture()).unwrap() else { + panic!("the first creation of a class captures it"); + }; + assert_eq!(bench.runs.get(), 1); + assert_eq!(first.generation(), store_generation(1)); + let class = class(); + let manifest = TemplateManifest::parse(&ManifestSpec::template().encode()).unwrap(); + let key = TemplateKey::new(&class, &manifest).unwrap(); + assert_eq!(first.key(), key.identity()); + assert_eq!(first.class(), class.identity().unwrap()); + assert_eq!(first.created_at(), manifest.created_at()); + assert_eq!(first.time(), manifest.time()); + assert_eq!( + (first.state_len(), first.memory_len()), + (manifest.state_len(), manifest.memory_len()) + ); + assert_eq!( + bench.store.index().unwrap().generations(), + std::slice::from_ref(&first) + ); + // The payload is the envelope: OpenVMM's snapshot, with the capture's record and the + // template's key beside it. + let payload = bench.store.payload_dir(first.generation()); + assert_eq!( + sorted(&payload), + [CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME] + ); + assert_eq!(sorted(&payload.join(SNAPSHOT_NAME)), SNAPSHOT_ARTIFACTS); + assert_eq!(bench.store.read_key(&first).unwrap(), key); + assert_eq!( + bench.store.snapshot_dir(&first).unwrap(), + payload.join(SNAPSHOT_NAME) + ); + assert_eq!( + json_of(&payload.join(CAPTURE_RECORD_NAME)), + json!({ + "format": STORE_FORMAT, + "store": bench.store.id().to_string(), + "class": class.identity().unwrap().to_string(), + "storageGeneration": first.storage_generation().to_string(), + }) + ); + assert_clean(&bench); + // A creation serves the class with the generation that it holds, and runs no VM. + assert_eq!( + bench.create(&Emulation::capture()).unwrap(), + Created::Existing(first.clone()) + ); + assert_eq!(bench.runs.get(), 1); + // A refresh always captures, with a fresh storage generation, and creations then serve the + // new generation. + let second = bench.refresh(&Emulation::capture()).unwrap(); + assert_eq!(bench.runs.get(), 2); + assert_eq!(second.generation(), store_generation(2)); + assert_ne!(second.storage_generation(), first.storage_generation()); + assert_eq!(second.key(), first.key()); + assert_eq!( + bench.create(&Emulation::capture()).unwrap(), + Created::Existing(second.clone()) + ); + assert_eq!(bench.indexed(), [1, 2]); + assert_eq!(bench.store.read_key(&first).unwrap(), key); +} + +#[test] +fn refreshes_leave_older_generations_and_their_leases_untouched() { + let bench = Bench::new(); + let first = bench.refresh(&Emulation::capture()).unwrap(); + let sandboxes = tempfile::tempdir().unwrap(); + let sandbox = provision(sandboxes.path(), "clone"); + let lease = bench + .store + .acquire(first.generation(), &sandbox, &files(), &after(1), None) + .unwrap() + .lease; + let payload = bench.store.payload_dir(first.generation()); + let before = (sorted(&payload), bench.store.read_key(&first).unwrap()); + let second = bench.refresh(&Emulation::capture()).unwrap(); + assert_eq!(bench.indexed(), [1, 2]); + assert_eq!( + bench.store.index().unwrap().get(first.generation()), + Some(&first) + ); + assert_eq!( + (sorted(&payload), bench.store.read_key(&first).unwrap()), + before + ); + assert_eq!( + bench.store.leases(first.generation()).unwrap(), + std::slice::from_ref(&lease) + ); + assert!(bench.store.leases(second.generation()).unwrap().is_empty()); + assert_eq!( + Lease::of(&SandboxStore::open(&sandbox).unwrap()).unwrap(), + Some(lease) + ); +} + +#[test] +fn creations_capture_unless_the_class_has_a_generation_that_restores_take() { + let bench = Bench::new(); + // A generation of another class serves none of this one's. + let other = publish_with( + &bench.store, + &network_class(), + &ManifestSpec::with_network(), + storage(9), + ); + let Created::Captured(first) = bench.create(&Emulation::capture()).unwrap() else { + panic!("a class without generations is captured"); + }; + assert_eq!(first.generation(), store_generation(2)); + // A quarantined generation serves no creation. + bench.store.tombstone(first.generation(), None).unwrap(); + let Created::Captured(second) = bench.create(&Emulation::capture()).unwrap() else { + panic!("a class whose generations are quarantined is captured"); + }; + assert_eq!(second.generation(), store_generation(3)); + // Neither does one that no restore takes at the host's time. + let late = HostTime { + monotonic_ns: after(0).monotonic_ns + MAX_DOWNTIME_NS + SECOND, + ..after(0) + }; + assert!(!second.eligibility(&late).is_eligible()); + let Created::Captured(third) = bench + .serve( + Some(&|| Ok(late)), + &Emulation::capture(), + &uninterrupted, + false, + ) + .unwrap() + else { + panic!("a class whose generations are ineligible is captured"); + }; + assert_eq!(third.generation(), store_generation(4)); + assert_eq!( + bench.create(&Emulation::capture()).unwrap(), + Created::Existing(third) + ); + assert_eq!(bench.runs.get(), 3); + assert_eq!( + bench.store.index().unwrap().get(other.generation()), + Some(&other) + ); +} + +#[test] +fn failures_before_the_check_retire_the_transaction() { + let bench = Bench::new(); + for (point, runs) in [(Point::Prepared, 0), (Point::Exited, 1)] { + let error = bench.interrupted(point, false); + assert_eq!(error.message(), format!("failed at {point:?}")); + assert_eq!(bench.runs.replace(0), runs, "{point:?}"); + assert_clean(&bench); + } + let failures = [ + ( + Emulation::guest_failure(), + ErrorKind::Exited, + "records the outcome guest-exit with status 3 of the operation capture", + ), + // OpenVMM may fail once it has published the snapshot, as when it cannot make the + // publication durable. + ( + Emulation { + outcome: Some(("capture", "vmm-failure", 1)), + ..Emulation::capture() + }, + ErrorKind::Exited, + "records the outcome vmm-failure with status 1", + ), + ( + Emulation { + outcome: None, + ..Emulation::capture() + }, + ErrorKind::Exited, + "ended without the outcome report", + ), + ( + Emulation { + outcome: Some(("run", "success", 0)), + ..Emulation::capture() + }, + ErrorKind::Exited, + "of the operation run", + ), + ( + Emulation { + manifest: None, + ..Emulation::capture() + }, + ErrorKind::Io, + "cannot open the snapshot of the capture", + ), + ( + Emulation { + missing: Some(MANIFEST_NAME), + ..Emulation::capture() + }, + ErrorKind::Io, + "cannot read the snapshot manifest", + ), + ( + Emulation { + missing: Some(MEMORY_NAME), + ..Emulation::capture() + }, + ErrorKind::Io, + "holds manifest.bin, state.bin, not exactly the manifest", + ), + ( + Emulation { + manifest: Some(ManifestSpec::with_network()), + ..Emulation::capture() + }, + ErrorKind::Io, + "the capture does not match its class", + ), + ( + Emulation { + extra: Some("scratch.img"), + ..Emulation::capture() + }, + ErrorKind::Io, + "manifest.bin, memory.bin, scratch.img, state.bin, not exactly the manifest", + ), + ( + Emulation { + state_len: Some(ManifestSpec::template().state_size - 512), + ..Emulation::capture() + }, + ErrorKind::Io, + "bytes that its manifest records", + ), + ]; + for (emulation, kind, message) in failures { + let error = bench.refresh(&emulation).unwrap_err(); + assert_eq!(error.kind(), kind, "{error}"); + assert!(error.message().contains(message), "{error}"); + assert_eq!(bench.runs.replace(0), 1); + assert_clean(&bench); + // The slot keeps the outcome report of the latest capture, and none of an earlier one. + assert_eq!( + bench.slot().join(OUTCOME_NAME).exists(), + emulation.outcome.is_some() + ); + } + assert!(bench.indexed().is_empty()); +} + +#[test] +fn what_openvmm_leaves_beside_the_snapshot_is_removed() { + let bench = Bench::new(); + let emulation = Emulation { + leftovers: true, + ..Emulation::capture() + }; + let generation = bench.refresh(&emulation).unwrap(); + assert_eq!( + sorted(&bench.store.payload_dir(generation.generation())), + [CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME] + ); + assert_clean(&bench); +} + +#[test] +fn checked_transactions_outlive_failures_until_recovery_publishes_them() { + let bench = Bench::new(); + bench.interrupted(Point::Checked, false); + assert_eq!( + sorted(&bench.transaction()), + [CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME] + ); + assert!(bench.indexed().is_empty()); + let published = bench.recover().unwrap(); + assert_eq!(published.len(), 1); + assert_eq!(bench.store.index().unwrap().generations(), published); + assert_clean(&bench); + // The next capture of the class publishes it too, and then serves it. + let bench = Bench::new(); + bench.interrupted(Point::Checked, false); + let Created::Existing(generation) = bench.create(&Emulation::capture()).unwrap() else { + panic!("a creation serves the generation that it recovered"); + }; + assert_eq!(generation.generation(), store_generation(1)); + assert_eq!(bench.runs.get(), 1); + assert_clean(&bench); +} + +#[test] +fn publications_interrupted_once_they_renamed_their_envelope_are_completed() { + let bench = Bench::new(); + bench.interrupted(Point::Renamed, false); + assert!(bench.indexed().is_empty()); + assert_eq!( + sorted(&bench.store.payload_dir(store_generation(1))), + [CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME] + ); + // The next capture completes it first. + let generation = bench.refresh(&Emulation::capture()).unwrap(); + assert_eq!(generation.generation(), store_generation(2)); + assert_eq!(bench.indexed(), [1, 2]); + // So does a recovery. + bench.interrupted(Point::Renamed, false); + let published = bench.recover().unwrap(); + assert_eq!(published.len(), 1); + assert_eq!(published[0].generation(), store_generation(3)); + assert_eq!(bench.indexed(), [1, 2, 3]); + // A recovery that completes one before it publishes a checked transaction returns both. + bench.interrupted(Point::Checked, false); + bench.unindex_last(); + assert_eq!(bench.indexed(), [1, 2]); + assert_eq!(numbers(&bench.recover().unwrap()), [3, 4]); + assert_eq!(bench.indexed(), [1, 2, 3, 4]); + assert_clean(&bench); +} + +#[test] +fn recoveries_make_what_they_publish_durable_before_they_index_it() { + let bench = Bench::new(); + bench.refresh(&Emulation::capture()).unwrap(); + bench.interrupted(Point::Checked, false); + bench.unindex_last(); + // Each directory that the recovery syncs, with what the index lists as it does. + let synced = RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced + .borrow_mut() + .push((dir.to_path_buf(), bench.indexed())); + sync_dir(dir) + }; + assert_eq!(numbers(&bench.recover_syncing(&sync).unwrap()), [1, 2]); + let generations = bench.store.dir.join(GENERATIONS_NAME); + let parent = bench.store.dir.parent().unwrap().to_path_buf(); + assert_eq!( + synced.into_inner(), + [ + // The transaction that the crash left checked. + (bench.transaction(), vec![]), + // The publication that the crash interrupted once it had renamed its envelope. + (bench.store.payload_dir(store_generation(1)), vec![]), + (generations.clone(), vec![]), + (bench.slot(), vec![]), + // The transaction's publication. + (generations, vec![]), + (bench.slot(), vec![]), + (bench.store.dir.clone(), vec![1, 2]), + (parent, vec![1, 2]), + ] + ); + assert_clean(&bench); +} + +#[test] +fn captures_make_the_store_durable_before_they_report_a_generation() { + let bench = Bench::new(); + // Each directory that the capture syncs, with what the index lists as it does. + let synced = RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced + .borrow_mut() + .push((dir.to_path_buf(), bench.indexed())); + sync_dir(dir) + }; + let run = |locked: &mut LockedStore, vm: &Vm<'_>| Emulation::capture().run(locked, vm); + let hooks = Hooks { + sync: &sync, + run: &run, + interrupt: &uninterrupted, + crash: false, + }; + let created = bench + .store + .capture(&bench.capture, None, deadline(), &hooks) + .unwrap(); + assert_eq!(created.generation().generation(), store_generation(1)); + let parent = bench.store.dir.parent().unwrap().to_path_buf(); + assert_eq!( + synced.into_inner(), + [ + // The checked transaction, and every directory above it, up to the store's parent, + // which the store's creation did not sync. + (bench.transaction(), vec![]), + (bench.slot(), vec![]), + (bench.store.captures_dir(), vec![]), + (bench.store.dir.clone(), vec![]), + (parent.clone(), vec![]), + // The rename into the store. + (bench.store.dir.join(GENERATIONS_NAME), vec![]), + (bench.slot(), vec![]), + // The index. + (bench.store.dir.clone(), vec![1]), + (parent, vec![1]), + ] + ); +} + +#[test] +fn creations_recover_the_index_before_they_look_for_a_generation() { + // A creation completes a publication that a crash interrupted, and serves its generation, + // the class's latest. + let bench = Bench::new(); + bench.refresh(&Emulation::capture()).unwrap(); + bench.interrupted(Point::Renamed, false); + assert_eq!(bench.indexed(), [1]); + let Created::Existing(generation) = bench.create(&Emulation::capture()).unwrap() else { + panic!("a creation serves the generation whose publication it completed"); + }; + assert_eq!(generation.generation(), store_generation(2)); + assert_eq!(bench.indexed(), [1, 2]); + assert_eq!(bench.runs.get(), 2); + // It serves no generation whose payload a crash lost, and captures one in its place. + let bench = Bench::new(); + bench.refresh(&Emulation::capture()).unwrap(); + fs::remove_dir_all(bench.store.payload_dir(store_generation(1))).unwrap(); + let Created::Captured(generation) = bench.create(&Emulation::capture()).unwrap() else { + panic!("a creation captures in place of a generation whose payload was lost"); + }; + assert_eq!(generation.generation(), store_generation(2)); + assert_eq!(bench.indexed(), [2]); + // A store whose index is full of generations whose payloads a crash lost still captures. + let bench = Bench::new(); + let first = bench.refresh(&Emulation::capture()).unwrap(); + let last = MAX_GENERATIONS as u64; + let mut index = bench.store.index().unwrap(); + index + .generations + .extend((2..=last).map(|generation| Generation { + generation: store_generation(generation), + ..first.clone() + })); + index.next = store_generation(last + 1); + bench.store.write_index(&index).unwrap(); + assert_eq!(bench.indexed().len(), MAX_GENERATIONS); + let generation = bench.refresh(&Emulation::capture()).unwrap(); + assert_eq!(generation.generation(), store_generation(last + 1)); + assert_eq!(bench.indexed(), [1, last + 1]); + assert_clean(&bench); +} + +#[test] +fn recoveries_drop_lost_generations_before_they_index_another() { + // A store whose index is full of generations that a crash lost, but one, beside a publication + // that a crash interrupted once it had renamed its envelope. + let bench = Bench::new(); + let first = bench.refresh(&Emulation::capture()).unwrap(); + bench.interrupted(Point::Renamed, false); + let last = MAX_GENERATIONS as u64; + fs::rename( + bench.store.payload_dir(store_generation(2)), + bench.store.payload_dir(store_generation(last + 1)), + ) + .unwrap(); + let mut index = bench.store.index().unwrap(); + index + .generations + .extend((2..=last).map(|generation| Generation { + generation: store_generation(generation), + ..first.clone() + })); + index.next = store_generation(last + 1); + bench.store.write_index(&index).unwrap(); + assert_eq!(bench.indexed().len(), MAX_GENERATIONS); + // The lost generations take none of the store's capacity from the publication. + assert_eq!(numbers(&bench.recover().unwrap()), [last + 1]); + assert_eq!(bench.indexed(), [1, last + 1]); + assert_clean(&bench); +} + +#[test] +fn creations_make_the_store_durable_before_they_serve_a_generation() { + let bench = Bench::new(); + let parent = bench.store.dir.parent().unwrap().to_path_buf(); + let run = |locked: &mut LockedStore, vm: &Vm<'_>| Emulation::capture().run(locked, vm); + // A capture whose host crashes as it syncs the store's parent, once the index lists the + // capture's generation. + let crashing = |dir: &Path| -> Result<()> { + if dir == parent && !bench.indexed().is_empty() { + return Err(Error::other("the host crashed")); + } + sync_dir(dir) + }; + let hooks = Hooks { + sync: &crashing, + run: &run, + interrupt: &uninterrupted, + crash: true, + }; + bench + .store + .capture(&bench.capture, None, deadline(), &hooks) + .unwrap_err(); + assert_eq!(bench.indexed(), [1]); + // A creation serves the generation once it has made the index, and the store's own entry in + // its parent, durable, although no recovery changed the index. + let synced = RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced.borrow_mut().push(dir.to_path_buf()); + sync_dir(dir) + }; + let hooks = Hooks { + sync: &sync, + run: &run, + interrupt: &uninterrupted, + crash: false, + }; + let now = || -> Result { Ok(after(1)) }; + let Created::Existing(generation) = bench + .store + .capture(&bench.capture, Some(&now), deadline(), &hooks) + .unwrap() + else { + panic!("a creation serves the generation that the index lists"); + }; + assert_eq!(generation.generation(), store_generation(1)); + assert_eq!(synced.into_inner(), [bench.store.dir.clone(), parent]); + assert_clean(&bench); +} + +#[test] +fn recovery_ends_the_captures_that_crashes_interrupt() { + for (point, left, indexed) in [ + ( + Point::Prepared, + vec![CAPTURE_RECORD_NAME, LOCK_NAME, SCRATCH_NAME], + vec![], + ), + ( + Point::Exited, + vec![CAPTURE_RECORD_NAME, LOCK_NAME, SCRATCH_NAME, SNAPSHOT_NAME], + vec![], + ), + ( + Point::Checked, + vec![CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME], + vec![1], + ), + (Point::Renamed, vec![], vec![1]), + (Point::Indexed, vec![], vec![1]), + ] { + let bench = Bench::new(); + bench.interrupted(point, true); + if left.is_empty() { + assert!(!bench.transaction().exists(), "{point:?}"); + } else { + assert_eq!(sorted(&bench.transaction()), left, "{point:?}"); + } + let published = bench.recover().unwrap(); + let recovered: Vec = published + .iter() + .map(|generation| generation.generation().get().get()) + .collect(); + let expected = if matches!(point, Point::Checked | Point::Renamed) { + vec![1] + } else { + vec![] + }; + assert_eq!(recovered, expected, "{point:?}"); + assert_eq!(bench.indexed(), indexed, "{point:?}"); + assert_clean(&bench); + // The slot captures again. + bench.refresh(&Emulation::capture()).unwrap(); + } +} + +#[test] +fn generations_whose_payloads_are_lost_are_dropped_from_the_index() { + let bench = Bench::new(); + let first = bench.refresh(&Emulation::capture()).unwrap(); + bench.refresh(&Emulation::capture()).unwrap(); + fs::remove_dir_all(bench.store.payload_dir(first.generation())).unwrap(); + bench.refresh(&Emulation::capture()).unwrap(); + assert_eq!(bench.indexed(), [2, 3]); + fs::remove_dir_all(bench.store.payload_dir(store_generation(3))).unwrap(); + assert!(bench.recover().unwrap().is_empty()); + assert_eq!(bench.indexed(), [2]); + // The index never assigns a store generation twice. + assert_eq!( + bench.store.index().unwrap().next_generation(), + store_generation(4) + ); +} + +#[test] +fn a_transaction_stays_while_it_records_a_launch() { + let bench = Bench::new(); + // A source that fails while its launch remains, as one whose VMM may still run: dropping a + // launch leaves its journal, as a crash would. + let failed = |locked: &mut LockedStore, vm: &Vm<'_>| -> Result<()> { + let capability = [7; crate::protocol::CAPABILITY_LEN]; + let launch = LaunchId::generate()?; + drop(locked.begin_launch(&launch, &vm.capture.class.runtime, &capability)?); + Err(Error::other("the VMM may still run")) + }; + let error = bench + .serve_with(None, &failed, &uninterrupted, false) + .unwrap_err(); + assert_eq!(error.message(), "the VMM may still run"); + // The transaction's sandbox store records the launch, and the slot none. + assert!(bench.transaction().join(JOURNAL_NAME).exists()); + assert!(!bench.slot().join(JOURNAL_NAME).exists()); + let left = sorted(&bench.transaction()); + for name in [CAPTURE_RECORD_NAME, SCRATCH_NAME, LOCK_NAME, LOG_NAME] { + assert!(left.iter().any(|left| left == name), "{name}: {left:?}"); + } + // The next capture ends the launch, which nothing holds, retires the transaction, and + // captures. + bench.refresh(&Emulation::capture()).unwrap(); + assert_clean(&bench); + assert_eq!(bench.indexed(), [1]); +} + +/// Returns whether this host lets the tests create a symbolic link to a file: Windows lets only +/// Developer Mode or a privilege create one. +fn can_link_files() -> bool { + let directory = tempfile::tempdir().unwrap(); + let target = directory.path().join("target"); + fs::write(&target, b"").unwrap(); + link_file(&target, &directory.path().join("link")) +} + +/// Asserts that `result` failed with `kind`, with a message that holds `message`, and that its +/// capture ran no VM, and left no transaction. +fn assert_refused(bench: &Bench, result: Result, kind: ErrorKind, message: &str) { + let error = result.unwrap_err(); + assert_eq!(error.kind(), kind, "{error}"); + assert!( + error.message().contains(message) || cause(&error).contains(message), + "{error}: {}", + cause(&error) + ); + assert_eq!(bench.runs.replace(0), 0); + assert!(bench.indexed().is_empty()); +} + +#[test] +fn links_and_unsafe_ownership_are_refused_before_the_vm_starts() { + // A placeholder that is a link, which OpenVMM would not follow, though the check of its + // binding follows it. + if can_link_files() { + let bench = Bench::with(|image| { + let link = image.with_file_name("link.erofs"); + assert!(link_file(image, &link)); + link + }); + let result = bench.create(&Emulation::capture()); + assert_refused( + &bench, + result, + ErrorKind::Unavailable, + "it is a link, or not a regular file", + ); + assert_clean(&bench); + } + // A placeholder that others may change. + let bench = Bench::with(|image| { + let_anyone_write(image); + image.to_path_buf() + }); + let result = bench.create(&Emulation::capture()); + let message = if cfg!(windows) { + "allows other users to change it" + } else { + "is writable by other users" + }; + assert_refused(&bench, result, ErrorKind::Unavailable, message); + assert_clean(&bench); + // A scratch that a link replaced, or that others may change. + let bench = Bench::new(); + let scratch = bench.transaction().join(SCRATCH_NAME); + let replaced = |point: Point| -> Result<()> { + if point == Point::Prepared { + let target = bench.slot().join("elsewhere.raw"); + sparse(&target, class().scratch.length); + fs::remove_file(&scratch).unwrap(); + assert!(link_file(&target, &scratch)); + } + Ok(()) + }; + if can_link_files() { + let result = bench.serve(None, &Emulation::capture(), &replaced, false); + assert_refused( + &bench, + result, + ErrorKind::Io, + "it is a link, or not a regular file", + ); + assert_clean(&bench); + } + let writable = |point: Point| -> Result<()> { + if point == Point::Prepared { + let_anyone_write(&scratch); + } + Ok(()) + }; + let result = bench.serve(None, &Emulation::capture(), &writable, false); + assert_refused(&bench, result, ErrorKind::Io, message); + assert_clean(&bench); + // A transaction that others may access, which the capture keeps rather than retire. + let exposed_message = if cfg!(windows) { + "allows other users" + } else { + "is accessible to other users" + }; + let transaction = bench.transaction(); + let exposed = |point: Point| -> Result<()> { + if point == Point::Prepared { + let_anyone_write(&transaction); + } + Ok(()) + }; + let result = bench.serve(None, &Emulation::capture(), &exposed, false); + assert_refused(&bench, result, ErrorKind::Io, exposed_message); + assert!(transaction.exists()); + let result = bench.create(&Emulation::capture()); + assert_refused(&bench, result, ErrorKind::Io, exposed_message); + fs::remove_dir_all(&transaction).unwrap(); + // A transaction that is a link. + let elsewhere = bench.slot().join("elsewhere"); + private_dir(&elsewhere); + link_dir(&elsewhere, &transaction); + let result = bench.create(&Emulation::capture()); + assert_refused(&bench, result, ErrorKind::Io, "is not a plain directory"); + assert!(fs::symlink_metadata(&transaction).is_ok()); +} + +#[test] +fn scratches_are_the_files_that_their_captures_created() { + // Another file of the scratch's length, renamed over the scratch. It is created while the + // scratch exists, so that the host cannot give it the scratch's identity again. + let bench = Bench::new(); + let scratch = bench.transaction().join(SCRATCH_NAME); + let replaced = |point: Point| -> Result<()> { + if point == Point::Prepared { + let other = bench.slot().join("other.raw"); + sparse(&other, class().scratch.length); + fs::rename(&other, &scratch).unwrap(); + } + Ok(()) + }; + let result = bench.serve(None, &Emulation::capture(), &replaced, false); + assert_refused( + &bench, + result, + ErrorKind::Io, + "is not the file that the capture created", + ); + assert_clean(&bench); + // A file whose ID differs from the scratch's only above its low 64 bits, as the 128-bit IDs + // of two files of a file system such as ReFS may: the check compares the whole ID. + let inputs = Inputs::check(&bench.capture).unwrap(); + let template = TemplatePlan::new( + &bench.capture.adapter, + &bench.capture.binding, + bench.capture.class.scratch.length, + inputs.resources, + &bench.transaction(), + StorageGeneration::generate().unwrap(), + ) + .unwrap(); + private_dir(&bench.transaction()); + let created = create_scratch(template.scratch_path(), template.scratch_length()).unwrap(); + check_scratch(&template, created).unwrap(); + let (device, file_id) = created; + let error = check_scratch(&template, (device, file_id ^ (1 << 64))).unwrap_err(); + assert!( + error + .message() + .contains("is not the file that the capture created"), + "{error}" + ); +} + +/// Returns whether this process holds the file at `path` open for writing. +#[cfg(target_os = "linux")] +fn held_for_writing(path: &Path) -> bool { + let target = fs::canonicalize(path).unwrap(); + fs::read_dir("/proc/self/fd") + .unwrap() + .filter_map(std::result::Result::ok) + .any(|entry| { + let flags = || { + let info = + fs::read_to_string(Path::new("/proc/self/fdinfo").join(entry.file_name())) + .ok()?; + let flags = info.lines().find_map(|line| line.strip_prefix("flags:"))?; + u32::from_str_radix(flags.trim(), 8).ok() + }; + fs::read_link(entry.path()).is_ok_and(|linked| linked == target) + && flags().is_some_and(|flags| flags & 0o3 != 0) + }) +} + +/// Opens the file at `path` as OpenVMM opens a block device of a capture on Windows: without +/// following a reparse point, for reading, and for writing if `write`, sharing reads and +/// deletion alone, which fails while another handle may write it. +#[cfg(windows)] +fn open_as_openvmm(path: &Path, write: bool) -> io::Result { + use std::os::windows::fs::OpenOptionsExt; + + /// `FILE_SHARE_READ | FILE_SHARE_DELETE`. + const SHARE_READ_DELETE: u32 = 0x1 | 0x4; + /// `FILE_FLAG_OPEN_REPARSE_POINT`. + const OPEN_REPARSE_POINT: u32 = 0x0020_0000; + OpenOptions::new() + .read(true) + .write(write) + .share_mode(SHARE_READ_DELETE) + .custom_flags(OPEN_REPARSE_POINT) + .open(path) +} + +#[test] +fn no_handle_through_which_the_host_could_write_the_block_devices_is_open_as_the_vm_starts() { + let bench = Bench::new(); + let checked = |locked: &mut LockedStore, vm: &Vm<'_>| -> Result<()> { + #[cfg(any(target_os = "linux", windows))] + let (placeholder, scratch) = ( + vm.template.plan().distro().path(), + vm.template.scratch_path(), + ); + #[cfg(target_os = "linux")] + { + assert!(!held_for_writing(placeholder)); + assert!(!held_for_writing(scratch)); + } + #[cfg(windows)] + { + open_as_openvmm(placeholder, false).unwrap(); + open_as_openvmm(scratch, true).unwrap(); + // The checked placeholder stays held, so that nothing writes it meanwhile. + let error = OpenOptions::new() + .write(true) + .open(placeholder) + .unwrap_err(); + assert_eq!(error.raw_os_error(), Some(32), "{error}"); + } + Emulation::capture().run(locked, vm) + }; + bench + .serve_with(None, &checked, &uninterrupted, false) + .unwrap(); + assert_eq!(bench.runs.get(), 1); + // The placeholder is released once the VM has ended. + OpenOptions::new() + .write(true) + .open(bench.capture.adapter.distro().path()) + .unwrap(); +} + +#[test] +fn captures_of_one_class_are_serialized() { + let active = AtomicUsize::new(0); + let most = AtomicUsize::new(0); + let runs = AtomicUsize::new(0); + let run = |locked: &mut LockedStore, vm: &Vm<'_>| -> Result<()> { + let now = active.fetch_add(1, Ordering::SeqCst) + 1; + most.fetch_max(now, Ordering::SeqCst); + runs.fetch_add(1, Ordering::SeqCst); + thread::sleep(Duration::from_millis(100)); + let ran = Emulation::capture().run(locked, vm); + active.fetch_sub(1, Ordering::SeqCst); + ran + }; + // Serves the capture of `bench` twice at once, as creations at the host's time that `now` + // returns, or as refreshes. + let race = + |bench: &Bench, now: Option<&(dyn Fn() -> Result + Sync)>| -> Vec { + let (store, capture) = (&bench.store, &bench.capture); + let barrier = Arc::new(Barrier::new(2)); + let serve = || { + let hooks = Hooks { + sync: &sync_dir, + run: &run, + interrupt: &uninterrupted, + crash: false, + }; + let now: Option<&dyn Fn() -> Result> = match now { + Some(now) => Some(now), + None => None, + }; + barrier.wait(); + store.capture(capture, now, deadline(), &hooks).unwrap() + }; + thread::scope(|scope| { + let threads: Vec<_> = (0..2).map(|_| scope.spawn(serve)).collect(); + threads + .into_iter() + .map(|thread| thread.join().unwrap()) + .collect() + }) + }; + // Refreshes capture one after the other. + let bench = Bench::new(); + let refreshed = race(&bench, None); + assert!( + refreshed + .iter() + .all(|created| matches!(created, Created::Captured(_))) + ); + assert_eq!(runs.swap(0, Ordering::SeqCst), 2); + assert_eq!(bench.indexed(), [1, 2]); + // Creations that race make one generation between them, although both start before the + // capture's anchor: the host's time is before it until a capture is published. + let bench = Bench::new(); + let store = &bench.store; + let now = || -> Result { + Ok(if store.index()?.generations().is_empty() { + before(1) + } else { + after(1) + }) + }; + assert_eq!(now().unwrap(), before(1)); + let created = race(&bench, Some(&now)); + assert_eq!(runs.load(Ordering::SeqCst), 1); + assert_eq!(created[0].generation(), created[1].generation()); + assert_eq!( + created + .iter() + .filter(|created| matches!(created, Created::Captured(_))) + .count(), + 1 + ); + assert_eq!(most.load(Ordering::SeqCst), 1); +} + +#[test] +fn recoveries_refuse_entries_of_the_captures_that_are_not_slots() { + let bench = Bench::new(); + assert!(bench.recover().unwrap().is_empty()); + bench.refresh(&Emulation::capture()).unwrap(); + fs::write(bench.store.captures_dir().join("notes.txt"), b"").unwrap(); + let error = bench.recover().unwrap_err(); + assert_eq!(error.code(), ErrorCode::ForeignFiles, "{error}"); + assert!(error.message().ends_with(": notes.txt"), "{error}"); +} + +#[cfg(target_os = "linux")] +#[test] +fn stores_too_deep_for_the_control_socket_of_a_capture_are_refused() { + let bench = Bench::new(); + let parent = bench._directory.path().join("d".repeat(80)); + fs::create_dir(&parent).unwrap(); + let bench = Bench { + store: Store::create(parent.join("templates")).unwrap(), + ..bench + }; + let error = bench.refresh(&Emulation::capture()).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Unsupported, "{error}"); + assert!( + error.message().contains("too deep for the control socket"), + "{error}" + ); + assert_eq!(bench.runs.get(), 0); +} diff --git a/nvx-rs/src/host/template/store/tests.rs b/nvx-rs/src/host/template/store/tests.rs index d37dc2ae8..fdd8d9447 100644 --- a/nvx-rs/src/host/template/store/tests.rs +++ b/nvx-rs/src/host/template/store/tests.rs @@ -951,3 +951,32 @@ fn the_store_is_private_to_the_current_user() { ); } } + +#[cfg(windows)] +#[test] +fn snapshots_that_openvmm_published_are_private_without_entries_for_their_files() { + let (_directory, store) = store(); + let published = publish(&store, &ManifestSpec::template(), storage(1)); + let payload = store.payload_dir(published.generation()); + // OpenVMM gives a snapshot's directory, and each of its files, a protected list of their own, + // which allows SYSTEM and the owner, and passes nothing on to the files created in it. + let openvmm = "D:P(A;;FA;;;SY)(A;;FA;;;OW)"; + let snapshot = payload.join(SNAPSHOT_NAME); + platform::set_access(&snapshot.join(MANIFEST_NAME), openvmm).unwrap(); + platform::set_access(&snapshot, openvmm).unwrap(); + assert_eq!(store.snapshot_dir(&published).unwrap(), snapshot); + // The host's own directories still pass entries on to the files created in them. + platform::set_access(&payload, openvmm).unwrap(); + let error = store.snapshot_dir(&published).unwrap_err(); + assert!( + error + .message() + .ends_with("passes no access control entries on to the files created in it"), + "{error}" + ); + // A snapshot whose list allows anyone else is refused. + platform::set_access(&payload, "D:P(A;OICI;FA;;;OW)").unwrap(); + platform::set_access(&snapshot, "D:P(A;;FA;;;SY)(A;;FA;;;OW)(A;;FR;;;WD)").unwrap(); + let error = store.snapshot_dir(&published).unwrap_err(); + assert!(error.message().ends_with("allows other users"), "{error}"); +} diff --git a/nvx-rs/src/testing/fake_openvmm.rs b/nvx-rs/src/testing/fake_openvmm.rs index 077cdf452..46e2b8608 100644 --- a/nvx-rs/src/testing/fake_openvmm.rs +++ b/nvx-rs/src/testing/fake_openvmm.rs @@ -8,6 +8,11 @@ //! and decodes requests with the guest agent's decoders, whose conformance vectors pin the wire //! format independently. //! +//! On Unix, as OpenVMM does, the fake refuses a control console unless its socket's directory is a +//! directory, not a link, that belongs to the current user, with exactly the mode 0700, and, if +//! the command line names a snapshot, the canonical directory that holds it: the one that +//! `--restore-snapshot` restores, or else the destination of `--snapshot-destination`. +//! //! Workloads are `/bin/sh -c SCRIPT`, `/bin/echo ARGS...`, `/bin/pwd`, or a program named `env` //! or `printenv` in any directory, which prints its environment. A script is a `;`-separated //! list of commands: `echo TEXT`, `echoerr TEXT`, `printerr TEXT`, which writes `TEXT` to @@ -89,6 +94,43 @@ //! its command line next to its snapshot, since it has no kernel. The outcome reports of sandboxes //! and of these runs take the form of OpenVMM's. //! +//! The fake also runs the VMs of templates' captures, whose command lines have no +//! `--microvm-lifecycle`, a `distro` layer and a `scratch` disk on sandbox block devices and no +//! share, `--snapshot-destination`, `--snapshot-tier platform`, +//! `--snapshot-block-identity generation` with a nonzero `--snapshot-generation-id` of 32 +//! hexadecimal digits, the options of a cold boot of a run with a `--memory-capacity` that is a +//! multiple of 128 MiB, `nvx_sandbox=1` alone on the kernel command line, `--virtio-console none`, +//! the control console with its capability on standard input, and an outcome report. On Unix, the +//! control console's socket must be in the directory of the snapshot's destination, as above. The +//! fake serves no control session of a capture, which the host opens none of. Like OpenVMM, it +//! opens each block device itself, at its path, without following a link, and on Windows sharing +//! only reads and deletion, so that its open fails while another handle may write the file; on +//! Linux, a read lease, which no process may take of a file that a handle holds open for writing, +//! finds such a handle. It records each device's role, path, and length, and whether it found a +//! writer, in the blocks dump. It then backs the RAM with a sparse file beside the destination, and +//! does what `fake-openvmm-capture.json` next to the kernel asks, a JSON object of these fields, +//! each optional but `manifest`: +//! +//! - `manifest`: the manifest to publish, which the fake copies, since it reads none; +//! - `stateLen`: the length of the saved state to publish, 0 by default; +//! - `outcome`: `publish`, the default, which stages the snapshot beside the destination, with the +//! RAM as a hard link to the file that backed it, renames it to the destination, and reports +//! the capture's success; `fail`, a guest that fails with `status` before it captures; +//! `publish-then-fail`, a VMM that publishes the snapshot and then fails to tear the VM down, +//! which reports `teardown-failure` and exits with 1; `hang` and `publish-then-hang`, a VM that +//! never ends; and `crash`, a VMM that exits with 3, without a report; +//! - `status`: the guest's status of `fail`, 1 by default; +//! - `active`: a file that the fake creates as it starts, failing if it exists, which holds its +//! process ID, and which it removes as it ends, unless it is killed, so that a test sees whether +//! two captures overlapped, and which VMM a capture left; +//! - `delayMs`: how long the guest boots before it captures, 0 by default; +//! - `keepMemory`: whether the fake leaves the file that backed the RAM, as OpenVMM leaves it when +//! it ends abruptly. +//! +//! On Unix, the snapshot's directory and files are private to the current user, as OpenVMM's are; +//! on Windows, they take the access of the destination's directory, while OpenVMM gives them its +//! own, which the store's unit tests apply. +//! //! Every `--mount-deny`, `--mount-allow`, and `--mount-write` path of a launch must exist in a //! shared directory. Like OpenVMM, the fake attributes each one to the share whose root contains //! it once it has resolved both, so a root that is relative to the working directory, as @@ -131,6 +173,30 @@ const GUEST_FEATURES: GuestFeatures = GuestFeatures::from_bits( /// The guest directory at which the host mounts the aggregate export. const GUEST_EXPORT: &str = "/run/nvx/hostfs/root"; +/// Bytes in a MiB. +const MIB: u64 = 1024 * 1024; + +/// The granularity of the RAM capacity of a capture's VM, which OpenVMM requires. +const MEMORY_CAPACITY_ALIGNMENT: u64 = 128 * MIB; + +/// The size of a logical block of a sandbox block device, of which a capture's devices' lengths +/// are multiples. +const LOGICAL_BLOCK_SIZE: u64 = 512; + +/// Name of the scenario of a template's capture, next to the kernel. +const CAPTURE_SCENARIO_NAME: &str = "fake-openvmm-capture.json"; + +/// Prefix of the name of the file that backs the RAM of a capture's VM, beside the snapshot's +/// destination, which the process ID of the VMM follows. +const MEMORY_FILE_PREFIX: &str = ".openvmm-microvm-memory-"; + +/// Name of the manifest of a snapshot. +const SNAPSHOT_MANIFEST: &str = "manifest.bin"; +/// Name of the RAM of a snapshot. +const SNAPSHOT_MEMORY: &str = "memory.bin"; +/// Name of the saved state of a snapshot. +const SNAPSHOT_STATE: &str = "state.bin"; + /// `NAME=VALUE` variables in the order in which a workload's `environ` lists them. type Environment = Vec<(String, String)>; @@ -198,6 +264,20 @@ enum Lifecycle { /// boots, or restores from a snapshot, on the terminal, which OpenVMM runs until /// the guest shuts down. Run, + /// No lifecycle, sandbox block devices, and a snapshot destination: the VM of a template's + /// capture, which its guest captures, after which OpenVMM exits. + Capture, +} + +/// The options of a template's capture. +#[derive(Debug, Clone)] +struct CaptureOptions { + /// The snapshot's destination, which must not exist. + destination: PathBuf, + /// The RAM, in bytes. + memory: u64, + /// The scenario of the capture, `fake-openvmm-capture.json` next to the kernel. + scenario: PathBuf, } /// Parsed OpenVMM-compatible command line and fake-specific test controls. @@ -261,6 +341,8 @@ struct Options { workload: Vec, /// The network policy of the command line, as OpenVMM summarizes it in its report. network: NetworkSummary, + /// The options of a template's capture, for [`Lifecycle::Capture`]. + capture: Option, } /// The network policy of a command line, as the `network_policy` section of OpenVMM's outcome @@ -336,7 +418,12 @@ pub fn main() -> ExitCode { } thread::sleep(options.block_delay); // OpenVMM opens its disks as it builds the VM, and holds them until it exits. - let _blocks = match open_blocks(&options) { + let opened = if options.lifecycle == Lifecycle::Capture { + open_capture_blocks(&options) + } else { + open_blocks(&options) + }; + let _blocks = match opened { Ok(blocks) => blocks, Err(message) => { eprintln!("{program}: {message}"); @@ -346,6 +433,7 @@ pub fn main() -> ExitCode { let ended = match options.lifecycle { Lifecycle::OneShot => Some(run_one_shot(&options)), Lifecycle::Run => Some(run_microvm(&options)), + Lifecycle::Capture => Some(run_capture(&options)), Lifecycle::Managed => None, }; if let Some(status) = ended { @@ -426,6 +514,141 @@ fn open_blocks(options: &Options) -> Result, String> { Ok(files) } +/// What the fake found of a block device of a template's capture as it opened it. +#[derive(serde::Serialize)] +struct CheckedBlock { + /// Its role: `distro` or `scratch`. + role: String, + /// The path at which the fake opened it. + path: String, + /// Its length, in bytes. + length: u64, + /// Whether a handle through which a process could write the file was open as the fake opened + /// it, or `null` if the fake could not tell. + writers: Option, +} + +/// Opens the block devices of a template's capture as OpenVMM opens them, at their paths: the +/// `distro` layer read-only and the scratch disk for reading and writing, each without following +/// a link, and on Windows sharing only reads and deletion. Checks that each is a regular file +/// whose length is a nonzero multiple of 512 bytes, without reading it, and records in the blocks +/// dump each one's role, path, and length, and whether a handle through which a process could +/// write it was open. Returns the open files, which the caller holds until it exits. +fn open_capture_blocks(options: &Options) -> Result, String> { + let mut files = Vec::new(); + let mut checked = Vec::new(); + for block in &options.blocks { + let failed = |error: io::Error| { + format!( + "cannot open the {} block device {}: {error}", + block.role, + block.path.display() + ) + }; + let (file, writers) = open_unshared(&block.path, !block.read_only).map_err(failed)?; + let metadata = file.metadata().map_err(failed)?; + let length = metadata.len(); + if !metadata.is_file() || length == 0 || !length.is_multiple_of(LOGICAL_BLOCK_SIZE) { + return Err(format!( + "the {} block device {} is not a regular file whose length is a nonzero multiple \ + of {LOGICAL_BLOCK_SIZE} bytes", + block.role, + block.path.display() + )); + } + checked.push(CheckedBlock { + role: block.role.clone(), + path: block.path.display().to_string(), + length, + writers, + }); + files.push(file); + } + let _ = fs::write( + &options.blocks_dump, + serde_json::to_vec(&checked).unwrap_or_default(), + ); + Ok(files) +} + +/// Opens the file `path` without following a link, for writing too if `write` is set, and returns +/// it, with whether a handle through which a process could write it was open, as +/// [`writers_of`] finds it. +#[cfg(unix)] +fn open_unshared(path: &Path, write: bool) -> io::Result<(fs::File, Option)> { + use std::os::unix::fs::OpenOptionsExt; + + let open = |write: bool| { + OpenOptions::new() + .read(true) + .write(write) + .custom_flags(nix::libc::O_NOFOLLOW) + .open(path) + }; + let reader = open(false)?; + let writers = writers_of(&reader); + let file = if write { open(true)? } else { reader }; + Ok((file, writers)) +} + +/// Opens the file `path` as OpenVMM opens a snapshot-capable block device on Windows: the file +/// itself rather than the target of a reparse point, for writing too if `write` is set, sharing +/// only reads and deletion. The open fails while another handle may write the file, so none was +/// open once it succeeds, and none can be opened while the fake holds the file. +#[cfg(windows)] +fn open_unshared(path: &Path, write: bool) -> io::Result<(fs::File, Option)> { + use std::os::windows::fs::OpenOptionsExt; + + use windows_sys::Win32::Storage::FileSystem::{ + FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ, + }; + + let file = OpenOptions::new() + .read(true) + .write(write) + .share_mode(FILE_SHARE_READ | FILE_SHARE_DELETE) + .custom_flags(FILE_FLAG_OPEN_REPARSE_POINT) + .open(path)?; + Ok((file, Some(false))) +} + +/// Returns whether a handle, of any process, holds the file that `reader` holds open for reading +/// only open for writing: `Some(false)` once the fake has taken a read lease of the file, which +/// Linux grants no process while a handle may write the file, and has given it up; and `None` if +/// the file's file system grants no lease. A process that another started holds copies of the +/// other's handles until it runs its program, so the fake tries again for a second. +#[cfg(target_os = "linux")] +fn writers_of(reader: &fs::File) -> Option { + use std::os::fd::AsRawFd; + + use nix::libc::{EAGAIN, F_RDLCK, F_SETLEASE, F_UNLCK, fcntl}; + + let deadline = Instant::now() + Duration::from_secs(1); + loop { + // SAFETY: the descriptor is open for as long as `reader` is borrowed, and F_SETLEASE + // takes an integer. + if unsafe { fcntl(reader.as_raw_fd(), F_SETLEASE, F_RDLCK) } == 0 { + // SAFETY: as above. + unsafe { fcntl(reader.as_raw_fd(), F_SETLEASE, F_UNLCK) }; + return Some(false); + } + if io::Error::last_os_error().raw_os_error() != Some(EAGAIN) { + return None; + } + if Instant::now() >= deadline { + return Some(true); + } + thread::sleep(Duration::from_millis(10)); + } +} + +/// Returns `None`: the fake cannot tell whether a handle of another process may write a file on +/// Unix hosts other than Linux. +#[cfg(all(unix, not(target_os = "linux")))] +fn writers_of(_reader: &fs::File) -> Option { + None +} + /// Runs the entrypoint of a one-shot sandbox with its output on the fake's own, as OpenVMM runs /// the guest on its console, writes the outcome report, and returns the status with which /// OpenVMM exits: the workload's. @@ -503,6 +726,259 @@ fn run_microvm(options: &Options) -> i32 { status } +/// The scenario of a template's capture, `fake-openvmm-capture.json` next to the kernel, as the +/// module's documentation describes it. +#[derive(serde::Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct Scenario { + /// The manifest that the snapshot publishes. + manifest: PathBuf, + /// The length of the saved state that the snapshot publishes. + #[serde(default)] + state_len: u64, + /// What the capture does. + #[serde(default)] + outcome: CaptureOutcome, + /// The guest's status of [`CaptureOutcome::Fail`]. + #[serde(default = "Scenario::default_status")] + status: i32, + /// The file that marks the capture active while it runs. + #[serde(default)] + active: Option, + /// How long the guest boots before it captures, in milliseconds. + #[serde(default)] + delay_ms: u64, + /// Whether the fake leaves the file that backed the RAM. + #[serde(default)] + keep_memory: bool, +} + +impl Scenario { + /// Returns the guest's status of a failure that the scenario does not set. + fn default_status() -> i32 { + 1 + } + + /// Reads the scenario at `path`. + fn read(path: &Path) -> Result { + let bytes = fs::read(path) + .map_err(|error| format!("cannot read the scenario {}: {error}", path.display()))?; + serde_json::from_slice(&bytes) + .map_err(|error| format!("the scenario {} is malformed: {error}", path.display())) + } +} + +/// What the VM of a template's capture does, as its scenario's `outcome` asks. +#[derive(serde::Deserialize, Clone, Copy, Debug, Default, PartialEq, Eq)] +#[serde(rename_all = "kebab-case")] +enum CaptureOutcome { + /// It publishes the snapshot, and reports the capture's success. + #[default] + Publish, + /// Its guest fails before it captures. + Fail, + /// It publishes the snapshot, and then fails to tear the VM down. + PublishThenFail, + /// It never ends. + Hang, + /// It publishes the snapshot, and then never ends. + PublishThenHang, + /// It crashes, without a report. + Crash, +} + +/// The marker of a capture that runs, which its scenario's `active` names: created as the capture +/// starts, which fails if the marker exists, and removed as the capture ends. +struct Active(PathBuf); + +impl Active { + /// Marks the capture active at `path`, with the fake's process ID. + fn begin(path: &Path) -> Result { + fs::File::create_new(path) + .and_then(|mut marker| write!(marker, "{}", std::process::id())) + .map_err(|error| { + format!( + "cannot mark the capture active at {}: {error}", + path.display() + ) + })?; + Ok(Self(path.to_owned())) + } +} + +impl Drop for Active { + fn drop(&mut self) { + let _ = fs::remove_file(&self.0); + } +} + +/// Runs the VM of a template's capture as OpenVMM runs it, and returns the status with which +/// OpenVMM exits. +/// +/// The fake reads the capability, backs the RAM with a sparse file beside the snapshot's +/// destination, and, once the guest has booted for the scenario's delay, does what the scenario +/// asks: it publishes the snapshot as [`publish_snapshot`] does, removes the file that backed the +/// RAM, unless the scenario keeps it, and writes the outcome report of the capture. A capture whose +/// guest fails reports `guest-exit` and exits with its status; a VMM that cannot publish the +/// snapshot reports `vmm-failure`, and one that fails once it has published it, +/// `teardown-failure`, both with 1; and a crash exits with 3, without a report, leaving the file +/// that backed the RAM. +fn run_capture(options: &Options) -> i32 { + let fail = |message: String| { + eprintln!("{}: {message}", options.program); + 2 + }; + let Some(capture) = &options.capture else { + return fail("a capture has no capture options".to_owned()); + }; + if let Err(message) = read_capability() { + return fail(message); + } + let scenario = match Scenario::read(&capture.scenario) { + Ok(scenario) => scenario, + Err(message) => return fail(message), + }; + let _active = match scenario.active.as_deref().map(Active::begin).transpose() { + Ok(active) => active, + Err(message) => return fail(message), + }; + let directory = capture.destination.parent().unwrap_or(Path::new(".")); + let memory = directory.join(format!("{MEMORY_FILE_PREFIX}{}", std::process::id())); + if let Err(error) = create_sparse(&memory, capture.memory) { + return fail(format!( + "cannot back the RAM with {}: {error}", + memory.display() + )); + } + thread::sleep(Duration::from_millis(scenario.delay_ms)); + let (category, status) = match scenario.outcome { + CaptureOutcome::Crash => return 3, + CaptureOutcome::Hang => hang(), + CaptureOutcome::Fail => ("guest-exit", scenario.status), + publishing => { + if let Err(error) = publish_snapshot(capture, &scenario, &memory) { + eprintln!( + "{}: cannot publish the snapshot {}: {error}", + options.program, + capture.destination.display() + ); + ("vmm-failure", 1) + } else if publishing == CaptureOutcome::PublishThenHang { + hang() + } else if publishing == CaptureOutcome::PublishThenFail { + ("teardown-failure", 1) + } else { + ("success", 0) + } + } + }; + if !scenario.keep_memory { + let _ = fs::remove_file(&memory); + } + write_report(options, "capture", category, status); + status +} + +/// Never returns, as a VM that never ends. +fn hang() -> ! { + loop { + thread::sleep(Duration::from_secs(3600)); + } +} + +/// Publishes the snapshot of a capture as OpenVMM publishes it: stages a copy of the scenario's +/// manifest, a sparse saved state of the scenario's length, and the RAM, a hard link to the file +/// `memory` that backed it, in a directory private to the current user beside the destination, and +/// renames it to the destination, which must not exist. +fn publish_snapshot( + capture: &CaptureOptions, + scenario: &Scenario, + memory: &Path, +) -> io::Result<()> { + let destination = &capture.destination; + let name = destination + .file_name() + .ok_or_else(|| io::Error::other("the destination has no name"))?; + let staging = destination.with_file_name(format!( + ".{}.staging-{}-0-0", + name.to_string_lossy(), + std::process::id() + )); + create_private_dir(&staging)?; + let manifest = fs::read(&scenario.manifest)?; + create_private_file(&staging.join(SNAPSHOT_MANIFEST))?.write_all(&manifest)?; + create_sparse(&staging.join(SNAPSHOT_STATE), scenario.state_len)?; + fs::hard_link(memory, staging.join(SNAPSHOT_MEMORY))?; + if fs::symlink_metadata(destination).is_ok() { + return Err(io::Error::from(io::ErrorKind::AlreadyExists)); + } + fs::rename(&staging, destination) +} + +/// Creates the directory `path`, private to the current user on Unix; on Windows, it takes the +/// access of its parent. +fn create_private_dir(path: &Path) -> io::Result<()> { + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + + fs::DirBuilder::new().mode(0o700).create(path) + } + #[cfg(not(unix))] + fs::create_dir(path) +} + +/// Creates the file `path`, which must not exist, for writing, private to the current user on +/// Unix; on Windows, it takes the access of its directory. +fn create_private_file(path: &Path) -> io::Result { + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, 0o600); + options.open(path) +} + +/// Creates the file `path`, which must not exist, as [`create_private_file`] does, as a sparse +/// file of `length` bytes. +fn create_sparse(path: &Path, length: u64) -> io::Result<()> { + let file = create_private_file(path)?; + #[cfg(windows)] + set_sparse(&file)?; + file.set_len(length) +} + +/// Marks the file that `file` holds open for writing sparse, which NTFS otherwise allocates whole +/// as the file is extended. +#[cfg(windows)] +fn set_sparse(file: &fs::File) -> io::Result<()> { + use std::os::windows::io::AsRawHandle; + + use windows_sys::Win32::Foundation::HANDLE; + use windows_sys::Win32::System::IO::DeviceIoControl; + + /// `FSCTL_SET_SPARSE`, in `winioctl.h`, which marks a file sparse without a buffer. + const FSCTL_SET_SPARSE: u32 = 0x0009_00c4; + let mut returned = 0; + // SAFETY: the handle is open for as long as `file` is borrowed, the control takes no input or + // output buffer, and the count's pointer is valid. + let set = unsafe { + DeviceIoControl( + file.as_raw_handle() as HANDLE, + FSCTL_SET_SPARSE, + std::ptr::null(), + 0, + std::ptr::null_mut(), + 0, + &mut returned, + std::ptr::null_mut(), + ) + }; + if set == 0 { + return Err(io::Error::last_os_error()); + } + Ok(()) +} + /// Returns the name of the running executable, such as `nvx-fake-openvmm`. fn program_name() -> String { std::env::args_os() @@ -547,7 +1023,7 @@ fn parse(program: &str, arguments: &[String]) -> Result { "--host-loopback", "--host-loopback-forward", ]; - // The options that only the microVMs of `nvx-rs run` take. + // The options that only the microVMs of `nvx-rs run` and of captures take. const RUN_VALUED: [&str; 7] = [ "--processors", "--cpu-profile", @@ -557,15 +1033,25 @@ fn parse(program: &str, arguments: &[String]) -> Result { "--restore-memory", "--restore-ready-path", ]; - // `nvx-rs run` boots or restores its microVM without a lifecycle, and is no sandbox. + // The options that only the VMs of templates' captures take. + const CAPTURE_VALUED: [&str; 4] = [ + "--snapshot-destination", + "--snapshot-tier", + "--snapshot-block-identity", + "--snapshot-generation-id", + ]; + // `nvx-rs run` boots or restores its microVM without a lifecycle, and is no sandbox; a + // capture boots its VM without a lifecycle, with sandbox block devices. let mut option_arguments = arguments.iter(); let mut run = true; + let mut capture = false; while let Some(argument) = option_arguments.next() { let name = argument.as_str(); if name == "--microvm-lifecycle" || name == "--microvm-sandbox-block" { run = false; } - if VALUED.contains(&name) || RUN_VALUED.contains(&name) { + capture |= name == "--snapshot-destination"; + if VALUED.contains(&name) || RUN_VALUED.contains(&name) || CAPTURE_VALUED.contains(&name) { let _ = option_arguments.next(); } } @@ -576,7 +1062,10 @@ fn parse(program: &str, arguments: &[String]) -> Result { let name = argument.as_str(); if name == "--single-process" || name == "--microvm-control-auth-stdin" { flags.insert(name); - } else if VALUED.contains(&name) || (run && RUN_VALUED.contains(&name)) { + } else if VALUED.contains(&name) + || ((run || capture) && RUN_VALUED.contains(&name)) + || (capture && CAPTURE_VALUED.contains(&name)) + { let value = iterator .next() .ok_or_else(|| format!("{name} requires a value"))?; @@ -598,7 +1087,12 @@ fn parse(program: &str, arguments: &[String]) -> Result { // The sandbox profile of `nvx-rs sandbox` puts its layers and scratch disk on sandbox block // devices, and may run one-shot. let sandbox = values.contains_key("--microvm-sandbox-block"); - let lifecycle = if run { + let lifecycle = if capture { + if values.contains_key("--microvm-lifecycle") { + return Err("a capture takes no --microvm-lifecycle".to_owned()); + } + Lifecycle::Capture + } else if run { Lifecycle::Run } else { match single("--microvm-lifecycle")? { @@ -611,7 +1105,7 @@ fn parse(program: &str, arguments: &[String]) -> Result { return Err("--single-process is required".to_owned()); } let (endpoint, report) = match lifecycle { - Lifecycle::Managed => { + Lifecycle::Managed | Lifecycle::Capture => { expect("--virtio-console", "none")?; if !flags.contains("--microvm-control-auth-stdin") { return Err("--microvm-control-auth-stdin is required".to_owned()); @@ -620,6 +1114,11 @@ fn parse(program: &str, arguments: &[String]) -> Result { .strip_prefix("listen=") .ok_or("--microvm-control-console must use listen=")? .to_owned(); + if endpoint.is_empty() { + return Err( + "--microvm-control-console must name its endpoint after listen=".to_owned(), + ); + } (endpoint, Some(PathBuf::from(single("--microvm-report")?))) } Lifecycle::OneShot | Lifecycle::Run => { @@ -648,18 +1147,45 @@ fn parse(program: &str, arguments: &[String]) -> Result { if !["kvm", "mshv", "whp"].contains(&hypervisor) { return Err(format!("unsupported hypervisor {hypervisor}")); } - // A run uses the default workload identity of the guest, and records its command line next to - // the kernel of a cold boot or to the snapshot of a restore. - let (workload_uid, anchor, restore) = if run { + // A run, or a capture, uses the default workload identity of the guest, and records its + // command line next to the kernel of a cold boot or to the snapshot of a restore. + let (workload_uid, anchor, restore) = if run || capture { if values.contains_key("--microvm-workload-identity") { - return Err("a microVM run takes no --microvm-workload-identity".to_owned()); + return Err(format!( + "{} takes no --microvm-workload-identity", + if capture { + "a capture" + } else { + "a microVM run" + } + )); } let (anchor, restore) = check_run_boot(&values)?; + if capture && restore.is_some() { + return Err("a capture boots its VM cold".to_owned()); + } (65534, anchor, restore) } else { let (workload_uid, kernel) = check_boot(&values)?; (workload_uid, kernel, None) }; + let capture_options = if capture { + if !sandbox { + return Err("a capture takes sandbox block devices".to_owned()); + } + Some(check_capture(&values, &anchor)?) + } else { + None + }; + // OpenVMM checks where a control console listens before it starts the VM, beside the snapshot + // that the VM restores, or else captures, if any. + #[cfg(unix)] + if matches!(lifecycle, Lifecycle::Managed | Lifecycle::Capture) { + let snapshot = ["--restore-snapshot", "--snapshot-destination"] + .into_iter() + .find_map(|option| values.get(option).and_then(|given| given.first())); + check_control_console(&endpoint, snapshot.map(|snapshot| Path::new(*snapshot)))?; + } if let Some(report) = &report && report.exists() { @@ -684,48 +1210,72 @@ fn parse(program: &str, arguments: &[String]) -> Result { } let blocks = check_sandbox_blocks(values.get("--microvm-sandbox-block").into_iter().flatten())?; - let mut workload: Vec = match token_values("nvx_entrypoint=").as_slice() { - [entrypoint] => vec![(*entrypoint).to_owned()], - _ => return Err("the kernel command line must set one nvx_entrypoint".to_owned()), - }; - workload.extend(token_values("nvx_arg=").into_iter().map(str::to_owned)); - if token_values("nvx_hostname=").len() != 1 { - return Err("the kernel command line must set one nvx_hostname".to_owned()); - } - let (maps, children, share_targets) = match lifecycle { - // A managed sandbox's guest agent maps its shares from the export, as the mapping - // table of the host's `MAPS` requests orders. - Lifecycle::Managed => { - if values.contains_key("--mount") { - return Err("a managed sandbox takes no --mount".to_owned()); - } - if !token_values("nvx_share=").is_empty() { - return Err("a managed sandbox takes no nvx_share=".to_owned()); + if capture { + // A template's VM boots the invariant configuration alone, without a tenant. + if tokens != ["nvx_sandbox=1"] { + return Err("a capture's kernel command line is nvx_sandbox=1 alone".to_owned()); + } + let roles: BTreeSet<&str> = blocks.iter().map(|block| block.role.as_str()).collect(); + if roles != BTreeSet::from(["distro", "scratch"]) { + return Err("a capture takes a distro layer and a scratch disk".to_owned()); + } + for option in [ + "--mount", + "--mount-aggregate", + "--mount-child", + "--mount-owner", + ] { + if values.contains_key(option) { + return Err(format!("a capture takes no {option}")); } - let maps = mapping_count(&token_values(MAPS_TOKEN))?; - (maps, mapped_children(&values, maps)?, Vec::new()) } - // A one-shot sandbox's shell binds them at boot, as their tokens place them. - Lifecycle::OneShot => { - if !token_values(MAPS_TOKEN).is_empty() { - return Err(format!("a one-shot sandbox takes no {MAPS_TOKEN}")); + (0, Vec::new(), Vec::new(), Vec::new(), blocks) + } else { + let mut workload: Vec = match token_values("nvx_entrypoint=").as_slice() { + [entrypoint] => vec![(*entrypoint).to_owned()], + _ => return Err("the kernel command line must set one nvx_entrypoint".to_owned()), + }; + workload.extend(token_values("nvx_arg=").into_iter().map(str::to_owned)); + if token_values("nvx_hostname=").len() != 1 { + return Err("the kernel command line must set one nvx_hostname".to_owned()); + } + let (maps, children, share_targets) = match lifecycle { + // A managed sandbox's guest agent maps its shares from the export, as the mapping + // table of the host's `MAPS` requests orders. + Lifecycle::Managed => { + if values.contains_key("--mount") { + return Err("a managed sandbox takes no --mount".to_owned()); + } + if !token_values("nvx_share=").is_empty() { + return Err("a managed sandbox takes no nvx_share=".to_owned()); + } + let maps = mapping_count(&token_values(MAPS_TOKEN))?; + (maps, mapped_children(&values, maps)?, Vec::new()) } - let (children, mut share_targets) = sandbox_shares(&values)?; - // The guest binds each directory child of an aggregate at the target of its - // `nvx_share=NAME,TARGET,ACCESS[,file]` token. - for share in token_values("nvx_share=") { - match share.split(',').collect::>().as_slice() { - [_, target, _] => share_targets.push((*target).to_owned()), - [_, _, _, "file"] => {} - _ => return Err(format!("invalid nvx_share={share}")), + // A one-shot sandbox's shell binds them at boot, as their tokens place them. + Lifecycle::OneShot => { + if !token_values(MAPS_TOKEN).is_empty() { + return Err(format!("a one-shot sandbox takes no {MAPS_TOKEN}")); } + let (children, mut share_targets) = sandbox_shares(&values)?; + // The guest binds each directory child of an aggregate at the target of its + // `nvx_share=NAME,TARGET,ACCESS[,file]` token. + for share in token_values("nvx_share=") { + match share.split(',').collect::>().as_slice() { + [_, target, _] => share_targets.push((*target).to_owned()), + [_, _, _, "file"] => {} + _ => return Err(format!("invalid nvx_share={share}")), + } + } + (0, children, share_targets) } - (0, children, share_targets) - } - // A sandbox block device makes a command line no run. - Lifecycle::Run => return Err("a microVM run takes no sandbox block device".to_owned()), - }; - (maps, children, share_targets, workload, blocks) + // A sandbox block device makes a command line no run, and a capture was handled above. + Lifecycle::Run | Lifecycle::Capture => { + return Err("a microVM run takes no sandbox block device".to_owned()); + } + }; + (maps, children, share_targets, workload, blocks) + } } else if run { // A run shares as a sandbox does, and its kernel command line is its user's. let (children, _) = sandbox_shares(&values)?; @@ -843,6 +1393,7 @@ fn parse(program: &str, arguments: &[String]) -> Result { share_targets, workload, network: network_summary(&values), + capture: capture_options, }) } @@ -951,6 +1502,135 @@ fn check_run_boot( Ok((kernel, None)) } +/// Checks the options of a template's capture, besides those of its boot, and returns them: RAM +/// and a RAM capacity in MiB, the capacity a multiple of 128 MiB no smaller than the RAM, a +/// `--snapshot-destination` that does not exist, in an existing directory, `--snapshot-tier +/// platform`, `--snapshot-block-identity generation`, and a `--snapshot-generation-id` of 32 +/// hexadecimal digits that are not all zero. The scenario of the capture is next to `kernel`. +fn check_capture( + values: &BTreeMap<&str, Vec<&str>>, + kernel: &Path, +) -> Result { + let bytes = |name: &str| { + let value = exactly_once(values, name)?; + value + .strip_suffix('M') + .and_then(|mebibytes| mebibytes.parse::().ok()) + .and_then(|mebibytes| mebibytes.checked_mul(MIB)) + .filter(|bytes| *bytes != 0) + .ok_or_else(|| format!("invalid {name} {value}")) + }; + let memory = bytes("--memory")?; + let capacity = bytes("--memory-capacity")?; + if !capacity.is_multiple_of(MEMORY_CAPACITY_ALIGNMENT) || capacity < memory { + return Err( + "--memory-capacity must be a multiple of 128 MiB no smaller than --memory".to_owned(), + ); + } + for (name, expected) in [ + ("--snapshot-tier", "platform"), + ("--snapshot-block-identity", "generation"), + ] { + let value = exactly_once(values, name)?; + if value != expected { + return Err(format!("{name} must be {expected}, not {value}")); + } + } + let generation = exactly_once(values, "--snapshot-generation-id")?; + if generation.len() != 32 + || !generation.bytes().all(|digit| digit.is_ascii_hexdigit()) + || generation.bytes().all(|digit| digit == b'0') + { + return Err(format!("invalid --snapshot-generation-id {generation}")); + } + let destination = PathBuf::from(exactly_once(values, "--snapshot-destination")?); + if !destination.parent().is_some_and(Path::is_dir) { + return Err(format!( + "the directory of the snapshot destination {} does not exist", + destination.display() + )); + } + if fs::symlink_metadata(&destination).is_ok() { + return Err(format!( + "the snapshot destination {} exists", + destination.display() + )); + } + Ok(CaptureOptions { + destination, + memory, + scenario: kernel.with_file_name(CAPTURE_SCENARIO_NAME), + }) +} + +/// Checks the socket `endpoint` of a control console as OpenVMM checks it on Unix before it +/// starts the VM: its directory, once its path is absolute, must be a directory, not a link, that +/// belongs to the effective user, with exactly the mode 0700, and, if the VM restores or captures +/// the snapshot directory `snapshot`, the canonical directory that holds it, in which a restore of +/// the snapshot listens again. +#[cfg(unix)] +fn check_control_console(endpoint: &str, snapshot: Option<&Path>) -> Result<(), String> { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + + let endpoint = Path::new(endpoint); + let absolute = if endpoint.is_absolute() { + endpoint.to_owned() + } else { + std::env::current_dir() + .map_err(|error| { + format!("failed to resolve current directory for control endpoint: {error}") + })? + .join(endpoint) + }; + let directory = absolute + .parent() + .ok_or("microVM control endpoint has no parent directory")?; + let metadata = fs::symlink_metadata(directory).map_err(|error| { + format!( + "failed to inspect microVM control endpoint parent {}: {error}", + directory.display() + ) + })?; + if !metadata.is_dir() + || metadata.file_type().is_symlink() + || metadata.uid() != nix::unistd::geteuid().as_raw() + || metadata.permissions().mode() & 0o7777 != 0o700 + { + return Err( + "microVM control endpoint parent must be a non-symlink directory owned by \ + OpenVMM with mode 0700" + .to_owned(), + ); + } + let Some(snapshot) = snapshot else { + return Ok(()); + }; + // OpenVMM finds the socket's identity in the canonical directory of its path. + let directory = fs::canonicalize(directory).map_err(|error| { + format!( + "failed to canonicalize microVM console endpoint parent {}: {error}", + directory.display() + ) + })?; + let parent = snapshot + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + .unwrap_or(Path::new(".")); + let namespace = fs::canonicalize(parent).map_err(|error| { + format!( + "failed to canonicalize snapshot parent {}: {error}", + parent.display() + ) + })?; + if directory != namespace { + return Err(format!( + "microVM console socket must be inside the snapshot parent namespace {}", + namespace.display() + )); + } + Ok(()) +} + /// Checks the sandbox block devices of a sandbox: each layer, `ROLE:file:PATH,ro` with a role of /// `distro`, `runtime`, or `custom`, at most once each, and one scratch disk, `scratch:file:PATH`, /// each an existing regular file, and returns them in order. @@ -2141,4 +2821,357 @@ mod tests { ); } } + + /// The command line of a template's capture, with `{dir}` in place of the directory of + /// [`parse_capture`]. + const CAPTURE: &[&str] = &[ + "--machine", + "microvm", + "--microvm-sandbox-block", + "distro:file:{dir}/placeholder,ro", + "--microvm-sandbox-block", + "scratch:file:{dir}/transaction/scratch.raw", + "--processors", + "2", + "--memory", + "512M", + "--memory-capacity", + "2048M", + "--snapshot-destination", + "{dir}/transaction/snapshot", + "--snapshot-tier", + "platform", + "--snapshot-block-identity", + "generation", + "--snapshot-generation-id", + "0123456789abcdef0123456789ABCDEF", + "--single-process", + "--hypervisor", + "kvm", + "--cpu-profile", + "auto", + "--kernel", + "{dir}/vmlinux", + "--initrd", + "{dir}/initramfs.cpio.gz", + "--cmdline", + "nvx_sandbox=1", + "--virtio-console", + "none", + "--microvm-control-console", + "listen={dir}/transaction/control.sock", + "--microvm-control-auth-stdin", + "--microvm-report", + "{dir}/transaction/outcome.json", + ]; + + /// Returns a directory with a kernel, an initramfs, a placeholder, and a transaction with a + /// scratch disk, the directory and the transaction private to the current user, as the + /// directory of a control console's socket must be, and the parse of the command line that + /// `arguments` gives with the path of that directory in place of `{dir}`. + fn parse_capture(arguments: &[&str]) -> (tempfile::TempDir, Result) { + parse_capture_in(arguments, |_| {}) + } + + /// Returns the directory of [`parse_capture`], and the parse of `arguments` there once + /// `prepare` has changed the directory. + fn parse_capture_in( + arguments: &[&str], + prepare: impl FnOnce(&Path), + ) -> (tempfile::TempDir, Result) { + let temporary = tempfile::tempdir().unwrap(); + let root = temporary.path(); + fs::write(root.join("vmlinux"), b"kernel").unwrap(); + fs::write(root.join("initramfs.cpio.gz"), b"initrd").unwrap(); + fs::write(root.join("placeholder"), [0; 512]).unwrap(); + fs::create_dir(root.join("transaction")).unwrap(); + fs::write(root.join("transaction").join("scratch.raw"), [0; 512]).unwrap(); + #[cfg(unix)] + for directory in [root.to_owned(), root.join("transaction")] { + use std::os::unix::fs::PermissionsExt; + fs::set_permissions(directory, fs::Permissions::from_mode(0o700)).unwrap(); + } + prepare(root); + let directory = root.to_str().unwrap(); + let arguments: Vec = arguments + .iter() + .map(|argument| argument.replace("{dir}", directory)) + .collect(); + let options = parse("openvmm", &arguments); + (temporary, options) + } + + /// Returns the command line of [`CAPTURE`] with `value` in place of the value of `option`. + fn capture_with(option: &str, value: &'static str) -> Vec<&'static str> { + let mut arguments = CAPTURE.to_vec(); + let at = arguments.iter().position(|given| *given == option).unwrap(); + arguments[at + 1] = value; + arguments + } + + #[test] + fn captures_boot_their_template_cold_on_their_block_devices() { + let (temporary, options) = parse_capture(CAPTURE); + let options = options.unwrap(); + let root = temporary.path(); + assert_eq!(options.lifecycle, Lifecycle::Capture); + assert!(options.sandbox); + assert_eq!(options.restore, None); + let blocks: Vec<_> = options + .blocks + .iter() + .map(|block| (block.role.as_str(), block.path.clone(), block.read_only)) + .collect(); + assert_eq!( + blocks, + [ + ("distro", root.join("placeholder"), true), + ( + "scratch", + root.join("transaction").join("scratch.raw"), + false + ), + ] + ); + let capture = options.capture.as_ref().unwrap(); + assert_eq!( + capture.destination, + root.join("transaction").join("snapshot") + ); + assert_eq!(capture.memory, 512 * MIB); + assert_eq!(capture.scenario, root.join(CAPTURE_SCENARIO_NAME)); + assert_eq!( + options.endpoint, + format!("{}/transaction/control.sock", root.to_str().unwrap()) + ); + assert_eq!( + options.blocks_dump, + root.join("fake-openvmm-transaction-blocks.json") + ); + assert!(options.workload.is_empty()); + assert!(options.children.is_empty()); + } + + #[cfg(unix)] + #[test] + fn captures_listen_in_the_directory_of_their_snapshot_destination() { + let (temporary, options) = parse_capture(&capture_with( + "--microvm-control-console", + "listen={dir}/control.sock", + )); + let namespace = fs::canonicalize(temporary.path().join("transaction")).unwrap(); + assert_eq!( + options.err(), + Some(format!( + "microVM console socket must be inside the snapshot parent namespace {}", + namespace.display() + )) + ); + } + + #[cfg(unix)] + #[test] + fn control_consoles_listen_in_a_private_directory_of_their_user() { + use std::os::unix::fs::{PermissionsExt, symlink}; + + let refusal = "microVM control endpoint parent must be a non-symlink directory owned by \ + OpenVMM with mode 0700"; + // OpenVMM takes exactly the mode 0700. + for mode in [0o750, 0o701, 0o1700] { + let (_temporary, options) = parse_capture_in(CAPTURE, |root| { + let transaction = root.join("transaction"); + fs::set_permissions(transaction, fs::Permissions::from_mode(mode)).unwrap(); + }); + assert_eq!(options.err().as_deref(), Some(refusal), "{mode:o}"); + } + // It follows no link to the socket's directory, even to a private one. + let (_temporary, options) = parse_capture_in( + &capture_with( + "--microvm-control-console", + "listen={dir}/link/control.sock", + ), + |root| symlink(root.join("transaction"), root.join("link")).unwrap(), + ); + assert_eq!(options.err().as_deref(), Some(refusal)); + let (temporary, options) = parse_capture(&capture_with( + "--microvm-control-console", + "listen={dir}/missing/control.sock", + )); + let missing = format!( + "failed to inspect microVM control endpoint parent {}: ", + temporary.path().join("missing").display() + ); + let error = options.err().unwrap(); + assert!(error.starts_with(&missing), "{error}"); + } + + #[cfg(unix)] + #[test] + fn a_restore_listens_in_the_directory_of_the_snapshot_that_it_restores() { + use std::os::unix::fs::PermissionsExt; + + let temporary = tempfile::tempdir().unwrap(); + let (clone, generation) = ( + temporary.path().join("clone"), + temporary.path().join("generation"), + ); + for directory in [&clone, &generation] { + fs::create_dir(directory).unwrap(); + fs::set_permissions(directory, fs::Permissions::from_mode(0o700)).unwrap(); + } + let endpoint = clone.join("control.sock"); + let endpoint = endpoint.to_str().unwrap(); + assert_eq!( + check_control_console(endpoint, Some(&clone.join("snapshot"))), + Ok(()) + ); + // A VM that restores a snapshot elsewhere cannot listen in its own directory. + assert_eq!( + check_control_console(endpoint, Some(&generation.join("snapshot"))), + Err(format!( + "microVM console socket must be inside the snapshot parent namespace {}", + fs::canonicalize(&generation).unwrap().display() + )) + ); + // Without a snapshot, the socket's directory alone counts. + assert_eq!(check_control_console(endpoint, None), Ok(())); + } + + #[test] + fn captures_refuse_what_a_template_does_not_take() { + let mut lifecycle = CAPTURE.to_vec(); + lifecycle.extend(["--microvm-lifecycle", "managed"]); + let mut runtime = CAPTURE.to_vec(); + runtime.extend([ + "--microvm-sandbox-block", + "runtime:file:{dir}/placeholder,ro", + ]); + let mut aggregate = CAPTURE.to_vec(); + aggregate.extend(["--mount-aggregate", "/run/nvx/shares"]); + let mut identity = CAPTURE.to_vec(); + identity.extend(["--microvm-workload-identity", "1000:1000"]); + let mut unbounded = CAPTURE.to_vec(); + let at = unbounded + .iter() + .position(|given| *given == "--memory-capacity") + .unwrap(); + unbounded.drain(at..at + 2); + let mut without_blocks = CAPTURE.to_vec(); + without_blocks.drain(2..6); + let invalid_generation = "invalid --snapshot-generation-id"; + let capacity = "--memory-capacity must be a multiple of 128 MiB no smaller than --memory"; + let cases: Vec<(Vec<&str>, String)> = + vec![ + ( + lifecycle, + "a capture takes no --microvm-lifecycle".to_owned(), + ), + ( + runtime, + "a capture takes a distro layer and a scratch disk".to_owned(), + ), + (aggregate, "a capture takes no --mount-aggregate".to_owned()), + ( + identity, + "a capture takes no --microvm-workload-identity".to_owned(), + ), + ( + unbounded, + "--memory-capacity must be given exactly once".to_owned(), + ), + ( + without_blocks, + "a capture takes sandbox block devices".to_owned(), + ), + ( + capture_with("--memory-capacity", "2000M"), + capacity.to_owned(), + ), + (capture_with("--memory-capacity", "256M"), capacity.to_owned()), + ( + capture_with("--snapshot-tier", "process"), + "--snapshot-tier must be platform, not process".to_owned(), + ), + ( + capture_with("--snapshot-block-identity", "path"), + "--snapshot-block-identity must be generation, not path".to_owned(), + ), + ( + capture_with("--snapshot-generation-id", "00000000000000000000000000000000"), + format!("{invalid_generation} 00000000000000000000000000000000"), + ), + ( + capture_with("--snapshot-generation-id", "0123456789abcdef0123456789abcde"), + format!("{invalid_generation} 0123456789abcdef0123456789abcde"), + ), + ( + capture_with("--snapshot-generation-id", "0123456789abcdef0123456789abcdeg"), + format!("{invalid_generation} 0123456789abcdef0123456789abcdeg"), + ), + ( + capture_with("--cmdline", "nvx_sandbox=1 nvx_hostname=tenant"), + "a capture's kernel command line is nvx_sandbox=1 alone".to_owned(), + ), + ( + capture_with("--snapshot-destination", "{dir}/transaction"), + "the snapshot destination {dir}/transaction exists".to_owned(), + ), + ( + capture_with("--snapshot-destination", "{dir}/missing/snapshot"), + "the directory of the snapshot destination {dir}/missing/snapshot does not exist" + .to_owned(), + ), + ( + capture_with("--virtio-console", "stdio"), + "--virtio-console must be none, not stdio".to_owned(), + ), + ( + capture_with("--microvm-control-console", "listen="), + "--microvm-control-console must name its endpoint after listen=".to_owned(), + ), + ]; + for (arguments, message) in cases { + let (temporary, options) = parse_capture(&arguments); + let message = message.replace("{dir}", temporary.path().to_str().unwrap()); + assert_eq!(options.err(), Some(message), "{arguments:?}"); + } + } + + #[test] + fn capture_scenarios_take_defaults_and_refuse_unknown_fields() { + let scenario: Scenario = serde_json::from_str(r#"{"manifest": "manifest.bin"}"#).unwrap(); + assert_eq!(scenario.manifest, Path::new("manifest.bin")); + assert_eq!(scenario.state_len, 0); + assert_eq!(scenario.outcome, CaptureOutcome::Publish); + assert_eq!(scenario.status, 1); + assert_eq!(scenario.active, None); + assert_eq!(scenario.delay_ms, 0); + assert!(!scenario.keep_memory); + let scenario: Scenario = serde_json::from_str( + r#"{"manifest": "m", "stateLen": 512, "outcome": "publish-then-hang", "status": 3, + "active": "a", "delayMs": 5, "keepMemory": true}"#, + ) + .unwrap(); + assert_eq!( + ( + scenario.state_len, + scenario.outcome, + scenario.status, + scenario.delay_ms, + scenario.keep_memory + ), + (512, CaptureOutcome::PublishThenHang, 3, 5, true) + ); + assert_eq!(scenario.active.as_deref(), Some(Path::new("a"))); + for refused in [ + r#"{}"#, + r#"{"manifest": "m", "outcome": "succeed"}"#, + r#"{"manifest": "m", "memory": 1}"#, + ] { + assert!( + serde_json::from_str::(refused).is_err(), + "{refused}" + ); + } + } } diff --git a/nvx-rs/tests/template_capture.rs b/nvx-rs/tests/template_capture.rs new file mode 100644 index 000000000..0036b1c10 --- /dev/null +++ b/nvx-rs/tests/template_capture.rs @@ -0,0 +1,697 @@ +//! Captures of templates through the supervisor, with the fake OpenVMM as their VMM: the block +//! devices that the VMM opens, which no handle of the host may write as it starts, the +//! generations that captures publish and that creations then serve, VMs that end without a +//! capture or never end, captures of one class that race, and an adapter that crashes while its +//! capture's VM runs, whose VMM a recovery ends. +//! +//! The fake OpenVMM publishes the snapshot whose manifest OpenVMM wrote for the fixture +//! `tests/fixtures/template/platform-kvm`, or fails, as the scenario next to the kernel asks; see +//! `nvx::testing::fake_openvmm`. The crash is real: a child process of this test binary, the +//! adapter, captures in the store, and the test kills it while its capture's VM runs. +#![cfg(all( + feature = "host", + feature = "testing", + any(target_os = "linux", windows) +))] + +use std::env; +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::sync::{Mutex, MutexGuard, PoisonError}; +use std::thread; +use std::time::{Duration, Instant}; + +use nvx::host::platform; +use nvx::host::process::{BootId, Process, ProcessRef}; +use nvx::host::sandbox::distro::{DISTRO_POLICY_REVISION, DistroBinding, PolicyVerdict}; +use nvx::host::sandbox::{LayerRole, SandboxDistro, SandboxLayer, SandboxPlan, SandboxUpper}; +#[cfg(target_os = "linux")] +use nvx::host::store::SOCKET_NAME; +use nvx::host::store::{ArtifactIdentity, OUTCOME_NAME, RuntimeIdentity, SandboxStore}; +use nvx::host::template::{ + BlockGeometry, CAPTURES_NAME, Capture, CaptureClass, Created, GENERATIONS_NAME, Generation, + HostTime, Invariants, SCRATCH_NAME, SNAPSHOT_NAME, Store, TRANSACTION_NAME, TemplateKey, + TemplateManifest, +}; +use nvx::protocol::root::RootImage; +use nvx::uuid::Uuid; +use nvx::{Artifacts, ErrorCode, ErrorKind, Hypervisor}; +use serde_json::{Value, json}; + +/// Bytes in a MiB. +const MIB: u64 = 1024 * 1024; +/// Nanoseconds in a second. +const SECOND_NS: u64 = 1_000_000_000; +/// The identity of the placeholder. +const IDENTITY: &str = "11111111-1111-1111-1111-111111111111"; +/// Name of the scenario of the fake OpenVMM's captures, next to the kernel. +const SCENARIO_NAME: &str = "fake-openvmm-capture.json"; +/// Name of the marker that the fake OpenVMM holds while a capture's VM runs, in the platform's +/// directory. +const ACTIVE_NAME: &str = "active"; +/// The longest wait of these tests for a capture, a recovery, or a process. +const TIMEOUT: Duration = Duration::from_secs(60); +/// Environment variable that marks the adapter that [`adapter_role`] runs, and names the +/// directory of its bench. +const ADAPTER_BENCH: &str = "NVX_TEMPLATE_CAPTURE_TEST_BENCH"; + +/// Serializes the tests that spawn processes: a process that one test spawns while another test's +/// log is inheritable for its own spawn could hold that log's claim for a while. +static SPAWNS: Mutex<()> = Mutex::new(()); + +/// Takes [`SPAWNS`]. +fn serialized() -> MutexGuard<'static, ()> { + SPAWNS.lock().unwrap_or_else(PoisonError::into_inner) +} + +/// Returns the directory of the fixture whose manifest OpenVMM wrote for a capture of [`class`]. +fn fixture() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/template/platform-kvm") +} + +/// Returns the manifest of [`fixture`]. +fn manifest() -> TemplateManifest { + TemplateManifest::read(&fixture()).unwrap() +} + +/// Returns the identity `sha256:` and 64 copies of `digit`. +fn artifact(digit: char) -> ArtifactIdentity { + ArtifactIdentity::sha256(&digit.to_string().repeat(64)).unwrap() +} + +/// Returns the class of the capture of [`fixture`]. +fn class() -> CaptureClass { + CaptureClass { + runtime: RuntimeIdentity::new(artifact('a'), artifact('b'), artifact('c')), + guest_build: ArtifactIdentity::new("guest-build-1").unwrap(), + hypervisor: Hypervisor::Kvm, + cpu_profile: "auto".to_owned(), + processors: 2, + boot_processors: 2, + memory: 512 * MIB, + memory_capacity: 2048 * MIB, + distro: BlockGeometry { + length: 1024 * MIB, + logical_block_size: 512, + physical_block_size: 4096, + }, + scratch: BlockGeometry { + length: 64 * MIB, + logical_block_size: 512, + physical_block_size: 512, + }, + network: None, + invariants: Invariants::new(), + } +} + +/// Returns the host's time `seconds` after the capture of [`fixture`], on the host and in the +/// boot that captured it. +fn after(seconds: u64) -> HostTime { + let manifest = manifest(); + let time = manifest.time(); + HostTime { + host_id: time.host_id, + boot_id: time.boot_id, + clock: time.host_clock, + monotonic_ns: time.capture_monotonic_ns + seconds * SECOND_NS, + utc_ns: time.capture_utc_ns + seconds * SECOND_NS, + } +} + +/// Returns a deadline that no capture of these tests reaches. +fn deadline() -> Instant { + Instant::now() + TIMEOUT +} + +/// Creates the directory `path`, private to the current user: on Windows, with a protected list +/// that allows the owner and SYSTEM alone, which the files created in it inherit. +fn private_dir(path: &Path) { + fs::create_dir(path).unwrap(); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + + fs::set_permissions(path, fs::Permissions::from_mode(0o700)).unwrap(); + } + #[cfg(windows)] + windows::set_access(path, "D:P(A;OICI;FA;;;OW)(A;OICI;FA;;;SY)"); +} + +/// Creates the sparse file `path` of `length` bytes, which no one else may write, whatever the +/// umask. +fn sparse(path: &Path, length: u64) { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, 0o600); + let file = options.open(path).unwrap(); + #[cfg(windows)] + windows::set_sparse(&file); + file.set_len(length).unwrap(); +} + +/// Returns the file at `path` as JSON. +fn json_of(path: &Path) -> Value { + serde_json::from_slice(&fs::read(path).unwrap()).unwrap() +} + +/// Returns the names of the entries of `dir`, sorted. +fn sorted(dir: &Path) -> Vec { + let mut names: Vec = fs::read_dir(dir) + .unwrap() + .map(|entry| entry.unwrap().file_name().into_string().unwrap()) + .collect(); + names.sort(); + names +} + +/// Returns the store generation of `generation`, as a number. +fn number(generation: &Generation) -> u64 { + generation.generation().get().get() +} + +/// A store, and the inputs of captures of [`class`] into it, in a temporary directory: the +/// platform's files in `platform`, a placeholder, a kernel, an initramfs, and the fake OpenVMM's +/// scenario, which a capture's VM follows, and the store in `store`. +struct Bench { + /// The temporary directory. + directory: tempfile::TempDir, + /// The store. + store: Store, + /// The inputs of a capture. + capture: Capture, +} + +impl Bench { + /// Returns a bench whose VMs publish the fixture's snapshot, as `scenario` modifies the + /// default scenario. + fn new(scenario: &Value) -> Self { + let directory = tempfile::tempdir().unwrap(); + let platform = directory.path().join("platform"); + private_dir(&platform); + sparse(&platform.join("placeholder.erofs"), class().distro.length); + fs::write(platform.join("vmlinux"), b"kernel").unwrap(); + fs::write(platform.join("initramfs.cpio.gz"), b"initrd").unwrap(); + let store = Store::create(directory.path().join("store")).unwrap(); + let bench = Self { + store, + capture: capture_in(&platform), + directory, + }; + bench.follow(scenario); + bench + } + + /// Returns the bench in `dir`, as another process made it. + fn open(dir: &Path) -> (Store, Capture) { + ( + Store::open(dir.join("store")).unwrap(), + capture_in(&dir.join("platform")), + ) + } + + /// Returns the platform's directory. + fn platform(&self) -> PathBuf { + self.directory.path().join("platform") + } + + /// Lets the next VMs follow the default scenario, as `fields` modify it: the fake OpenVMM + /// publishes the fixture's snapshot. + fn follow(&self, fields: &Value) { + let mut scenario = json!({ + "manifest": fixture().join("manifest.bin"), + "stateLen": manifest().state_len(), + }); + for (name, value) in fields.as_object().unwrap() { + scenario[name] = value.clone(); + } + fs::write(self.platform().join(SCENARIO_NAME), scenario.to_string()).unwrap(); + } + + /// Returns the path of the marker of an active VM. + fn active(&self) -> PathBuf { + self.platform().join(ACTIVE_NAME) + } + + /// Returns the slot of the class, the only one. + fn slot(&self) -> PathBuf { + let captures = self.store.dir().join(CAPTURES_NAME); + let slots = sorted(&captures); + assert_eq!(slots.len(), 1, "{slots:?}"); + captures.join(&slots[0]) + } + + /// Returns the path of the file `suffix` that the fake OpenVMM wrote of the slot's last VM, + /// which it names after the directory of the VM's outcome report: the capture's transaction, + /// from whose sandbox store the VM launched. + fn dump(&self, suffix: &str) -> PathBuf { + self.platform() + .join(format!("fake-openvmm-{TRANSACTION_NAME}{suffix}.json")) + } + + /// Creates a template at a second after the fixture's capture. + fn create(&self) -> nvx::Result { + self.store + .create_template(&self.capture, || Ok(after(1)), deadline()) + } + + /// Refreshes the template. + fn refresh(&self) -> nvx::Result { + self.store.refresh_template(&self.capture, deadline()) + } + + /// Returns the store generations that the index lists. + fn indexed(&self) -> Vec { + self.store + .index() + .unwrap() + .generations() + .iter() + .map(number) + .collect() + } + + /// Asserts that no capture of the class runs or remains: its slot holds no transaction and no + /// launch. + fn assert_idle(&self) { + let slot = self.slot(); + assert!(!slot.join(TRANSACTION_NAME).exists()); + assert!( + SandboxStore::open(&slot) + .unwrap() + .journal() + .unwrap() + .is_none() + ); + } +} + +/// Returns the inputs of a capture of [`class`] whose placeholder, kernel, initramfs, and +/// scenario are in `platform`, and whose VMM is the fake OpenVMM. +fn capture_in(platform: &Path) -> Capture { + let placeholder = platform.join("placeholder.erofs"); + let path = placeholder.to_str().unwrap(); + let layer = SandboxLayer::new(LayerRole::Distro, path, IDENTITY); + let binding = DistroBinding::import(&layer, PolicyVerdict::ACCEPTED).unwrap(); + let image = RootImage::erofs(Some(Uuid::parse(IDENTITY).unwrap())); + let upper = SandboxUpper::scratch(platform.join("clone.ext4").to_str().unwrap()); + Capture { + class: class(), + adapter: SandboxPlan::adapter(SandboxDistro::new(path, image), upper), + binding, + policy_revision: DISTRO_POLICY_REVISION, + artifacts: Artifacts::new( + env!("CARGO_BIN_EXE_nvx-fake-openvmm"), + platform.join("vmlinux"), + platform.join("initramfs.cpio.gz"), + ), + } +} + +/// Returns the value of `option` in the command line `arguments`. +fn value_of<'a>(arguments: &'a [String], option: &str) -> &'a str { + let at = arguments + .iter() + .position(|argument| argument == option) + .unwrap_or_else(|| panic!("{option} is missing from {arguments:?}")); + &arguments[at + 1] +} + +#[test] +fn captures_publish_generations_that_creations_then_serve() { + let _spawns = serialized(); + let bench = Bench::new(&json!({})); + let Created::Captured(generation) = bench.create().unwrap() else { + panic!("the first creation captures"); + }; + assert_eq!(bench.indexed(), [1]); + assert_eq!(number(&generation), 1); + // The VMM received the template's command line, with the generation's storage generation as + // its block devices' identity. + let arguments: Vec = serde_json::from_value(json_of(&bench.dump(""))).unwrap(); + assert_eq!(value_of(&arguments, "--cmdline"), "nvx_sandbox=1"); + assert_eq!( + value_of(&arguments, "--snapshot-generation-id"), + generation.storage_generation().to_string() + ); + let transaction = bench.slot().join(TRANSACTION_NAME); + assert_eq!( + Path::new(value_of(&arguments, "--snapshot-destination")), + transaction.join(SNAPSHOT_NAME) + ); + // The VM launched from the transaction's sandbox store, so that its control socket, on + // Linux, is beside the snapshot's destination, as OpenVMM requires, and its outcome report + // too, which the slot then keeps. + assert_eq!( + Path::new(value_of(&arguments, "--microvm-report")), + transaction.join(OUTCOME_NAME) + ); + #[cfg(target_os = "linux")] + assert_eq!( + value_of(&arguments, "--microvm-control-console"), + format!("listen={}", transaction.join(SOCKET_NAME).display()) + ); + assert!(bench.slot().join(OUTCOME_NAME).is_file()); + // The VMM opened the placeholder and the scratch at their paths, and found no handle that + // could write them: on Windows, its open would have failed, and on Linux, its read lease + // finds one, unless the file system takes no lease. + let blocks = json_of(&bench.dump("-blocks")); + let opened: Vec<(&str, PathBuf, u64)> = blocks + .as_array() + .unwrap() + .iter() + .map(|block| { + assert_ne!(block["writers"], true, "{block}"); + ( + block["role"].as_str().unwrap(), + PathBuf::from(block["path"].as_str().unwrap()), + block["length"].as_u64().unwrap(), + ) + }) + .collect(); + assert_eq!( + opened, + [ + ( + "distro", + bench.platform().join("placeholder.erofs"), + 1024 * MIB + ), + ("scratch", transaction.join(SCRATCH_NAME), 64 * MIB), + ] + ); + // The generation's payload holds the snapshot that the VMM published, with its key, and + // the slot keeps nothing of the capture. + let snapshot = bench.store.snapshot_dir(&generation).unwrap(); + assert_eq!( + sorted(&snapshot), + ["manifest.bin", "memory.bin", "state.bin"] + ); + let key = TemplateKey::new(&bench.capture.class, &manifest()).unwrap(); + assert_eq!(bench.store.read_key(&generation).unwrap(), key); + assert_eq!(generation.key(), key.identity()); + bench.assert_idle(); + // A second creation serves the generation without a VM. + fs::remove_file(bench.dump("")).unwrap(); + assert_eq!(bench.create().unwrap(), Created::Existing(generation)); + assert!(!bench.dump("").exists()); +} + +#[test] +fn refreshes_publish_new_generations_and_leave_the_older_ones() { + let _spawns = serialized(); + let bench = Bench::new(&json!({})); + let first = bench.create().unwrap().into_generation(); + // A VMM that ends abruptly once it has published leaves the file that backed the RAM, which + // the publication does not take. + bench.follow(&json!({ "keepMemory": true })); + let second = bench.refresh().unwrap(); + assert_eq!(number(&second), 2); + assert_ne!(second.storage_generation(), first.storage_generation()); + assert_eq!(bench.indexed(), [1, 2]); + for generation in [&first, &second] { + let snapshot = bench.store.snapshot_dir(generation).unwrap(); + assert_eq!( + sorted(&snapshot), + ["manifest.bin", "memory.bin", "state.bin"] + ); + } + assert_eq!( + sorted(&bench.store.dir().join(GENERATIONS_NAME)), + ["1", "2"] + ); + bench.assert_idle(); + assert_eq!(bench.create().unwrap(), Created::Existing(second)); +} + +#[test] +fn vms_that_end_without_a_capture_publish_nothing() { + let _spawns = serialized(); + let bench = Bench::new(&json!({})); + for scenario in [ + json!({ "outcome": "fail", "status": 3 }), + // The snapshot that the VMM published before it failed is not the capture's. + json!({ "outcome": "publish-then-fail" }), + json!({ "outcome": "crash" }), + ] { + bench.follow(&scenario); + let error = bench.refresh().unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Exited, "{scenario}: {error}"); + assert!(bench.indexed().is_empty(), "{scenario}"); + bench.assert_idle(); + } + // The slot serves the next capture. + bench.follow(&json!({})); + bench.refresh().unwrap(); + assert_eq!(bench.indexed(), [1]); +} + +/// Returns the process ID that the marker of an active VM at `path` holds. +fn active_pid(path: &Path) -> u32 { + fs::read_to_string(path).unwrap().parse().unwrap() +} + +#[test] +fn vms_that_never_end_are_torn_down_at_the_deadline() { + let _spawns = serialized(); + let bench = Bench::new(&json!({})); + for outcome in ["hang", "publish-then-hang"] { + bench.follow(&json!({ "outcome": outcome, "active": bench.active() })); + let deadline = Instant::now() + Duration::from_secs(2); + let error = bench + .store + .refresh_template(&bench.capture, deadline) + .unwrap_err(); + assert_eq!(error.kind(), ErrorKind::TimedOut, "{outcome}: {error}"); + // The VMM, which was killed, left its marker. + let pid = active_pid(&bench.active()); + assert_eq!( + platform::process_start_time(pid).unwrap(), + None, + "{outcome}" + ); + fs::remove_file(bench.active()).unwrap(); + assert!(bench.indexed().is_empty(), "{outcome}"); + bench.assert_idle(); + } +} + +#[test] +fn captures_of_one_class_are_serialized() { + let _spawns = serialized(); + // A VM that started while another ran would fail to mark itself active. + let bench = Bench::new(&json!({})); + bench.follow(&json!({ "active": bench.active(), "delayMs": 300 })); + let (store, capture) = (&bench.store, &bench.capture); + let generations: Vec = thread::scope(|scope| { + let refreshes: Vec<_> = (0..2) + .map(|_| scope.spawn(|| store.refresh_template(capture, deadline()))) + .collect(); + refreshes + .into_iter() + .map(|refresh| refresh.join().unwrap().unwrap()) + .collect() + }); + let mut published: Vec = generations.iter().map(number).collect(); + published.sort_unstable(); + assert_eq!(published, [1, 2]); + assert_eq!(bench.indexed(), [1, 2]); + assert!(!bench.active().exists()); + bench.assert_idle(); +} + +/// The adapter that [`a_vm_that_outlives_its_crashed_adapter_is_ended_by_recovery`] runs in a +/// process of its own: it refreshes the template of the bench that [`ADAPTER_BENCH`] names, until +/// the test kills it. +#[test] +#[ignore = "an adapter that the capture tests run in a process of its own"] +fn adapter_role() { + let Some(dir) = env::var_os(ADAPTER_BENCH) else { + return; + }; + let (store, capture) = Bench::open(Path::new(&dir)); + let result = store.refresh_template(&capture, deadline()); + panic!("the capture ended before the test killed its adapter: {result:?}"); +} + +/// A VMM that a crashed adapter left behind, which is killed when the test ends if it still runs. +struct Orphan(ProcessRef); + +impl Orphan { + /// Returns whether the VMM still runs. + fn runs(&self) -> bool { + platform::process_start_time(self.0.pid).unwrap() == Some(self.0.start_time) + } +} + +impl Drop for Orphan { + fn drop(&mut self) { + if let Ok(Some(process)) = Process::open(&self.0) { + let _ = process.kill(); + let _ = process.wait(Instant::now() + TIMEOUT); + } + } +} + +/// Runs [`adapter_role`] on `bench` until its capture's VM is active, the journal of the +/// transaction's sandbox store records the VMM, and, if `published`, the VM has published its +/// snapshot, then kills the adapter, as a crash would end it, and returns the VMM, which still +/// runs. +fn crash_adapter(bench: &Bench, published: bool) -> Orphan { + let mut adapter = Command::new(env::current_exe().unwrap()) + .args([ + "--exact", + "adapter_role", + "--ignored", + "--nocapture", + "--test-threads=1", + ]) + .env(ADAPTER_BENCH, bench.directory.path()) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .spawn() + .unwrap(); + let started = Instant::now(); + let recorded = loop { + assert!( + adapter.try_wait().unwrap().is_none(), + "the adapter ended before the test killed it" + ); + // The VM marks itself active before it publishes its snapshot. + let recorded = bench.active().exists().then(|| { + let transaction = bench.slot().join(TRANSACTION_NAME); + if published && !transaction.join(SNAPSHOT_NAME).is_dir() { + return None; + } + SandboxStore::open(transaction) + .ok()? + .journal() + .ok()?? + .process() + .cloned() + }); + if let Some(Some(process)) = recorded { + break process; + } + assert!( + started.elapsed() < TIMEOUT, + "the capture's VM did not start" + ); + thread::sleep(Duration::from_millis(20)); + }; + adapter.kill().unwrap(); + adapter.wait().unwrap(); + assert_eq!(recorded.pid, active_pid(&bench.active())); + assert_eq!(recorded.boot, BootId::current().unwrap()); + let orphan = Orphan(recorded); + assert!(orphan.runs()); + orphan +} + +#[test] +fn a_vm_that_outlives_its_crashed_adapter_is_ended_by_recovery() { + let _spawns = serialized(); + let bench = Bench::new(&json!({})); + // A VM that published its snapshot, but whose exit no capture verified, is not published + // either. + for outcome in ["hang", "publish-then-hang"] { + bench.follow(&json!({ "outcome": outcome, "active": bench.active() })); + let orphan = crash_adapter(&bench, outcome == "publish-then-hang"); + let deadline = deadline(); + // A scanner that briefly opens the launch's log can keep its claim held for a moment. + let recovered = loop { + match bench.store.recover_captures(deadline) { + Err(error) + if error.code() == ErrorCode::VmmIndeterminate && Instant::now() < deadline => + { + eprintln!("the recovery is indeterminate: {error}"); + thread::sleep(Duration::from_millis(100)); + } + recovered => break recovered.unwrap(), + } + }; + assert!(recovered.is_empty(), "{outcome}"); + assert!(!orphan.runs(), "{outcome}"); + assert!(bench.indexed().is_empty(), "{outcome}"); + bench.assert_idle(); + fs::remove_file(bench.active()).unwrap(); + } + // The slot serves the next capture. + bench.follow(&json!({})); + bench.refresh().unwrap(); + assert_eq!(bench.indexed(), [1]); +} + +/// What the tests need of Windows that the crate does not export. +#[cfg(windows)] +mod windows { + use std::fs::File; + use std::os::windows::ffi::OsStrExt; + use std::os::windows::io::AsRawHandle; + use std::path::Path; + use std::{io, ptr}; + + use windows_sys::Win32::Foundation::{HANDLE, LocalFree}; + use windows_sys::Win32::Security::Authorization::{ + ConvertStringSecurityDescriptorToSecurityDescriptorW, SDDL_REVISION_1, + }; + use windows_sys::Win32::Security::{ + DACL_SECURITY_INFORMATION, PROTECTED_DACL_SECURITY_INFORMATION, SetFileSecurityW, + }; + use windows_sys::Win32::System::IO::DeviceIoControl; + + /// Returns `text` as a NUL-terminated UTF-16 string. + fn wide(text: impl AsRef) -> Vec { + text.as_ref().encode_wide().chain([0]).collect() + } + + /// Replaces the access control list of the file or directory `path` with the protected list + /// of the descriptor `sddl`, in the security descriptor definition language. + pub(super) fn set_access(path: &Path, sddl: &str) { + let sddl = wide(sddl); + let name = wide(path); + let mut descriptor = ptr::null_mut(); + // SAFETY: the string is NUL-terminated, and the output pointer is valid. + let converted = unsafe { + ConvertStringSecurityDescriptorToSecurityDescriptorW( + sddl.as_ptr(), + SDDL_REVISION_1, + &mut descriptor, + ptr::null_mut(), + ) + }; + assert_ne!(converted, 0, "{}", io::Error::last_os_error()); + let information = DACL_SECURITY_INFORMATION | PROTECTED_DACL_SECURITY_INFORMATION; + // SAFETY: the name is NUL-terminated, and the descriptor is valid. + let set = unsafe { SetFileSecurityW(name.as_ptr(), information, descriptor) }; + let error = io::Error::last_os_error(); + // SAFETY: the conversion allocated the descriptor, which nothing uses any more. + unsafe { LocalFree(descriptor) }; + assert_ne!(set, 0, "{error}"); + } + + /// Marks the file that `file` holds open for writing sparse, which NTFS otherwise allocates + /// whole as the file is extended. + pub(super) fn set_sparse(file: &File) { + /// `FSCTL_SET_SPARSE`, in `winioctl.h`, which marks a file sparse without a buffer. + const FSCTL_SET_SPARSE: u32 = 0x0009_00c4; + let mut returned = 0; + // SAFETY: the handle is open for as long as `file` is borrowed, the control takes no + // input or output buffer, and the count's pointer is valid. + let set = unsafe { + DeviceIoControl( + file.as_raw_handle() as HANDLE, + FSCTL_SET_SPARSE, + ptr::null(), + 0, + ptr::null_mut(), + 0, + &mut returned, + ptr::null_mut(), + ) + }; + assert_ne!(set, 0, "{}", io::Error::last_os_error()); + } +}