diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index d89423d5e..828afc381 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -225,6 +225,10 @@ the root, of normal components, and the other probes take no arguments: | `rename DIRECTORY` | It renames DIRECTORY, a directory of the layers, to `.runtime-adapter-renamed` at the top of the root, and back if it can: `outcome` is `allowed`, `refused` once the overlay refused it with `EXDEV`, or `failed: PROBLEM` | | `fill` | It writes a file in the root until the upper is full, then removes it, and writes a file there again: the upper's size, `total`, and the bytes available in it before, `available`; the bytes that it wrote, `written`; the error that ended the writes, `error`: `ENOSPC` once the upper was full, or the problem; what the second file holds, `after`; and the guest's memory, `memory`, in bytes | | `devices` | The guest's disks, the block devices that a device backs: `disks`, each as `NAME:SECTORS:ACCESS`, its size in 512-byte sectors and its access mode, `ro` or `rw`, in the order of their names, separated by spaces | +| `reads` | The sectors that the guest read from each of its disks, by the disk's statistics: `disks`, each as `NAME:SECTORS`, in 512-byte sectors, in the order of their names, separated by spaces | +| `read-disks` | It reads each of the guest's disks from its start to its end, through the page cache: `disks`, each as `NAME:BYTES`, the bytes that it read, in the order of their names, separated by spaces | +| `memory` | The guest's memory: the size of its memory blocks, `block-size`, how many are online, `blocks`, and their bytes, `online`, and the memory that the kernel manages, `total`, each in bytes | +| `time` | The guest's time state, `/run/nvx/time/state`, which NVX's time daemon keeps: each line `KEY=VALUE` as a fact, with each `_` of its key as `-`, in order, leaving out a key that no fact may have, and a repeated one | | `first-partition` | The bytes of the first partition of a GPT disk that the assembly returned, without their trailing padding: how many, `length`, and the bytes, `bytes`, in hexadecimal, or `none` for both for an EROFS image | | `cgroups` | The file system at the cgroups' parent, `parent.type`, and its `parent.subtree-control`; whether each cgroup exists, `agent.exists` and `workload.exists`; `agent.memory.low`, `workload.memory.max`, and `workload.pids.max`; the number of processes in each, `agent.processes` and `workload.processes`; and `user-namespaces`, the guest's limit of user namespaces, or `unavailable` without them | | `processes` | For PID 1, `pid1`, the worker, `worker`, and PID 1's unrelated child, `sibling`: `NAME.pid`, and while the process exists, `NAME.state`, `NAME.ppid`, `NAME.root`, `NAME.cgroup`, `NAME.command`, and `NAME.namespaces`: `same` if it shares PID 1's, or the ones that differ | @@ -234,6 +238,19 @@ A value that a probe cannot read is `absent` if its file does not exist, and otherwise `unreadable: PROBLEM`. A new check of a root is a new probe in this table. +The worker of a clone also answers the probe `activation`, which takes no +arguments, itself, at any time once the clone is ready, executions running or +not, with when the events of the clone's activation and order happened, in +microseconds after the repair's acknowledgement of the restore: whether the +repair acknowledged the restore, `acknowledged`, `yes`, or `none` for a restore +whose host did not gate its input until the acknowledgement; and when the clone +took the reset that the restore queued, `reset`, when the host's first message +came, `message`, which must be the clone's `Activate`, when the clone was +ready, `ready`, when its sandbox started, `started`, and when the start of the +host's first execution came, `execution`. Each event is `none` until it +happened, `before` if it happened before the acknowledgement, and `unknown` +without one. The worker of a VM that boots cold runs no such probe. + The probes of `worker::mapping` inspect the mappings beneath the root, and map from the host export, as an adapter maps the host paths of its sandboxes. `map` inspects the mappings that the worker applied from the host's mapping table, @@ -799,7 +816,9 @@ after the repair's acknowledgement of the restore: stop. The record's workload identity, if it has one, is the whole sandbox's, as a root configuration's is: every execution runs as it, and a start that names an identity is refused. Without an export, the clone - refuses each part of a mapping table with `MapFailed`. + refuses each part of a mapping table with `MapFailed`. It also answers the + [`activation` probe](#probes), with when the events of its activation and + order happened after the repair's acknowledgement of the restore. Until the clone is ready, its session takes nothing but that `Activate`: another reset, any other message, a `Stop` and a `Ping` included, an @@ -1255,27 +1274,28 @@ guest's listeners, through publication or a forward. ### The step hook -For tests, `launch` takes a step hook, `hook::StepHook`, from the variable -`RUNTIME_ADAPTER_STEP_HOOK`, which pauses the launch at a named step, so that a -test can act between two steps, or exits the launch there without unwinding, -as a crash would end it: +For tests, `launch`, `reattach`, and `stop` take a step hook, `hook::StepHook`, +from the variable `RUNTIME_ADAPTER_STEP_HOOK`, which pauses the command at a +named step, so that a test can act between two steps, or exits the command +there without unwinding, as a crash would end it: - `STEP:pause:PATH` creates the file PATH, which holds the step's name, and - waits until the file is gone, then goes on: the launch leaves the time that - it waited out of its deadline, so that a test may pause it for as long as it + waits until the file is gone, then goes on: a launch leaves the time that it + waited out of its deadline, so that a test may pause it for as long as it needs; -- `STEP:exit` exits at once with status 5: the launch ends nothing that it +- `STEP:exit` exits at once with status 5: the command ends nothing that it started, such as the VMM, and the system releases what it holds, such as its descriptors or handles, and their locks. -The steps of a launch, in order, are: +The steps of a launch, in order, then that of a command that reattaches, are: -| Step | Where the launch is | +| Step | Where the command is | | --- | --- | | `checked` | It has checked the bindings of its images, and the export of its shares, if any, and holds the checked files, which claim the scratch image, but has begun nothing in the store, and spawned nothing. A launch without a layered root reaches it too. | | `spawned` | The VMM runs, and the launch's journal records it, but the launch has not connected to the VM. | | `ready` | The guest agent, of the build that the host ships, has assembled the layered root, and the launch has released the checked files, but has sent neither the mapping table nor `Start`. | | `commit` | The guest has started the sandbox, if the VM has a layered root, the guest agent has answered, and the launch is about to commit: its journal still records the VMM, which no runtime record names yet. | +| `found` | `reattach` or `stop` has found the VM running, and holds it, under the sandbox's lock, with the session in which a clone answered the `Activate` of its record again, if the VM is a clone's, but has sent the guest agent nothing more, and stopped nothing. | A value that names no step or action is a usage error. Without the variable, the hook does nothing. @@ -1331,6 +1351,117 @@ before it creates the store. A capture that fails leaves the store's generations as they were, and the command's diagnostic names the log of the VMM of the class's latest capture, when the class's slot keeps one. +### Clones + +`runtime-adapter-host start` starts a clone of a generation of a template store +in a sandbox store, through NVX's public API, `nvx::host::template`. Its +library side, `clone`, holds what the command's options choose of the clone, +`clone::Request`: its image, an EROFS layer of the length of the template's +placeholder, its upper, on an ext4 scratch image or in RAM, its memory target, +the template's RAM unless given, the workload identity as which its executions +run, its overlay mode, and its scratch's journal policy. `Request::plan` plans +the image over the upper as a launch plans a layered root, and +`Request::clone_request` returns NVX's `CloneRequest` of it, with the bindings +that the sandbox's store stages for the images, as `stage` stages them, so the +start hashes nothing. The clone's activation record names the image's own +machine ID, the hostname `runtime-adapter`, and the guest's defaults of the +other settings, unless the request names them. + +The command first reads the generation from the store's index, and, unless the +request names the clone's memory target, its compatibility key, whose RAM the +target is then, before it needs NVX's artifacts. A key that cannot be read is +left to NVX's plan, which reads it again, and attributes the failure, to a +malformed generation, which quarantines it, or to the host, as it does for +every clone. The store then leases the generation to the sandbox, +`Store::acquire`, at the host's time, which it refuses for a generation that a +tombstone quarantines, or that is not eligible then, by the monotonic clock of +the host's boot that captured the template, and by UTC otherwise. For tests, +the host's time may come from the variable `RUNTIME_ADAPTER_HOST_TIME`, as a +clock fixture, such as another boot's, in the form that `clone::host_time_text` +writes: +`host=HOST,boot=BOOT,clock=CLOCK,monotonic=NANOSECONDS,utc=NANOSECONDS`, with +the identities of the host and of its boot, 32 lowercase hexadecimal digits +each, the clock as a manifest spells it, `linux-boottime` or +`windows-interrupt-time`, and the readings of the monotonic clock and of UTC. +It affects the store's check alone: OpenVMM's restore samples the host's own +time. + +NVX's `ClonePlan` then checks the clone against the generation from the +metadata of its files alone, under the sandbox's lock, which the lease holds: +the bindings of its images, which no one else may change, the geometry of its +image, which must be the placeholder's, its length included, and that of its +scratch image, which must be the template's scratch's, and its memory target, +whole blocks of 128 MiB, from the template's RAM to its capacity. A clone that +the template cannot serve fails there, before any VMM starts. The command then records the clone in the +sandbox's store, as `clone::CloneRecord`, in the format of a launch's +[record](#layered-roots), in the mode `clone`, with the template's kernel +command line, the plan and the bindings, so that `probe` and `exec` read the +plan back, no root configuration, and the template store and the generation, so +that a later stop finds the store whose lease it releases. OpenVMM restores the +generation's snapshot in place, through NVX's supervisor, as the launch that the +lease reserves, and the restore sequence activates the clone through the +adapter's clone order, `host::CloneOrder`, which sends the clone's activation +record in an `Activate`, checks the guest agent's build in its `Ready`, and +sends `Start`, with no mapping table: the command maps nothing into a clone. + +A clone that fails, from its plan to its commit, fails with NVX's +`CloneFailure`, which attributes the failure, by the evidence that the host saw +of it, such as a length that the request supplied, or a status with which the +guest ended the VM, to the request, the staging of its files, the host, the +template, or a cause that the evidence does not show, and which, from the +clone's launch on, records it in the sandbox's store, with the stage at which +the clone failed, how far it got, and how the teardown that followed ended. The +command prints the failure's record, then releases the lease, `Store::release`, +which removes the records with it, once the failure's teardown is resolved, and +nothing of the clone runs: the clone had no VM, its VM exited, or NVX +terminated it. A failure whose teardown is unresolved, such as that of a commit, +or one that NVX could not record before it ended the clone's VM, leaves the +lease, and the records, to a later reattach. Once it released the +sandbox's lock, which comes after the template store's in NVX's lock order, the +command quarantines the generation through its tombstone, +`CloneFailure::quarantine`, for a failure of the template alone. A guest agent +that does not answer the command's `Ping` or `Echo` once the clone's sandbox +started fails the clone so, through `Started::fail`, which ends the launch, +with the evidence that NVX's restore sequence gives the failures of its own +sessions: the passing of the deadline, the loss of the session, or a failure +of the protocol. A start that fails otherwise, at a step of the adapter's own +before the clone's launch, such as the read of the images that the sandbox +stages, releases the lease, since nothing of the clone runs. + +The commands that reattach take the lock of a clone's sandbox through the path +of its lease, which they read under the lock, as a release requires. A sandbox +that holds a lease must not move: they refuse one whose lease names another +path, at which no sandbox holds the same lease, as the lease of a sandbox that +moved does, rather than take another sandbox there for it. They then reattach +to the clone with NVX's `reattach_clone`, in a new session of the adapter's +clone order, in which the guest agent answers the `Activate` of the clone's +record again, with the `Ready` that it answered once, and which they then use +as `reattach` attaches one. The reattach ends a clone that runs under another +runtime identity, without a word to its guest, or that fails the check of its +`Ready`, through its verified process identity, and finishes the record of an +earlier failure of the clone, whose teardown it resolves if it can. The +command then releases the lease of a clone whose VM ended without a failure, +and fails; ends as `start` does for a clone that failed; and keeps the lease +and the records, and fails, for a clone whose state cannot be determined. A +stop of a clone releases the lease once the store verified that the VM exited, +and the clone's records, which the release removes, told how the clone ended, +as `reattach_clone` reads them: a VM that had exited when the stop began ended +without its holder, as NVX's `Vmm::stop_unless_exited`, which then stops +nothing, tells by keeping the runtime record of its launch, and the stop ends +as the other commands do for a clone that they find so, while a VM that the +guest ended, once the stop began, with a status that is evidence of a failure, +such as its repair's, failed, which ends the stop as it ends `start`. +The commands release a lease through the template store that the clone's +record names, once the lease is of it. A sandbox whose start ended before it +recorded the clone holds a lease of a template store that no record names: it +records no clone, or an earlier clone's, which a release leaves, of another +template store perhaps. `reattach` and `stop` then release it with +`--template TEMPLATES`, through the template store TEMPLATES, in place of the +record's, which the release refuses unless the lease is of it. +A tombstone of the generation refuses new leases of it, and leaves those that +sandboxes hold to their clones, which reattach, stop, and release them as +before. + ### Commands Each command of `runtime-adapter-host` but `stage` takes NVX's artifacts from @@ -1416,19 +1547,42 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from the network before anything else, as a launch does, and checks the class before it creates the store, then prints the generation once the store has committed it. -- `reattach STORE` takes the sandbox's lock, and reattaches to its VM, which +- `reattach STORE [--template TEMPLATES]` takes the sandbox's lock, and + reattaches to its VM, which another process launched, and which it holds, although the VM is not its child, through its verified process identity. It reads the capability from the store, attaches a new session, in the session's next epoch, exchanges a - `Ping`, and exits, leaving the VM running. -- `stop STORE` reattaches and attaches as `reattach` does, then stops the VM: + `Ping`, and exits, leaving the VM running. It reattaches to a + [clone](#clones) in the session in which the clone answers the `Activate` of + its record again, and releases the lease of a clone that ended, through the + template store TEMPLATES, if given, in place of the one that the clone's + record names. +- `start STORE --template TEMPLATES --generation G --distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY]` + starts a [clone](#clones) of the generation G of the template store + TEMPLATES in the sandbox store STORE, which it creates if needed, and whose + `stage` staged its images, which no one else may change: the image PATH, an + EROFS file system, `erofs`, of which the guest expects the identity UUID, + over the ext4 scratch image PATH, or RAM, with a memory target of MIB mebibytes, the template's RAM by default, + the workload identity UID:GID, if given, as which every execution runs, and + whose account the guest checks before the clone is ready, and the overlay + mode MODE and the scratch's journal policy POLICY, as a launch takes them. + Once the restore sequence started the clone's sandbox, it exchanges a `Ping` + and the longest `Echo` with the guest agent, commits the launch, and exits, + leaving the VM running, and the sandbox holding the lease. A clone that + fails, such as one whose activation failed before it was ready, and whose VM + ended with the status of the failure's class, from 50 to 58, ends as + [Clones](#clones) describes, with the record of its failure. +- `stop STORE [--template TEMPLATES]` reattaches and attaches as `reattach` + does, then stops the VM: the adapter's `Stop`, which the guest agent confirms with `Stopped` before it ends the VM, is the stop's graceful step. It waits for the VM to exit, and checks the VM's outcome report: a success, with status 0 and a complete teardown. A VM whose guest agent does not confirm the stop is terminated, and the stop fails. The worker of a layered root tears the root down before it confirms, and reports a teardown that failed first: the stop then checks - that the VM ended with the teardown's status instead. + that the VM ended with the teardown's status instead. A stop of a clone + releases its lease once the VM exited, and the clone's records told how it + ended, as [Clones](#clones) describes. - `probe STORE PROBE...` reattaches and attaches as `reattach` does, then runs each PROBE in the VM's layered root, in order, and prints the facts that it found. A PROBE is the text of a [probe](#probes): its name, then its @@ -1449,9 +1603,9 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from DIR, or `/`, with exactly the environment of its `--env` entries, then, if the VM's [network](#networks) has a proxy, the proxy's URL as the guest sees it in `http_proxy`, `https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY`, unless - the entries name any of them. Before it attaches, the command checks that the - framing still carries each start with the proxy's URL, and runs none of the - executions if one no longer fits. It runs for at most a minute, with 16 KiB + the entries name any of them. Before it starts any execution, the command + checks that the framing still carries each start with the proxy's URL, and + runs none of the executions if one no longer fits. It runs for at most a minute, with 16 KiB chunks and the credit of 4 of them for its input and 8 for its output, which its standard output and error share. The command keeps at most 16 MiB of each stream, and cancels an execution whose stream would exceed them. Execution N, @@ -1461,11 +1615,16 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from broker instance only advance, while a broker restored from a snapshot has a new instance ID, so no command repeats the key of another's execution. -`reattach`, `stop`, `probe`, and `exec` find a VM that another runtime identity -launched mismatched, and then signal nothing: the VM keeps running. The commands -print what they find as `KEY=VALUE` lines: `reattached` (`live`, `mismatched`, -`exited`, or `indeterminate`), `differences` (the parts of the runtime identity -in which a mismatched VM differs), `pid` (the VMM's process ID), `instance` and +`reattach`, `stop`, `probe`, and `exec` find a VM that booted cold under +another runtime identity mismatched, and then signal nothing: the VM keeps +running. They end a clone that runs under another, as [Clones](#clones) +describes. The commands print what they find as `KEY=VALUE` lines: `reattached` +(`live`, `mismatched`, `exited`, or `indeterminate`, and of a clone, `live`, +`ended`, `failed`, or `indeterminate`), `differences` (the parts of the runtime +identity in which a mismatched VM differs), `progress` (the step that a clone, +which ended, reached, from `planned`, `launching`, `spawned`, `event`, +`acknowledged`, `activating`, `ready`, `starting`, and `started` to +`committed`), `pid` (the VMM's process ID), `instance` and `epoch` (the identity of the session that they attached to), `root` (`ready` or `failed`: how the guest answered the root configuration, or `failed` when a stop finds that the root's teardown failed), `build` (the build of the guest agent @@ -1479,25 +1638,48 @@ the probe named PROBE found), `exec.N.KEY` (what execution N did: its `launch`, `signal NUMBER`, or `none`; the `cause` for which a side ended it, such as `timed-out`, or `none`; the streams that it `truncated`, such as `stdout,stderr`, or `none`; whether it `settled`, `reusable` or `fenced`; and -its `stdout` and `stderr`), `stopped` (`graceful`, `forced`, or `timed-out`), +its `stdout` and `stderr`), `stopped` (`graceful`, `forced`, or `timed-out`, or +`exited` once a clone's VM, which the command found running, had exited when +the stop began, which stopped nothing), `outcome` (such as `success 0`, or `guest-exit 14` once the assembly failed), `teardown` (`complete` or `incomplete`: whether the VMM's teardown of the VM completed, which the outcome report records apart from the root's), `created` (`captured` once a capture made the template's generation, or `existing` once the store held it), `generation` (the generation's number in the template -store), and `key` and `class` (the identities of its compatibility key and of -its capture class, in hexadecimal). Values +store, which `create` served or `start` leased), `key` and `class` (the +identities of its compatibility key and of its capture class, in hexadecimal), +`lease` (the identity of the lease that `start` took, 32 hexadecimal digits), +`downtime` and `source` (the downtime of the leased generation's capture at the +host's time, in nanoseconds, and the clock that measured it, `monotonic` or +`utc`), `released` (the identity of the lease that a stop, or a start that +failed, released), `failure.KEY` (the record of a clone's failure: its +`outcome`, what the failure is attributed to, `request`, `staging`, `host`, +`unknown`, `template`, or `repair`; its `evidence`, what the host saw of it, +which attributes it, such as `supplied-length`; its `stage`, where the clone +failed, `plan`, `launch`, `restore`, `activation`, `start`, `commit`, `run`, or +`reattach`; its `progress`, the step that the clone reached, as `progress` +spells it; its `status`, the status with which the guest ended the VM, as the +VM's outcome report gives it, or `none`; its `teardown`, how the teardown that +followed ended, `no-vm`, `exited`, `terminated`, or `unresolved`; and its +`code`, the code of the failure's error, such as `runtime-mismatch`), and +`quarantined` (whether the command quarantined the generation of a clone that +failed: `yes`, for a failure of the template, or `no`). Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each byte of a program's output that is not part of UTF-8 text as `\xHH`. The commands exit with 0 once they did what they were asked, `exec` once every -execution finished, whatever its outcome, with 3 when they find the VM -mismatched, with 4 when the guest reported that a stage of the root failed, and -the VM ended with the stage's status and a complete teardown, with 5 when the -[step hook](#the-step-hook) ended a launch, with 6 when the guest failed a -mapping of the table, and the VM stopped gracefully, with a success and a -complete teardown, and with 1, after a diagnostic on standard error, when they -fail. Without arguments, the program reports the contract that it was built -against. +execution finished, whatever its outcome, with 3 when they find a VM that +booted cold mismatched, with 4 when the guest reported that a stage of the root +failed, and the VM ended with the stage's status and a complete teardown, with +5 when the [step hook](#the-step-hook) ended a command, with 6 when the guest +failed a mapping of the table, and the VM stopped gracefully, with a success +and a complete teardown, with 8 when a clone failed, once nothing of it runs +and the command released its lease, with 9 when a clone failed, and its +teardown is unresolved, so that the sandbox keeps the lease, and with 1, after +a diagnostic on standard error, when they +fail. A value of `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time is a +usage error, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step +hook, with status 2. Without arguments, the program reports the contract that +it was built against. ## Tests @@ -1839,7 +2021,31 @@ The tests cover: the activation with the status of the host's session; - the record's workload identity, as which every execution runs, while a start that names an identity is refused with `EINVAL`, as is a start that names - none once the record has none. + none once the record has none; +- the `activation` probe, which the worker answers itself, with no root + operation, executions running or not: the restore's reset, the host's first + message, the clone's `Ready`, and its start, in that order, after the + repair's acknowledgement, and the start of the first execution after them. + +`tests/clone_start.rs` checks, in the host-only configuration, on Linux and +Windows, without a hypervisor: + +- the command lines of `start` that the host program refuses, each with status + 2: missing options, a generation that is not a positive number, a scratch's + journal policy for an upper in RAM, invalid values, and the options of other + commands, and the options of `start` that other commands do not take, with a + clock fixture that is not a host's time too; +- a start whose template store does not exist, or does not hold the + generation, and whose image the plan refuses, each with status 1 before it + needs NVX's artifacts or creates the sandbox's store; +- the text form of a host's time, which `RUNTIME_ADAPTER_HOST_TIME` takes, + written and read back on both clocks, and the texts that are none; +- a clone's record: its JSON, with the plan that `layered::LaunchRecord` + reads back, the template store and the generation that a stop reads, a + launch's record that is no clone's, and records that a reader refuses; +- NVX's request of a clone, with the staged bindings, the memory target, and + the guest's defaults or the request's settings, and the images that the + sandbox's store does not stage, which it refuses. `tests/initramfs.rs` and `tests/runtime.rs` check, in the host-only configuration, on Linux and Windows: @@ -1942,10 +2148,11 @@ and Windows: Windows, the [step hook](#the-step-hook): the values of `RUNTIME_ADAPTER_STEP_HOOK` that set it, with each step and action, a path with colons, and those that set none, which the host program refuses with status -2; and a pause, which does nothing, and pauses for no time, at another step, -creates its file with its step's name at its own, waits until the file is -gone, on Windows until a pending deletion of the file completes, and says how -long it paused, and which a file that is there already fails. +2 for `launch`, `reattach`, and `stop`; and a pause, which does nothing, and +pauses for no time, at another step, creates its file with its step's name at +its own, waits until the file is gone, on Windows until a pending deletion of +the file completes, and says how long it paused, and which a file that is +there already fails. `tests/vm_steps.rs` checks the step hook on a real hypervisor, in the host-only configuration, with NVX's Ubuntu EROFS image over a copy of the @@ -1961,9 +2168,9 @@ scratch template: - a launch that the hook ends at `checked`, with status 5, which leaves nothing of a launch in the store, and whose exit released the scratch image's claim, since the next launch takes it; -- a launch paused at each step in turn, at which the store holds what the step - says, and the scratch image's claim is held at `checked`, and which then - goes on, and is stopped. +- a launch paused at each of its steps in turn, at which the store holds what + the step says, and the scratch image's claim is held at `checked`, and which + then goes on, and is stopped. `tests/vm_restart.rs` checks the restart of an orchestrator, on a real hypervisor, in the host-only configuration: @@ -2179,10 +2386,40 @@ transaction and records no launch, and a second creation prints `existing`, with the same generation, key, and class, without a capture: the slot's log of the VMM stays as the capture left it. +### Clone scenarios + +`tests/vm_clone.rs` starts [clones](#clones) of templates on a real hypervisor, +in the host-only configuration, through `runtime-adapter-host start`. Each test +captures a template of its own, of 1 processor and 256 MiB of RAM with a +capacity of 1 GiB, in a template store of its own, of a copy of NVX's Ubuntu +EROFS image that a directory of the platform holds, which the clones stage as +their image, and starts its clones, each in a sandbox store of its own, over a +copy of the scratch template, which no one else may change either, or in RAM: + +| Test | Clones | Expects | +| --- | --- | --- | +| `one_generation_serves_clones_of_each_upper_and_memory_target` | Two at once, over a scratch image at 256 MiB, and in RAM at 512 MiB with untrusted lowers, then one over a scratch image at 1 GiB | Each started, by the host's monotonic clock, with its own lease, which the store lists with the clone's sandbox, its memory target online, its upper, overlay mode, and image's contents, and the events of its activation in order; a stop releases each lease; the generation's snapshot unchanged, and its payload with nothing but the capture's files once the clones stopped | +| `a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts` | An image 4 KiB longer, one 4 KiB shorter, and one 100 bytes longer, than the placeholder, a scratch image of twice the template's scratch, memory targets of 2 GiB and of 320 MiB, unstaged images, and a generation that the store does not hold, then the generation once its key is gone | For those that NVX's plan refuses, status 8, with NVX's reason, and the record of a failure of the request, at the plan, without a VM, by the evidence of a length that the request supplied, of the image's geometry, which OpenVMM cannot attach, or of invalid targets, which quarantines nothing; for the others, status 1, without a record; any lease released, and no VMM's log or record in the sandbox; for the generation without its key, status 8, and the record of a failure of the template, at the plan, by the evidence of a malformed generation, which quarantines the generation, so that a later start fails without a lease | +| `a_clone_whose_activation_fails_ends_before_it_is_ready` | A scratch image of zeros, a journal that the record forbids, a workload identity whose account the image lacks, and, through NVX's restore sequence, a ping, a start, or the start of an execution before the `Activate` | Status 8, no `Ready`, and the record of a failure, at the activation, once the VM exited: of the request, by the evidence of an unusable root, with the guest's status 54, or of invalid settings, with 55, or, before the `Activate`, of an unknown cause, by the protocol's, with 51, which the sandbox records until the release; nothing quarantined, and the lease released, with the records | +| `a_clone_runs_its_executions_as_its_workload_identity_once_it_is_ready` | Over a scratch image, as nobody | An execution that runs as nobody, one that names an identity refused with `EINVAL`, and the first execution after the clone's start | +| `a_generation_is_eligible_by_the_clock_of_its_capture` | At the host's time, then with clock fixtures: UTC a day behind, another boot an hour after the capture by UTC, and the restore window passed by each clock, or UTC before the capture | The host's monotonic clock on the boot of the capture, whatever UTC says, UTC after a reboot, with the fixture's downtime, and the store's refusal without a lease outside the window | +| `a_clone_reads_no_more_of_its_disks_than_its_root_needs` | Over a scratch image, and in RAM | Less than an eighth of each disk read once the clone is ready, and each disk's whole length once the guest read it to its end | +| `a_clone_s_time_settles_after_it_is_ready` | In RAM | The time daemon's restore check `ok`, after the restore's generation, without a violation, after a downtime that the host's monotonic clock measured, and a clone that still serves | +| `each_command_reattaches_to_the_clone_from_a_new_process` | Over a scratch image, then `reattach`, `probe`, `exec`, and `reattach` again, each from a new process | The launch's records, committed, with the clone's VM and no failure; each command `live`, with the clone's VMM, in a new session of the same broker, in a later epoch; the events of the activation as they were; an execution as nobody; and a stop that releases the lease, with the records | +| `a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity` | In RAM, then a `reattach` with another guest agent's build | Status 8, `reattached=failed`, without a session, and the record of a failure of the host, at the reattach, by the evidence of the runtime's mismatch, once NVX terminated the VMM, which quarantines nothing; the lease released, with the records; then another clone that starts and stops | +| `a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released` | In RAM, then a `stop` that the step hook pauses at `found`, while the test terminates the clone's VMM | Status 8, `reattached=live` and `stopped=exited`, and the record of a failure of an unknown cause, at the run, by the evidence of the clone's lost status, without a status of the guest's, once the VM exited, which quarantines nothing; the lease released, with the records; then another clone that starts and stops | +| `a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease` | Over a scratch image, then a tombstone of its generation, through NVX's store | A start that the tombstone refuses, without a lease; the running clone, which reattaches, probes, runs an execution, and stops, which releases its lease; and the generation's snapshot unchanged | +| `the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store` | In RAM, in a sandbox that NVX's store leased a generation to, as a start that ended before it recorded the clone leaves it: first moved, with another sandbox at its lease's path, which holds a lease of its own, then none; then in place, without a record of a clone, then with the record of an earlier clone, of another template | For the moved sandbox, a `stop` and a `reattach` that refuse it, since the sandbox at its lease's path does not hold its lease, and leave that sandbox's lease; in place, a stop that finds the clone `ended` at `planned`, and keeps the lease, whose template store no record names; a stop with another template store, which the release refuses; a `reattach` or a `stop` with `--template` naming the lease's own, which releases it; and a start and a stop in the sandbox after each | + +A test that fails keeps its template's directory, with the template store and +the platform's image, as long as it keeps the directory of a clone that leases +the template, whose VM might still run. + ### Running the VM tests The VM tests, `tests/vm_restart.rs`, `tests/vm_steps.rs`, -`tests/vm_layered.rs`, and `tests/vm_template.rs`, share their sandboxes and +`tests/vm_layered.rs`, `tests/vm_template.rs`, and `tests/vm_clone.rs`, share +their sandboxes and the fixtures of their layered roots through `tests/common`. A test that fails ends the VM that it left running, and the host program that it left paused, and keeps the sandbox's directory, whose path it prints, unless it can confirm @@ -2228,8 +2465,8 @@ configuration. A change to the adapter is a test input of CI's NVX microVM tests. Their [workflow](../../.github/workflows/run-nvx-microvm-tests.yml) builds the host-only configuration on each backend, in the jobs that run the guest and -sandbox steps, and runs its ignored tests, the adapter's VM scenarios and -template captures, with `NVX_HYPERVISOR` naming the backend, +sandbox steps, and runs its ignored tests, the adapter's VM scenarios, template +captures, and clones, with `NVX_HYPERVISOR` naming the backend, `NVX_ARTIFACTS_DIR` the checkout, into which the job downloaded NVX's artifacts, their provenance, and the fixtures of the layered scenarios with the guest artifacts, as NVX's own layered scenarios take them, and diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 443189922..73addc7e2 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -95,10 +95,51 @@ //! - `runtime-adapter-host reattach STORE` reattaches to the VM of STORE, which an earlier process //! launched and which it holds although it is not its child, attaches a new session to it, in //! the session's next epoch, and exchanges a `Ping`, leaving the VM running. +//! - `runtime-adapter-host start STORE --template TEMPLATES --generation G --distro FORMAT:PATH +//! --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity +//! UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY]` starts a clone of the generation G +//! of the template store TEMPLATES, an `nvx::host::template::Store`, in the sandbox store STORE, +//! which it creates if needed; see `runtime_adapter::clone`. It reads the generation from the +//! store's index, and, unless MIB mebibytes are given, its compatibility key, whose RAM is then +//! the clone's memory target, before it needs NVX's artifacts. A key that cannot be read is left +//! to NVX's plan, which reads it again, and attributes the failure, to the generation or to the +//! host, as it does for every clone. The store then leases the generation to +//! the sandbox, at the host's time, with `Store::acquire`, which refuses a generation that a +//! tombstone quarantines, or that is not eligible then, and NVX's `ClonePlan` checks the clone +//! against the generation, from the metadata of its files alone: its image, the raw file PATH, +//! an EROFS filesystem, `erofs`, of which the guest expects the identity UUID, of the length of +//! the template's placeholder, its upper, the ext4 scratch image PATH, of the geometry of the +//! template's scratch, or RAM, with the bindings that STORE stages for the images, as `stage` +//! stages them, and its memory target, in whole blocks, from the template's RAM to its capacity. +//! A request that the template cannot serve fails there, before any VMM starts. The command +//! records the clone in STORE, with the template store and the generation; see +//! `runtime_adapter::clone::CloneRecord`. OpenVMM then restores the generation's snapshot in +//! place, with the clone's devices, through NVX's supervisor, as the launch that the lease +//! reserves, and the restore sequence of `nvx::host::template` activates the clone through the +//! adapter's clone order, `runtime_adapter::host::CloneOrder`, with the clone's activation +//! record: the workload identity UID:GID, if given, as which every execution runs, whose account +//! the guest checks in the clone's root before it is ready, and the overlay mode MODE and the +//! scratch's journal policy POLICY, as a launch takes them. Once the clone is ready, the command +//! starts its sandbox, exchanges a `Ping` and the longest `Echo` with the guest agent, commits +//! the launch, and exits, leaving the VM running, and the sandbox holding the lease, which a stop +//! releases. A clone that fails, from its plan to its commit, fails with NVX's +//! `nvx::host::template::CloneFailure`, which attributes the failure by its evidence, and records +//! it in STORE from the clone's launch on: the command prints the failure's record, releases the +//! lease once the failure's teardown is resolved, and nothing of the clone runs, or keeps it, and +//! the records, for a later reattach, while it is not, and quarantines the generation through +//! its tombstone, once it released the sandbox's lock, for a failure of the template. A guest +//! agent that does not answer the `Ping` or the `Echo` fails the clone so, through +//! `Started::fail`, as the restore sequence attributes the failures of its sessions. A start +//! that fails otherwise, at a step of the adapter's own before the clone's launch, releases the +//! lease. //! - `runtime-adapter-host stop STORE` reattaches and attaches as `reattach` does, then stops the //! VM: it asks the guest agent to stop, which first tears down a layered root, waits for the VM //! to exit, and checks the VM's outcome report. A VM whose guest agent does not confirm the stop -//! is terminated. +//! is terminated. A stop of a clone releases the lease once the VM exited, and the clone's +//! records told how it ended, which the release removes, with the template store that the +//! clone's record names: a clone whose VM had exited when the stop began ended without its +//! holder, and one whose VM the guest ended, once the stop began, with a status that is evidence +//! of a failure, such as its repair's, failed, which ends the stop as it ends `start`. //! - `runtime-adapter-host probe STORE PROBE...` reattaches and attaches as `reattach` does, then //! runs each PROBE in the VM's layered root, and prints the facts that it found. A PROBE is the //! text of a probe: its name, then its arguments, each after one space; see @@ -117,28 +158,48 @@ //! have none. It starts in the root's directory DIR, or `/`, with exactly the environment of its //! `--env` entries, then, if the VM's network has a proxy, the proxy's URL as the guest sees it //! in `http_proxy`, `https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY`, unless the entries name any -//! of them; see `runtime_adapter::layered::with_proxy`. Before it attaches, the command checks -//! that the framing still carries each start with the proxy's URL, and runs none of the -//! executions if one no longer fits. It runs for at most a minute, with 16 KiB chunks and the -//! credit of 4 of them for its input and 8 for its output, which its standard output and error -//! share. The command keeps at most 16 MiB of each stream, and cancels an execution whose stream -//! would exceed them; see `runtime_adapter::exec::MAX_CAPTURE`. Execution N, from 1, has the key -//! `(B + N, E)`: E is the epoch of the command's attachment, and B the first 8 bytes of the -//! broker instance's ID, as a little-endian number. The guest refuses a key that it heard of +//! of them; see `runtime_adapter::layered::with_proxy`. Before it starts any execution, the +//! command checks that the framing still carries each start with the proxy's URL, and runs none +//! of the executions if one no longer fits. It runs for at most a minute, with 16 KiB chunks +//! and the credit of 4 of them for its input and 8 for its output, which its standard output and +//! error share. The command keeps at most 16 MiB of each stream, and cancels an execution whose +//! stream would exceed them; see `runtime_adapter::exec::MAX_CAPTURE`. Execution N, from 1, has +//! the key `(B + N, E)`: E is the epoch of the command's attachment, and B the first 8 bytes of +//! the broker instance's ID, as a little-endian number. The guest refuses a key that it heard of //! before, and the epochs of a broker instance only advance, while a broker restored from a //! snapshot has a new instance ID, so no command repeats the key of another's execution. //! //! Each command but `stage` takes NVX's artifacts from the directory that `NVX_ARTIFACTS_DIR` //! names, a repository checkout or an extracted release, and the VM's runtime identity from NVX's //! identity of them, with the guest agent's build that `--guest-build BUILD` names, this crate's -//! version by default; see `runtime_adapter::runtime`. Reattach finds a VM that another runtime -//! identity launched mismatched, and then signals nothing: the VM keeps running. +//! version by default; see `runtime_adapter::runtime`. Reattach finds a VM that booted cold under +//! another runtime identity mismatched, and then signals nothing: the VM keeps running. +//! +//! The sandbox of a clone holds the clone's lease, through whose path the commands that reattach +//! take the sandbox's lock. They reattach to a clone with `nvx::host::template::reattach_clone`, +//! in a new session of the adapter's clone order, in which the clone answers the Activate of its +//! record again, with the Ready that it answered once, and which they then use as `reattach` +//! attaches one. The reattach ends a clone that runs under another runtime identity, without a +//! word to its guest, or that fails the check of its Ready, through its verified identity, and +//! finishes the record of an earlier failure of the clone, whose teardown it resolves if it can. +//! The command then releases the lease of a clone whose VM ended without a failure, and fails; +//! ends as `start` does for a clone that failed; and keeps the lease and the records, and fails, +//! for a clone whose state cannot be determined. It releases a lease through the template store +//! that the clone's record names, once the lease is of it. A sandbox whose start ended before it +//! recorded the clone holds a lease of a template store that no record names: it records no +//! clone, or an earlier clone's, which a release leaves, of another template store perhaps. +//! `reattach STORE --template TEMPLATES` and `stop STORE --template TEMPLATES` release such a +//! lease through the template store TEMPLATES, in place of the record's, which the release refuses +//! unless the lease is of it. //! //! The commands print what they find as `KEY=VALUE` lines: //! -//! - `reattached`: what reattach found: `live`, `mismatched`, `exited`, or `indeterminate`; +//! - `reattached`: what reattach found: `live`, `mismatched`, `exited`, or `indeterminate`, and of +//! a clone, `live`, `ended`, `failed`, or `indeterminate`; //! - `differences`: the parts of the runtime identity in which a mismatched VM differs, separated //! by commas; +//! - `progress`: the step that a clone, which ended, reached: `planned`, `launching`, `spawned`, +//! `event`, `acknowledged`, `activating`, `ready`, `starting`, `started`, or `committed`; //! - `pid`: the VMM's process ID; //! - `instance` and `epoch`: the broker instance, in hexadecimal, and the epoch of the session //! that the command attached to; @@ -162,48 +223,83 @@ //! comma, or `none`; //! - `settled`: `reusable` once the execution settled, or `fenced`; //! - `stdout` and `stderr`: the program's standard output and error; -//! - `stopped`: how the stop ended: `graceful`, `forced`, or `timed-out`; +//! - `stopped`: how the stop ended: `graceful`, `forced`, or `timed-out`, or `exited` once a +//! clone's VM, which the command found running, had exited when the stop began, which stopped +//! nothing; //! - `outcome`: the category and status code of the VM's outcome report, such as `success 0`; //! - `teardown`: whether the VM's teardown completed: `complete` or `incomplete`; //! - `created`: how `create` served the template: `captured`, with a generation that a capture //! made and the store published, or `existing`, with one that the store held; //! - `generation`, `key`, and `class`: the store generation of the template that `create` served, //! and the identities of its compatibility key and of its capture class, each 64 hexadecimal -//! digits. +//! digits, or the store generation that `start` leased; +//! - `lease`: the identity of the lease that `start` took, 32 hexadecimal digits; +//! - `downtime` and `source`: the downtime of the leased generation's capture at the host's time, +//! in nanoseconds, and the clock that measured it: `monotonic`, the host's monotonic clock, on +//! the host's boot that captured the template, or `utc` otherwise; +//! - `released`: the identity of the lease that a stop, or a start that failed, released; +//! - `failure.KEY`: the record of a clone's failure, with these KEYs: +//! - `outcome`: what the failure is attributed to: `request`, `staging`, `host`, `unknown`, +//! `template`, or `repair`; +//! - `evidence`: what the host saw of it, which attributes it, such as `supplied-length`; +//! - `stage`: where the clone failed: `plan`, `launch`, `restore`, `activation`, `start`, +//! `commit`, `run`, or `reattach`; +//! - `progress`: the step that the clone reached, as `progress` spells it; +//! - `status`: the status with which the guest ended the VM, as OpenVMM's outcome report gives +//! it, or `none`; +//! - `teardown`: how the teardown that followed ended: `no-vm`, `exited`, `terminated`, or +//! `unresolved`; +//! - `code`: the code of the failure's error, such as `runtime-mismatch`; +//! - `quarantined`: whether the command quarantined the generation of a clone that failed: +//! `yes`, for a failure of the template, or `no`. //! //! Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each byte of a //! program's output that is not part of UTF-8 text as `\xHH`, with two hexadecimal digits. //! -//! For tests, `launch` takes a step hook from the variable `RUNTIME_ADAPTER_STEP_HOOK`, which -//! pauses the launch at a named step, for a time that the launch leaves out of its deadline, or -//! exits it there without unwinding, with status 5; see `runtime_adapter::hook`. +//! For tests, `launch`, `reattach`, and `stop` take a step hook from the variable +//! `RUNTIME_ADAPTER_STEP_HOOK`, which pauses the command at a named step, for a time that a launch +//! leaves out of its deadline, or exits it there without unwinding, with status 5; see +//! `runtime_adapter::hook`. `start` takes the +//! host's time, at which the store checks the generation's eligibility, from the variable +//! `RUNTIME_ADAPTER_HOST_TIME`, if it is set, as a clock fixture, such as another boot's; see +//! `runtime_adapter::clone::HOST_TIME_VARIABLE`. //! //! They exit with 0 once they did what they were asked, `exec` once every execution finished, -//! whatever its outcome, with 3 when reattach finds the VM mismatched, with 4 when the guest -//! reported that a stage of the root failed, and the VM ended with the stage's status and a -//! complete teardown, with 5 when the step hook ends a launch, with 6 when the guest failed a -//! mapping of the table, and the VM stopped gracefully, with success and a complete teardown, and +//! whatever its outcome, with 3 when reattach finds a VM that booted cold mismatched, with 4 when +//! the guest reported that a stage of the root failed, and the VM ended with the stage's status +//! and a complete teardown, with 5 when the step hook ends a command, with 6 when the guest failed +//! a mapping of the table, and the VM stopped gracefully, with success and a complete teardown, +//! with 8 when a clone failed, once nothing of it runs and the command released its lease, with 9 +//! when a clone failed, and its teardown is unresolved, so that the sandbox keeps the lease, and //! with 1, after a diagnostic on standard error, when they fail. Anything else is a usage error, -//! with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook. +//! with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook, or of +//! `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time. use std::env; use std::error::Error; use std::ffi::OsString; use std::fs; use std::io; +use std::num::NonZeroU64; use std::path::{Path, PathBuf}; use std::process::ExitCode; use std::time::{Duration, Instant}; +use nvx::activation::StoreGeneration; use nvx::exec::{Cause, Key, Limits, Outcome, Status, Stream, Timers, Window}; use nvx::host::artifacts::{Artifacts, absolute}; use nvx::host::instance::generate_capability; use nvx::host::launch::{DEFAULT_MEMORY_MIB, LaunchSpec}; -use nvx::host::report; +use nvx::host::nvxs::HostConnectionError; +use nvx::host::report::{self, Report}; use nvx::host::session; -use nvx::host::store::{IdentityPart, LOG_NAME, LaunchId, LockedStore, SandboxStore}; +use nvx::host::store::{CallerFiles, IdentityPart, LOG_NAME, LaunchId, LockedStore, SandboxStore}; use nvx::host::supervisor::{self, Reattached, Starting, StopOutcome, Vmm, VmmCommand}; -use nvx::host::template::{CAPTURES_NAME, Created, HostTime, Store, TRANSACTION_NAME}; +use nvx::host::template::{ + Acquired, CAPTURES_NAME, CloneControl, CloneFailure, ClonePlan, Created, DowntimeSource, + Eligibility, Evidence, FailureRecord, HostTime, Lease, ReattachedClone, RestoreDeadlines, + Store, TRANSACTION_NAME, reattach_clone, +}; use nvx::protocol::CAPABILITY_LEN; use nvx::protocol::root::ImageFormat; use nvx::uuid::Uuid; @@ -212,12 +308,13 @@ use runtime_adapter::GUEST_BUILD; use runtime_adapter::capture::{ DEFAULT_CPU_PROFILE, DEFAULT_PROCESSORS, DEFAULT_SCRATCH_LENGTH, Request, }; +use runtime_adapter::clone::{CloneRecord, host_time_from_env}; use runtime_adapter::exec::{self, AccountMode, HostExecution, Start}; use runtime_adapter::frame::{MAX_BODY, Message}; use runtime_adapter::hook::{Step, StepHook}; -use runtime_adapter::host::{self, Client, ClientError, Executions, OrderError}; +use runtime_adapter::host::{self, Client, ClientError, CloneOrder, Executions, OrderError}; use runtime_adapter::initramfs; -use runtime_adapter::layered::{LaunchRecord, LayeredLaunch, read_network, with_proxy}; +use runtime_adapter::layered::{Bindings, LaunchRecord, LayeredLaunch, read_network, with_proxy}; use runtime_adapter::probe::{Probe, ProbeReply, escape}; use runtime_adapter::root::{JournalPolicy, OverlayMode, RootFailure}; use runtime_adapter::runtime::Runtime; @@ -258,6 +355,14 @@ const ROOT_FAILED: u8 = 4; /// The exit status of a launch whose guest failed a mapping of the table, once the VM stopped /// gracefully, with success and a complete teardown. const TABLE_FAILED: u8 = 6; +/// The exit status of a command whose clone failed, as NVX attributes and records the failure, +/// once nothing of the clone runs: the failure's teardown is resolved, and the command released +/// the clone's lease. +const CLONE_FAILED: u8 = 8; +/// The exit status of a command whose clone failed, once the failure's teardown is unresolved: a +/// VM of the clone may still run, or NVX could not record the failure before it ended the VM, and +/// the sandbox keeps the clone's lease, and NVX's records of its launch, for a later reattach. +const CLONE_UNRESOLVED: u8 = 9; // The timers and limits of `exec` below are values chosen for tests that make a valid // configuration, within the limits that the guest and the host allow: chunks of at most // `exec::MAX_CHUNK` bytes, at most `worker::MAX_EXECUTION_MEMORY` bytes held in the guest for @@ -296,7 +401,14 @@ const USAGE: &str = concat!( " [--processors N]\n", " [--memory MIB] [--memory-capacity MIB] [--scratch-length BYTES]", " [--network FILE]\n", - " runtime-adapter-host (reattach | stop) STORE [--guest-build BUILD]\n", + " runtime-adapter-host (reattach | stop) STORE [--guest-build BUILD]", + " [--template TEMPLATES]\n", + " runtime-adapter-host start STORE --template TEMPLATES --generation G", + " --distro FORMAT:PATH\n", + " --distro-identity UUID --upper (scratch:PATH | ram) [--guest-build BUILD]", + " [--memory MIB]\n", + " [--workload-identity UID:GID] [--overlay-mode MODE]", + " [--scratch-journal POLICY]\n", " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]\n", " runtime-adapter-host exec STORE [--guest-build BUILD] EXECUTION [';' EXECUTION]...\n", " EXECUTION: [--identity UID:GID [--groups GID,...] [--capabilities N,...]", @@ -357,6 +469,22 @@ enum Command { guest_build: String, /// Whether to stop the VM. stop: bool, + /// The template store through which to release a clone's lease, in place of the one that + /// the clone's record names. + template: Option, + /// The step hook, for tests, if `RUNTIME_ADAPTER_STEP_HOOK` sets one. + hook: Option, + }, + /// Start a clone of a generation of a template. + Start { + /// The sandbox's store. + store: PathBuf, + /// The guest agent's build. + guest_build: String, + /// The clone. + request: Box, + /// The host's time, for tests, if `RUNTIME_ADAPTER_HOST_TIME` sets one. + time: Option, }, /// Reattach to the VM, and run probes in its layered root. Probe { @@ -495,6 +623,16 @@ struct Template { network: Option, } +/// The clone of `start`, as its options give it. +struct CloneOptions { + /// The template store. + templates: PathBuf, + /// The generation of the template store that the clone leases. + generation: StoreGeneration, + /// The clone's image, upper, memory target, and settings. + request: runtime_adapter::clone::Request, +} + /// The options of a command. #[derive(Default)] struct Options { @@ -504,6 +642,10 @@ struct Options { hypervisor: Option, /// `--guest-build`. guest_build: Option, + /// `--template`. + template: Option, + /// `--generation`. + generation: Option, /// `--distro`: the image's format and its file. distro: Option<(ImageFormat, PathBuf)>, /// `--distro-identity`. @@ -586,7 +728,21 @@ fn main() -> ExitCode { store, guest_build, stop, - } => reattach(&store, &guest_build, stop), + template, + hook, + } => reattach( + &store, + &guest_build, + stop, + template.as_deref(), + hook.as_ref(), + ), + Command::Start { + store, + guest_build, + request, + time, + } => start(&store, &guest_build, &request, time), Command::Probe { store, guest_build, @@ -605,16 +761,17 @@ fn main() -> ExitCode { } /// Parses the program's arguments, with the step hook that `RUNTIME_ADAPTER_STEP_HOOK` sets for a -/// launch, or says why they are not a command. +/// launch, or a reattach or a stop, or says why they are not a command. fn parse(arguments: Vec) -> Result { let mut arguments = arguments.into_iter(); let Some(name) = arguments.next() else { return Ok(Command::Report); }; let name = match name.to_str() { - Some(name @ ("stage" | "launch" | "create" | "reattach" | "stop" | "probe" | "exec")) => { - name.to_owned() - } + Some( + name @ ("stage" | "launch" | "create" | "reattach" | "stop" | "start" | "probe" + | "exec"), + ) => name.to_owned(), _ => return Err(format!("unknown command {name:?}")), }; let store = PathBuf::from(arguments.next().ok_or_else(|| match name.as_str() { @@ -753,6 +910,35 @@ fn parse(arguments: Vec) -> Result { }), } } + "start" => { + let templates = options.template.ok_or("start needs --template")?; + let generation = options.generation.ok_or("start needs --generation")?; + let (format, distro) = options.distro.ok_or("start needs --distro")?; + let identity = options.identity.ok_or("start needs --distro-identity")?; + let scratch = options.upper.ok_or("start needs --upper")?; + if scratch.is_none() && options.scratch_journal.is_some() { + return Err("--scratch-journal needs --upper scratch:PATH".to_owned()); + } + Command::Start { + store, + guest_build, + request: Box::new(CloneOptions { + templates, + generation, + request: runtime_adapter::clone::Request { + distro, + format, + identity, + scratch, + memory: options.memory, + workload_identity: options.workload_identity, + overlay_mode: options.overlay_mode, + scratch_journal: options.scratch_journal, + }, + }), + time: host_time_from_env()?, + } + } "probe" if probes.is_empty() => return Err("probe needs a probe".to_owned()), "probe" => Command::Probe { store, @@ -763,6 +949,8 @@ fn parse(arguments: Vec) -> Result { store, guest_build, stop: stop == "stop", + template: options.template, + hook: StepHook::from_env()?, }, }) } @@ -788,7 +976,7 @@ fn parse_option( let hypervisor = text(value)?.parse().map_err(|error| describe(&error))?; options.hypervisor = Some(hypervisor); } - ("launch" | "stage" | "create", "--distro") if options.distro.is_none() => { + ("launch" | "stage" | "create" | "start", "--distro") if options.distro.is_none() => { let value = text(value)?; let distro = value.split_once(':').and_then(|(format, path)| { let format = ImageFormat::from_name(format)?; @@ -798,17 +986,19 @@ fn parse_option( format!("--distro takes FORMAT:PATH, with the format erofs or gpt, not {value:?}") })?); } - ("launch" | "stage" | "create", "--distro-identity") if options.identity.is_none() => { + ("launch" | "stage" | "create" | "start", "--distro-identity") + if options.identity.is_none() => + { let value = text(value)?; let identity = Uuid::parse(&value) .ok_or_else(|| format!("--distro-identity takes a UUID, not {value:?}"))?; options.identity = Some(identity); } - ("launch" | "stage", "--upper") if options.upper.is_none() => { + ("launch" | "stage" | "start", "--upper") if options.upper.is_none() => { let value = text(value)?; - // Staging binds a scratch image; only a launch puts the upper in RAM. + // Staging binds a scratch image; only a launch or a clone puts the upper in RAM. let upper = match value.split_once(':') { - _ if value == "ram" && name == "launch" => Some(None), + _ if value == "ram" && name != "stage" => Some(None), Some(("scratch", path)) if !path.is_empty() => Some(Some(PathBuf::from(path))), _ => None, }; @@ -820,14 +1010,14 @@ fn parse_option( ("launch", "--runtime-dir") if options.runtime.is_none() => { options.runtime = Some(text(value)?); } - ("launch", "--overlay-mode") if options.overlay_mode.is_none() => { + ("launch" | "start", "--overlay-mode") if options.overlay_mode.is_none() => { let value = text(value)?; let mode = OverlayMode::from_name(&value).ok_or_else(|| { format!("--overlay-mode takes trusted-lowers or untrusted-lowers, not {value:?}") })?; options.overlay_mode = Some(mode); } - ("launch", "--scratch-journal") if options.scratch_journal.is_none() => { + ("launch" | "start", "--scratch-journal") if options.scratch_journal.is_none() => { let value = text(value)?; let policy = JournalPolicy::from_name(&value).ok_or_else(|| { format!("--scratch-journal takes allowed or forbidden, not {value:?}") @@ -841,7 +1031,7 @@ fn parse_option( format!("--first-partition-bound takes a positive number of bytes, not {value:?}") })?); } - ("launch", "--workload-identity") if options.workload_identity.is_none() => { + ("launch" | "start", "--workload-identity") if options.workload_identity.is_none() => { let value = text(value)?; let identity = ids(&value) .filter(|(uid, gid)| [uid, gid].into_iter().all(|id| (1..u32::MAX).contains(id))); @@ -882,9 +1072,23 @@ fn parse_option( format!("--processors takes a positive number, not {value:?}") })?); } - ("create", "--memory") if options.memory.is_none() => { + ("create" | "start", "--memory") if options.memory.is_none() => { options.memory = Some(mebibytes(option, &text(value)?)?); } + ("start" | "reattach" | "stop", "--template") if options.template.is_none() => { + options.template = Some(PathBuf::from(value)); + } + ("start", "--generation") if options.generation.is_none() => { + let value = text(value)?; + let generation = value + .parse::() + .ok() + .filter(|_| value.bytes().all(|byte| byte.is_ascii_digit())); + options.generation = + Some(StoreGeneration::new(generation.ok_or_else(|| { + format!("--generation takes a positive number, not {value:?}") + })?)); + } ("create", "--memory-capacity") if options.memory_capacity.is_none() => { options.memory_capacity = Some(mebibytes(option, &text(value)?)?); } @@ -1270,13 +1474,13 @@ fn launch( Ok(ExitCode::SUCCESS) } -/// Runs the step hook `hook`, if any, at the launch's `step`, and returns how long it paused the -/// launch there, which the launch leaves out of its deadline. +/// Runs the step hook `hook`, if any, at the command's `step`, and returns how long it paused the +/// command there, which a launch leaves out of its deadline. fn step(hook: Option<&StepHook>, step: Step) -> Result { match hook { Some(hook) => hook .at(step) - .map_err(|error| format!("the step hook cannot pause the launch at {step}: {error}")), + .map_err(|error| format!("the step hook cannot pause the command at {step}: {error}")), None => Ok(Duration::ZERO), } } @@ -1510,50 +1714,636 @@ impl Drop for TemporaryDirectory { } } -/// Reattaches to the VM of the sandbox store `store`, with the guest agent's build `guest_build`, -/// attaches a session to it, and exchanges a `Ping`, then stops the VM if `stop` asks. -fn reattach(store: &Path, guest_build: &str, stop: bool) -> Result { +/// Starts a clone of the generation of the template store that `clone` names, as it asks, in the +/// sandbox store `store`, which it creates if needed, with the guest agent's build `guest_build`, +/// at the host's time `time`, or at the time that `HostTime::now` samples without one. +fn start( + store: &Path, + guest_build: &str, + clone: &CloneOptions, + time: Option, +) -> Result { + // A request that the adapter's plan refuses, and a generation that the template store does not + // hold, fail the start before it needs NVX's artifacts or the sandbox's store. + let plan = clone.request.plan().map_err(|error| describe(&error))?; + let templates = Store::open(&clone.templates).map_err(|error| describe(&error))?; + let generation = templates + .index() + .map_err(|error| describe(&error))? + .get(clone.generation) + .cloned() + .ok_or_else(|| { + format!( + "the template store {} holds no generation {}", + templates.dir().display(), + clone.generation + ) + })?; + // The clone's memory target is the request's, or else the template's RAM, which the + // generation's key gives. A key that cannot be read gives none: NVX's plan reads the key + // again, once the store leased the generation, and attributes the failure, to the generation + // or to the host, before it checks a memory target. + let memory = clone + .request + .memory + .or_else(|| templates.read_key(&generation).ok().map(|key| key.memory())); let runtime = runtime(guest_build)?; - let store = SandboxStore::open(store).map_err(|error| describe(&error))?; - let mut locked = store - .lock(Some(Instant::now() + LOCK_TIMEOUT)) + runtime + .artifacts + .probe() + .map_err(|error| describe(&error))?; + let now = match time { + Some(time) => time, + None => HostTime::now().map_err(|error| describe(&error))?, + }; + let sandbox = SandboxStore::create(store).map_err(|error| describe(&error))?; + let mut acquired = templates + .acquire( + generation.generation(), + sandbox.dir(), + &CallerFiles::none(), + &now, + Some(Instant::now() + LOCK_TIMEOUT), + ) + .map_err(|error| describe(&error))?; + println!("generation={}", generation.generation()); + println!("lease={}", acquired.lease.id()); + if let Eligibility::Eligible(downtime) = generation.eligibility(&now) { + println!("downtime={}", downtime.nanos); + println!("source={}", source_name(downtime.source)); + } + let clone = CloneLaunch { + templates: &templates, + request: &clone.request, + plan, + memory, + runtime: &runtime, + guest_build, + }; + match clone.run(&mut acquired) { + Ok(()) => Ok(ExitCode::SUCCESS), + Err(Failed::Clone(failure)) => clone_failed(Ok(&templates), acquired.locked, failure), + Err(Failed::Command(problem)) => { + // The command failed before the clone's launch, so nothing of it runs. + match release_lease(&templates, &mut acquired.locked) { + Ok(()) => Err(problem), + Err(release) => Err(format!("{problem}; {release}")), + } + } + } +} + +/// How the launch of a clone failed, once the store leased its generation. +enum Failed { + /// The clone failed, as NVX attributes the failure, and records it from the clone's launch on. + Clone(CloneFailure), + /// The command failed otherwise, with this problem, as a step of the adapter's own did before + /// the clone's launch. + Command(String), +} + +/// The clone that `start` launches, once the store leased the generation. +struct CloneLaunch<'a> { + /// The template store. + templates: &'a Store, + /// The clone, as the command's options give it. + request: &'a runtime_adapter::clone::Request, + /// The adapter's plan of the clone's image over its upper. + plan: nvx::host::sandbox::SandboxPlan, + /// The clone's memory target, in bytes, or none if the request names none, and the + /// generation's key, whose RAM it would be, cannot be read. + memory: Option, + /// The runtime of NVX's artifacts. + runtime: &'a Runtime, + /// The guest agent's build. + guest_build: &'a str, +} + +impl CloneLaunch<'_> { + /// Plans, records, and launches the clone in the sandbox of `acquired`, which leased the + /// generation, activates it, starts its sandbox, exchanges a `Ping` and the longest `Echo` + /// with its guest agent, and commits the launch, leaving the clone running. + /// + /// Fails with the clone's failure, as NVX's plan, launch, restore sequence, and commit return + /// it, or as `Started::fail` records it for a guest agent that does not answer the command + /// once the clone's sandbox started, which ended the launch, unless its teardown is + /// unresolved, and with the problem of a step of the adapter's own before the launch + /// otherwise. + fn run(self, acquired: &mut Acquired) -> Result<(), Failed> { + let Self { + templates, + request, + plan, + memory, + runtime, + guest_build, + } = self; + let staged = Staged::read(acquired.locked.dir()) + .map_err(|error| Failed::Command(describe(&error)))?; + let bindings = staged.map_or_else(Bindings::default, |staged| staged.bindings(&plan)); + // No template serves a memory target of 0, which stands for the one of a key that could + // not be read: the plan fails as it reads the key again, before it checks the target. + let clone_request = request + .clone_request(plan.clone(), &bindings, memory.unwrap_or(0)) + .map_err(|error| Failed::Command(binding_error(&error)))?; + let clone_plan = ClonePlan::new( + templates, + acquired, + &clone_request, + Instant::now() + LOCK_TIMEOUT, + ) + .map_err(Failed::Clone)?; + let record = CloneRecord { + template: templates.dir(), + generation: acquired.lease.generation(), + kernel_command_line: clone_plan.key().command_line(), + plan: &plan, + bindings: &bindings, + }; + record + .write(acquired.locked.dir()) + .map_err(|error| Failed::Command(describe(&error)))?; + let restoring = clone_plan + .launch(acquired, &runtime.artifacts.openvmm, &runtime.identity) + .map_err(Failed::Clone)?; + println!("pid={}", restoring.starting().process().reference().pid); + let mut order = clone_order(guest_build); + // The restore sequence, and the commit, end the launch that they fail. + let activated = restoring + .activate(&mut order, RestoreDeadlines::default()) + .map_err(Failed::Clone)?; + println!("root=ready"); + println!("build={}", escape(guest_build)); + let mut started = activated.start(&mut order).map_err(Failed::Clone)?; + println!("table=applied"); + println!("mapped=0"); + println!("sandbox=started"); + let deadline = Instant::now() + SESSION_TIMEOUT; + let client = started.session(); + let answered = client + .ping(deadline) + .and_then(|()| client.echo(&echo_body(), deadline)); + if let Err(error) = answered { + let (evidence, error) = unanswered(&order, error, deadline); + return Err(Failed::Clone(started.fail(evidence, error))); + } + started.commit().map_err(Failed::Clone)?; + Ok(()) + } +} + +/// Returns the evidence and the error of a failure of the session of a clone whose sandbox +/// started, in which the client failed, with `error`, a request of the command's own by +/// `deadline`, as NVX's restore sequence attributes the failures of its own sessions: the passing +/// of the deadline, the loss of a session that the clone's `order` takes for lost, or a failure of +/// the protocol otherwise. +fn unanswered( + order: &CloneOrder<'_>, + error: ClientError, + deadline: Instant, +) -> (Evidence, nvx::Error) { + const PROBLEM: &str = "the guest agent did not answer once the clone's sandbox started"; + let timed_out = matches!(error, ClientError::Session(HostConnectionError::TimedOut)); + if timed_out || Instant::now() >= deadline { + return ( + Evidence::Timeout, + nvx::Error::timed_out(PROBLEM).with_source(error), + ); + } + let error = OrderError::Client(error); + let evidence = if order.lost(&error) { + Evidence::Reset + } else { + Evidence::Protocol + }; + (evidence, nvx::Error::protocol(PROBLEM).with_source(error)) +} + +/// Returns the adapter's order of a clone, with the guest agent's build `build`, which the restore +/// sequence of a clone's start, and a reattach to it, take, the one place that makes it. A clone's +/// record announces no mapping table: the adapter maps nothing into a clone. +fn clone_order(build: &str) -> CloneOrder<'_> { + CloneOrder { build, table: &[] } +} + +/// Ends a command whose clone, in the sandbox of `locked`, failed with `failure`, as NVX +/// attributes and records the failure, and returns its exit status. +/// +/// It prints the failure's record. Once the failure's teardown is resolved, nothing of the clone +/// runs, and it releases the clone's lease through the template store `templates`, which removes +/// the records of the clone's launch, its failure's included, so that the sandbox may lease a +/// generation again. While the teardown is unresolved, the sandbox keeps the lease, and the +/// records, for a later reattach. Then, once it released the sandbox's lock, which comes after the +/// template store's in NVX's lock order, it quarantines the leased generation through its +/// tombstone if the failure is the template's, as `CloneFailure::quarantine` does, and prints +/// whether it did. +fn clone_failed( + templates: Result<&Store, String>, + mut locked: LockedStore, + failure: CloneFailure, +) -> Result { + print_clone_failure(failure.record()); + let mut problem = describe(failure.error()); + let log = locked.log_path(); + if log.is_file() { + problem = format!("{problem}; see the VMM's log {}", log.display()); + } + let resolved = failure.teardown().is_resolved(); + let released = match &templates { + Ok(templates) if resolved => release_lease(templates, &mut locked), + Ok(_) => Ok(()), + Err(problem) => Err(format!("the sandbox keeps its lease: {problem}")), + }; + drop(locked); + let quarantined = templates.and_then(|templates| { + failure + .quarantine(templates, Some(Instant::now() + LOCK_TIMEOUT)) + .map_err(|error| { + format!( + "the failure's generation was not quarantined: {}", + describe(&error) + ) + }) + }); + if let Ok(quarantined) = quarantined { + println!("quarantined={}", if quarantined { "yes" } else { "no" }); + } + match (released, quarantined) { + (Ok(()), Ok(_)) => { + eprintln!("runtime-adapter-host: {problem}"); + Ok(ExitCode::from(if resolved { + CLONE_FAILED + } else { + CLONE_UNRESOLVED + })) + } + (Err(release), _) => Err(format!("{problem}; {release}")), + (Ok(()), Err(quarantine)) => Err(format!("{problem}; {quarantine}")), + } +} + +/// Prints the `record` of a clone's failure: its outcome, the evidence that attributes it, the +/// stage at which the clone failed, how far the clone got, the guest's status, if OpenVMM's +/// outcome report gives one, how the teardown that followed ended, and the error's code. +fn print_clone_failure(record: &FailureRecord) { + println!("failure.outcome={}", record.outcome); + println!("failure.evidence={}", record.evidence); + println!("failure.stage={}", record.stage); + println!("failure.progress={}", record.progress); + println!( + "failure.status={}", + record + .status + .map_or("none".to_owned(), |status| status.to_string()) + ); + println!("failure.teardown={}", record.teardown.as_str()); + println!("failure.code={}", record.code); +} + +/// Releases the lease of the template store `templates` that the sandbox of `locked` holds, once +/// nothing of its clone's launch runs, and prints it. +fn release_lease(templates: &Store, locked: &mut LockedStore) -> Result<(), String> { + let released = templates + .release(locked) + .map_err(|error| format!("the sandbox keeps its lease: {}", describe(&error)))?; + if let Some(lease) = released { + println!("released={}", lease.id()); + } + Ok(()) +} + +/// Returns the name of `source`, as `start` prints it. +fn source_name(source: DowntimeSource) -> &'static str { + match source { + DowntimeSource::HostMonotonic => "monotonic", + DowntimeSource::Utc => "utc", + } +} + +/// Opens the template store whose lease the sandbox of `locked` holds: `named`, which the release +/// checks against the lease, or else the one that the record of the sandbox's clone names, once +/// the lease is of it. A sandbox whose start ended before it recorded its clone records none, or +/// an earlier clone, of another template store perhaps, which a release leaves: only `named` then +/// names the lease's. +fn clone_templates(locked: &LockedStore, named: Option<&Path>) -> Result { + if let Some(named) = named { + return Store::open(named).map_err(|error| describe(&error)); + } + let unnamed = |problem: String| { + format!( + "{problem}, so the template store of the lease of the sandbox in {} is unknown: name \ + it with --template TEMPLATES", + locked.dir().display() + ) + }; + let (templates, _) = CloneRecord::read_template(locked.dir()) + .map_err(|error| unnamed(describe(&error)))? + .ok_or_else(|| unnamed("the sandbox records no clone".to_owned()))?; + let store = Store::open(&templates).map_err(|error| unnamed(describe(&error)))?; + match Lease::of(locked).map_err(|error| describe(&error))? { + Some(lease) if lease.store() == store.id() => Ok(store), + _ => Err(unnamed(format!( + "the sandbox's record of its clone names the template store {}, which did not grant \ + its lease", + templates.display() + ))), + } +} + +/// Opens the sandbox store `path`, and takes its lock, through the path of the lease that the +/// sandbox holds, if it holds one, as a release requires, and returns whether the sandbox holds +/// one, as it reads it under the lock: an acquisition binds the lease under the sandbox's lock +/// too. +/// +/// A lease names the sandbox by its path with its links resolved, so the lock that a second +/// attempt takes through it names the sandbox as the lease does. The sandbox there must hold the +/// same lease: a sandbox that holds a lease must not move, and the lease of one that moved names +/// a path at which another sandbox, or none, may be, which the command must not take for it. +fn lock_sandbox(path: &Path) -> Result<(SandboxStore, LockedStore, bool), String> { + let deadline = Instant::now() + LOCK_TIMEOUT; + let store = SandboxStore::open(path).map_err(|error| describe(&error))?; + let locked = store + .lock(Some(deadline)) + .map_err(|error| describe(&error))?; + let lease = match Lease::of(&store).map_err(|error| describe(&error))? { + None => return Ok((store, locked, false)), + Some(lease) if lease.sandbox() == locked.dir() => return Ok((store, locked, true)), + Some(lease) => lease, + }; + // The lock through the lease's path is the same lock when both paths name the sandbox. + drop(locked); + drop(store); + let named = lease.sandbox(); + let moved = |problem: &str| { + format!( + "the lease of the sandbox in {} names the sandbox in {}, {problem}: a sandbox that \ + holds a lease must not move", + path.display(), + named.display() + ) + }; + let store = SandboxStore::open(named).map_err(|error| moved(&describe(&error)))?; + let locked = store + .lock(Some(deadline)) .map_err(|error| describe(&error))?; - let vmm = match hold(&mut locked, &runtime)? { - Ok(vmm) => vmm, + match Lease::of(&store).map_err(|error| describe(&error))? { + Some(held) if held == lease && held.sandbox() == locked.dir() => Ok((store, locked, true)), + Some(held) if held == lease => Err(format!( + "the lease of the sandbox in {} does not name it by the path through which its lock is \ + taken", + named.display() + )), + _ => Err(moved("which does not hold it")), + } +} + +/// The VM of a sandbox that a command found, as [`find`] returns it, under the sandbox's lock, +/// which the command holds. +struct Found { + /// The sandbox's store. + store: SandboxStore, + /// The sandbox's lock, which names a clone's sandbox by its lease's path. + locked: LockedStore, + /// The VM. + vmm: Vmm, + /// The session of the adapter's framing that the command holds of the VM. + session: Session, + /// Whether the VM is a clone, whose lease the sandbox holds. + leased: bool, +} + +/// The session that a command holds of the VM that it found. +enum Session { + /// None yet, of a VM that booted cold, to which the command attaches one once it needs it, as + /// [`Found::attach`] does. + Cold, + /// The one in which a clone answered the Activate of its record again, with the Ready that it + /// answered once, as `reattach_clone` checks it. + Clone(Box), +} + +impl Found { + /// Returns the session that the command holds of the VM, attaching one, with the sandbox's + /// capability, by `deadline`, to a VM that booted cold. + fn attach(&mut self, deadline: Instant) -> Result { + match std::mem::replace(&mut self.session, Session::Cold) { + Session::Clone(client) => Ok(*client), + Session::Cold => { + let capability = self + .store + .read_capability() + .map_err(|error| describe(&error))?; + connect(&self.vmm, &capability, deadline) + } + } + } +} + +/// Finds the VM of the sandbox store `path`, under the runtime identity of `runtime`, with the +/// guest agent's build `guest_build`, and prints what it found. Returns the VM, which runs, or the +/// status with which the command ends otherwise. +/// +/// It takes the sandbox's lock through the path of the lease that the sandbox holds, if it holds +/// one, which it reads under the lock, and refuses a sandbox whose lease names a path at which no +/// sandbox holds the same lease, as [`lock_sandbox`] does. The VM of a sandbox without a lease +/// booted cold: it holds the VM again, as `supervisor::reattach` does, and returns a mismatched +/// VM's status, which it leaves running. A sandbox with a lease holds a clone, which it reattaches +/// to with `reattach_clone`, in a new session of the adapter's clone order, in which the clone +/// answers the Activate of its record again, with its Ready, and which ends a clone that it may +/// not keep: one that runs under another runtime identity, without a word to its guest, or that +/// fails the check of its Ready. A clone that ended releases its lease, and one that failed, or +/// whose earlier failure the reattach finishes, ends the command as [`clone_failed`] does, through +/// the template store that [`clone_templates`] opens with `templates`. A clone whose state cannot +/// be determined keeps its lease and its records, and fails the command; see [`conclude`]. +fn find( + path: &Path, + runtime: &Runtime, + guest_build: &str, + templates: Option<&Path>, +) -> Result, String> { + let (store, mut locked, leased) = lock_sandbox(path)?; + if !leased { + let vmm = match hold(&mut locked, runtime)? { + Ok(vmm) => vmm, + Err(status) => return Ok(Err(status)), + }; + return Ok(Ok(Found { + store, + locked, + vmm, + session: Session::Cold, + leased: false, + })); + } + let reattached = reattach_leased(&mut locked, runtime, guest_build)?; + let found = match &reattached { + ReattachedClone::Live { .. } => "live", + ReattachedClone::Ended { .. } => "ended", + ReattachedClone::Failed(_) => "failed", + ReattachedClone::Indeterminate(_) => "indeterminate", + _ => return Err(UNKNOWN_STATE.to_owned()), + }; + println!("reattached={found}"); + let ReattachedClone::Live { vmm, session } = reattached else { + return conclude(reattached, locked, templates).map(Err); + }; + println!("pid={}", vmm.record().process().pid); + print_identity(&session); + Ok(Ok(Found { + store, + locked, + vmm: *vmm, + session: Session::Clone(Box::new(session)), + leased: true, + })) +} + +/// What a command says of a clone that `reattach_clone` found in a state that this build does not +/// know. +const UNKNOWN_STATE: &str = "the reattach found the clone in a state that this build does not know"; + +/// Reattaches to the clone of the sandbox of `locked`, under the runtime identity of `runtime`, +/// with `reattach_clone`, in a new session of the adapter's clone order, with the guest agent's +/// build `guest_build`. +fn reattach_leased( + locked: &mut LockedStore, + runtime: &Runtime, + guest_build: &str, +) -> Result, String> { + let mut order = clone_order(guest_build); + let deadline = Instant::now() + SESSION_TIMEOUT; + reattach_clone( + locked, + &runtime.identity, + &mut order, + deadline, + FORCED_TIMEOUT, + ) + .map_err(|error| describe(&error)) +} + +/// Ends a command on the clone of the sandbox of `locked` that `reattached`, what +/// `reattach_clone` found of it, tells does not run, and returns the command's exit status, or +/// fails. It releases the lease of a clone that ended, through the template store `templates`, or +/// the one that the clone's record names, as [`clone_templates`] opens it, and fails; ends as +/// [`clone_failed`] does for a clone that failed; and keeps the lease and the records, and fails, +/// for a clone whose state cannot be determined, or that runs. +fn conclude( + reattached: ReattachedClone, + mut locked: LockedStore, + templates: Option<&Path>, +) -> Result { + match reattached { + ReattachedClone::Ended { progress, report } => { + println!("progress={progress}"); + if let Some(report) = &report { + print_outcome(report); + } + let templates = clone_templates(&locked, templates)?; + release_lease(&templates, &mut locked)?; + Err(format!( + "no VM of the clone runs: it ended at its step {progress}, and the sandbox's lease \ + is released" + )) + } + ReattachedClone::Failed(failure) => { + let templates = clone_templates(&locked, templates); + clone_failed(templates.as_ref().map_err(Clone::clone), locked, failure) + } + ReattachedClone::Indeterminate(reason) => Err(format!( + "whether the clone's VM runs cannot be determined, so the sandbox keeps its lease: \ + {reason}" + )), + ReattachedClone::Live { .. } => Err( + "the clone's VM runs again, which the command did not expect, so the sandbox keeps \ + its lease" + .to_owned(), + ), + _ => Err(UNKNOWN_STATE.to_owned()), + } +} + +/// Reattaches to the VM of the sandbox store `store`, with the guest agent's build `guest_build`, +/// runs the step hook `hook`, if any, once it found the VM, at [`Step::Found`], then attaches a +/// session to the VM, and exchanges a `Ping`, then stops the VM if `stop` asks. +/// +/// The stop of a clone leaves the clone's records, which the release of its lease removes, to +/// tell how the clone ended first, as `reattach_clone` reads them. A clone's VM that had exited +/// when the stop began ended without its holder, as it may at any time: the stop, +/// `Vmm::stop_unless_exited`, stops nothing then, and keeps the runtime record of its launch, +/// which tells so, and the command prints `stopped=exited`, and ends as [`find`] does for a clone +/// that it finds so. A VM that exited once the stop began ended through the stop, unless the +/// guest's status that ended it is evidence of a failure, which ends the command as +/// [`clone_failed`] does. The lease of a clone that ended otherwise is released through the +/// template store `templates`, or the one that the clone's record names, as [`clone_templates`] +/// opens it. A VM that booted cold, whose end the command attributes to nothing, stops with +/// `Vmm::stop`, which ends one that had exited by then as one that it stopped: its outcome report +/// then tells how its guest ended. +fn reattach( + store: &Path, + guest_build: &str, + stop: bool, + templates: Option<&Path>, + hook: Option<&StepHook>, +) -> Result { + let runtime = runtime(guest_build)?; + let mut found = match find(store, &runtime, guest_build, templates)? { + Ok(found) => found, Err(status) => return Ok(status), }; + step(hook, Step::Found)?; let deadline = Instant::now() + SESSION_TIMEOUT; - let attached = store - .read_capability() - .map_err(|error| describe(&error)) - .and_then(|capability| connect(&vmm, &capability, deadline)) - .and_then(|mut client| { - client - .ping(deadline) - .map_err(|error| format!("the guest agent did not answer: {}", describe(&error)))?; - Ok(client) - }); + let attached = found.attach(deadline).and_then(|mut client| { + client + .ping(deadline) + .map_err(|error| format!("the guest agent did not answer: {}", describe(&error)))?; + Ok(client) + }); if !stop { return attached.map(|_| ExitCode::SUCCESS); } + let Found { + store, + mut locked, + vmm, + leased, + .. + } = found; // A VM whose guest agent cannot confirm the stop is terminated. let mut failure = None; - let stopped = vmm - .stop( + let graceful = |_: &Vmm, deadline: Instant| -> Result<(), String> { + failure = attached?.stop(deadline).map_err(|error| { + format!( + "the guest agent did not confirm the stop: {}", + describe(&error) + ) + })?; + Ok(()) + }; + let stopped = if leased { + // A clone's VM that had exited when the stop began ended without its holder: the stop + // stops nothing then, and keeps the runtime record of its launch, which tells so. + vmm.stop_unless_exited( &mut locked, Instant::now() + STOP_TIMEOUT, Some(FORCED_TIMEOUT), - |_, deadline| { - failure = attached?.stop(deadline).map_err(|error| { - format!( - "the guest agent did not confirm the stop: {}", - describe(&error) - ) - })?; - Ok(()) - }, + graceful, ) - .map_err(|error| describe(&error))?; + } else { + vmm.stop( + &mut locked, + Instant::now() + STOP_TIMEOUT, + Some(FORCED_TIMEOUT), + graceful, + ) + .map(Some) + } + .map_err(|error| describe(&error))?; + let Some(stopped) = stopped else { + println!("stopped=exited"); + let reattached = reattach_leased(&mut locked, &runtime, guest_build)?; + return conclude(reattached, locked, templates); + }; println!( "stopped={}", match stopped.outcome { @@ -1562,6 +2352,16 @@ fn reattach(store: &Path, guest_build: &str, stop: bool) -> Result "timed-out", } ); + // The store verified that the VM exited, unless the stop timed out. + if leased && stopped.outcome != StopOutcome::TimedOut { + match reattach_leased(&mut locked, &runtime, guest_build)? { + ReattachedClone::Ended { .. } => { + let templates = clone_templates(&locked, templates)?; + release_lease(&templates, &mut locked)?; + } + reattached => return conclude(reattached, locked, templates), + } + } if let Some(problem) = stopped.graceful_error { return Err(problem); } @@ -1586,17 +2386,12 @@ fn reattach(store: &Path, guest_build: &str, stop: bool) -> Result Result { let runtime = runtime(guest_build)?; - let store = SandboxStore::open(store).map_err(|error| describe(&error))?; - let mut locked = store - .lock(Some(Instant::now() + LOCK_TIMEOUT)) - .map_err(|error| describe(&error))?; - let vmm = match hold(&mut locked, &runtime)? { - Ok(vmm) => vmm, + let mut found = match find(store, &runtime, guest_build, None)? { + Ok(found) => found, Err(status) => return Ok(status), }; let deadline = Instant::now() + SESSION_TIMEOUT; - let capability = store.read_capability().map_err(|error| describe(&error))?; - let mut client = connect(&vmm, &capability, deadline)?; + let mut client = found.attach(deadline)?; for probe in probes { let reply = client .probe(probe, deadline) @@ -1618,20 +2413,16 @@ fn probe(store: &Path, guest_build: &str, probes: &[Probe]) -> Result Result { let runtime = runtime(guest_build)?; - let store = SandboxStore::open(store).map_err(|error| describe(&error))?; - let mut locked = store - .lock(Some(Instant::now() + LOCK_TIMEOUT)) - .map_err(|error| describe(&error))?; - let vmm = match hold(&mut locked, &runtime)? { - Ok(vmm) => vmm, + let mut found = match find(store, &runtime, guest_build, None)? { + Ok(found) => found, Err(status) => return Ok(status), }; // Each execution finds the URL of the proxy of the VM's network, if any, in its environment, // from the plan that the launch recorded. The proxy's variables lengthen a start, so the - // command checks every start again before it attaches, rather than fail once it sent the - // starts before it. A start takes its key once the command attached, and a key's encoding has - // the same size whatever its value. - let plan = LaunchRecord::read_plan(store.dir()).map_err(|error| describe(&error))?; + // command checks every start again before it starts any, rather than fail once it sent the + // starts before it, and, for a VM that booted cold, before it attaches. A start takes its key + // once the command attached, and a key's encoding has the same size whatever its value. + let plan = LaunchRecord::read_plan(found.store.dir()).map_err(|error| describe(&error))?; let mut starts = Vec::with_capacity(executions.len()); for (number, execution) in (1..).zip(executions) { let start = execution.start(Key::new(0, 0)); @@ -1645,8 +2436,7 @@ fn exec(store: &Path, guest_build: &str, executions: &[Execution]) -> Result Result<() store.outcome_path().display() ) })?; + print_outcome(&outcome); + if ( + outcome.category.as_deref(), + outcome.status_code, + outcome.teardown_complete, + ) != (Some(category), Some(status), Some(true)) + { + return Err(format!( + "the VM's outcome is not {category} {status} with a complete teardown" + )); + } + Ok(()) +} + +/// Prints the VM's `outcome` report: its category and status code, and whether its teardown is +/// complete. +fn print_outcome(outcome: &Report) { let found = outcome.category.as_deref().unwrap_or("none"); let code = outcome .status_code @@ -1804,14 +2611,6 @@ fn check_outcome(store: &SandboxStore, category: &str, status: i64) -> Result<() Some(false) | None => "incomplete", } ); - if (found, outcome.status_code, outcome.teardown_complete) - != (category, Some(status), Some(true)) - { - return Err(format!( - "the VM's outcome is not {category} {status} with a complete teardown" - )); - } - Ok(()) } /// Returns the runtime of NVX's artifacts in the directory that `NVX_ARTIFACTS_DIR` names, with @@ -1849,6 +2648,13 @@ fn attach( ) -> Result { let client = Client::attach(transport, capability, deadline) .map_err(|error| format!("cannot attach to the VM: {}", describe(&error)))?; + print_identity(&client); + Ok(client) +} + +/// Prints the identity of the session that `client` attached: its broker instance, in +/// hexadecimal, and its epoch. +fn print_identity(client: &Client) { let identity = client.identity(); let instance: String = identity .instance_id @@ -1857,7 +2663,6 @@ fn attach( .collect(); println!("instance={instance}"); println!("epoch={}", identity.epoch); - Ok(client) } /// Returns the names of `parts`, separated by commas. diff --git a/tests/runtime-adapter/src/clone.rs b/tests/runtime-adapter/src/clone.rs new file mode 100644 index 000000000..6773f9885 --- /dev/null +++ b/tests/runtime-adapter/src/clone.rs @@ -0,0 +1,372 @@ +//! The host program's clones: VMs that a restore of a generation of a template starts, which +//! NVX's template store leases to a sandbox, as `runtime-adapter-host start` asks. +//! +//! A [`Request`] holds what the host program's options choose of a clone: its image, an EROFS +//! layer of the length of the template's placeholder, its upper, on an ext4 scratch image or in +//! RAM, its memory target, the workload identity as which its executions run, its overlay mode, +//! and its scratch's journal policy. [`Request::plan`] plans the image over the upper as a launch +//! plans a layered root, and [`Request::clone_request`] returns NVX's +//! `nvx::host::template::CloneRequest` of it, with the bindings that the sandbox's store stages +//! for its images, as `staged::Staged` records them: staging hashes, and a start never does. +//! NVX's `ClonePlan` then checks the request against the leased generation before any VMM starts, +//! from the metadata of the clone's files alone. +//! +//! A clone's [record](CloneRecord) names the template store and the generation that the clone +//! leases, beside the plan and the bindings, as the record of a launch names what it launched, so +//! that a later process finds the store whose lease a stop releases. +//! +//! For tests, the host's time at which the store checks the generation's eligibility may come from +//! [`HOST_TIME_VARIABLE`], whose form [`host_time_text`] writes and [`parse_host_time`] reads: +//! a clock fixture, such as the time of another boot of the host, which no test can reach +//! otherwise. + +use std::fmt::Write as _; +use std::num::NonZeroU64; +use std::path::{Path, PathBuf}; +use std::{env, fs}; + +use nvx::activation::{ + Journal, MachineIdMode, OverlayMode as RecordOverlay, StoreGeneration, WorkloadIdentity, + WorkloadView, +}; +use nvx::host::process::BootId; +use nvx::host::sandbox::SandboxPlan; +use nvx::host::sandbox::distro::DISTRO_POLICY_REVISION; +use nvx::host::template::{CloneRequest, HostClock, HostTime}; +use nvx::protocol::root::ImageFormat; +use nvx::uuid::Uuid; +use nvx::{Error, Result}; +use serde_json::{Value, json}; + +use crate::layered::{Bindings, RECORD_FORMAT, RECORD_NAME, plan, unstaged, write_record}; +use crate::root::{JournalPolicy, OverlayMode}; +use crate::staged::invalid; + +/// The mode of a clone's [record](CloneRecord). +pub const CLONE_MODE: &str = "clone"; + +/// The hostname that a clone's activation record names. The guest leaves the hostname to the +/// activation of the workload's view, as it leaves the machine ID, which the record keeps as the +/// image's own. +pub const CLONE_HOSTNAME: &str = "runtime-adapter"; + +/// The variable from which `runtime-adapter-host start` takes the host's time, for tests, in the +/// form that [`host_time_text`] writes, in place of the time that `HostTime::now` samples. +pub const HOST_TIME_VARIABLE: &str = "RUNTIME_ADAPTER_HOST_TIME"; + +/// The host clocks that a time contract names, by their spelling in a manifest. +const CLOCKS: [HostClock; 2] = [HostClock::LinuxBoottime, HostClock::WindowsInterruptTime]; + +/// What the host program asks of a clone. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Request { + /// The image, which the guest expects of the template's placeholder's length. + pub distro: PathBuf, + /// The image's format. + pub format: ImageFormat, + /// The identity that the guest expects of the image. + pub identity: Uuid, + /// The ext4 scratch image of the upper, or `None` for an upper in RAM. + pub scratch: Option, + /// The memory target, in bytes, or `None` for the template's RAM. + pub memory: Option, + /// The workload identity as which every execution runs, if any: its user ID and its group ID. + pub workload_identity: Option<(u32, u32)>, + /// The overlay's mode, or `None` for the guest's default, which trusts the lowers. + pub overlay_mode: Option, + /// Whether the scratch may have a journal, or `None` for the guest's default, which allows + /// one. + pub scratch_journal: Option, +} + +impl Request { + /// Returns the adapter's plan of the clone's image over its upper, as a launch plans a layered + /// root, with the paths absolute and normalized, checked against the host with + /// `SandboxPlan::validated`. + /// + /// Fails as a launch plans its root: with [`ErrorKind::Unsupported`](nvx::ErrorKind::Unsupported) + /// for a path that is not Unicode, and otherwise as `SandboxPlan::validated` does, such as for + /// an image that is not a regular file. + pub fn plan(&self) -> Result { + plan( + &self.distro, + self.format, + self.identity, + self.scratch.as_deref(), + ) + } + + /// Returns NVX's request of the clone of `plan`, the [plan](Self::plan) of this request, with + /// the bindings that the sandbox's store stages for its images, `bindings`, as + /// `staged::Staged::bindings` finds them, and the memory target `memory`, in bytes. + /// + /// The record of the clone's activation names the workload identity, if any, the + /// [`CLONE_HOSTNAME`], the image's own machine ID, the overlay's mode, and the scratch's + /// journal policy, each the guest's default unless the request names it, as a root + /// configuration's are. + /// + /// Fails with [`ErrorKind::Unavailable`](nvx::ErrorKind::Unavailable) if the store stages no + /// binding for the image, or for the scratch image of an upper on one, at its path. + pub fn clone_request( + &self, + plan: SandboxPlan, + bindings: &Bindings, + memory: u64, + ) -> Result { + let binding = bindings + .distro + .clone() + .ok_or_else(|| unstaged("image", plan.distro().path()))?; + let scratch = match plan.upper().scratch_path() { + Some(path) => Some( + bindings + .scratch + .clone() + .ok_or_else(|| unstaged("scratch image", path))?, + ), + None => None, + }; + let overlay = match self.overlay_mode { + None | Some(OverlayMode::TrustedLowers) => RecordOverlay::TrustedLowers, + Some(OverlayMode::UntrustedLowers) => RecordOverlay::UntrustedLowers, + }; + let journal = match self.scratch_journal { + None | Some(JournalPolicy::Allowed) => Journal::Allowed, + Some(JournalPolicy::Forbidden) => Journal::Forbidden, + }; + Ok(CloneRequest { + adapter: plan, + binding, + policy_revision: DISTRO_POLICY_REVISION, + scratch, + memory, + workload: WorkloadView { + hostname: CLONE_HOSTNAME.as_bytes().to_vec(), + machine_id: MachineIdMode::Image, + identity: self + .workload_identity + .map(|(uid, gid)| WorkloadIdentity { uid, gid }), + }, + overlay, + journal, + }) + } +} + +/// What the host program records in a sandbox's store of the clone that it starts there, in +/// [`RECORD_NAME`], before it spawns the VMM, as the record of a launch, +/// `layered::LaunchRecord`, in its format, [`RECORD_FORMAT`]: +/// +/// ```json +/// {"format": 1, "mode": "clone", "kernelCommandLine": "nvx_sandbox=1 ...", +/// "plan": PLAN, "bindings": {"distro": DISTRO_BINDING, "scratch": SCRATCH_BINDING}, +/// "rootConfiguration": null, +/// "template": {"store": "/templates/store", "generation": 1}} +/// ``` +/// +/// The kernel command line is the template's, which the clone restores, the plan and the bindings +/// those of the clone's image and upper, as a layered launch records them, so that +/// `layered::LaunchRecord::read_plan` reads the plan back, and a clone sends no root +/// configuration: its activation record, which NVX keeps in the sandbox, holds its settings. +/// `template` names the template store, by its absolute path, and the generation that the clone +/// leases. +#[derive(Clone, Copy, Debug)] +pub struct CloneRecord<'a> { + /// The template store. + pub template: &'a Path, + /// The generation that the clone leases. + pub generation: StoreGeneration, + /// The template's kernel command line. + pub kernel_command_line: &'a str, + /// The plan of the clone's image over its upper. + pub plan: &'a SandboxPlan, + /// The bindings that the clone's plan checked. + pub bindings: &'a Bindings, +} + +impl CloneRecord<'_> { + /// Returns the record as JSON. + /// + /// Fails with [`ErrorKind::Unsupported`](nvx::ErrorKind::Unsupported) for a template store + /// whose path is not Unicode, and with [`ErrorKind::Other`](nvx::ErrorKind::Other) if the + /// plan or a binding has no JSON form. + pub fn to_json(&self) -> Result { + let plan = serde_json::to_value(self.plan).map_err(|error| { + Error::other("the plan of the clone has no JSON form").with_source(error) + })?; + let store = self.template.to_str().ok_or_else(|| { + Error::unsupported(format!( + "the path of the template store {} is not Unicode text", + self.template.display() + )) + })?; + Ok(json!({ + "format": RECORD_FORMAT, + "mode": CLONE_MODE, + "kernelCommandLine": self.kernel_command_line, + "plan": plan, + "bindings": self.bindings.to_json()?, + "rootConfiguration": null, + "template": {"store": store, "generation": self.generation.get().get()}, + })) + } + + /// Writes the record in `directory`, the sandbox's store, in place of an earlier record. + /// + /// Fails as [`to_json`](Self::to_json) does, and with [`ErrorKind::Io`](nvx::ErrorKind::Io) + /// if the record cannot be written. + pub fn write(&self, directory: &Path) -> Result<()> { + write_record(&directory.join(RECORD_NAME), &self.to_json()?) + } + + /// Reads the template store and the generation that the record in `directory`, the sandbox's + /// store, names, or returns `None` if the store holds no record, or one of a launch that is no + /// clone. + /// + /// Fails with [`ErrorKind::Io`](nvx::ErrorKind::Io) if the record cannot be read, and with + /// [`ErrorKind::Unavailable`](nvx::ErrorKind::Unavailable) if it is not JSON, is not in format + /// [`RECORD_FORMAT`], or is a clone's that does not name its template store by a path and its + /// generation by a positive number. + pub fn read_template(directory: &Path) -> Result> { + let path = directory.join(RECORD_NAME); + let text = match fs::read_to_string(&path) { + Ok(text) => text, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => { + return Err(Error::io(format!("cannot read {}", path.display())).with_source(error)); + } + }; + let unreadable = || { + Error::unavailable(format!( + "cannot read the template of the launch record {}", + path.display() + )) + }; + let value = serde_json::from_str::(&text) + .map_err(|error| unreadable().with_source(error))?; + if value.get("format").and_then(Value::as_u64) != Some(RECORD_FORMAT) { + let problem = format!("is not in format {RECORD_FORMAT}"); + return Err(unreadable().with_source(invalid(&problem))); + } + if value.get("mode").and_then(Value::as_str) != Some(CLONE_MODE) { + return Ok(None); + } + let template = value.get("template"); + let store = template + .and_then(|template| template.get("store")) + .and_then(Value::as_str); + let generation = template + .and_then(|template| template.get("generation")) + .and_then(Value::as_u64) + .and_then(NonZeroU64::new); + match store.zip(generation) { + Some((store, generation)) => Ok(Some(( + PathBuf::from(store), + StoreGeneration::new(generation), + ))), + None => Err(unreadable().with_source(invalid( + "does not name its template store by a path and its generation by a positive \ + number", + ))), + } + } +} + +/// Returns `time` in the form of [`HOST_TIME_VARIABLE`]: +/// `host=HOST,boot=BOOT,clock=CLOCK,monotonic=NANOSECONDS,utc=NANOSECONDS`, with the host's +/// identity and its boot's, each 32 lowercase hexadecimal digits, the clock as a manifest spells +/// it, `linux-boottime` or `windows-interrupt-time`, and the readings of the monotonic clock and +/// of UTC, in nanoseconds, UTC's since the Unix epoch. +pub fn host_time_text(time: &HostTime) -> String { + let mut host = String::with_capacity(32); + for byte in time.host_id { + let _ = write!(host, "{byte:02x}"); + } + format!( + "host={host},boot={},clock={},monotonic={},utc={}", + time.boot_id, + time.clock.as_str(), + time.monotonic_ns, + time.utc_ns + ) +} + +/// Reads a host's time from `text`, in the form that [`host_time_text`] writes, with its fields +/// in that order, or says why it is not one. +pub fn parse_host_time(text: &str) -> std::result::Result { + let invalid = || { + format!( + "{HOST_TIME_VARIABLE} takes host=HOST,boot=BOOT,clock=CLOCK,monotonic=NANOSECONDS,\ + utc=NANOSECONDS, not {text:?}" + ) + }; + let mut fields = text.split(','); + let mut field = |name: &str| { + fields + .next() + .and_then(|field| field.strip_prefix(name)) + .and_then(|field| field.strip_prefix('=')) + .ok_or_else(invalid) + }; + let host = field("host")?; + let boot = field("boot")?; + let clock = field("clock")?; + let monotonic = field("monotonic")?; + let utc = field("utc")?; + if fields.next().is_some() { + return Err(invalid()); + } + let host_id = hexadecimal_id(host).ok_or_else(invalid)?; + // A boot's identity has the same text form, which its parser also checks. + hexadecimal_id(boot).ok_or_else(invalid)?; + let boot_id = boot.parse::().map_err(|_| invalid())?; + let clock = CLOCKS + .into_iter() + .find(|known| known.as_str() == clock) + .ok_or_else(invalid)?; + let number = |text: &str| { + text.bytes() + .all(|byte| byte.is_ascii_digit()) + .then(|| text.parse::().ok()) + .flatten() + .ok_or_else(invalid) + }; + Ok(HostTime { + host_id, + boot_id, + clock, + monotonic_ns: number(monotonic)?, + utc_ns: number(utc)?, + }) +} + +/// Returns the host's time that [`HOST_TIME_VARIABLE`] sets, or `None` without the variable, or +/// says why its value is not a host's time. +pub fn host_time_from_env() -> std::result::Result, String> { + match env::var_os(HOST_TIME_VARIABLE) { + None => Ok(None), + Some(value) => { + let text = value + .into_string() + .map_err(|value| format!("{HOST_TIME_VARIABLE} is not Unicode: {value:?}"))?; + parse_host_time(&text).map(Some) + } + } +} + +/// Returns the 16 bytes that `text`, 32 lowercase hexadecimal digits, spells, or `None` if it is +/// not that. +fn hexadecimal_id(text: &str) -> Option<[u8; 16]> { + if text.len() != 32 + || !text + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return None; + } + let mut bytes = [0; 16]; + for (index, byte) in bytes.iter_mut().enumerate() { + *byte = u8::from_str_radix(&text[2 * index..2 * index + 2], 16).ok()?; + } + Some(bytes) +} diff --git a/tests/runtime-adapter/src/hook.rs b/tests/runtime-adapter/src/hook.rs index 68dadb766..b18bf3084 100644 --- a/tests/runtime-adapter/src/hook.rs +++ b/tests/runtime-adapter/src/hook.rs @@ -1,6 +1,7 @@ -//! The host program's step hook, for tests: at a named [`Step`] of a launch, it pauses the -//! program, so that a test can act between two steps, as when it replaces a staged image once the -//! launch has checked it, or it exits the program without unwinding, as a crash would end it there. +//! The host program's step hook, for tests: at a named [`Step`] of a launch, or of a command that +//! reattaches to a VM, it pauses the program, so that a test can act between two steps, as when it +//! replaces a staged image once the launch has checked it, or ends a clone's VM once a stop found +//! it, or it exits the program without unwinding, as a crash would end it there. //! //! The variable [`STEP_HOOK`] names the step, and what the hook does at it, its [`Action`]: //! @@ -12,8 +13,9 @@ //! the system releases when a process exits, such as its descriptors or handles, and their //! locks. //! -//! Without the variable, the hook does nothing. `runtime-adapter-host launch` reads it, and refuses -//! a value that names no step or action, as it refuses a command line that is not a command. +//! Without the variable, the hook does nothing. `runtime-adapter-host launch`, `reattach`, and +//! `stop` read it, and refuse a value that names no step or action, as they refuse a command line +//! that is not a command. use std::env; use std::ffi::OsStr; @@ -32,8 +34,8 @@ pub const HOOK_EXIT: u8 = 5; /// How often a paused program checks whether its pause file is gone. const PAUSE_POLL: Duration = Duration::from_millis(10); -/// A named step of a launch, at which the step hook acts: right after what the launch has done by -/// then, and before what it does next. +/// A named step of a launch, or of a command that reattaches to a VM, at which the step hook acts: +/// right after what the command has done by then, and before what it does next. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum Step { /// `checked`: the launch has checked the bindings of its images, and the export of its shares, @@ -52,11 +54,26 @@ pub enum Step { /// has answered, and the launch is about to commit, so its journal still records the VMM, /// which no runtime record names yet. Commit, + /// `found`: `reattach` or `stop` has found the sandbox's VM running, and holds it, under the + /// sandbox's lock, with the session in which a clone answered the Activate of its record + /// again, if the VM is a clone's, but has sent the guest agent nothing more, and stopped + /// nothing. + Found, } impl Step { - /// The steps, in the order in which a launch reaches them. - pub const ALL: [Self; 4] = [Self::Checked, Self::Spawned, Self::Ready, Self::Commit]; + /// The steps of a launch, in the order in which it reaches them. + pub const LAUNCH: [Self; 4] = [Self::Checked, Self::Spawned, Self::Ready, Self::Commit]; + + /// The steps: those of a launch, in the order in which it reaches them, then that of a command + /// that reattaches. + pub const ALL: [Self; 5] = [ + Self::Checked, + Self::Spawned, + Self::Ready, + Self::Commit, + Self::Found, + ]; /// Returns the step's name. pub const fn as_str(self) -> &'static str { @@ -65,6 +82,7 @@ impl Step { Self::Spawned => "spawned", Self::Ready => "ready", Self::Commit => "commit", + Self::Found => "found", } } diff --git a/tests/runtime-adapter/src/layered.rs b/tests/runtime-adapter/src/layered.rs index f22888950..e3e330d5f 100644 --- a/tests/runtime-adapter/src/layered.rs +++ b/tests/runtime-adapter/src/layered.rs @@ -436,7 +436,7 @@ pub(crate) fn unicode(path: &Path) -> Result<&str> { /// Returns the error of a launch whose `what`, its image or its scratch image, at `path`, the /// sandbox's store does not stage. -fn unstaged(what: &str, path: &Path) -> Error { +pub(crate) fn unstaged(what: &str, path: &Path) -> Error { Error::unavailable(format!( "the sandbox's store stages no {what} at {}", path.display() @@ -510,14 +510,16 @@ pub(crate) fn write_record(path: &Path, record: &Value) -> Result<()> { /// ``` /// /// The `mode` is `layered`, or `blockless` for a VM without a layered root, whose `plan`, -/// `bindings`, and `rootConfiguration` are `null`. The `plan` is the persisted form of the VM's -/// `SandboxPlan`, which [`read_plan`](Self::read_plan) reads back, whose `network` keeps the VM's -/// network, in the form that [`read_network`] reads, or `null` without one. The `bindings` are -/// those that the launch checked, which the store stages for the plan's images, as -/// [`Staged::bindings`](crate::staged::Staged::bindings) finds them, in the serde forms of NVX's -/// `DistroBinding` and `ScratchBinding`; the scratch image's is `null` for an upper in RAM, which -/// has no scratch image. The `rootConfiguration` is the encoding of the root configuration that the -/// host sends, as it sends it, whether or not the guest accepts it. +/// `bindings`, and `rootConfiguration` are `null`, or `clone` for a clone's record, which +/// [`CloneRecord`](crate::clone::CloneRecord) writes, with the members of a layered root's, but a +/// `rootConfiguration` of `null`, and the template that the clone leases. The `plan` is the +/// persisted form of the VM's `SandboxPlan`, which [`read_plan`](Self::read_plan) reads back, +/// whose `network` keeps the VM's network, in the form that [`read_network`] reads, or `null` +/// without one. The `bindings` are those that the launch checked, which the store stages for the +/// plan's images, as [`Staged::bindings`](crate::staged::Staged::bindings) finds them, in the serde +/// forms of NVX's `DistroBinding` and `ScratchBinding`; the scratch image's is `null` for an upper +/// in RAM, which has no scratch image. The `rootConfiguration` is the encoding of the root +/// configuration that the host sends, as it sends it, whether or not the guest accepts it. #[derive(Clone, Copy, Debug)] pub struct LaunchRecord<'a> { /// The VM's kernel command line. diff --git a/tests/runtime-adapter/src/lib.rs b/tests/runtime-adapter/src/lib.rs index 0f10a0340..46695dd30 100644 --- a/tests/runtime-adapter/src/lib.rs +++ b/tests/runtime-adapter/src/lib.rs @@ -9,8 +9,8 @@ //! messages about a layered [`root`], its [`probe`]s, and [`exec`]utions, and the kernel command //! line and statuses of its [`template`] mode; //! - host-only, with the `host` feature: the host API too, and the host program, in the `host`, -//! `runtime`, `initramfs`, `staged`, `sharing`, `layered`, `capture`, and `hook` modules and the -//! `runtime-adapter-host` binary; +//! `runtime`, `initramfs`, `staged`, `sharing`, `layered`, `capture`, `clone`, and `hook` modules +//! and the `runtime-adapter-host` binary; //! - guest-only, with the `guest` feature: the guest API too, and the guest agent, in the `guest` //! and `worker` modules and the `runtime-adapter-guest` binary, a static //! `x86_64-unknown-linux-musl` executable. @@ -53,10 +53,14 @@ //! activates the clone with the host's `Activate`, whose record the guest library applies, answers //! `Ready` with the activation, and serves the host's order as after a cold boot. On the host, the //! `host` module's clone order is the framing through which the restore sequence of -//! `nvx::host::template` activates a clone and starts its sandbox. +//! `nvx::host::template` activates a clone and starts its sandbox, and the host program's +//! `runtime-adapter-host start` starts a clone of a generation in a sandbox of its own, which +//! leases the generation, in the `clone` module. #[cfg(feature = "host")] pub mod capture; +#[cfg(feature = "host")] +pub mod clone; pub mod exec; pub mod frame; #[cfg(all(feature = "guest", target_os = "linux"))] diff --git a/tests/runtime-adapter/src/probe.rs b/tests/runtime-adapter/src/probe.rs index 37cab9906..316e8e52a 100644 --- a/tests/runtime-adapter/src/probe.rs +++ b/tests/runtime-adapter/src/probe.rs @@ -206,7 +206,7 @@ impl Facts { } /// Returns whether `key` is a valid key of a fact: lowercase ASCII letters, digits, `.`, and `-`. -fn valid_key(key: &str) -> bool { +pub(crate) fn valid_key(key: &str) -> bool { !key.is_empty() && key.bytes().all(|byte| { byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'.' || byte == b'-' diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index 02ca0240a..6bbea49e3 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -96,7 +96,9 @@ //! announces, the start, the executions, the probes, and the stop. The record's workload //! identity, if it has one, is the whole sandbox's, as a root configuration's is, so that //! executions name none, and a clone's executions run in the cgroups of the template's -//! bootstrap. +//! bootstrap. It also answers the probe `activation::ACTIVATION_PROBE`, with when the events of +//! the clone's activation and order happened, after the repair's acknowledgement of the +//! restore. //! //! The application applies none of the record's other settings: its export, its network, its //! hostname, and its machine ID. Without an export, a clone refuses each part of a mapping table @@ -527,7 +529,9 @@ pub fn work( /// before its capture, and whose first epoch it acknowledged, which the reset that the restore /// queued starts again: the worker attaches no session. `template` holds the inputs of the root's /// assembly that the template captured, with which the worker [applies](RootOperations::apply) -/// the record of the host's `Activate`. +/// the record of the host's `Activate`. `acknowledged` is when the repair acknowledged the +/// restore, `nvx::guest::clone::Repaired::acknowledged_at`, if it did, from which the +/// `activation::ACTIVATION_PROBE` reports the events of the clone. /// /// It ends as [`work`] does, and with [`Worked::Unactivated`] once the activation failed, or /// [`Worked::Abandoned`] once the broker closed the stream before the clone was ready, after which @@ -540,8 +544,9 @@ pub fn work_clone( operations: &mut dyn RootOperations, activation: Activation, template: RootTemplate, + acknowledged: Option, ) -> Result { - let state = CloneState::new(activation, template); + let state = CloneState::new(activation, template, acknowledged); serve( Connection::resume(stream, session)?, operations, diff --git a/tests/runtime-adapter/src/worker/activation.rs b/tests/runtime-adapter/src/worker/activation.rs index 8105462dc..4684e1995 100644 --- a/tests/runtime-adapter/src/worker/activation.rs +++ b/tests/runtime-adapter/src/worker/activation.rs @@ -6,6 +6,23 @@ //! //! The activation's failures are final, and end the worker, and with it the VM, with the status of //! `nvx::protocol::status` that [`failure_status`] selects by the failure's class. +//! +//! The worker of a clone also answers the [`ACTIVATION_PROBE`] once the clone is ready, at any +//! time, executions running or not, with when the events of the clone's activation and order +//! happened, in microseconds after the repair's acknowledgement of the restore, before which the +//! host's input must not reach the clone: +//! +//! - `acknowledged`: `yes` once the repair acknowledged the restore, and `none` for a restore +//! whose host did not gate its input until the acknowledgement, whose repair acknowledged +//! nothing; +//! - `reset`: when the clone took the reset that the restore queued; +//! - `message`: when the host's first message came, which must be the clone's `Activate`; +//! - `ready`: when the clone was ready, once the record applied; +//! - `started`: when the clone's sandbox started; +//! - `execution`: when the start of the host's first execution came. +//! +//! Each event is `none` until it happened, `before` if it happened before the acknowledgement, and +//! `unknown` without one. use std::io::{Read, Write}; use std::os::fd::AsFd; @@ -27,6 +44,7 @@ use super::{RootOperations, Worked, serve_root}; use crate::GUEST_BUILD; use crate::frame::{FrameError, Message}; use crate::guest::{ServeError, Step}; +use crate::probe::{Facts, Probe}; use crate::root::Ready; /// How long a clone's activation may take, from the repair's acknowledgement of the restore until @@ -36,6 +54,11 @@ use crate::root::Ready; /// host did not activate by then ends. pub const ACTIVATION_TIMEOUT: Duration = Duration::from_secs(120); +/// The probe that the worker of a clone answers with when the events of the clone's activation and +/// order happened, as the [module documentation](self) describes, and which the worker of a VM +/// that boots cold does not run. +pub const ACTIVATION_PROBE: &str = "activation"; + /// Returns the status with which the worker of a clone, and with it the VM, ends once the clone's /// activation failed with `failure`: the status of `nvx::protocol::status` of the failure's /// class, by its cause: @@ -90,6 +113,8 @@ pub(super) struct CloneState { applied: Option, /// The worker's order, once the clone is ready. order: Option, + /// When the events of the clone's activation and order happened. + timeline: Timeline, } /// The application of the record of the Activate that a clone's activation accepted, which ran, @@ -107,13 +132,22 @@ struct Applied { impl CloneState { /// Returns the state of the clone whose activation is `activation`, in its transport - /// bootstrap, and whose template captured `template`. - pub(super) fn new(activation: Activation, template: RootTemplate) -> Self { + /// bootstrap, whose template captured `template`, and whose repair acknowledged the restore at + /// `acknowledged`, if it did. + pub(super) fn new( + activation: Activation, + template: RootTemplate, + acknowledged: Option, + ) -> Self { Self { activation, template, applied: None, order: None, + timeline: Timeline { + acknowledged, + ..Timeline::default() + }, } } @@ -122,9 +156,9 @@ impl CloneState { /// clone is ready, as when the host attaches again. Returns how the worker ends once the reset /// failed the activation: another reset before the clone is ready. pub(super) fn reset(&mut self) -> Option { - let action = self - .activation - .input(Instant::now(), ActivationInput::Reset); + let now = Instant::now(); + self.timeline.reset.get_or_insert(now); + let action = self.activation.input(now, ActivationInput::Reset); action.err().map(|failure| unactivated(&failure)) } @@ -177,8 +211,8 @@ impl CloneState { /// `Activate`, byte for byte, joins the application, and any other message fails the /// activation. The worker answers an `Activate` that the activation completed, and each retry /// that joined its application, with `Ready`, which reports the activation, as it answers a - /// retry once the clone is ready. Once the activation failed, the worker drops its executions, - /// and ends. + /// retry once the clone is ready, and the [`ACTIVATION_PROBE`] itself. Once the activation + /// failed, the worker drops its executions, and ends. pub(super) fn handle( &mut self, operations: &mut dyn RootOperations, @@ -187,6 +221,7 @@ impl CloneState { now: Instant, message: Message, ) -> Result, ServeError> { + self.timeline.message.get_or_insert(now); let input = match &message { Message::Activate(record) => ActivationInput::Activate(record), _ => ActivationInput::Request, @@ -208,17 +243,23 @@ impl CloneState { } ActivationAction::Ready(activation) => Step::Continue(vec![ready(activation)]), ActivationAction::Serve => { + if let Some(answer) = self.timeline.answer(&message) { + return Ok(Step::Continue(vec![answer])); + } + if matches!(message, Message::ExecStart(_)) { + self.timeline.execution.get_or_insert(now); + } let order = self .order .as_mut() .expect("a ready activation has the worker's order"); let started = order.is_started(); let step = serve_root(operations, order, executions, connection, now, message)?; - if !started - && order.is_started() - && let Err(failure) = self.activation.start() - { - return Ok(self.end(executions, &failure)); + if !started && order.is_started() { + self.timeline.started = Some(now); + if let Err(failure) = self.activation.start() { + return Ok(self.end(executions, &failure)); + } } step } @@ -264,6 +305,7 @@ impl CloneState { }; Some(match self.activation.complete(Instant::now(), result) { Ok(activation) => { + self.timeline.ready = Some(Instant::now()); let identity = identity.map(|identity| (identity.uid(), identity.gid())); // The executions' listeners are published, the entry's default, which refuses // the most, as a root configuration that names no publication has them. @@ -291,6 +333,73 @@ fn ready(activation: ReadyActivation) -> Message { Message::Ready(ready.with_activation(activation).encode()) } +/// When the events of a clone's activation and order happened, which the [`ACTIVATION_PROBE`] +/// reports, as the [module documentation](self) describes. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +struct Timeline { + /// When the repair acknowledged the restore, if it did. + acknowledged: Option, + /// When the clone took the reset that the restore queued. + reset: Option, + /// When the host's first message came. + message: Option, + /// When the clone was ready. + ready: Option, + /// When the clone's sandbox started. + started: Option, + /// When the start of the host's first execution came. + execution: Option, +} + +impl Timeline { + /// Returns the worker's answer to `message` if it is the [`ACTIVATION_PROBE`]: the timeline's + /// [facts](Self::facts), or the problem of a probe with arguments. + fn answer(&self, message: &Message) -> Option { + let Message::Probe(body) = message else { + return None; + }; + let probe = Probe::decode(body).ok()?; + if probe.name() != ACTIVATION_PROBE { + return None; + } + Some(if probe.arguments().is_empty() { + Message::Report(self.facts().encode()) + } else { + let problem = format!("the {ACTIVATION_PROBE} probe takes no arguments"); + Message::ProbeFailed(problem.into_bytes()) + }) + } + + /// Returns the timeline's facts, as the [module documentation](self) describes them. + fn facts(&self) -> Facts { + let mut facts = Facts::new(); + let acknowledged = if self.acknowledged.is_some() { + "yes" + } else { + "none" + }; + facts.push("acknowledged", acknowledged); + let events = [ + ("reset", self.reset), + ("message", self.message), + ("ready", self.ready), + ("started", self.started), + ("execution", self.execution), + ]; + for (name, at) in events { + let value = match (self.acknowledged, at) { + (_, None) => "none".to_owned(), + (None, Some(_)) => "unknown".to_owned(), + (Some(acknowledged), Some(at)) => at + .checked_duration_since(acknowledged) + .map_or("before".to_owned(), |after| after.as_micros().to_string()), + }; + facts.push(name, value); + } + facts + } +} + /// Returns how the worker ends once the activation failed with `failure`: with its /// [status](failure_status), and its description. fn unactivated(failure: &ActivationFailure) -> Worked { @@ -489,4 +598,64 @@ mod tests { assert!(ACTIVATION_TIMEOUT >= 2 * template.device_timeout); assert_eq!(root_template(&template).timeout, template.device_timeout); } + + #[test] + fn the_activation_probe_reports_the_clone_s_events_after_the_acknowledgement() { + let acknowledged = Instant::now(); + let after = |micros| acknowledged + Duration::from_micros(micros); + let timeline = Timeline { + acknowledged: Some(acknowledged), + reset: Some(after(10)), + message: Some(after(250)), + ready: Some(after(4_000)), + started: Some(after(4_100)), + execution: None, + }; + let mut expected = Facts::new(); + let facts = [ + ("acknowledged", "yes"), + ("reset", "10"), + ("message", "250"), + ("ready", "4000"), + ("started", "4100"), + ("execution", "none"), + ]; + for (key, value) in facts { + expected.push(key, value); + } + assert_eq!(timeline.facts(), expected); + let probe = Message::Probe(ACTIVATION_PROBE.as_bytes().to_vec()); + let report = Message::Report(expected.encode()); + assert_eq!(timeline.answer(&probe), Some(report)); + let with_argument = Message::Probe(b"activation now".to_vec()); + let problem = b"the activation probe takes no arguments".to_vec(); + assert_eq!( + timeline.answer(&with_argument), + Some(Message::ProbeFailed(problem)) + ); + // The probes of the root, and other messages, are the order's. + for message in [ + Message::Probe(b"mounts".to_vec()), + Message::Probe(b"activations".to_vec()), + Message::Ping, + ] { + assert_eq!(timeline.answer(&message), None, "{message:?}"); + } + + // An event before the acknowledgement, and a timeline without one. + let early = Timeline { + acknowledged: Some(after(100)), + ..timeline + }; + assert_eq!(early.facts().get("reset"), Some("before")); + assert_eq!(early.facts().get("message"), Some("150")); + let unacknowledged = Timeline { + acknowledged: None, + ..timeline + }; + let facts = unacknowledged.facts(); + assert_eq!(facts.get("acknowledged"), Some("none")); + assert_eq!(facts.get("reset"), Some("unknown")); + assert_eq!(facts.get("execution"), Some("none")); + } } diff --git a/tests/runtime-adapter/src/worker/capture.rs b/tests/runtime-adapter/src/worker/capture.rs index 2cebab22e..025030fd8 100644 --- a/tests/runtime-adapter/src/worker/capture.rs +++ b/tests/runtime-adapter/src/worker/capture.rs @@ -352,9 +352,17 @@ pub fn capture_on_terminal( Capture::Repaired { repaired, fence } => (repaired, fence), }; // The activation's deadline starts at the repair's acknowledgement of the restore. + let acknowledged = repaired.acknowledged_at(); let activation = Activation::new(&repaired, fence, ACTIVATION_TIMEOUT, Instant::now()); let template = root_template(&config); - let worked = work_clone(open_tty(tty)?, session, operations, activation, template)?; + let worked = work_clone( + open_tty(tty)?, + session, + operations, + activation, + template, + acknowledged, + )?; // A clone that ends before it is ready, or whose activation failed, ends even if the tty // sends nothing. let unready = matches!(worked, Worked::Abandoned | Worked::Unactivated { .. }); diff --git a/tests/runtime-adapter/src/worker/probes.rs b/tests/runtime-adapter/src/worker/probes.rs index fba1499ed..427765da6 100644 --- a/tests/runtime-adapter/src/worker/probes.rs +++ b/tests/runtime-adapter/src/worker/probes.rs @@ -9,6 +9,10 @@ //! | `rename DIRECTORY` | It renames DIRECTORY, a directory of the layers, to `.runtime-adapter-renamed` at the top of the root, and back if it can: `outcome` is `allowed`, `refused` once the overlay refused it with `EXDEV`, or `failed: PROBLEM` | //! | `fill` | It writes a file in the root until the upper is full, then removes it, and writes a file there again: the upper's size, `total`, and the bytes available in it before, `available`, in bytes; the bytes that it wrote, `written`; the error that ended the writes, `error`: `ENOSPC` once the upper was full, or the problem; what the second file holds, `after`; and the guest's memory, `memory`, in bytes | //! | `devices` | The guest's disks, the block devices that a device backs: `disks`, each as `NAME:SECTORS:ACCESS`, its size in 512-byte sectors and its access mode, `ro` or `rw`, in the order of their names, separated by spaces | +//! | `reads` | The sectors that the guest read from each of its disks, by the disk's statistics: `disks`, each as `NAME:SECTORS`, in 512-byte sectors, in the order of their names, separated by spaces | +//! | `read-disks` | It reads each of the guest's disks from its start to its end, through the page cache: `disks`, each as `NAME:BYTES`, the bytes that it read, in the order of their names, separated by spaces | +//! | `memory` | The guest's memory: the size of its memory blocks, `block-size`, how many are online, `blocks`, and their bytes, `online`, and the memory that the kernel manages, `total`, each in bytes | +//! | `time` | The guest's time state, [`TIME_STATE`], which NVX's time service keeps: each line `KEY=VALUE` as a fact, with each `_` of its key as `-`, in order, leaving out a key that no fact may have, and a repeated one | //! | `first-partition` | The bytes of the first partition of a GPT disk that the assembly returned, without their trailing padding: how many, `length`, and the bytes, `bytes`, in hexadecimal, or `none` for both for an EROFS image | //! | `cgroups` | The file system at the cgroups' parent, `parent.type`, and its `parent.subtree-control`; whether each cgroup exists, `agent.exists` and `workload.exists`; `agent.memory.low`, `workload.memory.max`, and `workload.pids.max`; the number of processes in each, `agent.processes` and `workload.processes`; and `user-namespaces`, the guest's limit of user namespaces, or `unavailable` without them | //! | `processes` | For PID 1, `pid1`, the worker, `worker`, and PID 1's unrelated child, `sibling`: `NAME.pid`, and while the process exists, `NAME.state`, `NAME.ppid`, `NAME.root`, `NAME.cgroup`, `NAME.command`, and `NAME.namespaces`: `same` if it shares PID 1's, or the ones that differ | @@ -17,11 +21,13 @@ //! FILE and DIRECTORY are relative paths in the root, of normal components, and the other probes take //! no arguments. A value that a probe cannot read is `absent` if its file does not exist, and //! otherwise `unreadable: PROBLEM`. The probes of the mappings beneath the root, and of the host -//! export that they come from, are those of the [`mapping`](super::mapping) module. +//! export that they come from, are those of the [`mapping`](super::mapping) module. The worker of +//! a clone also answers the probe of the clone's activation, `activation::ACTIVATION_PROBE`, as +//! the `activation` module describes. use std::ffi::OsString; use std::fs::{self, File}; -use std::io::{self, Write}; +use std::io::{self, Read, Write}; use std::os::unix::ffi::OsStringExt; use std::os::unix::fs::PermissionsExt; use std::path::{Component, Path, PathBuf}; @@ -29,7 +35,7 @@ use std::path::{Component, Path, PathBuf}; use nix::errno::Errno; use nix::sys::statvfs; -use crate::probe::{Facts, Probe}; +use crate::probe::{Facts, Probe, valid_key}; /// The file that the `contents` probe writes at the top of the root. pub const PROBE_FILE: &str = ".runtime-adapter-probe"; @@ -46,10 +52,18 @@ const FILL_CHUNK: usize = 1 << 20; /// How many bytes the `fill` probe writes beyond those available in the upper before it stops, /// which a file system that keeps its size never takes. const FILL_SLACK: u64 = 16 << 20; -/// The guest's memory information, whose `MemTotal` the `fill` probe reports. +/// The guest's memory information, whose `MemTotal` the `fill` and `memory` probes report. const MEMORY_INFORMATION: &str = "/proc/meminfo"; +/// The guest's memory blocks, of which the `memory` probe counts those online. +const MEMORY_BLOCKS: &str = "/sys/devices/system/memory"; /// The guest's block devices. const BLOCK_DEVICES: &str = "/sys/block"; +/// The directory of the guest's device nodes, through which the `read-disks` probe reads its disks. +const DEVICE_NODES: &str = "/dev"; +/// How many bytes the `read-disks` probe reads at a time. +const READ_CHUNK: usize = 1 << 20; +/// The guest's time state, which the `time` probe reports. +pub const TIME_STATE: &str = "/run/nvx/time/state"; /// The mount table of the worker's mount namespace. pub(super) const MOUNTINFO: &str = "/proc/self/mountinfo"; /// The guest's limit of user namespaces, which only a kernel with user namespaces has. @@ -122,6 +136,10 @@ pub fn run(probe: &Probe, context: &Context<'_>, held: &mut Vec) -> Result "rename", "fill", "devices", + "reads", + "read-disks", + "memory", + "time", "first-partition", "cgroups", "processes", @@ -143,6 +161,10 @@ pub fn run(probe: &Probe, context: &Context<'_>, held: &mut Vec) -> Result ("rename", Some(directory)) => rename(context, directory), ("fill", _) => fill(context), ("devices", _) => devices(), + ("reads", _) => reads(), + ("read-disks", _) => read_disks(), + ("memory", _) => memory(), + ("time", _) => time(), ("first-partition", _) => Ok(first_partition(context)), ("cgroups", _) => cgroups(context), ("processes", _) => processes(context), @@ -331,28 +353,144 @@ fn fill(context: &Context<'_>) -> io::Result { /// Runs the `devices` probe. fn devices() -> io::Result { - let mut disks = Vec::new(); - for entry in fs::read_dir(BLOCK_DEVICES)? { - let directory = entry?.path(); - // A disk that a device backs, rather than a loop or RAM device. - if !directory.join("device").exists() { - continue; - } - let name = directory.file_name().unwrap_or_default().to_string_lossy(); + let mut found = Vec::new(); + for (name, directory) in disks()? { let access = match read(&directory.join("ro")).as_str() { "0" => "rw".to_owned(), "1" => "ro".to_owned(), other => other.to_owned(), }; let sectors = read(&directory.join("size")); - disks.push(format!("{name}:{sectors}:{access}")); + found.push(format!("{name}:{sectors}:{access}")); + } + let mut facts = Facts::new(); + facts.push("disks", found.join(" ")); + Ok(facts) +} + +/// Runs the `reads` probe. +fn reads() -> io::Result { + let mut found = Vec::new(); + for (name, directory) in disks()? { + let stat = read(&directory.join("stat")); + let sectors = sectors_read(&stat).map_or(stat, |sectors| sectors.to_string()); + found.push(format!("{name}:{sectors}")); + } + let mut facts = Facts::new(); + facts.push("disks", found.join(" ")); + Ok(facts) +} + +/// Runs the `read-disks` probe. +fn read_disks() -> io::Result { + let mut found = Vec::new(); + let mut chunk = vec![0; READ_CHUNK]; + for (name, _) in disks()? { + let path = Path::new(DEVICE_NODES).join(&name); + let mut disk = File::open(&path).map_err(|error| { + io::Error::new( + error.kind(), + format!("cannot open {}: {error}", path.display()), + ) + })?; + let mut bytes = 0u64; + loop { + match disk.read(&mut chunk) { + Ok(0) => break, + Ok(count) => bytes += count as u64, + Err(error) if error.kind() == io::ErrorKind::Interrupted => {} + Err(error) => { + let problem = format!("cannot read {}: {error}", path.display()); + return Err(io::Error::new(error.kind(), problem)); + } + } + } + found.push(format!("{name}:{bytes}")); } - disks.sort(); let mut facts = Facts::new(); - facts.push("disks", disks.join(" ")); + facts.push("disks", found.join(" ")); Ok(facts) } +/// Runs the `memory` probe. +fn memory() -> io::Result { + let blocks = Path::new(MEMORY_BLOCKS); + let text = fs::read_to_string(blocks.join("block_size_bytes"))?; + let block_size = u64::from_str_radix(text.trim(), 16).map_err(|_| { + io::Error::new( + io::ErrorKind::InvalidData, + format!("{MEMORY_BLOCKS}/block_size_bytes holds no block size: {text:?}"), + ) + })?; + let mut online = 0u64; + for entry in fs::read_dir(blocks)? { + let entry = entry?; + let name = entry.file_name(); + let block = name + .to_str() + .and_then(|name| name.strip_prefix("memory")) + .is_some_and(|number| !number.is_empty() && number.bytes().all(|b| b.is_ascii_digit())); + if block && read(&entry.path().join("online")) == "1" { + online += 1; + } + } + let total = fs::read_to_string(MEMORY_INFORMATION)?; + let total = memory_total(&total).ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidData, + format!("{MEMORY_INFORMATION} has no MemTotal in kB"), + ) + })?; + let mut facts = Facts::new(); + facts.push("block-size", block_size.to_string()); + facts.push("blocks", online.to_string()); + facts.push("online", online.saturating_mul(block_size).to_string()); + facts.push("total", total.to_string()); + Ok(facts) +} + +/// Runs the `time` probe. +fn time() -> io::Result { + Ok(time_facts(&fs::read_to_string(TIME_STATE)?)) +} + +/// Returns the facts of the guest's time state `state`, as the `time` probe reports them. +fn time_facts(state: &str) -> Facts { + let mut facts = Facts::new(); + for line in state.lines() { + let Some((key, value)) = line.split_once('=') else { + continue; + }; + let key = key.replace('_', "-"); + if valid_key(&key) && facts.get(&key).is_none() { + facts.push(key, value); + } + } + facts +} + +/// Returns the guest's disks, the block devices that a device backs, rather than loop or RAM +/// devices, each with its directory of [`BLOCK_DEVICES`], in the order of their names. +fn disks() -> io::Result> { + let mut disks = Vec::new(); + for entry in fs::read_dir(BLOCK_DEVICES)? { + let directory = entry?.path(); + if !directory.join("device").exists() { + continue; + } + let name = directory.file_name().unwrap_or_default().to_string_lossy(); + disks.push((name.into_owned(), directory)); + } + disks.sort(); + Ok(disks) +} + +/// Returns the sectors read of a block device's statistics, `/sys/block/NAME/stat`, its third +/// field. +fn sectors_read(stat: &str) -> Option { + stat.split_whitespace().nth(2)?.parse().ok() +} + /// Runs the `first-partition` probe. fn first_partition(context: &Context<'_>) -> Facts { let (length, bytes) = match context.first_partition { @@ -737,6 +875,13 @@ rw,lowerdir=/run/a\\040b/layers/distro,metacopy=on,xino=on run(&with_argument, &context, &mut held), Err("the first-partition probe takes no arguments".to_owned()) ); + for name in ["reads", "read-disks", "memory", "time"] { + let with_argument = Probe::parse(&format!("{name} vda")).unwrap(); + assert_eq!( + run(&with_argument, &context, &mut held), + Err(format!("the {name} probe takes no arguments")) + ); + } for (text, problem) in [ ("copy-up", "the copy-up probe takes one argument"), ("rename a b", "the rename probe takes one argument"), @@ -771,6 +916,33 @@ rw,lowerdir=/run/a\\040b/layers/distro,metacopy=on,xino=on assert!(held.is_empty()); } + #[test] + fn the_sectors_read_are_the_third_field_of_a_disk_s_statistics() { + let stat = " 163 0 8530 120 0 0 0 0 \ + 0 168 120 0 0 0 0 0 0"; + assert_eq!(sectors_read(stat), Some(8530)); + assert_eq!(sectors_read("1 2"), None); + assert_eq!(sectors_read("1 2 x"), None); + assert_eq!(sectors_read("1 2 -3"), None); + assert_eq!(sectors_read("absent"), None); + } + + #[test] + fn the_time_state_s_lines_are_facts_with_their_keys_in_the_facts_form() { + let state = "version=1\nrestore_status=ok\nlast_downtime_source=monotonic\n\ + Bad_Key=x\nempty=\nrestore_status=fail\nno value\nlast_step_ns=-2\n"; + let facts = time_facts(state); + let expected = [ + ("version", "1"), + ("restore-status", "ok"), + ("last-downtime-source", "monotonic"), + ("empty", ""), + ("last-step-ns", "-2"), + ]; + assert_eq!(facts.iter().collect::>(), expected); + assert!(time_facts("").is_empty()); + } + #[test] fn the_first_partition_is_reported_in_hexadecimal() { let lowers = [ diff --git a/tests/runtime-adapter/tests/clone_start.rs b/tests/runtime-adapter/tests/clone_start.rs new file mode 100644 index 000000000..b8dd9a78e --- /dev/null +++ b/tests/runtime-adapter/tests/clone_start.rs @@ -0,0 +1,515 @@ +//! The host program's `start` refuses options that do not make a start, and a clock fixture that is +//! not a host's time, as usage errors, and a generation that the template store does not hold, or +//! a template store that does not exist, before it needs NVX's artifacts or creates the sandbox's +//! store. A clone's record names its template store and its generation, which a stop reads back, +//! beside the plan that the other commands read, and a host's time keeps its text form, the form +//! of the clock fixture. The tests need no hypervisor. +#![cfg(feature = "host")] + +use std::fs; +use std::num::NonZeroU64; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use nvx::activation::StoreGeneration; +use nvx::host::artifacts::Artifacts; +use nvx::host::template::{HostClock, HostTime, Store}; +use nvx::protocol::root::ImageFormat; +use nvx::uuid::Uuid; +use runtime_adapter::clone::{ + CLONE_MODE, CloneRecord, HOST_TIME_VARIABLE, Request, host_time_text, parse_host_time, +}; +use runtime_adapter::layered::{Bindings, LaunchRecord, RECORD_FORMAT, RECORD_NAME}; +use runtime_adapter::staged::Staged; +use serde_json::{Value, json}; + +/// The identity that the tests' images have. +const IDENTITY: &str = "0a490905-41be-ed5f-984e-63eb7e5e20b1"; + +/// Returns a directory of the test `name`, empty, below Cargo's temporary directory. +fn directory(name: &str) -> PathBuf { + let directory = Path::new(env!("CARGO_TARGET_TMPDIR")).join(name); + let _ = fs::remove_dir_all(&directory); + fs::create_dir_all(&directory).unwrap(); + directory +} + +/// Returns the identity that the tests' images have. +fn identity() -> Uuid { + Uuid::parse(IDENTITY).unwrap() +} + +/// Writes a file of `length` zeros at `path`, and returns the path. +fn image(path: PathBuf, length: usize) -> PathBuf { + fs::write(&path, vec![0; length]).unwrap(); + path +} + +/// Runs the host program with `arguments`, with the clock fixture `time`, if any, and without +/// NVX's artifacts. +fn host(arguments: &[&str], time: Option<&str>) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")); + command + .args(arguments) + .env_remove(Artifacts::ENV_DIR) + .env_remove(HOST_TIME_VARIABLE); + if let Some(time) = time { + command.env(HOST_TIME_VARIABLE, time); + } + command.output().unwrap() +} + +/// Returns a host's time of the tests, by `clock`. +fn host_time(clock: HostClock) -> HostTime { + HostTime { + host_id: [0x0f, 0xa1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x5c, 0xff], + boot_id: "00112233445566778899aabbccddeeff".parse().unwrap(), + clock, + monotonic_ns: 5_000_000_123, + utc_ns: 1_790_841_600_000_000_000, + } +} + +#[test] +fn the_host_program_refuses_options_that_do_not_make_a_start() { + let start = [ + "start", + "store", + "--template", + "templates", + "--generation", + "1", + "--distro", + "erofs:image", + "--distro-identity", + IDENTITY, + ]; + let with = |options: &[&'static str]| [&start[..], options].concat(); + let cases: Vec<(Vec<&str>, &str)> = vec![ + (vec!["start"], "start needs a sandbox store"), + (vec!["start", "store"], "start needs --template"), + (start[..4].to_vec(), "start needs --generation"), + (start[..6].to_vec(), "start needs --distro"), + (start[..8].to_vec(), "start needs --distro-identity"), + (start.to_vec(), "start needs --upper"), + ( + vec!["start", "store", "--generation", "0"], + "--generation takes a positive number, not \"0\"", + ), + ( + vec!["start", "store", "--generation", "+1"], + "--generation takes a positive number, not \"+1\"", + ), + ( + vec!["start", "store", "--generation", "18446744073709551616"], + "--generation takes a positive number", + ), + ( + with(&["--upper", "ram", "--scratch-journal", "forbidden"]), + "--scratch-journal needs --upper scratch:PATH", + ), + ( + with(&["--upper", "scratch:"]), + "--upper takes scratch:PATH or ram, not \"scratch:\"", + ), + ( + with(&["--upper", "ram", "--memory", "0"]), + "--memory takes a positive number of MiB, not \"0\"", + ), + ( + with(&["--upper", "ram", "--workload-identity", "0:1000"]), + "--workload-identity takes UID:GID, two numbers from 1 to 4294967294", + ), + ( + with(&["--upper", "ram", "--overlay-mode", "lowers"]), + "--overlay-mode takes trusted-lowers or untrusted-lowers", + ), + ( + with(&["--upper", "ram", "--template", "other"]), + "unexpected option \"--template\" of start", + ), + ( + with(&["--upper", "ram", "extra"]), + "unexpected argument \"extra\" of start", + ), + ( + vec!["launch", "store", "--template", "templates"], + "unexpected option \"--template\" of launch", + ), + ( + vec!["probe", "store", "memory", "--template", "templates"], + "unexpected option \"--template\" of probe", + ), + ( + vec!["create", "store", "--generation", "1"], + "unexpected option \"--generation\" of create", + ), + ( + vec!["stage", "store", "--upper", "ram"], + "--upper of stage takes scratch:PATH, not \"ram\"", + ), + ( + vec!["stop", "store", "--memory", "512"], + "unexpected option \"--memory\" of stop", + ), + ]; + // Each option that a start does not take. + let others: [(&str, &str); 8] = [ + ("--guest-agent", "agent"), + ("--hypervisor", "whp"), + ("--runtime-dir", "/run/a"), + ("--first-partition-bound", "4096"), + ("--share", "a:/b"), + ("--network", "network.json"), + ("--cpu-profile", "auto"), + ("--memory-capacity", "512"), + ]; + let others = others.map(|(option, value)| { + let arguments = with(&["--upper", "ram", option, value]); + ( + arguments, + format!("unexpected option \"{option}\" of start"), + ) + }); + let cases = cases + .into_iter() + .map(|(arguments, problem)| (arguments, problem.to_owned())) + .chain(others); + for (arguments, problem) in cases { + let output = host(&arguments, None); + assert_eq!(output.status.code(), Some(2), "{arguments:?}: {output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&problem) && stderr.contains("usage: runtime-adapter-host"), + "{arguments:?}: {stderr}" + ); + } + + // A clock fixture that is not a host's time is a usage error too, of a start that the options + // make. + let arguments = with(&["--upper", "ram"]); + for time in [ + "now", + "host=0,boot=0,clock=linux-boottime,monotonic=0,utc=0", + ] { + let output = host(&arguments, Some(time)); + assert_eq!(output.status.code(), Some(2), "{time:?}: {output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&format!("{HOST_TIME_VARIABLE} takes host=HOST,boot=BOOT")), + "{time:?}: {stderr}" + ); + } +} + +#[test] +fn a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_anything_else() { + // The start reads the template store's index before it needs NVX's artifacts, which the + // commands do not name, and before it creates the sandbox's store. + let directory = directory("start-unindexed"); + let distro = image(directory.join("distro.erofs"), 4096); + let distro = format!("erofs:{}", distro.display()); + let templates = directory.join("templates"); + let store = directory.join("store"); + let start = |templates: &Path, generation: &str| { + let arguments = [ + "start", + store.to_str().unwrap(), + "--template", + templates.to_str().unwrap(), + "--generation", + generation, + "--distro", + &distro, + "--distro-identity", + IDENTITY, + "--upper", + "ram", + ]; + host(&arguments, None) + }; + let failed = |output: &Output, problem: &str| { + assert_eq!(output.status.code(), Some(1), "{output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains(problem), "{stderr}"); + assert!(!stderr.contains("usage:"), "{stderr}"); + assert!(!store.exists(), "the start created the sandbox's store"); + }; + + // No template store. + let output = start(&templates, "1"); + failed(&output, "cannot open the template store"); + + // A template store that holds no generation. + Store::create(&templates).unwrap(); + for generation in ["1", "2"] { + let output = start(&templates, generation); + let problem = format!( + "the template store {} holds no generation {generation}", + Store::open(&templates).unwrap().dir().display() + ); + failed(&output, &problem); + } + + // An image that the plan refuses fails the start before the store is read. + let missing = format!("erofs:{}", directory.join("missing.erofs").display()); + let none = directory.join("none"); + let arguments = [ + "start", + store.to_str().unwrap(), + "--template", + none.to_str().unwrap(), + "--generation", + "1", + "--distro", + &missing, + "--distro-identity", + IDENTITY, + "--upper", + "ram", + ]; + let output = host(&arguments, None); + assert_eq!(output.status.code(), Some(1), "{output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(!stderr.contains("template store"), "{stderr}"); + assert!(!store.exists()); +} + +#[test] +fn a_host_s_time_keeps_its_text_form() { + let linux = host_time(HostClock::LinuxBoottime); + let text = host_time_text(&linux); + assert_eq!( + text, + "host=0fa10000000000000000000000005cff,boot=00112233445566778899aabbccddeeff,\ + clock=linux-boottime,monotonic=5000000123,utc=1790841600000000000" + ); + assert_eq!(parse_host_time(&text), Ok(linux)); + let windows = host_time(HostClock::WindowsInterruptTime); + let text = host_time_text(&windows); + assert!(text.contains(",clock=windows-interrupt-time,"), "{text}"); + assert_eq!(parse_host_time(&text), Ok(windows)); + let extremes = HostTime { + monotonic_ns: 0, + utc_ns: u64::MAX, + ..linux + }; + assert_eq!(parse_host_time(&host_time_text(&extremes)), Ok(extremes)); + + // Anything else is no host's time: each field in its place, its identities in lowercase + // hexadecimal, a clock that a manifest names, and readings of digits alone, in 64 bits. + let text = host_time_text(&linux); + let refused = [ + String::new(), + text.replace("host=0fa1", "host=0FA1"), + text.replace("host=0fa1", "host=0fa"), + text.replace("boot=0011", "boot=001g"), + text.replace("clock=linux-boottime", "clock=boottime"), + text.replace("monotonic=5000000123", "monotonic=+5000000123"), + text.replace("monotonic=5000000123", "monotonic="), + text.replace("utc=1790841600000000000", "utc=18446744073709551616"), + text.replace("utc=", "UTC="), + text.replace(",clock=linux-boottime", ""), + format!("{text},extra=1"), + format!("{text},"), + format!(" {text}"), + { + let (host, rest) = text.split_once(',').unwrap(); + format!("{rest},{host}") + }, + ]; + for text in refused { + let problem = parse_host_time(&text).unwrap_err(); + assert!( + problem.starts_with(&format!("{HOST_TIME_VARIABLE} takes host=HOST")), + "{text:?}: {problem}" + ); + } +} + +#[test] +fn a_clone_s_record_names_its_template_store_and_generation_beside_its_plan() { + let directory = directory("clone-record"); + let distro = image(directory.join("distro.erofs"), 4096); + let scratch = image(directory.join("scratch.ext4"), 8192); + let request = Request { + distro: distro.clone(), + format: ImageFormat::Erofs, + identity: identity(), + scratch: Some(scratch.clone()), + memory: None, + workload_identity: Some((1000, 1000)), + overlay_mode: None, + scratch_journal: None, + }; + let plan = request.plan().unwrap(); + let staged = Staged::stage(&distro, ImageFormat::Erofs, identity(), Some(&scratch)).unwrap(); + let bindings = staged.bindings(&plan); + let templates = directory.join("templates"); + let generation = StoreGeneration::new(NonZeroU64::new(7).unwrap()); + let record = CloneRecord { + template: &templates, + generation, + kernel_command_line: "nvx_sandbox=1 nvx_snapshot_tier=platform", + plan: &plan, + bindings: &bindings, + }; + let sandbox = directory.join("store"); + fs::create_dir(&sandbox).unwrap(); + assert_eq!(CloneRecord::read_template(&sandbox).unwrap(), None); + record.write(&sandbox).unwrap(); + + let written: Value = + serde_json::from_str(&fs::read_to_string(sandbox.join(RECORD_NAME)).unwrap()).unwrap(); + assert_eq!(written["format"], json!(RECORD_FORMAT)); + assert_eq!(written["mode"], json!(CLONE_MODE)); + assert_eq!( + written["kernelCommandLine"], + json!("nvx_sandbox=1 nvx_snapshot_tier=platform") + ); + assert_eq!(written["plan"], serde_json::to_value(&plan).unwrap()); + assert_eq!(written["bindings"], bindings.to_json().unwrap()); + assert_eq!(written["rootConfiguration"], Value::Null); + assert_eq!( + written["template"], + json!({"store": templates.to_str().unwrap(), "generation": 7}) + ); + assert_eq!( + CloneRecord::read_template(&sandbox).unwrap(), + Some((templates.clone(), generation)) + ); + assert_eq!( + LaunchRecord::read_plan(&sandbox).unwrap(), + Some(plan.clone()) + ); + + // A record of a launch that is no clone names no template store; one of a clone must name + // both, by a path and a positive number. + let launched = LaunchRecord { + kernel_command_line: "nvx_sandbox=1", + layered: None, + }; + launched.write(&sandbox).unwrap(); + assert_eq!(CloneRecord::read_template(&sandbox).unwrap(), None); + for template in [ + json!(null), + json!({"store": "/templates"}), + json!({"store": 7, "generation": 1}), + json!({"store": "/templates", "generation": 0}), + json!({"store": "/templates", "generation": "1"}), + ] { + let mut broken = written.clone(); + broken["template"] = template.clone(); + fs::write(sandbox.join(RECORD_NAME), broken.to_string()).unwrap(); + let error = CloneRecord::read_template(&sandbox).unwrap_err(); + assert_eq!(error.kind(), nvx::ErrorKind::Unavailable, "{template}"); + } + let mut other = written.clone(); + other["format"] = json!(RECORD_FORMAT + 1); + fs::write(sandbox.join(RECORD_NAME), other.to_string()).unwrap(); + let error = CloneRecord::read_template(&sandbox).unwrap_err(); + assert_eq!(error.kind(), nvx::ErrorKind::Unavailable); + fs::write(sandbox.join(RECORD_NAME), "{").unwrap(); + let error = CloneRecord::read_template(&sandbox).unwrap_err(); + assert_eq!(error.kind(), nvx::ErrorKind::Unavailable); +} + +#[test] +fn a_clone_s_request_takes_the_staged_bindings_and_the_guest_s_defaults() { + let directory = directory("clone-request"); + let distro = image(directory.join("distro.erofs"), 4096); + let scratch = image(directory.join("scratch.ext4"), 8192); + let request = Request { + distro: distro.clone(), + format: ImageFormat::Erofs, + identity: identity(), + scratch: Some(scratch.clone()), + memory: None, + workload_identity: None, + overlay_mode: None, + scratch_journal: None, + }; + let plan = request.plan().unwrap(); + let staged = Staged::stage(&distro, ImageFormat::Erofs, identity(), Some(&scratch)).unwrap(); + let bindings = staged.bindings(&plan); + let memory = 512 << 20; + let clone = request + .clone_request(plan.clone(), &bindings, memory) + .unwrap(); + assert_eq!(clone.adapter, plan); + assert_eq!(clone.binding, staged.distro.binding); + assert_eq!(clone.scratch, Some(staged.scratch.unwrap().binding)); + assert_eq!(clone.memory, memory); + assert_eq!( + clone.policy_revision, + nvx::host::sandbox::distro::DISTRO_POLICY_REVISION + ); + assert_eq!(clone.workload.hostname, b"runtime-adapter"); + assert_eq!( + clone.workload.machine_id, + nvx::activation::MachineIdMode::Image + ); + assert_eq!(clone.workload.identity, None); + assert_eq!(clone.overlay, nvx::activation::OverlayMode::TrustedLowers); + assert_eq!(clone.journal, nvx::activation::Journal::Allowed); + + // A request that names its settings. + let named = Request { + workload_identity: Some((65534, 65533)), + overlay_mode: Some(runtime_adapter::root::OverlayMode::UntrustedLowers), + scratch_journal: Some(runtime_adapter::root::JournalPolicy::Forbidden), + ..request.clone() + }; + let clone = named + .clone_request(plan.clone(), &bindings, memory) + .unwrap(); + assert_eq!( + clone.workload.identity, + Some(nvx::activation::WorkloadIdentity { + uid: 65534, + gid: 65533 + }) + ); + assert_eq!(clone.overlay, nvx::activation::OverlayMode::UntrustedLowers); + assert_eq!(clone.journal, nvx::activation::Journal::Forbidden); + + // An image, or a scratch image, that the store does not stage. + for (bindings, what) in [ + (Bindings::default(), "image"), + ( + Bindings { + scratch: None, + ..bindings.clone() + }, + "scratch image", + ), + ] { + let error = request + .clone_request(plan.clone(), &bindings, memory) + .unwrap_err(); + assert_eq!(error.kind(), nvx::ErrorKind::Unavailable, "{what}"); + assert!( + error + .to_string() + .starts_with(&format!("the sandbox's store stages no {what} at ")), + "{error}" + ); + } + + // A clone in RAM takes no scratch's binding, but the image's still. + let in_ram = Request { + scratch: None, + ..request + }; + let plan = in_ram.plan().unwrap(); + let error = in_ram + .clone_request(plan.clone(), &Bindings::default(), memory) + .unwrap_err(); + assert_eq!(error.kind(), nvx::ErrorKind::Unavailable); + let staged = Staged::stage(&distro, ImageFormat::Erofs, identity(), None).unwrap(); + let clone = in_ram + .clone_request(plan.clone(), &staged.bindings(&plan), memory) + .unwrap(); + assert_eq!(clone.scratch, None); + assert_eq!(clone.adapter, plan); +} diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 31e414656..b72653a1e 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -3983,7 +3983,7 @@ mod clones { use nvx::protocol::status; use runtime_adapter::frame::MAX_ACTIVATE_BODY; use runtime_adapter::worker::UNACTIVATED_DRAIN; - use runtime_adapter::worker::activation::ACTIVATION_TIMEOUT; + use runtime_adapter::worker::activation::{ACTIVATION_PROBE, ACTIVATION_TIMEOUT}; use runtime_adapter::worker::capture::{self, Admission}; use super::*; @@ -4054,12 +4054,11 @@ mod clones { /// Returns the activation of a clone that the worker repaired, with [`RESOURCES`], behind the /// worker's fence, which the capture's barrier handed over closed, and whose deadline ends - /// `timeout` from now, as from the repair's acknowledgement of the restore. - fn repaired(timeout: Duration) -> Activation { + /// `timeout` after `acknowledged`, as from the repair's acknowledgement of the restore. + fn repaired(acknowledged: Instant, timeout: Duration) -> Activation { let mut fence = Admission::default(); fence.close().unwrap(); - let now = Instant::now(); - Activation::with_resources(RESOURCES, Some(now), fence, timeout, now) + Activation::with_resources(RESOURCES, Some(acknowledged), fence, timeout, acknowledged) } /// Returns the inputs of the assembly of the clones' roots that their template captured: @@ -4108,13 +4107,22 @@ mod clones { } /// Runs a clone's worker in a thread, with the root operations `root`, on `guest`, the guest's - /// end of the stream, and returns the thread, as [`start_clone`] does. + /// end of the stream, and returns the thread, as [`start_clone`] does. The clone's repair + /// acknowledged the restore now. fn run_clone(guest: UnixStream, root: FakeRoot, timeout: Duration) -> Working { let session = captured_session(); - let activation = repaired(timeout); + let acknowledged = Instant::now(); + let activation = repaired(acknowledged, timeout); thread::spawn(move || { let mut root = root; - let worked = worker::work_clone(guest, session, &mut root, activation, template()); + let worked = worker::work_clone( + guest, + session, + &mut root, + activation, + template(), + Some(acknowledged), + ); (root, worked) }) } @@ -4689,4 +4697,55 @@ mod clones { assert_eq!(worked.unwrap(), Worked::Stopped); assert_eq!(entrances(&root), [entrance(identity)]); } + + /// Returns the facts of the guest's `report`. + fn reported(report: &Message) -> Facts { + let Message::Report(body) = report else { + panic!("the guest did not report: {report:?}"); + }; + Facts::decode(body).unwrap() + } + + /// Returns the microseconds after the acknowledgement of the restore that the clone's `facts` + /// report for its event `name`. + fn after(facts: &Facts, name: &str) -> u64 { + let value = facts.get(name); + value + .and_then(|value| value.parse().ok()) + .unwrap_or_else(|| panic!("the clone's {name} is {value:?}: {facts:?}")) + } + + #[test] + fn a_clone_reports_when_its_events_happened_after_the_acknowledgement() { + // The worker answers the probe of the clone's activation itself, with no root operation, + // executions running or not: the restore's reset came after the repair's acknowledgement, + // then the host's first message, the clone's Activate, then its Ready and its start, and + // no execution yet. + let record = record(1, Some(WORKLOAD)); + let mut host = Host::activated(FakeRoot::booting(&layout()), &record); + let probe = Message::Probe(ACTIVATION_PROBE.as_bytes().to_vec()); + host.send(std::slice::from_ref(&probe)); + host.run_until(|host| !host.replies.is_empty()); + let facts = reported(&host.replies.remove(0)); + assert_eq!(facts.get("acknowledged"), Some("yes")); + assert_eq!(facts.get("execution"), Some("none")); + let events = ["reset", "message", "ready", "started"].map(|name| after(&facts, name)); + assert!(events.is_sorted(), "{facts:?}"); + + // The start of the first execution came once the sandbox started. + host.start(start_of(1, shell("sleep 1")), Vec::new()); + host.send(&[probe]); + host.run_until(|host| !host.replies.is_empty()); + let facts = reported(&host.replies.remove(0)); + assert!(after(&facts, "execution") >= events[3], "{facts:?}"); + host.run_until(|host| host.outcome(1).is_some()); + check_lossless(&host, 1, Status::Exited(0)); + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + assert!( + !root.calls.iter().any(|call| matches!(call, Call::Probe(_))), + "{:?}", + root.calls + ); + } } diff --git a/tests/runtime-adapter/tests/step_hook.rs b/tests/runtime-adapter/tests/step_hook.rs index 8f92e9ffc..cdf3622e8 100644 --- a/tests/runtime-adapter/tests/step_hook.rs +++ b/tests/runtime-adapter/tests/step_hook.rs @@ -1,7 +1,7 @@ //! The host program's step hook, for tests: the values of `RUNTIME_ADAPTER_STEP_HOOK` that set it, //! and those that the host program refuses, and a pause, which waits until its file is gone, on //! Windows until a pending deletion of the file completes, and says how long it paused. The tests -//! need no hypervisor; the VM tests pause launches with the hook, and end them. +//! need no hypervisor; the VM tests pause launches and stops with the hook, and end them. #![cfg(feature = "host")] use std::ffi::OsStr; @@ -18,6 +18,11 @@ use runtime_adapter::hook::{Action, HOOK_EXIT, STEP_HOOK, Step, StepHook}; fn a_step_hook_names_a_step_and_what_it_does_there() { assert_eq!( Step::ALL.map(Step::as_str), + ["checked", "spawned", "ready", "commit", "found"] + ); + // A launch reaches each step but that of a command that reattaches, in order. + assert_eq!( + Step::LAUNCH.map(Step::as_str), ["checked", "spawned", "ready", "commit"] ); // Each step, with each action, reads back as it is written. @@ -49,7 +54,7 @@ fn a_step_hook_names_a_step_and_what_it_does_there() { for (value, problem) in [ ( "", - "names the step \"\", not one of checked, spawned, ready, commit", + "names the step \"\", not one of checked, spawned, ready, commit, found", ), ("checkd:exit", "names the step \"checkd\""), ("Checked:exit", "names the step \"Checked\""), @@ -170,21 +175,32 @@ fn a_pause_waits_until_the_deletion_of_its_file_completes() { } #[test] -fn the_host_program_refuses_a_launch_whose_step_hook_it_cannot_read() { - for (value, problem) in [ - ("nowhere:exit", "names the step \"nowhere\""), - ("ready:pause", "takes STEP:pause:PATH or STEP:exit"), - ] { - let output = Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")) - .args(["launch", "store", "--guest-agent", "agent"]) - .env(STEP_HOOK, value) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(2), "{value:?}: {output:?}"); - let stderr = String::from_utf8_lossy(&output.stderr); - assert!( - stderr.contains(problem) && stderr.contains("usage: runtime-adapter-host"), - "{value:?}: {stderr}" - ); +fn the_host_program_refuses_a_command_whose_step_hook_it_cannot_read() { + let commands: [&[&str]; 3] = [ + &["launch", "store", "--guest-agent", "agent"], + &["reattach", "store"], + &["stop", "store"], + ]; + for command in commands { + for (value, problem) in [ + ("nowhere:exit", "names the step \"nowhere\""), + ("ready:pause", "takes STEP:pause:PATH or STEP:exit"), + ] { + let output = Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")) + .args(command) + .env(STEP_HOOK, value) + .output() + .unwrap(); + assert_eq!( + output.status.code(), + Some(2), + "{command:?} {value:?}: {output:?}" + ); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(problem) && stderr.contains("usage: runtime-adapter-host"), + "{command:?} {value:?}: {stderr}" + ); + } } } diff --git a/tests/runtime-adapter/tests/vm_clone.rs b/tests/runtime-adapter/tests/vm_clone.rs new file mode 100644 index 000000000..f6cbfe897 --- /dev/null +++ b/tests/runtime-adapter/tests/vm_clone.rs @@ -0,0 +1,1662 @@ +//! The host program starts clones of templates on a real hypervisor, through public APIs only: VMs +//! that a restore of a generation of a template store starts, in place, which the store leases to +//! a sandbox, and which the guest agent's clone mode activates, as `runtime-adapter-host start` +//! asks. +//! +//! Each scenario captures a [`Template`] of its own, with 256 MiB of RAM, of a capacity of +//! 1 GiB, in a template store of its own, as `vm_template` does, of the platform's copy of the +//! fixtures' Ubuntu EROFS image, and starts its clones, each in a sandbox store of its own, which +//! stages the clone's images, as a launch's does: that copy, and for a clone over a scratch image, +//! a copy of the scratch template, which, as NVX requires of a clone's images, no one else may +//! change either. +//! +//! - One generation serves clones over a scratch image and in RAM, at once and in turn, at each +//! memory target from the template's RAM to its capacity, which the guest has online. The store +//! leases the generation to each, a stop releases it, and the clones change nothing of the +//! generation's snapshot, and leave nothing in its payload. +//! - The store and NVX's plan refuse a clone that the template cannot serve before any VMM starts, +//! each a failure of the request, as NVX attributes it by its evidence, with no VM: an image or a +//! scratch image of another length than the template's, an image of a geometry that OpenVMM +//! refuses, a memory target beyond its capacity or of a part of a block. The start then releases +//! the lease, as it does once the adapter refuses images that the sandbox does not stage. A +//! generation whose key is gone fails the plan too, as the template's failure, and the start +//! quarantines it through its tombstone. +//! - A clone whose activation fails ends before it is ready, with the status of the failure's +//! class, which attributes the failure, and the VMM's exit: a scratch image that holds no ext4 +//! file system, a journal that the clone's record forbids, a workload identity whose account the +//! image lacks, and a host that sends anything but the clone's Activate first, a ping, a start, or +//! an execution. The sandbox records the failure until the release of its lease. +//! - A clone runs its executions as its record's workload identity, once it is ready: the repair +//! acknowledged the restore before the clone took its reset, the host's first message came +//! after that, and the clone's Ready, its start, and its first execution came in that order. +//! - Each command after the start reattaches to the clone from a new process: the clone answers +//! the Activate of its record again, in a new session of the same broker, with its Ready, without +//! activating again, and serves the command. +//! - A host program of another runtime ends the clone through its verified identity, without a +//! word to its guest, and releases its lease: the template still serves its other clones. +//! - A clone whose VM ends without a word of its guest once a stop found it running fails, as NVX +//! attributes the loss of its status, before the stop, which stops nothing, releases its lease: +//! the template still serves clones. +//! - A tombstone quarantines the generation of a running clone: the generation takes no new lease, +//! while the clone keeps running, reattaches, stops, and releases its lease, and the generation's +//! snapshot stays as the capture left it. +//! - A start that ended once the store leased the generation, before it recorded the clone, +//! leaves a lease whose template store no record names, or only the record of an earlier clone, +//! of another template: a stop keeps it, and only its own template store, which `--template` +//! names, releases it, after which the sandbox leases a generation again. A sandbox that holds a +//! lease must not move: a command on one that moved refuses it, rather than take for it the +//! sandbox at the lease's path, which holds another lease, or none. +//! - The store checks the generation's eligibility at the host's time: on the boot that captured +//! the template, by the host's monotonic clock, whatever UTC says, and after a reboot by UTC, +//! which clock fixtures stand for. It refuses a generation that is no longer, or not yet, +//! eligible, without a lease. +//! - A clone reads no more of its disks on its restore than its root needs, which a read of each +//! disk to its end shows the guest's statistics count. +//! - A clone's time settles after it is ready, with no violation, after the downtime that the +//! host's monotonic clock measured. +//! +//! The tests boot VMs, so they are ignored. They read what the `common` module lists, and take +//! the template's image and the scratch template from its fixtures. +//! +//! ```text +//! cargo test --locked --features host --test vm_clone -- --ignored +//! ``` +#![cfg(feature = "host")] + +mod common; + +use std::collections::BTreeMap; +use std::env; +use std::ffi::{OsStr, OsString}; +use std::fs::{self, File}; +use std::io; +use std::mem; +use std::num::NonZeroU64; +use std::path::{Path, PathBuf}; +use std::thread; +use std::time::{Duration, Instant, SystemTime}; + +use nvx::activation::{ReadyActivation, StoreGeneration}; +use nvx::exec::{Key, Limits, Timers, Window}; +use nvx::host::artifacts::Artifacts; +use nvx::host::platform::Transport; +use nvx::host::process::{BootId, Process}; +use nvx::host::store::{CallerFiles, LOG_NAME, SandboxStore}; +use nvx::host::template::{ + CAPTURE_RECORD_NAME, CloneControl, ClonePlan, CloneRecords, Evidence, GENERATIONS_NAME, + Generation, HostTime, KEY_NAME, Lease, MAX_DOWNTIME_NS, MEMORY_BLOCK_SIZE, Outcome, Progress, + RestoreDeadlines, SNAPSHOT_NAME, Stage, Store, Teardown, +}; +use nvx::protocol::CAPABILITY_LEN; +use nvx::protocol::root::ImageFormat; +use nvx::protocol::status; +use nvx::uuid::Uuid; +use runtime_adapter::GUEST_BUILD; +use runtime_adapter::capture::DEFAULT_SCRATCH_LENGTH; +use runtime_adapter::clone::{HOST_TIME_VARIABLE, Request, host_time_text}; +use runtime_adapter::exec::{HostMessage, Start}; +use runtime_adapter::frame::Message; +use runtime_adapter::hook::{Action, STEP_HOOK, Step, StepHook}; +use runtime_adapter::host::Client; +use runtime_adapter::runtime::Runtime; +use runtime_adapter::staged::Staged; + +use common::{Fixtures, Printed, Sandbox}; + +/// The exit status of the host program's command that failed, after a diagnostic. +const FAILED: i32 = 1; +/// The exit status of a command whose clone failed, once nothing of the clone runs, and the +/// command released its lease. +const CLONE_FAILED: i32 = 8; +/// The guest agent's build of a host program of another runtime. +const ANOTHER_BUILD: &str = "another-build"; +/// Bytes in a mebibyte. +const MIB: u64 = 1024 * 1024; +/// The RAM of the scenarios' templates, in MiB. +const TEMPLATE_MEMORY: u64 = 256; +/// The RAM capacity of the scenarios' templates, in MiB. +const TEMPLATE_CAPACITY: u64 = 1024; +/// The generation of each scenario's template store that its template is. +const GENERATION: StoreGeneration = StoreGeneration::new(NonZeroU64::MIN); +/// The workload identity of the clones that run as one: nobody's, in the Ubuntu image. +const NOBODY: &str = "65534:65534"; +/// A workload identity whose account the Ubuntu image lacks. +const STRANGER: &str = "4242:4242"; +/// Bytes in a sector of the guest's disks, which their statistics count. +const SECTOR: u64 = 512; +/// How long a clone's time may take to settle once the clone is ready. +const TIME_SETTLEMENT: Duration = Duration::from_secs(30); +/// Nanoseconds in an hour. +const HOUR_NS: u64 = 3_600_000_000_000; +/// The error number with which the guest refuses an execution that names an identity in a +/// sandbox that has its own: `EINVAL`. +const EINVAL: i32 = 22; + +/// The template of a scenario: the generation that a template store of its own captured, in the +/// scenario's sandbox, of the platform's copy of the fixtures' Ubuntu EROFS image, with +/// [`TEMPLATE_MEMORY`] MiB of RAM, of a capacity of [`TEMPLATE_CAPACITY`] MiB. +struct Template { + /// The scenario's sandbox, whose store is the template store, until the template drops. + sandbox: Option, + /// The platform's copy of the image, the template's placeholder, which the clones stage as + /// their image too. + image: PathBuf, + /// The image's identity. + identity: String, + /// The fixtures. + fixtures: Fixtures, +} + +impl Template { + /// Creates the sandbox of the scenario `name`, stages the platform's copy of the image, and + /// captures the template in the sandbox's store. + /// + /// On Linux, a clone's control socket, `generations/1/LAUNCH.sock` in the template store, + /// after the 32 hexadecimal digits of the launch, must fit the path of a Unix socket, so the + /// scenarios' names are short. + fn create(name: &str) -> Self { + let mut sandbox = Sandbox::new(name); + let fixtures = Fixtures::find(); + let image = stage_platform(&sandbox.root.join("platform"), &fixtures); + let distro = erofs(&image); + let (memory, capacity) = (TEMPLATE_MEMORY.to_string(), TEMPLATE_CAPACITY.to_string()); + let options = [ + OsStr::new("--distro"), + &distro, + OsStr::new("--distro-identity"), + OsStr::new(&fixtures.identity), + OsStr::new("--memory"), + OsStr::new(&memory), + OsStr::new("--memory-capacity"), + OsStr::new(&capacity), + ]; + let created = sandbox.create(&options, 0); + assert_eq!(created["created"], "captured", "{created:?}"); + assert_eq!(created["generation"], GENERATION.to_string(), "{created:?}"); + Self { + identity: fixtures.identity.clone(), + sandbox: Some(sandbox), + image, + fixtures, + } + } + + /// Returns the template store's directory. + fn dir(&self) -> &Path { + let sandbox = self.sandbox.as_ref(); + &sandbox + .expect("the sandbox lasts until the template drops") + .store + } + + /// Returns the template store. + fn store(&self) -> Store { + Store::open(self.dir()).unwrap() + } + + /// Returns the template's generation, as the store's index lists it. + fn generation(&self) -> Generation { + let index = self.store().index().unwrap(); + index.get(GENERATION).cloned().expect("the store holds it") + } + + /// Returns the leases of the template's generation. + fn leases(&self) -> Vec { + self.store().leases(GENERATION).unwrap() + } + + /// Returns the payload of the template's generation. + fn payload(&self) -> PathBuf { + self.dir() + .join(GENERATIONS_NAME) + .join(GENERATION.to_string()) + } + + /// Returns a copy of the platform's image of `length` bytes, beside it, without a manifest, + /// at `name`. + fn variant(&self, name: &str, length: u64) -> PathBuf { + let copy = self.image.with_file_name(name); + fs::copy(&self.image, ©).unwrap(); + File::options() + .write(true) + .open(©) + .unwrap() + .set_len(length) + .unwrap(); + copy + } + + /// Stages the template's image in the store of `sandbox`, which it creates, for a clone in + /// RAM. + fn stage(&self, sandbox: &mut Sandbox) { + let distro = erofs(&self.image); + let options = [ + OsStr::new("--distro"), + &distro, + OsStr::new("--distro-identity"), + OsStr::new(&self.identity), + ]; + sandbox.run("stage", &options, 0); + } + + /// Starts a clone in a new sandbox of the scenario `name`, as `start` asks, checks that the + /// start exits with `status`, and returns the clone. + fn start(&self, name: &str, start: StartOf<'_>, status: i32) -> Cloned { + let mut sandbox = Sandbox::new(name); + let distro = erofs(start.image.unwrap_or(&self.image)); + // NVX restores a clone only over images that no one else may change, so the scratch image + // lies in a directory of the sandbox's own, as the platform's image lies in the platform's. + let images = sandbox.root.join("images"); + SandboxStore::create(&images).unwrap(); + let scratch = images.join("scratch.ext4"); + match start.upper { + Upper::Ram => {} + Upper::Scratch => { + private_copy(&self.fixtures.scratch, &scratch); + } + Upper::Resized(length) => { + let file = private_copy(&self.fixtures.scratch, &scratch); + file.set_len(length).unwrap(); + } + Upper::Zeros => private_file(&scratch) + .set_len(DEFAULT_SCRATCH_LENGTH) + .unwrap(), + } + let upper = match start.upper { + Upper::Ram => OsString::from("ram"), + _ => prefixed("scratch:", &scratch), + }; + let identity = OsStr::new(&self.identity); + if !start.unstaged { + let mut staging = vec![ + OsStr::new("--distro"), + &distro, + OsStr::new("--distro-identity"), + identity, + ]; + if start.upper != Upper::Ram { + staging.extend([OsStr::new("--upper"), &upper]); + } + sandbox.run("stage", &staging, 0); + } + let generation = GENERATION.to_string(); + let mut options = vec![ + OsStr::new("--template"), + self.dir().as_os_str(), + OsStr::new("--generation"), + OsStr::new(&generation), + OsStr::new("--distro"), + &distro, + OsStr::new("--distro-identity"), + identity, + OsStr::new("--upper"), + &upper, + ]; + options.extend(start.options.iter().map(OsStr::new)); + let time = start.time.map(|time| OsString::from(host_time_text(&time))); + let environment: Vec<(&str, &OsStr)> = time + .iter() + .map(|time| (HOST_TIME_VARIABLE, time.as_os_str())) + .collect(); + let started = sandbox.start("start", &options, &environment); + let started = sandbox.finish(started, status); + Cloned { sandbox, started } + } +} + +impl Drop for Template { + /// Removes the template's sandbox, as a sandbox's drop does, unless a clone whose VM might + /// still run leases the generation: one whose sandbox a test that failed kept, since it could + /// not confirm that the clone's VM ended. Its VMM restores the generation's snapshot in place, + /// and reads the platform's image, which the template's sandbox holds too, so the template + /// stays with it. A clone's sandbox drops before the template, which the scenario creates + /// first. + fn drop(&mut self) { + let Some(sandbox) = self.sandbox.take() else { + return; + }; + let kept = Store::open(&sandbox.store) + .and_then(|store| store.leases(GENERATION)) + .map(|leases| { + leases + .iter() + .map(Lease::sandbox) + .filter(|clone| clone.exists()) + .map(|clone| clone.display().to_string()) + .collect::>() + }); + let reason = match kept { + Ok(kept) if kept.is_empty() => { + drop(sandbox); + return; + } + Ok(kept) => format!("the clones in {} might still run", kept.join(", ")), + Err(error) => format!("its leases cannot be read: {}", common::describe(&error)), + }; + eprintln!("kept {} because {reason}", sandbox.root.display()); + mem::forget(sandbox); + } +} + +/// The upper of a clone. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +enum Upper { + /// RAM. + #[default] + Ram, + /// A copy of the scratch template, an empty ext4 file system of the template's scratch + /// length. + Scratch, + /// A copy of the scratch template of this many bytes. + Resized(u64), + /// Zeros of the template's scratch length, which hold no file system. + Zeros, +} + +/// What a scenario asks of the start of a clone. +#[derive(Default)] +struct StartOf<'a> { + /// The image, or the template's image by default. + image: Option<&'a Path>, + /// The upper. + upper: Upper, + /// The start's other options. + options: &'a [&'a str], + /// The host's time at which the store checks the generation's eligibility, as the clock + /// fixture, or the time that the host program samples by default. + time: Option, + /// Whether the sandbox's store stages no image. + unstaged: bool, +} + +/// A clone of a scenario's template, in a sandbox of its own, as its start left it. +struct Cloned { + /// The clone's sandbox. + sandbox: Sandbox, + /// What the start printed. + started: Printed, +} + +impl Cloned { + /// Checks that the start started the clone, as the [module documentation](self) describes, + /// leasing the generation, and recorded its VM, which runs. + fn assert_started(&self) { + let started = &self.started; + assert_eq!(started["generation"], GENERATION.to_string(), "{started:?}"); + let lease = &started["lease"]; + assert!( + lease.len() == 32 && lease.bytes().all(|byte| byte.is_ascii_hexdigit()), + "{started:?}" + ); + started["downtime"].parse::().unwrap(); + let facts = ["root", "build", "table", "mapped", "sandbox"].map(|key| &started[key]); + assert_eq!( + facts, + ["ready", GUEST_BUILD, "applied", "0", "started"], + "{started:?}" + ); + let record = self.sandbox.record().expect("the start recorded its VM"); + assert_eq!(started["pid"], record.process().pid.to_string()); + assert!(!started.contains_key("released"), "{started:?}"); + } + + /// Runs `probes` in the clone's root, and returns what they found, each fact as + /// `PROBE.KEY`. + fn probe(&mut self, probes: &[&str]) -> Printed { + let probes = probes.iter().map(OsStr::new).collect::>(); + let printed = self.sandbox.run("probe", &probes, 0); + assert_eq!(printed["reattached"], "live"); + assert_eq!(printed["pid"], self.started["pid"]); + printed + } + + /// Runs one execution of `arguments`, its options and its program, in the clone's root, and + /// returns what it did, each fact as `exec.1.KEY`. + fn exec(&mut self, arguments: &[&str]) -> Printed { + let arguments = arguments.iter().map(OsStr::new).collect::>(); + let printed = self.sandbox.run("exec", &arguments, 0); + assert_eq!(printed["reattached"], "live"); + printed + } + + /// Stops the clone, and checks that it stopped gracefully, and that the stop released the + /// clone's lease. + fn stop(&mut self) { + let stopped = self.sandbox.run("stop", &[], 0); + let facts = ["stopped", "outcome", "teardown"].map(|key| &stopped[key]); + assert_eq!(facts, ["graceful", "success 0", "complete"], "{stopped:?}"); + assert_eq!(stopped["released"], self.started["lease"], "{stopped:?}"); + assert!(self.sandbox.record().is_none()); + let store = SandboxStore::open(&self.sandbox.store).unwrap(); + assert!(Lease::of(&store).unwrap().is_none()); + } + + /// Checks that the start refused the clone before any VMM started, and released the lease + /// that it took, if it took one, with a diagnostic that says `problem`. + fn assert_refused(&self, problem: &str) { + let started = &self.started; + assert!(!started.contains_key("pid"), "{started:?}"); + assert_eq!(started.get("released"), started.get("lease"), "{started:?}"); + let stderr = self.sandbox.stderr(); + assert!(stderr.contains(problem), "{problem:?} in {stderr}"); + self.sandbox.assert_unlaunched(); + let store = SandboxStore::open(&self.sandbox.store).unwrap(); + assert!(Lease::of(&store).unwrap().is_none()); + } +} + +/// Stages the platform's copy of the fixtures' Ubuntu EROFS image in the directory `platform`, +/// beside a copy of its manifest, and returns it. +/// +/// A platform stages its trusted artifacts where no one else may change them, and NVX refuses an +/// image that anyone else may change. So the directory is private to the current user, as NVX +/// makes a sandbox's store, whose access control list on Windows passes on to the files created in +/// it. +fn stage_platform(platform: &Path, fixtures: &Fixtures) -> PathBuf { + SandboxStore::create(platform).unwrap(); + let [image, _] = [&fixtures.distro, &fixtures.manifest].map(|source| { + let copy = platform.join(source.file_name().unwrap()); + private_copy(source, ©); + copy + }); + image +} + +/// Creates the file `path`, which no one else may change, in a directory private to the current +/// user, and returns it, open for writing: of the mode 0644 on Unix, whatever the umask, and on +/// Windows of the directory's access control list. +fn private_file(path: &Path) -> File { + let file = File::create(path).unwrap(); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + file.set_permissions(fs::Permissions::from_mode(0o644)) + .unwrap(); + } + file +} + +/// Copies the file `source` to `copy`, which [`private_file`] creates, and returns the copy, +/// open for writing. +fn private_copy(source: &Path, copy: &Path) -> File { + let mut file = private_file(copy); + io::copy(&mut File::open(source).unwrap(), &mut file).unwrap(); + file +} + +/// Returns `path` after `prefix`, as an option's value names a file. +fn prefixed(prefix: &str, path: &Path) -> OsString { + let mut value = OsString::from(prefix); + value.push(path); + value +} + +/// Returns the value of `--distro` for the EROFS image at `path`. +fn erofs(path: &Path) -> OsString { + prefixed("erofs:", path) +} + +/// Returns the names of the entries of the directory `path`, sorted. +fn names(path: &Path) -> Vec { + let mut names: Vec = fs::read_dir(path) + .unwrap() + .map(|entry| entry.unwrap().file_name().into_string().unwrap()) + .collect(); + names.sort(); + names +} + +/// Returns each file below the directory `snapshot`, by its path in it, with its length and its +/// modification time. +fn snapshot_files(snapshot: &Path) -> Vec<(PathBuf, u64, SystemTime)> { + let mut files = Vec::new(); + let mut directories = vec![snapshot.to_owned()]; + while let Some(directory) = directories.pop() { + for entry in fs::read_dir(&directory).unwrap() { + let entry = entry.unwrap(); + let metadata = entry.metadata().unwrap(); + if metadata.is_dir() { + directories.push(entry.path()); + continue; + } + let path = entry.path().strip_prefix(snapshot).unwrap().to_owned(); + files.push((path, metadata.len(), metadata.modified().unwrap())); + } + } + files.sort(); + files +} + +/// Checks that the payload of the template's generation holds what a capture published alone, +/// and, while clones run, their control sockets on Linux: no copy of a scratch image, say. +fn assert_payload(template: &Template, running: bool) { + let published = [CAPTURE_RECORD_NAME, KEY_NAME, SNAPSHOT_NAME]; + let names = names(&template.payload()); + let (template_files, others): (Vec<_>, Vec<_>) = names + .iter() + .partition(|name| published.contains(&name.as_str())); + assert_eq!(template_files, published, "{names:?}"); + let sockets = running && cfg!(target_os = "linux"); + assert!( + others + .iter() + .all(|name| sockets && name.len() == 37 && name.ends_with(".sock")), + "{names:?}" + ); +} + +/// Checks that a clone's `memory` probe found the memory target `target`, in MiB, online. +fn assert_memory(facts: &Printed, target: u64) { + let number = |key: &str| facts[&format!("memory.{key}")].parse::().unwrap(); + let block = number("block-size"); + assert!( + block > 0 && MEMORY_BLOCK_SIZE.is_multiple_of(block), + "{facts:?}" + ); + assert_eq!(number("online"), target * MIB, "{facts:?}"); + assert_eq!(number("blocks") * block, target * MIB, "{facts:?}"); + assert!(number("total") <= target * MIB, "{facts:?}"); +} + +/// Checks the root of a clone over `upper`, as its `mounts` and `contents` probes found it, in +/// the overlay mode that trusts the lowers, or not. +fn assert_root(facts: &Printed, upper: Upper, trusted: bool) { + assert_eq!(facts["mounts.distro.count"], "1", "{facts:?}"); + assert_eq!(facts["mounts.distro.type"], "erofs", "{facts:?}"); + let (used, unused, kind) = match upper { + Upper::Ram => ("tmpfs", "scratch", "tmpfs"), + _ => ("scratch", "tmpfs", "ext4"), + }; + assert_eq!(facts[&format!("mounts.{used}.count")], "1", "{facts:?}"); + assert_eq!(facts[&format!("mounts.{used}.type")], kind, "{facts:?}"); + assert_eq!(facts[&format!("mounts.{unused}.count")], "0", "{facts:?}"); + assert_eq!(facts["mounts.root.type"], "overlay", "{facts:?}"); + let overlay = facts["mounts.root.super"].split(',').collect::>(); + for option in ["metacopy=on", "redirect_dir=on", "xino=on"] { + assert_eq!( + overlay.contains(&option), + trusted, + "{option} in {overlay:?}" + ); + } + let contents = ["os-release.id", "write.root", "write.upper", "lower.write"] + .map(|key| &facts[&format!("contents.{key}")]); + assert_eq!( + contents, + ["ubuntu", "runtime-adapter", "runtime-adapter", "refused"], + "{facts:?}" + ); +} + +/// Checks the events of a clone's activation that its `activation` probe found: the repair +/// acknowledged the restore, and the clone took its reset, the host's first message, its Ready, +/// and its start, in that order, after the acknowledgement. Returns when each came, in +/// microseconds after the acknowledgement. +fn assert_timeline(facts: &Printed) -> [u64; 4] { + assert_eq!(facts["activation.acknowledged"], "yes", "{facts:?}"); + let events = ["reset", "message", "ready", "started"].map(|event| { + let value = &facts[&format!("activation.{event}")]; + value + .parse::() + .unwrap_or_else(|_| panic!("the clone's {event} is {value}: {facts:?}")) + }); + assert!(events.is_sorted(), "{facts:?}"); + events +} + +/// Returns the record of a clone's failure that a command printed, as `failure.KEY` lines: its +/// outcome, its evidence, the stage at which the clone failed, how far the clone got, the guest's +/// status, and how the teardown ended, each `-` if the command did not print it. +fn failure(printed: &Printed) -> [&str; 6] { + [ + "outcome", "evidence", "stage", "progress", "status", "teardown", + ] + .map(|key| { + printed + .get(&format!("failure.{key}")) + .map_or("-", String::as_str) + }) +} + +/// Returns the disks of a probe's `disks` fact, `NAME:NUMBER` or `NAME:NUMBER:ACCESS` each, with +/// their numbers. +fn disks(fact: &str) -> BTreeMap { + fact.split(' ') + .map(|disk| { + let mut parts = disk.split(':'); + let name = parts.next().unwrap().to_owned(); + let number = parts.next().and_then(|number| number.parse().ok()); + ( + name, + number.unwrap_or_else(|| panic!("{disk:?} in {fact:?}")), + ) + }) + .collect() +} + +/// Returns a boot of this host other than `boot`. +fn other_boot(boot: BootId) -> BootId { + let text = boot.to_string(); + let first = if text.starts_with('0') { "1" } else { "0" }; + format!("{first}{}", &text[1..]).parse().unwrap() +} + +/// The adapter's framing of a clone's control session that sends `early`, which the host must not +/// send before the clone is ready, in place of the clone's Activate, and, once the clone ended its +/// session, waits for the clone's VMM to exit, so that the restore sequence, which then ends the +/// launch, ends it once the VM ended on its own. +struct Early { + /// What the host sends first. + early: Message, + /// The clone's VMM. + vmm: Process, + /// Whether the clone's VMM exited once the clone ended the session. + exited: bool, +} + +impl CloneControl for Early { + type Session = Client; + type Error = io::Error; + + fn attach( + &mut self, + transport: Box, + capability: &[u8; CAPABILITY_LEN], + deadline: Instant, + ) -> io::Result { + Client::attach(transport, capability, deadline).map_err(io::Error::other) + } + + fn activate( + &mut self, + session: &mut Client, + _record: &[u8], + deadline: Instant, + ) -> io::Result> { + session + .send(&self.early, Some(deadline)) + .map_err(io::Error::other)?; + let answer = session.receive(Some(deadline)); + self.exited = self.vmm.wait(deadline)?; + Err(match answer { + Ok(message) => io::Error::other(format!("the clone answered {message:?}")), + Err(error) => io::Error::other(error), + }) + } + + fn start(&mut self, _session: &mut Client, _deadline: Instant) -> io::Result<()> { + Err(io::Error::other("the clone is never ready")) + } + + fn lost(&self, _error: &io::Error) -> bool { + false + } +} + +/// Starts a clone of `template` in RAM, in a sandbox of its own, through NVX's public API, whose +/// host sends `early` before the clone's Activate, and checks that the clone ended before it was +/// ready, with the status of the host's session, which attributes the failure to no one, as the +/// sandbox records it, and that the VMM exited, then releases the clone's lease, which removes the +/// record. +fn refuses_early(template: &Template, early: Message) { + let mut sandbox = Sandbox::new("cl-e"); + template.stage(&mut sandbox); + let store = template.store(); + let artifacts = env::var_os(Artifacts::ENV_DIR).unwrap(); + let runtime = Runtime::from_dir(artifacts, GUEST_BUILD).unwrap(); + let request = Request { + distro: template.image.clone(), + format: ImageFormat::Erofs, + identity: Uuid::parse(&template.identity).unwrap(), + scratch: None, + memory: None, + workload_identity: None, + overlay_mode: None, + scratch_journal: None, + }; + let plan = request.plan().unwrap(); + let staged = Staged::read(&sandbox.store).unwrap().unwrap(); + let key = store.read_key(&template.generation()).unwrap(); + let clone = request + .clone_request(plan.clone(), &staged.bindings(&plan), key.memory()) + .unwrap(); + let now = HostTime::now().unwrap(); + let mut acquired = store + .acquire(GENERATION, &sandbox.store, &CallerFiles::none(), &now, None) + .unwrap(); + let deadline = Instant::now() + Duration::from_secs(10); + let planned = ClonePlan::new(&store, &mut acquired, &clone, deadline).unwrap(); + let restoring = planned + .launch(&mut acquired, &runtime.artifacts.openvmm, &runtime.identity) + .unwrap(); + let vmm = Process::open(restoring.starting().process().reference()) + .unwrap() + .expect("the clone's VMM runs"); + let mut control = Early { + early: early.clone(), + vmm, + exited: false, + }; + let failure = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err(); + let log = || fs::read_to_string(sandbox.store.join(LOG_NAME)).unwrap_or_default(); + let problem = common::describe(&failure); + assert!( + !problem.contains("the clone answered"), + "{early:?}: {problem}" + ); + assert!(control.exited, "{early:?}: {problem}\n{}", log()); + let record = failure.record(); + let ended = ( + record.outcome, + record.evidence, + record.stage, + record.progress, + record.status, + &record.teardown, + ); + let expected = ( + Outcome::Unknown, + Evidence::Protocol, + Stage::Activation, + Progress::Activating, + Some(status::ACTIVATION_PROTOCOL), + &Teardown::Exited, + ); + assert_eq!(ended, expected, "{early:?}: {problem}\n{}", log()); + assert_eq!(record.lease, acquired.lease.id()); + // The sandbox records the failure, which keeps the clone from launching again, until the + // release of its lease. + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(record)); + assert!(records.journal.is_none() && records.runtime.is_none()); + let released = store.release(&mut acquired.locked).unwrap(); + assert_eq!(released.map(|lease| lease.id()), Some(acquired.lease.id())); + assert!(CloneRecords::read(&acquired.locked).unwrap().is_none()); + assert!(template.leases().is_empty()); +} + +/// Returns the limits of the executions of the scenarios that the host starts itself. +fn limits() -> Limits { + Limits::new(16 * 1024, Window::frames(4), Window::frames(8)) + .unwrap() + .shared_output() +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn one_generation_serves_clones_of_each_upper_and_memory_target() { + let template = Template::create("cl-reuse"); + let snapshot = template.payload().join(SNAPSHOT_NAME); + let captured = snapshot_files(&snapshot); + + // Two clones at once: one over a scratch image, at the template's RAM, which trusts its + // lowers, as the guest does by default, and one in RAM, at twice the template's RAM, which + // does not. + let scratch = StartOf { + upper: Upper::Scratch, + ..StartOf::default() + }; + let mut over_scratch = template.start("cl-a", scratch, 0); + let options = ["--memory", "512", "--overlay-mode", "untrusted-lowers"]; + let in_ram = StartOf { + options: &options, + ..StartOf::default() + }; + let mut in_ram = template.start("cl-b", in_ram, 0); + for cloned in [&over_scratch, &in_ram] { + cloned.assert_started(); + // On the boot that captured the template, the host's monotonic clock measures the + // downtime. + assert_eq!( + cloned.started["source"], "monotonic", + "{:?}", + cloned.started + ); + } + assert_ne!(over_scratch.started["lease"], in_ram.started["lease"]); + + // The store leases the generation to both, by their sandboxes' paths. + let mut leases = template + .leases() + .iter() + .map(|lease| (lease.id().to_string(), lease.sandbox().to_owned())) + .collect::>(); + leases.sort(); + let mut expected = [&over_scratch, &in_ram].map(|cloned| { + let root = cloned.sandbox.root.file_name().unwrap(); + ( + cloned.started["lease"].clone(), + Path::new(root).join("store"), + ) + }); + expected.sort(); + assert_eq!(leases.len(), expected.len(), "{leases:?}"); + for ((lease, sandbox), (expected, store)) in leases.iter().zip(&expected) { + assert_eq!(lease, expected); + assert!(sandbox.ends_with(store), "{}", sandbox.display()); + } + assert_payload(&template, true); + + let probes = ["memory", "mounts", "contents", "activation"]; + let facts = over_scratch.probe(&probes); + assert_memory(&facts, TEMPLATE_MEMORY); + assert_root(&facts, Upper::Scratch, true); + assert_timeline(&facts); + let facts = in_ram.probe(&probes); + assert_memory(&facts, 512); + assert_root(&facts, Upper::Ram, false); + assert_timeline(&facts); + over_scratch.stop(); + assert_eq!(template.leases().len(), 1); + in_ram.stop(); + assert!(template.leases().is_empty()); + + // Then another over a scratch image, at the template's RAM capacity. + let options = ["--memory", "1024"]; + let at_capacity = StartOf { + upper: Upper::Scratch, + options: &options, + ..StartOf::default() + }; + let mut at_capacity = template.start("cl-c", at_capacity, 0); + at_capacity.assert_started(); + let facts = at_capacity.probe(&["memory", "mounts", "contents"]); + assert_memory(&facts, TEMPLATE_CAPACITY); + assert_root(&facts, Upper::Scratch, true); + at_capacity.stop(); + assert!(template.leases().is_empty()); + + // The clones changed nothing of the generation's snapshot, which they restored in place, and + // left nothing in its payload. + assert_eq!(snapshot_files(&snapshot), captured); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts() { + let template = Template::create("cl-refuse"); + let length = fs::metadata(&template.image).unwrap().len(); + let longer = template.variant("longer.erofs", length + 4096); + let shorter = template.variant("shorter.erofs", length - 4096); + // OpenVMM attaches block devices of whole 512-byte sectors alone. + let ragged = template.variant("ragged.erofs", length + 100); + let placeholder = |found: u64| { + format!("is {found} bytes long, not the {length} bytes of the template's placeholder") + }; + let resized = 2 * DEFAULT_SCRATCH_LENGTH; + let beyond = ["--memory", "2048"]; + let part = ["--memory", "320"]; + let cases = [ + ( + StartOf { + image: Some(&longer), + ..StartOf::default() + }, + "supplied-length", + placeholder(length + 4096), + ), + ( + StartOf { + image: Some(&shorter), + upper: Upper::Scratch, + ..StartOf::default() + }, + "supplied-length", + placeholder(length - 4096), + ), + ( + StartOf { + image: Some(&ragged), + ..StartOf::default() + }, + "supplied-geometry", + format!("of {} bytes cannot be attached", length + 100), + ), + ( + StartOf { + upper: Upper::Resized(resized), + ..StartOf::default() + }, + "supplied-length", + format!( + "is {resized} bytes long, not the {DEFAULT_SCRATCH_LENGTH} bytes of the \ + template's scratch" + ), + ), + ( + StartOf { + options: &beyond, + ..StartOf::default() + }, + "invalid-targets", + format!("cannot restore with {} bytes of RAM", 2048 * MIB), + ), + ( + StartOf { + options: &part, + ..StartOf::default() + }, + "invalid-targets", + format!("cannot restore with {} bytes of RAM", 320 * MIB), + ), + ]; + for (start, evidence, problem) in cases { + let refused = template.start("cl-r", start, CLONE_FAILED); + // NVX's plan refused the request before anything of the clone's launch, so the start + // released the lease that the store granted. + assert!(refused.started.contains_key("lease"), "{problem}"); + assert_eq!( + failure(&refused.started), + ["request", evidence, "plan", "planned", "none", "no-vm"], + "{problem}: {:?}", + refused.started + ); + assert_eq!(refused.started["quarantined"], "no"); + refused.assert_refused(&problem); + assert!(template.leases().is_empty()); + } + + // The adapter refuses images that the sandbox does not stage, and releases the lease. + let start = StartOf { + unstaged: true, + ..StartOf::default() + }; + let refused = template.start("cl-r", start, FAILED); + assert!(refused.started.contains_key("lease")); + assert!(!refused.started.contains_key("failure.outcome")); + refused.assert_refused("; stage the images first"); + assert!(template.leases().is_empty()); + + // The store holds no other generation. + let mut sandbox = Sandbox::new("cl-r"); + template.stage(&mut sandbox); + let options = [ + OsStr::new("--template"), + template.dir().as_os_str(), + OsStr::new("--generation"), + OsStr::new("2"), + OsStr::new("--distro"), + &erofs(&template.image), + OsStr::new("--distro-identity"), + OsStr::new(&template.identity), + OsStr::new("--upper"), + OsStr::new("ram"), + ]; + let printed = sandbox.run("start", &options, FAILED); + assert!(printed.is_empty(), "{printed:?}"); + assert!(sandbox.stderr().contains("holds no generation 2")); + sandbox.assert_unlaunched(); + + // A generation whose key is gone is malformed: NVX's plan, which reads the key once the store + // leased the generation, attributes the failure to the template, and the start quarantines + // the generation through its tombstone, which refuses a later start without a lease. + fs::remove_file(template.payload().join(KEY_NAME)).unwrap(); + let refused = template.start("cl-r", StartOf::default(), CLONE_FAILED); + assert!( + refused.started.contains_key("lease"), + "{:?}", + refused.started + ); + assert_eq!( + failure(&refused.started), + [ + "template", + "malformed-generation", + "plan", + "planned", + "none", + "no-vm" + ], + "{:?}", + refused.started + ); + assert_eq!(refused.started["quarantined"], "yes"); + refused.assert_refused(" is missing"); + assert!(template.leases().is_empty()); + assert_eq!(template.store().tombstones().unwrap(), [GENERATION]); + let quarantined = template.start("cl-r", StartOf::default(), FAILED); + assert!(quarantined.started.is_empty(), "{:?}", quarantined.started); + quarantined.assert_refused(&format!( + "a tombstone quarantines the generation {GENERATION} of the template store" + )); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_clone_whose_activation_fails_ends_before_it_is_ready() { + let template = Template::create("cl-unready"); + let forbidden = ["--scratch-journal", "forbidden"]; + let stranger = ["--workload-identity", STRANGER]; + let cases = [ + ( + StartOf { + upper: Upper::Zeros, + ..StartOf::default() + }, + ("unusable-root", status::ACTIVATION_ROOT), + ), + ( + StartOf { + upper: Upper::Scratch, + options: &forbidden, + ..StartOf::default() + }, + ("unusable-root", status::ACTIVATION_ROOT), + ), + ( + StartOf { + options: &stranger, + ..StartOf::default() + }, + ("invalid-settings", status::ACTIVATION_WORKLOAD), + ), + ]; + for (start, (evidence, expected)) in cases { + let ended = template.start("cl-u", start, CLONE_FAILED); + let printed = &ended.started; + // The VM started, and ended before the clone was ready, with the status of the failure's + // class, which attributes the failure to the request, and the start released the lease. + assert!(printed.contains_key("pid"), "{printed:?}"); + assert!(!printed.contains_key("root"), "{printed:?}"); + let status = expected.to_string(); + assert_eq!( + failure(printed), + [ + "request", + evidence, + "activation", + "activating", + status.as_str(), + "exited" + ], + "{printed:?}" + ); + assert_eq!(printed["quarantined"], "no"); + assert_eq!(printed["released"], printed["lease"], "{printed:?}"); + assert!(ended.sandbox.record().is_none()); + // The release removed the records of the clone's launch. + let store = SandboxStore::open(&ended.sandbox.store).unwrap(); + assert!(CloneRecords::read(&store).unwrap().is_none()); + assert!(template.leases().is_empty()); + } + + // Until the clone is ready, it takes nothing but its Activate: a ping, a start, and the start + // of an execution, before it, each end the clone with the status of the host's session. + let timers = Timers::new(Duration::from_secs(1), Duration::from_secs(5)); + let execution = Start::new(Key::new(1, 1), limits(), timers, [b"/bin/true".to_vec()]); + for early in [ + Message::Ping, + Message::Start, + HostMessage::Start(execution).encode(), + ] { + refuses_early(&template, early); + } + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_clone_runs_its_executions_as_its_workload_identity_once_it_is_ready() { + let template = Template::create("cl-ident"); + let options = ["--workload-identity", NOBODY]; + let start = StartOf { + upper: Upper::Scratch, + options: &options, + ..StartOf::default() + }; + let mut cloned = template.start("cl-i", start, 0); + cloned.assert_started(); + let facts = cloned.probe(&["activation"]); + let events = assert_timeline(&facts); + assert_eq!(facts["activation.execution"], "none", "{facts:?}"); + + // Every execution runs as the record's workload identity, and one that names its own is + // refused before its program starts. + let ran = cloned.exec(&["--", "/bin/sh", "-c", "id -u; id -g"]); + let facts = ["launch", "status", "stdout", "stderr"].map(|key| &ran[&format!("exec.1.{key}")]); + assert_eq!( + facts, + ["started", "exit 0", "65534\\n65534\\n", ""], + "{ran:?}" + ); + let refused = cloned.exec(&["--identity", "1000:1000", "--", "/bin/true"]); + let launch = format!("failed 0 {EINVAL}"); + assert_eq!(refused["exec.1.launch"], launch, "{refused:?}"); + + // The first execution came once the clone's sandbox started. + let facts = cloned.probe(&["activation"]); + let execution = facts["activation.execution"].parse::().unwrap(); + assert!(execution >= events[3], "{facts:?}"); + assert_eq!(assert_timeline(&facts), events); + cloned.stop(); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_generation_is_eligible_by_the_clock_of_its_capture() { + let template = Template::create("cl-clock"); + let contract = template.generation().time().clone(); + let now = HostTime::now().unwrap(); + // The scenario runs on the host's boot that captured the template. + assert_eq!( + (now.host_id, now.boot_id, now.clock), + (contract.host_id, contract.boot_id, contract.host_clock) + ); + let started = |time: Option| { + let start = StartOf { + time, + ..StartOf::default() + }; + let mut cloned = template.start("cl-k", start, 0); + cloned.assert_started(); + cloned.stop(); + cloned.started + }; + + // On that boot, the host's monotonic clock measures the downtime, whatever UTC says. + let printed = started(None); + assert_eq!(printed["source"], "monotonic", "{printed:?}"); + let behind = HostTime { + utc_ns: contract.capture_utc_ns - 24 * HOUR_NS, + ..now + }; + let printed = started(Some(behind)); + assert_eq!(printed["source"], "monotonic", "{printed:?}"); + let downtime = behind.monotonic_ns - contract.capture_monotonic_ns; + assert_eq!(printed["downtime"], downtime.to_string()); + + // After a reboot, UTC does. + let rebooted = HostTime { + boot_id: other_boot(contract.boot_id), + monotonic_ns: 1, + utc_ns: contract.capture_utc_ns + HOUR_NS, + ..now + }; + let printed = started(Some(rebooted)); + assert_eq!(printed["source"], "utc", "{printed:?}"); + assert_eq!(printed["downtime"], HOUR_NS.to_string()); + + // A generation that is no longer eligible, or not yet, takes no lease. + let cases = [ + ( + HostTime { + monotonic_ns: contract.capture_monotonic_ns + MAX_DOWNTIME_NS + 1, + ..now + }, + "is no longer eligible: after a downtime of", + "by the host's monotonic clock", + ), + ( + HostTime { + utc_ns: contract.capture_utc_ns + MAX_DOWNTIME_NS + 1, + ..rebooted + }, + "is no longer eligible: after a downtime of", + "by UTC", + ), + ( + HostTime { + utc_ns: contract.capture_utc_ns - 1, + ..rebooted + }, + "is not eligible yet: by UTC", + "1 ns before the anchor of the capture", + ), + ]; + for (time, state, measured) in cases { + let start = StartOf { + time: Some(time), + ..StartOf::default() + }; + let refused = template.start("cl-k", start, FAILED); + assert!(refused.started.is_empty(), "{:?}", refused.started); + let generation = format!("the generation {GENERATION} of the template store"); + refused.assert_refused(&generation); + let stderr = refused.sandbox.stderr(); + assert!( + stderr.contains(state) && stderr.contains(measured), + "{stderr}" + ); + assert!(template.leases().is_empty()); + } +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_clone_reads_no_more_of_its_disks_than_its_root_needs() { + let template = Template::create("cl-reads"); + for upper in [Upper::Scratch, Upper::Ram] { + let start = StartOf { + upper, + ..StartOf::default() + }; + let mut cloned = template.start("cl-d", start, 0); + cloned.assert_started(); + let facts = cloned.probe(&["devices", "reads"]); + let sizes = disks(&facts["devices.disks"]); + let restored = disks(&facts["reads.disks"]); + assert_eq!( + sizes.keys().collect::>(), + restored.keys().collect::>(), + "{facts:?}" + ); + // The image and the scratch device, whose lengths are the template's. + let mut lengths = sizes + .values() + .map(|sectors| sectors * SECTOR) + .collect::>(); + lengths.sort_unstable(); + let mut expected = [ + fs::metadata(&template.image).unwrap().len(), + DEFAULT_SCRATCH_LENGTH, + ]; + expected.sort_unstable(); + assert_eq!(lengths, expected, "{facts:?}"); + for (disk, sectors) in &sizes { + assert!( + restored[disk] * 8 < *sectors, + "{upper:?}: the clone read {} of the {sectors} sectors of {disk}", + restored[disk] + ); + } + + // A read of each disk to its end, which the statistics count. + let facts = cloned.probe(&["read-disks"]); + let read = disks(&facts["read-disks.disks"]); + let whole = sizes + .iter() + .map(|(disk, sectors)| (disk.clone(), sectors * SECTOR)) + .collect::>(); + assert_eq!(read, whole, "{facts:?}"); + let facts = cloned.probe(&["reads"]); + let after = disks(&facts["reads.disks"]); + for (disk, sectors) in &sizes { + assert!( + after[disk] >= restored[disk] + sectors / 2, + "{upper:?}: {disk} read {} sectors, then {}, of {sectors}", + restored[disk], + after[disk] + ); + } + cloned.stop(); + } + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_clone_s_time_settles_after_it_is_ready() { + let template = Template::create("cl-time"); + let mut cloned = template.start("cl-t", StartOf::default(), 0); + cloned.assert_started(); + let deadline = Instant::now() + TIME_SETTLEMENT; + let facts = loop { + let facts = cloned.probe(&["time"]); + let status = facts.get("time.restore-status").map(String::as_str); + if !matches!(status, None | Some("pending")) { + break facts; + } + assert!(Instant::now() < deadline, "{facts:?}"); + thread::sleep(Duration::from_millis(100)); + }; + let settled = [ + "restore-status", + "violations", + "last-downtime-source", + "restore-generation", + ] + .map(|key| &facts[&format!("time.{key}")]); + let generation = &facts["time.generation"]; + assert_eq!( + settled, + ["ok", "0", "monotonic", generation.as_str()], + "{facts:?}" + ); + let downtime = facts["time.last-downtime-ns"].parse::().unwrap(); + assert!(downtime > 0, "{facts:?}"); + + // The clone still serves its order. + let facts = cloned.probe(&["activation", "memory"]); + assert_timeline(&facts); + assert_memory(&facts, TEMPLATE_MEMORY); + cloned.stop(); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn each_command_reattaches_to_the_clone_from_a_new_process() { + let template = Template::create("cl-again"); + let start = StartOf { + upper: Upper::Scratch, + ..StartOf::default() + }; + let mut cloned = template.start("cl-g", start, 0); + cloned.assert_started(); + // The launch committed: the sandbox records the clone's VM, and how far the clone got, and no + // failure. + let store = SandboxStore::open(&cloned.sandbox.store).unwrap(); + let records = CloneRecords::read(&store).unwrap().unwrap(); + assert_eq!(records.lease.id().to_string(), cloned.started["lease"]); + assert_eq!(records.progress, Progress::Committed); + assert!(records.runtime.is_some(), "{records:?}"); + assert!(records.journal.is_none() && records.failure.is_none()); + + // Each command, from a new process, reattaches to the clone in a new session of the same + // broker, in which the clone answers the Activate of its record again with its Ready, without + // activating again, so that the events of its activation stay as they were. + let first = cloned.sandbox.run("reattach", &[], 0); + assert_eq!(first["reattached"], "live", "{first:?}"); + assert_eq!(first["pid"], cloned.started["pid"]); + let facts = cloned.probe(&["activation"]); + let events = assert_timeline(&facts); + let ran = cloned.exec(&["--identity", NOBODY, "--", "/bin/sh", "-c", "id -u"]); + let facts_ran = ["launch", "status", "stdout"].map(|key| &ran[&format!("exec.1.{key}")]); + assert_eq!(facts_ran, ["started", "exit 0", "65534\\n"], "{ran:?}"); + let last = cloned.sandbox.run("reattach", &[], 0); + let epochs = [&first, &facts, &ran, &last].map(|printed| { + assert_eq!(printed["instance"], first["instance"], "{printed:?}"); + printed["epoch"].parse::().unwrap() + }); + assert!( + epochs.windows(2).all(|pair| pair[0] < pair[1]), + "{epochs:?}" + ); + let facts = cloned.probe(&["activation"]); + let again = assert_timeline(&facts); + assert_eq!(again, events, "{facts:?}"); + let execution = facts["activation.execution"].parse::().unwrap(); + assert!(execution >= events[3], "{facts:?}"); + cloned.stop(); + let store = SandboxStore::open(&cloned.sandbox.store).unwrap(); + assert!(CloneRecords::read(&store).unwrap().is_none()); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity() { + let template = Template::create("cl-other"); + let cloned = template.start("cl-o", StartOf::default(), 0); + cloned.assert_started(); + let vmm = cloned.sandbox.vmm().expect("the clone's VMM runs"); + let mut sandbox = cloned.sandbox; + + // The reattach finds the clone's VMM under another runtime identity, which might not speak + // this host's protocol, so it ends it through its verified identity, without a session, and + // the clone fails, by the host's evidence, which quarantines nothing. + let options = [OsStr::new("--guest-build"), OsStr::new(ANOTHER_BUILD)]; + let ended = sandbox.run("reattach", &options, CLONE_FAILED); + assert_eq!(ended["reattached"], "failed", "{ended:?}"); + assert_eq!( + failure(&ended), + [ + "host", + "runtime-mismatch", + "reattach", + "committed", + "none", + "terminated" + ], + "{ended:?}" + ); + assert_eq!(ended["failure.code"], "runtime-mismatch"); + assert_eq!(ended["quarantined"], "no"); + assert_eq!(ended["released"], cloned.started["lease"], "{ended:?}"); + assert!(!ended.contains_key("epoch"), "{ended:?}"); + assert!(sandbox.stderr().contains("differs in initramfs")); + assert!( + vmm.wait(Instant::now() + Duration::from_secs(10)).unwrap(), + "the clone's VMM still runs" + ); + assert!(sandbox.record().is_none()); + let store = SandboxStore::open(&sandbox.store).unwrap(); + assert!(CloneRecords::read(&store).unwrap().is_none()); + assert!(template.leases().is_empty()); + + // The template still serves clones. + let mut another = template.start("cl-p", StartOf::default(), 0); + another.assert_started(); + another.stop(); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released() { + let template = Template::create("cl-gone"); + let cloned = template.start("cl-x", StartOf::default(), 0); + cloned.assert_started(); + let vmm = cloned.sandbox.vmm().expect("the clone's VMM runs"); + let mut sandbox = cloned.sandbox; + + // The stop finds the clone running, in a new session, and pauses there, while the clone's VMM + // ends without a word of its guest, as a crash of the VMM would end it. + let pause = sandbox.root.join("found.pause"); + let hook = OsString::from( + StepHook { + step: Step::Found, + action: Action::Pause(pause.clone()), + } + .to_string(), + ); + let mut stopping = sandbox.start("stop", &[], &[(STEP_HOOK, &hook)]); + assert_eq!(sandbox.paused(&mut stopping, &pause), "found"); + vmm.kill().unwrap(); + assert!( + vmm.wait(Instant::now() + Duration::from_secs(10)).unwrap(), + "the clone's VMM still runs" + ); + fs::remove_file(&pause).unwrap(); + + // The stop stops nothing, and the clone's VM ended without its holder, with no report of the + // guest's own end: the clone fails, as NVX attributes the loss of its status, which + // quarantines nothing, before the stop releases the lease. + let ended = sandbox.finish(stopping, CLONE_FAILED); + let found = ["reattached", "pid", "stopped"].map(|key| &ended[key]); + assert_eq!( + found, + ["live", cloned.started["pid"].as_str(), "exited"], + "{ended:?}" + ); + assert_eq!( + failure(&ended), + [ + "unknown", + "lost-status", + "run", + "committed", + "none", + "exited" + ], + "{ended:?}" + ); + assert_eq!(ended["failure.code"], "exited"); + assert_eq!(ended["quarantined"], "no"); + assert_eq!(ended["released"], cloned.started["lease"], "{ended:?}"); + let stderr = sandbox.stderr(); + assert!( + stderr.contains("without an outcome report of the guest's own end"), + "{stderr}" + ); + assert!(sandbox.record().is_none()); + let store = SandboxStore::open(&sandbox.store).unwrap(); + assert!(CloneRecords::read(&store).unwrap().is_none()); + assert!(template.leases().is_empty()); + assert!(template.store().tombstones().unwrap().is_empty()); + + // The template still serves clones. + let mut another = template.start("cl-y", StartOf::default(), 0); + another.assert_started(); + another.stop(); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease() { + let template = Template::create("cl-tomb"); + let snapshot = template.payload().join(SNAPSHOT_NAME); + let captured = snapshot_files(&snapshot); + let start = StartOf { + upper: Upper::Scratch, + ..StartOf::default() + }; + let mut cloned = template.start("cl-q", start, 0); + cloned.assert_started(); + let store = template.store(); + let deadline = Instant::now() + Duration::from_secs(10); + store.tombstone(GENERATION, Some(deadline)).unwrap(); + assert_eq!(store.tombstones().unwrap(), [GENERATION]); + + // The generation takes no new lease. + let refused = template.start("cl-n", StartOf::default(), FAILED); + assert!(refused.started.is_empty(), "{:?}", refused.started); + refused.assert_refused(&format!( + "a tombstone quarantines the generation {GENERATION} of the template store" + )); + + // The clone keeps running on its lease: it reattaches, serves its order, and stops, which + // releases the lease. + let facts = cloned.probe(&["activation", "memory"]); + assert_timeline(&facts); + assert_memory(&facts, TEMPLATE_MEMORY); + let ran = cloned.exec(&["--identity", NOBODY, "--", "/bin/true"]); + assert_eq!(ran["exec.1.status"], "exit 0", "{ran:?}"); + assert_eq!(template.leases().len(), 1); + cloned.stop(); + assert!(template.leases().is_empty()); + + // The tombstone changed nothing of the generation's snapshot. + assert_eq!(snapshot_files(&snapshot), captured); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store() { + let template = Template::create("cl-lost"); + // A start that ended once the store leased the generation, before it recorded the clone, + // leaves the lease, and no record that names the template store. + let mut sandbox = Sandbox::new("cl-l"); + template.stage(&mut sandbox); + let lease = lease_to(&template, &sandbox); + + // A sandbox that holds a lease must not move. A command on one that moved refuses it, rather + // than take for it the sandbox that the lease's path holds then: one with a lease of its own, + // then one without a lease. + let named = sandbox.store.clone(); + let moved = sandbox.root.join("moved"); + fs::rename(&named, &moved).unwrap(); + SandboxStore::create(&named).unwrap(); + let other = lease_to(&template, &sandbox); + sandbox.store = moved.clone(); + let refused = sandbox.run("stop", &[], FAILED); + assert!(refused.is_empty(), "{refused:?}"); + let stderr = sandbox.stderr(); + assert!(stderr.contains("which does not hold it"), "{stderr}"); + assert_eq!(template.leases().len(), 2); + sandbox.store = named.clone(); + let options = [OsStr::new("--template"), template.dir().as_os_str()]; + let released = sandbox.run("stop", &options, FAILED); + assert_eq!(released["released"], other, "{released:?}"); + sandbox.store = moved.clone(); + let refused = sandbox.run("reattach", &[], FAILED); + assert!(refused.is_empty(), "{refused:?}"); + let stderr = sandbox.stderr(); + assert!(stderr.contains("which does not hold it"), "{stderr}"); + let leases = template.leases(); + assert_eq!(leases.len(), 1); + assert_eq!(leases[0].id().to_string(), lease); + // The sandbox moves back, as its lease names it. + fs::remove_dir_all(&named).unwrap(); + fs::rename(&moved, &named).unwrap(); + sandbox.store = named; + + // The stop finds that the clone never launched, and keeps the lease without its template + // store. + let kept = sandbox.run("stop", &[], FAILED); + assert_eq!(kept["reattached"], "ended", "{kept:?}"); + assert_eq!(kept["progress"], "planned", "{kept:?}"); + assert!(!kept.contains_key("released"), "{kept:?}"); + let stderr = sandbox.stderr(); + assert!(stderr.contains("records no clone"), "{stderr}"); + assert!(stderr.contains("--template TEMPLATES"), "{stderr}"); + assert_eq!(template.leases().len(), 1); + + // Another template store does not release it. + let other = sandbox.root.join("other"); + Store::create(&other).unwrap(); + let options = [OsStr::new("--template"), other.as_os_str()]; + let refused = sandbox.run("stop", &options, FAILED); + assert!(!refused.contains_key("released"), "{refused:?}"); + let stderr = sandbox.stderr(); + assert!(stderr.contains("not of this one"), "{stderr}"); + assert_eq!(template.leases().len(), 1); + + // Its own does, and the sandbox leases the generation again, to a clone whose record the + // release then leaves. + let options = [OsStr::new("--template"), template.dir().as_os_str()]; + let released = sandbox.run("reattach", &options, FAILED); + assert_eq!(released["released"], lease, "{released:?}"); + assert!(template.leases().is_empty()); + let started = start_in(&mut sandbox, &template, template.dir()); + assert_ne!(started["lease"], lease, "{started:?}"); + let stopped = sandbox.run("stop", &[], 0); + assert_eq!(stopped["released"], started["lease"], "{stopped:?}"); + assert!(template.leases().is_empty()); + + // A start of another template's clone in the sandbox that ended as the first did leaves a + // lease of that template's store, which the earlier clone's record does not name. + let another = Template::create("cl-gone"); + let lease = lease_to(&another, &sandbox); + let kept = sandbox.run("stop", &[], FAILED); + assert!(!kept.contains_key("released"), "{kept:?}"); + let stderr = sandbox.stderr(); + assert!(stderr.contains("which did not grant its lease"), "{stderr}"); + assert!(stderr.contains("--template TEMPLATES"), "{stderr}"); + assert_eq!(another.leases().len(), 1); + let options = [OsStr::new("--template"), another.dir().as_os_str()]; + let released = sandbox.run("stop", &options, FAILED); + assert_eq!(released["released"], lease, "{released:?}"); + assert!(another.leases().is_empty()); + let started = start_in(&mut sandbox, &template, another.dir()); + let stopped = sandbox.run("stop", &[], 0); + assert_eq!(stopped["released"], started["lease"], "{stopped:?}"); + assert!(another.leases().is_empty()); + assert_payload(&template, false); + assert_payload(&another, false); +} + +/// Leases the generation of `template` to the store of `sandbox`, through NVX's store, as a start +/// does before it records its clone, and returns the lease's identity. +fn lease_to(template: &Template, sandbox: &Sandbox) -> String { + let now = HostTime::now().unwrap(); + let acquired = template + .store() + .acquire(GENERATION, &sandbox.store, &CallerFiles::none(), &now, None) + .unwrap(); + acquired.lease.id().to_string() +} + +/// Starts a clone in RAM, in the store of `sandbox`, which staged the image of `template`, of the +/// generation of the template store `templates`, checks that it started, and returns what the +/// start printed. +fn start_in(sandbox: &mut Sandbox, template: &Template, templates: &Path) -> Printed { + let generation = GENERATION.to_string(); + let distro = erofs(&template.image); + let options = [ + OsStr::new("--template"), + templates.as_os_str(), + OsStr::new("--generation"), + OsStr::new(&generation), + OsStr::new("--distro"), + &distro, + OsStr::new("--distro-identity"), + OsStr::new(&template.identity), + OsStr::new("--upper"), + OsStr::new("ram"), + ]; + sandbox.run("start", &options, 0) +} diff --git a/tests/runtime-adapter/tests/vm_steps.rs b/tests/runtime-adapter/tests/vm_steps.rs index c805b71a2..bc4821d7d 100644 --- a/tests/runtime-adapter/tests/vm_steps.rs +++ b/tests/runtime-adapter/tests/vm_steps.rs @@ -185,7 +185,7 @@ fn the_hook_pauses_a_launch_at_each_of_its_steps() { let staged = Staged::read(&sandbox.store).unwrap().unwrap(); let scratch_binding = staged.scratch.unwrap().binding; - for step in Step::ALL { + for step in Step::LAUNCH { let pause = sandbox.root.join(format!("{step}.pause")); let hook = hook(step, Action::Pause(pause.clone())); let mut launch = sandbox.start_launch(&images, &[(STEP_HOOK, &hook)]);