diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 03f84aedd..b6e14a233 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -18,16 +18,16 @@ through NVX's supervisor, and finds a VM again after it restarts, as an orchestrator does. A VM may have a layered root: an image under an upper, which the host program -stages and plans with NVX's typed plan for an adapter's own guest, whose -staged bindings a launch checks before it hands OpenVMM the checked files, and -which the guest agent's worker assembles through `nvx::guest::root`, in a -process of its own rather than PID 1, as the host program's root configuration -asks. The host program may share host directories and files with the root, -which it exports with NVX's `filesystem::plan`, and which the worker maps -beneath the root from the host's mapping table, before the host starts the -sandbox. The host program then checks the root from inside the VM with the -worker's probes, in a [table of scenarios](#layered-scenarios), to which each -new check adds a row. +stages and plans with NVX's typed plan for an adapter's own guest, whose staged +bindings a launch checks before it hands OpenVMM the checked files, and which +the guest agent's worker assembles through `nvx::guest::root`, in a process of +its own rather than PID 1, as the host program's root configuration asks. The +host program may share host directories and files with the root, which it +exports with NVX's `filesystem::plan`, and which the worker maps beneath the +root from the host's mapping table, before the host starts the sandbox. The host +program then checks the root from inside the VM with the worker's probes, and +runs executions in it, several at once over one session, in a +[table of scenarios](#layered-scenarios), to which each new check adds a row. ## Configurations @@ -309,6 +309,20 @@ bytes, 2 if standard error did, and 4 if the execution is reusable. Stage 0 is the worker's refusal of the start, and a later stage is the binding's number of the stage of `nvx::guest::spawn` that failed, such as 21 for its `exec`. +`exec::HostExecution` is the host's side of one execution, in every +configuration: the engine's `Peer`, with a sink that takes the program's output +as it arrives, unless its caller stalls it, so that the guest gets no credit +for more, and a source that hands over the program's input, if the start says +that the host sends it, then ends it. Its caller sends the start, hands it each +of the guest's messages about the execution with `deliver`, and sends the +messages that `serve` returns, until the peer finished, serving it again once +its `deadline`, the peer's next timer, passes. The sink keeps at most +`exec::MAX_CAPTURE`, 16 MiB, of each stream: once more of a stream arrives, the +host's side cancels the execution, and its peer discards the rest once the grace +ends, so that the outcome marks the stream truncated, with the cause +`cancelled`. The host program's [driver](#host-program) and the tests of the +worker run it. + ### Root-profile entries The probe of `worker::entries`, which takes no arguments either, enters the @@ -675,10 +689,53 @@ with a new message, in a new attachment. A failure of the session, a send that fails or times out, a malformed message, an answer whose body does not decode, and a message that only the host program -sends end the client, and its later calls fail. A receive that times out -doesn't: it keeps what arrived. A request ends the client on any failure after -the framing accepted it, a timeout included, because an answer that arrived -later would answer the wrong request. +sends, an execution's start, input, end of input, or cancellation included, +end the client, and its later calls fail. A receive that times out doesn't: it +keeps what arrived. A request ends the client on any failure after the framing +accepted it, a timeout included, because an answer that arrived later would +answer the wrong request. + +`host::Executions` is the host program's driver of +[executions](#executions), over one attachment of the client: + +- `start` sends an execution's start, once it checks that the start decodes, + as the guest decodes it. A start that does not, such as one with a relative + working directory, and one of an ID that the driver started already, fail + before anything is sent, and leave the client usable: the guest would end its + session on either. So do a start that says that the host sends the program's + input, one whose output windows count bytes, or more than + `host::MAX_OUTPUT_FRAMES`, 32, frames, one beyond `exec::MAX_EXECUTIONS`, 8, + executions, as the guest runs no more at once, and any once the driver ran: + a driver runs one batch, whose executions it starts before it runs them; +- `run` carries the guest's messages to each execution's + `exec::HostExecution`, and theirs to the guest, until every execution + finished. One thread runs them all: each receive waits at most until the + earliest timer of the executions, which so fires in time whatever the guest + sends, and no execution's messages wait for another's. A message of the + guest about no execution, about one that the driver did not start, whose body + does not decode, or that breaks the engine's rules, ends the run, as a + deadline that passes first does: in a receive, which leaves the client + usable, or in a send. Each send waits, as the client's do, until the + transport takes the message or the run's deadline passes, and no timer fires + meanwhile, so a guest that stops reading the session holds the run until its + deadline, when the send's timeout ends the client. + +Even a guest that reads can hold a send while the host does not read: the +session's broker passes the guest's credit for the host's records on only +behind the guest's own records, and takes no more of the host's records than +that credit allows. So the driver sends little before it reads: no program's +input, which one thread could not send while it reads the programs' output; the +starts of at most 8 executions, whose answers wait for the run, as no program's +output flows before the run's credit does; and, in the run, besides a +cancellation for each execution, at most a credit for each message that it +reads, of output windows of at most 32 frames, so that the guest's messages +ahead of its own credit stay few. The guest may still send messages about the +executions of a run that gave up on one or failed, so a later batch takes an +attachment of its own, as each command of the host program does. + +The guest ends its session on a second start of the same key, even in a later +epoch of the session, so the keys of a driver's executions must be new to the +guest. ### The host's order @@ -949,7 +1006,7 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from ends its VM. LAYERED, - `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [SHARES]`, + `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [--workload-identity UID:GID] [SHARES]`, gives the VM a [layered root](#layered-roots). Its image is the file PATH, an EROFS file system, `erofs`, or a GPT disk, `gpt`, of which the guest expects the identity UUID: the file system's UUID, or the disk's GUID. Its upper is @@ -958,11 +1015,14 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from MODE, `trusted-lowers` or `untrusted-lowers`, checks a scratch's journal against the policy POLICY, `allowed` or `forbidden`, which only a scratch takes, and refuses a first partition larger than BYTES, which only a GPT disk - takes, with the guest's defaults unless given. Before it spawns the VM, the - launch checks the bindings that STORE stages for the image and the scratch - image, and fails if STORE stages none for either, or one no longer holds, as - [the checked handoff](#the-checked-handoff) describes. It records the - bindings that it checked. + takes, with the guest's defaults unless given. A workload identity, the user + ID UID and the group ID GID, two numbers from 1 to 4294967294, is the identity + as which every execution of the sandbox runs: the guest checks its account in + the assembled root, without which the root's `identity` stage fails. Before it + spawns the VM, the launch checks the bindings that STORE stages for the image + and the scratch image, and fails if STORE stages none for either, or one no + longer holds, as [the checked handoff](#the-checked-handoff) describes. It + records the bindings that it checked. SHARES, `--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]... [--share-root DIR]...`, @@ -1003,29 +1063,59 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from each PROBE in the VM's layered root, in order, and prints the facts that it found. A PROBE is the text of a [probe](#probes): its name, then its arguments, each after one space. A probe that fails fails the command. - -`reattach`, `stop`, and `probe` find a VM that another runtime identity -launched mismatched, and then signal nothing: the VM keeps running. The -commands print what they find as `KEY=VALUE` lines: `reattached` (`live`, -`mismatched`, `exited`, or `indeterminate`), `differences` (the parts of the -runtime identity in which a mismatched VM differs), `pid` (the VMM's process -ID), `instance` and `epoch` (the identity of the session that they attached -to), `root` (`ready` or `failed`: how the guest answered the root -configuration, or `failed` when a stop finds that the root's teardown failed), -`build` (the build of the guest agent that assembled the root), `stage` and -`problem` (the stage of the root that failed, such as `assembly`, and the -problem, or the problem of a mapping that failed), `table` (`applied` or -`failed`: how the guest took the mapping table), `mapped` (how many of the -table's mappings the guest applied), `sandbox` (`started` once the guest -started the sandbox), `PROBE.KEY` (the fact KEY that the probe named PROBE found), -`stopped` (`graceful`, `forced`, or `timed-out`), `outcome` (such as -`success 0`, or `guest-exit 14` once the assembly failed), and `teardown` -(`complete` or `incomplete`: whether the VMM's teardown of the VM completed, -which the outcome report records apart from the root's). Values escape `\`, newlines, and -carriage returns as `\\`, `\n`, and `\r`. The commands exit with 0 once they -did what they were asked, with 3 when they find the VM mismatched, with 4 when -the guest reported that a stage of the root failed, and the VM ended with the -stage's status and a complete teardown, with 5 when the +- `exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as + `reattach` does, then runs each EXECUTION in the VM's layered root, all at + once, over that one session, through `host::Executions`, and prints their + outputs and outcomes once every one finished. It runs at most 8 executions, + as many as the guest runs at once. An EXECUTION, + `[--identity UID:GID [--groups GID,...] [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, + runs PROGRAM with its ARGs, up to a lone `;`, which a shell takes quoted, as + `find -exec` takes them, without standard input. Without `--identity`, it + names no identity, and runs as the sandbox's workload identity, which the + guest otherwise refuses; with it, it runs as the user ID UID and the group ID + GID, with the supplementary groups GIDs and the capabilities numbered N, such + as 0 for `CAP_CHOWN`, if given, and the guest checks its account, unless + `--numeric` says that the IDs have none. It starts in the root's directory + DIR, or `/`, with exactly the environment of its `--env` entries. It runs for + at most a minute, with 16 KiB chunks and the credit of 4 of them for its + input and 8 for its output, which its standard output and error share. The + command keeps at most 16 MiB of each stream, and cancels an execution whose + stream would exceed them. + Execution N, from 1, has the key `(B + N, E)`: E is the epoch of the + command's attachment, and B the first 8 bytes of the broker instance's ID, as + a little-endian number. The guest refuses a key that it heard of before, and + the epochs of a broker instance only advance, while a broker restored from a + snapshot has a new instance ID, so no command repeats the key of another's + execution. + +`reattach`, `stop`, `probe`, and `exec` find a VM that another runtime identity +launched mismatched, and then signal nothing: the VM keeps running. The commands +print what they find as `KEY=VALUE` lines: `reattached` (`live`, `mismatched`, +`exited`, or `indeterminate`), `differences` (the parts of the runtime identity +in which a mismatched VM differs), `pid` (the VMM's process ID), `instance` and +`epoch` (the identity of the session that they attached to), `root` (`ready` or +`failed`: how the guest answered the root configuration, or `failed` when a stop +finds that the root's teardown failed), `build` (the build of the guest agent +that assembled the root), `stage` and `problem` (the stage of the root that +failed, such as `assembly`, and the problem, or the problem of a mapping that +failed), `table` (`applied` or `failed`: how the guest took the mapping table), +`mapped` (how many of the table's mappings the guest applied), `sandbox` +(`started` once the guest started the sandbox), `PROBE.KEY` (the fact KEY that +the probe named PROBE found), `exec.N.KEY` (what execution N did: its `launch`, +`started`, `failed STAGE ERRNO`, or `none`; its `status`, `exit CODE`, +`signal NUMBER`, or `none`; the `cause` for which a side ended it, such as +`timed-out`, or `none`; the streams that it `truncated`, such as +`stdout,stderr`, or `none`; whether it `settled`, `reusable` or `fenced`; and +its `stdout` and `stderr`), `stopped` (`graceful`, `forced`, or `timed-out`), +`outcome` (such as `success 0`, or `guest-exit 14` once the assembly failed), +and `teardown` (`complete` or `incomplete`: whether the VMM's teardown of the VM +completed, which the outcome report records apart from the root's). Values +escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each +byte of a program's output that is not part of UTF-8 text as `\xHH`. The +commands exit with 0 once they did what they were asked, `exec` once every +execution finished, whatever its outcome, with 3 when they find the VM +mismatched, with 4 when the guest reported that a stage of the root failed, and +the VM ended with the stage's status and a complete teardown, with 5 when the [step hook](#the-step-hook) ended a launch, with 6 when the guest failed a mapping of the table, and the VM stopped gracefully, with a success and a complete teardown, and with 1, after a diagnostic on standard error, when they @@ -1051,13 +1141,16 @@ independent adapter's does: The unit tests of `frame`, `root`, `probe`, and `exec` run in every configuration. They cover each kind's encoding, messages in pieces of any size, -a message of each kind split at every byte, the headers that the framing -refuses before their body arrives, and the root configurations, readinesses, -parts of mapping tables, failures, probes, facts, and messages of executions -that decode, and those that don't. In the guest-only configuration, -the unit tests of `worker::probes` cover what the probes read: the mount table -with its escapes, the kernel's records of a process, `os-release`, and the -guest's memory, and the probes and arguments that the worker refuses. Those of +a message of each kind split at every byte, the headers that the framing refuses +before their body arrives, and the root configurations, readinesses, parts of +mapping tables, failures, probes, facts, and messages of executions that decode, +and those that don't, and the host's side of an execution, which still sends the +cancellation of one that it gives up on in the step that ends it, without grace +or settlement time, and which cancels one whose stream outgrows the 16 MiB that +it keeps. In the guest-only configuration, the unit tests of `worker::probes` +cover what the probes read: the mount table with its escapes, the kernel's +records of a process, `os-release`, and the guest's memory, and the probes and +arguments that the worker refuses. Those of `worker::executions`, which spawn each program without entering a root, as a fake root does, hand an execution the host's cancellation once its timeout's instant passed, after its program ended before it, as a worker that @@ -1126,7 +1219,22 @@ Windows. They cover: nor `Start`, but `Stop`, and learns how the stop ended, with or without a teardown that failed, or as the session ends; and `Maps` or `Start` answered with another kind, or failures whose bodies do not decode, which end the - client. + client; +- the driver of executions, whose guest runs the runner of NVX's execution + engine for each start, as the worker does, without a program: two + executions at once, one with more output than its credit holds, each + byte-exact, with its outcome, and one whose launch the guest refuses, whose + records carry their messages whole or split; the starts that the guest would + not decode, a second start of an ID, a start whose input the host would + send, starts whose output windows count bytes or more than 32 frames, a + start beyond 8 executions, and a start once the driver ran, of which the + driver sends nothing, with a client that stays usable; a message about an + execution that it did not start, about none, whose body does not decode, or + that breaks the engine's rules, and the end of the stream, each of which ends + the run; a timeout that fires while the guest says nothing, after which the + execution is cancelled, then fenced; and a run whose deadline passes first, + in a receive, which leaves the client usable, or in a send that waits for a + guest that stopped reading, which ends it. Through a relay between its test broker and a pseudo-terminal, `tests/guest_session.rs` also checks that `guest::serve_terminal` and @@ -1171,19 +1279,19 @@ the binary exited, so that none of them outlives the test or holds its output open, which tests check too. `tests/guest_session.rs` also runs the worker's executions behind its test -broker, whose host side starts the sandbox, then drives the engine's peers, with -frame-counted credit and one output pool. The fake root prepares each -execution's entry, as the worker describes it, in a directory of the tests' own -whose account databases hold the accounts that the tests name, and then spawns -the program without entering a root, which needs privileges, in the tests' own -root and identity. Unless a test says otherwise, the sandbox has a workload -identity, as which every execution runs. Where the tests can create cgroups, as -root in a cgroup v2 hierarchy with the `pids` controller, the fake root gives -the worker a workload cgroup of the test's own, below which each execution runs -in a cgroup of its own, so that an execution whose program ends on its own -settles, and is reusable. Elsewhere, the fake root names no cgroups, and each -program runs in a process group alone, so that such an execution is fenced. The -tests cover: +broker, whose host side starts the sandbox, then drives each execution's +`exec::HostExecution`, with frame-counted credit and one output pool. The fake +root prepares each execution's entry, as the worker describes it, in a directory +of the tests' own whose account databases hold the accounts that the tests name, +and then spawns the program without entering a root, which needs privileges, in +the tests' own root and identity. Unless a test says otherwise, the sandbox has +a workload identity, as which every execution runs. Where the tests can create +cgroups, as root in a cgroup v2 hierarchy with the `pids` controller, the fake +root gives the worker a workload cgroup of the test's own, below which each +execution runs in a cgroup of its own, so that an execution whose program ends +on its own settles, and is reusable. Elsewhere, the fake root names no cgroups, +and each program runs in a process group alone, so that such an execution is +fenced. The tests cover: - three executions at once, each byte-exact: standard output beyond a MiB, 1.5 MiB of input through `cat`, and both streams of a program that fails; @@ -1313,7 +1421,8 @@ Windows: - `runtime-adapter-host launch` with shares that it cannot export, which it refuses with status 1 before it creates the store, with artifacts that boot nothing; -- the command lines that the host program refuses, each with status 2. +- the command lines that the host program refuses, each with status 2, those of + `exec` and of `--workload-identity` included. `tests/step_hook.rs` checks, in the host-only configuration, on Linux and Windows, the [step hook](#the-step-hook): the values of @@ -1408,22 +1517,25 @@ stages its images, then launches them, and is a row of the table that its | `unstaged_images_are_refused_before_spawn` | The image over the scratch, which the scenario does not stage | A launch that fails before it spawns the VM, since the store stages no image at its path, and that leaves no launch journal, VMM log, runtime record, or launch record; then, once the scenario stages the images, the root that the worker assembles, and a clean teardown | | `a_changed_image_is_refused_before_spawn` | The GPT disk over the scratch, which the scenario writes in place after it stages it | A launch that fails before it spawns the VM, since the disk's modification time differs from its binding's, and that leaves nothing of a launch in the store; then, once the scenario stages the disk again, the root, and a clean teardown | | `a_replaced_scratch_is_refused_before_spawn` | The image over the scratch, which another copy of the template replaces after the scenario stages it | A launch that fails before it spawns the VM, since the scratch image's identity differs from its binding's, and that leaves nothing of a launch in the store; then, once the scenario stages the images again, the root, and a clean teardown | +| `executions_run_in_the_root` | The image over the scratch, without a workload identity, in which `exec` runs executions that name their identities | Over one session, four executions as `nobody` that meet only while all of them run, beside one as root that writes more than a MiB to its standard output and a little to its standard error, each byte-exact, ending on its own with 0, and reusable. Over a later session, with new keys: a program that leaves a descendant in a session of its own, which the program's end, as PID 1 of its namespace, ends, so that no output is lost; `nobody`'s IDs and groups, with empty permitted, effective, and bounding sets; root with `CAP_CHOWN` alone in each of them; exactly the start's environment; the start's working directory; the entry's default hostname, `nvx-sandbox`, and PID 1; and a start without an identity, and one with `CAP_SYS_ADMIN`, which the guest refuses at stage 0 with `EINVAL` and `EPERM`, fenced and without output | +| `executions_run_as_the_workload_identity` | The image over the scratch, with the workload identity of `nobody` | Executions that name no identity, which run as `nobody`, without capabilities, with exactly the start's environment, and a start that names an identity, which the guest refuses at stage 0 with `EINVAL` | +| `accountless_workload_identity_fails_its_stage` | The image over the scratch, with a workload identity whose account the image lacks | The identity stage's failure once the guest assembled the root, which the launch reports, and a VM that ends with 16, which unmounted the scratch | A row names the root, the image, the GPT disk's top-layer variant, the identity that the launch expects of the image, the upper, the runtime directory, the overlay mode, the scratch's journal policy, the bound of the -first partition, and the host paths that the launch shares, and what the -scenario expects: a root that the guest assembles, with a check that probes -it, a teardown that fails after a check, the stage that fails, with its -problem, a mapping of the table that the guest fails, after how many, with its -problem, shares that the host program refuses before it spawns the VM, with -the problem, or a launch that the host program refuses before it spawns the -VM, with the change that the scenario makes after it stages the images, and -the problem. Each scenario that launches a VM checks the launch's record, with -the bindings that the store stages, which the launch checked, and the number -of mappings that the root configuration announced, and, over a scratch, -whether the VM left the scratch's journal to recover, which tells whether the -guest unmounted the scratch. +first partition, the sandbox's workload identity, and the host paths that the +launch shares, and what the scenario expects: a root that the guest assembles, +with a check that probes it or runs executions in it, a teardown that fails +after a check, the stage that fails, with its problem, a mapping of the table +that the guest fails, after how many, with its problem, shares that the host +program refuses before it spawns the VM, with the problem, or a launch that the +host program refuses before it spawns the VM, with the change that the scenario +makes after it stages the images, and the problem. Each scenario that launches a +VM checks the launch's record, with the bindings that the store stages, which +the launch checked, and the number of mappings that the root configuration +announced, and, over a scratch, whether the VM left the scratch's journal to +recover, which tells whether the guest unmounted the scratch. A scenario with the mapping table writes its sources in its directory, below `sources`: a directory `dir`, with the files `marker` and `secret` and an empty diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 29070500e..e580b4ecf 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -21,20 +21,23 @@ //! //! LAYERED, `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) //! [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] -//! [--first-partition-bound BYTES] [SHARES]`, gives the VM a layered root, which the guest -//! agent's worker assembles; see `runtime_adapter::layered` and `runtime_adapter::worker`. Its -//! image is the raw file PATH, an EROFS filesystem, `erofs`, or a GPT disk, `gpt`, of which the -//! guest expects the identity UUID: the file system's UUID, or the disk's GUID. Its upper is the -//! ext4 scratch image PATH, or RAM. The guest assembles the root below its runtime directory -//! DIR, `/run/runtime-adapter` by default, in the overlay mode MODE, `trusted-lowers` or -//! `untrusted-lowers`, checks a scratch's journal against the policy POLICY, `allowed` or -//! `forbidden`, and refuses a first partition of a GPT disk larger than BYTES, with the guest's -//! defaults, `trusted-lowers`, `allowed`, and 64 KiB, unless given. Before it spawns the VM, the +//! [--first-partition-bound BYTES] [--workload-identity UID:GID] [SHARES]`, gives the VM a +//! layered root, which the guest agent's worker assembles; see `runtime_adapter::layered` and +//! `runtime_adapter::worker`. Its image is the raw file PATH, an EROFS filesystem, `erofs`, or a +//! GPT disk, `gpt`, of which the guest expects the identity UUID: the file system's UUID, or the +//! disk's GUID. Its upper is the ext4 scratch image PATH, or RAM. The guest assembles the root +//! below its runtime directory DIR, `/run/runtime-adapter` by default, in the overlay mode MODE, +//! `trusted-lowers` or `untrusted-lowers`, checks a scratch's journal against the policy POLICY, +//! `allowed` or `forbidden`, and refuses a first partition of a GPT disk larger than BYTES, with +//! the guest's defaults, `trusted-lowers`, `allowed`, and 64 KiB, unless given. A workload +//! identity, the user ID UID and the group ID GID, two numbers from 1 to 4294967294, is the +//! identity as which every execution of the sandbox runs: the guest checks its account in the +//! assembled root, without which the root's `identity` stage fails. Before it spawns the VM, the //! launch checks the bindings that STORE stages for the image and the scratch image, from their //! metadata alone, and fails if STORE stages none for either, or one no longer holds, as for an //! image that changed after it was staged. It hands OpenVMM the files that the checks opened, -//! which it holds until the guest has assembled the root, or until it has torn down a launch -//! that failed, and records the bindings. +//! which it holds until the guest has assembled the root, or until it has torn down a launch that +//! failed, and records the bindings. //! //! SHARES, `--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]... //! [--share-root DIR]...`, shares each host directory or regular file HOST with the root, at the @@ -69,6 +72,26 @@ //! runs each PROBE in the VM's layered root, and prints the facts that it found. A PROBE is the //! text of a probe: its name, then its arguments, each after one space; see //! `runtime_adapter::probe` and `runtime_adapter::worker::probes`. +//! - `runtime-adapter-host exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as +//! `reattach` does, then runs each EXECUTION in the VM's layered root, all at once, over that one +//! session, through `runtime_adapter::host::Executions`, and prints their outputs and outcomes +//! once every one finished. It runs at most 8 executions, as many as the guest runs at once; see +//! `runtime_adapter::exec::MAX_EXECUTIONS`. An EXECUTION, `[--identity UID:GID [--groups GID,...] +//! [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, +//! runs PROGRAM with its ARGs, up to a lone `;`, as `find -exec` takes them, without standard +//! input. Without `--identity`, it names no identity, and runs as the sandbox's workload +//! identity, which the guest otherwise refuses; with it, it runs as the user ID UID and the group +//! ID GID, with the supplementary groups GIDs and the capabilities numbered N, such as 0 for +//! `CAP_CHOWN`, if given, and the guest checks its account, unless `--numeric` says that the IDs +//! have none. It starts in the root's directory DIR, or `/`, with exactly the environment of its +//! `--env` entries. It runs for at most a minute, with 16 KiB chunks and the credit of 4 of them +//! for its input and 8 for its output, which its standard output and error share. The command +//! keeps at most 16 MiB of each stream, and cancels an execution whose stream would exceed them; +//! see `runtime_adapter::exec::MAX_CAPTURE`. Execution N, from 1, has the key `(B + N, E)`: E is +//! the epoch of the command's attachment, and B the first 8 bytes of the broker instance's ID, as +//! a little-endian number. The guest refuses a key that it heard of before, and the epochs of a +//! broker instance only advance, while a broker restored from a snapshot has a new instance ID, +//! so no command repeats the key of another's execution. //! //! Each command but `stage` takes NVX's artifacts from the directory that `NVX_ARTIFACTS_DIR` //! names, a repository checkout or an extracted release, and the VM's runtime identity from NVX's @@ -93,23 +116,35 @@ //! - `mapped`: how many of the table's mappings the guest applied; //! - `sandbox`: `started` once the guest started the sandbox; //! - `PROBE.KEY`: the fact KEY that the probe named PROBE found; +//! - `exec.N.KEY`: what execution N did, with these KEYs: +//! - `launch`: `started`, `failed STAGE ERRNO`, with the number of the guest's stage that failed +//! and the error number, or `none` once the guest reported neither; +//! - `status`: how the program ended: `exit CODE`, `signal NUMBER`, or `none`; +//! - `cause`: why a side ended the execution, `cancelled`, `dropped`, `timed-out`, `reset`, or +//! `disconnected`, `other` for a cause that `nvx` adds later, or `none` for an execution that +//! ended on its own; +//! - `truncated`: the streams that lost bytes, `stdout`, `stderr`, or both, separated by a +//! comma, or `none`; +//! - `settled`: `reusable` once the execution settled, or `fenced`; +//! - `stdout` and `stderr`: the program's standard output and error; //! - `stopped`: how the stop ended: `graceful`, `forced`, or `timed-out`; //! - `outcome`: the category and status code of the VM's outcome report, such as `success 0`; //! - `teardown`: whether the VM's teardown completed: `complete` or `incomplete`. //! -//! Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`. +//! Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each byte of a +//! program's output that is not part of UTF-8 text as `\xHH`, with two hexadecimal digits. //! //! For tests, `launch` takes a step hook from the variable `RUNTIME_ADAPTER_STEP_HOOK`, which //! pauses the launch at a named step, for a time that the launch leaves out of its deadline, or //! exits it there without unwinding, with status 5; see `runtime_adapter::hook`. //! -//! They exit with 0 once they did what they were asked, with 3 when reattach finds the VM -//! mismatched, with 4 when the guest reported that a stage of the root failed, and the VM ended -//! with the stage's status and a complete teardown, with 5 when the step hook ends a launch, with -//! 6 when the guest failed a mapping of the table, and the VM stopped gracefully, with success and -//! a complete teardown, and with 1, after a diagnostic on standard error, when they fail. Anything -//! else is a usage error, with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no -//! step hook. +//! They exit with 0 once they did what they were asked, `exec` once every execution finished, +//! whatever its outcome, with 3 when reattach finds the VM mismatched, with 4 when the guest +//! reported that a stage of the root failed, and the VM ended with the stage's status and a +//! complete teardown, with 5 when the step hook ends a launch, with 6 when the guest failed a +//! mapping of the table, and the VM stopped gracefully, with success and a complete teardown, and +//! with 1, after a diagnostic on standard error, when they fail. Anything else is a usage error, +//! with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook. use std::env; use std::error::Error; @@ -118,6 +153,7 @@ use std::path::{Path, PathBuf}; use std::process::ExitCode; use std::time::{Duration, Instant}; +use nvx::exec::{Cause, Key, Limits, Outcome, Status, Stream, Timers, Window}; use nvx::host::artifacts::Artifacts; use nvx::host::instance::generate_capability; use nvx::host::launch::LaunchSpec; @@ -130,9 +166,10 @@ use nvx::protocol::root::ImageFormat; use nvx::uuid::Uuid; use nvx::{ErrorKind, Hypervisor}; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::frame::MAX_BODY; +use runtime_adapter::exec::{self, AccountMode, HostExecution, Start}; +use runtime_adapter::frame::{MAX_BODY, Message}; use runtime_adapter::hook::{Step, StepHook}; -use runtime_adapter::host::{self, Client, ClientError, OrderError}; +use runtime_adapter::host::{self, Client, ClientError, Executions, OrderError}; use runtime_adapter::initramfs; use runtime_adapter::layered::{LaunchRecord, LayeredLaunch}; use runtime_adapter::probe::{Probe, ProbeReply, escape}; @@ -164,6 +201,24 @@ const ROOT_FAILED: u8 = 4; /// The exit status of a launch whose guest failed a mapping of the table, once the VM stopped /// gracefully, with success and a complete teardown. const TABLE_FAILED: u8 = 6; +// The timers and limits of `exec` below are values chosen for tests that make a valid +// configuration, within the limits that the guest and the host allow: chunks of at most +// `exec::MAX_CHUNK` bytes, at most `worker::MAX_EXECUTION_MEMORY` bytes held in the guest for +// each execution, and output windows of at most `host::MAX_OUTPUT_FRAMES` frames. +/// How long an execution of `exec` may run before its sides end it. +const EXEC_TIMEOUT: Duration = Duration::from_secs(60); +/// How long the pending output of an execution of `exec` still flows once a side ended it. +const EXEC_GRACE: Duration = Duration::from_secs(1); +/// How long an execution of `exec` may take to settle, once its program or a side ended it. +const EXEC_SETTLEMENT: Duration = Duration::from_secs(5); +/// The most bytes of a chunk of an execution of `exec`. +const EXEC_CHUNK: u32 = 16 * 1024; +/// The chunks of an execution's input that the guest may hold for `exec`. +const EXEC_INPUT_FRAMES: u32 = 4; +/// The chunks of an execution's output, of both streams, that `exec` may hold. +const EXEC_OUTPUT_FRAMES: u32 = 8; +/// The argument that ends an execution's arguments, after which another execution may follow. +const EXEC_END: &str = ";"; /// The command lines that the program accepts. const USAGE: &str = concat!( "usage: runtime-adapter-host\n", @@ -175,10 +230,15 @@ const USAGE: &str = concat!( " --upper (scratch:PATH | ram) [--runtime-dir DIR]\n", " [--overlay-mode MODE] [--scratch-journal POLICY]", " [--first-partition-bound BYTES]\n", + " [--workload-identity UID:GID]\n", " [--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]...", " [--share-root DIR]...]]\n", " runtime-adapter-host (reattach | stop) STORE [--guest-build BUILD]\n", - " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]", + " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]\n", + " runtime-adapter-host exec STORE [--guest-build BUILD] EXECUTION [';' EXECUTION]...\n", + " EXECUTION: [--identity UID:GID [--groups GID,...] [--capabilities N,...]", + " [--numeric]]\n", + " [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...", ); /// What the program was asked to do. @@ -231,6 +291,73 @@ enum Command { /// The probes, in order. probes: Vec, }, + /// Reattach to the VM, and run executions in its layered root. + Exec { + /// The sandbox's store. + store: PathBuf, + /// The guest agent's build. + guest_build: String, + /// The executions, in order. + executions: Vec, + }, +} + +/// An execution of `exec`, as its options and arguments give it. +#[derive(Default)] +struct Execution { + /// `--identity`: the user ID and the group ID. + identity: Option<(u32, u32)>, + /// `--groups`. + groups: Option>, + /// `--capabilities`: the set, a bit for each number. + capabilities: Option, + /// `--numeric`. + numeric: bool, + /// `--cwd`. + working_directory: Option, + /// `--env`, in order. + environment: Vec, + /// The program's path, then its arguments. + arguments: Vec, +} + +impl Execution { + /// Returns the start of the execution, as the execution `key`. + fn start(&self, key: Key) -> Start { + let arguments = self.arguments.iter().map(String::as_str); + let mut start = Start::new(key, exec_limits(), exec_timers(), arguments) + .with_environment(self.environment.iter().map(String::as_str)); + start.identity = self.identity.map(|(uid, gid)| exec::Identity { + groups: self.groups.clone().unwrap_or_default(), + capabilities: self.capabilities.unwrap_or(0), + account: if self.numeric { + AccountMode::Numeric + } else { + AccountMode::Existing + }, + ..exec::Identity::new(uid, gid) + }); + start.working_directory = self.working_directory.clone().map(String::into_bytes); + start + } +} + +/// Returns the limits of an execution of `exec`: chunks of [`EXEC_CHUNK`] bytes, and frame-counted +/// credit, of [`EXEC_INPUT_FRAMES`] for the input, and of [`EXEC_OUTPUT_FRAMES`] in one pool that +/// standard output and error share. +fn exec_limits() -> Limits { + Limits::new( + EXEC_CHUNK, + Window::frames(EXEC_INPUT_FRAMES), + Window::frames(EXEC_OUTPUT_FRAMES), + ) + .expect("the limits of `exec` hold no empty window, and a chunk that the framing carries") + .shared_output() +} + +/// Returns the timers of both sides of an execution of `exec`. +fn exec_timers() -> Timers { + Timers::new(EXEC_GRACE, EXEC_SETTLEMENT).timeout(EXEC_TIMEOUT) } /// The layered root of a launch, as its options give it. @@ -251,6 +378,8 @@ struct Layered { scratch_journal: Option, /// The most bytes that the first partition of a GPT disk may hold, unless the guest's default. first_partition_bound: Option, + /// The workload identity of the whole sandbox, if any: its user ID and its group ID. + workload_identity: Option<(u32, u32)>, /// The host paths that the launch shares with the root. sharing: HostFsSharing, } @@ -278,6 +407,8 @@ struct Options { scratch_journal: Option, /// `--first-partition-bound`. first_partition_bound: Option, + /// `--workload-identity`. + workload_identity: Option<(u32, u32)>, /// `--share`, `--deny`, `--reserved`, and `--share-root`, in order. sharing: HostFsSharing, } @@ -331,6 +462,11 @@ fn main() -> ExitCode { guest_build, probes, } => probe(&store, &guest_build, &probes), + Command::Exec { + store, + guest_build, + executions, + } => exec(&store, &guest_build, &executions), }; result.unwrap_or_else(|problem| { eprintln!("runtime-adapter-host: {problem}"); @@ -346,7 +482,9 @@ fn parse(arguments: Vec) -> Result { return Ok(Command::Report); }; let name = match name.to_str() { - Some(name @ ("stage" | "launch" | "reattach" | "stop" | "probe")) => name.to_owned(), + Some(name @ ("stage" | "launch" | "reattach" | "stop" | "probe" | "exec")) => { + name.to_owned() + } _ => return Err(format!("unknown command {name:?}")), }; let store = PathBuf::from( @@ -354,6 +492,9 @@ fn parse(arguments: Vec) -> Result { .next() .ok_or_else(|| format!("{name} needs a sandbox store"))?, ); + if name == "exec" { + return parse_exec(store, arguments); + } let mut options = Options::default(); let mut probes = Vec::new(); while let Some(argument) = arguments.next() { @@ -419,6 +560,7 @@ fn parse(arguments: Vec) -> Result { overlay_mode: options.overlay_mode, scratch_journal: options.scratch_journal, first_partition_bound: options.first_partition_bound, + workload_identity: options.workload_identity, sharing: options.sharing, })) } @@ -435,6 +577,7 @@ fn parse(arguments: Vec) -> Result { "--first-partition-bound", options.first_partition_bound.is_some(), ), + ("--workload-identity", options.workload_identity.is_some()), ("--share", !options.sharing.shares().is_empty()), ]; if let Some((option, _)) = layered.iter().find(|(_, given)| *given) { @@ -540,6 +683,17 @@ fn parse_option( format!("--first-partition-bound takes a positive number of bytes, not {value:?}") })?); } + ("launch", "--workload-identity") if options.workload_identity.is_none() => { + let value = text(value)?; + let identity = ids(&value) + .filter(|(uid, gid)| [uid, gid].into_iter().all(|id| (1..u32::MAX).contains(id))); + options.workload_identity = Some(identity.ok_or_else(|| { + format!( + "--workload-identity takes UID:GID, two numbers from 1 to {}, not {value:?}", + u32::MAX - 1 + ) + })?); + } ("launch", "--share") => { let value = text(value)?; let share = Share::parse(&value).ok_or_else(|| { @@ -561,6 +715,136 @@ fn parse_option( Ok(()) } +/// Parses the arguments of `exec` after its sandbox store `store`: its executions, each ended by +/// a lone `;` or the end of the arguments, with the options before them, or says why they are not +/// the command's. +fn parse_exec( + store: PathBuf, + arguments: impl Iterator, +) -> Result { + let mut arguments = arguments.map(|argument| { + argument + .into_string() + .map_err(|argument| format!("the argument {argument:?} of exec is not Unicode")) + }); + let mut guest_build = None; + let mut executions = Vec::new(); + loop { + let mut execution = Execution::default(); + loop { + let option = arguments + .next() + .ok_or("exec needs `-- PROGRAM` after the options of each execution")??; + match option.as_str() { + "--" => break, + "--numeric" if !execution.numeric => { + execution.numeric = true; + continue; + } + _ => {} + } + let value = arguments + .next() + .ok_or_else(|| format!("{option:?} needs a value"))??; + match option.as_str() { + "--guest-build" if guest_build.is_none() => guest_build = Some(value), + "--identity" if execution.identity.is_none() => { + let identity = ids(&value) + .ok_or_else(|| format!("--identity takes UID:GID, not {value:?}"))?; + execution.identity = Some(identity); + } + "--groups" if execution.groups.is_none() => { + let groups = + numbers(&value, |group| group.parse::().ok()).ok_or_else(|| { + format!("--groups takes GIDs, separated by commas, not {value:?}") + })?; + execution.groups = Some(groups); + } + "--capabilities" if execution.capabilities.is_none() => { + let capabilities = numbers(&value, |number| { + number.parse::().ok().filter(|number| *number < 64) + }) + .ok_or_else(|| { + format!( + "--capabilities takes capability numbers from 0 to 63, separated by \ + commas, not {value:?}" + ) + })?; + let set = capabilities + .into_iter() + .fold(0, |set, number| set | 1 << number); + execution.capabilities = Some(set); + } + "--cwd" if execution.working_directory.is_none() => { + execution.working_directory = Some(value); + } + "--env" => execution.environment.push(value), + _ => return Err(format!("unexpected option {option:?} of exec")), + } + } + let mut more = false; + for argument in arguments.by_ref() { + let argument = argument?; + if argument == EXEC_END { + more = true; + break; + } + execution.arguments.push(argument); + } + if execution.arguments.is_empty() { + return Err("exec needs a program after `--`".to_owned()); + } + if execution.identity.is_none() { + let qualifiers = [ + ("--groups", execution.groups.is_some()), + ("--capabilities", execution.capabilities.is_some()), + ("--numeric", execution.numeric), + ]; + if let Some((option, _)) = qualifiers.iter().find(|(_, given)| *given) { + return Err(format!("{option} needs --identity")); + } + } + // The guest ends its session on a start that it cannot decode, so the program refuses + // one that breaks the rules of the encoding, as a relative working directory or an + // environment entry without a name does, or that the framing does not carry, before it + // attaches. + let encoded = execution.start(Key::new(0, 0)).encode(); + Start::decode(&encoded).map_err(|error| error.problem().to_owned())?; + Message::ExecStart(encoded) + .encode() + .map_err(|error| format!("the start of an execution is too long: {error}"))?; + executions.push(execution); + if !more { + break; + } + } + // The driver runs no more executions at once than the guest does. + if executions.len() > exec::MAX_EXECUTIONS { + return Err(format!( + "exec runs at most {} executions, not {}", + exec::MAX_EXECUTIONS, + executions.len() + )); + } + Ok(Command::Exec { + store, + guest_build: guest_build.unwrap_or_else(|| GUEST_BUILD.to_owned()), + executions, + }) +} + +/// Returns the user ID and the group ID of `value`, `UID:GID`, or `None` if it is not that. +fn ids(value: &str) -> Option<(u32, u32)> { + let (uid, gid) = value.split_once(':')?; + Some((uid.parse().ok()?, gid.parse().ok()?)) +} + +/// Returns the numbers of `value`, separated by commas, which `parse` reads, or `None` if one is +/// not a number that it accepts, or `value` is empty. +fn numbers(value: &str, parse: impl Fn(&str) -> Option) -> Option> { + value.split(',').map(parse).collect() +} + /// Stages the image `distro`, in `format`, of which the guest expects `identity`, and the ext4 /// scratch image `scratch`, if any, in the sandbox store `store`, which it creates if needed. fn stage( @@ -616,6 +900,7 @@ fn launch( launch.config.overlay_mode = layered.overlay_mode; launch.config.scratch_journal = layered.scratch_journal; launch.config.first_partition_bound = layered.first_partition_bound; + launch.config.workload_identity = layered.workload_identity; // The launch refuses shares that it cannot export before it touches the store. if let Some(export) = layered.sharing.plan()? { launch.share(export)?; @@ -970,6 +1255,125 @@ fn probe(store: &Path, guest_build: &str, probes: &[Probe]) -> Result Result { + let runtime = runtime(guest_build)?; + let store = SandboxStore::open(store).map_err(|error| describe(&error))?; + let mut locked = store + .lock(Some(Instant::now() + LOCK_TIMEOUT)) + .map_err(|error| describe(&error))?; + let vmm = match hold(&mut locked, &runtime)? { + Ok(vmm) => vmm, + Err(status) => return Ok(status), + }; + let deadline = Instant::now() + SESSION_TIMEOUT; + let capability = store.read_capability().map_err(|error| describe(&error))?; + let mut client = connect(&vmm, &capability, deadline)?; + // The keys of the attachment's executions are new to the guest: see the module documentation. + let identity = client.identity(); + let (base, _) = identity.instance_id.split_at(8); + let base = u64::from_le_bytes(base.try_into().expect("an instance ID has 16 bytes")); + let mut running = Executions::new(); + let mut ids = Vec::new(); + for (number, execution) in (1..).zip(executions) { + let key = Key::new(base.wrapping_add(number), identity.epoch); + running + .start(&mut client, execution.start(key), exec_timers(), deadline) + .map_err(|error| format!("cannot start execution {number}: {}", describe(&error)))?; + ids.push(key.id()); + } + // Every execution finishes once it ended, which its timeout bounds, and its output's grace and + // its settlement passed, or once the guest's last messages about it arrived. + let deadline = Instant::now() + EXEC_TIMEOUT + EXEC_GRACE + EXEC_SETTLEMENT + SESSION_TIMEOUT; + running + .run(&mut client, deadline) + .map_err(|error| format!("the executions failed: {}", describe(&error)))?; + for (number, id) in (1..).zip(ids) { + let execution = running.get(id).expect("the driver started every execution"); + print_execution(number, execution); + } + Ok(ExitCode::SUCCESS) +} + +/// Prints what the execution numbered `number`, `execution`, which finished, did. +fn print_execution(number: u64, execution: &HostExecution) { + let Some(outcome) = execution.outcome() else { + unreachable!("every execution finished"); + }; + let launch = match outcome.launch_failure() { + Some(failure) => format!("failed {} {}", failure.stage(), failure.error()), + None if execution.peer().has_started() => "started".to_owned(), + None => "none".to_owned(), + }; + let status = match outcome.status() { + Some(Status::Exited(code)) => format!("exit {code}"), + Some(Status::Signaled(signal)) => format!("signal {signal}"), + None => "none".to_owned(), + }; + let truncated = Stream::ALL + .into_iter() + .filter(|stream| outcome.is_truncated(*stream)) + .map(stream_name) + .collect::>(); + let settled = if outcome.is_reusable() { + "reusable" + } else { + "fenced" + }; + println!("exec.{number}.launch={launch}"); + println!("exec.{number}.status={status}"); + println!("exec.{number}.cause={}", cause_name(&outcome)); + println!( + "exec.{number}.truncated={}", + if truncated.is_empty() { + "none".to_owned() + } else { + truncated.join(",") + } + ); + println!("exec.{number}.settled={settled}"); + for stream in Stream::ALL { + let output = escape_bytes(execution.output(stream)); + println!("exec.{number}.{}={output}", stream_name(stream)); + } +} + +/// Returns the name of why a side ended the execution whose outcome is `outcome`, or `none`. +fn cause_name(outcome: &Outcome) -> &'static str { + match outcome.cause() { + None => "none", + Some(Cause::Cancelled) => "cancelled", + Some(Cause::Dropped) => "dropped", + Some(Cause::TimedOut) => "timed-out", + Some(Cause::Reset) => "reset", + Some(Cause::Disconnected) => "disconnected", + Some(_) => "other", + } +} + +/// Returns the name of `stream`: `stdout` or `stderr`. +fn stream_name(stream: Stream) -> &'static str { + match stream { + Stream::Stdout => "stdout", + Stream::Stderr => "stderr", + } +} + +/// Returns `bytes` as a value: the UTF-8 text that they hold, escaped as [`escape`] escapes it, and +/// each other byte as `\xHH`, with two hexadecimal digits. +fn escape_bytes(bytes: &[u8]) -> String { + let mut escaped = String::with_capacity(bytes.len()); + for chunk in bytes.utf8_chunks() { + escaped.push_str(&escape(chunk.valid())); + for byte in chunk.invalid() { + escaped.push_str(&format!("\\x{byte:02x}")); + } + } + escaped +} + /// Reattaches to the VM of `locked`, with the runtime identity of `runtime`, and prints what it /// found. Returns the VM, or the status with which the command ends once it finds the VM /// mismatched, which it leaves running. diff --git a/tests/runtime-adapter/src/exec.rs b/tests/runtime-adapter/src/exec.rs index 083061453..badf98a95 100644 --- a/tests/runtime-adapter/src/exec.rs +++ b/tests/runtime-adapter/src/exec.rs @@ -1,7 +1,9 @@ //! The adapter's messages about executions: programs that the guest runs for the host, whose //! standard streams and outcome the execution engine of `nvx::exec` carries. The host's side of an //! execution is the engine's [`Peer`], and the guest's side its [`Runner`], which the guest agent's -//! worker binds to the program through `nvx::guest::exec::binding`. +//! worker binds to the program through `nvx::guest::exec::binding`. A [`HostExecution`] holds the +//! peer with a sink of the program's output and a source of its input, as the host program's +//! driver, `host::Executions`, and the in-process tests of the worker run it. //! //! Each message of the engine travels in a frame of its own kind. Its body starts with the //! execution's [`Key`]: the execution's ID, then its generation, each a `u64`: @@ -71,6 +73,14 @@ pub const KEY_LEN: usize = 16; /// longest body holds after the key and the stream. pub const MAX_CHUNK: u32 = (MAX_BODY - KEY_LEN - 1) as u32; +/// The most executions that the guest's worker runs at once, beyond which the host's driver +/// starts none while those that it started have not finished. +pub const MAX_EXECUTIONS: usize = 8; + +/// The most bytes of each output stream of an execution that a [`HostExecution`] keeps: it +/// cancels an execution whose stream would exceed them, so that its host holds no more. +pub const MAX_CAPTURE: usize = 16 * 1024 * 1024; + /// The code of a cause that the framing does not know, which no side decodes. const UNKNOWN_CAUSE: u8 = 0xff; @@ -575,6 +585,173 @@ impl<'a> GuestMessage<'a> { } } +/// The host's side of an execution: its [`Peer`], with a sink that takes the program's output as +/// it arrives, unless its caller stalls it, and a source that hands over the program's input, if +/// the start says that the host sends it, then ends it. +/// +/// The sink keeps at most [`MAX_CAPTURE`] bytes of each stream. Once more of a stream arrives, the +/// host's side cancels the execution, and its peer discards the rest once the grace ends, so that +/// the outcome marks the stream truncated, with the cause [`Cause::Cancelled`]. +/// +/// Its caller sends the execution's [`Start`], hands it each of the guest's messages about the +/// execution with [`deliver`](Self::deliver), and sends the frames that [`serve`](Self::serve) +/// returns, until the peer [finished](Self::outcome). The host program's driver, +/// `host::Executions`, runs several at once over the session. +#[derive(Debug)] +pub struct HostExecution { + /// The execution's start. + start: Start, + /// The host's side of the execution. + peer: Peer, + /// Whether the sink takes nothing. + stalled: bool, + /// The bytes of each stream that the sink took. + output: [Vec; 2], + /// The input that the source hands over, if the start says that the host sends it. + input: Vec, + /// How many bytes of the input the source handed over. + sent: usize, + /// Whether the source ended the input. + ended: bool, + /// The peer's events, in order. + events: Vec, + /// Whether a stream outgrew [`MAX_CAPTURE`], so that the host's side cancelled the execution. + overflowed: bool, +} + +impl HostExecution { + /// Returns the host's side of the execution of `start`, created at `now`, from which the + /// timeout of its peer's `timers` runs, with a source of `input`. + pub fn new(start: Start, timers: Timers, input: Vec, now: Instant) -> Self { + let peer = Peer::new(start.key, start.limits, timers, now); + Self { + start, + peer, + stalled: false, + output: [Vec::new(), Vec::new()], + input, + sent: 0, + ended: false, + events: Vec::new(), + overflowed: false, + } + } + + /// Returns the execution's start. + pub fn start(&self) -> &Start { + &self.start + } + + /// Returns the host's side of the execution. + pub fn peer(&self) -> &Peer { + &self.peer + } + + /// Returns the host's side of the execution, with which the caller ends it, or tells it that + /// the session was reset or lost. + pub fn peer_mut(&mut self) -> &mut Peer { + &mut self.peer + } + + /// Makes the sink take nothing while `stalled` holds, so that the program's output waits in + /// the peer, which grants the guest no credit for it. + pub fn stall(&mut self, stalled: bool) { + self.stalled = stalled; + } + + /// Returns the bytes of `stream` that the sink took, at most [`MAX_CAPTURE`]. + pub fn output(&self, stream: Stream) -> &[u8] { + &self.output[position(stream)] + } + + /// Returns the peer's events, in order. + pub fn events(&self) -> &[peer::Event] { + &self.events + } + + /// Returns the execution's outcome, once the peer finished. + pub fn outcome(&self) -> Option { + self.events.iter().find_map(|event| match event { + peer::Event::Finished(outcome) => Some(*outcome), + _ => None, + }) + } + + /// Returns when the peer's next timer fires, for the caller to [serve](Self::serve) the + /// execution then, or `None` while no timer runs. + pub fn deadline(&self) -> Option { + self.peer.deadline() + } + + /// Hands the guest's `message` about the execution to the peer at `now`, and returns what the + /// peer returns. + pub fn deliver( + &mut self, + message: &GuestMessage<'_>, + now: Instant, + ) -> Result { + message.deliver(&mut self.peer, now) + } + + /// Takes every step that the source, the sink, and the peer can take at `now`, and returns + /// the peer's messages for the guest, as frames, in order, the cancellation of an execution + /// that the peer gave up on in this step included. + pub fn serve(&mut self, now: Instant) -> Vec { + if self.start.input && !self.ended { + self.sent += self.peer.input(&self.input[self.sent..]); + if self.sent == self.input.len() { + self.peer.end_input(); + self.ended = true; + } + } + if !self.stalled { + for stream in Stream::ALL { + let output = &mut self.output[position(stream)]; + loop { + let pending = self.peer.pending_output(stream); + if pending.is_empty() { + break; + } + let room = MAX_CAPTURE - output.len(); + if room == 0 { + if !self.overflowed { + self.overflowed = true; + self.peer.cancel(now); + } + break; + } + let length = pending.len().min(room); + output.extend_from_slice(&pending[..length]); + self.peer.consumed(stream, length); + } + } + } + self.peer.tick(now); + // The messages go first: a peer without grace or settlement time gives up on an + // execution in the step that ends it, and its last event, once taken, discards the + // cancellation that waits to be sent. + let mut buffer = vec![0; self.start.limits.chunk()]; + let mut messages = Vec::new(); + while let Some(message) = self.peer.next_message(&mut buffer) { + if let Some(message) = HostMessage::from_peer(self.start.key, message, &buffer) { + messages.push(message.encode()); + } + } + while let Some(event) = self.peer.next_event() { + self.events.push(event); + } + messages + } +} + +/// Returns the position of `stream` among [`Stream::ALL`]. +fn position(stream: Stream) -> usize { + match stream { + Stream::Stdout => 0, + Stream::Stderr => 1, + } +} + /// Returns `key` as a name for problems: its ID, then its generation. pub fn name(key: Key) -> String { format!("{}.{}", key.id(), key.generation()) diff --git a/tests/runtime-adapter/src/exec/tests.rs b/tests/runtime-adapter/src/exec/tests.rs index db112acfb..24555130d 100644 --- a/tests/runtime-adapter/src/exec/tests.rs +++ b/tests/runtime-adapter/src/exec/tests.rs @@ -1,5 +1,7 @@ //! The encodings of the execution messages: round trips, the layout of a start, the bodies that -//! break the rules, and a peer and a runner that exchange their messages through the frames. +//! break the rules, and a peer and a runner that exchange their messages through the frames; and +//! the host's side of an execution, which sends the cancellation of one that it gives up on at +//! once, and cancels one whose stream outgrows what it keeps. use std::time::{Duration, Instant}; @@ -8,7 +10,10 @@ use nvx::exec::{ Timers, Window, peer, runner, }; -use super::{AccountMode, GuestMessage, HostMessage, Identity, KEY_LEN, MAX_CHUNK, Start, name}; +use super::{ + AccountMode, GuestMessage, HostExecution, HostMessage, Identity, KEY_LEN, MAX_CAPTURE, + MAX_CHUNK, Start, name, +}; use crate::frame::{MAX_BODY, Message}; /// The key of most executions in these tests. @@ -655,3 +660,96 @@ fn a_peer_and_a_runner_exchange_their_messages_through_the_frames() { .collect::>(); assert_eq!(interleaved, input); } + +#[test] +fn the_host_s_side_sends_the_cancellation_of_an_execution_that_it_gives_up_on_at_once() { + // Without grace or settlement time, the host's side gives up on an execution in the step that + // ends it, for its timeout or its caller's cancellation, and still sends the cancellation. + let now = Instant::now(); + let immediate = Timers::new(Duration::ZERO, Duration::ZERO); + let timeout = Duration::from_secs(1); + for (timers, cause) in [ + (immediate.timeout(timeout), Cause::TimedOut), + (immediate, Cause::Cancelled), + ] { + let start = Start::new(KEY, shared(), timers, ["/bin/program"]); + let mut execution = HostExecution::new(start, timers, Vec::new(), now); + // The credit that opens the output's pool. + assert_eq!(execution.serve(now).len(), 1, "{cause:?}"); + assert!(execution.outcome().is_none(), "{cause:?}"); + let later = now + timeout; + if cause == Cause::Cancelled { + execution.peer_mut().cancel(later); + } + assert_eq!( + execution.serve(later), + [HostMessage::Cancel(KEY, cause).encode()], + "{cause:?}" + ); + let outcome = execution.outcome().expect("the host's side gave up"); + assert_eq!((outcome.cause(), outcome.status()), (Some(cause), None)); + assert!(!outcome.is_reusable(), "{cause:?}"); + assert!(execution.serve(later).is_empty(), "{cause:?}"); + } +} + +#[test] +fn the_host_s_side_keeps_at_most_max_capture_bytes_of_a_stream_and_cancels_the_rest() { + // A program that writes a chunk to its standard output at each step, without end: the host's + // side keeps the first MAX_CAPTURE bytes, then cancels the execution, and its peer, without + // grace, discards the rest at once. The runner takes and drops what the program writes once + // it discards the stream, so each step writes once. + let now = Instant::now(); + let timers = Timers::new(Duration::ZERO, Duration::from_secs(5)); + let start = Start::new(KEY, shared(), timers, ["/usr/bin/yes"]); + let mut host = HostExecution::new(start, timers, Vec::new(), now); + let mut runner = Runner::new(KEY, shared(), timers, now); + runner.launched(Ok(())); + let written = vec![b'y'; shared().chunk()]; + let mut buffer = vec![0u8; shared().chunk()]; + let mut running = true; + for _ in 0..2 * MAX_CAPTURE / written.len() { + if host.outcome().is_some() { + break; + } + for frame in host.serve(now) { + match HostMessage::decode(&frame).unwrap().unwrap() { + HostMessage::Credit(key, pool, units) => runner.credit(key, pool, units), + HostMessage::Cancel(key, cause) => runner.cancel(now, key, cause), + message => panic!("{message:?}"), + } + .unwrap(); + } + if running { + runner.output(Stream::Stdout, &written); + } + while let Some(request) = runner.next_request() { + match request { + runner::Request::Terminate(_) => { + running = false; + for stream in Stream::ALL { + runner.output_closed(stream); + } + runner.exited(now, Some(Status::Signaled(15))); + } + runner::Request::Settle => runner.settled(Settlement::Settled), + _ => {} + } + } + while let Some(message) = runner.next_message(&mut buffer) { + let frame = GuestMessage::from_runner(KEY, message, &buffer) + .unwrap() + .encode(); + let decoded = GuestMessage::decode(&frame).unwrap().unwrap(); + host.deliver(&decoded, now).unwrap(); + } + } + let outcome = host.outcome().expect("the execution finished"); + let kept = host.output(Stream::Stdout); + assert_eq!(kept.len(), MAX_CAPTURE); + assert!(kept.iter().all(|byte| *byte == b'y')); + assert!(host.output(Stream::Stderr).is_empty()); + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert!(outcome.is_truncated(Stream::Stdout)); + assert!(!outcome.is_truncated(Stream::Stderr)); +} diff --git a/tests/runtime-adapter/src/host.rs b/tests/runtime-adapter/src/host.rs index 84ba3103a..11cb1211c 100644 --- a/tests/runtime-adapter/src/host.rs +++ b/tests/runtime-adapter/src/host.rs @@ -1,7 +1,8 @@ //! The adapter's host program: a client of the guest agent that attaches to the host's end of the //! control session between the host and the guest, and exchanges the adapter's own //! [framing](crate::frame) with the guest agent, through `nvx::host::nvxs` alone, over any platform -//! transport. +//! transport, and the driver of the [executions](crate::exec) that the host program runs over it, +//! [`Executions`]. //! //! # The host's order //! @@ -22,6 +23,7 @@ //! waits for the VM to end, and ends a VM whose guest did not stop, as it ends one after any other //! failure of the order, an [`OrderError`]. +mod executions; mod map_reply; mod order_error; @@ -37,6 +39,7 @@ use crate::frame::{BodyError, Decoder, FrameError, Message}; use crate::probe::{Facts, Probe, ProbeReply}; use crate::root::{MapFailure, Ready, RootConfig, RootFailure, RootReply, encode_maps}; +pub use executions::{Executions, ExecutionsError, MAX_OUTPUT_FRAMES}; pub use map_reply::MapReply; pub use order_error::OrderError; @@ -117,7 +120,7 @@ impl From for ClientError { /// returns the guest agent's next message, whether it spans records or shares one with the next /// message. [`ping`](Self::ping), [`echo`](Self::echo), [`configure_root`](Self::configure_root), /// [`map`](Self::map), [`start`](Self::start), [`probe`](Self::probe), and [`stop`](Self::stop) -/// each send a request and wait for its answer. +/// each send a request and wait for its answer, and [`Executions`] runs executions over the client. /// /// Besides the session's own buffers, the client holds at most the rest of one of the guest's /// records and one partial message, whose header [`Decoder`] checks before it takes any byte of @@ -208,7 +211,11 @@ impl Client { | Message::RootConfig(_) | Message::Probe(_) | Message::Maps(_) - | Message::Start), + | Message::Start + | Message::ExecStart(_) + | Message::ExecInput(_) + | Message::ExecInputEnd(_) + | Message::ExecCancel(_)), ) => { return Err(self.fail(ClientError::Unexpected(message.kind()))); } diff --git a/tests/runtime-adapter/src/host/executions.rs b/tests/runtime-adapter/src/host/executions.rs new file mode 100644 index 000000000..bfd269eae --- /dev/null +++ b/tests/runtime-adapter/src/host/executions.rs @@ -0,0 +1,280 @@ +//! The host program's driver of executions: the host's side of each, a [`HostExecution`], over one +//! attachment of a [`Client`] to the control session. + +use std::collections::BTreeMap; +use std::fmt; +use std::time::Instant; + +use nvx::exec::{Key, Timers, Unit, Violation}; +use nvx::host::nvxs::HostConnectionError; + +use super::{Client, ClientError}; +use crate::exec::{GuestMessage, HostExecution, MAX_EXECUTIONS, Start, name}; +use crate::frame::{BodyError, Message}; + +/// The most frames of each output pool of an execution of [`Executions`], whose output windows +/// count frames: then the credits that the driver sends for the messages that the guest of +/// [`MAX_EXECUTIONS`] executions may send ahead of its own credit fit the guest's receive window, +/// `nvx::protocol::OUTER_MIN_RECEIVE_WINDOW`, several times over. +pub const MAX_OUTPUT_FRAMES: u32 = 32; + +/// A failure of [`Executions`]. +#[derive(Debug)] +#[non_exhaustive] +pub enum ExecutionsError { + /// The client failed. + Client(ClientError), + /// A start whose encoding does not decode, which the driver did not send. + Start(BodyError), + /// A start of an execution whose ID the driver started already, which it did not send. + Duplicate(u64), + /// A start that says that the host sends the program's input, which the driver does not + /// send, and so did not send the start. + Input, + /// A start whose output windows count bytes, or more than [`MAX_OUTPUT_FRAMES`] frames, + /// which the driver did not send. + Window, + /// A start beyond [`MAX_EXECUTIONS`] executions of the driver, which it did not send. + Full, + /// A start once the driver ran its executions, which it did not send. + Ran, + /// The guest sent a message about an execution whose body does not decode. + Body(BodyError), + /// The guest sent a message of this kind, which is about no execution. + Unexpected(u8), + /// The guest sent a message about an execution that the driver did not start. + Unknown(Key), + /// The host's side of the execution refused the guest's message about it. + Violation(Key, Violation), + /// The deadline passed before every execution finished. + TimedOut, +} + +impl fmt::Display for ExecutionsError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Client(error) => error.fmt(formatter), + Self::Start(error) => write!(formatter, "the host refuses to send the start: {error}"), + Self::Duplicate(id) => write!(formatter, "the host starts execution {id} twice"), + Self::Input => formatter.write_str("the host sends no execution's input"), + Self::Window => write!( + formatter, + "the host takes output windows of at most {MAX_OUTPUT_FRAMES} frames" + ), + Self::Full => write!( + formatter, + "the host runs at most {MAX_EXECUTIONS} executions at once" + ), + Self::Ran => { + formatter.write_str("the host starts no execution once it ran its executions") + } + Self::Body(error) => error.fmt(formatter), + Self::Unexpected(kind) => write!( + formatter, + "the guest sent a message of kind {kind}, which is about no execution" + ), + Self::Unknown(key) => write!( + formatter, + "the guest sent a message about execution {}, which the host did not start", + name(*key) + ), + Self::Violation(key, violation) => write!( + formatter, + "the guest's message about execution {} breaks the engine's rules: {violation}", + name(*key) + ), + Self::TimedOut => formatter.write_str("the executions did not finish in time"), + } + } +} + +impl std::error::Error for ExecutionsError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Client(error) => Some(error), + Self::Start(error) | Self::Body(error) => Some(error), + Self::Violation(_, violation) => Some(violation), + Self::Duplicate(_) + | Self::Input + | Self::Window + | Self::Full + | Self::Ran + | Self::Unexpected(_) + | Self::Unknown(_) + | Self::TimedOut => None, + } + } +} + +impl From for ExecutionsError { + fn from(error: ClientError) -> Self { + Self::Client(error) + } +} + +/// The host's side of the executions that the host program runs over one attachment of the control +/// session: a [`HostExecution`] for each, by ID, whose sink takes the program's output as it +/// arrives. +/// +/// [`start`](Self::start) sends the start of an execution, and [`run`](Self::run) carries the +/// guest's messages to the executions, and theirs to the guest, until every execution finished. +/// A driver runs one batch: it starts every execution before it runs them, and none after. The +/// guest may still send messages about the executions of a run that gave up on one or failed, so +/// a later batch takes an attachment of its own, as each command of the host program does. +/// +/// One thread runs them all: each receive waits at most until the earliest timer of the +/// executions, which so fires in time whatever the guest sends, and the guest's messages about one +/// execution never wait for another's. Each send waits, as the client's do, until the transport +/// takes the message or the run's deadline passes, and no timer fires meanwhile, so a guest that +/// stops reading the session holds the run until its deadline, when the send's timeout ends the +/// client. +/// +/// Even a guest that reads can hold a send while the host does not read: the session's broker +/// passes the guest's credit for the host's records on only behind the guest's own records, and +/// takes no more of the host's records than that credit allows. So the driver sends little before +/// it reads: no program's input, which one thread could not send while it reads the programs' +/// output; the starts of at most [`MAX_EXECUTIONS`] executions, as many as the guest runs at once, +/// whose answers wait for the run, as no program's output flows before the run's credit does; +/// and, in the run, besides a cancellation for each execution, at most a credit for each message +/// that it reads, of output windows of at most [`MAX_OUTPUT_FRAMES`] frames, so that the guest's +/// messages ahead of its own credit stay few. +/// +/// The guest ends its session on a second start of the same key, even in a later epoch of the +/// session, so the keys of the executions must differ from those of every execution that the guest +/// heard of before. A failure ends the run, and leaves the client as its own failures do. +#[derive(Debug, Default)] +pub struct Executions { + /// The host's side of each execution, by ID. + executions: BTreeMap, + /// Whether the driver ran its executions. + ran: bool, +} + +impl Executions { + /// Returns a driver without executions. + pub fn new() -> Self { + Self::default() + } + + /// Starts the execution of `start` over `client`, whose peer runs with `timers`: sends the + /// start by `deadline`. The timeout of `timers` runs from then. + /// + /// These starts fail before anything is sent, and leave the client usable, as a message that + /// the framing refuses does: one of an ID that the driver started already, or whose encoding + /// does not decode, as one with a relative working directory or an environment entry without + /// a name, on either of which the guest would end its session; and, as [`Executions`] + /// describes, one that says that the host sends the program's input, one whose output windows + /// count bytes or more than [`MAX_OUTPUT_FRAMES`] frames, one beyond [`MAX_EXECUTIONS`] + /// executions, and any once the driver ran. + pub fn start( + &mut self, + client: &mut Client, + start: Start, + timers: Timers, + deadline: Instant, + ) -> Result<(), ExecutionsError> { + if self.ran { + return Err(ExecutionsError::Ran); + } + let id = start.key.id(); + if self.executions.contains_key(&id) { + return Err(ExecutionsError::Duplicate(id)); + } + if start.input { + return Err(ExecutionsError::Input); + } + let output = start.limits.output(); + if !matches!(output.unit(), Unit::Frames) || output.size() > MAX_OUTPUT_FRAMES { + return Err(ExecutionsError::Window); + } + if self.executions.len() >= MAX_EXECUTIONS { + return Err(ExecutionsError::Full); + } + let body = start.encode(); + Start::decode(&body).map_err(ExecutionsError::Start)?; + let execution = HostExecution::new(start, timers, Vec::new(), Instant::now()); + client.send(&Message::ExecStart(body), Some(deadline))?; + self.executions.insert(id, execution); + Ok(()) + } + + /// Carries the guest's messages to the executions, and theirs to the guest, over `client`, + /// until every execution finished, and fails with [`ExecutionsError::TimedOut`] once + /// `deadline` passed first: in a receive, which leaves the client usable, or in a send, which + /// ends it. + pub fn run(&mut self, client: &mut Client, deadline: Instant) -> Result<(), ExecutionsError> { + self.ran = true; + loop { + let now = Instant::now(); + for execution in self.executions.values_mut() { + for message in execution.serve(now) { + client.send(&message, Some(deadline)).map_err(failed)?; + } + } + if self.is_finished() { + return Ok(()); + } + if now >= deadline { + return Err(ExecutionsError::TimedOut); + } + let wake = self + .executions + .values() + .filter_map(HostExecution::deadline) + .fold(deadline, Instant::min); + match client.receive(Some(wake)) { + Ok(message) => self.dispatch(&message, Instant::now())?, + Err(ClientError::Session(HostConnectionError::TimedOut)) => {} + Err(error) => return Err(error.into()), + } + } + } + + /// Returns the host's side of the execution `id`, if the driver started it. + pub fn get(&self, id: u64) -> Option<&HostExecution> { + self.executions.get(&id) + } + + /// Returns the host's side of the execution `id`, if the driver started it, with which the + /// caller may end it. + pub fn get_mut(&mut self, id: u64) -> Option<&mut HostExecution> { + self.executions.get_mut(&id) + } + + /// Returns the host's side of each execution, in the order of their IDs. + pub fn iter(&self) -> impl Iterator { + self.executions.values() + } + + /// Returns whether every execution finished. + pub fn is_finished(&self) -> bool { + self.executions + .values() + .all(|execution| execution.outcome().is_some()) + } + + /// Hands the guest's `message` to the host's side of its execution at `now`. + fn dispatch(&mut self, message: &Message, now: Instant) -> Result<(), ExecutionsError> { + let decoded = GuestMessage::decode(message) + .ok_or(ExecutionsError::Unexpected(message.kind()))? + .map_err(ExecutionsError::Body)?; + let key = decoded.key(); + let execution = self + .executions + .get_mut(&key.id()) + .ok_or(ExecutionsError::Unknown(key))?; + execution + .deliver(&decoded, now) + .map_err(|violation| ExecutionsError::Violation(key, violation))?; + Ok(()) + } +} + +/// Returns the failure of a run whose send failed with `error`: [`ExecutionsError::TimedOut`] +/// once the send's deadline, the run's, passed. +fn failed(error: ClientError) -> ExecutionsError { + match error { + ClientError::Session(HostConnectionError::TimedOut) => ExecutionsError::TimedOut, + error => ExecutionsError::Client(error), + } +} diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index c2db2b503..9f92c0923 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -247,8 +247,9 @@ pub const EXECUTING: &str = "the worker runs executions, and probes nothing unti /// The most executions that the worker runs at once, those that the host abandoned included, and /// those that ended but keep their places: with cgroups, those whose cgroups the worker has not /// removed, as those of the executions that did not settle, and without them, those whose -/// programs the worker forgot before it collected their statuses. -pub const MAX_EXECUTIONS: usize = 8; +/// programs the worker forgot before it collected their statuses. The host's side knows it as +/// [`exec::MAX_EXECUTIONS`](crate::exec::MAX_EXECUTIONS). +pub const MAX_EXECUTIONS: usize = crate::exec::MAX_EXECUTIONS; /// The most bytes that one execution may hold in the worker: what its runner holds, a chunk of /// each output stream and the input's window. The output that waits for the session's stream diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 556bd7fb7..1382fabb1 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -22,21 +22,21 @@ //! signals leave, whatever the names of its children. //! //! The worker's executions run behind the test broker too, whose host side drives the engine's -//! peers: the tests cover several executions at once, byte-exact, a stalled sink and a full -//! stream to the host, which stall only what waits on them, executions that take turns at the -//! stream, timeouts, cancellations, and stops, the starts, messages, and probes that the worker -//! refuses, generations and resets, the descendants that a program leaves in its process group, -//! and executions on a tty. The fake root prepares each execution's entry, as the worker -//! describes it, in a directory of the tests' own that holds account databases, and then spawns -//! the program without entering a root, which needs privileges, in the tests' own root and -//! identity: the tests cover the identity that exactly one source names, the sandbox's or the -//! start's, the entry's inputs, a program's working directory and environment, and the starts -//! that the worker refuses for their identities. Where the tests can create cgroups, each fake -//! root gives the worker a workload cgroup of the test's own, below which the executions run in -//! cgroups of their own, and settle: the tests then also cover a descendant that leaves its -//! program's process group, which the settlement ends, and an execution that does not settle, -//! which keeps its place until the host stops the worker. Elsewhere, the programs run in process -//! groups alone. +//! peers through the host program's own `exec::HostExecution`: the tests cover several executions +//! at once, byte-exact, a stalled sink and a full stream to the host, which stall only what waits +//! on them, executions that take turns at the stream, timeouts, cancellations, and stops, the +//! starts, messages, and probes that the worker refuses, generations and resets, the descendants +//! that a program leaves in its process group, and executions on a tty. The fake root prepares +//! each execution's entry, as the worker describes it, in a directory of the tests' own that holds +//! account databases, and then spawns the program without entering a root, which needs +//! privileges, in the tests' own root and identity: the tests cover the identity that exactly one +//! source names, the sandbox's or the start's, the entry's inputs, a program's working directory +//! and environment, and the starts that the worker refuses for their identities. Where the tests +//! can create cgroups, each fake root gives the worker a workload cgroup of the test's own, below +//! which the executions run in cgroups of their own, and settle: the tests then also cover a +//! descendant that leaves its program's process group, which the settlement ends, and an execution +//! that does not settle, which keeps its place until the host stops the worker. Elsewhere, the +//! programs run in process groups alone. #![cfg(all(feature = "guest", target_os = "linux"))] use std::collections::BTreeMap; @@ -61,9 +61,7 @@ use nix::pty; use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, Id, WaitPidFlag, WaitStatus}; use nix::unistd::Pid; -use nvx::exec::{ - Cause, Key, LaunchFailure, Limits, Outcome, Peer, Pool, Status, Stream, Timers, Window, peer, -}; +use nvx::exec::{Cause, Key, LaunchFailure, Limits, Outcome, Pool, Status, Stream, Timers, Window}; use nvx::guest::cgroup::Layout; use nvx::guest::entry::{self, Capabilities, Capability, Prepared}; use nvx::guest::mappings::{Access, Mapping}; @@ -79,7 +77,9 @@ use nvx::protocol::{ }; use nvx::uuid::Uuid; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::exec::{self, AccountMode, GuestMessage, HostMessage, MAX_CHUNK, Start}; +use runtime_adapter::exec::{ + self, AccountMode, GuestMessage, HostExecution, HostMessage, MAX_CHUNK, Start, +}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, Message}; use runtime_adapter::guest::{self, NO_ROOT, RECEIVE_WINDOW, ServeError, Served}; use runtime_adapter::probe::{Facts, Probe}; @@ -2649,14 +2649,6 @@ fn lines(first: usize, last: usize) -> Vec { .into_bytes() } -/// Returns `stream`'s position in [`Stream::ALL`]. -fn position(stream: Stream) -> usize { - match stream { - Stream::Stdout => 0, - Stream::Stderr => 1, - } -} - /// Waits until no process of the process group `group` runs, as once an execution ended the /// program that leads it. fn gone(group: u32) { @@ -2699,93 +2691,24 @@ fn reaped(id: u32) { } } -/// The host's side of one of the worker's executions in a test: its peer, with a sink that takes -/// the output unless it stalls, and a source that hands over the input, then ends it. -struct HostExecution { - /// The execution's start. - start: Start, - /// The host's side of the execution. - peer: Peer, - /// Whether the sink takes nothing. - stalled: bool, - /// The bytes of each stream that the sink took. - output: [Vec; 2], - /// The input that the source hands over, if the start says that the host sends it. - input: Vec, - /// How many bytes of the input the source handed over. - sent: usize, - /// Whether the source ended the input. - ended: bool, - /// The peer's events, in order. - events: Vec, -} - -impl HostExecution { - /// Returns the host's side of the execution of `start`, created at `now`, with a source of - /// `input`. - fn new(start: Start, input: Vec, now: Instant) -> Self { - let peer = Peer::new(start.key, start.limits, timers(), now); - Self { - start, - peer, - stalled: false, - output: [Vec::new(), Vec::new()], - input, - sent: 0, - ended: false, - events: Vec::new(), - } - } +/// What the tests read of the host's side of one of the worker's executions, a +/// [`HostExecution`], besides what it tells itself. +trait Taken { + /// Returns the bytes of each stream that the sink took, in the order of [`Stream::ALL`]. + fn outputs(&self) -> [Vec; 2]; - /// Takes every step that the source, the sink, and the peer can take at `now`, and returns - /// the peer's messages for the guest. - fn serve(&mut self, now: Instant) -> Vec { - if self.start.input && !self.ended { - self.sent += self.peer.input(&self.input[self.sent..]); - if self.sent == self.input.len() { - self.peer.end_input(); - self.ended = true; - } - } - if !self.stalled { - for stream in Stream::ALL { - loop { - let pending = self.peer.pending_output(stream); - if pending.is_empty() { - break; - } - let length = pending.len(); - self.output[position(stream)].extend_from_slice(pending); - self.peer.consumed(stream, length); - } - } - } - self.peer.tick(now); - while let Some(event) = self.peer.next_event() { - self.events.push(event); - } - let mut buffer = vec![0; self.start.limits.chunk()]; - let mut messages = Vec::new(); - while let Some(message) = self.peer.next_message(&mut buffer) { - let message = HostMessage::from_peer(self.start.key, message, &buffer) - .expect("the framing carries every message of the peer"); - messages.push(message.encode()); - } - messages - } + /// Returns the process ID that the program printed on its first line, that of the process + /// group that it leads. + fn group(&self) -> u32; +} - /// Returns the execution's outcome, once the peer finished. - fn outcome(&self) -> Option { - self.events.iter().find_map(|event| match event { - peer::Event::Finished(outcome) => Some(*outcome), - _ => None, - }) +impl Taken for HostExecution { + fn outputs(&self) -> [Vec; 2] { + Stream::ALL.map(|stream| self.output(stream).to_vec()) } - /// Returns the process ID that the program printed on its first line, that of the process - /// group that it leads. fn group(&self) -> u32 { - let output = String::from_utf8_lossy(&self.output[0]); + let output = String::from_utf8_lossy(self.output(Stream::Stdout)); output.lines().next().unwrap().parse().unwrap() } } @@ -2841,8 +2764,9 @@ impl Host { fn start_then(&mut self, start: Start, input: Vec, after: &[Message]) { let mut messages = vec![HostMessage::Start(start.clone()).encode()]; messages.extend_from_slice(after); - let execution = HostExecution::new(start, input, Instant::now()); - self.executions.insert(execution.start.key.id(), execution); + let execution = HostExecution::new(start, timers(), input, Instant::now()); + self.executions + .insert(execution.start().key.id(), execution); self.send(&messages); } @@ -2883,8 +2807,8 @@ impl Host { .executions .get_mut(&decoded.key().id()) .expect("the guest's message names an execution that the host started"); - decoded - .deliver(&mut execution.peer, now) + execution + .deliver(&decoded, now) .expect("the peer accepts the guest's message"); } } @@ -2960,7 +2884,7 @@ fn check_lossless(host: &Host, id: u64, status: Status) { assert!(!outcome.is_truncated(stream), "{id}: {stream:?}"); } assert_eq!(outcome.is_reusable(), cgroups(), "{id}"); - let peer = &host.executions[&id].peer; + let peer = host.executions[&id].peer(); assert!(peer.high_water() <= peer.memory_bound(), "{id}"); } @@ -2981,16 +2905,19 @@ fn the_worker_runs_executions_at_once_with_frame_counted_credit_and_one_output_p ); let both = "seq 1 30000; seq 30001 60000 >&2; exit 7"; host.start(start_of(3, shell(both)), Vec::new()); - host.run_until(|host| !host.executions[&1].output[0].is_empty()); + host.run_until(|host| !host.executions[&1].output(Stream::Stdout).is_empty()); host.send(&[Message::Ping]); host.run_until(|host| { host.replies.contains(&Message::Pong) && (1..=3).all(|id| host.outcome(id).is_some()) }); assert_eq!(host.replies, [Message::Pong]); - assert_eq!(host.executions[&1].output, [lines(1, 300_000), Vec::new()]); - assert_eq!(host.executions[&2].output, [input, Vec::new()]); assert_eq!( - host.executions[&3].output, + host.executions[&1].outputs(), + [lines(1, 300_000), Vec::new()] + ); + assert_eq!(host.executions[&2].outputs(), [input, Vec::new()]); + assert_eq!( + host.executions[&3].outputs(), [lines(1, 30_000), lines(30_001, 60_000)] ); check_lossless(&host, 1, Status::Exited(0)); @@ -3006,7 +2933,7 @@ fn a_stalled_sink_stalls_only_its_own_execution_while_timers_cancellations_and_c // program blocks once its pipe is full. let mut host = Host::new(); host.start(start_of(1, shell("exec cat /dev/zero")), Vec::new()); - host.execution(1).stalled = true; + host.execution(1).stall(true); // Another execution streams to its end meanwhile, a third one's timeout ends it in time, and // the host's ping is answered. host.start(start_of(2, shell("seq 1 200000")), Vec::new()); @@ -3025,18 +2952,21 @@ fn a_stalled_sink_stalls_only_its_own_execution_while_timers_cancellations_and_c && host.outcome(2).is_some() && host.outcome(3).is_some() }); - assert_eq!(host.executions[&2].output, [lines(1, 200_000), Vec::new()]); + assert_eq!( + host.executions[&2].outputs(), + [lines(1, 200_000), Vec::new()] + ); check_lossless(&host, 2, Status::Exited(0)); let timed = host.outcome(3).unwrap(); assert_eq!(timed.cause(), Some(Cause::TimedOut)); assert!(started.elapsed() < timeout + GRACE + Duration::from_secs(2)); let stalled = &host.executions[&1]; - assert!(stalled.outcome().is_none() && stalled.peer.buffered() > 0); - assert!(stalled.output[0].is_empty()); + assert!(stalled.outcome().is_none() && stalled.peer().buffered() > 0); + assert!(stalled.output(Stream::Stdout).is_empty()); // The stalled execution's cancellation ends it within its grace, with its output truncated. let cancelled = Instant::now(); - host.execution(1).peer.cancel(cancelled); + host.execution(1).peer_mut().cancel(cancelled); host.run_until(|host| host.outcome(1).is_some()); assert!(cancelled.elapsed() < GRACE + Duration::from_secs(2)); let outcome = host.outcome(1).unwrap(); @@ -3105,7 +3035,9 @@ fn a_full_stream_to_the_host_holds_back_only_output_while_cancellations_and_stop Vec::new(), ); } - host.run_until(|host| (1..=4).all(|id| host.executions[&id].output[0].contains(&b'\n'))); + host.run_until(|host| { + (1..=4).all(|id| host.executions[&id].output(Stream::Stdout).contains(&b'\n')) + }); let groups = (1..=4) .map(|id| host.executions[&id].group()) .collect::>(); @@ -3119,7 +3051,7 @@ fn a_full_stream_to_the_host_holds_back_only_output_while_cancellations_and_stop // the others, while the stream stays full. assert!(host.broker.room >= 1024, "{}", host.broker.room); let now = Instant::now(); - host.execution(1).peer.cancel(now); + host.execution(1).peer_mut().cancel(now); let cancel = host.execution(1).serve(now); host.send(&cancel); gone(groups[0]); @@ -3154,7 +3086,7 @@ fn executions_that_write_without_end_take_turns_at_the_stream_to_the_host() { } // Each takes a fair part of what the stream carries. - let taken = |host: &Host, id: u64| host.executions[&id].output[0].len(); + let taken = |host: &Host, id: u64| host.executions[&id].output(Stream::Stdout).len(); let total = |host: &Host| taken(host, 1) + taken(host, 2); host.run_until(|host| total(host) >= 16 * 1024 * 1024); let total = total(&host); @@ -3174,7 +3106,11 @@ fn the_worker_refuses_starts_beyond_its_limits_with_a_failed_launch() { for id in 1..=most { host.start(start_of(id, shell("exec sleep 60")), Vec::new()); } - host.run_until(|host| host.executions.values().all(|side| side.peer.has_started())); + host.run_until(|host| { + host.executions + .values() + .all(|side| side.peer().has_started()) + }); host.start(start_of(most + 1, shell("exit 0")), Vec::new()); host.run_until(|host| host.outcome(most + 1).is_some()); let refused = host.outcome(most + 1).unwrap(); @@ -3182,13 +3118,13 @@ fn the_worker_refuses_starts_beyond_its_limits_with_a_failed_launch() { assert_eq!(refused.launch_failure(), Some(again)); assert_eq!(refused.status(), None); assert!(!refused.is_reusable()); - assert!(!host.executions[&(most + 1)].peer.has_started()); + assert!(!host.executions[&(most + 1)].peer().has_started()); // Once those ended, it refuses an execution that would hold more than it holds for one, and // reports a program that it cannot find. let now = Instant::now(); for id in 1..=most { - host.execution(id).peer.cancel(now); + host.execution(id).peer_mut().cancel(now); } host.run_until(|host| (1..=most).all(|id| host.outcome(id).is_some())); for id in 1..=most { @@ -3283,7 +3219,7 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { host.run_until(|host| { [1, 2] .iter() - .all(|id| host.executions[id].output[0].ends_with(b"\n")) + .all(|id| host.executions[id].output(Stream::Stdout).ends_with(b"\n")) }); let (first, second) = (host.executions[&1].group(), host.executions[&2].group()); @@ -3299,7 +3235,7 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { host.start_then(later, Vec::new(), &[stale]); host.run_until(|host| host.outcome(2).is_some()); assert_eq!( - host.executions[&2].output, + host.executions[&2].outputs(), [b"later\n".to_vec(), Vec::new()] ); check_lossless(&host, 2, Status::Exited(0)); @@ -3309,12 +3245,12 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { // executions of the new epoch. host.broker.reset_in_flight(Identity { epoch: 2, ..FIRST }); let now = Instant::now(); - host.execution(1).peer.reset(now); + host.execution(1).peer_mut().reset(now); gone(first); host.start(start_of(3, shell("echo again")), Vec::new()); host.run_until(|host| host.outcome(1).is_some() && host.outcome(3).is_some()); assert_eq!( - host.executions[&3].output, + host.executions[&3].outputs(), [b"again\n".to_vec(), Vec::new()] ); check_lossless(&host, 3, Status::Exited(0)); @@ -3327,7 +3263,7 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { fn the_worker_ends_its_executions_when_the_host_stops_it() { let mut host = Host::new(); host.start(start_of(1, shell("echo $$; exec sleep 60")), Vec::new()); - host.run_until(|host| host.executions[&1].output[0].ends_with(b"\n")); + host.run_until(|host| host.executions[&1].output(Stream::Stdout).ends_with(b"\n")); let group = host.executions[&1].group(); let (root, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); @@ -3347,9 +3283,9 @@ fn the_worker_collects_the_status_of_a_program_that_outlives_its_execution() { Start::new(Key::new(1, 1), shared_limits(), instant, sleeper), Vec::new(), ); - host.run_until(|host| host.executions[&1].output[0].ends_with(b"\n")); + host.run_until(|host| host.executions[&1].output(Stream::Stdout).ends_with(b"\n")); let program = host.executions[&1].group(); - host.execution(1).peer.cancel(Instant::now()); + host.execution(1).peer_mut().cancel(Instant::now()); host.run_until(|host| host.outcome(1).is_some()); let outcome = host.outcome(1).unwrap(); assert_eq!(outcome.cause(), Some(Cause::Cancelled)); @@ -3383,7 +3319,7 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { // A stop once the worker collected the status of such a program leaves no process of the // group running either. host.start(start_of(2, leaving()), Vec::new()); - host.run_until(|host| host.executions[&2].output[0].ends_with(b"\n")); + host.run_until(|host| host.executions[&2].output(Stream::Stdout).ends_with(b"\n")); let second = host.executions[&2].group(); reaped(second); // The fake root lives until the check, as its workload cgroup, if any, ends what is left in it @@ -3397,7 +3333,7 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { /// standard output of the execution `id`, and the directory of the program's cgroup from the next /// line, which `/proc/self/cgroup` gave. fn process_and_cgroup(host: &Host, id: u64, line: usize) -> (u32, PathBuf) { - let output = String::from_utf8(host.executions[&id].output[0].clone()).unwrap(); + let output = String::from_utf8(host.executions[&id].output(Stream::Stdout).to_vec()).unwrap(); let mut lines = output.lines().skip(line); let process = lines.next().unwrap().parse().unwrap(); let cgroup = lines.next().unwrap().strip_prefix("0::/").unwrap(); @@ -3432,7 +3368,7 @@ fn an_execution_s_settlement_ends_a_descendant_that_left_its_process_group() { for id in 2..=most + 1 { host.start(start_of(id, shell("exec sleep 60")), Vec::new()); } - host.run_until(|host| (2..=most + 1).all(|id| host.executions[&id].peer.has_started())); + host.run_until(|host| (2..=most + 1).all(|id| host.executions[&id].peer().has_started())); let (_, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); } @@ -3498,7 +3434,7 @@ fn an_execution_that_does_not_settle_keeps_its_place_until_the_host_stops_the_wo ); host.start(start, Vec::new()); host.run_until(|host| { - let output = &host.executions[&1].output[0]; + let output = host.executions[&1].output(Stream::Stdout); output.iter().filter(|&&byte| byte == b'\n').count() == 2 }); let (program, cgroup) = process_and_cgroup(&host, 1, 0); @@ -3545,7 +3481,7 @@ fn fill_but_one(host: &mut Host, first: u64) { } host.run_until(|host| { ids.clone() - .all(|id| host.executions[&id].peer.has_started()) + .all(|id| host.executions[&id].peer().has_started()) }); let refused = ids.end; host.start(start_of(refused, shell("exit 0")), Vec::new()); @@ -3554,7 +3490,7 @@ fn fill_but_one(host: &mut Host, first: u64) { assert_eq!(host.outcome(refused).unwrap().launch_failure(), Some(again)); let now = Instant::now(); for id in ids.clone() { - host.execution(id).peer.cancel(now); + host.execution(id).peer_mut().cancel(now); } host.run_until(|host| ids.clone().all(|id| host.outcome(id).is_some())); for id in ids { @@ -3583,9 +3519,12 @@ fn an_execution_runs_in_its_working_directory_with_exactly_its_environment() { check_lossless(&host, id, Status::Exited(0)); } let printed = environment.map(|entry| format!("{entry}\n")).concat(); - assert_eq!(host.executions[&1].output[0], printed.as_bytes()); assert_eq!( - host.executions[&2].output[0], + host.executions[&1].output(Stream::Stdout), + printed.as_bytes() + ); + assert_eq!( + host.executions[&2].output(Stream::Stdout), format!("{directory}\n").as_bytes() ); let (root, worked) = host.stop(); @@ -3716,7 +3655,7 @@ fn the_worker_refuses_at_stage_0_a_start_whose_identity_breaks_the_rules() { } host.run_until(|host| { ids.clone() - .all(|id| host.executions[&id].peer.has_started()) + .all(|id| host.executions[&id].peer().has_started()) }); let (_, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); @@ -3750,7 +3689,7 @@ fn an_execution_that_starts_with_the_root_counts_its_timeout_from_its_own_start( timed, shell("exec sleep 0.2"), ); - let execution = HostExecution::new(start.clone(), Vec::new(), Instant::now()); + let execution = HostExecution::new(start.clone(), timers(), Vec::new(), Instant::now()); let mut host = Host { broker, working: Some(working), @@ -3779,7 +3718,7 @@ fn a_host_that_a_test_drops_while_its_broker_pauses_ends_the_worker_and_its_prog start_of(1, shell("echo $$; exec cat /dev/zero")), Vec::new(), ); - host.run_until(|host| host.executions[&1].output[0].contains(&b'\n')); + host.run_until(|host| host.executions[&1].output(Stream::Stdout).contains(&b'\n')); let group = host.executions[&1].group(); host.broker.pause(); let working = host.working.take().unwrap(); @@ -3864,7 +3803,7 @@ fn the_worker_runs_executions_on_a_tty() { host.start(start_of(1, shell(script)), Vec::new()); host.run_until(|host| host.outcome(1).is_some()); assert_eq!( - host.executions[&1].output, + host.executions[&1].outputs(), [lines(1, 100_000), lines(1, 1000)] ); check_lossless(&host, 1, Status::Exited(0)); diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index 7ffc22fca..a9a7424c0 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -10,15 +10,22 @@ //! decoder, the requests of a layered root: its configuration, its probes, and a stop whose //! teardown failed, and the host's order of a layered root: the guest agent's readiness and build, //! the mapping table in parts, and `Start`, and each failure of the order, after which the host -//! sends nothing more, or stops the guest. +//! sends nothing more, or stops the guest. The broker's guest also runs the runners of NVX's +//! execution engine for the executions that the host program's driver runs, so that the tests cover +//! the driver's output, credit, outcomes, and timers, the refusals that end its run, and a guest +//! that stops reading. #![cfg(feature = "host")] -use std::collections::VecDeque; +use std::collections::{BTreeMap, VecDeque}; use std::io; use std::sync::{Arc, Condvar, Mutex, MutexGuard}; use std::thread::{self, JoinHandle}; use std::time::{Duration, Instant}; +use nvx::exec::{ + Cause, Key, LaunchFailure, Limits, Outcome, Runner, Settlement, Status, Stream, Timers, + Violation, Window, runner, +}; use nvx::host::nvxs::HostConnectionError; use nvx::host::platform::Transport; use nvx::protocol::nvxs::{ @@ -29,8 +36,11 @@ use nvx::protocol::{ OUTER_MAX_PAYLOAD, OUTER_MIN_RECEIVE_WINDOW, }; use runtime_adapter::GUEST_BUILD; +use runtime_adapter::exec::{GuestMessage, HostMessage, MAX_EXECUTIONS, Start}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, MAX_TEXT_BODY, Message}; -use runtime_adapter::host::{self, Client, ClientError, MapReply, OrderError}; +use runtime_adapter::host::{ + self, Client, ClientError, Executions, ExecutionsError, MAX_OUTPUT_FRAMES, MapReply, OrderError, +}; use runtime_adapter::probe::{Facts, Probe, ProbeReply}; use runtime_adapter::root::{ MapFailure, Ready, RootConfig, RootFailure, RootReply, RootStage, decode_maps, encode_maps, @@ -1192,6 +1202,7 @@ fn the_client_refuses_messages_that_its_framing_does_not_allow() { // The client takes the guest agent's messages, and refuses those that only the host program // sends. + let key = Key::new(1, 1); let host_messages = [ Message::Ping, Message::Stop, @@ -1199,6 +1210,10 @@ fn the_client_refuses_messages_that_its_framing_does_not_allow() { Message::Probe(b"mounts".to_vec()), Message::Maps(maps_part()), Message::Start, + HostMessage::Start(start_of(1)).encode(), + HostMessage::Input(key, b"input").encode(), + HostMessage::InputEnd(key).encode(), + HostMessage::Cancel(key, Cause::Cancelled).encode(), ]; for message in host_messages { let (mut broker, mut client) = attached(FIRST, usize::MAX); @@ -1804,3 +1819,467 @@ fn a_mapping_that_the_guest_fails_ends_the_table_and_the_host_stops_the_guest() "{description}" ); } + +/// The chunk of the executions of the tests of the host's driver. +const CHUNK: u32 = 16 * 1024; + +/// Returns the limits of the executions of the tests of the host's driver, as the host program +/// chooses them: chunks of 16 KiB, and frame-counted credit, of 4 frames of input, and of 8 frames +/// of output in one pool that standard output and error share. +fn limits() -> Limits { + Limits::new(CHUNK, Window::frames(4), Window::frames(8)) + .unwrap() + .shared_output() +} + +/// Returns the timers of both sides of the executions of the tests of the host's driver. +fn timers() -> Timers { + Timers::new(Duration::from_millis(300), Duration::from_secs(5)) +} + +/// Returns the start of the execution `id` of the first generation, of a program that the test's +/// guest plays. +fn start_of(id: u64) -> Start { + Start::new(Key::new(id, 1), limits(), timers(), ["/bin/program"]) +} + +/// Returns the position of `stream` among [`Stream::ALL`]. +fn position(stream: Stream) -> usize { + match stream { + Stream::Stdout => 0, + Stream::Stderr => 1, + } +} + +/// What the program of an execution that the test's guest plays does. +#[derive(Clone, Debug)] +struct Workload { + /// Whether the program starts, or why its launch failed. + launch: Result<(), LaunchFailure>, + /// What the program writes to its standard output and error, in the order of [`Stream::ALL`]. + output: [Vec; 2], + /// How the program ends, once it wrote its output. + status: Status, +} + +/// The guest's side of an execution that the test's guest plays: the runner of NVX's execution +/// engine, which takes the program's output as its credit lets it, as the worker's does. +struct Played { + /// The guest's side of the execution. + runner: Runner, + /// What the program does. + workload: Workload, + /// Whether the runner learned how the launch went. + launched: bool, + /// The bytes of each stream that the runner took. + written: [usize; 2], + /// Whether the program ended. + ended: bool, +} + +impl Played { + /// Returns the guest's side of the execution of `runner`, whose program does `workload`. + fn new(runner: Runner, workload: Workload) -> Self { + Self { + runner, + workload, + launched: false, + written: [0, 0], + ended: false, + } + } + + /// Takes every step that the program and the runner can take at `now`, as the worker takes + /// them, and returns the runner's messages, as frames. + fn step(&mut self, now: Instant) -> Vec { + if !self.launched { + self.launched = true; + self.runner.launched(self.workload.launch); + if self.workload.launch.is_err() { + for stream in Stream::ALL { + self.runner.output_closed(stream); + } + self.runner.exited(now, None); + self.ended = true; + } + } + if !self.ended { + for stream in Stream::ALL { + let output = &self.workload.output[position(stream)]; + let written = &mut self.written[position(stream)]; + *written += self.runner.output(stream, &output[*written..]); + } + if self.written == self.workload.output.each_ref().map(Vec::len) { + for stream in Stream::ALL { + self.runner.output_closed(stream); + } + self.runner.exited(now, Some(self.workload.status)); + self.ended = true; + } + } + self.runner.tick(now); + while let Some(request) = self.runner.next_request() { + if let runner::Request::Settle = request { + self.runner.settled(match self.workload.launch { + Ok(()) => Settlement::Settled, + Err(_) => Settlement::Failed, + }); + } + } + let key = self.runner.key(); + let mut buffer = vec![0; self.runner.limits().chunk()]; + let mut messages = Vec::new(); + while let Some(message) = self.runner.next_message(&mut buffer) { + if let Some(message) = GuestMessage::from_runner(key, message, &buffer) { + messages.push(message.encode()); + } + } + messages + } +} + +/// Plays the guest's side of the host's executions until `count` of them finished: a runner for +/// each start, whose program does what `workload` says of the start. Takes the host's messages, +/// sends the runners' in DATA records of at most `size` bytes, and returns the starts, in order. +fn play( + broker: &mut Broker, + count: usize, + size: usize, + workload: impl Fn(&Start) -> Workload, +) -> Vec { + let mut starts = Vec::new(); + let mut played = BTreeMap::::new(); + loop { + let now = Instant::now(); + let messages = played + .values_mut() + .flat_map(|played| played.step(now)) + .collect::>(); + broker.send_data(&encode(&messages), size); + if played.len() == count && played.values().all(|played| played.runner.is_finished()) { + return starts; + } + for message in broker.take() { + let decoded = HostMessage::decode(&message) + .expect("the host sends messages about executions") + .expect("the host's messages decode"); + let message = match decoded { + HostMessage::Start(start) => { + let runner = Runner::new(start.key, start.limits, start.timers, now); + played.insert(start.key.id(), Played::new(runner, workload(&start))); + starts.push(start); + continue; + } + message => message, + }; + let key = message.key(); + let runner = &mut played + .get_mut(&key.id()) + .expect("the host started the execution") + .runner; + match message { + HostMessage::Credit(key, pool, units) => runner.credit(key, pool, units), + HostMessage::Input(key, bytes) => runner.input(key, bytes), + HostMessage::InputEnd(key) => runner.end_input(key), + HostMessage::Cancel(key, cause) => runner.cancel(now, key, cause), + HostMessage::Start(_) => unreachable!("the start came first"), + } + .expect("the runner accepts the host's message"); + } + } +} + +/// Starts the executions of `starts` with `client`, each with its own timers, and runs them in a +/// thread until `deadline`, which returns the client and the driver. +fn run( + client: Client, + starts: Vec, + deadline: Instant, +) -> JoinHandle<(Client, Result)> { + spawn(client, move |client| { + let mut executions = Executions::new(); + for start in starts { + let timers = start.timers; + executions.start(client, start, timers, deadline)?; + } + executions.run(client, deadline)?; + Ok(executions) + }) +} + +#[test] +fn the_driver_runs_executions_at_once_and_collects_their_output_and_outcomes() { + // The first program writes more to its standard output than the guest's credit holds, so that + // its output flows as the host takes it, and a little to its standard error, then exits with + // 3. The second one does not start. The guest's records carry its messages whole, or in + // pieces that split them, and each read of the host's takes at most 4 KiB. + let output = (0..=u8::MAX).cycle().take(300 * 1024).collect::>(); + let written = Workload { + launch: Ok(()), + output: [output.clone(), b"error".to_vec()], + status: Status::Exited(3), + }; + let refused = Workload { + launch: Err(LaunchFailure::new(0, 13)), + output: [Vec::new(), Vec::new()], + status: Status::Exited(0), + }; + for size in [OUTER_MAX_PAYLOAD, 1000] { + let (mut broker, client) = attached(FIRST, 4096); + let running = run(client, vec![start_of(1), start_of(2)], deadline()); + let starts = play(&mut broker, 2, size, |start| match start.key.id() { + 1 => written.clone(), + _ => refused.clone(), + }); + assert_eq!(starts, [start_of(1), start_of(2)]); + let (_, executions) = joined(running); + let executions = executions.unwrap(); + assert!(executions.is_finished()); + let ids = executions + .iter() + .map(|execution| execution.start().key.id()) + .collect::>(); + assert_eq!(ids, [1, 2]); + + let first = executions.get(1).unwrap(); + assert!(first.peer().has_started()); + assert_eq!(first.output(Stream::Stdout), output); + assert_eq!(first.output(Stream::Stderr), b"error"); + let outcome = first.outcome().unwrap(); + assert_eq!(outcome.status(), Some(Status::Exited(3))); + assert_eq!((outcome.cause(), outcome.launch_failure()), (None, None)); + assert!(outcome.is_reusable()); + for stream in Stream::ALL { + assert!(!outcome.is_truncated(stream), "{stream:?}"); + } + + let second = executions.get(2).unwrap(); + assert!(!second.peer().has_started()); + let outcome = second.outcome().unwrap(); + assert_eq!(outcome.launch_failure(), Some(LaunchFailure::new(0, 13))); + assert_eq!(outcome.status(), None); + assert!(!outcome.is_reusable()); + for stream in Stream::ALL { + assert!(second.output(stream).is_empty(), "{stream:?}"); + } + } +} + +#[test] +fn the_driver_refuses_starts_before_it_sends_them() { + let (mut broker, mut client) = attached(FIRST, usize::MAX); + let mut executions = Executions::new(); + // The guest ends its session on a start that it cannot decode. + let undecodable = [ + start_of(1).with_working_directory("relative"), + start_of(1).with_environment(["=value"]), + Start::new(Key::new(1, 1), limits(), timers(), [""]), + ]; + for start in undecodable { + let refused = executions.start(&mut client, start.clone(), timers(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Start(_))), + "{start:?}: {refused:?}" + ); + } + // A start whose input the host would send while it reads the guest's output, and starts whose + // output windows would let the guest send many messages ahead of its credit for the host's. + let refused = executions.start(&mut client, start_of(1).with_input(), timers(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Input)), + "{refused:?}" + ); + let program = ["/bin/program"]; + for output in [ + Window::bytes(64 * 1024), + Window::frames(MAX_OUTPUT_FRAMES + 1), + ] { + let limits = Limits::new(CHUNK, Window::frames(4), output).unwrap(); + let start = Start::new(Key::new(1, 1), limits, timers(), program); + let refused = executions.start(&mut client, start, timers(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Window)), + "{output:?}: {refused:?}" + ); + } + // A second start of an ID, and a start beyond the executions that the guest runs at once, + // after starts of which one has the widest output windows that the driver takes. + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + let mut starts = (1..=most).map(start_of).collect::>(); + let widest = Limits::new(CHUNK, Window::frames(4), Window::frames(MAX_OUTPUT_FRAMES)).unwrap(); + starts[0] = Start::new(Key::new(1, 1), widest, timers(), program); + for start in &starts { + executions + .start(&mut client, start.clone(), timers(), deadline()) + .unwrap(); + } + let refused = executions.start(&mut client, start_of(1), timers(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Duplicate(1))), + "{refused:?}" + ); + let refused = executions.start(&mut client, start_of(most + 1), timers(), deadline()); + assert!(matches!(refused, Err(ExecutionsError::Full)), "{refused:?}"); + // The guest received those starts alone, and the client stays usable. + let sent = starts + .into_iter() + .map(|start| HostMessage::Start(start).encode()) + .collect::>(); + assert_eq!(broker.messages(MAX_EXECUTIONS), sent); + let pinging = spawn(client, |client| client.ping(deadline())); + assert_eq!(broker.answer(OUTER_MAX_PAYLOAD), Message::Ping); + joined(pinging).1.unwrap(); + + // A driver runs one batch: once it ran, it starts nothing more. + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], deadline()); + let refused = Workload { + launch: Err(LaunchFailure::new(0, 11)), + output: [Vec::new(), Vec::new()], + status: Status::Exited(0), + }; + let played = play(&mut broker, 1, OUTER_MAX_PAYLOAD, |_| refused.clone()); + assert_eq!(played, [start_of(1)]); + let (mut client, executions) = joined(running); + let mut executions = executions.unwrap(); + assert!(executions.is_finished()); + let refused = executions.start(&mut client, start_of(2), timers(), deadline()); + assert!(matches!(refused, Err(ExecutionsError::Ran)), "{refused:?}"); +} + +#[test] +fn the_driver_ends_its_run_on_a_message_that_breaks_the_rules() { + /// A check of the failure that ended a run. + type Check = fn(&ExecutionsError) -> bool; + let key = Key::new(1, 1); + let key_body = [key.id().to_le_bytes(), key.generation().to_le_bytes()].concat(); + let cases: [(&str, Vec, Check); 4] = [ + ( + "a message about an execution that the host did not start", + vec![GuestMessage::Started(Key::new(2, 1)).encode()], + |error| matches!(error, ExecutionsError::Unknown(key) if *key == Key::new(2, 1)), + ), + ( + "a message about no execution", + vec![Message::Pong], + |error| matches!(error, ExecutionsError::Unexpected(kind) if *kind == Message::Pong.kind()), + ), + ( + "output of a stream that does not exist", + vec![Message::ExecOutput([&key_body[..], &[9], b"x"].concat())], + |error| matches!(error, ExecutionsError::Body(_)), + ), + ( + "an outcome before the ends of the streams", + vec![ + GuestMessage::Started(key).encode(), + GuestMessage::Terminal(key, Outcome::new(Some(Status::Exited(0)))).encode(), + ], + |error| { + matches!( + error, + ExecutionsError::Violation(failed, Violation::Order) if *failed == Key::new(1, 1) + ) + }, + ), + ]; + for (name, messages, check) in cases { + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], deadline()); + broker.messages(1); + broker.send_data(&encode(&messages), OUTER_MAX_PAYLOAD); + let (_, ran) = joined(running); + let failed = ran.as_ref().err().is_some_and(check); + assert!(failed, "{name}: {:?}", ran.err()); + } + + // The end of the guest's stream ends the run too. + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], deadline()); + broker.messages(1); + broker.close(); + let (_, ran) = joined(running); + let closed = matches!( + ran, + Err(ExecutionsError::Client(ClientError::Session( + HostConnectionError::Closed + ))) + ); + assert!(closed, "{:?}", ran.err()); +} + +#[test] +fn the_driver_fires_the_timers_of_its_executions_while_the_guest_says_nothing() { + // The execution's timeout passes while the guest sends nothing: the host cancels it, waits for + // its outcome for the grace and the settlement time, then gives up on it, and fences it. + let (grace, settlement, timeout) = ( + Duration::from_millis(50), + Duration::from_millis(100), + Duration::from_millis(200), + ); + let timed = Timers::new(grace, settlement).timeout(timeout); + let key = Key::new(1, 1); + let start = Start::new(key, limits(), timed, ["/bin/program"]); + let (mut broker, client) = attached(FIRST, usize::MAX); + let started = Instant::now(); + let running = run(client, vec![start.clone()], deadline()); + // The start, the credit of the output's pool, then the cancellation. + let messages = broker.messages(3); + assert_eq!(messages[0], HostMessage::Start(start).encode()); + assert_eq!( + messages[2], + HostMessage::Cancel(key, Cause::TimedOut).encode() + ); + let (_, executions) = joined(running); + assert!(started.elapsed() >= timeout + grace + settlement); + let executions = executions.unwrap(); + let outcome = executions.get(1).unwrap().outcome().unwrap(); + assert_eq!(outcome.cause(), Some(Cause::TimedOut)); + assert_eq!(outcome.status(), None); + assert!(!outcome.is_reusable()); + + // A run whose deadline passes before its executions finish fails, in a receive, which leaves + // the client usable: the guest takes the start's credit, then answers a ping. + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], Instant::now() + SHORT); + broker.messages(1); + let (client, ran) = joined(running); + assert!( + matches!(ran, Err(ExecutionsError::TimedOut)), + "{:?}", + ran.err() + ); + broker.messages(1); + let pinging = spawn(client, |client| client.ping(deadline())); + assert_eq!(broker.answer(OUTER_MAX_PAYLOAD), Message::Ping); + joined(pinging).1.unwrap(); +} + +#[test] +fn the_driver_s_sends_wait_for_a_guest_that_stops_reading_until_the_deadline_of_its_run() { + // The guest takes the start, then stops reading, and the host's stream holds 16 bytes, so the + // driver's next send, the credit of the output's pool, waits for the transport until the + // run's deadline, whose timeout ends the client. The run's deadline counts from the start's + // send, which the guest took. + let (mut broker, client) = attach_over(connect_bounded(FIRST, usize::MAX, 16)); + let running = spawn(client, |client| { + let mut executions = Executions::new(); + executions.start(client, start_of(1), timers(), deadline())?; + let started = Instant::now(); + let ran = executions.run(client, started + SHORT); + Ok::<_, ExecutionsError>((started.elapsed(), ran)) + }); + assert_eq!( + broker.messages(1), + [HostMessage::Start(start_of(1)).encode()] + ); + let (mut client, result) = joined(running); + let (elapsed, ran) = result.unwrap(); + assert!(elapsed >= SHORT, "{elapsed:?}"); + assert!( + matches!(ran, Err(ExecutionsError::TimedOut)), + "{:?}", + ran.err() + ); + assert_ended(&mut client); +} diff --git a/tests/runtime-adapter/tests/layered_launch.rs b/tests/runtime-adapter/tests/layered_launch.rs index 4569de3d9..52b32b7b3 100644 --- a/tests/runtime-adapter/tests/layered_launch.rs +++ b/tests/runtime-adapter/tests/layered_launch.rs @@ -3,8 +3,9 @@ //! OpenVMM the checked files, plans the export of the host paths that it shares with the root //! inside the directories that the adapter authorizes, attaches it, and verifies it before the //! launch, composes OpenVMM's arguments without NVX's managed lifecycle, records what it launches, -//! with the bindings that it checked, and refuses options that do not make a command, and shares -//! that it cannot export before it touches the store. The tests need no hypervisor. +//! with the bindings that it checked, and refuses options that do not make a command, shares that +//! it cannot export before it touches the store, executions whose starts the guest would not take, +//! and more executions than the guest runs at once. The tests need no hypervisor. #![cfg(feature = "host")] use std::ffi::OsString; @@ -24,6 +25,7 @@ use nvx::protocol::root::{ImageFormat, RootImage, RootLayout, RootUpper}; use nvx::protocol::{HostMap, MAX_HOST_MAPPINGS}; use nvx::uuid::Uuid; use nvx::{ErrorCode, ErrorKind, Hypervisor}; +use runtime_adapter::exec::MAX_EXECUTIONS; use runtime_adapter::hook::STEP_HOOK; use runtime_adapter::layered::{Bindings, LaunchRecord, LayeredLaunch, RECORD_NAME, SANDBOX_TOKEN}; use runtime_adapter::root::RootConfig; @@ -849,7 +851,7 @@ fn the_host_program_refuses_shares_before_it_touches_the_store() { fn the_host_program_refuses_options_that_do_not_make_a_command() { let launch = ["launch", "store", "--guest-agent", "agent"]; let uuid = "0a490905-41be-ed5f-984e-63eb7e5e20b1"; - let cases: [(&[&str], &str); 30] = [ + let cases: [(&[&str], &str); 34] = [ ( &["--distro", "erofs:image", "--upper", "ram"], "--distro needs --distro-identity", @@ -864,6 +866,22 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { "--distro-identity needs --distro", ), (&["--runtime-dir", "/run/a"], "--runtime-dir needs --distro"), + ( + &["--workload-identity", "1000:1000"], + "--workload-identity needs --distro", + ), + ( + &["--workload-identity", "0:1000"], + "--workload-identity takes UID:GID, two numbers from 1 to 4294967294, not \"0:1000\"", + ), + ( + &["--workload-identity", "1000:4294967295"], + "--workload-identity takes UID:GID", + ), + ( + &["--workload-identity", "1000"], + "--workload-identity takes UID:GID", + ), ( &["--overlay-mode", "untrusted-lowers"], "--overlay-mode needs --distro", @@ -1022,6 +1040,78 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { "unexpected option \"--share\" of reattach", ), ]); + // Each execution of `exec` has options of its own, before `--` and its program, and its + // arguments end at a lone `;`. The program refuses a start that the guest would not decode, + // on which the guest would end its session, and more executions than the guest runs at once, + // before it attaches. + let program = ["--", "/bin/true"]; + let exec = |options: &[&'static str]| [&["exec", "store"], options, &program[..]].concat(); + commands.extend([ + ( + vec!["exec", "store"], + "exec needs `-- PROGRAM` after the options of each execution", + ), + ( + vec!["exec", "store", "--"], + "exec needs a program after `--`", + ), + ( + exec(&[]).into_iter().chain([";"]).collect(), + "exec needs `-- PROGRAM` after the options of each execution", + ), + (exec(&["--groups", "1"]), "--groups needs --identity"), + ( + exec(&["--capabilities", "0"]), + "--capabilities needs --identity", + ), + (exec(&["--numeric"]), "--numeric needs --identity"), + ( + exec(&["--identity", "1000"]), + "--identity takes UID:GID, not \"1000\"", + ), + ( + exec(&["--identity", "0:0", "--groups", "1,,2"]), + "--groups takes GIDs, separated by commas, not \"1,,2\"", + ), + ( + exec(&["--identity", "0:0", "--capabilities", "64"]), + "--capabilities takes capability numbers from 0 to 63", + ), + ( + exec(&["--cwd", "tmp"]), + "names a working directory that is not absolute", + ), + ( + exec(&["--env", "=value"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + exec(&["--cwd", "/a", "--cwd", "/b"]), + "unexpected option \"--cwd\" of exec", + ), + ( + exec(&["--distro", "erofs:image"]), + "unexpected option \"--distro\" of exec", + ), + (vec!["exec", "store", "--env"], "\"--env\" needs a value"), + ( + [ + exec(&["--identity", "0:0", "--groups", "1"]), + vec![";"], + exec(&["--groups", "1"])[2..].to_vec(), + ] + .concat(), + "--groups needs --identity", + ), + ( + [ + vec!["exec", "store"], + vec![program.to_vec(); MAX_EXECUTIONS + 1].join(&";"), + ] + .concat(), + "exec runs at most", + ), + ]); for (arguments, problem) in commands { let output = Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")) .args(&arguments) diff --git a/tests/runtime-adapter/tests/vm_layered.rs b/tests/runtime-adapter/tests/vm_layered.rs index fad70eda4..93908e3aa 100644 --- a/tests/runtime-adapter/tests/vm_layered.rs +++ b/tests/runtime-adapter/tests/vm_layered.rs @@ -4,10 +4,10 @@ //! //! Each scenario of the table that [`scenarios!`] declares is an ignored test, which stages the //! images of a [`Root`], launches a VM with it, and checks what it [`Expect`]s: a root that the -//! guest assembles, which the scenario probes and the stop tears down, the stage of the root that -//! fails, or a launch that the host program refuses before it spawns the VM, since the scenario -//! changed what it staged. A new scenario is a new row, with its own checks, through the probes of -//! the worker. +//! guest assembles, which the scenario probes, or runs executions in through the host program's +//! `exec`, and the stop tears down, the stage of the root that fails, or a launch that the host +//! program refuses before it spawns the VM, since the scenario changed what it staged. A new +//! scenario is a new row, with its own checks, through the probes of the worker or executions. //! //! The tests boot VMs, so they are ignored. Besides what the `common` module lists, they read its //! fixtures, those of NVX's own layered scenarios, from NVX's artifacts in `NVX_ARTIFACTS_DIR`. @@ -130,6 +130,31 @@ const MAP_TABLE: &str = "map dir=srv/maps/dir inner=srv/maps/dir/inner file=srv/ /// `file.txt`: the plan exports the outermost shares as its children, in the order of their host /// paths. const MAP_CHECKS: &str = "map-checks 2 1"; +/// The identity, besides root's, as which the scenarios' executions run: the image's `nobody`, of +/// the group `nogroup`, which NVX's Ubuntu image always holds. +const NOBODY: (u32, u32) = (65534, 65534); +/// An identity whose account the image lacks. +const ACCOUNTLESS: (u32, u32) = (4242, 4242); +/// Linux's error number of an operation that is not permitted. +const EPERM: i32 = 1; +/// Linux's error number of an invalid argument. +const EINVAL: i32 = 22; +/// The number of executions that meet only while all of them run. +const MEETING: usize = 4; +/// A script that starts a descendant that leads a session of its own and holds the script's +/// output: the descendant prints its process ID, which is its session's ID, then moves its output +/// to the script's, which ends the command substitution that reads the ID. The script prints +/// `started` once `/proc` shows that the descendant leads its session. +const DESCENDANT_SCRIPT: &str = "exec 3>&1; \ + descendant=$(setsid sh -c 'echo $$; exec sleep 600 >&3 3>&-' &); \ + read -r pid comm state parent group session rest , /// The bound of the first partition, as the launch names it, unless the guest's default. first_partition_bound: Option, + /// The sandbox's workload identity, its user ID and its group ID, as the launch names it, if + /// any. + workload: Option<(u32, u32)>, /// The host paths that the launch shares with the root. table: Table, } @@ -206,6 +234,7 @@ const ROOT: Root = Root { overlay: None, journal: None, first_partition_bound: None, + workload: None, table: Table::None, }; @@ -289,6 +318,9 @@ impl Root { if let Some(bound) = self.first_partition_bound { options.extend(["--first-partition-bound".into(), bound.to_string().into()]); } + if let Some((uid, gid)) = self.workload { + options.extend(["--workload-identity".into(), format!("{uid}:{gid}").into()]); + } options } @@ -399,7 +431,8 @@ enum Expect { /// that failed with this problem, which leaves a scratch mounted when the VM ends. TeardownFails(fn(&mut Vm<'_>), &'static str), /// The guest reports that the stage failed with this problem, and the VM ends with its - /// status, before the guest mounted a scratch. + /// status, without a scratch mounted: the guest fails each stage but the identity's before it + /// mounts the scratch, and unmounts the root that it assembled once the identity's failed. Fails(RootStage, &'static str), /// The guest assembles the root, applies this many mappings of the table, and fails the next /// with a problem that holds this text: the launch sends no more of the table, nor `Start`, and @@ -489,6 +522,23 @@ impl Vm<'_> { assert_eq!(printed["pid"], self.launched["pid"]); printed } + + /// Runs `executions` at once in the VM's root, each the options and the arguments of one + /// execution of the host program's `exec`, and returns what they did, each fact as + /// `exec.N.KEY`, for execution N, from 1. + fn exec(&mut self, executions: &[Vec]) -> Printed { + let mut options = Vec::new(); + for (index, execution) in executions.iter().enumerate() { + if index > 0 { + options.push(OsStr::new(";")); + } + options.extend(execution.iter().map(OsStr::new)); + } + let printed = self.sandbox.run("exec", &options, 0); + assert_eq!(printed["reattached"], "live"); + assert_eq!(printed["pid"], self.launched["pid"]); + printed + } } /// Declares the table of scenarios: each row, `NAME: SCENARIO;`, is an ignored test, `NAME`, which @@ -758,6 +808,33 @@ scenarios! { "changed after its scratch binding was made: its identity differs", ), }; + /// Executions that the host program runs at once over one session, in a sandbox without a + /// workload identity, so that each start names its own: output beyond 1 MiB, whole; several + /// executions that meet only while all of them run; a descendant in a session of its own, + /// which holds its program's output, and which the end of its program ends; a user other than + /// root, and root with an allowed capability alone; a working directory, an exact + /// environment, and the default hostname, each as PID 1 of its namespaces. The guest + /// refuses a start without an identity, and one with a capability outside the entry's + /// allowlist, at stage 0. + executions_run_in_the_root: Scenario { + root: ROOT, + expect: Expect::Ready(check_executions), + }; + /// Executions in a sandbox whose workload identity the launch names, which every execution + /// runs as: the guest refuses a start that names an identity of its own. + executions_run_as_the_workload_identity: Scenario { + root: Root { workload: Some(NOBODY), ..ROOT }, + expect: Expect::Ready(check_workload_executions), + }; + /// A workload identity without an account in the image, which fails the identity stage once + /// the guest assembled the root, before it answers the root configuration. + accountless_workload_identity_fails_its_stage: Scenario { + root: Root { workload: Some(ACCOUNTLESS), ..ROOT }, + expect: Expect::Fails( + RootStage::Identity, + "cannot enter the root as the workload identity 4242:4242: ", + ), + }; } /// Runs the scenario `name`, `scenario`. @@ -807,7 +884,7 @@ fn run(name: &str, scenario: Scenario) { assert_eq!(sandbox.record(), None); assert!( !(over_scratch && needs_recovery(&scratch)), - "the guest mounted the scratch" + "the guest left the scratch mounted" ); return; } @@ -973,6 +1050,7 @@ fn check_record(sandbox: &Sandbox, root: &Root, distro: &Distro, scratch: &Path) overlay_mode: root.overlay, scratch_journal: root.journal, mapping_count: root.table.mapping_count(), + workload_identity: root.workload, ..RootConfig::default() }; let sent = String::from_utf8(config.encode()).unwrap(); @@ -2108,3 +2186,187 @@ fn check_root_profiles(vm: &mut Vm<'_>) { assert_eq!(fact("after.machine-id"), ROOT_MACHINE_ID); assert_eq!(fact("after.mounts"), "0"); } + +/// Returns the options and the arguments of an execution of the host program's `exec`: `options`, +/// then `--`, then the program and its arguments, `command`. +fn execution(options: &[&str], command: &[&str]) -> Vec { + options + .iter() + .chain(&["--"]) + .chain(command) + .map(|argument| (*argument).to_owned()) + .collect() +} + +/// Returns `identity` as the value of the option `--identity` of `exec`: `UID:GID`. +fn ids((uid, gid): (u32, u32)) -> String { + format!("{uid}:{gid}") +} + +/// Returns `text` as the host program prints a value: with `\`, newlines, and carriage returns +/// escaped. +fn escaped(text: &str) -> String { + text.replace('\\', "\\\\") + .replace('\n', "\\n") + .replace('\r', "\\r") +} + +/// Returns the output of `seq first last`. +fn lines(first: usize, last: usize) -> String { + (first..=last).map(|number| format!("{number}\n")).collect() +} + +/// Returns what the [`IDENTITY_SCRIPT`] prints for the user ID `uid`, the group ID `gid`, the +/// groups `groups`, and the capabilities `capabilities`, a set of each capability's bit. +fn identity_facts(uid: u32, gid: u32, groups: &str, capabilities: u64) -> String { + let set = capability_set(capabilities); + format!("{uid}\n{gid}\n{groups}\nCapPrm: {set}\nCapEff: {set}\nCapBnd: {set}\n") +} + +/// Returns the fact KEY of execution `number` that `exec` printed in `printed`. +fn fact<'a>(printed: &'a Printed, number: usize, key: &str) -> &'a str { + let key = format!("exec.{number}.{key}"); + printed + .get(&key) + .unwrap_or_else(|| panic!("{key} is missing: {printed:?}")) +} + +/// Checks that execution `number` of what `exec` printed in `printed` started, ended on its own +/// with the exit code `code`, lost no byte, and settled, so that its place is reusable, and that +/// it wrote `stdout` and `stderr`. +fn assert_ran(printed: &Printed, number: usize, code: i32, stdout: &str, stderr: &str) { + let status = format!("exit {code}"); + let facts = + ["launch", "status", "cause", "truncated", "settled"].map(|key| fact(printed, number, key)); + assert_eq!( + facts, + ["started", &*status, "none", "none", "reusable"], + "{number}: {printed:?}" + ); + assert_eq!(fact(printed, number, "stdout"), escaped(stdout), "{number}"); + assert_eq!(fact(printed, number, "stderr"), escaped(stderr), "{number}"); +} + +/// Checks that the guest refused execution `number` of what `exec` printed in `printed` at stage +/// 0 with the error number `error`, before its program started: the execution is fenced, and has +/// no output. +fn assert_refused(printed: &Printed, number: usize, error: i32) { + let launch = format!("failed 0 {error}"); + let facts = + ["launch", "status", "settled", "stdout", "stderr"].map(|key| fact(printed, number, key)); + assert_eq!( + facts, + [&*launch, "none", "fenced", "", ""], + "{number}: {printed:?}" + ); +} + +/// Checks the VM of the `executions_run_in_the_root` scenario, whose starts name their identities. +fn check_executions(vm: &mut Vm<'_>) { + let nobody = ids(NOBODY); + + // Several executions at once, which meet only while all of them run: each leaves its mark in + // the root's `/tmp`, and waits for the others' marks. Beside them, one writes more than a MiB + // to its standard output, and a little to its standard error, which arrive whole. + let meet = format!( + "mkdir -p /tmp/meeting && touch /tmp/meeting/$NUMBER && \ + until [ \"$(ls /tmp/meeting | wc -l)\" -ge {MEETING} ]; do sleep 0.01; done && \ + echo \"$NUMBER met\"" + ); + let mut executions = (1..=MEETING) + .map(|number| { + let environment = format!("NUMBER={number}"); + execution( + &["--identity", &nobody, "--env", &environment], + &["/bin/sh", "-c", &meet], + ) + }) + .collect::>(); + let writer = "seq 1 200000; seq 1 1000 >&2"; + executions.push(execution( + &["--identity", "0:0"], + &["/bin/sh", "-c", writer], + )); + let printed = vm.exec(&executions); + for number in 1..=MEETING { + assert_ran(&printed, number, 0, &format!("{number} met\n"), ""); + } + let output = lines(1, 200_000); + assert!(output.len() > 1 << 20, "{}", output.len()); + assert_ran(&printed, MEETING + 1, 0, &output, &lines(1, 1000)); + + // In a later session, whose keys are new to the guest: + // 1. a descendant that leads a session of its own, and holds its program's output, which the + // end of its program, PID 1 of its namespace, ends, so that the execution loses no byte; + // 2. a user other than root, without capabilities, in existing-account mode; + // 3. root with `CAP_CHOWN` alone, in its permitted, effective, and bounding sets; + // 4. exactly the environment that the start names; + // 5. the working directory that the start names; + // 6. the entry's default hostname, `nvx-sandbox`, as PID 1 of the execution's namespaces; + // 7. no identity, in a sandbox without a workload identity, which the guest refuses; + // 8. a capability outside the entry's allowlist, `CAP_SYS_ADMIN`, which the entry refuses. + let executions = [ + execution( + &["--identity", "0:0"], + &["/bin/sh", "-c", DESCENDANT_SCRIPT], + ), + execution( + &["--identity", &nobody], + &["/bin/sh", "-c", IDENTITY_SCRIPT], + ), + execution( + &["--identity", "0:0", "--capabilities", "0"], + &["/bin/sh", "-c", IDENTITY_SCRIPT], + ), + execution( + &[ + "--identity", + &nobody, + "--env", + "FIRST=1", + "--env", + "SECOND=two words", + ], + &["/usr/bin/env"], + ), + execution( + &["--identity", &nobody, "--cwd", "/usr/share"], + &["/bin/pwd"], + ), + execution( + &["--identity", &nobody], + &["/bin/sh", "-c", "cat /proc/sys/kernel/hostname; echo $$"], + ), + execution(&[], &["/bin/true"]), + execution( + &["--identity", "0:0", "--capabilities", "21"], + &["/bin/true"], + ), + ]; + let printed = vm.exec(&executions); + assert_ran(&printed, 1, 0, "started\n", ""); + let nobody_facts = identity_facts(NOBODY.0, NOBODY.1, "65534", 0); + assert_ran(&printed, 2, 0, &nobody_facts, ""); + assert_ran(&printed, 3, 0, &identity_facts(0, 0, "0", 1), ""); + assert_ran(&printed, 4, 0, "FIRST=1\nSECOND=two words\n", ""); + assert_ran(&printed, 5, 0, "/usr/share\n", ""); + assert_ran(&printed, 6, 0, "nvx-sandbox\n1\n", ""); + assert_refused(&printed, 7, EINVAL); + assert_refused(&printed, 8, EPERM); +} + +/// Checks the VM of the `executions_run_as_the_workload_identity` scenario, whose sandbox's +/// workload identity is [`NOBODY`]. +fn check_workload_executions(vm: &mut Vm<'_>) { + assert_eq!(vm.config.workload_identity, Some(NOBODY)); + let executions = [ + execution(&[], &["/bin/sh", "-c", IDENTITY_SCRIPT]), + execution(&["--env", "ONLY=this"], &["/usr/bin/env"]), + execution(&["--identity", &ids(NOBODY)], &["/bin/true"]), + ]; + let printed = vm.exec(&executions); + let nobody_facts = identity_facts(NOBODY.0, NOBODY.1, "65534", 0); + assert_ran(&printed, 1, 0, &nobody_facts, ""); + assert_ran(&printed, 2, 0, "ONLY=this\n", ""); + assert_refused(&printed, 3, EINVAL); +}