From bd9a1e41932ccc0883a56d8424d69a049e6edc57 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 02:10:24 -0700 Subject: [PATCH 1/7] tests: run the runtime adapter's executions in VMs through its host program The guest's worker runs executions in the assembled root, under their identities, but the host program could not start one, so no test ran an execution in a VM, and nothing entered a real root. The host program's new exec command reattaches to a VM and runs several executions at once over one session. A single-threaded driver, host::Executions, serves the host's side of each, an exec::HostExecution: the engine's peer, with a sink of the program's output and a source of its input. The driver refuses a start that the guest would not decode, and a second start of an ID, before it sends anything, since the guest ends its session on either. Execution N's key is (B + N, E): E is the attachment's epoch and B the broker instance's ID, so no command repeats another's key, even after a broker restored from a snapshot starts its epochs over. The launch gains --workload-identity, the identity as which every execution of the sandbox runs. The client now also refuses the messages about executions that only the host sends. The VM scenarios run executions in the root over one session: output beyond a MiB, executions that meet only while all of them run, a descendant in a session of its own, the identities that starts name, root with one allowed capability, an exact environment, a working directory, the configured hostname, and the refusals of stage 0. Others run executions as the sandbox's workload identity, and fail the identity stage for one without an account. The in-memory tests of the host cover the driver's output, outcomes, refusals, and timers, and the in-process tests of the guest now drive the library's HostExecution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 217 ++++++--- .../src/bin/runtime-adapter-host.rs | 442 +++++++++++++++++- tests/runtime-adapter/src/exec.rs | 151 +++++- tests/runtime-adapter/src/host.rs | 13 +- tests/runtime-adapter/src/host/executions.rs | 210 +++++++++ tests/runtime-adapter/tests/guest_session.rs | 235 ++++------ tests/runtime-adapter/tests/host_session.rs | 405 +++++++++++++++- tests/runtime-adapter/tests/layered_launch.rs | 86 +++- tests/runtime-adapter/tests/vm_layered.rs | 266 ++++++++++- 9 files changed, 1768 insertions(+), 257 deletions(-) create mode 100644 tests/runtime-adapter/src/host/executions.rs diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 03f84aedd..f749bf514 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -18,16 +18,16 @@ through NVX's supervisor, and finds a VM again after it restarts, as an orchestrator does. A VM may have a layered root: an image under an upper, which the host program -stages and plans with NVX's typed plan for an adapter's own guest, whose -staged bindings a launch checks before it hands OpenVMM the checked files, and -which the guest agent's worker assembles through `nvx::guest::root`, in a -process of its own rather than PID 1, as the host program's root configuration -asks. The host program may share host directories and files with the root, -which it exports with NVX's `filesystem::plan`, and which the worker maps -beneath the root from the host's mapping table, before the host starts the -sandbox. The host program then checks the root from inside the VM with the -worker's probes, in a [table of scenarios](#layered-scenarios), to which each -new check adds a row. +stages and plans with NVX's typed plan for an adapter's own guest, whose staged +bindings a launch checks before it hands OpenVMM the checked files, and which +the guest agent's worker assembles through `nvx::guest::root`, in a process of +its own rather than PID 1, as the host program's root configuration asks. The +host program may share host directories and files with the root, which it +exports with NVX's `filesystem::plan`, and which the worker maps beneath the +root from the host's mapping table, before the host starts the sandbox. The host +program then checks the root from inside the VM with the worker's probes, and +runs executions in it, several at once over one session, in a +[table of scenarios](#layered-scenarios), to which each new check adds a row. ## Configurations @@ -309,6 +309,16 @@ bytes, 2 if standard error did, and 4 if the execution is reusable. Stage 0 is the worker's refusal of the start, and a later stage is the binding's number of the stage of `nvx::guest::spawn` that failed, such as 21 for its `exec`. +`exec::HostExecution` is the host's side of one execution, in every +configuration: the engine's `Peer`, with a sink that takes the program's output +as it arrives, unless its caller stalls it, so that the guest gets no credit +for more, and a source that hands over the program's input, if the start says +that the host sends it, then ends it. Its caller sends the start, hands it each +of the guest's messages about the execution with `deliver`, and sends the +messages that `serve` returns, until the peer finished, serving it again once +its `deadline`, the peer's next timer, passes. The host program's +[driver](#host-program) and the tests of the worker run it. + ### Root-profile entries The probe of `worker::entries`, which takes no arguments either, enters the @@ -675,10 +685,32 @@ with a new message, in a new attachment. A failure of the session, a send that fails or times out, a malformed message, an answer whose body does not decode, and a message that only the host program -sends end the client, and its later calls fail. A receive that times out -doesn't: it keeps what arrived. A request ends the client on any failure after -the framing accepted it, a timeout included, because an answer that arrived -later would answer the wrong request. +sends, an execution's start, input, end of input, or cancellation included, +end the client, and its later calls fail. A receive that times out doesn't: it +keeps what arrived. A request ends the client on any failure after the framing +accepted it, a timeout included, because an answer that arrived later would +answer the wrong request. + +`host::Executions` is the host program's driver of +[executions](#executions), over one attachment of the client: + +- `start` sends an execution's start, once it checks that the start decodes, + as the guest decodes it. A start that does not, such as one with a relative + working directory, and one of an ID that the driver started already, fail + before anything is sent, and leave the client usable: the guest would end its + session on either; +- `run` carries the guest's messages to each execution's + `exec::HostExecution`, and theirs to the guest, until every execution + finished. One thread runs them all: each receive waits at most until the + earliest timer of the executions, which so fires in time whatever the guest + sends, and no execution's messages wait for another's. A message of the + guest about no execution, about one that the driver did not start, whose body + does not decode, or that breaks the engine's rules, ends the run, as a + deadline that passes first does. + +The guest ends its session on a second start of the same key, even in a later +epoch of the session, so the keys of a driver's executions must be new to the +guest. ### The host's order @@ -949,7 +981,7 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from ends its VM. LAYERED, - `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [SHARES]`, + `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [--workload-identity UID:GID] [SHARES]`, gives the VM a [layered root](#layered-roots). Its image is the file PATH, an EROFS file system, `erofs`, or a GPT disk, `gpt`, of which the guest expects the identity UUID: the file system's UUID, or the disk's GUID. Its upper is @@ -958,11 +990,14 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from MODE, `trusted-lowers` or `untrusted-lowers`, checks a scratch's journal against the policy POLICY, `allowed` or `forbidden`, which only a scratch takes, and refuses a first partition larger than BYTES, which only a GPT disk - takes, with the guest's defaults unless given. Before it spawns the VM, the - launch checks the bindings that STORE stages for the image and the scratch - image, and fails if STORE stages none for either, or one no longer holds, as - [the checked handoff](#the-checked-handoff) describes. It records the - bindings that it checked. + takes, with the guest's defaults unless given. A workload identity, the user + ID UID and the group ID GID, two numbers from 1 to 4294967294, is the identity + as which every execution of the sandbox runs: the guest checks its account in + the assembled root, without which the root's `identity` stage fails. Before it + spawns the VM, the launch checks the bindings that STORE stages for the image + and the scratch image, and fails if STORE stages none for either, or one no + longer holds, as [the checked handoff](#the-checked-handoff) describes. It + records the bindings that it checked. SHARES, `--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]... [--share-root DIR]...`, @@ -1003,29 +1038,56 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from each PROBE in the VM's layered root, in order, and prints the facts that it found. A PROBE is the text of a [probe](#probes): its name, then its arguments, each after one space. A probe that fails fails the command. - -`reattach`, `stop`, and `probe` find a VM that another runtime identity -launched mismatched, and then signal nothing: the VM keeps running. The -commands print what they find as `KEY=VALUE` lines: `reattached` (`live`, -`mismatched`, `exited`, or `indeterminate`), `differences` (the parts of the -runtime identity in which a mismatched VM differs), `pid` (the VMM's process -ID), `instance` and `epoch` (the identity of the session that they attached -to), `root` (`ready` or `failed`: how the guest answered the root -configuration, or `failed` when a stop finds that the root's teardown failed), -`build` (the build of the guest agent that assembled the root), `stage` and -`problem` (the stage of the root that failed, such as `assembly`, and the -problem, or the problem of a mapping that failed), `table` (`applied` or -`failed`: how the guest took the mapping table), `mapped` (how many of the -table's mappings the guest applied), `sandbox` (`started` once the guest -started the sandbox), `PROBE.KEY` (the fact KEY that the probe named PROBE found), -`stopped` (`graceful`, `forced`, or `timed-out`), `outcome` (such as -`success 0`, or `guest-exit 14` once the assembly failed), and `teardown` -(`complete` or `incomplete`: whether the VMM's teardown of the VM completed, -which the outcome report records apart from the root's). Values escape `\`, newlines, and -carriage returns as `\\`, `\n`, and `\r`. The commands exit with 0 once they -did what they were asked, with 3 when they find the VM mismatched, with 4 when -the guest reported that a stage of the root failed, and the VM ended with the -stage's status and a complete teardown, with 5 when the +- `exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as + `reattach` does, then runs each EXECUTION in the VM's layered root, all at + once, over that one session, through `host::Executions`, and prints their + outputs and outcomes once every one finished. An EXECUTION, + `[--identity UID:GID [--groups GID,...] [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, + runs PROGRAM with its ARGs, up to a lone `;`, which a shell takes quoted, as + `find -exec` takes them, without standard input. Without `--identity`, it + names no identity, and runs as the sandbox's workload identity, which the + guest otherwise refuses; with it, it runs as the user ID UID and the group ID + GID, with the supplementary groups GIDs and the capabilities numbered N, such + as 0 for `CAP_CHOWN`, if given, and the guest checks its account, unless + `--numeric` says that the IDs have none. It starts in the root's directory + DIR, or `/`, with exactly the environment of its `--env` entries. It runs for + at most a minute, with 16 KiB chunks and the credit of 4 of them for its + input and 8 for its output, which its standard output and error share. + Execution N, from 1, has the key `(B + N, E)`: E is the epoch of the + command's attachment, and B the first 8 bytes of the broker instance's ID, as + a little-endian number. The guest refuses a key that it heard of before, and + the epochs of a broker instance only advance, while a broker restored from a + snapshot has a new instance ID, so no command repeats the key of another's + execution. + +`reattach`, `stop`, `probe`, and `exec` find a VM that another runtime identity +launched mismatched, and then signal nothing: the VM keeps running. The commands +print what they find as `KEY=VALUE` lines: `reattached` (`live`, `mismatched`, +`exited`, or `indeterminate`), `differences` (the parts of the runtime identity +in which a mismatched VM differs), `pid` (the VMM's process ID), `instance` and +`epoch` (the identity of the session that they attached to), `root` (`ready` or +`failed`: how the guest answered the root configuration, or `failed` when a stop +finds that the root's teardown failed), `build` (the build of the guest agent +that assembled the root), `stage` and `problem` (the stage of the root that +failed, such as `assembly`, and the problem, or the problem of a mapping that +failed), `table` (`applied` or `failed`: how the guest took the mapping table), +`mapped` (how many of the table's mappings the guest applied), `sandbox` +(`started` once the guest started the sandbox), `PROBE.KEY` (the fact KEY that +the probe named PROBE found), `exec.N.KEY` (what execution N did: its `launch`, +`started`, `failed STAGE ERRNO`, or `none`; its `status`, `exit CODE`, +`signal NUMBER`, or `none`; the `cause` for which a side ended it, such as +`timed-out`, or `none`; the streams that it `truncated`, such as +`stdout,stderr`, or `none`; whether it `settled`, `reusable` or `fenced`; and +its `stdout` and `stderr`), `stopped` (`graceful`, `forced`, or `timed-out`), +`outcome` (such as `success 0`, or `guest-exit 14` once the assembly failed), +and `teardown` (`complete` or `incomplete`: whether the VMM's teardown of the VM +completed, which the outcome report records apart from the root's). Values +escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each +byte of a program's output that is not part of UTF-8 text as `\xHH`. The +commands exit with 0 once they did what they were asked, `exec` once every +execution finished, whatever its outcome, with 3 when they find the VM +mismatched, with 4 when the guest reported that a stage of the root failed, and +the VM ended with the stage's status and a complete teardown, with 5 when the [step hook](#the-step-hook) ended a launch, with 6 when the guest failed a mapping of the table, and the VM stopped gracefully, with a success and a complete teardown, and with 1, after a diagnostic on standard error, when they @@ -1126,7 +1188,18 @@ Windows. They cover: nor `Start`, but `Stop`, and learns how the stop ended, with or without a teardown that failed, or as the session ends; and `Maps` or `Start` answered with another kind, or failures whose bodies do not decode, which end the - client. + client; +- the driver of executions, whose guest runs the runner of NVX's execution + engine for each start, as the worker does, without a program: two + executions at once, one with more output than its credit holds, each + byte-exact, with its outcome, and one whose launch the guest refuses, whose + records carry their messages whole or split; the starts that the guest would + not decode, and a second start of an ID, of which the driver sends nothing, + with a client that stays usable; a message about an execution that it did + not start, about none, whose body does not decode, or that breaks the + engine's rules, and the end of the stream, each of which ends the run; and a + timeout that fires while the guest says nothing, after which the execution + is cancelled, then fenced, and a run whose deadline passes first. Through a relay between its test broker and a pseudo-terminal, `tests/guest_session.rs` also checks that `guest::serve_terminal` and @@ -1171,19 +1244,19 @@ the binary exited, so that none of them outlives the test or holds its output open, which tests check too. `tests/guest_session.rs` also runs the worker's executions behind its test -broker, whose host side starts the sandbox, then drives the engine's peers, with -frame-counted credit and one output pool. The fake root prepares each -execution's entry, as the worker describes it, in a directory of the tests' own -whose account databases hold the accounts that the tests name, and then spawns -the program without entering a root, which needs privileges, in the tests' own -root and identity. Unless a test says otherwise, the sandbox has a workload -identity, as which every execution runs. Where the tests can create cgroups, as -root in a cgroup v2 hierarchy with the `pids` controller, the fake root gives -the worker a workload cgroup of the test's own, below which each execution runs -in a cgroup of its own, so that an execution whose program ends on its own -settles, and is reusable. Elsewhere, the fake root names no cgroups, and each -program runs in a process group alone, so that such an execution is fenced. The -tests cover: +broker, whose host side starts the sandbox, then drives each execution's +`exec::HostExecution`, with frame-counted credit and one output pool. The fake +root prepares each execution's entry, as the worker describes it, in a directory +of the tests' own whose account databases hold the accounts that the tests name, +and then spawns the program without entering a root, which needs privileges, in +the tests' own root and identity. Unless a test says otherwise, the sandbox has +a workload identity, as which every execution runs. Where the tests can create +cgroups, as root in a cgroup v2 hierarchy with the `pids` controller, the fake +root gives the worker a workload cgroup of the test's own, below which each +execution runs in a cgroup of its own, so that an execution whose program ends +on its own settles, and is reusable. Elsewhere, the fake root names no cgroups, +and each program runs in a process group alone, so that such an execution is +fenced. The tests cover: - three executions at once, each byte-exact: standard output beyond a MiB, 1.5 MiB of input through `cat`, and both streams of a program that fails; @@ -1313,7 +1386,8 @@ Windows: - `runtime-adapter-host launch` with shares that it cannot export, which it refuses with status 1 before it creates the store, with artifacts that boot nothing; -- the command lines that the host program refuses, each with status 2. +- the command lines that the host program refuses, each with status 2, those of + `exec` and of `--workload-identity` included. `tests/step_hook.rs` checks, in the host-only configuration, on Linux and Windows, the [step hook](#the-step-hook): the values of @@ -1408,22 +1482,25 @@ stages its images, then launches them, and is a row of the table that its | `unstaged_images_are_refused_before_spawn` | The image over the scratch, which the scenario does not stage | A launch that fails before it spawns the VM, since the store stages no image at its path, and that leaves no launch journal, VMM log, runtime record, or launch record; then, once the scenario stages the images, the root that the worker assembles, and a clean teardown | | `a_changed_image_is_refused_before_spawn` | The GPT disk over the scratch, which the scenario writes in place after it stages it | A launch that fails before it spawns the VM, since the disk's modification time differs from its binding's, and that leaves nothing of a launch in the store; then, once the scenario stages the disk again, the root, and a clean teardown | | `a_replaced_scratch_is_refused_before_spawn` | The image over the scratch, which another copy of the template replaces after the scenario stages it | A launch that fails before it spawns the VM, since the scratch image's identity differs from its binding's, and that leaves nothing of a launch in the store; then, once the scenario stages the images again, the root, and a clean teardown | +| `executions_run_in_the_root` | The image over the scratch, without a workload identity, in which `exec` runs executions that name their identities | Over one session, four executions as `nobody` that meet only while all of them run, beside one as root that writes more than a MiB to its standard output and a little to its standard error, each byte-exact, ending on its own with 0, and reusable. Over a later session, with new keys: a program that leaves a descendant in a session of its own, which the program's end, as PID 1 of its namespace, ends, so that no output is lost; `nobody`'s IDs and groups, with empty permitted, effective, and bounding sets; root with `CAP_CHOWN` alone in each of them; exactly the start's environment; the start's working directory; the configured hostname, `nvx-sandbox`, and PID 1; and a start without an identity, and one with `CAP_SYS_ADMIN`, which the guest refuses at stage 0 with `EINVAL` and `EPERM`, fenced and without output | +| `executions_run_as_the_workload_identity` | The image over the scratch, with the workload identity of `nobody` | Executions that name no identity, which run as `nobody`, without capabilities, with exactly the start's environment, and a start that names an identity, which the guest refuses at stage 0 with `EINVAL` | +| `accountless_workload_identity_fails_its_stage` | The image over the scratch, with a workload identity whose account the image lacks | The identity stage's failure once the guest assembled the root, which the launch reports, and a VM that ends with 16, which unmounted the scratch | A row names the root, the image, the GPT disk's top-layer variant, the identity that the launch expects of the image, the upper, the runtime directory, the overlay mode, the scratch's journal policy, the bound of the -first partition, and the host paths that the launch shares, and what the -scenario expects: a root that the guest assembles, with a check that probes -it, a teardown that fails after a check, the stage that fails, with its -problem, a mapping of the table that the guest fails, after how many, with its -problem, shares that the host program refuses before it spawns the VM, with -the problem, or a launch that the host program refuses before it spawns the -VM, with the change that the scenario makes after it stages the images, and -the problem. Each scenario that launches a VM checks the launch's record, with -the bindings that the store stages, which the launch checked, and the number -of mappings that the root configuration announced, and, over a scratch, -whether the VM left the scratch's journal to recover, which tells whether the -guest unmounted the scratch. +first partition, the sandbox's workload identity, and the host paths that the +launch shares, and what the scenario expects: a root that the guest assembles, +with a check that probes it or runs executions in it, a teardown that fails +after a check, the stage that fails, with its problem, a mapping of the table +that the guest fails, after how many, with its problem, shares that the host +program refuses before it spawns the VM, with the problem, or a launch that the +host program refuses before it spawns the VM, with the change that the scenario +makes after it stages the images, and the problem. Each scenario that launches a +VM checks the launch's record, with the bindings that the store stages, which +the launch checked, and the number of mappings that the root configuration +announced, and, over a scratch, whether the VM left the scratch's journal to +recover, which tells whether the guest unmounted the scratch. A scenario with the mapping table writes its sources in its directory, below `sources`: a directory `dir`, with the files `marker` and `secret` and an empty diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 29070500e..59709832e 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -21,20 +21,23 @@ //! //! LAYERED, `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) //! [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] -//! [--first-partition-bound BYTES] [SHARES]`, gives the VM a layered root, which the guest -//! agent's worker assembles; see `runtime_adapter::layered` and `runtime_adapter::worker`. Its -//! image is the raw file PATH, an EROFS filesystem, `erofs`, or a GPT disk, `gpt`, of which the -//! guest expects the identity UUID: the file system's UUID, or the disk's GUID. Its upper is the -//! ext4 scratch image PATH, or RAM. The guest assembles the root below its runtime directory -//! DIR, `/run/runtime-adapter` by default, in the overlay mode MODE, `trusted-lowers` or -//! `untrusted-lowers`, checks a scratch's journal against the policy POLICY, `allowed` or -//! `forbidden`, and refuses a first partition of a GPT disk larger than BYTES, with the guest's -//! defaults, `trusted-lowers`, `allowed`, and 64 KiB, unless given. Before it spawns the VM, the +//! [--first-partition-bound BYTES] [--workload-identity UID:GID] [SHARES]`, gives the VM a +//! layered root, which the guest agent's worker assembles; see `runtime_adapter::layered` and +//! `runtime_adapter::worker`. Its image is the raw file PATH, an EROFS filesystem, `erofs`, or a +//! GPT disk, `gpt`, of which the guest expects the identity UUID: the file system's UUID, or the +//! disk's GUID. Its upper is the ext4 scratch image PATH, or RAM. The guest assembles the root +//! below its runtime directory DIR, `/run/runtime-adapter` by default, in the overlay mode MODE, +//! `trusted-lowers` or `untrusted-lowers`, checks a scratch's journal against the policy POLICY, +//! `allowed` or `forbidden`, and refuses a first partition of a GPT disk larger than BYTES, with +//! the guest's defaults, `trusted-lowers`, `allowed`, and 64 KiB, unless given. A workload +//! identity, the user ID UID and the group ID GID, two numbers from 1 to 4294967294, is the +//! identity as which every execution of the sandbox runs: the guest checks its account in the +//! assembled root, without which the root's `identity` stage fails. Before it spawns the VM, the //! launch checks the bindings that STORE stages for the image and the scratch image, from their //! metadata alone, and fails if STORE stages none for either, or one no longer holds, as for an //! image that changed after it was staged. It hands OpenVMM the files that the checks opened, -//! which it holds until the guest has assembled the root, or until it has torn down a launch -//! that failed, and records the bindings. +//! which it holds until the guest has assembled the root, or until it has torn down a launch that +//! failed, and records the bindings. //! //! SHARES, `--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]... //! [--share-root DIR]...`, shares each host directory or regular file HOST with the root, at the @@ -69,6 +72,24 @@ //! runs each PROBE in the VM's layered root, and prints the facts that it found. A PROBE is the //! text of a probe: its name, then its arguments, each after one space; see //! `runtime_adapter::probe` and `runtime_adapter::worker::probes`. +//! - `runtime-adapter-host exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as +//! `reattach` does, then runs each EXECUTION in the VM's layered root, all at once, over that +//! one session, through `runtime_adapter::host::Executions`, and prints their outputs and +//! outcomes once every one finished. An EXECUTION, `[--identity UID:GID [--groups GID,...] +//! [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, +//! runs PROGRAM with its ARGs, up to a lone `;`, as `find -exec` takes them, without standard +//! input. Without `--identity`, it names no identity, and runs as the sandbox's workload +//! identity, which the guest otherwise refuses; with it, it runs as the user ID UID and the +//! group ID GID, with the supplementary groups GIDs and the capabilities numbered N, such as 0 +//! for `CAP_CHOWN`, if given, and the guest checks its account, unless `--numeric` says that +//! the IDs have none. It starts in the root's directory DIR, or `/`, with exactly the +//! environment of its `--env` entries. It runs for at most a minute, with 16 KiB chunks and the +//! credit of 4 of them for its input and 8 for its output, which its standard output and error +//! share. Execution N, from 1, has the key `(B + N, E)`: E is the epoch of the command's +//! attachment, and B the first 8 bytes of the broker instance's ID, as a little-endian number. +//! The guest refuses a key that it heard of before, and the epochs of a broker instance only +//! advance, while a broker restored from a snapshot has a new instance ID, so no command +//! repeats the key of another's execution. //! //! Each command but `stage` takes NVX's artifacts from the directory that `NVX_ARTIFACTS_DIR` //! names, a repository checkout or an extracted release, and the VM's runtime identity from NVX's @@ -93,23 +114,35 @@ //! - `mapped`: how many of the table's mappings the guest applied; //! - `sandbox`: `started` once the guest started the sandbox; //! - `PROBE.KEY`: the fact KEY that the probe named PROBE found; +//! - `exec.N.KEY`: what execution N did, with these KEYs: +//! - `launch`: `started`, `failed STAGE ERRNO`, with the number of the guest's stage that failed +//! and the error number, or `none` once the guest reported neither; +//! - `status`: how the program ended: `exit CODE`, `signal NUMBER`, or `none`; +//! - `cause`: why a side ended the execution, `cancelled`, `dropped`, `timed-out`, `reset`, or +//! `disconnected`, `other` for a cause that `nvx` adds later, or `none` for an execution that +//! ended on its own; +//! - `truncated`: the streams that lost bytes, `stdout`, `stderr`, or both, separated by a +//! comma, or `none`; +//! - `settled`: `reusable` once the execution settled, or `fenced`; +//! - `stdout` and `stderr`: the program's standard output and error; //! - `stopped`: how the stop ended: `graceful`, `forced`, or `timed-out`; //! - `outcome`: the category and status code of the VM's outcome report, such as `success 0`; //! - `teardown`: whether the VM's teardown completed: `complete` or `incomplete`. //! -//! Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`. +//! Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each byte of a +//! program's output that is not part of UTF-8 text as `\xHH`, with two hexadecimal digits. //! //! For tests, `launch` takes a step hook from the variable `RUNTIME_ADAPTER_STEP_HOOK`, which //! pauses the launch at a named step, for a time that the launch leaves out of its deadline, or //! exits it there without unwinding, with status 5; see `runtime_adapter::hook`. //! -//! They exit with 0 once they did what they were asked, with 3 when reattach finds the VM -//! mismatched, with 4 when the guest reported that a stage of the root failed, and the VM ended -//! with the stage's status and a complete teardown, with 5 when the step hook ends a launch, with -//! 6 when the guest failed a mapping of the table, and the VM stopped gracefully, with success and -//! a complete teardown, and with 1, after a diagnostic on standard error, when they fail. Anything -//! else is a usage error, with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no -//! step hook. +//! They exit with 0 once they did what they were asked, `exec` once every execution finished, +//! whatever its outcome, with 3 when reattach finds the VM mismatched, with 4 when the guest +//! reported that a stage of the root failed, and the VM ended with the stage's status and a +//! complete teardown, with 5 when the step hook ends a launch, with 6 when the guest failed a +//! mapping of the table, and the VM stopped gracefully, with success and a complete teardown, and +//! with 1, after a diagnostic on standard error, when they fail. Anything else is a usage error, +//! with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook. use std::env; use std::error::Error; @@ -118,6 +151,7 @@ use std::path::{Path, PathBuf}; use std::process::ExitCode; use std::time::{Duration, Instant}; +use nvx::exec::{Cause, Key, Limits, Outcome, Status, Stream, Timers, Window}; use nvx::host::artifacts::Artifacts; use nvx::host::instance::generate_capability; use nvx::host::launch::LaunchSpec; @@ -130,9 +164,10 @@ use nvx::protocol::root::ImageFormat; use nvx::uuid::Uuid; use nvx::{ErrorKind, Hypervisor}; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::frame::MAX_BODY; +use runtime_adapter::exec::{self, AccountMode, HostExecution, Start}; +use runtime_adapter::frame::{MAX_BODY, Message}; use runtime_adapter::hook::{Step, StepHook}; -use runtime_adapter::host::{self, Client, ClientError, OrderError}; +use runtime_adapter::host::{self, Client, ClientError, Executions, OrderError}; use runtime_adapter::initramfs; use runtime_adapter::layered::{LaunchRecord, LayeredLaunch}; use runtime_adapter::probe::{Probe, ProbeReply, escape}; @@ -164,6 +199,20 @@ const ROOT_FAILED: u8 = 4; /// The exit status of a launch whose guest failed a mapping of the table, once the VM stopped /// gracefully, with success and a complete teardown. const TABLE_FAILED: u8 = 6; +/// How long an execution of `exec` may run before its sides end it. +const EXEC_TIMEOUT: Duration = Duration::from_secs(60); +/// How long the pending output of an execution of `exec` still flows once a side ended it. +const EXEC_GRACE: Duration = Duration::from_secs(1); +/// How long an execution of `exec` may take to settle, once its program or a side ended it. +const EXEC_SETTLEMENT: Duration = Duration::from_secs(5); +/// The most bytes of a chunk of an execution of `exec`. +const EXEC_CHUNK: u32 = 16 * 1024; +/// The chunks of an execution's input that the guest may hold for `exec`. +const EXEC_INPUT_FRAMES: u32 = 4; +/// The chunks of an execution's output, of both streams, that `exec` may hold. +const EXEC_OUTPUT_FRAMES: u32 = 8; +/// The argument that ends an execution's arguments, after which another execution may follow. +const EXEC_END: &str = ";"; /// The command lines that the program accepts. const USAGE: &str = concat!( "usage: runtime-adapter-host\n", @@ -175,10 +224,15 @@ const USAGE: &str = concat!( " --upper (scratch:PATH | ram) [--runtime-dir DIR]\n", " [--overlay-mode MODE] [--scratch-journal POLICY]", " [--first-partition-bound BYTES]\n", + " [--workload-identity UID:GID]\n", " [--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]...", " [--share-root DIR]...]]\n", " runtime-adapter-host (reattach | stop) STORE [--guest-build BUILD]\n", - " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]", + " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]\n", + " runtime-adapter-host exec STORE [--guest-build BUILD] EXECUTION [';' EXECUTION]...\n", + " EXECUTION: [--identity UID:GID [--groups GID,...] [--capabilities N,...]", + " [--numeric]]\n", + " [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...", ); /// What the program was asked to do. @@ -231,6 +285,73 @@ enum Command { /// The probes, in order. probes: Vec, }, + /// Reattach to the VM, and run executions in its layered root. + Exec { + /// The sandbox's store. + store: PathBuf, + /// The guest agent's build. + guest_build: String, + /// The executions, in order. + executions: Vec, + }, +} + +/// An execution of `exec`, as its options and arguments give it. +#[derive(Default)] +struct Execution { + /// `--identity`: the user ID and the group ID. + identity: Option<(u32, u32)>, + /// `--groups`. + groups: Option>, + /// `--capabilities`: the set, a bit for each number. + capabilities: Option, + /// `--numeric`. + numeric: bool, + /// `--cwd`. + working_directory: Option, + /// `--env`, in order. + environment: Vec, + /// The program's path, then its arguments. + arguments: Vec, +} + +impl Execution { + /// Returns the start of the execution, as the execution `key`. + fn start(&self, key: Key) -> Start { + let arguments = self.arguments.iter().map(String::as_str); + let mut start = Start::new(key, exec_limits(), exec_timers(), arguments) + .with_environment(self.environment.iter().map(String::as_str)); + start.identity = self.identity.map(|(uid, gid)| exec::Identity { + groups: self.groups.clone().unwrap_or_default(), + capabilities: self.capabilities.unwrap_or(0), + account: if self.numeric { + AccountMode::Numeric + } else { + AccountMode::Existing + }, + ..exec::Identity::new(uid, gid) + }); + start.working_directory = self.working_directory.clone().map(String::into_bytes); + start + } +} + +/// Returns the limits of an execution of `exec`: chunks of [`EXEC_CHUNK`] bytes, and frame-counted +/// credit, of [`EXEC_INPUT_FRAMES`] for the input, and of [`EXEC_OUTPUT_FRAMES`] in one pool that +/// standard output and error share. +fn exec_limits() -> Limits { + Limits::new( + EXEC_CHUNK, + Window::frames(EXEC_INPUT_FRAMES), + Window::frames(EXEC_OUTPUT_FRAMES), + ) + .expect("the limits of `exec` hold no empty window, and a chunk that the framing carries") + .shared_output() +} + +/// Returns the timers of both sides of an execution of `exec`. +fn exec_timers() -> Timers { + Timers::new(EXEC_GRACE, EXEC_SETTLEMENT).timeout(EXEC_TIMEOUT) } /// The layered root of a launch, as its options give it. @@ -251,6 +372,8 @@ struct Layered { scratch_journal: Option, /// The most bytes that the first partition of a GPT disk may hold, unless the guest's default. first_partition_bound: Option, + /// The workload identity of the whole sandbox, if any: its user ID and its group ID. + workload_identity: Option<(u32, u32)>, /// The host paths that the launch shares with the root. sharing: HostFsSharing, } @@ -278,6 +401,8 @@ struct Options { scratch_journal: Option, /// `--first-partition-bound`. first_partition_bound: Option, + /// `--workload-identity`. + workload_identity: Option<(u32, u32)>, /// `--share`, `--deny`, `--reserved`, and `--share-root`, in order. sharing: HostFsSharing, } @@ -331,6 +456,11 @@ fn main() -> ExitCode { guest_build, probes, } => probe(&store, &guest_build, &probes), + Command::Exec { + store, + guest_build, + executions, + } => exec(&store, &guest_build, &executions), }; result.unwrap_or_else(|problem| { eprintln!("runtime-adapter-host: {problem}"); @@ -346,7 +476,9 @@ fn parse(arguments: Vec) -> Result { return Ok(Command::Report); }; let name = match name.to_str() { - Some(name @ ("stage" | "launch" | "reattach" | "stop" | "probe")) => name.to_owned(), + Some(name @ ("stage" | "launch" | "reattach" | "stop" | "probe" | "exec")) => { + name.to_owned() + } _ => return Err(format!("unknown command {name:?}")), }; let store = PathBuf::from( @@ -354,6 +486,9 @@ fn parse(arguments: Vec) -> Result { .next() .ok_or_else(|| format!("{name} needs a sandbox store"))?, ); + if name == "exec" { + return parse_exec(store, arguments); + } let mut options = Options::default(); let mut probes = Vec::new(); while let Some(argument) = arguments.next() { @@ -419,6 +554,7 @@ fn parse(arguments: Vec) -> Result { overlay_mode: options.overlay_mode, scratch_journal: options.scratch_journal, first_partition_bound: options.first_partition_bound, + workload_identity: options.workload_identity, sharing: options.sharing, })) } @@ -435,6 +571,7 @@ fn parse(arguments: Vec) -> Result { "--first-partition-bound", options.first_partition_bound.is_some(), ), + ("--workload-identity", options.workload_identity.is_some()), ("--share", !options.sharing.shares().is_empty()), ]; if let Some((option, _)) = layered.iter().find(|(_, given)| *given) { @@ -540,6 +677,17 @@ fn parse_option( format!("--first-partition-bound takes a positive number of bytes, not {value:?}") })?); } + ("launch", "--workload-identity") if options.workload_identity.is_none() => { + let value = text(value)?; + let identity = ids(&value) + .filter(|(uid, gid)| [uid, gid].into_iter().all(|id| (1..u32::MAX).contains(id))); + options.workload_identity = Some(identity.ok_or_else(|| { + format!( + "--workload-identity takes UID:GID, two numbers from 1 to {}, not {value:?}", + u32::MAX - 1 + ) + })?); + } ("launch", "--share") => { let value = text(value)?; let share = Share::parse(&value).ok_or_else(|| { @@ -561,6 +709,128 @@ fn parse_option( Ok(()) } +/// Parses the arguments of `exec` after its sandbox store `store`: its executions, each ended by +/// a lone `;` or the end of the arguments, with the options before them, or says why they are not +/// the command's. +fn parse_exec( + store: PathBuf, + arguments: impl Iterator, +) -> Result { + let mut arguments = arguments.map(|argument| { + argument + .into_string() + .map_err(|argument| format!("the argument {argument:?} of exec is not Unicode")) + }); + let mut guest_build = None; + let mut executions = Vec::new(); + loop { + let mut execution = Execution::default(); + loop { + let option = arguments + .next() + .ok_or("exec needs `-- PROGRAM` after the options of each execution")??; + match option.as_str() { + "--" => break, + "--numeric" if !execution.numeric => { + execution.numeric = true; + continue; + } + _ => {} + } + let value = arguments + .next() + .ok_or_else(|| format!("{option:?} needs a value"))??; + match option.as_str() { + "--guest-build" if guest_build.is_none() => guest_build = Some(value), + "--identity" if execution.identity.is_none() => { + let identity = ids(&value) + .ok_or_else(|| format!("--identity takes UID:GID, not {value:?}"))?; + execution.identity = Some(identity); + } + "--groups" if execution.groups.is_none() => { + let groups = + numbers(&value, |group| group.parse::().ok()).ok_or_else(|| { + format!("--groups takes GIDs, separated by commas, not {value:?}") + })?; + execution.groups = Some(groups); + } + "--capabilities" if execution.capabilities.is_none() => { + let capabilities = numbers(&value, |number| { + number.parse::().ok().filter(|number| *number < 64) + }) + .ok_or_else(|| { + format!( + "--capabilities takes capability numbers from 0 to 63, separated by \ + commas, not {value:?}" + ) + })?; + let set = capabilities + .into_iter() + .fold(0, |set, number| set | 1 << number); + execution.capabilities = Some(set); + } + "--cwd" if execution.working_directory.is_none() => { + execution.working_directory = Some(value); + } + "--env" => execution.environment.push(value), + _ => return Err(format!("unexpected option {option:?} of exec")), + } + } + let mut more = false; + for argument in arguments.by_ref() { + let argument = argument?; + if argument == EXEC_END { + more = true; + break; + } + execution.arguments.push(argument); + } + if execution.arguments.is_empty() { + return Err("exec needs a program after `--`".to_owned()); + } + if execution.identity.is_none() { + let qualifiers = [ + ("--groups", execution.groups.is_some()), + ("--capabilities", execution.capabilities.is_some()), + ("--numeric", execution.numeric), + ]; + if let Some((option, _)) = qualifiers.iter().find(|(_, given)| *given) { + return Err(format!("{option} needs --identity")); + } + } + // The guest ends its session on a start that it cannot decode, so the program refuses + // one that breaks the rules of the encoding, as a relative working directory or an + // environment entry without a name does, or that the framing does not carry, before it + // attaches. + let encoded = execution.start(Key::new(0, 0)).encode(); + Start::decode(&encoded).map_err(|error| error.problem().to_owned())?; + Message::ExecStart(encoded) + .encode() + .map_err(|error| format!("the start of an execution is too long: {error}"))?; + executions.push(execution); + if !more { + break; + } + } + Ok(Command::Exec { + store, + guest_build: guest_build.unwrap_or_else(|| GUEST_BUILD.to_owned()), + executions, + }) +} + +/// Returns the user ID and the group ID of `value`, `UID:GID`, or `None` if it is not that. +fn ids(value: &str) -> Option<(u32, u32)> { + let (uid, gid) = value.split_once(':')?; + Some((uid.parse().ok()?, gid.parse().ok()?)) +} + +/// Returns the numbers of `value`, separated by commas, which `parse` reads, or `None` if one is +/// not a number that it accepts, or `value` is empty. +fn numbers(value: &str, parse: impl Fn(&str) -> Option) -> Option> { + value.split(',').map(parse).collect() +} + /// Stages the image `distro`, in `format`, of which the guest expects `identity`, and the ext4 /// scratch image `scratch`, if any, in the sandbox store `store`, which it creates if needed. fn stage( @@ -616,6 +886,7 @@ fn launch( launch.config.overlay_mode = layered.overlay_mode; launch.config.scratch_journal = layered.scratch_journal; launch.config.first_partition_bound = layered.first_partition_bound; + launch.config.workload_identity = layered.workload_identity; // The launch refuses shares that it cannot export before it touches the store. if let Some(export) = layered.sharing.plan()? { launch.share(export)?; @@ -970,6 +1241,131 @@ fn probe(store: &Path, guest_build: &str, probes: &[Probe]) -> Result Result { + let runtime = runtime(guest_build)?; + let store = SandboxStore::open(store).map_err(|error| describe(&error))?; + let mut locked = store + .lock(Some(Instant::now() + LOCK_TIMEOUT)) + .map_err(|error| describe(&error))?; + let vmm = match hold(&mut locked, &runtime)? { + Ok(vmm) => vmm, + Err(status) => return Ok(status), + }; + let deadline = Instant::now() + SESSION_TIMEOUT; + let capability = store.read_capability().map_err(|error| describe(&error))?; + let mut client = connect(&vmm, &capability, deadline)?; + // The keys of the attachment's executions are new to the guest: see the module documentation. + let identity = client.identity(); + let (base, _) = identity.instance_id.split_at(8); + let base = u64::from_le_bytes(base.try_into().expect("an instance ID has 16 bytes")); + let mut running = Executions::new(); + let mut ids = Vec::new(); + for (number, execution) in (1..).zip(executions) { + let key = Key::new(base.wrapping_add(number), identity.epoch); + running + .start( + &mut client, + execution.start(key), + exec_timers(), + Vec::new(), + deadline, + ) + .map_err(|error| format!("cannot start execution {number}: {}", describe(&error)))?; + ids.push(key.id()); + } + // Every execution finishes once it ended, which its timeout bounds, and its output's grace and + // its settlement passed, or once the guest's last messages about it arrived. + let deadline = Instant::now() + EXEC_TIMEOUT + EXEC_GRACE + EXEC_SETTLEMENT + SESSION_TIMEOUT; + running + .run(&mut client, deadline) + .map_err(|error| format!("the executions failed: {}", describe(&error)))?; + for (number, id) in (1..).zip(ids) { + let execution = running.get(id).expect("the driver started every execution"); + print_execution(number, execution); + } + Ok(ExitCode::SUCCESS) +} + +/// Prints what the execution numbered `number`, `execution`, which finished, did. +fn print_execution(number: u64, execution: &HostExecution) { + let Some(outcome) = execution.outcome() else { + unreachable!("every execution finished"); + }; + let launch = match outcome.launch_failure() { + Some(failure) => format!("failed {} {}", failure.stage(), failure.error()), + None if execution.peer().has_started() => "started".to_owned(), + None => "none".to_owned(), + }; + let status = match outcome.status() { + Some(Status::Exited(code)) => format!("exit {code}"), + Some(Status::Signaled(signal)) => format!("signal {signal}"), + None => "none".to_owned(), + }; + let truncated = Stream::ALL + .into_iter() + .filter(|stream| outcome.is_truncated(*stream)) + .map(stream_name) + .collect::>(); + let settled = if outcome.is_reusable() { + "reusable" + } else { + "fenced" + }; + println!("exec.{number}.launch={launch}"); + println!("exec.{number}.status={status}"); + println!("exec.{number}.cause={}", cause_name(&outcome)); + println!( + "exec.{number}.truncated={}", + if truncated.is_empty() { + "none".to_owned() + } else { + truncated.join(",") + } + ); + println!("exec.{number}.settled={settled}"); + for stream in Stream::ALL { + let output = escape_bytes(execution.output(stream)); + println!("exec.{number}.{}={output}", stream_name(stream)); + } +} + +/// Returns the name of why a side ended the execution whose outcome is `outcome`, or `none`. +fn cause_name(outcome: &Outcome) -> &'static str { + match outcome.cause() { + None => "none", + Some(Cause::Cancelled) => "cancelled", + Some(Cause::Dropped) => "dropped", + Some(Cause::TimedOut) => "timed-out", + Some(Cause::Reset) => "reset", + Some(Cause::Disconnected) => "disconnected", + Some(_) => "other", + } +} + +/// Returns the name of `stream`: `stdout` or `stderr`. +fn stream_name(stream: Stream) -> &'static str { + match stream { + Stream::Stdout => "stdout", + Stream::Stderr => "stderr", + } +} + +/// Returns `bytes` as a value: the UTF-8 text that they hold, escaped as [`escape`] escapes it, and +/// each other byte as `\xHH`, with two hexadecimal digits. +fn escape_bytes(bytes: &[u8]) -> String { + let mut escaped = String::with_capacity(bytes.len()); + for chunk in bytes.utf8_chunks() { + escaped.push_str(&escape(chunk.valid())); + for byte in chunk.invalid() { + escaped.push_str(&format!("\\x{byte:02x}")); + } + } + escaped +} + /// Reattaches to the VM of `locked`, with the runtime identity of `runtime`, and prints what it /// found. Returns the VM, or the status with which the command ends once it finds the VM /// mismatched, which it leaves running. diff --git a/tests/runtime-adapter/src/exec.rs b/tests/runtime-adapter/src/exec.rs index 083061453..05aaab117 100644 --- a/tests/runtime-adapter/src/exec.rs +++ b/tests/runtime-adapter/src/exec.rs @@ -1,7 +1,9 @@ //! The adapter's messages about executions: programs that the guest runs for the host, whose //! standard streams and outcome the execution engine of `nvx::exec` carries. The host's side of an //! execution is the engine's [`Peer`], and the guest's side its [`Runner`], which the guest agent's -//! worker binds to the program through `nvx::guest::exec::binding`. +//! worker binds to the program through `nvx::guest::exec::binding`. A [`HostExecution`] holds the +//! peer with a sink of the program's output and a source of its input, as the host program's +//! driver, `host::Executions`, and the in-process tests of the worker run it. //! //! Each message of the engine travels in a frame of its own kind. Its body starts with the //! execution's [`Key`]: the execution's ID, then its generation, each a `u64`: @@ -575,6 +577,153 @@ impl<'a> GuestMessage<'a> { } } +/// The host's side of an execution: its [`Peer`], with a sink that takes the program's output as +/// it arrives, unless its caller stalls it, and a source that hands over the program's input, if +/// the start says that the host sends it, then ends it. +/// +/// Its caller sends the execution's [`Start`], hands it each of the guest's messages about the +/// execution with [`deliver`](Self::deliver), and sends the frames that [`serve`](Self::serve) +/// returns, until the peer [finished](Self::outcome). The host program's driver, +/// `host::Executions`, runs several at once over the session. +#[derive(Debug)] +pub struct HostExecution { + /// The execution's start. + start: Start, + /// The host's side of the execution. + peer: Peer, + /// Whether the sink takes nothing. + stalled: bool, + /// The bytes of each stream that the sink took. + output: [Vec; 2], + /// The input that the source hands over, if the start says that the host sends it. + input: Vec, + /// How many bytes of the input the source handed over. + sent: usize, + /// Whether the source ended the input. + ended: bool, + /// The peer's events, in order. + events: Vec, +} + +impl HostExecution { + /// Returns the host's side of the execution of `start`, created at `now`, from which the + /// timeout of its peer's `timers` runs, with a source of `input`. + pub fn new(start: Start, timers: Timers, input: Vec, now: Instant) -> Self { + let peer = Peer::new(start.key, start.limits, timers, now); + Self { + start, + peer, + stalled: false, + output: [Vec::new(), Vec::new()], + input, + sent: 0, + ended: false, + events: Vec::new(), + } + } + + /// Returns the execution's start. + pub fn start(&self) -> &Start { + &self.start + } + + /// Returns the host's side of the execution. + pub fn peer(&self) -> &Peer { + &self.peer + } + + /// Returns the host's side of the execution, with which the caller ends it, or tells it that + /// the session was reset or lost. + pub fn peer_mut(&mut self) -> &mut Peer { + &mut self.peer + } + + /// Makes the sink take nothing while `stalled` holds, so that the program's output waits in + /// the peer, which grants the guest no credit for it. + pub fn stall(&mut self, stalled: bool) { + self.stalled = stalled; + } + + /// Returns the bytes of `stream` that the sink took. + pub fn output(&self, stream: Stream) -> &[u8] { + &self.output[position(stream)] + } + + /// Returns the peer's events, in order. + pub fn events(&self) -> &[peer::Event] { + &self.events + } + + /// Returns the execution's outcome, once the peer finished. + pub fn outcome(&self) -> Option { + self.events.iter().find_map(|event| match event { + peer::Event::Finished(outcome) => Some(*outcome), + _ => None, + }) + } + + /// Returns when the peer's next timer fires, for the caller to [serve](Self::serve) the + /// execution then, or `None` while no timer runs. + pub fn deadline(&self) -> Option { + self.peer.deadline() + } + + /// Hands the guest's `message` about the execution to the peer at `now`, and returns what the + /// peer returns. + pub fn deliver( + &mut self, + message: &GuestMessage<'_>, + now: Instant, + ) -> Result { + message.deliver(&mut self.peer, now) + } + + /// Takes every step that the source, the sink, and the peer can take at `now`, and returns + /// the peer's messages for the guest, as frames, in order. + pub fn serve(&mut self, now: Instant) -> Vec { + if self.start.input && !self.ended { + self.sent += self.peer.input(&self.input[self.sent..]); + if self.sent == self.input.len() { + self.peer.end_input(); + self.ended = true; + } + } + if !self.stalled { + for stream in Stream::ALL { + loop { + let pending = self.peer.pending_output(stream); + if pending.is_empty() { + break; + } + let length = pending.len(); + self.output[position(stream)].extend_from_slice(pending); + self.peer.consumed(stream, length); + } + } + } + self.peer.tick(now); + while let Some(event) = self.peer.next_event() { + self.events.push(event); + } + let mut buffer = vec![0; self.start.limits.chunk()]; + let mut messages = Vec::new(); + while let Some(message) = self.peer.next_message(&mut buffer) { + if let Some(message) = HostMessage::from_peer(self.start.key, message, &buffer) { + messages.push(message.encode()); + } + } + messages + } +} + +/// Returns the position of `stream` among [`Stream::ALL`]. +fn position(stream: Stream) -> usize { + match stream { + Stream::Stdout => 0, + Stream::Stderr => 1, + } +} + /// Returns `key` as a name for problems: its ID, then its generation. pub fn name(key: Key) -> String { format!("{}.{}", key.id(), key.generation()) diff --git a/tests/runtime-adapter/src/host.rs b/tests/runtime-adapter/src/host.rs index 84ba3103a..fb79445d9 100644 --- a/tests/runtime-adapter/src/host.rs +++ b/tests/runtime-adapter/src/host.rs @@ -1,7 +1,8 @@ //! The adapter's host program: a client of the guest agent that attaches to the host's end of the //! control session between the host and the guest, and exchanges the adapter's own //! [framing](crate::frame) with the guest agent, through `nvx::host::nvxs` alone, over any platform -//! transport. +//! transport, and the driver of the [executions](crate::exec) that the host program runs over it, +//! [`Executions`]. //! //! # The host's order //! @@ -22,6 +23,7 @@ //! waits for the VM to end, and ends a VM whose guest did not stop, as it ends one after any other //! failure of the order, an [`OrderError`]. +mod executions; mod map_reply; mod order_error; @@ -37,6 +39,7 @@ use crate::frame::{BodyError, Decoder, FrameError, Message}; use crate::probe::{Facts, Probe, ProbeReply}; use crate::root::{MapFailure, Ready, RootConfig, RootFailure, RootReply, encode_maps}; +pub use executions::{Executions, ExecutionsError}; pub use map_reply::MapReply; pub use order_error::OrderError; @@ -117,7 +120,7 @@ impl From for ClientError { /// returns the guest agent's next message, whether it spans records or shares one with the next /// message. [`ping`](Self::ping), [`echo`](Self::echo), [`configure_root`](Self::configure_root), /// [`map`](Self::map), [`start`](Self::start), [`probe`](Self::probe), and [`stop`](Self::stop) -/// each send a request and wait for its answer. +/// each send a request and wait for its answer, and [`Executions`] runs executions over the client. /// /// Besides the session's own buffers, the client holds at most the rest of one of the guest's /// records and one partial message, whose header [`Decoder`] checks before it takes any byte of @@ -208,7 +211,11 @@ impl Client { | Message::RootConfig(_) | Message::Probe(_) | Message::Maps(_) - | Message::Start), + | Message::Start + | Message::ExecStart(_) + | Message::ExecInput(_) + | Message::ExecInputEnd(_) + | Message::ExecCancel(_)), ) => { return Err(self.fail(ClientError::Unexpected(message.kind()))); } diff --git a/tests/runtime-adapter/src/host/executions.rs b/tests/runtime-adapter/src/host/executions.rs new file mode 100644 index 000000000..0c26e4395 --- /dev/null +++ b/tests/runtime-adapter/src/host/executions.rs @@ -0,0 +1,210 @@ +//! The host program's driver of executions: the host's side of each, a [`HostExecution`], over one +//! attachment of a [`Client`] to the control session. + +use std::collections::BTreeMap; +use std::fmt; +use std::time::Instant; + +use nvx::exec::{Key, Timers, Violation}; +use nvx::host::nvxs::HostConnectionError; + +use super::{Client, ClientError}; +use crate::exec::{GuestMessage, HostExecution, Start, name}; +use crate::frame::{BodyError, Message}; + +/// A failure of [`Executions`]. +#[derive(Debug)] +#[non_exhaustive] +pub enum ExecutionsError { + /// The client failed. + Client(ClientError), + /// A start whose encoding does not decode, which the driver did not send. + Start(BodyError), + /// A start of an execution whose ID the driver started already, which it did not send. + Duplicate(u64), + /// The guest sent a message about an execution whose body does not decode. + Body(BodyError), + /// The guest sent a message of this kind, which is about no execution. + Unexpected(u8), + /// The guest sent a message about an execution that the driver did not start. + Unknown(Key), + /// The host's side of the execution refused the guest's message about it. + Violation(Key, Violation), + /// The deadline passed before every execution finished. + TimedOut, +} + +impl fmt::Display for ExecutionsError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Client(error) => error.fmt(formatter), + Self::Start(error) => write!(formatter, "the host refuses to send the start: {error}"), + Self::Duplicate(id) => write!(formatter, "the host starts execution {id} twice"), + Self::Body(error) => error.fmt(formatter), + Self::Unexpected(kind) => write!( + formatter, + "the guest sent a message of kind {kind}, which is about no execution" + ), + Self::Unknown(key) => write!( + formatter, + "the guest sent a message about execution {}, which the host did not start", + name(*key) + ), + Self::Violation(key, violation) => write!( + formatter, + "the guest's message about execution {} breaks the engine's rules: {violation}", + name(*key) + ), + Self::TimedOut => formatter.write_str("the executions did not finish in time"), + } + } +} + +impl std::error::Error for ExecutionsError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Client(error) => Some(error), + Self::Start(error) | Self::Body(error) => Some(error), + Self::Violation(_, violation) => Some(violation), + Self::Duplicate(_) | Self::Unexpected(_) | Self::Unknown(_) | Self::TimedOut => None, + } + } +} + +impl From for ExecutionsError { + fn from(error: ClientError) -> Self { + Self::Client(error) + } +} + +/// The host's side of the executions that the host program runs over one attachment of the control +/// session: a [`HostExecution`] for each, by ID, whose sink takes the program's output as it +/// arrives. +/// +/// [`start`](Self::start) sends the start of an execution, and [`run`](Self::run) carries the +/// guest's messages to the executions, and theirs to the guest, until every execution finished. +/// One thread runs them all: each receive waits at most until the earliest timer of the +/// executions, which so fires in time whatever the guest sends, and the guest's messages about one +/// execution never wait for another's. +/// +/// The guest ends its session on a second start of the same key, even in a later epoch of the +/// session, so the keys of the executions must differ from those of every execution that the guest +/// heard of before. A failure ends the run, and leaves the client as its own failures do. +#[derive(Debug, Default)] +pub struct Executions { + /// The host's side of each execution, by ID. + executions: BTreeMap, +} + +impl Executions { + /// Returns a driver without executions. + pub fn new() -> Self { + Self::default() + } + + /// Starts the execution of `start` over `client`, whose peer runs with `timers`, and whose + /// source hands over `input` if the start says that the host sends the program's input: sends + /// the start by `deadline`. The timeout of `timers` runs from then. + /// + /// A start of an ID that the driver started already, or whose encoding does not decode, as + /// one with a relative working directory or an environment entry without a name, fails + /// before anything is sent, and leaves the client usable, as a message that the framing + /// refuses does: the guest would end its session on either. + pub fn start( + &mut self, + client: &mut Client, + start: Start, + timers: Timers, + input: Vec, + deadline: Instant, + ) -> Result<(), ExecutionsError> { + let id = start.key.id(); + if self.executions.contains_key(&id) { + return Err(ExecutionsError::Duplicate(id)); + } + let body = start.encode(); + Start::decode(&body).map_err(ExecutionsError::Start)?; + let execution = HostExecution::new(start, timers, input, Instant::now()); + client.send(&Message::ExecStart(body), Some(deadline))?; + self.executions.insert(id, execution); + Ok(()) + } + + /// Carries the guest's messages to the executions, and theirs to the guest, over `client`, + /// until every execution finished, and fails with [`ExecutionsError::TimedOut`] once + /// `deadline` passed first. + pub fn run(&mut self, client: &mut Client, deadline: Instant) -> Result<(), ExecutionsError> { + loop { + let now = Instant::now(); + for execution in self.executions.values_mut() { + for message in execution.serve(now) { + client.send(&message, Some(deadline)).map_err(failed)?; + } + } + if self.is_finished() { + return Ok(()); + } + if now >= deadline { + return Err(ExecutionsError::TimedOut); + } + let wake = self + .executions + .values() + .filter_map(HostExecution::deadline) + .fold(deadline, Instant::min); + match client.receive(Some(wake)) { + Ok(message) => self.dispatch(&message, Instant::now())?, + Err(ClientError::Session(HostConnectionError::TimedOut)) => {} + Err(error) => return Err(error.into()), + } + } + } + + /// Returns the host's side of the execution `id`, if the driver started it. + pub fn get(&self, id: u64) -> Option<&HostExecution> { + self.executions.get(&id) + } + + /// Returns the host's side of the execution `id`, if the driver started it, with which the + /// caller may end it. + pub fn get_mut(&mut self, id: u64) -> Option<&mut HostExecution> { + self.executions.get_mut(&id) + } + + /// Returns the host's side of each execution, in the order of their IDs. + pub fn iter(&self) -> impl Iterator { + self.executions.values() + } + + /// Returns whether every execution finished. + pub fn is_finished(&self) -> bool { + self.executions + .values() + .all(|execution| execution.outcome().is_some()) + } + + /// Hands the guest's `message` to the host's side of its execution at `now`. + fn dispatch(&mut self, message: &Message, now: Instant) -> Result<(), ExecutionsError> { + let decoded = GuestMessage::decode(message) + .ok_or(ExecutionsError::Unexpected(message.kind()))? + .map_err(ExecutionsError::Body)?; + let key = decoded.key(); + let execution = self + .executions + .get_mut(&key.id()) + .ok_or(ExecutionsError::Unknown(key))?; + execution + .deliver(&decoded, now) + .map_err(|violation| ExecutionsError::Violation(key, violation))?; + Ok(()) + } +} + +/// Returns the failure of a run whose send failed with `error`: [`ExecutionsError::TimedOut`] +/// once the send's deadline, the run's, passed. +fn failed(error: ClientError) -> ExecutionsError { + match error { + ClientError::Session(HostConnectionError::TimedOut) => ExecutionsError::TimedOut, + error => ExecutionsError::Client(error), + } +} diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 556bd7fb7..1382fabb1 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -22,21 +22,21 @@ //! signals leave, whatever the names of its children. //! //! The worker's executions run behind the test broker too, whose host side drives the engine's -//! peers: the tests cover several executions at once, byte-exact, a stalled sink and a full -//! stream to the host, which stall only what waits on them, executions that take turns at the -//! stream, timeouts, cancellations, and stops, the starts, messages, and probes that the worker -//! refuses, generations and resets, the descendants that a program leaves in its process group, -//! and executions on a tty. The fake root prepares each execution's entry, as the worker -//! describes it, in a directory of the tests' own that holds account databases, and then spawns -//! the program without entering a root, which needs privileges, in the tests' own root and -//! identity: the tests cover the identity that exactly one source names, the sandbox's or the -//! start's, the entry's inputs, a program's working directory and environment, and the starts -//! that the worker refuses for their identities. Where the tests can create cgroups, each fake -//! root gives the worker a workload cgroup of the test's own, below which the executions run in -//! cgroups of their own, and settle: the tests then also cover a descendant that leaves its -//! program's process group, which the settlement ends, and an execution that does not settle, -//! which keeps its place until the host stops the worker. Elsewhere, the programs run in process -//! groups alone. +//! peers through the host program's own `exec::HostExecution`: the tests cover several executions +//! at once, byte-exact, a stalled sink and a full stream to the host, which stall only what waits +//! on them, executions that take turns at the stream, timeouts, cancellations, and stops, the +//! starts, messages, and probes that the worker refuses, generations and resets, the descendants +//! that a program leaves in its process group, and executions on a tty. The fake root prepares +//! each execution's entry, as the worker describes it, in a directory of the tests' own that holds +//! account databases, and then spawns the program without entering a root, which needs +//! privileges, in the tests' own root and identity: the tests cover the identity that exactly one +//! source names, the sandbox's or the start's, the entry's inputs, a program's working directory +//! and environment, and the starts that the worker refuses for their identities. Where the tests +//! can create cgroups, each fake root gives the worker a workload cgroup of the test's own, below +//! which the executions run in cgroups of their own, and settle: the tests then also cover a +//! descendant that leaves its program's process group, which the settlement ends, and an execution +//! that does not settle, which keeps its place until the host stops the worker. Elsewhere, the +//! programs run in process groups alone. #![cfg(all(feature = "guest", target_os = "linux"))] use std::collections::BTreeMap; @@ -61,9 +61,7 @@ use nix::pty; use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, Id, WaitPidFlag, WaitStatus}; use nix::unistd::Pid; -use nvx::exec::{ - Cause, Key, LaunchFailure, Limits, Outcome, Peer, Pool, Status, Stream, Timers, Window, peer, -}; +use nvx::exec::{Cause, Key, LaunchFailure, Limits, Outcome, Pool, Status, Stream, Timers, Window}; use nvx::guest::cgroup::Layout; use nvx::guest::entry::{self, Capabilities, Capability, Prepared}; use nvx::guest::mappings::{Access, Mapping}; @@ -79,7 +77,9 @@ use nvx::protocol::{ }; use nvx::uuid::Uuid; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::exec::{self, AccountMode, GuestMessage, HostMessage, MAX_CHUNK, Start}; +use runtime_adapter::exec::{ + self, AccountMode, GuestMessage, HostExecution, HostMessage, MAX_CHUNK, Start, +}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, Message}; use runtime_adapter::guest::{self, NO_ROOT, RECEIVE_WINDOW, ServeError, Served}; use runtime_adapter::probe::{Facts, Probe}; @@ -2649,14 +2649,6 @@ fn lines(first: usize, last: usize) -> Vec { .into_bytes() } -/// Returns `stream`'s position in [`Stream::ALL`]. -fn position(stream: Stream) -> usize { - match stream { - Stream::Stdout => 0, - Stream::Stderr => 1, - } -} - /// Waits until no process of the process group `group` runs, as once an execution ended the /// program that leads it. fn gone(group: u32) { @@ -2699,93 +2691,24 @@ fn reaped(id: u32) { } } -/// The host's side of one of the worker's executions in a test: its peer, with a sink that takes -/// the output unless it stalls, and a source that hands over the input, then ends it. -struct HostExecution { - /// The execution's start. - start: Start, - /// The host's side of the execution. - peer: Peer, - /// Whether the sink takes nothing. - stalled: bool, - /// The bytes of each stream that the sink took. - output: [Vec; 2], - /// The input that the source hands over, if the start says that the host sends it. - input: Vec, - /// How many bytes of the input the source handed over. - sent: usize, - /// Whether the source ended the input. - ended: bool, - /// The peer's events, in order. - events: Vec, -} - -impl HostExecution { - /// Returns the host's side of the execution of `start`, created at `now`, with a source of - /// `input`. - fn new(start: Start, input: Vec, now: Instant) -> Self { - let peer = Peer::new(start.key, start.limits, timers(), now); - Self { - start, - peer, - stalled: false, - output: [Vec::new(), Vec::new()], - input, - sent: 0, - ended: false, - events: Vec::new(), - } - } +/// What the tests read of the host's side of one of the worker's executions, a +/// [`HostExecution`], besides what it tells itself. +trait Taken { + /// Returns the bytes of each stream that the sink took, in the order of [`Stream::ALL`]. + fn outputs(&self) -> [Vec; 2]; - /// Takes every step that the source, the sink, and the peer can take at `now`, and returns - /// the peer's messages for the guest. - fn serve(&mut self, now: Instant) -> Vec { - if self.start.input && !self.ended { - self.sent += self.peer.input(&self.input[self.sent..]); - if self.sent == self.input.len() { - self.peer.end_input(); - self.ended = true; - } - } - if !self.stalled { - for stream in Stream::ALL { - loop { - let pending = self.peer.pending_output(stream); - if pending.is_empty() { - break; - } - let length = pending.len(); - self.output[position(stream)].extend_from_slice(pending); - self.peer.consumed(stream, length); - } - } - } - self.peer.tick(now); - while let Some(event) = self.peer.next_event() { - self.events.push(event); - } - let mut buffer = vec![0; self.start.limits.chunk()]; - let mut messages = Vec::new(); - while let Some(message) = self.peer.next_message(&mut buffer) { - let message = HostMessage::from_peer(self.start.key, message, &buffer) - .expect("the framing carries every message of the peer"); - messages.push(message.encode()); - } - messages - } + /// Returns the process ID that the program printed on its first line, that of the process + /// group that it leads. + fn group(&self) -> u32; +} - /// Returns the execution's outcome, once the peer finished. - fn outcome(&self) -> Option { - self.events.iter().find_map(|event| match event { - peer::Event::Finished(outcome) => Some(*outcome), - _ => None, - }) +impl Taken for HostExecution { + fn outputs(&self) -> [Vec; 2] { + Stream::ALL.map(|stream| self.output(stream).to_vec()) } - /// Returns the process ID that the program printed on its first line, that of the process - /// group that it leads. fn group(&self) -> u32 { - let output = String::from_utf8_lossy(&self.output[0]); + let output = String::from_utf8_lossy(self.output(Stream::Stdout)); output.lines().next().unwrap().parse().unwrap() } } @@ -2841,8 +2764,9 @@ impl Host { fn start_then(&mut self, start: Start, input: Vec, after: &[Message]) { let mut messages = vec![HostMessage::Start(start.clone()).encode()]; messages.extend_from_slice(after); - let execution = HostExecution::new(start, input, Instant::now()); - self.executions.insert(execution.start.key.id(), execution); + let execution = HostExecution::new(start, timers(), input, Instant::now()); + self.executions + .insert(execution.start().key.id(), execution); self.send(&messages); } @@ -2883,8 +2807,8 @@ impl Host { .executions .get_mut(&decoded.key().id()) .expect("the guest's message names an execution that the host started"); - decoded - .deliver(&mut execution.peer, now) + execution + .deliver(&decoded, now) .expect("the peer accepts the guest's message"); } } @@ -2960,7 +2884,7 @@ fn check_lossless(host: &Host, id: u64, status: Status) { assert!(!outcome.is_truncated(stream), "{id}: {stream:?}"); } assert_eq!(outcome.is_reusable(), cgroups(), "{id}"); - let peer = &host.executions[&id].peer; + let peer = host.executions[&id].peer(); assert!(peer.high_water() <= peer.memory_bound(), "{id}"); } @@ -2981,16 +2905,19 @@ fn the_worker_runs_executions_at_once_with_frame_counted_credit_and_one_output_p ); let both = "seq 1 30000; seq 30001 60000 >&2; exit 7"; host.start(start_of(3, shell(both)), Vec::new()); - host.run_until(|host| !host.executions[&1].output[0].is_empty()); + host.run_until(|host| !host.executions[&1].output(Stream::Stdout).is_empty()); host.send(&[Message::Ping]); host.run_until(|host| { host.replies.contains(&Message::Pong) && (1..=3).all(|id| host.outcome(id).is_some()) }); assert_eq!(host.replies, [Message::Pong]); - assert_eq!(host.executions[&1].output, [lines(1, 300_000), Vec::new()]); - assert_eq!(host.executions[&2].output, [input, Vec::new()]); assert_eq!( - host.executions[&3].output, + host.executions[&1].outputs(), + [lines(1, 300_000), Vec::new()] + ); + assert_eq!(host.executions[&2].outputs(), [input, Vec::new()]); + assert_eq!( + host.executions[&3].outputs(), [lines(1, 30_000), lines(30_001, 60_000)] ); check_lossless(&host, 1, Status::Exited(0)); @@ -3006,7 +2933,7 @@ fn a_stalled_sink_stalls_only_its_own_execution_while_timers_cancellations_and_c // program blocks once its pipe is full. let mut host = Host::new(); host.start(start_of(1, shell("exec cat /dev/zero")), Vec::new()); - host.execution(1).stalled = true; + host.execution(1).stall(true); // Another execution streams to its end meanwhile, a third one's timeout ends it in time, and // the host's ping is answered. host.start(start_of(2, shell("seq 1 200000")), Vec::new()); @@ -3025,18 +2952,21 @@ fn a_stalled_sink_stalls_only_its_own_execution_while_timers_cancellations_and_c && host.outcome(2).is_some() && host.outcome(3).is_some() }); - assert_eq!(host.executions[&2].output, [lines(1, 200_000), Vec::new()]); + assert_eq!( + host.executions[&2].outputs(), + [lines(1, 200_000), Vec::new()] + ); check_lossless(&host, 2, Status::Exited(0)); let timed = host.outcome(3).unwrap(); assert_eq!(timed.cause(), Some(Cause::TimedOut)); assert!(started.elapsed() < timeout + GRACE + Duration::from_secs(2)); let stalled = &host.executions[&1]; - assert!(stalled.outcome().is_none() && stalled.peer.buffered() > 0); - assert!(stalled.output[0].is_empty()); + assert!(stalled.outcome().is_none() && stalled.peer().buffered() > 0); + assert!(stalled.output(Stream::Stdout).is_empty()); // The stalled execution's cancellation ends it within its grace, with its output truncated. let cancelled = Instant::now(); - host.execution(1).peer.cancel(cancelled); + host.execution(1).peer_mut().cancel(cancelled); host.run_until(|host| host.outcome(1).is_some()); assert!(cancelled.elapsed() < GRACE + Duration::from_secs(2)); let outcome = host.outcome(1).unwrap(); @@ -3105,7 +3035,9 @@ fn a_full_stream_to_the_host_holds_back_only_output_while_cancellations_and_stop Vec::new(), ); } - host.run_until(|host| (1..=4).all(|id| host.executions[&id].output[0].contains(&b'\n'))); + host.run_until(|host| { + (1..=4).all(|id| host.executions[&id].output(Stream::Stdout).contains(&b'\n')) + }); let groups = (1..=4) .map(|id| host.executions[&id].group()) .collect::>(); @@ -3119,7 +3051,7 @@ fn a_full_stream_to_the_host_holds_back_only_output_while_cancellations_and_stop // the others, while the stream stays full. assert!(host.broker.room >= 1024, "{}", host.broker.room); let now = Instant::now(); - host.execution(1).peer.cancel(now); + host.execution(1).peer_mut().cancel(now); let cancel = host.execution(1).serve(now); host.send(&cancel); gone(groups[0]); @@ -3154,7 +3086,7 @@ fn executions_that_write_without_end_take_turns_at_the_stream_to_the_host() { } // Each takes a fair part of what the stream carries. - let taken = |host: &Host, id: u64| host.executions[&id].output[0].len(); + let taken = |host: &Host, id: u64| host.executions[&id].output(Stream::Stdout).len(); let total = |host: &Host| taken(host, 1) + taken(host, 2); host.run_until(|host| total(host) >= 16 * 1024 * 1024); let total = total(&host); @@ -3174,7 +3106,11 @@ fn the_worker_refuses_starts_beyond_its_limits_with_a_failed_launch() { for id in 1..=most { host.start(start_of(id, shell("exec sleep 60")), Vec::new()); } - host.run_until(|host| host.executions.values().all(|side| side.peer.has_started())); + host.run_until(|host| { + host.executions + .values() + .all(|side| side.peer().has_started()) + }); host.start(start_of(most + 1, shell("exit 0")), Vec::new()); host.run_until(|host| host.outcome(most + 1).is_some()); let refused = host.outcome(most + 1).unwrap(); @@ -3182,13 +3118,13 @@ fn the_worker_refuses_starts_beyond_its_limits_with_a_failed_launch() { assert_eq!(refused.launch_failure(), Some(again)); assert_eq!(refused.status(), None); assert!(!refused.is_reusable()); - assert!(!host.executions[&(most + 1)].peer.has_started()); + assert!(!host.executions[&(most + 1)].peer().has_started()); // Once those ended, it refuses an execution that would hold more than it holds for one, and // reports a program that it cannot find. let now = Instant::now(); for id in 1..=most { - host.execution(id).peer.cancel(now); + host.execution(id).peer_mut().cancel(now); } host.run_until(|host| (1..=most).all(|id| host.outcome(id).is_some())); for id in 1..=most { @@ -3283,7 +3219,7 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { host.run_until(|host| { [1, 2] .iter() - .all(|id| host.executions[id].output[0].ends_with(b"\n")) + .all(|id| host.executions[id].output(Stream::Stdout).ends_with(b"\n")) }); let (first, second) = (host.executions[&1].group(), host.executions[&2].group()); @@ -3299,7 +3235,7 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { host.start_then(later, Vec::new(), &[stale]); host.run_until(|host| host.outcome(2).is_some()); assert_eq!( - host.executions[&2].output, + host.executions[&2].outputs(), [b"later\n".to_vec(), Vec::new()] ); check_lossless(&host, 2, Status::Exited(0)); @@ -3309,12 +3245,12 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { // executions of the new epoch. host.broker.reset_in_flight(Identity { epoch: 2, ..FIRST }); let now = Instant::now(); - host.execution(1).peer.reset(now); + host.execution(1).peer_mut().reset(now); gone(first); host.start(start_of(3, shell("echo again")), Vec::new()); host.run_until(|host| host.outcome(1).is_some() && host.outcome(3).is_some()); assert_eq!( - host.executions[&3].output, + host.executions[&3].outputs(), [b"again\n".to_vec(), Vec::new()] ); check_lossless(&host, 3, Status::Exited(0)); @@ -3327,7 +3263,7 @@ fn a_reset_or_a_later_generation_abandons_executions_whose_programs_end() { fn the_worker_ends_its_executions_when_the_host_stops_it() { let mut host = Host::new(); host.start(start_of(1, shell("echo $$; exec sleep 60")), Vec::new()); - host.run_until(|host| host.executions[&1].output[0].ends_with(b"\n")); + host.run_until(|host| host.executions[&1].output(Stream::Stdout).ends_with(b"\n")); let group = host.executions[&1].group(); let (root, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); @@ -3347,9 +3283,9 @@ fn the_worker_collects_the_status_of_a_program_that_outlives_its_execution() { Start::new(Key::new(1, 1), shared_limits(), instant, sleeper), Vec::new(), ); - host.run_until(|host| host.executions[&1].output[0].ends_with(b"\n")); + host.run_until(|host| host.executions[&1].output(Stream::Stdout).ends_with(b"\n")); let program = host.executions[&1].group(); - host.execution(1).peer.cancel(Instant::now()); + host.execution(1).peer_mut().cancel(Instant::now()); host.run_until(|host| host.outcome(1).is_some()); let outcome = host.outcome(1).unwrap(); assert_eq!(outcome.cause(), Some(Cause::Cancelled)); @@ -3383,7 +3319,7 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { // A stop once the worker collected the status of such a program leaves no process of the // group running either. host.start(start_of(2, leaving()), Vec::new()); - host.run_until(|host| host.executions[&2].output[0].ends_with(b"\n")); + host.run_until(|host| host.executions[&2].output(Stream::Stdout).ends_with(b"\n")); let second = host.executions[&2].group(); reaped(second); // The fake root lives until the check, as its workload cgroup, if any, ends what is left in it @@ -3397,7 +3333,7 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { /// standard output of the execution `id`, and the directory of the program's cgroup from the next /// line, which `/proc/self/cgroup` gave. fn process_and_cgroup(host: &Host, id: u64, line: usize) -> (u32, PathBuf) { - let output = String::from_utf8(host.executions[&id].output[0].clone()).unwrap(); + let output = String::from_utf8(host.executions[&id].output(Stream::Stdout).to_vec()).unwrap(); let mut lines = output.lines().skip(line); let process = lines.next().unwrap().parse().unwrap(); let cgroup = lines.next().unwrap().strip_prefix("0::/").unwrap(); @@ -3432,7 +3368,7 @@ fn an_execution_s_settlement_ends_a_descendant_that_left_its_process_group() { for id in 2..=most + 1 { host.start(start_of(id, shell("exec sleep 60")), Vec::new()); } - host.run_until(|host| (2..=most + 1).all(|id| host.executions[&id].peer.has_started())); + host.run_until(|host| (2..=most + 1).all(|id| host.executions[&id].peer().has_started())); let (_, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); } @@ -3498,7 +3434,7 @@ fn an_execution_that_does_not_settle_keeps_its_place_until_the_host_stops_the_wo ); host.start(start, Vec::new()); host.run_until(|host| { - let output = &host.executions[&1].output[0]; + let output = host.executions[&1].output(Stream::Stdout); output.iter().filter(|&&byte| byte == b'\n').count() == 2 }); let (program, cgroup) = process_and_cgroup(&host, 1, 0); @@ -3545,7 +3481,7 @@ fn fill_but_one(host: &mut Host, first: u64) { } host.run_until(|host| { ids.clone() - .all(|id| host.executions[&id].peer.has_started()) + .all(|id| host.executions[&id].peer().has_started()) }); let refused = ids.end; host.start(start_of(refused, shell("exit 0")), Vec::new()); @@ -3554,7 +3490,7 @@ fn fill_but_one(host: &mut Host, first: u64) { assert_eq!(host.outcome(refused).unwrap().launch_failure(), Some(again)); let now = Instant::now(); for id in ids.clone() { - host.execution(id).peer.cancel(now); + host.execution(id).peer_mut().cancel(now); } host.run_until(|host| ids.clone().all(|id| host.outcome(id).is_some())); for id in ids { @@ -3583,9 +3519,12 @@ fn an_execution_runs_in_its_working_directory_with_exactly_its_environment() { check_lossless(&host, id, Status::Exited(0)); } let printed = environment.map(|entry| format!("{entry}\n")).concat(); - assert_eq!(host.executions[&1].output[0], printed.as_bytes()); assert_eq!( - host.executions[&2].output[0], + host.executions[&1].output(Stream::Stdout), + printed.as_bytes() + ); + assert_eq!( + host.executions[&2].output(Stream::Stdout), format!("{directory}\n").as_bytes() ); let (root, worked) = host.stop(); @@ -3716,7 +3655,7 @@ fn the_worker_refuses_at_stage_0_a_start_whose_identity_breaks_the_rules() { } host.run_until(|host| { ids.clone() - .all(|id| host.executions[&id].peer.has_started()) + .all(|id| host.executions[&id].peer().has_started()) }); let (_, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); @@ -3750,7 +3689,7 @@ fn an_execution_that_starts_with_the_root_counts_its_timeout_from_its_own_start( timed, shell("exec sleep 0.2"), ); - let execution = HostExecution::new(start.clone(), Vec::new(), Instant::now()); + let execution = HostExecution::new(start.clone(), timers(), Vec::new(), Instant::now()); let mut host = Host { broker, working: Some(working), @@ -3779,7 +3718,7 @@ fn a_host_that_a_test_drops_while_its_broker_pauses_ends_the_worker_and_its_prog start_of(1, shell("echo $$; exec cat /dev/zero")), Vec::new(), ); - host.run_until(|host| host.executions[&1].output[0].contains(&b'\n')); + host.run_until(|host| host.executions[&1].output(Stream::Stdout).contains(&b'\n')); let group = host.executions[&1].group(); host.broker.pause(); let working = host.working.take().unwrap(); @@ -3864,7 +3803,7 @@ fn the_worker_runs_executions_on_a_tty() { host.start(start_of(1, shell(script)), Vec::new()); host.run_until(|host| host.outcome(1).is_some()); assert_eq!( - host.executions[&1].output, + host.executions[&1].outputs(), [lines(1, 100_000), lines(1, 1000)] ); check_lossless(&host, 1, Status::Exited(0)); diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index 7ffc22fca..792e75e00 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -10,15 +10,21 @@ //! decoder, the requests of a layered root: its configuration, its probes, and a stop whose //! teardown failed, and the host's order of a layered root: the guest agent's readiness and build, //! the mapping table in parts, and `Start`, and each failure of the order, after which the host -//! sends nothing more, or stops the guest. +//! sends nothing more, or stops the guest. The broker's guest also runs the runners of NVX's +//! execution engine for the executions that the host program's driver runs, so that the tests cover +//! the driver's output, credit, outcomes, and timers, and the refusals that end its run. #![cfg(feature = "host")] -use std::collections::VecDeque; +use std::collections::{BTreeMap, VecDeque}; use std::io; use std::sync::{Arc, Condvar, Mutex, MutexGuard}; use std::thread::{self, JoinHandle}; use std::time::{Duration, Instant}; +use nvx::exec::{ + Cause, Key, LaunchFailure, Limits, Outcome, Runner, Settlement, Status, Stream, Timers, + Violation, Window, runner, +}; use nvx::host::nvxs::HostConnectionError; use nvx::host::platform::Transport; use nvx::protocol::nvxs::{ @@ -29,8 +35,11 @@ use nvx::protocol::{ OUTER_MAX_PAYLOAD, OUTER_MIN_RECEIVE_WINDOW, }; use runtime_adapter::GUEST_BUILD; +use runtime_adapter::exec::{GuestMessage, HostMessage, Start}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, MAX_TEXT_BODY, Message}; -use runtime_adapter::host::{self, Client, ClientError, MapReply, OrderError}; +use runtime_adapter::host::{ + self, Client, ClientError, Executions, ExecutionsError, MapReply, OrderError, +}; use runtime_adapter::probe::{Facts, Probe, ProbeReply}; use runtime_adapter::root::{ MapFailure, Ready, RootConfig, RootFailure, RootReply, RootStage, decode_maps, encode_maps, @@ -1192,6 +1201,7 @@ fn the_client_refuses_messages_that_its_framing_does_not_allow() { // The client takes the guest agent's messages, and refuses those that only the host program // sends. + let key = Key::new(1, 1); let host_messages = [ Message::Ping, Message::Stop, @@ -1199,6 +1209,10 @@ fn the_client_refuses_messages_that_its_framing_does_not_allow() { Message::Probe(b"mounts".to_vec()), Message::Maps(maps_part()), Message::Start, + HostMessage::Start(start_of(1)).encode(), + HostMessage::Input(key, b"input").encode(), + HostMessage::InputEnd(key).encode(), + HostMessage::Cancel(key, Cause::Cancelled).encode(), ]; for message in host_messages { let (mut broker, mut client) = attached(FIRST, usize::MAX); @@ -1804,3 +1818,388 @@ fn a_mapping_that_the_guest_fails_ends_the_table_and_the_host_stops_the_guest() "{description}" ); } + +/// The chunk of the executions of the tests of the host's driver. +const CHUNK: u32 = 16 * 1024; + +/// Returns the limits of the executions of the tests of the host's driver, as the host program +/// chooses them: chunks of 16 KiB, and frame-counted credit, of 4 frames of input, and of 8 frames +/// of output in one pool that standard output and error share. +fn limits() -> Limits { + Limits::new(CHUNK, Window::frames(4), Window::frames(8)) + .unwrap() + .shared_output() +} + +/// Returns the timers of both sides of the executions of the tests of the host's driver. +fn timers() -> Timers { + Timers::new(Duration::from_millis(300), Duration::from_secs(5)) +} + +/// Returns the start of the execution `id` of the first generation, of a program that the test's +/// guest plays. +fn start_of(id: u64) -> Start { + Start::new(Key::new(id, 1), limits(), timers(), ["/bin/program"]) +} + +/// Returns the position of `stream` among [`Stream::ALL`]. +fn position(stream: Stream) -> usize { + match stream { + Stream::Stdout => 0, + Stream::Stderr => 1, + } +} + +/// What the program of an execution that the test's guest plays does. +#[derive(Clone, Debug)] +struct Workload { + /// Whether the program starts, or why its launch failed. + launch: Result<(), LaunchFailure>, + /// What the program writes to its standard output and error, in the order of [`Stream::ALL`]. + output: [Vec; 2], + /// How the program ends, once it wrote its output. + status: Status, +} + +/// The guest's side of an execution that the test's guest plays: the runner of NVX's execution +/// engine, which takes the program's output as its credit lets it, as the worker's does. +struct Played { + /// The guest's side of the execution. + runner: Runner, + /// What the program does. + workload: Workload, + /// Whether the runner learned how the launch went. + launched: bool, + /// The bytes of each stream that the runner took. + written: [usize; 2], + /// Whether the program ended. + ended: bool, +} + +impl Played { + /// Returns the guest's side of the execution of `runner`, whose program does `workload`. + fn new(runner: Runner, workload: Workload) -> Self { + Self { + runner, + workload, + launched: false, + written: [0, 0], + ended: false, + } + } + + /// Takes every step that the program and the runner can take at `now`, as the worker takes + /// them, and returns the runner's messages, as frames. + fn step(&mut self, now: Instant) -> Vec { + if !self.launched { + self.launched = true; + self.runner.launched(self.workload.launch); + if self.workload.launch.is_err() { + for stream in Stream::ALL { + self.runner.output_closed(stream); + } + self.runner.exited(now, None); + self.ended = true; + } + } + if !self.ended { + for stream in Stream::ALL { + let output = &self.workload.output[position(stream)]; + let written = &mut self.written[position(stream)]; + *written += self.runner.output(stream, &output[*written..]); + } + if self.written == self.workload.output.each_ref().map(Vec::len) { + for stream in Stream::ALL { + self.runner.output_closed(stream); + } + self.runner.exited(now, Some(self.workload.status)); + self.ended = true; + } + } + self.runner.tick(now); + while let Some(request) = self.runner.next_request() { + if let runner::Request::Settle = request { + self.runner.settled(match self.workload.launch { + Ok(()) => Settlement::Settled, + Err(_) => Settlement::Failed, + }); + } + } + let key = self.runner.key(); + let mut buffer = vec![0; self.runner.limits().chunk()]; + let mut messages = Vec::new(); + while let Some(message) = self.runner.next_message(&mut buffer) { + if let Some(message) = GuestMessage::from_runner(key, message, &buffer) { + messages.push(message.encode()); + } + } + messages + } +} + +/// Plays the guest's side of the host's executions until `count` of them finished: a runner for +/// each start, whose program does what `workload` says of the start. Takes the host's messages, +/// sends the runners' in DATA records of at most `size` bytes, and returns the starts, in order. +fn play( + broker: &mut Broker, + count: usize, + size: usize, + workload: impl Fn(&Start) -> Workload, +) -> Vec { + let mut starts = Vec::new(); + let mut played = BTreeMap::::new(); + loop { + let now = Instant::now(); + let messages = played + .values_mut() + .flat_map(|played| played.step(now)) + .collect::>(); + broker.send_data(&encode(&messages), size); + if played.len() == count && played.values().all(|played| played.runner.is_finished()) { + return starts; + } + for message in broker.take() { + let decoded = HostMessage::decode(&message) + .expect("the host sends messages about executions") + .expect("the host's messages decode"); + let message = match decoded { + HostMessage::Start(start) => { + let runner = Runner::new(start.key, start.limits, start.timers, now); + played.insert(start.key.id(), Played::new(runner, workload(&start))); + starts.push(start); + continue; + } + message => message, + }; + let key = message.key(); + let runner = &mut played + .get_mut(&key.id()) + .expect("the host started the execution") + .runner; + match message { + HostMessage::Credit(key, pool, units) => runner.credit(key, pool, units), + HostMessage::Input(key, bytes) => runner.input(key, bytes), + HostMessage::InputEnd(key) => runner.end_input(key), + HostMessage::Cancel(key, cause) => runner.cancel(now, key, cause), + HostMessage::Start(_) => unreachable!("the start came first"), + } + .expect("the runner accepts the host's message"); + } + } +} + +/// Starts the executions of `starts` with `client`, each with its own timers, and runs them in a +/// thread until `deadline`, which returns the client and the driver. +fn run( + client: Client, + starts: Vec, + deadline: Instant, +) -> JoinHandle<(Client, Result)> { + spawn(client, move |client| { + let mut executions = Executions::new(); + for start in starts { + let timers = start.timers; + executions.start(client, start, timers, Vec::new(), deadline)?; + } + executions.run(client, deadline)?; + Ok(executions) + }) +} + +#[test] +fn the_driver_runs_executions_at_once_and_collects_their_output_and_outcomes() { + // The first program writes more to its standard output than the guest's credit holds, so that + // its output flows as the host takes it, and a little to its standard error, then exits with + // 3. The second one does not start. The guest's records carry its messages whole, or in + // pieces that split them, and each read of the host's takes at most 4 KiB. + let output = (0..=u8::MAX).cycle().take(300 * 1024).collect::>(); + let written = Workload { + launch: Ok(()), + output: [output.clone(), b"error".to_vec()], + status: Status::Exited(3), + }; + let refused = Workload { + launch: Err(LaunchFailure::new(0, 13)), + output: [Vec::new(), Vec::new()], + status: Status::Exited(0), + }; + for size in [OUTER_MAX_PAYLOAD, 1000] { + let (mut broker, client) = attached(FIRST, 4096); + let running = run(client, vec![start_of(1), start_of(2)], deadline()); + let starts = play(&mut broker, 2, size, |start| match start.key.id() { + 1 => written.clone(), + _ => refused.clone(), + }); + assert_eq!(starts, [start_of(1), start_of(2)]); + let (_, executions) = joined(running); + let executions = executions.unwrap(); + assert!(executions.is_finished()); + let ids = executions + .iter() + .map(|execution| execution.start().key.id()) + .collect::>(); + assert_eq!(ids, [1, 2]); + + let first = executions.get(1).unwrap(); + assert!(first.peer().has_started()); + assert_eq!(first.output(Stream::Stdout), output); + assert_eq!(first.output(Stream::Stderr), b"error"); + let outcome = first.outcome().unwrap(); + assert_eq!(outcome.status(), Some(Status::Exited(3))); + assert_eq!((outcome.cause(), outcome.launch_failure()), (None, None)); + assert!(outcome.is_reusable()); + for stream in Stream::ALL { + assert!(!outcome.is_truncated(stream), "{stream:?}"); + } + + let second = executions.get(2).unwrap(); + assert!(!second.peer().has_started()); + let outcome = second.outcome().unwrap(); + assert_eq!(outcome.launch_failure(), Some(LaunchFailure::new(0, 13))); + assert_eq!(outcome.status(), None); + assert!(!outcome.is_reusable()); + for stream in Stream::ALL { + assert!(second.output(stream).is_empty(), "{stream:?}"); + } + } +} + +#[test] +fn the_driver_refuses_starts_that_the_guest_would_not_take_and_sends_nothing() { + let (mut broker, mut client) = attached(FIRST, usize::MAX); + let mut executions = Executions::new(); + // The guest ends its session on a start that it cannot decode. + let undecodable = [ + start_of(1).with_working_directory("relative"), + start_of(1).with_environment(["=value"]), + Start::new(Key::new(1, 1), limits(), timers(), [""]), + ]; + for start in undecodable { + let refused = + executions.start(&mut client, start.clone(), timers(), Vec::new(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Start(_))), + "{start:?}: {refused:?}" + ); + } + // A second start of an ID. + executions + .start(&mut client, start_of(1), timers(), Vec::new(), deadline()) + .unwrap(); + let refused = executions.start(&mut client, start_of(1), timers(), Vec::new(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Duplicate(1))), + "{refused:?}" + ); + // The guest received the one start alone, and the client stays usable. + assert_eq!( + broker.messages(1), + [HostMessage::Start(start_of(1)).encode()] + ); + let pinging = spawn(client, |client| client.ping(deadline())); + assert_eq!(broker.answer(OUTER_MAX_PAYLOAD), Message::Ping); + joined(pinging).1.unwrap(); +} + +#[test] +fn the_driver_ends_its_run_on_a_message_that_breaks_the_rules() { + /// A check of the failure that ended a run. + type Check = fn(&ExecutionsError) -> bool; + let key = Key::new(1, 1); + let key_body = [key.id().to_le_bytes(), key.generation().to_le_bytes()].concat(); + let cases: [(&str, Vec, Check); 4] = [ + ( + "a message about an execution that the host did not start", + vec![GuestMessage::Started(Key::new(2, 1)).encode()], + |error| matches!(error, ExecutionsError::Unknown(key) if *key == Key::new(2, 1)), + ), + ( + "a message about no execution", + vec![Message::Pong], + |error| matches!(error, ExecutionsError::Unexpected(kind) if *kind == Message::Pong.kind()), + ), + ( + "output of a stream that does not exist", + vec![Message::ExecOutput([&key_body[..], &[9], b"x"].concat())], + |error| matches!(error, ExecutionsError::Body(_)), + ), + ( + "an outcome before the ends of the streams", + vec![ + GuestMessage::Started(key).encode(), + GuestMessage::Terminal(key, Outcome::new(Some(Status::Exited(0)))).encode(), + ], + |error| { + matches!( + error, + ExecutionsError::Violation(failed, Violation::Order) if *failed == Key::new(1, 1) + ) + }, + ), + ]; + for (name, messages, check) in cases { + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], deadline()); + broker.messages(1); + broker.send_data(&encode(&messages), OUTER_MAX_PAYLOAD); + let (_, ran) = joined(running); + let failed = ran.as_ref().err().is_some_and(check); + assert!(failed, "{name}: {:?}", ran.err()); + } + + // The end of the guest's stream ends the run too. + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], deadline()); + broker.messages(1); + broker.close(); + let (_, ran) = joined(running); + let closed = matches!( + ran, + Err(ExecutionsError::Client(ClientError::Session( + HostConnectionError::Closed + ))) + ); + assert!(closed, "{:?}", ran.err()); +} + +#[test] +fn the_driver_fires_the_timers_of_its_executions_while_the_guest_says_nothing() { + // The execution's timeout passes while the guest sends nothing: the host cancels it, waits for + // its outcome for the grace and the settlement time, then gives up on it, and fences it. + let (grace, settlement, timeout) = ( + Duration::from_millis(50), + Duration::from_millis(100), + Duration::from_millis(200), + ); + let timed = Timers::new(grace, settlement).timeout(timeout); + let key = Key::new(1, 1); + let start = Start::new(key, limits(), timed, ["/bin/program"]); + let (mut broker, client) = attached(FIRST, usize::MAX); + let started = Instant::now(); + let running = run(client, vec![start.clone()], deadline()); + // The start, the credit of the output's pool, then the cancellation. + let messages = broker.messages(3); + assert_eq!(messages[0], HostMessage::Start(start).encode()); + assert_eq!( + messages[2], + HostMessage::Cancel(key, Cause::TimedOut).encode() + ); + let (_, executions) = joined(running); + assert!(started.elapsed() >= timeout + grace + settlement); + let executions = executions.unwrap(); + let outcome = executions.get(1).unwrap().outcome().unwrap(); + assert_eq!(outcome.cause(), Some(Cause::TimedOut)); + assert_eq!(outcome.status(), None); + assert!(!outcome.is_reusable()); + + // A run whose deadline passes before its executions finish fails. + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start_of(1)], Instant::now() + SHORT); + broker.messages(1); + let (_, ran) = joined(running); + assert!( + matches!(ran, Err(ExecutionsError::TimedOut)), + "{:?}", + ran.err() + ); +} diff --git a/tests/runtime-adapter/tests/layered_launch.rs b/tests/runtime-adapter/tests/layered_launch.rs index 4569de3d9..be523e0e2 100644 --- a/tests/runtime-adapter/tests/layered_launch.rs +++ b/tests/runtime-adapter/tests/layered_launch.rs @@ -3,8 +3,9 @@ //! OpenVMM the checked files, plans the export of the host paths that it shares with the root //! inside the directories that the adapter authorizes, attaches it, and verifies it before the //! launch, composes OpenVMM's arguments without NVX's managed lifecycle, records what it launches, -//! with the bindings that it checked, and refuses options that do not make a command, and shares -//! that it cannot export before it touches the store. The tests need no hypervisor. +//! with the bindings that it checked, and refuses options that do not make a command, shares that +//! it cannot export before it touches the store, and executions whose starts the guest would not +//! take. The tests need no hypervisor. #![cfg(feature = "host")] use std::ffi::OsString; @@ -849,7 +850,7 @@ fn the_host_program_refuses_shares_before_it_touches_the_store() { fn the_host_program_refuses_options_that_do_not_make_a_command() { let launch = ["launch", "store", "--guest-agent", "agent"]; let uuid = "0a490905-41be-ed5f-984e-63eb7e5e20b1"; - let cases: [(&[&str], &str); 30] = [ + let cases: [(&[&str], &str); 34] = [ ( &["--distro", "erofs:image", "--upper", "ram"], "--distro needs --distro-identity", @@ -864,6 +865,22 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { "--distro-identity needs --distro", ), (&["--runtime-dir", "/run/a"], "--runtime-dir needs --distro"), + ( + &["--workload-identity", "1000:1000"], + "--workload-identity needs --distro", + ), + ( + &["--workload-identity", "0:1000"], + "--workload-identity takes UID:GID, two numbers from 1 to 4294967294, not \"0:1000\"", + ), + ( + &["--workload-identity", "1000:4294967295"], + "--workload-identity takes UID:GID", + ), + ( + &["--workload-identity", "1000"], + "--workload-identity takes UID:GID", + ), ( &["--overlay-mode", "untrusted-lowers"], "--overlay-mode needs --distro", @@ -1022,6 +1039,69 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { "unexpected option \"--share\" of reattach", ), ]); + // Each execution of `exec` has options of its own, before `--` and its program, and its + // arguments end at a lone `;`. The program refuses a start that the guest would not decode, + // on which the guest would end its session, before it attaches. + let program = ["--", "/bin/true"]; + let exec = |options: &[&'static str]| [&["exec", "store"], options, &program[..]].concat(); + commands.extend([ + ( + vec!["exec", "store"], + "exec needs `-- PROGRAM` after the options of each execution", + ), + ( + vec!["exec", "store", "--"], + "exec needs a program after `--`", + ), + ( + exec(&[]).into_iter().chain([";"]).collect(), + "exec needs `-- PROGRAM` after the options of each execution", + ), + (exec(&["--groups", "1"]), "--groups needs --identity"), + ( + exec(&["--capabilities", "0"]), + "--capabilities needs --identity", + ), + (exec(&["--numeric"]), "--numeric needs --identity"), + ( + exec(&["--identity", "1000"]), + "--identity takes UID:GID, not \"1000\"", + ), + ( + exec(&["--identity", "0:0", "--groups", "1,,2"]), + "--groups takes GIDs, separated by commas, not \"1,,2\"", + ), + ( + exec(&["--identity", "0:0", "--capabilities", "64"]), + "--capabilities takes capability numbers from 0 to 63", + ), + ( + exec(&["--cwd", "tmp"]), + "names a working directory that is not absolute", + ), + ( + exec(&["--env", "=value"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + exec(&["--cwd", "/a", "--cwd", "/b"]), + "unexpected option \"--cwd\" of exec", + ), + ( + exec(&["--distro", "erofs:image"]), + "unexpected option \"--distro\" of exec", + ), + (vec!["exec", "store", "--env"], "\"--env\" needs a value"), + ( + [ + exec(&["--identity", "0:0", "--groups", "1"]), + vec![";"], + exec(&["--groups", "1"])[2..].to_vec(), + ] + .concat(), + "--groups needs --identity", + ), + ]); for (arguments, problem) in commands { let output = Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")) .args(&arguments) diff --git a/tests/runtime-adapter/tests/vm_layered.rs b/tests/runtime-adapter/tests/vm_layered.rs index fad70eda4..cfe3723ca 100644 --- a/tests/runtime-adapter/tests/vm_layered.rs +++ b/tests/runtime-adapter/tests/vm_layered.rs @@ -4,10 +4,10 @@ //! //! Each scenario of the table that [`scenarios!`] declares is an ignored test, which stages the //! images of a [`Root`], launches a VM with it, and checks what it [`Expect`]s: a root that the -//! guest assembles, which the scenario probes and the stop tears down, the stage of the root that -//! fails, or a launch that the host program refuses before it spawns the VM, since the scenario -//! changed what it staged. A new scenario is a new row, with its own checks, through the probes of -//! the worker. +//! guest assembles, which the scenario probes, or runs executions in through the host program's +//! `exec`, and the stop tears down, the stage of the root that fails, or a launch that the host +//! program refuses before it spawns the VM, since the scenario changed what it staged. A new +//! scenario is a new row, with its own checks, through the probes of the worker or executions. //! //! The tests boot VMs, so they are ignored. Besides what the `common` module lists, they read its //! fixtures, those of NVX's own layered scenarios, from NVX's artifacts in `NVX_ARTIFACTS_DIR`. @@ -130,6 +130,23 @@ const MAP_TABLE: &str = "map dir=srv/maps/dir inner=srv/maps/dir/inner file=srv/ /// `file.txt`: the plan exports the outermost shares as its children, in the order of their host /// paths. const MAP_CHECKS: &str = "map-checks 2 1"; +/// The identity, besides root's, as which the scenarios' executions run: the image's `nobody`, of +/// the group `nogroup`, which NVX's Ubuntu image always holds. +const NOBODY: (u32, u32) = (65534, 65534); +/// An identity whose account the image lacks. +const ACCOUNTLESS: (u32, u32) = (4242, 4242); +/// Linux's error number of an operation that is not permitted. +const EPERM: i32 = 1; +/// Linux's error number of an invalid argument. +const EINVAL: i32 = 22; +/// The number of executions that meet only while all of them run. +const MEETING: usize = 4; +/// A script that prints its process's user ID, group ID, and groups, and its permitted, +/// effective, and bounding capability sets, which the shell reads itself. +const IDENTITY_SCRIPT: &str = "id -u; id -g; id -G; \ + while read -r key value; do \ + case $key in CapPrm:|CapEff:|CapBnd:) echo \"$key $value\";; esac; \ + done , /// The bound of the first partition, as the launch names it, unless the guest's default. first_partition_bound: Option, + /// The sandbox's workload identity, its user ID and its group ID, as the launch names it, if + /// any. + workload: Option<(u32, u32)>, /// The host paths that the launch shares with the root. table: Table, } @@ -206,6 +226,7 @@ const ROOT: Root = Root { overlay: None, journal: None, first_partition_bound: None, + workload: None, table: Table::None, }; @@ -289,6 +310,9 @@ impl Root { if let Some(bound) = self.first_partition_bound { options.extend(["--first-partition-bound".into(), bound.to_string().into()]); } + if let Some((uid, gid)) = self.workload { + options.extend(["--workload-identity".into(), format!("{uid}:{gid}").into()]); + } options } @@ -399,7 +423,8 @@ enum Expect { /// that failed with this problem, which leaves a scratch mounted when the VM ends. TeardownFails(fn(&mut Vm<'_>), &'static str), /// The guest reports that the stage failed with this problem, and the VM ends with its - /// status, before the guest mounted a scratch. + /// status, without a scratch mounted: the guest fails each stage but the identity's before it + /// mounts the scratch, and unmounts the root that it assembled once the identity's failed. Fails(RootStage, &'static str), /// The guest assembles the root, applies this many mappings of the table, and fails the next /// with a problem that holds this text: the launch sends no more of the table, nor `Start`, and @@ -489,6 +514,23 @@ impl Vm<'_> { assert_eq!(printed["pid"], self.launched["pid"]); printed } + + /// Runs `executions` at once in the VM's root, each the options and the arguments of one + /// execution of the host program's `exec`, and returns what they did, each fact as + /// `exec.N.KEY`, for execution N, from 1. + fn exec(&mut self, executions: &[Vec]) -> Printed { + let mut options = Vec::new(); + for (index, execution) in executions.iter().enumerate() { + if index > 0 { + options.push(OsStr::new(";")); + } + options.extend(execution.iter().map(OsStr::new)); + } + let printed = self.sandbox.run("exec", &options, 0); + assert_eq!(printed["reattached"], "live"); + assert_eq!(printed["pid"], self.launched["pid"]); + printed + } } /// Declares the table of scenarios: each row, `NAME: SCENARIO;`, is an ignored test, `NAME`, which @@ -758,6 +800,33 @@ scenarios! { "changed after its scratch binding was made: its identity differs", ), }; + /// Executions that the host program runs at once over one session, in a sandbox without a + /// workload identity, so that each start names its own: output beyond 1 MiB, whole; several + /// executions that meet only while all of them run; a descendant in a session of its own, + /// which holds its program's output, and which the end of its program ends; a user other than + /// root, and root with an allowed capability alone; a working directory, an exact + /// environment, and the configured hostname, each as PID 1 of its namespaces. The guest + /// refuses a start without an identity, and one with a capability outside the entry's + /// allowlist, at stage 0. + executions_run_in_the_root: Scenario { + root: ROOT, + expect: Expect::Ready(check_executions), + }; + /// Executions in a sandbox whose workload identity the launch names, which every execution + /// runs as: the guest refuses a start that names an identity of its own. + executions_run_as_the_workload_identity: Scenario { + root: Root { workload: Some(NOBODY), ..ROOT }, + expect: Expect::Ready(check_workload_executions), + }; + /// A workload identity without an account in the image, which fails the identity stage once + /// the guest assembled the root, before it answers the root configuration. + accountless_workload_identity_fails_its_stage: Scenario { + root: Root { workload: Some(ACCOUNTLESS), ..ROOT }, + expect: Expect::Fails( + RootStage::Identity, + "cannot enter the root as the workload identity 4242:4242: ", + ), + }; } /// Runs the scenario `name`, `scenario`. @@ -807,7 +876,7 @@ fn run(name: &str, scenario: Scenario) { assert_eq!(sandbox.record(), None); assert!( !(over_scratch && needs_recovery(&scratch)), - "the guest mounted the scratch" + "the guest left the scratch mounted" ); return; } @@ -973,6 +1042,7 @@ fn check_record(sandbox: &Sandbox, root: &Root, distro: &Distro, scratch: &Path) overlay_mode: root.overlay, scratch_journal: root.journal, mapping_count: root.table.mapping_count(), + workload_identity: root.workload, ..RootConfig::default() }; let sent = String::from_utf8(config.encode()).unwrap(); @@ -2108,3 +2178,187 @@ fn check_root_profiles(vm: &mut Vm<'_>) { assert_eq!(fact("after.machine-id"), ROOT_MACHINE_ID); assert_eq!(fact("after.mounts"), "0"); } + +/// Returns the options and the arguments of an execution of the host program's `exec`: `options`, +/// then `--`, then the program and its arguments, `command`. +fn execution(options: &[&str], command: &[&str]) -> Vec { + options + .iter() + .chain(&["--"]) + .chain(command) + .map(|argument| (*argument).to_owned()) + .collect() +} + +/// Returns `identity` as the value of the option `--identity` of `exec`: `UID:GID`. +fn ids((uid, gid): (u32, u32)) -> String { + format!("{uid}:{gid}") +} + +/// Returns `text` as the host program prints a value: with `\`, newlines, and carriage returns +/// escaped. +fn escaped(text: &str) -> String { + text.replace('\\', "\\\\") + .replace('\n', "\\n") + .replace('\r', "\\r") +} + +/// Returns the output of `seq first last`. +fn lines(first: usize, last: usize) -> String { + (first..=last).map(|number| format!("{number}\n")).collect() +} + +/// Returns what the [`IDENTITY_SCRIPT`] prints for the user ID `uid`, the group ID `gid`, the +/// groups `groups`, and the capabilities `capabilities`, a set of each capability's bit. +fn identity_facts(uid: u32, gid: u32, groups: &str, capabilities: u64) -> String { + let set = capability_set(capabilities); + format!("{uid}\n{gid}\n{groups}\nCapPrm: {set}\nCapEff: {set}\nCapBnd: {set}\n") +} + +/// Returns the fact KEY of execution `number` that `exec` printed in `printed`. +fn fact<'a>(printed: &'a Printed, number: usize, key: &str) -> &'a str { + let key = format!("exec.{number}.{key}"); + printed + .get(&key) + .unwrap_or_else(|| panic!("{key} is missing: {printed:?}")) +} + +/// Checks that execution `number` of what `exec` printed in `printed` started, ended on its own +/// with the exit code `code`, lost no byte, and settled, so that its place is reusable, and that +/// it wrote `stdout` and `stderr`. +fn assert_ran(printed: &Printed, number: usize, code: i32, stdout: &str, stderr: &str) { + let status = format!("exit {code}"); + let facts = + ["launch", "status", "cause", "truncated", "settled"].map(|key| fact(printed, number, key)); + assert_eq!( + facts, + ["started", &*status, "none", "none", "reusable"], + "{number}: {printed:?}" + ); + assert_eq!(fact(printed, number, "stdout"), escaped(stdout), "{number}"); + assert_eq!(fact(printed, number, "stderr"), escaped(stderr), "{number}"); +} + +/// Checks that the guest refused execution `number` of what `exec` printed in `printed` at stage +/// 0 with the error number `error`, before its program started: the execution is fenced, and has +/// no output. +fn assert_refused(printed: &Printed, number: usize, error: i32) { + let launch = format!("failed 0 {error}"); + let facts = + ["launch", "status", "settled", "stdout", "stderr"].map(|key| fact(printed, number, key)); + assert_eq!( + facts, + [&*launch, "none", "fenced", "", ""], + "{number}: {printed:?}" + ); +} + +/// Checks the VM of the `executions_run_in_the_root` scenario, whose starts name their identities. +fn check_executions(vm: &mut Vm<'_>) { + let nobody = ids(NOBODY); + + // Several executions at once, which meet only while all of them run: each leaves its mark in + // the root's `/tmp`, and waits for the others' marks. Beside them, one writes more than a MiB + // to its standard output, and a little to its standard error, which arrive whole. + let meet = format!( + "mkdir -p /tmp/meeting && touch /tmp/meeting/$NUMBER && \ + until [ \"$(ls /tmp/meeting | wc -l)\" -ge {MEETING} ]; do sleep 0.01; done && \ + echo \"$NUMBER met\"" + ); + let mut executions = (1..=MEETING) + .map(|number| { + let environment = format!("NUMBER={number}"); + execution( + &["--identity", &nobody, "--env", &environment], + &["/bin/sh", "-c", &meet], + ) + }) + .collect::>(); + let writer = "seq 1 200000; seq 1 1000 >&2"; + executions.push(execution( + &["--identity", "0:0"], + &["/bin/sh", "-c", writer], + )); + let printed = vm.exec(&executions); + for number in 1..=MEETING { + assert_ran(&printed, number, 0, &format!("{number} met\n"), ""); + } + let output = lines(1, 200_000); + assert!(output.len() > 1 << 20, "{}", output.len()); + assert_ran(&printed, MEETING + 1, 0, &output, &lines(1, 1000)); + + // In a later session, whose keys are new to the guest: + // 1. a descendant in a session of its own, which holds its program's output, and which the + // end of its program, PID 1 of its namespace, ends, so that the execution loses no byte; + // 2. a user other than root, without capabilities, in existing-account mode; + // 3. root with `CAP_CHOWN` alone, in its permitted, effective, and bounding sets; + // 4. exactly the environment that the start names; + // 5. the working directory that the start names; + // 6. the configured hostname, as PID 1 of the execution's namespaces; + // 7. no identity, in a sandbox without a workload identity, which the guest refuses; + // 8. a capability outside the entry's allowlist, `CAP_SYS_ADMIN`, which the entry refuses. + let executions = [ + execution( + &["--identity", "0:0"], + &["/bin/sh", "-c", "setsid sleep 600 & echo started"], + ), + execution( + &["--identity", &nobody], + &["/bin/sh", "-c", IDENTITY_SCRIPT], + ), + execution( + &["--identity", "0:0", "--capabilities", "0"], + &["/bin/sh", "-c", IDENTITY_SCRIPT], + ), + execution( + &[ + "--identity", + &nobody, + "--env", + "FIRST=1", + "--env", + "SECOND=two words", + ], + &["/usr/bin/env"], + ), + execution( + &["--identity", &nobody, "--cwd", "/usr/share"], + &["/bin/pwd"], + ), + execution( + &["--identity", &nobody], + &["/bin/sh", "-c", "cat /proc/sys/kernel/hostname; echo $$"], + ), + execution(&[], &["/bin/true"]), + execution( + &["--identity", "0:0", "--capabilities", "21"], + &["/bin/true"], + ), + ]; + let printed = vm.exec(&executions); + assert_ran(&printed, 1, 0, "started\n", ""); + let nobody_facts = identity_facts(NOBODY.0, NOBODY.1, "65534", 0); + assert_ran(&printed, 2, 0, &nobody_facts, ""); + assert_ran(&printed, 3, 0, &identity_facts(0, 0, "0", 1), ""); + assert_ran(&printed, 4, 0, "FIRST=1\nSECOND=two words\n", ""); + assert_ran(&printed, 5, 0, "/usr/share\n", ""); + assert_ran(&printed, 6, 0, "nvx-sandbox\n1\n", ""); + assert_refused(&printed, 7, EINVAL); + assert_refused(&printed, 8, EPERM); +} + +/// Checks the VM of the `executions_run_as_the_workload_identity` scenario, whose sandbox's +/// workload identity is [`NOBODY`]. +fn check_workload_executions(vm: &mut Vm<'_>) { + assert_eq!(vm.config.workload_identity, Some(NOBODY)); + let executions = [ + execution(&[], &["/bin/sh", "-c", IDENTITY_SCRIPT]), + execution(&["--env", "ONLY=this"], &["/usr/bin/env"]), + execution(&["--identity", &ids(NOBODY)], &["/bin/true"]), + ]; + let printed = vm.exec(&executions); + let nobody_facts = identity_facts(NOBODY.0, NOBODY.1, "65534", 0); + assert_ran(&printed, 1, 0, &nobody_facts, ""); + assert_ran(&printed, 2, 0, "ONLY=this\n", ""); + assert_refused(&printed, 3, EINVAL); +} From a2154900fa4dd29b711cf124a9fa322962c8df5e Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 02:34:34 -0700 Subject: [PATCH 2/7] tests: send the cancellation of an execution that the host gives up on at once A peer without grace or settlement time gives up on an execution in the step that ends it, for its timeout or its caller's cancellation. HostExecution::serve took the peer's events before its messages, and the last event, once taken, discarded the cancellation that waited to be sent, so the guest's program ran on after the host's side finished. serve now takes the messages first, as the example of nvx::exec does. The driver's sends wait for the transport, as the client's do, until the run's deadline, and no timer fires meanwhile. Its documentation now says so, and that a deadline that passes in a receive leaves the client usable, while one that passes in a send ends it, which the tests now check, the latter with a guest that stops reading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 30 +++++++++----- tests/runtime-adapter/src/exec.rs | 12 ++++-- tests/runtime-adapter/src/exec/tests.rs | 41 +++++++++++++++++++- tests/runtime-adapter/src/host/executions.rs | 9 ++++- tests/runtime-adapter/tests/host_session.rs | 34 ++++++++++++++-- 5 files changed, 105 insertions(+), 21 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index f749bf514..1dc02954f 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -706,7 +706,13 @@ answer the wrong request. sends, and no execution's messages wait for another's. A message of the guest about no execution, about one that the driver did not start, whose body does not decode, or that breaks the engine's rules, ends the run, as a - deadline that passes first does. + deadline that passes first does: in a receive, which leaves the client + usable, or in a send. Each send waits, as the client's do, until the + transport takes the message or the run's deadline passes, and no timer fires + meanwhile, so a guest that stops reading the session holds the run until its + deadline, when the send's timeout ends the client. The adapter's guest agent + keeps reading the session while its own messages wait, until more than 4 MiB + of them do. The guest ends its session on a second start of the same key, even in a later epoch of the session, so the keys of a driver's executions must be new to the @@ -1113,13 +1119,15 @@ independent adapter's does: The unit tests of `frame`, `root`, `probe`, and `exec` run in every configuration. They cover each kind's encoding, messages in pieces of any size, -a message of each kind split at every byte, the headers that the framing -refuses before their body arrives, and the root configurations, readinesses, -parts of mapping tables, failures, probes, facts, and messages of executions -that decode, and those that don't. In the guest-only configuration, -the unit tests of `worker::probes` cover what the probes read: the mount table -with its escapes, the kernel's records of a process, `os-release`, and the -guest's memory, and the probes and arguments that the worker refuses. Those of +a message of each kind split at every byte, the headers that the framing refuses +before their body arrives, and the root configurations, readinesses, parts of +mapping tables, failures, probes, facts, and messages of executions that decode, +and those that don't, and the host's side of an execution, which still sends the +cancellation of one that it gives up on in the step that ends it, without grace +or settlement time. In the guest-only configuration, the unit tests of +`worker::probes` cover what the probes read: the mount table with its escapes, +the kernel's records of a process, `os-release`, and the guest's memory, and the +probes and arguments that the worker refuses. Those of `worker::executions`, which spawn each program without entering a root, as a fake root does, hand an execution the host's cancellation once its timeout's instant passed, after its program ended before it, as a worker that @@ -1197,9 +1205,11 @@ Windows. They cover: not decode, and a second start of an ID, of which the driver sends nothing, with a client that stays usable; a message about an execution that it did not start, about none, whose body does not decode, or that breaks the - engine's rules, and the end of the stream, each of which ends the run; and a + engine's rules, and the end of the stream, each of which ends the run; a timeout that fires while the guest says nothing, after which the execution - is cancelled, then fenced, and a run whose deadline passes first. + is cancelled, then fenced; and a run whose deadline passes first, in a + receive, which leaves the client usable, or in a send that waits for a guest + that stopped reading, which ends it. Through a relay between its test broker and a pseudo-terminal, `tests/guest_session.rs` also checks that `guest::serve_terminal` and diff --git a/tests/runtime-adapter/src/exec.rs b/tests/runtime-adapter/src/exec.rs index 05aaab117..135e161af 100644 --- a/tests/runtime-adapter/src/exec.rs +++ b/tests/runtime-adapter/src/exec.rs @@ -679,7 +679,8 @@ impl HostExecution { } /// Takes every step that the source, the sink, and the peer can take at `now`, and returns - /// the peer's messages for the guest, as frames, in order. + /// the peer's messages for the guest, as frames, in order, the cancellation of an execution + /// that the peer gave up on in this step included. pub fn serve(&mut self, now: Instant) -> Vec { if self.start.input && !self.ended { self.sent += self.peer.input(&self.input[self.sent..]); @@ -702,9 +703,9 @@ impl HostExecution { } } self.peer.tick(now); - while let Some(event) = self.peer.next_event() { - self.events.push(event); - } + // The messages go first: a peer without grace or settlement time gives up on an + // execution in the step that ends it, and its last event, once taken, discards the + // cancellation that waits to be sent. let mut buffer = vec![0; self.start.limits.chunk()]; let mut messages = Vec::new(); while let Some(message) = self.peer.next_message(&mut buffer) { @@ -712,6 +713,9 @@ impl HostExecution { messages.push(message.encode()); } } + while let Some(event) = self.peer.next_event() { + self.events.push(event); + } messages } } diff --git a/tests/runtime-adapter/src/exec/tests.rs b/tests/runtime-adapter/src/exec/tests.rs index db112acfb..5f24c2d68 100644 --- a/tests/runtime-adapter/src/exec/tests.rs +++ b/tests/runtime-adapter/src/exec/tests.rs @@ -1,5 +1,7 @@ //! The encodings of the execution messages: round trips, the layout of a start, the bodies that -//! break the rules, and a peer and a runner that exchange their messages through the frames. +//! break the rules, and a peer and a runner that exchange their messages through the frames; and +//! the host's side of an execution, which sends the cancellation of one that it gives up on at +//! once. use std::time::{Duration, Instant}; @@ -8,7 +10,10 @@ use nvx::exec::{ Timers, Window, peer, runner, }; -use super::{AccountMode, GuestMessage, HostMessage, Identity, KEY_LEN, MAX_CHUNK, Start, name}; +use super::{ + AccountMode, GuestMessage, HostExecution, HostMessage, Identity, KEY_LEN, MAX_CHUNK, Start, + name, +}; use crate::frame::{MAX_BODY, Message}; /// The key of most executions in these tests. @@ -655,3 +660,35 @@ fn a_peer_and_a_runner_exchange_their_messages_through_the_frames() { .collect::>(); assert_eq!(interleaved, input); } + +#[test] +fn the_host_s_side_sends_the_cancellation_of_an_execution_that_it_gives_up_on_at_once() { + // Without grace or settlement time, the host's side gives up on an execution in the step that + // ends it, for its timeout or its caller's cancellation, and still sends the cancellation. + let now = Instant::now(); + let immediate = Timers::new(Duration::ZERO, Duration::ZERO); + let timeout = Duration::from_secs(1); + for (timers, cause) in [ + (immediate.timeout(timeout), Cause::TimedOut), + (immediate, Cause::Cancelled), + ] { + let start = Start::new(KEY, shared(), timers, ["/bin/program"]); + let mut execution = HostExecution::new(start, timers, Vec::new(), now); + // The credit that opens the output's pool. + assert_eq!(execution.serve(now).len(), 1, "{cause:?}"); + assert!(execution.outcome().is_none(), "{cause:?}"); + let later = now + timeout; + if cause == Cause::Cancelled { + execution.peer_mut().cancel(later); + } + assert_eq!( + execution.serve(later), + [HostMessage::Cancel(KEY, cause).encode()], + "{cause:?}" + ); + let outcome = execution.outcome().expect("the host's side gave up"); + assert_eq!((outcome.cause(), outcome.status()), (Some(cause), None)); + assert!(!outcome.is_reusable(), "{cause:?}"); + assert!(execution.serve(later).is_empty(), "{cause:?}"); + } +} diff --git a/tests/runtime-adapter/src/host/executions.rs b/tests/runtime-adapter/src/host/executions.rs index 0c26e4395..8b577b1ba 100644 --- a/tests/runtime-adapter/src/host/executions.rs +++ b/tests/runtime-adapter/src/host/executions.rs @@ -85,7 +85,11 @@ impl From for ExecutionsError { /// guest's messages to the executions, and theirs to the guest, until every execution finished. /// One thread runs them all: each receive waits at most until the earliest timer of the /// executions, which so fires in time whatever the guest sends, and the guest's messages about one -/// execution never wait for another's. +/// execution never wait for another's. Each send waits, as the client's do, until the transport +/// takes the message or the run's deadline passes, and no timer fires meanwhile: a guest that +/// stops reading the session holds the run until its deadline, when the run fails, and the send's +/// timeout ends the client. The adapter's guest agent keeps reading the session while its own +/// messages wait, until more than 4 MiB of them do, as its worker describes. /// /// The guest ends its session on a second start of the same key, even in a later epoch of the /// session, so the keys of the executions must differ from those of every execution that the guest @@ -132,7 +136,8 @@ impl Executions { /// Carries the guest's messages to the executions, and theirs to the guest, over `client`, /// until every execution finished, and fails with [`ExecutionsError::TimedOut`] once - /// `deadline` passed first. + /// `deadline` passed first: in a receive, which leaves the client usable, or in a send, which + /// ends it. pub fn run(&mut self, client: &mut Client, deadline: Instant) -> Result<(), ExecutionsError> { loop { let now = Instant::now(); diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index 792e75e00..cae7e5d96 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -12,7 +12,8 @@ //! the mapping table in parts, and `Start`, and each failure of the order, after which the host //! sends nothing more, or stops the guest. The broker's guest also runs the runners of NVX's //! execution engine for the executions that the host program's driver runs, so that the tests cover -//! the driver's output, credit, outcomes, and timers, and the refusals that end its run. +//! the driver's output, credit, outcomes, and timers, the refusals that end its run, and a guest +//! that stops reading. #![cfg(feature = "host")] use std::collections::{BTreeMap, VecDeque}; @@ -2192,14 +2193,41 @@ fn the_driver_fires_the_timers_of_its_executions_while_the_guest_says_nothing() assert_eq!(outcome.status(), None); assert!(!outcome.is_reusable()); - // A run whose deadline passes before its executions finish fails. + // A run whose deadline passes before its executions finish fails, in a receive, which leaves + // the client usable: the guest takes the start's credit, then answers a ping. let (mut broker, client) = attached(FIRST, usize::MAX); let running = run(client, vec![start_of(1)], Instant::now() + SHORT); broker.messages(1); - let (_, ran) = joined(running); + let (client, ran) = joined(running); assert!( matches!(ran, Err(ExecutionsError::TimedOut)), "{:?}", ran.err() ); + broker.messages(1); + let pinging = spawn(client, |client| client.ping(deadline())); + assert_eq!(broker.answer(OUTER_MAX_PAYLOAD), Message::Ping); + joined(pinging).1.unwrap(); +} + +#[test] +fn the_driver_s_sends_wait_for_a_guest_that_stops_reading_until_the_deadline_of_its_run() { + // The guest takes the start, then stops reading, and the host's stream holds 16 bytes, so the + // driver's next send, the credit of the output's pool, waits for the transport until the + // run's deadline, whose timeout ends the client. + let (mut broker, client) = attach_over(connect_bounded(FIRST, usize::MAX, 16)); + let started = Instant::now(); + let running = run(client, vec![start_of(1)], started + SHORT); + assert_eq!( + broker.messages(1), + [HostMessage::Start(start_of(1)).encode()] + ); + let (mut client, ran) = joined(running); + assert!(started.elapsed() >= SHORT); + assert!( + matches!(ran, Err(ExecutionsError::TimedOut)), + "{:?}", + ran.err() + ); + assert_ended(&mut client); } From 5492e2ffcf487e1b5abfe86e6c5b64fd1ae098fb Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 03:06:13 -0700 Subject: [PATCH 3/7] tests: bound what the runtime adapter's driver sends before it reads The session's broker passes the guest's credit for the host's records on only behind the guest's own records, which wait while the host does not read, and takes no more of the host's records than that credit allows. So a single-threaded host that sends without reading can wait for credit that only its reading would free, even with a guest that reads. The exec command sent every start before it read anything, and a batch of hundreds, which the guest refuses beyond the executions that it runs at once, could fill the way back with refusals and hold the next start until its deadline. The driver now runs at most exec::MAX_EXECUTIONS executions that have not finished, as many as the guest runs at once. Their answers to their starts wait for the run, since no output flows before the run's credit does, and exec refuses more executions as a usage error. The driver also sends no program's input, which one thread could not send while it reads the programs' output: it refuses a start that says that the host sends input, and Executions::start no longer takes any. The worker's limit of executions now comes from that shared constant. The test of a send that waits for a guest that stops reading counts the run's deadline from the start's send, so that the deadline passes in the send that it checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 37 +++++---- .../src/bin/runtime-adapter-host.rs | 19 +++-- tests/runtime-adapter/src/exec.rs | 4 + tests/runtime-adapter/src/host/executions.rs | 58 +++++++++++--- tests/runtime-adapter/src/worker.rs | 5 +- tests/runtime-adapter/tests/host_session.rs | 78 ++++++++++++++----- tests/runtime-adapter/tests/layered_launch.rs | 16 +++- 7 files changed, 159 insertions(+), 58 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 1dc02954f..146490953 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -698,7 +698,9 @@ answer the wrong request. as the guest decodes it. A start that does not, such as one with a relative working directory, and one of an ID that the driver started already, fail before anything is sent, and leave the client usable: the guest would end its - session on either; + session on either. So do a start that says that the host sends the program's + input, and one beyond `exec::MAX_EXECUTIONS`, 8, executions that the driver + started and that have not finished, as the guest runs no more at once; - `run` carries the guest's messages to each execution's `exec::HostExecution`, and theirs to the guest, until every execution finished. One thread runs them all: each receive waits at most until the @@ -710,9 +712,15 @@ answer the wrong request. usable, or in a send. Each send waits, as the client's do, until the transport takes the message or the run's deadline passes, and no timer fires meanwhile, so a guest that stops reading the session holds the run until its - deadline, when the send's timeout ends the client. The adapter's guest agent - keeps reading the session while its own messages wait, until more than 4 MiB - of them do. + deadline, when the send's timeout ends the client. + +Even a guest that reads can hold a send while the host does not read: the +session's broker passes the guest's credit for the host's records on only +behind the guest's own records, and takes no more of the host's records than +that credit allows. So the driver sends little before it reads: no program's +input, which one thread could not send while it reads the programs' output, and +the starts of at most 8 executions that have not finished, whose answers wait +for the run, as no program's output flows before the run's credit does. The guest ends its session on a second start of the same key, even in a later epoch of the session, so the keys of a driver's executions must be new to the @@ -1047,7 +1055,8 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from - `exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as `reattach` does, then runs each EXECUTION in the VM's layered root, all at once, over that one session, through `host::Executions`, and prints their - outputs and outcomes once every one finished. An EXECUTION, + outputs and outcomes once every one finished. It runs at most 8 executions, + as many as the guest runs at once. An EXECUTION, `[--identity UID:GID [--groups GID,...] [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, runs PROGRAM with its ARGs, up to a lone `;`, which a shell takes quoted, as `find -exec` takes them, without standard input. Without `--identity`, it @@ -1202,14 +1211,16 @@ Windows. They cover: executions at once, one with more output than its credit holds, each byte-exact, with its outcome, and one whose launch the guest refuses, whose records carry their messages whole or split; the starts that the guest would - not decode, and a second start of an ID, of which the driver sends nothing, - with a client that stays usable; a message about an execution that it did - not start, about none, whose body does not decode, or that breaks the - engine's rules, and the end of the stream, each of which ends the run; a - timeout that fires while the guest says nothing, after which the execution - is cancelled, then fenced; and a run whose deadline passes first, in a - receive, which leaves the client usable, or in a send that waits for a guest - that stopped reading, which ends it. + not decode, a second start of an ID, a start whose input the host would + send, and a start beyond 8 executions that have not finished, of which the + driver sends nothing, with a client that stays usable, and the places of the + executions that finished, which another start takes; a message about an + execution that it did not start, about none, whose body does not decode, or + that breaks the engine's rules, and the end of the stream, each of which ends + the run; a timeout that fires while the guest says nothing, after which the + execution is cancelled, then fenced; and a run whose deadline passes first, + in a receive, which leaves the client usable, or in a send that waits for a + guest that stopped reading, which ends it. Through a relay between its test broker and a pseudo-terminal, `tests/guest_session.rs` also checks that `guest::serve_terminal` and diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 59709832e..db1fb629d 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -75,7 +75,8 @@ //! - `runtime-adapter-host exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as //! `reattach` does, then runs each EXECUTION in the VM's layered root, all at once, over that //! one session, through `runtime_adapter::host::Executions`, and prints their outputs and -//! outcomes once every one finished. An EXECUTION, `[--identity UID:GID [--groups GID,...] +//! outcomes once every one finished. It runs at most 8 executions, as many as the guest runs at +//! once; see `runtime_adapter::exec::MAX_EXECUTIONS`. An EXECUTION, `[--identity UID:GID [--groups GID,...] //! [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, //! runs PROGRAM with its ARGs, up to a lone `;`, as `find -exec` takes them, without standard //! input. Without `--identity`, it names no identity, and runs as the sandbox's workload @@ -812,6 +813,14 @@ fn parse_exec( break; } } + // The driver runs no more executions at once than the guest does. + if executions.len() > exec::MAX_EXECUTIONS { + return Err(format!( + "exec runs at most {} executions, not {}", + exec::MAX_EXECUTIONS, + executions.len() + )); + } Ok(Command::Exec { store, guest_build: guest_build.unwrap_or_else(|| GUEST_BUILD.to_owned()), @@ -1266,13 +1275,7 @@ fn exec(store: &Path, guest_build: &str, executions: &[Execution]) -> Result error.fmt(formatter), Self::Start(error) => write!(formatter, "the host refuses to send the start: {error}"), Self::Duplicate(id) => write!(formatter, "the host starts execution {id} twice"), + Self::Input => formatter.write_str("the host sends no execution's input"), + Self::Full => write!( + formatter, + "the host runs {MAX_EXECUTIONS} executions that have not finished already" + ), Self::Body(error) => error.fmt(formatter), Self::Unexpected(kind) => write!( formatter, @@ -66,7 +77,12 @@ impl std::error::Error for ExecutionsError { Self::Client(error) => Some(error), Self::Start(error) | Self::Body(error) => Some(error), Self::Violation(_, violation) => Some(violation), - Self::Duplicate(_) | Self::Unexpected(_) | Self::Unknown(_) | Self::TimedOut => None, + Self::Duplicate(_) + | Self::Input + | Self::Full + | Self::Unexpected(_) + | Self::Unknown(_) + | Self::TimedOut => None, } } } @@ -86,10 +102,17 @@ impl From for ExecutionsError { /// One thread runs them all: each receive waits at most until the earliest timer of the /// executions, which so fires in time whatever the guest sends, and the guest's messages about one /// execution never wait for another's. Each send waits, as the client's do, until the transport -/// takes the message or the run's deadline passes, and no timer fires meanwhile: a guest that -/// stops reading the session holds the run until its deadline, when the run fails, and the send's -/// timeout ends the client. The adapter's guest agent keeps reading the session while its own -/// messages wait, until more than 4 MiB of them do, as its worker describes. +/// takes the message or the run's deadline passes, and no timer fires meanwhile, so a guest that +/// stops reading the session holds the run until its deadline, when the send's timeout ends the +/// client. +/// +/// Even a guest that reads can hold a send while the host does not read: the session's broker +/// passes the guest's credit for the host's records on only behind the guest's own records, and +/// takes no more of the host's records than that credit allows. So the driver sends little before +/// it reads: no program's input, which one thread could not send while it reads the programs' +/// output, and the starts of at most [`MAX_EXECUTIONS`] executions that have not finished, as +/// many as the guest runs at once, whose answers wait for the run, as no program's output flows +/// before the run's credit does. /// /// The guest ends its session on a second start of the same key, even in a later epoch of the /// session, so the keys of the executions must differ from those of every execution that the guest @@ -106,29 +129,40 @@ impl Executions { Self::default() } - /// Starts the execution of `start` over `client`, whose peer runs with `timers`, and whose - /// source hands over `input` if the start says that the host sends the program's input: sends - /// the start by `deadline`. The timeout of `timers` runs from then. + /// Starts the execution of `start` over `client`, whose peer runs with `timers`: sends the + /// start by `deadline`. The timeout of `timers` runs from then. /// /// A start of an ID that the driver started already, or whose encoding does not decode, as /// one with a relative working directory or an environment entry without a name, fails /// before anything is sent, and leaves the client usable, as a message that the framing - /// refuses does: the guest would end its session on either. + /// refuses does: the guest would end its session on either. So does a start that says that + /// the host sends the program's input, and one beyond [`MAX_EXECUTIONS`] executions that have + /// not finished, which the driver does not send, as [`Executions`] describes. pub fn start( &mut self, client: &mut Client, start: Start, timers: Timers, - input: Vec, deadline: Instant, ) -> Result<(), ExecutionsError> { let id = start.key.id(); if self.executions.contains_key(&id) { return Err(ExecutionsError::Duplicate(id)); } + if start.input { + return Err(ExecutionsError::Input); + } + let running = self + .executions + .values() + .filter(|execution| execution.outcome().is_none()) + .count(); + if running >= MAX_EXECUTIONS { + return Err(ExecutionsError::Full); + } let body = start.encode(); Start::decode(&body).map_err(ExecutionsError::Start)?; - let execution = HostExecution::new(start, timers, input, Instant::now()); + let execution = HostExecution::new(start, timers, Vec::new(), Instant::now()); client.send(&Message::ExecStart(body), Some(deadline))?; self.executions.insert(id, execution); Ok(()) diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index c2db2b503..9f92c0923 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -247,8 +247,9 @@ pub const EXECUTING: &str = "the worker runs executions, and probes nothing unti /// The most executions that the worker runs at once, those that the host abandoned included, and /// those that ended but keep their places: with cgroups, those whose cgroups the worker has not /// removed, as those of the executions that did not settle, and without them, those whose -/// programs the worker forgot before it collected their statuses. -pub const MAX_EXECUTIONS: usize = 8; +/// programs the worker forgot before it collected their statuses. The host's side knows it as +/// [`exec::MAX_EXECUTIONS`](crate::exec::MAX_EXECUTIONS). +pub const MAX_EXECUTIONS: usize = crate::exec::MAX_EXECUTIONS; /// The most bytes that one execution may hold in the worker: what its runner holds, a chunk of /// each output stream and the input's window. The output that waits for the session's stream diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index cae7e5d96..069d06e14 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -36,7 +36,7 @@ use nvx::protocol::{ OUTER_MAX_PAYLOAD, OUTER_MIN_RECEIVE_WINDOW, }; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::exec::{GuestMessage, HostMessage, Start}; +use runtime_adapter::exec::{GuestMessage, HostMessage, MAX_EXECUTIONS, Start}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, MAX_TEXT_BODY, Message}; use runtime_adapter::host::{ self, Client, ClientError, Executions, ExecutionsError, MapReply, OrderError, @@ -2000,7 +2000,7 @@ fn run( let mut executions = Executions::new(); for start in starts { let timers = start.timers; - executions.start(client, start, timers, Vec::new(), deadline)?; + executions.start(client, start, timers, deadline)?; } executions.run(client, deadline)?; Ok(executions) @@ -2066,7 +2066,7 @@ fn the_driver_runs_executions_at_once_and_collects_their_output_and_outcomes() { } #[test] -fn the_driver_refuses_starts_that_the_guest_would_not_take_and_sends_nothing() { +fn the_driver_refuses_starts_before_it_sends_them() { let (mut broker, mut client) = attached(FIRST, usize::MAX); let mut executions = Executions::new(); // The guest ends its session on a start that it cannot decode. @@ -2076,30 +2076,61 @@ fn the_driver_refuses_starts_that_the_guest_would_not_take_and_sends_nothing() { Start::new(Key::new(1, 1), limits(), timers(), [""]), ]; for start in undecodable { - let refused = - executions.start(&mut client, start.clone(), timers(), Vec::new(), deadline()); + let refused = executions.start(&mut client, start.clone(), timers(), deadline()); assert!( matches!(refused, Err(ExecutionsError::Start(_))), "{start:?}: {refused:?}" ); } - // A second start of an ID. - executions - .start(&mut client, start_of(1), timers(), Vec::new(), deadline()) - .unwrap(); - let refused = executions.start(&mut client, start_of(1), timers(), Vec::new(), deadline()); + // A start whose input the host would send while it reads the guest's output. + let refused = executions.start(&mut client, start_of(1).with_input(), timers(), deadline()); assert!( - matches!(refused, Err(ExecutionsError::Duplicate(1))), + matches!(refused, Err(ExecutionsError::Input)), "{refused:?}" ); - // The guest received the one start alone, and the client stays usable. - assert_eq!( - broker.messages(1), - [HostMessage::Start(start_of(1)).encode()] + // A second start of an ID, and a start beyond the executions that the guest runs at once. + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + for id in 1..=most { + executions + .start(&mut client, start_of(id), timers(), deadline()) + .unwrap(); + } + let refused = executions.start(&mut client, start_of(1), timers(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Duplicate(1))), + "{refused:?}" ); + let refused = executions.start(&mut client, start_of(most + 1), timers(), deadline()); + assert!(matches!(refused, Err(ExecutionsError::Full)), "{refused:?}"); + // The guest received those starts alone, and the client stays usable. + let starts = (1..=most) + .map(|id| HostMessage::Start(start_of(id)).encode()) + .collect::>(); + assert_eq!(broker.messages(MAX_EXECUTIONS), starts); let pinging = spawn(client, |client| client.ping(deadline())); assert_eq!(broker.answer(OUTER_MAX_PAYLOAD), Message::Ping); joined(pinging).1.unwrap(); + + // The executions that finished leave their places: once the guest refused as many launches, + // the driver starts another execution. + let (mut broker, client) = attached(FIRST, usize::MAX); + let starts = (1..=most).map(start_of).collect::>(); + let running = run(client, starts.clone(), deadline()); + let refused = Workload { + launch: Err(LaunchFailure::new(0, 11)), + output: [Vec::new(), Vec::new()], + status: Status::Exited(0), + }; + let played = play(&mut broker, MAX_EXECUTIONS, OUTER_MAX_PAYLOAD, |_| { + refused.clone() + }); + assert_eq!(played, starts); + let (mut client, executions) = joined(running); + let mut executions = executions.unwrap(); + assert!(executions.is_finished()); + executions + .start(&mut client, start_of(most + 1), timers(), deadline()) + .unwrap(); } #[test] @@ -2214,16 +2245,23 @@ fn the_driver_fires_the_timers_of_its_executions_while_the_guest_says_nothing() fn the_driver_s_sends_wait_for_a_guest_that_stops_reading_until_the_deadline_of_its_run() { // The guest takes the start, then stops reading, and the host's stream holds 16 bytes, so the // driver's next send, the credit of the output's pool, waits for the transport until the - // run's deadline, whose timeout ends the client. + // run's deadline, whose timeout ends the client. The run's deadline counts from the start's + // send, which the guest took. let (mut broker, client) = attach_over(connect_bounded(FIRST, usize::MAX, 16)); - let started = Instant::now(); - let running = run(client, vec![start_of(1)], started + SHORT); + let running = spawn(client, |client| { + let mut executions = Executions::new(); + executions.start(client, start_of(1), timers(), deadline())?; + let started = Instant::now(); + let ran = executions.run(client, started + SHORT); + Ok::<_, ExecutionsError>((started.elapsed(), ran)) + }); assert_eq!( broker.messages(1), [HostMessage::Start(start_of(1)).encode()] ); - let (mut client, ran) = joined(running); - assert!(started.elapsed() >= SHORT); + let (mut client, result) = joined(running); + let (elapsed, ran) = result.unwrap(); + assert!(elapsed >= SHORT, "{elapsed:?}"); assert!( matches!(ran, Err(ExecutionsError::TimedOut)), "{:?}", diff --git a/tests/runtime-adapter/tests/layered_launch.rs b/tests/runtime-adapter/tests/layered_launch.rs index be523e0e2..52b32b7b3 100644 --- a/tests/runtime-adapter/tests/layered_launch.rs +++ b/tests/runtime-adapter/tests/layered_launch.rs @@ -4,8 +4,8 @@ //! inside the directories that the adapter authorizes, attaches it, and verifies it before the //! launch, composes OpenVMM's arguments without NVX's managed lifecycle, records what it launches, //! with the bindings that it checked, and refuses options that do not make a command, shares that -//! it cannot export before it touches the store, and executions whose starts the guest would not -//! take. The tests need no hypervisor. +//! it cannot export before it touches the store, executions whose starts the guest would not take, +//! and more executions than the guest runs at once. The tests need no hypervisor. #![cfg(feature = "host")] use std::ffi::OsString; @@ -25,6 +25,7 @@ use nvx::protocol::root::{ImageFormat, RootImage, RootLayout, RootUpper}; use nvx::protocol::{HostMap, MAX_HOST_MAPPINGS}; use nvx::uuid::Uuid; use nvx::{ErrorCode, ErrorKind, Hypervisor}; +use runtime_adapter::exec::MAX_EXECUTIONS; use runtime_adapter::hook::STEP_HOOK; use runtime_adapter::layered::{Bindings, LaunchRecord, LayeredLaunch, RECORD_NAME, SANDBOX_TOKEN}; use runtime_adapter::root::RootConfig; @@ -1041,7 +1042,8 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { ]); // Each execution of `exec` has options of its own, before `--` and its program, and its // arguments end at a lone `;`. The program refuses a start that the guest would not decode, - // on which the guest would end its session, before it attaches. + // on which the guest would end its session, and more executions than the guest runs at once, + // before it attaches. let program = ["--", "/bin/true"]; let exec = |options: &[&'static str]| [&["exec", "store"], options, &program[..]].concat(); commands.extend([ @@ -1101,6 +1103,14 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { .concat(), "--groups needs --identity", ), + ( + [ + vec!["exec", "store"], + vec![program.to_vec(); MAX_EXECUTIONS + 1].join(&";"), + ] + .concat(), + "exec runs at most", + ), ]); for (arguments, problem) in commands { let output = Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")) From 7b500452775340caf5f1f23d4bf436ebe36676b5 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 03:26:18 -0700 Subject: [PATCH 4/7] tests: run one bounded batch of executions per runtime adapter driver Small sends did not yet bound what the driver sends before the guest's credit for the host's records can reach the broker. An output window of many frames of few bytes lets the guest queue thousands of output messages ahead of its credit, while the driver sends a credit for each one that it reads. A start after a run may also wait for credit behind the output of executions that the run gave up on, or left running when its deadline passed. The driver now takes only output windows that count frames, at most host::MAX_OUTPUT_FRAMES of them, so that the credits for the guest's messages ahead of its own credit stay well within the guest's receive window. It also runs one batch: it starts every execution before it runs them and none after, so a later batch takes an attachment of its own, as each command of the host program does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 23 ++++--- tests/runtime-adapter/src/host.rs | 2 +- tests/runtime-adapter/src/host/executions.rs | 69 ++++++++++++++------ tests/runtime-adapter/tests/host_session.rs | 52 +++++++++------ 4 files changed, 99 insertions(+), 47 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 146490953..9cac3a1be 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -699,8 +699,10 @@ answer the wrong request. working directory, and one of an ID that the driver started already, fail before anything is sent, and leave the client usable: the guest would end its session on either. So do a start that says that the host sends the program's - input, and one beyond `exec::MAX_EXECUTIONS`, 8, executions that the driver - started and that have not finished, as the guest runs no more at once; + input, one whose output windows count bytes, or more than + `host::MAX_OUTPUT_FRAMES`, 32, frames, one beyond `exec::MAX_EXECUTIONS`, 8, + executions, as the guest runs no more at once, and any once the driver ran: + a driver runs one batch, whose executions it starts before it runs them; - `run` carries the guest's messages to each execution's `exec::HostExecution`, and theirs to the guest, until every execution finished. One thread runs them all: each receive waits at most until the @@ -718,9 +720,14 @@ Even a guest that reads can hold a send while the host does not read: the session's broker passes the guest's credit for the host's records on only behind the guest's own records, and takes no more of the host's records than that credit allows. So the driver sends little before it reads: no program's -input, which one thread could not send while it reads the programs' output, and -the starts of at most 8 executions that have not finished, whose answers wait -for the run, as no program's output flows before the run's credit does. +input, which one thread could not send while it reads the programs' output; the +starts of at most 8 executions, whose answers wait for the run, as no program's +output flows before the run's credit does; and, in the run, besides a +cancellation for each execution, at most a credit for each message that it +reads, of output windows of at most 32 frames, so that the guest's messages +ahead of its own credit stay few. The guest may still send messages about the +executions of a run that gave up on one or failed, so a later batch takes an +attachment of its own, as each command of the host program does. The guest ends its session on a second start of the same key, even in a later epoch of the session, so the keys of a driver's executions must be new to the @@ -1212,9 +1219,9 @@ Windows. They cover: byte-exact, with its outcome, and one whose launch the guest refuses, whose records carry their messages whole or split; the starts that the guest would not decode, a second start of an ID, a start whose input the host would - send, and a start beyond 8 executions that have not finished, of which the - driver sends nothing, with a client that stays usable, and the places of the - executions that finished, which another start takes; a message about an + send, starts whose output windows count bytes or more than 32 frames, a + start beyond 8 executions, and a start once the driver ran, of which the + driver sends nothing, with a client that stays usable; a message about an execution that it did not start, about none, whose body does not decode, or that breaks the engine's rules, and the end of the stream, each of which ends the run; a timeout that fires while the guest says nothing, after which the diff --git a/tests/runtime-adapter/src/host.rs b/tests/runtime-adapter/src/host.rs index fb79445d9..11cb1211c 100644 --- a/tests/runtime-adapter/src/host.rs +++ b/tests/runtime-adapter/src/host.rs @@ -39,7 +39,7 @@ use crate::frame::{BodyError, Decoder, FrameError, Message}; use crate::probe::{Facts, Probe, ProbeReply}; use crate::root::{MapFailure, Ready, RootConfig, RootFailure, RootReply, encode_maps}; -pub use executions::{Executions, ExecutionsError}; +pub use executions::{Executions, ExecutionsError, MAX_OUTPUT_FRAMES}; pub use map_reply::MapReply; pub use order_error::OrderError; diff --git a/tests/runtime-adapter/src/host/executions.rs b/tests/runtime-adapter/src/host/executions.rs index 00cf8df1d..bfd269eae 100644 --- a/tests/runtime-adapter/src/host/executions.rs +++ b/tests/runtime-adapter/src/host/executions.rs @@ -5,13 +5,19 @@ use std::collections::BTreeMap; use std::fmt; use std::time::Instant; -use nvx::exec::{Key, Timers, Violation}; +use nvx::exec::{Key, Timers, Unit, Violation}; use nvx::host::nvxs::HostConnectionError; use super::{Client, ClientError}; use crate::exec::{GuestMessage, HostExecution, MAX_EXECUTIONS, Start, name}; use crate::frame::{BodyError, Message}; +/// The most frames of each output pool of an execution of [`Executions`], whose output windows +/// count frames: then the credits that the driver sends for the messages that the guest of +/// [`MAX_EXECUTIONS`] executions may send ahead of its own credit fit the guest's receive window, +/// `nvx::protocol::OUTER_MIN_RECEIVE_WINDOW`, several times over. +pub const MAX_OUTPUT_FRAMES: u32 = 32; + /// A failure of [`Executions`]. #[derive(Debug)] #[non_exhaustive] @@ -25,9 +31,13 @@ pub enum ExecutionsError { /// A start that says that the host sends the program's input, which the driver does not /// send, and so did not send the start. Input, - /// A start beyond [`MAX_EXECUTIONS`] executions that the driver started and that have not - /// finished, which it did not send. + /// A start whose output windows count bytes, or more than [`MAX_OUTPUT_FRAMES`] frames, + /// which the driver did not send. + Window, + /// A start beyond [`MAX_EXECUTIONS`] executions of the driver, which it did not send. Full, + /// A start once the driver ran its executions, which it did not send. + Ran, /// The guest sent a message about an execution whose body does not decode. Body(BodyError), /// The guest sent a message of this kind, which is about no execution. @@ -47,10 +57,17 @@ impl fmt::Display for ExecutionsError { Self::Start(error) => write!(formatter, "the host refuses to send the start: {error}"), Self::Duplicate(id) => write!(formatter, "the host starts execution {id} twice"), Self::Input => formatter.write_str("the host sends no execution's input"), + Self::Window => write!( + formatter, + "the host takes output windows of at most {MAX_OUTPUT_FRAMES} frames" + ), Self::Full => write!( formatter, - "the host runs {MAX_EXECUTIONS} executions that have not finished already" + "the host runs at most {MAX_EXECUTIONS} executions at once" ), + Self::Ran => { + formatter.write_str("the host starts no execution once it ran its executions") + } Self::Body(error) => error.fmt(formatter), Self::Unexpected(kind) => write!( formatter, @@ -79,7 +96,9 @@ impl std::error::Error for ExecutionsError { Self::Violation(_, violation) => Some(violation), Self::Duplicate(_) | Self::Input + | Self::Window | Self::Full + | Self::Ran | Self::Unexpected(_) | Self::Unknown(_) | Self::TimedOut => None, @@ -99,6 +118,10 @@ impl From for ExecutionsError { /// /// [`start`](Self::start) sends the start of an execution, and [`run`](Self::run) carries the /// guest's messages to the executions, and theirs to the guest, until every execution finished. +/// A driver runs one batch: it starts every execution before it runs them, and none after. The +/// guest may still send messages about the executions of a run that gave up on one or failed, so +/// a later batch takes an attachment of its own, as each command of the host program does. +/// /// One thread runs them all: each receive waits at most until the earliest timer of the /// executions, which so fires in time whatever the guest sends, and the guest's messages about one /// execution never wait for another's. Each send waits, as the client's do, until the transport @@ -110,9 +133,11 @@ impl From for ExecutionsError { /// passes the guest's credit for the host's records on only behind the guest's own records, and /// takes no more of the host's records than that credit allows. So the driver sends little before /// it reads: no program's input, which one thread could not send while it reads the programs' -/// output, and the starts of at most [`MAX_EXECUTIONS`] executions that have not finished, as -/// many as the guest runs at once, whose answers wait for the run, as no program's output flows -/// before the run's credit does. +/// output; the starts of at most [`MAX_EXECUTIONS`] executions, as many as the guest runs at once, +/// whose answers wait for the run, as no program's output flows before the run's credit does; +/// and, in the run, besides a cancellation for each execution, at most a credit for each message +/// that it reads, of output windows of at most [`MAX_OUTPUT_FRAMES`] frames, so that the guest's +/// messages ahead of its own credit stay few. /// /// The guest ends its session on a second start of the same key, even in a later epoch of the /// session, so the keys of the executions must differ from those of every execution that the guest @@ -121,6 +146,8 @@ impl From for ExecutionsError { pub struct Executions { /// The host's side of each execution, by ID. executions: BTreeMap, + /// Whether the driver ran its executions. + ran: bool, } impl Executions { @@ -132,12 +159,13 @@ impl Executions { /// Starts the execution of `start` over `client`, whose peer runs with `timers`: sends the /// start by `deadline`. The timeout of `timers` runs from then. /// - /// A start of an ID that the driver started already, or whose encoding does not decode, as - /// one with a relative working directory or an environment entry without a name, fails - /// before anything is sent, and leaves the client usable, as a message that the framing - /// refuses does: the guest would end its session on either. So does a start that says that - /// the host sends the program's input, and one beyond [`MAX_EXECUTIONS`] executions that have - /// not finished, which the driver does not send, as [`Executions`] describes. + /// These starts fail before anything is sent, and leave the client usable, as a message that + /// the framing refuses does: one of an ID that the driver started already, or whose encoding + /// does not decode, as one with a relative working directory or an environment entry without + /// a name, on either of which the guest would end its session; and, as [`Executions`] + /// describes, one that says that the host sends the program's input, one whose output windows + /// count bytes or more than [`MAX_OUTPUT_FRAMES`] frames, one beyond [`MAX_EXECUTIONS`] + /// executions, and any once the driver ran. pub fn start( &mut self, client: &mut Client, @@ -145,6 +173,9 @@ impl Executions { timers: Timers, deadline: Instant, ) -> Result<(), ExecutionsError> { + if self.ran { + return Err(ExecutionsError::Ran); + } let id = start.key.id(); if self.executions.contains_key(&id) { return Err(ExecutionsError::Duplicate(id)); @@ -152,12 +183,11 @@ impl Executions { if start.input { return Err(ExecutionsError::Input); } - let running = self - .executions - .values() - .filter(|execution| execution.outcome().is_none()) - .count(); - if running >= MAX_EXECUTIONS { + let output = start.limits.output(); + if !matches!(output.unit(), Unit::Frames) || output.size() > MAX_OUTPUT_FRAMES { + return Err(ExecutionsError::Window); + } + if self.executions.len() >= MAX_EXECUTIONS { return Err(ExecutionsError::Full); } let body = start.encode(); @@ -173,6 +203,7 @@ impl Executions { /// `deadline` passed first: in a receive, which leaves the client usable, or in a send, which /// ends it. pub fn run(&mut self, client: &mut Client, deadline: Instant) -> Result<(), ExecutionsError> { + self.ran = true; loop { let now = Instant::now(); for execution in self.executions.values_mut() { diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index 069d06e14..a9a7424c0 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -39,7 +39,7 @@ use runtime_adapter::GUEST_BUILD; use runtime_adapter::exec::{GuestMessage, HostMessage, MAX_EXECUTIONS, Start}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, MAX_TEXT_BODY, Message}; use runtime_adapter::host::{ - self, Client, ClientError, Executions, ExecutionsError, MapReply, OrderError, + self, Client, ClientError, Executions, ExecutionsError, MAX_OUTPUT_FRAMES, MapReply, OrderError, }; use runtime_adapter::probe::{Facts, Probe, ProbeReply}; use runtime_adapter::root::{ @@ -2082,17 +2082,35 @@ fn the_driver_refuses_starts_before_it_sends_them() { "{start:?}: {refused:?}" ); } - // A start whose input the host would send while it reads the guest's output. + // A start whose input the host would send while it reads the guest's output, and starts whose + // output windows would let the guest send many messages ahead of its credit for the host's. let refused = executions.start(&mut client, start_of(1).with_input(), timers(), deadline()); assert!( matches!(refused, Err(ExecutionsError::Input)), "{refused:?}" ); - // A second start of an ID, and a start beyond the executions that the guest runs at once. + let program = ["/bin/program"]; + for output in [ + Window::bytes(64 * 1024), + Window::frames(MAX_OUTPUT_FRAMES + 1), + ] { + let limits = Limits::new(CHUNK, Window::frames(4), output).unwrap(); + let start = Start::new(Key::new(1, 1), limits, timers(), program); + let refused = executions.start(&mut client, start, timers(), deadline()); + assert!( + matches!(refused, Err(ExecutionsError::Window)), + "{output:?}: {refused:?}" + ); + } + // A second start of an ID, and a start beyond the executions that the guest runs at once, + // after starts of which one has the widest output windows that the driver takes. let most = u64::try_from(MAX_EXECUTIONS).unwrap(); - for id in 1..=most { + let mut starts = (1..=most).map(start_of).collect::>(); + let widest = Limits::new(CHUNK, Window::frames(4), Window::frames(MAX_OUTPUT_FRAMES)).unwrap(); + starts[0] = Start::new(Key::new(1, 1), widest, timers(), program); + for start in &starts { executions - .start(&mut client, start_of(id), timers(), deadline()) + .start(&mut client, start.clone(), timers(), deadline()) .unwrap(); } let refused = executions.start(&mut client, start_of(1), timers(), deadline()); @@ -2103,34 +2121,30 @@ fn the_driver_refuses_starts_before_it_sends_them() { let refused = executions.start(&mut client, start_of(most + 1), timers(), deadline()); assert!(matches!(refused, Err(ExecutionsError::Full)), "{refused:?}"); // The guest received those starts alone, and the client stays usable. - let starts = (1..=most) - .map(|id| HostMessage::Start(start_of(id)).encode()) + let sent = starts + .into_iter() + .map(|start| HostMessage::Start(start).encode()) .collect::>(); - assert_eq!(broker.messages(MAX_EXECUTIONS), starts); + assert_eq!(broker.messages(MAX_EXECUTIONS), sent); let pinging = spawn(client, |client| client.ping(deadline())); assert_eq!(broker.answer(OUTER_MAX_PAYLOAD), Message::Ping); joined(pinging).1.unwrap(); - // The executions that finished leave their places: once the guest refused as many launches, - // the driver starts another execution. + // A driver runs one batch: once it ran, it starts nothing more. let (mut broker, client) = attached(FIRST, usize::MAX); - let starts = (1..=most).map(start_of).collect::>(); - let running = run(client, starts.clone(), deadline()); + let running = run(client, vec![start_of(1)], deadline()); let refused = Workload { launch: Err(LaunchFailure::new(0, 11)), output: [Vec::new(), Vec::new()], status: Status::Exited(0), }; - let played = play(&mut broker, MAX_EXECUTIONS, OUTER_MAX_PAYLOAD, |_| { - refused.clone() - }); - assert_eq!(played, starts); + let played = play(&mut broker, 1, OUTER_MAX_PAYLOAD, |_| refused.clone()); + assert_eq!(played, [start_of(1)]); let (mut client, executions) = joined(running); let mut executions = executions.unwrap(); assert!(executions.is_finished()); - executions - .start(&mut client, start_of(most + 1), timers(), deadline()) - .unwrap(); + let refused = executions.start(&mut client, start_of(2), timers(), deadline()); + assert!(matches!(refused, Err(ExecutionsError::Ran)), "{refused:?}"); } #[test] From 23f7fe9c02e58bc1f22b16e1bb1f6fff0d0295b7 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 03:59:27 -0700 Subject: [PATCH 5/7] tests: bound what the runtime adapter keeps of an execution's output The host's side of an execution took all of a stream's output into memory, so a program that writes without end grew the host program's memory without bound until its deadline. HostExecution now keeps at most exec::MAX_CAPTURE, 16 MiB, of each stream. Once more of a stream arrives, it cancels the execution, and its peer discards the rest when the grace ends, so the outcome marks the stream truncated, with the cause cancelled. The VM scenario whose program leaves a descendant in a session of its own printed `started` as soon as it forked setsid, so the program could end before the descendant called setsid(2) or held its output. The program now waits until the descendant has printed its process ID and moved its output to the program's, and checks in /proc that it leads its session, before it prints `started`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 21 ++++-- .../src/bin/runtime-adapter-host.rs | 31 ++++----- tests/runtime-adapter/src/exec.rs | 26 ++++++- tests/runtime-adapter/src/exec/tests.rs | 67 ++++++++++++++++++- tests/runtime-adapter/tests/vm_layered.rs | 12 +++- 5 files changed, 127 insertions(+), 30 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 9cac3a1be..39be7269d 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -316,8 +316,12 @@ for more, and a source that hands over the program's input, if the start says that the host sends it, then ends it. Its caller sends the start, hands it each of the guest's messages about the execution with `deliver`, and sends the messages that `serve` returns, until the peer finished, serving it again once -its `deadline`, the peer's next timer, passes. The host program's -[driver](#host-program) and the tests of the worker run it. +its `deadline`, the peer's next timer, passes. The sink keeps at most +`exec::MAX_CAPTURE`, 16 MiB, of each stream: once more of a stream arrives, the +host's side cancels the execution, and its peer discards the rest once the grace +ends, so that the outcome marks the stream truncated, with the cause +`cancelled`. The host program's [driver](#host-program) and the tests of the +worker run it. ### Root-profile entries @@ -1074,7 +1078,9 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from `--numeric` says that the IDs have none. It starts in the root's directory DIR, or `/`, with exactly the environment of its `--env` entries. It runs for at most a minute, with 16 KiB chunks and the credit of 4 of them for its - input and 8 for its output, which its standard output and error share. + input and 8 for its output, which its standard output and error share. The + command keeps at most 16 MiB of each stream, and cancels an execution whose + stream would exceed them. Execution N, from 1, has the key `(B + N, E)`: E is the epoch of the command's attachment, and B the first 8 bytes of the broker instance's ID, as a little-endian number. The guest refuses a key that it heard of before, and @@ -1140,10 +1146,11 @@ before their body arrives, and the root configurations, readinesses, parts of mapping tables, failures, probes, facts, and messages of executions that decode, and those that don't, and the host's side of an execution, which still sends the cancellation of one that it gives up on in the step that ends it, without grace -or settlement time. In the guest-only configuration, the unit tests of -`worker::probes` cover what the probes read: the mount table with its escapes, -the kernel's records of a process, `os-release`, and the guest's memory, and the -probes and arguments that the worker refuses. Those of +or settlement time, and which cancels one whose stream outgrows the 16 MiB that +it keeps. In the guest-only configuration, the unit tests of `worker::probes` +cover what the probes read: the mount table with its escapes, the kernel's +records of a process, `os-release`, and the guest's memory, and the probes and +arguments that the worker refuses. Those of `worker::executions`, which spawn each program without entering a root, as a fake root does, hand an execution the host's cancellation once its timeout's instant passed, after its program ended before it, as a worker that diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index db1fb629d..7136b2001 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -73,24 +73,25 @@ //! text of a probe: its name, then its arguments, each after one space; see //! `runtime_adapter::probe` and `runtime_adapter::worker::probes`. //! - `runtime-adapter-host exec STORE EXECUTION [; EXECUTION]...` reattaches and attaches as -//! `reattach` does, then runs each EXECUTION in the VM's layered root, all at once, over that -//! one session, through `runtime_adapter::host::Executions`, and prints their outputs and -//! outcomes once every one finished. It runs at most 8 executions, as many as the guest runs at -//! once; see `runtime_adapter::exec::MAX_EXECUTIONS`. An EXECUTION, `[--identity UID:GID [--groups GID,...] +//! `reattach` does, then runs each EXECUTION in the VM's layered root, all at once, over that one +//! session, through `runtime_adapter::host::Executions`, and prints their outputs and outcomes +//! once every one finished. It runs at most 8 executions, as many as the guest runs at once; see +//! `runtime_adapter::exec::MAX_EXECUTIONS`. An EXECUTION, `[--identity UID:GID [--groups GID,...] //! [--capabilities N,...] [--numeric]] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...`, //! runs PROGRAM with its ARGs, up to a lone `;`, as `find -exec` takes them, without standard //! input. Without `--identity`, it names no identity, and runs as the sandbox's workload -//! identity, which the guest otherwise refuses; with it, it runs as the user ID UID and the -//! group ID GID, with the supplementary groups GIDs and the capabilities numbered N, such as 0 -//! for `CAP_CHOWN`, if given, and the guest checks its account, unless `--numeric` says that -//! the IDs have none. It starts in the root's directory DIR, or `/`, with exactly the -//! environment of its `--env` entries. It runs for at most a minute, with 16 KiB chunks and the -//! credit of 4 of them for its input and 8 for its output, which its standard output and error -//! share. Execution N, from 1, has the key `(B + N, E)`: E is the epoch of the command's -//! attachment, and B the first 8 bytes of the broker instance's ID, as a little-endian number. -//! The guest refuses a key that it heard of before, and the epochs of a broker instance only -//! advance, while a broker restored from a snapshot has a new instance ID, so no command -//! repeats the key of another's execution. +//! identity, which the guest otherwise refuses; with it, it runs as the user ID UID and the group +//! ID GID, with the supplementary groups GIDs and the capabilities numbered N, such as 0 for +//! `CAP_CHOWN`, if given, and the guest checks its account, unless `--numeric` says that the IDs +//! have none. It starts in the root's directory DIR, or `/`, with exactly the environment of its +//! `--env` entries. It runs for at most a minute, with 16 KiB chunks and the credit of 4 of them +//! for its input and 8 for its output, which its standard output and error share. The command +//! keeps at most 16 MiB of each stream, and cancels an execution whose stream would exceed them; +//! see `runtime_adapter::exec::MAX_CAPTURE`. Execution N, from 1, has the key `(B + N, E)`: E is +//! the epoch of the command's attachment, and B the first 8 bytes of the broker instance's ID, as +//! a little-endian number. The guest refuses a key that it heard of before, and the epochs of a +//! broker instance only advance, while a broker restored from a snapshot has a new instance ID, +//! so no command repeats the key of another's execution. //! //! Each command but `stage` takes NVX's artifacts from the directory that `NVX_ARTIFACTS_DIR` //! names, a repository checkout or an extracted release, and the VM's runtime identity from NVX's diff --git a/tests/runtime-adapter/src/exec.rs b/tests/runtime-adapter/src/exec.rs index 088153aa3..badf98a95 100644 --- a/tests/runtime-adapter/src/exec.rs +++ b/tests/runtime-adapter/src/exec.rs @@ -77,6 +77,10 @@ pub const MAX_CHUNK: u32 = (MAX_BODY - KEY_LEN - 1) as u32; /// starts none while those that it started have not finished. pub const MAX_EXECUTIONS: usize = 8; +/// The most bytes of each output stream of an execution that a [`HostExecution`] keeps: it +/// cancels an execution whose stream would exceed them, so that its host holds no more. +pub const MAX_CAPTURE: usize = 16 * 1024 * 1024; + /// The code of a cause that the framing does not know, which no side decodes. const UNKNOWN_CAUSE: u8 = 0xff; @@ -585,6 +589,10 @@ impl<'a> GuestMessage<'a> { /// it arrives, unless its caller stalls it, and a source that hands over the program's input, if /// the start says that the host sends it, then ends it. /// +/// The sink keeps at most [`MAX_CAPTURE`] bytes of each stream. Once more of a stream arrives, the +/// host's side cancels the execution, and its peer discards the rest once the grace ends, so that +/// the outcome marks the stream truncated, with the cause [`Cause::Cancelled`]. +/// /// Its caller sends the execution's [`Start`], hands it each of the guest's messages about the /// execution with [`deliver`](Self::deliver), and sends the frames that [`serve`](Self::serve) /// returns, until the peer [finished](Self::outcome). The host program's driver, @@ -607,6 +615,8 @@ pub struct HostExecution { ended: bool, /// The peer's events, in order. events: Vec, + /// Whether a stream outgrew [`MAX_CAPTURE`], so that the host's side cancelled the execution. + overflowed: bool, } impl HostExecution { @@ -623,6 +633,7 @@ impl HostExecution { sent: 0, ended: false, events: Vec::new(), + overflowed: false, } } @@ -648,7 +659,7 @@ impl HostExecution { self.stalled = stalled; } - /// Returns the bytes of `stream` that the sink took. + /// Returns the bytes of `stream` that the sink took, at most [`MAX_CAPTURE`]. pub fn output(&self, stream: Stream) -> &[u8] { &self.output[position(stream)] } @@ -695,13 +706,22 @@ impl HostExecution { } if !self.stalled { for stream in Stream::ALL { + let output = &mut self.output[position(stream)]; loop { let pending = self.peer.pending_output(stream); if pending.is_empty() { break; } - let length = pending.len(); - self.output[position(stream)].extend_from_slice(pending); + let room = MAX_CAPTURE - output.len(); + if room == 0 { + if !self.overflowed { + self.overflowed = true; + self.peer.cancel(now); + } + break; + } + let length = pending.len().min(room); + output.extend_from_slice(&pending[..length]); self.peer.consumed(stream, length); } } diff --git a/tests/runtime-adapter/src/exec/tests.rs b/tests/runtime-adapter/src/exec/tests.rs index 5f24c2d68..24555130d 100644 --- a/tests/runtime-adapter/src/exec/tests.rs +++ b/tests/runtime-adapter/src/exec/tests.rs @@ -1,7 +1,7 @@ //! The encodings of the execution messages: round trips, the layout of a start, the bodies that //! break the rules, and a peer and a runner that exchange their messages through the frames; and //! the host's side of an execution, which sends the cancellation of one that it gives up on at -//! once. +//! once, and cancels one whose stream outgrows what it keeps. use std::time::{Duration, Instant}; @@ -11,8 +11,8 @@ use nvx::exec::{ }; use super::{ - AccountMode, GuestMessage, HostExecution, HostMessage, Identity, KEY_LEN, MAX_CHUNK, Start, - name, + AccountMode, GuestMessage, HostExecution, HostMessage, Identity, KEY_LEN, MAX_CAPTURE, + MAX_CHUNK, Start, name, }; use crate::frame::{MAX_BODY, Message}; @@ -692,3 +692,64 @@ fn the_host_s_side_sends_the_cancellation_of_an_execution_that_it_gives_up_on_at assert!(execution.serve(later).is_empty(), "{cause:?}"); } } + +#[test] +fn the_host_s_side_keeps_at_most_max_capture_bytes_of_a_stream_and_cancels_the_rest() { + // A program that writes a chunk to its standard output at each step, without end: the host's + // side keeps the first MAX_CAPTURE bytes, then cancels the execution, and its peer, without + // grace, discards the rest at once. The runner takes and drops what the program writes once + // it discards the stream, so each step writes once. + let now = Instant::now(); + let timers = Timers::new(Duration::ZERO, Duration::from_secs(5)); + let start = Start::new(KEY, shared(), timers, ["/usr/bin/yes"]); + let mut host = HostExecution::new(start, timers, Vec::new(), now); + let mut runner = Runner::new(KEY, shared(), timers, now); + runner.launched(Ok(())); + let written = vec![b'y'; shared().chunk()]; + let mut buffer = vec![0u8; shared().chunk()]; + let mut running = true; + for _ in 0..2 * MAX_CAPTURE / written.len() { + if host.outcome().is_some() { + break; + } + for frame in host.serve(now) { + match HostMessage::decode(&frame).unwrap().unwrap() { + HostMessage::Credit(key, pool, units) => runner.credit(key, pool, units), + HostMessage::Cancel(key, cause) => runner.cancel(now, key, cause), + message => panic!("{message:?}"), + } + .unwrap(); + } + if running { + runner.output(Stream::Stdout, &written); + } + while let Some(request) = runner.next_request() { + match request { + runner::Request::Terminate(_) => { + running = false; + for stream in Stream::ALL { + runner.output_closed(stream); + } + runner.exited(now, Some(Status::Signaled(15))); + } + runner::Request::Settle => runner.settled(Settlement::Settled), + _ => {} + } + } + while let Some(message) = runner.next_message(&mut buffer) { + let frame = GuestMessage::from_runner(KEY, message, &buffer) + .unwrap() + .encode(); + let decoded = GuestMessage::decode(&frame).unwrap().unwrap(); + host.deliver(&decoded, now).unwrap(); + } + } + let outcome = host.outcome().expect("the execution finished"); + let kept = host.output(Stream::Stdout); + assert_eq!(kept.len(), MAX_CAPTURE); + assert!(kept.iter().all(|byte| *byte == b'y')); + assert!(host.output(Stream::Stderr).is_empty()); + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert!(outcome.is_truncated(Stream::Stdout)); + assert!(!outcome.is_truncated(Stream::Stderr)); +} diff --git a/tests/runtime-adapter/tests/vm_layered.rs b/tests/runtime-adapter/tests/vm_layered.rs index cfe3723ca..896ff65c8 100644 --- a/tests/runtime-adapter/tests/vm_layered.rs +++ b/tests/runtime-adapter/tests/vm_layered.rs @@ -141,6 +141,14 @@ const EPERM: i32 = 1; const EINVAL: i32 = 22; /// The number of executions that meet only while all of them run. const MEETING: usize = 4; +/// A script that starts a descendant that leads a session of its own and holds the script's +/// output: the descendant prints its process ID, which is its session's ID, then moves its output +/// to the script's, which ends the command substitution that reads the ID. The script prints +/// `started` once `/proc` shows that the descendant leads its session. +const DESCENDANT_SCRIPT: &str = "exec 3>&1; \ + descendant=$(setsid sh -c 'echo $$; exec sleep 600 >&3 3>&-' &); \ + read -r pid comm state parent group session rest Date: Sat, 24 Oct 2026 10:01:55 -0700 Subject: [PATCH 6/7] tests: name the default hostname that the runtime adapter's executions take The executions take nvx::guest::entry's default hostname, which no configuration changes, rather than a hostname that the kernel command line configures. The VM scenario that checks it now describes it so. Its check, nvx-sandbox, is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 2 +- tests/runtime-adapter/tests/vm_layered.rs | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 39be7269d..b6e14a233 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -1517,7 +1517,7 @@ stages its images, then launches them, and is a row of the table that its | `unstaged_images_are_refused_before_spawn` | The image over the scratch, which the scenario does not stage | A launch that fails before it spawns the VM, since the store stages no image at its path, and that leaves no launch journal, VMM log, runtime record, or launch record; then, once the scenario stages the images, the root that the worker assembles, and a clean teardown | | `a_changed_image_is_refused_before_spawn` | The GPT disk over the scratch, which the scenario writes in place after it stages it | A launch that fails before it spawns the VM, since the disk's modification time differs from its binding's, and that leaves nothing of a launch in the store; then, once the scenario stages the disk again, the root, and a clean teardown | | `a_replaced_scratch_is_refused_before_spawn` | The image over the scratch, which another copy of the template replaces after the scenario stages it | A launch that fails before it spawns the VM, since the scratch image's identity differs from its binding's, and that leaves nothing of a launch in the store; then, once the scenario stages the images again, the root, and a clean teardown | -| `executions_run_in_the_root` | The image over the scratch, without a workload identity, in which `exec` runs executions that name their identities | Over one session, four executions as `nobody` that meet only while all of them run, beside one as root that writes more than a MiB to its standard output and a little to its standard error, each byte-exact, ending on its own with 0, and reusable. Over a later session, with new keys: a program that leaves a descendant in a session of its own, which the program's end, as PID 1 of its namespace, ends, so that no output is lost; `nobody`'s IDs and groups, with empty permitted, effective, and bounding sets; root with `CAP_CHOWN` alone in each of them; exactly the start's environment; the start's working directory; the configured hostname, `nvx-sandbox`, and PID 1; and a start without an identity, and one with `CAP_SYS_ADMIN`, which the guest refuses at stage 0 with `EINVAL` and `EPERM`, fenced and without output | +| `executions_run_in_the_root` | The image over the scratch, without a workload identity, in which `exec` runs executions that name their identities | Over one session, four executions as `nobody` that meet only while all of them run, beside one as root that writes more than a MiB to its standard output and a little to its standard error, each byte-exact, ending on its own with 0, and reusable. Over a later session, with new keys: a program that leaves a descendant in a session of its own, which the program's end, as PID 1 of its namespace, ends, so that no output is lost; `nobody`'s IDs and groups, with empty permitted, effective, and bounding sets; root with `CAP_CHOWN` alone in each of them; exactly the start's environment; the start's working directory; the entry's default hostname, `nvx-sandbox`, and PID 1; and a start without an identity, and one with `CAP_SYS_ADMIN`, which the guest refuses at stage 0 with `EINVAL` and `EPERM`, fenced and without output | | `executions_run_as_the_workload_identity` | The image over the scratch, with the workload identity of `nobody` | Executions that name no identity, which run as `nobody`, without capabilities, with exactly the start's environment, and a start that names an identity, which the guest refuses at stage 0 with `EINVAL` | | `accountless_workload_identity_fails_its_stage` | The image over the scratch, with a workload identity whose account the image lacks | The identity stage's failure once the guest assembled the root, which the launch reports, and a VM that ends with 16, which unmounted the scratch | diff --git a/tests/runtime-adapter/tests/vm_layered.rs b/tests/runtime-adapter/tests/vm_layered.rs index 896ff65c8..93908e3aa 100644 --- a/tests/runtime-adapter/tests/vm_layered.rs +++ b/tests/runtime-adapter/tests/vm_layered.rs @@ -813,7 +813,7 @@ scenarios! { /// executions that meet only while all of them run; a descendant in a session of its own, /// which holds its program's output, and which the end of its program ends; a user other than /// root, and root with an allowed capability alone; a working directory, an exact - /// environment, and the configured hostname, each as PID 1 of its namespaces. The guest + /// environment, and the default hostname, each as PID 1 of its namespaces. The guest /// refuses a start without an identity, and one with a capability outside the entry's /// allowlist, at stage 0. executions_run_in_the_root: Scenario { @@ -2302,7 +2302,7 @@ fn check_executions(vm: &mut Vm<'_>) { // 3. root with `CAP_CHOWN` alone, in its permitted, effective, and bounding sets; // 4. exactly the environment that the start names; // 5. the working directory that the start names; - // 6. the configured hostname, as PID 1 of the execution's namespaces; + // 6. the entry's default hostname, `nvx-sandbox`, as PID 1 of the execution's namespaces; // 7. no identity, in a sandbox without a workload identity, which the guest refuses; // 8. a capability outside the entry's allowlist, `CAP_SYS_ADMIN`, which the entry refuses. let executions = [ From b4d666a461611e6997b8d2da09da30d62b8b9355 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 26 Oct 2026 16:36:41 -0700 Subject: [PATCH 7/7] tests: say that the host program's limits of `exec` are chosen for tests The host program gives the executions of `exec` timers and limits whose values are chosen for tests, as a valid configuration within the limits that the guest and the host allow. A comment now says so, and names those limits: `exec::MAX_CHUNK`, the most bytes of a chunk; `worker::MAX_EXECUTION_MEMORY`, the most bytes that an execution holds in the guest; and `host::MAX_OUTPUT_FRAMES`, the most frames of an output window. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/src/bin/runtime-adapter-host.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 7136b2001..e580b4ecf 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -201,6 +201,10 @@ const ROOT_FAILED: u8 = 4; /// The exit status of a launch whose guest failed a mapping of the table, once the VM stopped /// gracefully, with success and a complete teardown. const TABLE_FAILED: u8 = 6; +// The timers and limits of `exec` below are values chosen for tests that make a valid +// configuration, within the limits that the guest and the host allow: chunks of at most +// `exec::MAX_CHUNK` bytes, at most `worker::MAX_EXECUTION_MEMORY` bytes held in the guest for +// each execution, and output windows of at most `host::MAX_OUTPUT_FRAMES` frames. /// How long an execution of `exec` may run before its sides end it. const EXEC_TIMEOUT: Duration = Duration::from_secs(60); /// How long the pending output of an execution of `exec` still flows once a side ended it.