diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 1fd5644c8..03f84aedd 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -108,13 +108,19 @@ zeros: | `overlay-mode` | The overlay's mode: `trusted-lowers` or `untrusted-lowers` | no | | `scratch-journal` | Whether an ext4 scratch may have a journal: `allowed` or `forbidden` | no | | `mapping-count` | The number of mappings of the host's [mapping table](#ready-mapping-tables-and-start), at most 4,096 | no | +| `workload-uid` | The user ID of the sandbox's workload identity, as which every execution runs | with `workload-gid` | +| `workload-gid` | The group ID of the sandbox's workload identity | with `workload-uid` | The paths are absolute and canonical guest paths of at most 4,095 bytes, without a NUL or a newline: not `/`, and without an empty, `.`, or `..` component, or a trailing `/`. The runtime directory holds no `,`, `:`, `\`, or `=`, which the overlay's options reserve, and the two cgroups are siblings, -whose parent is not `/`. The optional numbers are positive. Without an -overlay mode or a journal policy, the guest takes the defaults of +whose parent is not `/`. The optional numbers are positive. A workload +identity has both IDs, each below 4,294,967,295, which the kernel reads as +"unchanged", and no supplementary groups or capabilities, as a layered +sandbox's identity does; the worker checks its account once it assembled the +root, and every execution then runs as it, naming no identity of its own. +Without an overlay mode or a journal policy, the guest takes the defaults of `nvx::guest::root`: an overlay that trusts its lowers, and a scratch that may have a journal. Without a bound of the first partition, the guest bounds it at `root::DEFAULT_FIRST_PARTITION_BOUND`, 64 KiB. Without a mapping count, the @@ -126,7 +132,8 @@ caller names others: the runtime directory `/run/runtime-adapter`, the cgroups `/sys/fs/cgroup/agent` and `/sys/fs/cgroup/workload`, a wait of 30 seconds, an agent's `memory.low` of 16 MiB, as NVX's own guest reserves, and a workload's `memory.max` of 128 MiB and `pids.max` of 128, with neither a bound of the -first partition, an overlay mode, nor a journal policy, and no mapping. +first partition, an overlay mode, a journal policy, nor a workload identity, and no +mapping. A stage of the root that fails is a `root::RootFailure`: the stage's code, one byte, then the problem, as UTF-8 text. The guest's worker, and with it the VM, @@ -139,6 +146,7 @@ then ends with the stage's status: | 3 | `bootstrap` | Bootstrapping the guest for the root: its cgroups and their resource values | 13 | | 4 | `assembly` | Assembling the root, such as from an image of another identity than the boot tokens expect, and mounting the host export that the kernel command line names | 14 | | 5 | `teardown` | Tearing the root down, such as while a process holds it open | 15 | +| 6 | `identity` | Checking the account of the sandbox's workload identity in the assembled root, in existing-account mode, such as one that the image lacks | 16 | ### Ready, mapping tables, and Start @@ -272,11 +280,26 @@ the key and a stream; the input's window, then the output's, each a unit, 0 for bytes or 1 for frames, then its size, a `u32`; and whether the streams share one output pool, 0 or 1. It then holds the worker's timers, in milliseconds, each a `u64`: the grace, the settlement time, and whether a -timeout follows, 0 or 1, then the timeout, or 0. Last come whether the host +timeout follows, 0 or 1, then the timeout, or 0. Then come whether the host sends the program's standard input, 0 or 1, and the program's arguments: their count, a `u16`, then each argument's length, a `u32`, and its bytes. The first argument is the program's path, which is not searched for in `PATH`, and no -argument holds a NUL byte. +argument holds a NUL byte. The start ends with how the program enters the +worker's root: + +- whether it names the program's identity, `exec::Identity`, 0 or 1, and if + it does, the user ID and the group ID, each a `u32`, the account mode, + `exec::AccountMode`, 0 for an existing account or 1 for numeric IDs, the + capabilities, a `u64` whose bit N stands for the capability numbered N, then + the supplementary groups: their count, a `u16`, then each group, a `u32`. + Exactly one source names an execution's identity: its start, or the + sandbox's workload identity, which the root configuration names; +- whether it names the program's working directory, 0 or 1, and if it does, + the directory's length, a `u32`, and its bytes: an absolute path in the + root, without a NUL byte. Otherwise the program starts in the root's `/`; +- the program's whole environment: the count of its entries, a `u16`, then + each entry's length, a `u32`, and its bytes, `NAME=VALUE`, with a name that + is not empty, and without a NUL byte. An outcome holds the program's status: 0 for none, 1 for an exit, or 2 for a signal, then the exit's code or the signal's number, an `i32`, or 0; the cause, @@ -410,7 +433,9 @@ then: then bootstraps the guest's cgroups with `nvx::guest::root::bootstrap`, which moves the guest's processes into the agent's cgroup, assembles the root with `nvx::guest::root::assemble`, and mounts the export out of the - root with `Root::mount_export`; + root with `Root::mount_export`. If the configuration names a workload + identity, it then checks the identity's account in the assembled root, in + existing-account mode, with `nvx::guest::entry`; 4. answers `Ready`, with the guest agent's build, or `RootFailed` with the stage that failed, and then ends; 5. applies the host's [mapping table](#the-mapping-table-and-the-start) @@ -430,22 +455,23 @@ Before the root configuration, the worker accepts only the configuration, or a layers of a GPT disk, whose first partition holds at most the bound of the configuration, over an ext4 scratch or a tmpfs in RAM, in the overlay mode of the configuration. At the layout stage, it refuses a configuration that -configures a scratch for an upper in RAM, bounds a first partition for an -EROFS image, or announces mappings while the kernel command line names no -host export, from which they come, and a command line whose tokens of the -export break their rules, as `HostExport::parse` checks them. An export that -fails to mount fails the assembly. It ends with 0 once the host stopped it, -after a teardown that succeeded, with the stage's status, from 11 to 15, once -a stage failed, and with 1 once its session failed or the broker closed the -tty, each but the first after a diagnostic on standard error. Once the broker -closed the stream at a record boundary, which may close only the broker's end, -the worker drops its executions, and writes what it queued, as once the host -stopped it. Before it exits, it closes what its probes hold and the host -export's root, then unmounts a root that it still holds, as -`nvx::guest::root::Root` does when it drops, ignoring failures. A teardown -that failed leaves the root mounted until the VM ends. The worker runs the -root's operations through `worker::RootOperations`, so that tests run it -against a fake root. +configures a scratch for an upper in RAM, bounds a first partition for an EROFS +image, or announces mappings while the kernel command line names no host export, +from which they come, and a command line whose tokens of the export break their +rules, as `HostExport::parse` checks them. An export that fails to mount fails +the assembly. A workload identity whose account the assembled root lacks fails +the identity stage. The worker ends with 0 once the host stopped it, after a +teardown that succeeded, with the stage's status, from 11 to 16, once a stage +failed, and with 1 once its session failed or the broker closed the tty, each +but the first after a diagnostic on standard error. Once the broker closed the +stream at a record boundary, which may close only the broker's end, the worker +drops its executions, and writes what it queued, as once the host stopped it. +Before it exits, it closes what its probes hold and the host export's root, then +unmounts a root that it still holds, as `nvx::guest::root::Root` does when it +drops, ignoring failures. A teardown that failed leaves the root mounted until +the VM ends. The worker runs the root's operations through +`worker::RootOperations`, and enters its executions through +`worker::EntryOperations`, so that tests run it against a fake root. #### The mapping table and the start @@ -522,28 +548,73 @@ credit that it needs, never keeps the others' output from the host. runs until it ends, detached from the host, which receives nothing more about it. - The worker admits at most `worker::MAX_EXECUTIONS`, 8, executions at once, - abandoned ones included, and those whose programs' statuses it has not - collected, and none that would hold more than + abandoned ones included, and those that keep their places once they ended, + as the items below describe, and none that would hold more than `worker::MAX_EXECUTION_MEMORY`, 1 MiB: its runner's buffers, a chunk of each - output stream and the input's window. It refuses a start beyond those, or - one whose program it cannot spawn, with a terminal record whose launch failed - at stage 0, with `EAGAIN`, `ENOMEM`, or the spawn's `errno`. -- Each program runs in the worker's own root and working directories, with its - environment, not in the layered root, with its standard input from the host - or the null device, and leads a process group of its own, which its - execution kills as a whole. -- The worker collects the status of every program itself, without waiting for - it, and learns that a program ended from a pidfd of its own. Before it - collects the status, it kills what is left of the program's process group, - as NVX's settlement through a cgroup kills what is left in the cgroup: until - then, the program, a zombie, keeps its process ID, which is the group's, from - any other process. So a descendant that stayed in the group neither outlives - the program nor holds its output open. The program runs without a cgroup, - though, so nothing proves that a descendant that left the group is gone, and - each outcome is fenced: once the program ended, a stream that such a process - may still hold has the settlement time and the grace to close, after which - the runner gives up on it, while a stream that no process holds any more - loses nothing, however slowly the host takes it. + output stream and the input's window. It refuses a start beyond those, one + whose identity or entry it refuses, as the next item describes, or one whose + cgroup it cannot prepare, or whose program it cannot spawn, with a terminal + record whose launch failed at stage 0, with `EAGAIN`, `ENOMEM`, the `errno` + of the refusal, or that of the preparation or the spawn. +- Each program enters the layered root through `nvx::guest::entry`, as a + `worker::Entrance` describes it: in new mount, PID, and UTS namespaces, with + the root moved over `/`, the entry's default hostname, `nvx-sandbox`, the + image's own machine ID, and no time status, so that the entry binds nothing + into the root, and with published listeners, the entry's default, until the + adapter tells whether its listeners are published. It runs as the identity + that exactly one source names: the sandbox's workload identity, with no + supplementary groups or capabilities, or, in a sandbox without one, the + identity that its start names, with its supplementary groups, its + capabilities, and its account mode. A start that names an identity in a + sandbox with a workload identity, or none in a sandbox without one, is refused + with `EINVAL`. Before the worker prepares the execution's cgroup, the entry + checks the identity, and reads its account in existing-account mode: it + refuses with `EPERM` a capability outside its allowlist, any capability of a + user ID other than 0, and `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while + listeners are published, with `ENOENT` an identity without the user ID's + passwd entry or the group ID's group entry, with the error of an account + database or a root that it cannot read, and with `EINVAL` otherwise. The + program starts in the working directory of its start, or in the root's `/`, + with exactly the environment of its start, none of the worker's own, and with + its standard input from the host or the null device. It leads a session of its + own, without a controlling terminal, and the process group that the session + creates, and, as PID 1 of its PID namespace, its end ends every process of the + namespace. +- Each execution runs in a cgroup of its own below the workload cgroup of the + root configuration, `execution-N`, where N counts the cgroups that the worker + prepared. The worker prepares it with + `nvx::guest::cgroup::ExecutionCgroup::prepare`, and the program joins it as + it spawns, before it runs, so that its descendants start in it, however they + leave its process group or session. The execution settles through that + cgroup alone, which leaves the other executions running: once the program + ended, as its status, which the execution collects itself, tells, the + execution kills what is left in the cgroup, and settles once the cgroup is + empty. So no descendant outlives the program or holds its output open, and + an execution whose program ends on its own loses nothing, however slowly the + host takes its output, and is reusable. +- An execution keeps its place among `worker::MAX_EXECUTIONS` until its cgroup + is empty and the worker removed it, which the worker does once the runner + returned the terminal outcome of an execution that settled. One that did not + settle, whose outcome is fenced, keeps its place and its cgroup until the + host stops the worker, as does a start whose program the worker spawned, but + whose execution it could not have: until the worker collected the status of + such a program, the program may still join the cgroup, so that a removal + could hide it once it ended. +- Without cgroups, as where the in-process tests cannot create them, the + root's operations name none, and the program's process group stands in for + its cgroup: its execution kills the group as a whole. The worker collects the + status of every program itself then, without waiting for it, and learns that + a program ended from a pidfd of its own. Before it collects the status, it + kills what is left of the program's process group, as the settlement through + a cgroup kills what is left in the cgroup: until then, the program, a + zombie, keeps its process ID, which is the group's, from any other process. + So a descendant that stayed in the group neither outlives the program nor + holds its output open. Nothing proves that a descendant that left the group + is gone, though, so each outcome is fenced: once the program ended, a stream + that such a process may still hold has the settlement time and the grace to + close, after which the runner gives up on it, while a stream that no process + holds any more loses nothing, however slowly the host takes it. Such an + execution keeps its place until the worker collected its program's status. - Before it hands an execution the host's cancellation, the worker gives the execution a turn, as the loop does, which collects the status of a program that ended, and reports the program's end to the runner. So the end of a @@ -554,11 +625,14 @@ credit that it needs, never keeps the others' output from the host. - The worker forgets an execution once its runner returned the terminal outcome, which may come before the program's end, as once the runner gave up on a program that a kill did not end in time: forgetting it kills the - program and its process group again, and the worker collects the program's - status once it ended. On a stop, or once its session ended otherwise, the - worker drops every execution, which kills the process group of each program - whose status it has not collected, and waits up to a second in all for those - programs to end. + program, and what is left in its cgroup or its process group, again, and the + worker collects the program's status once it ended. On a stop, or once its + session ended otherwise, the worker drops every execution, which kills the + program of each one that has not ended, with what is left in its cgroup or + its process group, and what is left in the cgroups of the executions that did + not settle, and waits up to a second in all for those programs to end and + those cgroups to empty, removing each cgroup that emptied once it collected + the statuses of those programs. The worker's `entries` probe runs two more commands of the guest agent: `runtime-adapter-guest entries ROOT CGROUP`, its helper, which runs the probe's @@ -984,7 +1058,8 @@ that decode, and those that don't. In the guest-only configuration, the unit tests of `worker::probes` cover what the probes read: the mount table with its escapes, the kernel's records of a process, `os-release`, and the guest's memory, and the probes and arguments that the worker refuses. Those of -`worker::executions` hand an execution the host's cancellation once its +`worker::executions`, which spawn each program without entering a root, as a +fake root does, hand an execution the host's cancellation once its timeout's instant passed, after its program ended before it, as a worker that did not run meanwhile would, and check that the program's end stays its own. Those of `worker` run PID 1's collection alone, in a run of the test binary of @@ -1062,16 +1137,18 @@ byte value, and that `init` fails at once on a path that is no tty. test broker: a root that the worker assembles, probes, and tears down on a stop, across a reset of the session; an image over RAM, in the overlay mode of the configuration; a GPT disk, with the configuration's bound of the first -partition; the host export that it mounts, the mapping table that it applies -in order, in parts, across a reset of the session, and the start; a table of -zero mappings, which does no work, with or without an export; the parts that -do not continue the table, which it refuses as a whole, and a mapping that -the root refuses, which latches the root, after either of which it refuses -the later parts and the start; the executions and starts that it refuses out -of order, before the start and after it; the failure of each stage, the tokens -of a host export that break their rules, mappings without an export, and an -export that fails to mount among them, which the worker reports before it ends -with the stage's status, without the operations of the later stages; a teardown +partition; the host export that it mounts, the mapping table that it applies in +order, in parts, across a reset of the session, and the start; a table of zero +mappings, which does no work, with or without an export; the parts that do not +continue the table, which it refuses as a whole, and a mapping that the root +refuses, which latches the root, after either of which it refuses the later +parts and the start; the executions and starts that it refuses out of order, +before the start and after it; the failure of each stage, the tokens of a host +export that break their rules, mappings without an export, an export that fails +to mount, and a workload identity whose account the root lacks among them, which +the worker reports before it ends with the stage's status, without the +operations of the later stages; the account check of a sandbox's workload +identity, once the root is assembled, with the default hostname; a teardown that fails; before the root, the configuration and the `Stop` that it accepts, and the messages that it refuses; a second configuration and a broker that closes the stream, which leave the root that it assembled to its operations; @@ -1094,8 +1171,19 @@ the binary exited, so that none of them outlives the test or holds its output open, which tests check too. `tests/guest_session.rs` also runs the worker's executions behind its test -broker, whose host side starts the sandbox, then drives the engine's peers, -with frame-counted credit and one output pool: +broker, whose host side starts the sandbox, then drives the engine's peers, with +frame-counted credit and one output pool. The fake root prepares each +execution's entry, as the worker describes it, in a directory of the tests' own +whose account databases hold the accounts that the tests name, and then spawns +the program without entering a root, which needs privileges, in the tests' own +root and identity. Unless a test says otherwise, the sandbox has a workload +identity, as which every execution runs. Where the tests can create cgroups, as +root in a cgroup v2 hierarchy with the `pids` controller, the fake root gives +the worker a workload cgroup of the test's own, below which each execution runs +in a cgroup of its own, so that an execution whose program ends on its own +settles, and is reusable. Elsewhere, the fake root names no cgroups, and each +program runs in a process group alone, so that such an execution is fenced. The +tests cover: - three executions at once, each byte-exact: standard output beyond a MiB, 1.5 MiB of input through `cat`, and both streams of a program that fails; @@ -1125,10 +1213,35 @@ with frame-counted credit and one output pool: statuses, and a program whose status the worker collects once it ended, after its execution's runner gave up on it; - programs that end while a descendant holds their output: the worker ends - the descendant, so that the output ends at once, losing nothing, and a stop - once the worker collected such a program's status leaves no process of its - group running; + the descendant, through the execution's cgroup or the program's process + group, so that the output ends at once, losing nothing, and a stop once the + worker collected such a program's status leaves no process of its group + running; +- with cgroups, a descendant in a session of its own, which holds a program's + output once the program ended, and which the execution's settlement ends, so + that the output ends at once, losing nothing, and the execution settles; + the worker removed the execution's cgroup before it sent the outcome, and + runs its limit of executions at once again; +- with cgroups, an execution that does not settle within its settlement time, + because a zombie whose parent is outside its cgroup occupies the cgroup: it + keeps its place, so that the worker runs one execution fewer than its limit, + even once the cgroup emptied, until the host stops the worker, which then + removes the cgroup; - a program that a real-time signal ends, whose status the worker collects; +- a program that runs in the working directory of its start, with exactly the + environment of its start; +- the identities that starts name in a sandbox without a workload identity, + with their supplementary groups, capabilities, and account modes, and the + default hostname, all of which the worker's entry takes; +- the starts that the worker refuses at stage 0 for their identities: one that + names an identity in a sandbox with a workload identity, or none in a + sandbox without one, with `EINVAL`; a capability outside the entry's + allowlist, one of a user ID other than 0, and `CAP_NET_BIND_SERVICE` and + `CAP_NET_RAW` while listeners are published, with `EPERM`; an identity + without a passwd entry, or whose primary group has no group entry, with + `ENOENT`; and one whose passwd entry has another primary group, or whose + user ID the kernel reads as "unchanged", with `EINVAL`. None of them takes a + place, so the worker then runs its limit of executions at once; - an execution whose start comes in the payload of the root's configuration and the sandbox's start, whose timeout counts from its own start, however long the root's assembly takes; diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs b/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs index da8206ffd..761c4c460 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs @@ -18,7 +18,7 @@ //! - `runtime-adapter-guest worker TTY [SIBLING]` runs the worker on the tty TTY: it assembles the //! layered root that the host configures, and serves the host program until the host stops it. //! SIBLING is the process ID of PID 1's unrelated child, which its probes check. It exits with 0 -//! once the host stopped it, with 11 to 15 once a stage of the root failed, and with 1 once its +//! once the host stopped it, with 11 to 16 once a stage of the root failed, and with 1 once its //! session failed or the broker closed the tty, each but the first after a diagnostic on //! standard error. //! - `runtime-adapter-guest idle` sleeps until a signal ends it. diff --git a/tests/runtime-adapter/src/exec.rs b/tests/runtime-adapter/src/exec.rs index f17d97d85..083061453 100644 --- a/tests/runtime-adapter/src/exec.rs +++ b/tests/runtime-adapter/src/exec.rs @@ -33,6 +33,19 @@ //! each argument's length, a `u32`, and its bytes. The first argument is the program's path, and //! no argument holds a NUL byte. //! +//! The start ends with how the program enters the guest's root: +//! +//! - whether it names the program's [`Identity`], 0 or 1, and if it does, the user ID and the +//! group ID, each a `u32`, the [`AccountMode`], 0 for an existing account or 1 for numeric IDs, +//! the capabilities, a `u64` whose bit N stands for the capability numbered N, then the +//! supplementary groups: their count, a `u16`, then each group, a `u32`; +//! - whether it names the program's working directory, 0 or 1, and if it does, the directory's +//! length, a `u32`, and its bytes: an absolute path without a NUL byte. Otherwise the program +//! starts in the root's `/`; +//! - the program's whole environment: the count of its entries, a `u16`, then each entry's +//! length, a `u32`, and its bytes, `NAME=VALUE`, with a name that is not empty, and without a +//! NUL byte. +//! //! An outcome, after the key, holds the program's status: 0 for none, 1 for an exit, or 2 for a //! signal, then the exit's code or the signal's number, an `i32`, or 0; the cause, or 0 for none; //! whether the launch failed, 0 or 1, then its stage, a `u32`, and its `errno`, an `i32`, or 0 for @@ -82,11 +95,20 @@ pub struct Start { pub input: bool, /// The program's path, which is also its first argument, then its other arguments. pub arguments: Vec>, + /// The identity as which the program runs, if the start names one. Exactly one source names + /// it: the start, or the workload identity of the guest's whole sandbox. + pub identity: Option, + /// The program's working directory, an absolute path in the guest's root, if the program + /// does not start in the root's `/`. + pub working_directory: Option>, + /// The program's whole environment, as `NAME=VALUE` entries. + pub environment: Vec>, } impl Start { /// Returns the start of the execution `key`, with `limits` and the guest's `timers`, of the - /// program that `arguments` names, without standard input. + /// program that `arguments` names, without standard input, without an identity, in the + /// root's `/`, and with an empty environment. pub fn new>>( key: Key, limits: Limits, @@ -99,6 +121,9 @@ impl Start { timers, input: false, arguments: arguments.into_iter().map(Into::into).collect(), + identity: None, + working_directory: None, + environment: Vec::new(), } } @@ -108,6 +133,27 @@ impl Start { self } + /// Names the identity as which the program runs. + pub fn with_identity(mut self, identity: Identity) -> Self { + self.identity = Some(identity); + self + } + + /// Names the program's working directory. + pub fn with_working_directory(mut self, directory: impl Into>) -> Self { + self.working_directory = Some(directory.into()); + self + } + + /// Gives the program the environment of the `NAME=VALUE` entries `environment`. + pub fn with_environment>>( + mut self, + environment: impl IntoIterator, + ) -> Self { + self.environment = environment.into_iter().map(Into::into).collect(); + self + } + /// Returns the start's encoding, the body of an `ExecStart`. pub fn encode(&self) -> Vec { let mut body = key_body(self.key); @@ -129,18 +175,44 @@ impl Start { put_u64(&mut body, timeout.map_or(0, milliseconds)); body.push(u8::from(self.input)); // A start with more arguments than `u16` counts exceeds the framing's limit too. - let count = u16::try_from(self.arguments.len()).unwrap_or(u16::MAX); - body.extend_from_slice(&count.to_le_bytes()); + put_count(&mut body, self.arguments.len()); for argument in &self.arguments { - put_u32(&mut body, u32::try_from(argument.len()).unwrap_or(u32::MAX)); - body.extend_from_slice(argument); + put_bytes(&mut body, argument); + } + match &self.identity { + None => body.push(0), + Some(identity) => { + body.push(1); + put_u32(&mut body, identity.uid); + put_u32(&mut body, identity.gid); + body.push(identity.account.code()); + put_u64(&mut body, identity.capabilities); + put_count(&mut body, identity.groups.len()); + for group in &identity.groups { + put_u32(&mut body, *group); + } + } + } + match &self.working_directory { + None => body.push(0), + Some(directory) => { + body.push(1); + put_bytes(&mut body, directory); + } + } + put_count(&mut body, self.environment.len()); + for entry in &self.environment { + put_bytes(&mut body, entry); } body } /// Decodes the body of an `ExecStart`, and checks it: its limits, of which none is empty and - /// whose chunk is at most [`MAX_CHUNK`] bytes, its flags, and its arguments, of which there is - /// at least one, the program's path, which is not empty, and none of which holds a NUL byte. + /// whose chunk is at most [`MAX_CHUNK`] bytes, its flags, its arguments, of which there is + /// at least one, the program's path, which is not empty, and none of which holds a NUL byte, + /// its identity's account mode, its working directory, which is absolute and holds no NUL + /// byte, and its environment, whose entries are `NAME=VALUE`, with a name that is not empty, + /// and hold no NUL byte. /// /// Fails with a [`BodyError`] that names the first problem. pub fn decode(body: &[u8]) -> Result { @@ -159,13 +231,64 @@ impl Start { let count = reader.u16()?; let mut arguments = Vec::with_capacity(usize::from(count).min(body.len())); for _ in 0..count { - let length = reader.u32()?; - let argument = reader.bytes(usize::try_from(length).unwrap_or(usize::MAX))?; + let argument = reader.counted()?; if argument.contains(&0) { return Err(refused("has an argument with a NUL byte".to_owned())); } arguments.push(argument.to_vec()); } + let identity = if flag(reader.u8()?).map_err(refused)? { + let uid = reader.u32()?; + let gid = reader.u32()?; + let account = AccountMode::from_code(reader.u8()?).map_err(refused)?; + let capabilities = reader.u64()?; + let count = reader.u16()?; + let mut groups = Vec::with_capacity(usize::from(count).min(body.len())); + for _ in 0..count { + groups.push(reader.u32()?); + } + Some(Identity { + uid, + gid, + groups, + capabilities, + account, + }) + } else { + None + }; + let working_directory = if flag(reader.u8()?).map_err(refused)? { + let directory = reader.counted()?; + if directory.first() != Some(&b'/') { + return Err(refused( + "names a working directory that is not absolute".to_owned(), + )); + } + if directory.contains(&0) { + return Err(refused( + "has a working directory with a NUL byte".to_owned(), + )); + } + Some(directory.to_vec()) + } else { + None + }; + let count = reader.u16()?; + let mut environment = Vec::with_capacity(usize::from(count).min(body.len())); + for _ in 0..count { + let entry = reader.counted()?; + if entry.contains(&0) { + return Err(refused( + "has an environment entry with a NUL byte".to_owned(), + )); + } + if entry.iter().position(|&byte| byte == b'=').unwrap_or(0) == 0 { + return Err(refused( + "has an environment entry that is not NAME=VALUE".to_owned(), + )); + } + environment.push(entry.to_vec()); + } reader.finish()?; if chunk > MAX_CHUNK { return Err(refused(format!( @@ -196,10 +319,74 @@ impl Start { timers, input: with_input, arguments, + identity, + working_directory, + environment, }) } } +/// The identity as which an execution's program runs, which a start names unless the guest's +/// sandbox has a workload identity: its IDs, its capabilities, and how the guest checks its +/// account, as the guest's entry into its root takes them. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Identity { + /// The user ID. + pub uid: u32, + /// The group ID. + pub gid: u32, + /// The supplementary groups. + pub groups: Vec, + /// The capabilities, which only user ID 0 may hold: bit N for the capability numbered N. + pub capabilities: u64, + /// How the guest checks the identity's account. + pub account: AccountMode, +} + +impl Identity { + /// Returns the identity of user ID `uid` and group ID `gid`, without supplementary groups or + /// capabilities, whose account the guest checks in [existing](AccountMode::Existing) mode. + pub fn new(uid: u32, gid: u32) -> Self { + Self { + uid, + gid, + groups: Vec::new(), + capabilities: 0, + account: AccountMode::Existing, + } + } +} + +/// How the guest checks the account of an execution's [`Identity`]. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum AccountMode { + /// The root's `/etc/passwd` holds exactly one entry for the user ID, whose primary group is + /// the group ID, which the root's `/etc/group` holds. + #[default] + Existing, + /// The IDs are used as given, without an account. + Numeric, +} + +impl AccountMode { + /// Returns the mode's code in a start: 0 for an existing account, 1 for numeric IDs. + fn code(self) -> u8 { + match self { + Self::Existing => 0, + Self::Numeric => 1, + } + } + + /// Returns the mode that `code` encodes, or the problem. + fn from_code(code: u8) -> Result { + match code { + 0 => Ok(Self::Existing), + 1 => Ok(Self::Numeric), + _ => Err(format!("names the unknown account mode {code}")), + } + } +} + /// A message of the host's side of an execution, its [`Peer`], to the guest's. #[derive(Clone, Debug, PartialEq, Eq)] pub enum HostMessage<'a> { @@ -418,6 +605,12 @@ impl<'a> Reader<'a> { Ok(taken) } + /// Takes the next bytes, after their length, a `u32`. + fn counted(&mut self) -> Result<&'a [u8], BodyError> { + let length = self.u32()?; + self.bytes(usize::try_from(length).unwrap_or(usize::MAX)) + } + /// Takes the next `N` bytes. fn array(&mut self) -> Result<[u8; N], BodyError> { Ok(self.bytes(N)?.try_into().expect("N bytes")) @@ -755,5 +948,18 @@ fn put_u64(body: &mut Vec, value: u64) { body.extend_from_slice(&value.to_le_bytes()); } +/// Appends the count `count` of a start's items to `body`, as a `u16`. A start with more items +/// than a `u16` counts exceeds the framing's limit too. +fn put_count(body: &mut Vec, count: usize) { + let count = u16::try_from(count).unwrap_or(u16::MAX); + body.extend_from_slice(&count.to_le_bytes()); +} + +/// Appends `bytes` to `body`, after their length, a `u32`. +fn put_bytes(body: &mut Vec, bytes: &[u8]) { + put_u32(body, u32::try_from(bytes.len()).unwrap_or(u32::MAX)); + body.extend_from_slice(bytes); +} + #[cfg(test)] mod tests; diff --git a/tests/runtime-adapter/src/exec/tests.rs b/tests/runtime-adapter/src/exec/tests.rs index 2e3492aac..db112acfb 100644 --- a/tests/runtime-adapter/src/exec/tests.rs +++ b/tests/runtime-adapter/src/exec/tests.rs @@ -8,7 +8,7 @@ use nvx::exec::{ Timers, Window, peer, runner, }; -use super::{GuestMessage, HostMessage, KEY_LEN, MAX_CHUNK, Start, name}; +use super::{AccountMode, GuestMessage, HostMessage, Identity, KEY_LEN, MAX_CHUNK, Start, name}; use crate::frame::{MAX_BODY, Message}; /// The key of most executions in these tests. @@ -27,7 +27,18 @@ fn timers() -> Timers { Timers::new(Duration::from_secs(1), Duration::from_secs(5)) } -/// Returns starts with every unit, flag, and kind of argument. +/// Returns an identity with every field: supplementary groups, capabilities, and numeric IDs. +fn numeric() -> Identity { + Identity { + groups: vec![0, 50, u32::MAX], + capabilities: 1 | 1 << 13 | 1 << 63, + account: AccountMode::Numeric, + ..Identity::new(4242, u32::MAX) + } +} + +/// Returns starts with every unit, flag, and kind of argument, of identity, of working +/// directory, and of environment. fn starts() -> Vec { let bytes = Limits::new(1, Window::bytes(1), Window::bytes(u32::MAX)).unwrap(); let largest = Limits::new(MAX_CHUNK, Window::frames(u32::MAX), Window::frames(1)).unwrap(); @@ -39,14 +50,20 @@ fn starts() -> Vec { bytes, Timers::new(Duration::ZERO, Duration::from_millis(u64::MAX)).timeout(Duration::ZERO), [b"/x".to_vec(), Vec::new(), (1..=u8::MAX).collect()], - ), + ) + .with_identity(numeric()) + .with_working_directory("/") + .with_environment(["A=", "B==", "PATH=/usr/bin:/bin"]), Start::new( Key::new(0, 0), largest, timers().timeout(Duration::from_millis(1500)), ["/bin/true"], ) - .with_input(), + .with_input() + .with_identity(Identity::new(0, 0)) + .with_working_directory(std::iter::once(b'/').chain(1..=u8::MAX).collect::>()) + .with_environment([(1..=u8::MAX).collect::>()]), ] } @@ -97,16 +114,56 @@ fn a_start_s_layout_is_its_key_limits_timers_input_and_arguments() { expected.extend_from_slice(b"/p"); expected.extend_from_slice(&1u32.to_le_bytes()); expected.extend_from_slice(b"a"); + // No identity, no working directory, and an empty environment. + expected.extend_from_slice(&[0, 0, 0, 0]); assert_eq!(start.encode(), expected); } +#[test] +fn a_start_s_layout_ends_with_its_identity_working_directory_and_environment() { + let start = Start::new(KEY, shared(), timers(), ["/p"]) + .with_identity(Identity { + groups: vec![50, 60], + capabilities: 1 << 13 | 1, + account: AccountMode::Numeric, + ..Identity::new(1000, 100) + }) + .with_working_directory("/w") + .with_environment(["A=1", "B="]); + let head = Start::new(KEY, shared(), timers(), ["/p"]).encode(); + let mut expected = head[..head.len() - 4].to_vec(); + expected.push(1); + expected.extend_from_slice(&1000u32.to_le_bytes()); + expected.extend_from_slice(&100u32.to_le_bytes()); + expected.push(1); + expected.extend_from_slice(&0x2001u64.to_le_bytes()); + expected.extend_from_slice(&2u16.to_le_bytes()); + expected.extend_from_slice(&50u32.to_le_bytes()); + expected.extend_from_slice(&60u32.to_le_bytes()); + expected.push(1); + expected.extend_from_slice(&2u32.to_le_bytes()); + expected.extend_from_slice(b"/w"); + expected.extend_from_slice(&2u16.to_le_bytes()); + expected.extend_from_slice(&3u32.to_le_bytes()); + expected.extend_from_slice(b"A=1"); + expected.extend_from_slice(&2u32.to_le_bytes()); + expected.extend_from_slice(b"B="); + assert_eq!(start.encode(), expected); + // An existing account is 0. + let existing = Start::new(KEY, shared(), timers(), ["/p"]).with_identity(Identity::new(0, 0)); + assert_eq!(existing.encode()[head.len() - 4 + 9], 0); +} + #[test] fn starts_that_break_the_rules_are_refused() { let valid = Start::new(KEY, shared(), timers(), ["/bin/sh", "-c", "exit"]).encode(); + let full = starts()[2].encode(); // Every start that ends early, and one with a byte too many. - for length in 0..valid.len() { - let problem = Start::decode(&valid[..length]).unwrap_err(); - assert_eq!(problem.problem(), "an execution's start ends early"); + for body in [&valid, &full] { + for length in 0..body.len() { + let problem = Start::decode(&body[..length]).unwrap_err(); + assert_eq!(problem.problem(), "an execution's start ends early"); + } } let mut longer = valid.clone(); longer.push(0); @@ -154,6 +211,7 @@ fn starts_that_break_the_rules_are_refused() { // Arguments: none, an empty program, a NUL byte, and a length beyond the body. let mut none = valid[..57].to_vec(); none.extend_from_slice(&0u16.to_le_bytes()); + none.extend_from_slice(&[0, 0, 0, 0]); assert_eq!( Start::decode(&none).unwrap_err().problem(), refused("names no program") @@ -174,6 +232,64 @@ fn starts_that_break_the_rules_are_refused() { Start::decode(&beyond).unwrap_err().problem(), "an execution's start ends early" ); + // The flags of the identity and the working directory, and the account mode, which the + // identity's IDs precede. + let head = &valid[..valid.len() - 4]; + let ending = |tail: &[u8]| { + let body = [head, tail].concat(); + Start::decode(&body).unwrap_err().problem().to_owned() + }; + assert_eq!(ending(&[2, 0, 0, 0]), refused("has the flag 2")); + assert_eq!(ending(&[0, 3, 0, 0]), refused("has the flag 3")); + let mut identity = vec![1]; + identity.extend_from_slice(&[0; 8]); + identity.push(2); + identity.extend_from_slice(&[0; 10]); + identity.extend_from_slice(&[0, 0, 0]); + assert_eq!( + ending(&identity), + refused("names the unknown account mode 2") + ); + // Working directories that are not absolute, or hold a NUL byte, and environment entries + // that are not `NAME=VALUE`, or hold a NUL byte. + let start = || Start::new(KEY, shared(), timers(), ["/bin/true"]); + let cases = [ + ( + start().with_working_directory("relative"), + "names a working directory that is not absolute", + ), + ( + start().with_working_directory(""), + "names a working directory that is not absolute", + ), + ( + start().with_working_directory("/a\0b"), + "has a working directory with a NUL byte", + ), + ( + start().with_environment(["A=1", "NAME"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + start().with_environment(["=value"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + start().with_environment([""]), + "has an environment entry that is not NAME=VALUE", + ), + ( + start().with_environment(["A=b\0c"]), + "has an environment entry with a NUL byte", + ), + ]; + for (start, problem) in cases { + assert_eq!( + Start::decode(&start.encode()).unwrap_err().problem(), + refused(problem), + "{start:?}" + ); + } } /// Returns outcomes with every status, cause, launch failure, and flag. diff --git a/tests/runtime-adapter/src/root.rs b/tests/runtime-adapter/src/root.rs index 41dcb762d..3435d5056 100644 --- a/tests/runtime-adapter/src/root.rs +++ b/tests/runtime-adapter/src/root.rs @@ -7,8 +7,8 @@ //! the rest that the root needs, which only the adapter defines, after the host attaches: the //! runtime directory, the cgroups, their resource values, the sizes that the block devices must //! have, how long the guest waits for them, the most bytes that the first partition of a GPT disk -//! may hold, how the guest assembles the overlay over the upper, and how many mappings the host's -//! mapping table holds. +//! may hold, how the guest assembles the overlay over the upper, how many mappings the host's +//! mapping table holds, and the workload identity of the whole sandbox, if it has one. //! //! A root configuration is UTF-8 text: one `NAME=VALUE` line per field, each ended by a newline, //! in any order, and each at most once. Numbers are decimal, without a sign or leading zeros: @@ -29,12 +29,17 @@ //! | `overlay-mode` | The overlay's [mode](OverlayMode): `trusted-lowers`, the guest's default, or `untrusted-lowers` | no | //! | `scratch-journal` | Whether an ext4 scratch may have a journal, a [`JournalPolicy`]: `allowed`, the guest's default, or `forbidden` | no | //! | `mapping-count` | The number of mappings of the host's mapping table, at most [`MAX_HOST_MAPPINGS`] | no | +//! | `workload-uid` | The user ID of the sandbox's workload identity, as which every execution runs | with `workload-gid` | +//! | `workload-gid` | The group ID of the sandbox's workload identity | with `workload-uid` | //! //! The paths are absolute and canonical guest paths: not `/`, without an empty, `.`, or `..` //! component, or a trailing `/`. The runtime directory holds no `,`, `:`, `\`, or `=`, which the //! overlay's options reserve, and the two cgroups have the same parent. The optional numbers are -//! positive. [`RootConfig::decode`] refuses every other configuration, which the guest reports as -//! a failure of [`RootStage::Configuration`]. +//! positive. A workload identity has both IDs, each below 4,294,967,295, which the kernel reads +//! as "unchanged", and no supplementary groups or capabilities; the guest checks its account in +//! the assembled root, a failure of [`RootStage::Identity`], and an execution then names no +//! identity of its own. [`RootConfig::decode`] refuses every other configuration, which the guest +//! reports as a failure of [`RootStage::Configuration`]. //! //! A [`RootFailure`] is a stage's code, one byte, then the problem as UTF-8 text. //! @@ -120,8 +125,12 @@ const OVERLAY_MODE: &str = "overlay-mode"; const SCRATCH_JOURNAL: &str = "scratch-journal"; /// See [`VERSION`]. const MAPPING_COUNT: &str = "mapping-count"; +/// See [`VERSION`]. +const WORKLOAD_UID: &str = "workload-uid"; +/// See [`VERSION`]. +const WORKLOAD_GID: &str = "workload-gid"; /// Every field, in the order in which [`RootConfig::encode`] writes them. -const FIELDS: [&str; 14] = [ +const FIELDS: [&str; 16] = [ VERSION, RUNTIME, AGENT_CGROUP, @@ -136,6 +145,8 @@ const FIELDS: [&str; 14] = [ OVERLAY_MODE, SCRATCH_JOURNAL, MAPPING_COUNT, + WORKLOAD_UID, + WORKLOAD_GID, ]; /// The overlay mode that a configuration names, which the guest's assembly applies: the @@ -230,6 +241,9 @@ pub struct RootConfig { /// The number of mappings of the host's mapping table, at most [`MAX_HOST_MAPPINGS`], or 0 /// for none. pub mapping_count: u32, + /// The workload identity of the whole sandbox, its user ID and group ID, if it has one: every + /// execution runs as it, and names no identity of its own. + pub workload_identity: Option<(u32, u32)>, } impl Default for RootConfig { @@ -237,7 +251,7 @@ impl Default for RootConfig { /// [`DEFAULT_WORKLOAD_CGROUP`], [`DEFAULT_DEVICE_TIMEOUT`], [`DEFAULT_AGENT_MEMORY_LOW`], /// [`DEFAULT_WORKLOAD_MEMORY_MAX`], and [`DEFAULT_WORKLOAD_PIDS_MAX`], with no sizes, the /// guest's default bound of the first partition, overlay mode, and journal policy, and no - /// mapping. + /// mapping or workload identity. fn default() -> Self { Self { runtime: DEFAULT_RUNTIME.to_owned(), @@ -253,6 +267,7 @@ impl Default for RootConfig { overlay_mode: None, scratch_journal: None, mapping_count: 0, + workload_identity: None, } } } @@ -297,6 +312,12 @@ impl RootConfig { if self.mapping_count > 0 { fields.push((MAPPING_COUNT, self.mapping_count.to_string())); } + if let Some((uid, gid)) = self.workload_identity { + fields.extend([ + (WORKLOAD_UID, uid.to_string()), + (WORKLOAD_GID, gid.to_string()), + ]); + } fields } @@ -412,6 +433,26 @@ impl RootConfig { "announces {mapping_count} mappings, more than {MAX_HOST_MAPPINGS}" ))); } + let id = |name: &str| -> Result, BodyError> { + let Some(value) = number(name)? else { + return Ok(None); + }; + match u32::try_from(value) { + Ok(id) if id < u32::MAX => Ok(Some(id)), + _ => Err(refused(format!( + "has a {name} above 4294967294, which no identity has: {value}" + ))), + } + }; + let workload_identity = match (id(WORKLOAD_UID)?, id(WORKLOAD_GID)?) { + (Some(uid), Some(gid)) => Some((uid, gid)), + (None, None) => None, + _ => { + return Err(refused(format!( + "names a workload identity without both {WORKLOAD_UID} and {WORKLOAD_GID}" + ))); + } + }; Ok(Self { runtime, agent_cgroup, @@ -427,6 +468,7 @@ impl RootConfig { scratch_journal, // The count is at most `MAX_HOST_MAPPINGS`, so it fits a `u32`. mapping_count: mapping_count as u32, + workload_identity, }) } } @@ -486,19 +528,24 @@ pub enum RootStage { Assembly, /// Tearing the root down. Teardown, + /// Checking the account of the sandbox's workload identity in the assembled root, in + /// existing-account mode, once the root is assembled. + Identity, } impl RootStage { - /// Every stage, in order. - pub const ALL: [Self; 5] = [ + /// Every stage, in the order of their codes. + pub const ALL: [Self; 6] = [ Self::Configuration, Self::Layout, Self::Bootstrap, Self::Assembly, Self::Teardown, + Self::Identity, ]; - /// Returns the stage's code in a [`RootFailure`]'s body: its position, from 1. + /// Returns the stage's code in a [`RootFailure`]'s body: its position in [`ALL`](Self::ALL), + /// from 1. pub const fn code(self) -> u8 { match self { Self::Configuration => 1, @@ -506,6 +553,7 @@ impl RootStage { Self::Bootstrap => 3, Self::Assembly => 4, Self::Teardown => 5, + Self::Identity => 6, } } @@ -514,8 +562,8 @@ impl RootStage { Self::ALL.into_iter().find(|stage| stage.code() == code) } - /// Returns the stage's name: `configuration`, `layout`, `bootstrap`, `assembly`, or - /// `teardown`. + /// Returns the stage's name: `configuration`, `layout`, `bootstrap`, `assembly`, + /// `teardown`, or `identity`. pub const fn as_str(self) -> &'static str { match self { Self::Configuration => "configuration", @@ -523,6 +571,7 @@ impl RootStage { Self::Bootstrap => "bootstrap", Self::Assembly => "assembly", Self::Teardown => "teardown", + Self::Identity => "identity", } } @@ -532,8 +581,8 @@ impl RootStage { } /// Returns the status with which the guest's worker, and with it the VM, ends once this stage - /// failed: [`FAILURE_STATUS_BASE`] plus its code, from 11 for the configuration to 15 for - /// the teardown. + /// failed: [`FAILURE_STATUS_BASE`] plus its code, from 11 for the configuration to 16 for + /// the identity. pub const fn exit_status(self) -> u8 { FAILURE_STATUS_BASE + self.code() } @@ -834,6 +883,7 @@ mod tests { overlay_mode: Some(OverlayMode::UntrustedLowers), scratch_journal: Some(JournalPolicy::Forbidden), mapping_count: 3, + workload_identity: Some((1000, 100)), ..RootConfig::default() } } @@ -855,7 +905,9 @@ mod tests { first-partition-bound=4096\n\ overlay-mode=untrusted-lowers\n\ scratch-journal=forbidden\n\ - mapping-count=3\n" + mapping-count=3\n\ + workload-uid=1000\n\ + workload-gid=100\n" ); // Without a mapping, the configuration names no count. let unmapped = String::from_utf8(RootConfig::default().encode()).unwrap(); @@ -873,7 +925,11 @@ mod tests { workload_pids_max: None, ..RootConfig::default() }; - for config in [full(), minimal, RootConfig::default()] { + let largest = RootConfig { + workload_identity: Some((u32::MAX - 1, u32::MAX - 1)), + ..RootConfig::default() + }; + for config in [full(), minimal, largest, RootConfig::default()] { assert_eq!(RootConfig::decode(&config.encode()), Ok(config)); } // The fields may come in any order. @@ -1058,6 +1114,30 @@ mod tests { format!("{base}mapping-count=18446744073709551615\n").into_bytes(), "announces 18446744073709551615 mappings, more than 4096", ), + ( + format!("{base}workload-uid=0\nworkload-gid=1000\n").into_bytes(), + "has a workload-uid that is not a positive decimal number: \"0\"", + ), + ( + format!("{base}workload-uid=1000\nworkload-gid=0\n").into_bytes(), + "has a workload-gid that is not a positive decimal number: \"0\"", + ), + ( + format!("{base}workload-uid=4294967295\nworkload-gid=1000\n").into_bytes(), + "has a workload-uid above 4294967294, which no identity has: 4294967295", + ), + ( + format!("{base}workload-uid=1000\nworkload-gid=4294967296\n").into_bytes(), + "has a workload-gid above 4294967294", + ), + ( + format!("{base}workload-uid=1000\n").into_bytes(), + "names a workload identity without both workload-uid and workload-gid", + ), + ( + format!("{base}workload-gid=1000\n").into_bytes(), + "names a workload identity without both workload-uid and workload-gid", + ), ]; for (body, problem) in cases { let error = RootConfig::decode(&body).unwrap_err(); @@ -1095,6 +1175,7 @@ mod tests { (RootStage::Bootstrap, 3, "bootstrap", 13), (RootStage::Assembly, 4, "assembly", 14), (RootStage::Teardown, 5, "teardown", 15), + (RootStage::Identity, 6, "identity", 16), ]; for (stage, code, name, status) in expected { assert_eq!(stage.code(), code); @@ -1104,8 +1185,9 @@ mod tests { assert_eq!(RootStage::from_name(name), Some(stage)); assert_eq!(stage.exit_status(), status); } + assert_eq!(RootStage::ALL.map(RootStage::code), [1, 2, 3, 4, 5, 6]); assert_eq!(RootStage::from_code(0), None); - assert_eq!(RootStage::from_code(6), None); + assert_eq!(RootStage::from_code(7), None); assert_eq!(RootStage::from_name("mapping"), None); } @@ -1131,7 +1213,7 @@ mod tests { for (body, problem) in [ (&b""[..], "names no stage"), (b"\x00problem", "unknown stage 0"), - (b"\x06problem", "unknown stage 6"), + (b"\x07problem", "unknown stage 7"), (b"\x01\xffproblem", "not UTF-8 text"), ] { let error = RootFailure::decode(body).unwrap_err(); diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index f6ecf540c..c2db2b503 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -21,9 +21,11 @@ //! 1. It opens the tty and attaches the session. //! 2. It receives the host's root configuration, a [`RootConfig`], and checks it. //! 3. It reads the boot layout of the root from the kernel command line with -//! `nvx::protocol::root::RootLayout::parse`, and the host export, if the command line names -//! one, with `nvx::protocol::export::HostExport::parse`, then bootstraps the guest, assembles -//! the root, and mounts the export out of the root. +//! `nvx::protocol::root::RootLayout::parse`, and the host export, if the command line names one, +//! with `nvx::protocol::export::HostExport::parse`, then bootstraps the guest, assembles the +//! root, and mounts the export out of the root. If the configuration names a workload identity +//! for the whole sandbox, it then checks the identity's account in the assembled root, in +//! existing-account mode, through `nvx::guest::entry`. //! 4. It answers `Ready`, with the guest agent's build, [`GUEST_BUILD`], or `RootFailed` with the //! stage that failed, and then ends. //! 5. It applies the host's mapping table beneath the root, from the export, in the parts that @@ -44,11 +46,12 @@ //! configures a scratch for a root over RAM, bounds a first partition for an EROFS image, or //! announces mappings while the kernel command line names no host export, from which they come, //! and a command line whose tokens of the export break their rules. An export that fails to mount -//! fails the assembly. +//! fails the assembly. A workload identity whose account the assembled root lacks fails the +//! identity stage. //! //! The worker, and with it the VM, ends with 0 once the host stopped it, after a teardown that //! succeeded, with the stage's [exit status](RootStage::exit_status) once a stage failed, from 11 -//! for the configuration to 15 for the teardown, and with [`SESSION_STATUS`] once its session +//! for the configuration to 16 for the identity, and with [`SESSION_STATUS`] once its session //! failed or the broker closed it. Once the broker closed the stream at a record boundary, which //! may close only the broker's end, the worker drops its executions, and writes what it queued, //! as once the host stopped it. @@ -114,22 +117,60 @@ //! session abandons every execution: the worker drops it, and it runs until it ends, detached //! from the host, which receives nothing more about it. //! - The worker admits at most [`MAX_EXECUTIONS`] executions at once, abandoned ones included, -//! and those whose programs' statuses it has not collected, and none that would hold more than -//! [`MAX_EXECUTION_MEMORY`] bytes. It refuses a start beyond those, or one whose program it -//! cannot spawn, with a terminal record whose launch failed at stage 0, with `EAGAIN`, -//! `ENOMEM`, or the spawn's `errno`. -//! - Each program runs in the worker's root and working directories, with its environment, and -//! leads a process group of its own, which its execution kills as a whole. -//! - The worker collects the status of every program itself, without waiting for it, and learns -//! that a program ended from a pidfd of its own. Before it collects the status, it kills what -//! is left of the program's process group, as NVX's settlement through a cgroup kills what is -//! left in the cgroup: until then, the program, a zombie, keeps its process ID, which is the -//! group's, from any other process. So a descendant that stayed in the group neither outlives -//! the program nor holds its output open. The program runs without a cgroup, though, so nothing -//! proves that a descendant that left the group is gone, and each outcome is fenced: once the -//! program ended, a stream that such a process may still hold has the settlement time and the -//! grace to close, after which the runner gives up on it, while a stream that no process holds -//! any more loses nothing, however slowly the host takes it. +//! and those that keep their places once they ended, as the items below describe, and none +//! that would hold more than [`MAX_EXECUTION_MEMORY`] bytes. It refuses a start beyond those, +//! one whose identity or entry it refuses, as the next item describes, or one whose cgroup it +//! cannot prepare, or whose program it cannot spawn, with a terminal record whose launch failed +//! at stage 0, with `EAGAIN`, `ENOMEM`, the `errno` of the refusal, or that of the preparation +//! or the spawn. +//! - Each program enters the assembled root through `nvx::guest::entry`, as an [`Entrance`] +//! describes it: in new mount, PID, and UTS namespaces, with the root moved over `/`, the entry's +//! default hostname, `nvx-sandbox`, the image's own machine ID, and no time status, so that the +//! entry binds nothing into the root, and with published listeners, the entry's default. It runs +//! as the identity that exactly one source names: the sandbox's workload identity, which the root +//! configuration names, with no supplementary groups or capabilities, or, in a sandbox without +//! one, the identity that its start names, with its supplementary groups, its capabilities, and +//! its account mode. A start that names an identity in a sandbox with a workload identity, or +//! none in a sandbox without one, is refused with `EINVAL`. Before the worker prepares the +//! execution's cgroup, the entry checks the identity, and reads its account in existing-account +//! mode: it refuses with `EPERM` a capability outside its allowlist, any capability of a user ID +//! other than 0, and `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while listeners are published, with +//! `ENOENT` an identity without the user ID's passwd entry or the group ID's group entry, with +//! the error of an account database or a root that it cannot read, and with `EINVAL` otherwise. +//! The program starts in the working directory of its start, or in the root's `/`, with exactly +//! the environment of its start, and leads a session of its own, without a controlling terminal, +//! and the process group that the session creates. As PID 1 of its PID namespace, its end ends +//! every process of the namespace. +//! - Each execution runs in a cgroup of its own below the workload cgroup, `execution-N`, where N +//! counts the cgroups that the worker prepared. The worker prepares it with +//! `nvx::guest::cgroup::ExecutionCgroup::prepare`, and the program joins it as it spawns, +//! before it runs, so that its descendants start in it, however they leave its process group or +//! session. The execution settles through that cgroup alone, which leaves the other executions +//! running: once the program ended, as its status, which the execution collects itself, +//! tells, the execution kills what is left in the cgroup, and settles once the cgroup is empty. +//! So no descendant outlives the program or holds its output open, and an execution whose +//! program ends on its own loses nothing, however slowly the host takes its output, and is +//! reusable. +//! - An execution keeps its place among [`MAX_EXECUTIONS`] until its cgroup is empty and the +//! worker removed it, which the worker does once the runner returned the terminal outcome of an +//! execution that settled. One that did not settle, whose outcome is fenced, keeps its place +//! and its cgroup until the host stops the worker, as does a start whose program the worker +//! spawned, but whose execution it could not have: until the worker collected the status of +//! such a program, the program may still join the cgroup, so that a removal could hide it once +//! it ended. +//! - Without cgroups, as where the in-process tests cannot create them, the [root's +//! operations](RootOperations::cgroups) name none, and the program's process group stands in +//! for its cgroup: its execution kills the group as a whole. The worker collects the status of +//! every program itself then, without waiting for it, and learns that a program ended from a +//! pidfd of its own. Before it collects the status, it kills what is left of the program's +//! process group, as the settlement through a cgroup kills what is left in the cgroup: until +//! then, the program, a zombie, keeps its process ID, which is the group's, from any other +//! process. So a descendant that stayed in the group neither outlives the program nor holds its +//! output open. Nothing proves that a descendant that left the group is gone, though, so each +//! outcome is fenced: once the program ended, a stream that such a process may still hold has +//! the settlement time and the grace to close, after which the runner gives up on it, while a +//! stream that no process holds any more loses nothing, however slowly the host takes it. Such +//! an execution keeps its place until the worker collected its program's status. //! - Before it hands an execution the host's cancellation, the worker gives the execution a turn, //! as the loop does, which collects the status of a program that ended, and reports the //! program's end to the runner. So the end of a program that ended before its timeout while the @@ -138,11 +179,13 @@ //! timeout's instant. //! - The worker forgets an execution once its runner returned the terminal outcome, which may //! come before the program's end, as once the runner gave up on a program that a kill did not -//! end in time: forgetting it kills the program and its process group again, and the worker -//! collects the program's status once it ended. On a stop, or once its session ended -//! otherwise, the worker drops every execution, which kills the process group of each program -//! whose status it has not collected, and waits up to a second in all for those programs to -//! end. +//! end in time: forgetting it kills the program, and what is left in its cgroup or its process +//! group, again, and the worker collects the program's status once it ended. On a stop, or +//! once its session ended otherwise, the worker drops every execution, which kills the program +//! of each one that has not ended, with what is left in its cgroup or its process group, and +//! what is left in the cgroups of the executions that did not settle, and waits up to a second +//! in all for those programs to end and those cgroups to empty, removing each cgroup that +//! emptied once it collected the statuses of those programs. mod connection; pub mod entries; @@ -168,12 +211,14 @@ use nix::sys::wait::{self, Id, WaitPidFlag, WaitStatus}; use nix::unistd::Pid; use nvx::gpt::Expected; use nvx::guest::cgroup::Layout; +use nvx::guest::entry::{self, Entry, Identity, Prepared}; use nvx::guest::mappings::{Mapping, SourceRoot}; use nvx::guest::nvxs::GuestConnectionError; use nvx::guest::process::Child; use nvx::guest::root::{ self, Assembly, Bootstrap, Device, Image, Journal, Resources, Root, Upper, UpperKind, }; +use nvx::guest::spawn::{self, Spawned}; use nvx::protocol::export::HostExport; use nvx::protocol::root::{ DISTRO_MMIO_BASE, ImageFormat, RootLayout, RootUpper, SCRATCH_MMIO_BASE, @@ -189,7 +234,7 @@ use crate::root::{ RootStage, }; use connection::{Connection, Received}; -use executions::Executions; +use executions::{Executions, Sandbox}; use order::Order; /// The status with which the worker ends once its session failed or the broker closed it, and @@ -200,7 +245,9 @@ pub const SESSION_STATUS: u8 = 1; pub const EXECUTING: &str = "the worker runs executions, and probes nothing until they end"; /// The most executions that the worker runs at once, those that the host abandoned included, and -/// those whose programs the worker forgot before it collected their statuses. +/// those that ended but keep their places: with cgroups, those whose cgroups the worker has not +/// removed, as those of the executions that did not settle, and without them, those whose +/// programs the worker forgot before it collected their statuses. pub const MAX_EXECUTIONS: usize = 8; /// The most bytes that one execution may hold in the worker: what its runner holds, a chunk of @@ -216,10 +263,45 @@ pub const MAX_HISTORY: usize = 1024; /// The kernel command line, from which the worker reads the boot layout of the root. const COMMAND_LINE: &str = "/proc/cmdline"; +/// How one execution enters the assembled root: the inputs of its `nvx::guest::entry::Entry` that +/// the worker chooses. The entry keeps its defaults for the rest: new mount, PID, and UTS +/// namespaces, with the root moved over `/`, the image's own machine ID and no time status, which +/// leave `Entry::bind_runtime` nothing to bind, published listeners, which refuse +/// `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW`, and every device node. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Entrance { + /// The identity: the sandbox's workload identity, or the one that the execution's start names. + pub identity: Identity, + /// The hostname, the sandbox's: the entry's default. + pub hostname: Vec, +} + +impl Entrance { + /// Returns the entry into the root at `root`. + pub fn entry(&self, root: impl Into) -> Entry { + Entry::new(root, self.identity.clone()).hostname(self.hostname.clone()) + } +} + +/// The operations through which the executions enter the assembled root: those of +/// `nvx::guest::entry` in a VM, in [`LibraryRoot`], and a fake's in tests, which cannot enter a +/// root without privileges. +pub trait EntryOperations { + /// Prepares the entry of an execution into the assembled root, as `entrance` describes it, + /// with `Entry::prepare`: checks its inputs, and reads and checks the identity's account in + /// existing-account mode. + fn prepare(&mut self, entrance: &Entrance) -> Result; + + /// Spawns `command`, whose child enters the execution that `prepared` prepared. A fake that + /// cannot enter a root spawns the command in its own root instead, as its own identity. + fn spawn(&mut self, command: spawn::Command, prepared: Prepared) -> io::Result; +} + /// The operations of a layered root that the worker runs: those of `nvx::guest::root` in a VM, in /// [`LibraryRoot`], and a fake's in tests. Each fails with the problem, as text, and -/// [`map`](Self::map) with the mapping that failed too. -pub trait RootOperations { +/// [`map`](Self::map) with the mapping that failed too, but the [entry's](EntryOperations), which +/// fail as `nvx::guest::entry` does. +pub trait RootOperations: EntryOperations { /// Returns the kernel command line, from which the worker reads the boot layout of the root, /// and the host export. fn command_line(&mut self) -> Result, String>; @@ -242,6 +324,12 @@ pub trait RootOperations { /// Fails with a [`MapError`] that counts the mappings before the one that failed. fn map(&mut self, mappings: &[Mapping]) -> Result<(), MapError>; + /// Returns the cgroups in which the worker runs its executions once the root is ready: the + /// layout that the bootstrap prepared, below whose workload cgroup each execution gets a + /// cgroup of its own. Without them, as in tests that cannot create cgroups, each program runs + /// in a process group alone. + fn cgroups(&self) -> Option; + /// Runs `probe` on the assembled root, and returns what it found. fn probe(&mut self, probe: &Probe) -> Result; @@ -337,7 +425,7 @@ pub fn work( message, )?, None => { - let (step, ready) = configure_root(operations, message)?; + let (step, ready) = configure_root(operations, &mut executions, message)?; order = ready; step } @@ -436,15 +524,19 @@ fn wait( } /// Handles `message` before the root is ready: a root configuration, whose root it assembles -/// through `operations`, and which it answers with its readiness, or a `Stop`. Returns the step, -/// with the worker's order once the root is ready. +/// through `operations`, after which `executions` run in the cgroups that the operations prepared, +/// as the sandbox that the configuration and the kernel command line describe, and which it answers +/// with its readiness, or a `Stop`. Returns the step, with the worker's order once the root is +/// ready. fn configure_root( operations: &mut dyn RootOperations, + executions: &mut Executions, message: Message, ) -> Result<(Step, Option), ServeError> { match message { Message::RootConfig(body) => Ok(match assemble(operations, &body) { - Ok(config) => { + Ok((config, sandbox)) => { + executions.run_in(operations.cgroups(), sandbox); let ready = Ready::new(GUEST_BUILD).expect("this crate's version is a build"); let step = Step::Continue(vec![Message::Ready(ready.encode())]); (step, Some(Order::new(config.mapping_count))) @@ -460,8 +552,8 @@ fn configure_root( } /// Handles `message` at `now` once the root is ready, through `operations`, in the worker's -/// `order`, and hands a message about an execution to `executions`, which queue what they answer -/// on `connection`. +/// `order`, and hands a message about an execution to `executions`, which enter the root through +/// `operations`, and queue what they answer on `connection`. fn serve_root( operations: &mut dyn RootOperations, order: &mut Order, @@ -476,7 +568,7 @@ fn serve_root( return Err(ServeError::Unexpected(message.kind())); } let decoded = decoded.map_err(|error| ServeError::Execution(error.problem().to_owned()))?; - executions.deliver(now, decoded, connection)?; + executions.deliver(now, decoded, operations, connection)?; return Ok(Step::Continue(Vec::new())); } Ok(match message { @@ -512,10 +604,15 @@ fn serve_root( } /// Checks the root configuration `body`, reads the boot layout of the root and the host export, -/// bootstraps the guest, assembles the root, and mounts the export out of it, if the kernel -/// command line names one, through `operations`. Returns the configuration, or the failure of the -/// stage that failed: the export's mount ends the assembly. -fn assemble(operations: &mut dyn RootOperations, body: &[u8]) -> Result { +/// bootstraps the guest, assembles the root, mounts the export out of it, if the kernel command +/// line names one, and checks the account of the sandbox's workload identity there, if the +/// configuration names one, through `operations`. Returns the configuration, with the sandbox in +/// which the executions run, or the failure of the stage that failed: the export's mount ends the +/// assembly. +fn assemble( + operations: &mut dyn RootOperations, + body: &[u8], +) -> Result<(RootConfig, Sandbox), RootFailure> { let config = RootConfig::decode(body) .map_err(|error| RootFailure::new(RootStage::Configuration, error.problem()))?; let (layout, export) = layout(operations, &config) @@ -531,7 +628,20 @@ fn assemble(operations: &mut dyn RootOperations, body: &[u8]) -> Result Option { + self.bootstrap + .as_ref() + .map(|bootstrap| bootstrap.layout().clone()) + } + /// Runs one of the [`probes`], of the [`mapping`] probes, which inspect the root's mappings /// and map from the host export, or of the [`entries`] probes, which enter it. fn probe(&mut self, probe: &Probe) -> Result { @@ -800,6 +917,22 @@ impl RootOperations for LibraryRoot { } } +impl EntryOperations for LibraryRoot { + /// Runs `Entry::prepare` on the assembled root. + fn prepare(&mut self, entrance: &Entrance) -> Result { + let Some(root) = &self.root else { + let missing = io::Error::new(io::ErrorKind::NotFound, "the worker holds no root"); + return Err(entry::Error::Root(missing)); + }; + entrance.entry(root.path()).prepare() + } + + /// Spawns `command`, whose child enters the execution with `Command::enter`. + fn spawn(&mut self, command: spawn::Command, prepared: Prepared) -> io::Result { + command.enter(prepared).spawn() + } +} + /// Runs PID 1 of a VM with a layered root, as the [module documentation](self) describes, with /// the worker on the tty `tty`, and returns the worker's status, or [`SESSION_STATUS`] if a child /// cannot start or the worker's status cannot be collected. diff --git a/tests/runtime-adapter/src/worker/executions.rs b/tests/runtime-adapter/src/worker/executions.rs index dfae13982..a92782a98 100644 --- a/tests/runtime-adapter/src/worker/executions.rs +++ b/tests/runtime-adapter/src/worker/executions.rs @@ -1,8 +1,10 @@ //! The worker's executions, as the worker's module documentation describes them: the table that //! reaches each one by its ID, the generations that the host started, their admission, the -//! refusal of a start, the loop's turns at every execution, the collection of each program's -//! status, before which the worker ends what is left of the program's process group, and that of -//! the statuses of the programs that the worker forgot. +//! refusal of a start, the loop's turns at every execution, the cgroup of each execution, through +//! which it settles, and which keeps its place until the worker removes it, and, without +//! cgroups, the collection of each program's status, before which the worker ends what is left +//! of the program's process group, and that of the statuses of the programs that the worker +//! forgot. use std::collections::{BTreeMap, VecDeque}; use std::ffi::CString; @@ -17,14 +19,16 @@ use nix::poll::{self, PollFd, PollFlags, PollTimeout}; use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, WaitPidFlag, WaitStatus}; use nvx::exec::{Key, LaunchFailure, Runner, Settlement, Stream, Violation, runner}; +use nvx::guest::cgroup::{ExecutionCgroup, Layout, Scope}; +use nvx::guest::entry::{self, AccountError, Capabilities, Capability, Prepared}; use nvx::guest::exec::binding::{Binding, Event}; use nvx::guest::exec::{Execution, Options, Pipes, Source}; use nvx::guest::process::Child; use nvx::guest::spawn::Command; use super::connection::{BACKLOG, Connection}; -use super::{MAX_EXECUTION_MEMORY, MAX_EXECUTIONS, MAX_HISTORY}; -use crate::exec::{GuestMessage, HostMessage, MAX_CHUNK, Start, name}; +use super::{Entrance, EntryOperations, MAX_EXECUTION_MEMORY, MAX_EXECUTIONS, MAX_HISTORY}; +use crate::exec::{AccountMode, GuestMessage, HostMessage, Identity, MAX_CHUNK, Start, name}; use crate::guest::ServeError; /// The device that a program without the host's input reads. @@ -56,6 +60,17 @@ pub(super) struct Executions { /// When the worker stops waiting for the programs once it dropped every execution: a while /// after it first did. cleared: Option, + /// The cgroups in which the executions run, once the root is ready: the layout below whose + /// workload cgroup each execution gets a cgroup of its own. Without them, each program runs + /// in a process group alone. + cgroups: Option, + /// The sandbox in which the executions run. + sandbox: Sandbox, + /// How many cgroups of executions the worker prepared, which numbers the next one. + prepared: u64, + /// The cgroups of the executions that did not settle, each of which keeps its place among + /// [`MAX_EXECUTIONS`] until the worker drops every execution. + fenced: Vec, /// The ID from which the next turn serves the executions that the host reaches, before /// those of lower IDs: the ID after that of the execution that the last turn served first, /// so that the executions take turns at being served first. @@ -73,18 +88,49 @@ pub(super) struct Executions { buffer: Vec, } -/// One of the worker's executions: the binding of its runner to its program, which leaves the -/// program's status to the worker, and the worker's own handle of the program. +/// The sandbox in which the executions run once the root is ready: what every execution's +/// [`Entrance`] shares. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(super) struct Sandbox { + /// The executions' hostname. + pub(super) hostname: Vec, + /// The sandbox's workload identity, its user ID and group ID, if the root configuration + /// names one, as which every execution runs. + pub(super) identity: Option<(u32, u32)>, +} + +impl Sandbox { + /// Returns the sandbox whose workload identity is `identity`, if it has one. Its executions + /// take the entry's default hostname, [`DEFAULT_HOSTNAME`](entry::DEFAULT_HOSTNAME), as + /// nothing configures another: this is the one place that names their hostname. + pub(super) fn new(identity: Option<(u32, u32)>) -> Self { + Self { + hostname: entry::DEFAULT_HOSTNAME.as_bytes().to_vec(), + identity, + } + } +} + +/// One of the worker's executions: the binding of its runner to its program, and either the +/// execution's cgroup, through which it settles, or, without cgroups, the worker's own handle of +/// the program, whose status the binding leaves to the worker. struct Run { /// The binding of the execution's runner to its program. binding: Binding, - /// The program, through a pidfd of the worker's own, which polls readable once the program - /// ended, until the worker collected its status. + /// Without cgroups, the program, through a pidfd of the worker's own, which polls readable + /// once the program ended, until the worker collected its status. program: Option, + /// The execution's cgroup, if the worker runs it in one. + cgroup: Option, + /// Whether the runner's terminal outcome tells that the execution settled, so that its + /// cgroup is empty, and the worker may remove it. + settled: bool, } impl Executions { - /// Returns no executions. + /// Returns no executions, which run in process groups alone, with the default hostname and + /// without a workload identity, until [`run_in`](Self::run_in) names their cgroups and their + /// sandbox. pub(super) fn new() -> Self { Self { attached: BTreeMap::new(), @@ -92,6 +138,10 @@ impl Executions { reaping: Vec::new(), reaped: None, cleared: None, + cgroups: None, + sandbox: Sandbox::new(None), + prepared: 0, + fenced: Vec::new(), turn: 0, owed: None, generations: BTreeMap::new(), @@ -100,21 +150,31 @@ impl Executions { } } + /// Runs the executions that start from now on in `cgroups`, and in `sandbox`: each in a + /// cgroup of its own, below the workload cgroup, or, without cgroups, in a process group + /// alone. + pub(super) fn run_in(&mut self, cgroups: Option, sandbox: Sandbox) { + self.cgroups = cgroups; + self.sandbox = sandbox; + } + /// Hands `message` from the host to its execution at `now`, and queues on `connection` the - /// terminal record of a start that the worker refuses. Before a cancellation, it gives the - /// execution a turn at `now`, and queues its messages on `connection`: the end of a program - /// that ended meanwhile, even before its timeout, while the worker did not run, reaches the - /// runner first, so that it stays the program's own, without the cause of the cancellation, - /// or of a timeout that `now` reached. + /// terminal record of a start that the worker refuses. A start's program enters the root + /// through `operations`. Before a cancellation, it gives the execution a turn at `now`, and + /// queues its messages on `connection`: the end of a program that ended meanwhile, even + /// before its timeout, while the worker did not run, reaches the runner first, so that it + /// stays the program's own, without the cause of the cancellation, or of a timeout that + /// `now` reached. pub(super) fn deliver( &mut self, now: Instant, message: HostMessage<'_>, + operations: &mut dyn EntryOperations, connection: &mut Connection, ) -> Result<(), ServeError> { let key = message.key(); if let HostMessage::Start(start) = message { - return self.start(now, start, connection); + return self.start(now, start, operations, connection); } // A message of an execution that the worker no longer runs is late. let Self { @@ -142,8 +202,9 @@ impl Executions { /// Takes a turn at `now`: collects the status of each program that ended, takes every step /// that the executions can take, queues their messages for the host on `connection`, - /// forgets each execution once it returned its terminal outcome, and collects the statuses - /// of the programs that it forgot, of those that ended. + /// forgets each execution once it returned its terminal outcome, which removes the cgroup of + /// one that settled, and collects the statuses of the programs that it forgot, of those that + /// ended. /// /// The executions that the host reaches take turns at being served first: a turn starts with /// the execution after the one that the last turn started with, in the order of their IDs, @@ -162,7 +223,6 @@ impl Executions { let Self { attached, detached, - reaping, turn, owed, buffer, @@ -184,17 +244,32 @@ impl Executions { *turn = first.wrapping_add(1); } *owed = more.then_some(now); - attached.retain(|_, run| running(run, reaping)); for run in detached.iter_mut() { let key = run.binding.runner().key(); while let Some(event) = run.binding.next(now, buffer) { - // A detached runner returns only its terminal outcome, which no host takes. - if !matches!(event, Event::Message(_)) { - report(key, &event); + match event { + // A detached runner returns only its terminal outcome, which no host takes. + Event::Message(runner::Message::Terminal(outcome)) => { + run.settled = outcome.is_reusable(); + } + Event::Message(_) => {} + event => report(key, &event), } } } - detached.retain(|run| running(run, reaping)); + let finished = attached + .iter() + .filter(|(_, run)| run.binding.is_finished()) + .map(|(&id, _)| id) + .collect::>(); + let mut forgotten = finished + .into_iter() + .filter_map(|id| attached.remove(&id)) + .collect::>(); + forgotten.extend(detached.extract_if(.., |run| run.binding.is_finished())); + for run in forgotten { + self.forget(run); + } self.reap(now); Ok(()) } @@ -208,29 +283,44 @@ impl Executions { } /// Returns whether the worker runs no execution: none that the host reaches, and none that it - /// abandoned. It may still collect the statuses of programs that it forgot. + /// abandoned. It may still collect the statuses of programs that it forgot, and hold the + /// cgroups of executions that did not settle. pub(super) fn is_empty(&self) -> bool { self.attached.is_empty() && self.detached.is_empty() } - /// Drops every execution, which ends the program and its process group of each one whose - /// program's status the worker has not collected, and waits for those programs to end, so - /// that the worker collects their statuses, at most [`REAP_WAIT`] after it first dropped - /// every execution: a later call, as once the executions drop after a stop, waits no longer. - /// The worker ended the process groups of the others when it collected their programs' - /// statuses. + /// Drops every execution, which ends the program and its process group, or the processes of + /// its cgroup, of each one that has not ended, and waits until the programs ended and their + /// cgroups emptied, so that the worker collects the programs' statuses and removes the + /// cgroups, those of the executions that did not settle included, whose processes it ends + /// too, at most [`REAP_WAIT`] after it first dropped every execution: a later call, as once + /// the executions drop after a stop, waits no longer. It removes the cgroups once it + /// collected the status of every program that it forgot: until then, such a program may + /// still join its cgroup, as its launch may not have reported, and a removal could hide it + /// once it ended. Without cgroups, the worker ended the process groups of the other programs + /// when it collected their statuses. pub(super) fn clear(&mut self) { - let Self { - attached, - detached, - reaping, - .. - } = self; - for run in std::mem::take(attached) + let runs = std::mem::take(&mut self.attached) .into_values() - .chain(detached.drain(..)) - { - forget(&run, reaping); + .chain(self.detached.drain(..)) + .collect::>(); + for run in runs { + let Run { + binding, cgroup, .. + } = run; + if binding.execution().status().is_none() { + self.reaping.push(binding.id()); + } + self.fenced.extend(cgroup); + } + // An execution that settled gave its cgroup back, which its drop then leaves alone. + for cgroup in &self.fenced { + if let Err(error) = cgroup.kill(Scope::Execution) { + eprintln!( + "runtime-adapter-guest: cannot end the processes of {}: {error}", + cgroup.path().display() + ); + } } let deadline = *self .cleared @@ -238,7 +328,10 @@ impl Executions { loop { let now = Instant::now(); self.reap(now); - if self.reaping.is_empty() || now >= deadline { + if self.reaping.is_empty() { + self.fenced.retain(|cgroup| !removed(cgroup)); + } + if (self.reaping.is_empty() && self.fenced.is_empty()) || now >= deadline { return; } thread::sleep(REAP_INTERVAL); @@ -292,6 +385,51 @@ impl Executions { self.attached.values().chain(&self.detached) } + /// Returns how many of the places of [`MAX_EXECUTIONS`] the executions take: one for each + /// execution that the worker runs, and, with cgroups, one for each cgroup of an execution that + /// did not settle, or, without them, one for each program that the worker forgot before it + /// collected its status. + fn places(&self) -> usize { + let held = if self.cgroups.is_some() { + self.fenced.len() + } else { + self.reaping.len() + }; + self.attached.len() + self.detached.len() + held + } + + /// Forgets `run`, whose runner returned its terminal outcome. Unless the worker collected + /// the program's status, the program's ID goes to `reaping`, so that the worker collects the + /// status once the program ended. The worker removes the cgroup of an execution that settled, + /// which frees its place, and keeps that of one that did not, or that it cannot remove, which + /// keeps its place until the worker drops every execution. The binding drops last, which ends + /// a program that has not ended, and its process group, or the processes of its cgroup unless + /// the execution settled: one that settled gave its cgroup back, which its drop leaves alone. + fn forget(&mut self, run: Run) { + let Run { + binding, + cgroup, + settled, + .. + } = run; + if binding.execution().status().is_none() { + self.reaping.push(binding.id()); + } + let Some(cgroup) = cgroup else { + return; + }; + if settled { + match cgroup.remove() { + Ok(()) => return, + Err(error) => eprintln!( + "runtime-adapter-guest: execution {}: cannot remove its cgroup: {error}", + name(binding.runner().key()) + ), + } + } + self.fenced.push(cgroup); + } + /// Collects at `now` the statuses of the programs that the worker forgot, of those that /// ended. fn reap(&mut self, now: Instant) { @@ -306,13 +444,15 @@ impl Executions { self.reaped = Some(now); } - /// Starts the execution of `start` at `now`, or refuses it with a terminal record that it - /// queues on `connection`. A start of an earlier generation than the latest that the host - /// started of its ID is stale, and one of the same generation fails. + /// Starts the execution of `start` at `now`, whose program enters the root through + /// `operations`, or refuses it with a terminal record that it queues on `connection`. A start + /// of an earlier generation than the latest that the host started of its ID is stale, and one + /// of the same generation fails. fn start( &mut self, now: Instant, start: Start, + operations: &mut dyn EntryOperations, connection: &mut Connection, ) -> Result<(), ServeError> { let key = start.key; @@ -339,25 +479,89 @@ impl Executions { self.detached.push(abandoned); } let runner = Runner::new(key, start.limits, start.timers, now); - let executions = self.attached.len() + self.detached.len() + self.reaping.len(); - let admitted = if executions >= MAX_EXECUTIONS { + let admitted = if self.places() >= MAX_EXECUTIONS { Err(Errno::EAGAIN as i32) } else if runner.memory_bound() > MAX_EXECUTION_MEMORY { Err(Errno::ENOMEM as i32) } else { - spawn(&start, &mut self.reaping).map_err(|error| error.raw_os_error().unwrap_or(0)) + self.launch(&start, operations) + .map_err(|error| error.raw_os_error().unwrap_or(0)) }; match admitted { - Ok((execution, program)) => { + Ok((execution, program, cgroup)) => { let binding = Binding::new(execution, runner); - let program = Some(program); - self.attached.insert(key.id(), Run { binding, program }); + let run = Run { + binding, + program, + cgroup, + settled: false, + }; + self.attached.insert(key.id(), run); Ok(()) } Err(error) => refuse(now, runner, error, connection), } } + /// Starts the program of `start`, which enters the root through `operations` as its + /// [entrance](Self::entrance) describes, once the entry prepared its execution: with cgroups, + /// in a cgroup of its own that the worker prepares below the workload cgroup, and otherwise + /// with the worker's own handle of the program. A start whose entry the root's entry refuses + /// fails with the `errno` that [`refusal`] returns, before the worker prepares a cgroup. The + /// worker removes the cgroup of a program that it could not spawn. One that it spawned, but + /// whose execution could not be had, may still join the cgroup until the worker collected its + /// status, so that a removal could hide it once it ended: its cgroup keeps its place, as that + /// of an execution that did not settle. + fn launch( + &mut self, + start: &Start, + operations: &mut dyn EntryOperations, + ) -> io::Result<(Execution, Option, Option)> { + let entrance = self.entrance(start)?; + let prepared = operations + .prepare(&entrance) + .map_err(|error| io::Error::from(refusal(&error)))?; + let Some(cgroups) = &self.cgroups else { + let (execution, program) = spawn(start, prepared, None, operations, &mut self.reaping) + .map_err(|unstarted| unstarted.error)?; + return Ok((execution, program, None)); + }; + self.prepared += 1; + let name = format!("execution-{}", self.prepared); + let cgroup = ExecutionCgroup::prepare(cgroups, Some(&name), &[])?; + match spawn( + start, + prepared, + Some(&cgroup), + operations, + &mut self.reaping, + ) { + Ok((execution, program)) => Ok((execution, program, Some(cgroup))), + Err(Unstarted { error, spawned }) => { + if spawned || cgroup.remove().is_err() { + self.fenced.push(cgroup); + } + Err(error) + } + } + } + + /// Returns how the program of `start` enters the root: with the sandbox's hostname, as the + /// identity that exactly one source names, the sandbox's workload identity, if it has one, or + /// otherwise the identity that the start names. A start that names an identity in a sandbox + /// with a workload identity, or none in a sandbox without one, is `EINVAL`. + fn entrance(&self, start: &Start) -> io::Result { + let identity = match (self.sandbox.identity, &start.identity) { + (Some((uid, gid)), None) => entry::Identity::new(uid, gid), + (None, Some(identity)) => entry_identity(identity), + _ => return Err(Errno::EINVAL.into()), + }; + Ok(Entrance { + identity, + hostname: self.sandbox.hostname.clone(), + }) + } + /// Remembers that the host started `key`, the latest generation of its ID, and forgets the /// ID that it heard of first once it remembers more than [`MAX_HISTORY`]. fn remember(&mut self, key: Key) { @@ -386,13 +590,16 @@ impl Drop for Executions { impl Run { /// Collects the program's status without blocking once the program ended, and hands it to - /// the binding. First, it kills what is left of the process group that the program leads, as - /// the binding's settlement through a cgroup kills what is left in the cgroup, so that no - /// descendant that stayed in the group outlives the program, or holds its output open: until - /// its status is collected, the program, a zombie, keeps its process ID, which is the - /// group's, from any other process, so that the kill reaches that group alone. + /// the binding, if the worker collects it: without cgroups. First, it kills what is left of + /// the process group that the program leads, as the binding's settlement through a cgroup + /// kills what is left in the cgroup, so that no descendant that stayed in the group outlives + /// the program, or holds its output open: until its status is collected, the program, a + /// zombie, keeps its process ID, which is the group's, from any other process, so that the + /// kill reaches that group alone. fn collect(&mut self) { - let Self { binding, program } = self; + let Self { + binding, program, .. + } = self; let Some(child) = program else { return; }; @@ -447,10 +654,11 @@ impl Run { } /// Serves the execution in a turn at `now`: takes every step that it can take, and queues its - /// messages for the host on `connection`. Each read of its program's output goes into - /// `buffer`, until it read [`SHARE`] bytes in the turn, or while the connection is - /// backlogged: an empty buffer then lets it read no output, and return none. Returns whether - /// it read its whole share, and so may have more output to read in the next turn. + /// messages for the host on `connection`, and notes whether its terminal outcome tells that + /// it settled. Each read of its program's output goes into `buffer`, until it read [`SHARE`] + /// bytes in the turn, or while the connection is backlogged: an empty buffer then lets it + /// read no output, and return none. Returns whether it read its whole share, and so may have + /// more output to read in the next turn. fn serve( &mut self, now: Instant, @@ -471,8 +679,10 @@ impl Run { }; match event { Event::Message(message) => { - if let runner::Message::Output { length: bytes, .. } = message { - read += bytes; + match message { + runner::Message::Output { length: bytes, .. } => read += bytes, + runner::Message::Terminal(outcome) => self.settled = outcome.is_reusable(), + _ => {} } if let Some(message) = GuestMessage::from_runner(key, message, lent) { connection.send(&message.encode())?; @@ -485,62 +695,114 @@ impl Run { } } -/// Returns whether the runner of `run` has not returned its terminal outcome, and otherwise -/// [forgets](forget) the execution. -fn running(run: &Run, reaping: &mut Vec) -> bool { - if run.binding.is_finished() { - forget(run, reaping); - return false; - } - true +/// Why the worker could not start a program. +struct Unstarted { + /// The error. + error: io::Error, + /// Whether the worker spawned the program before the error. + spawned: bool, } -/// Adds the process ID of the program of `run` to `reaping`, unless the worker collected the -/// program's status: the worker drops the execution next, whose binding kills the program and its -/// process group, which the program's ID still names, and the worker collects the program's -/// status once the program ended. -fn forget(run: &Run, reaping: &mut Vec) { - if run.binding.execution().status().is_none() { - reaping.push(run.binding.id()); +impl From for Unstarted { + /// Returns the failure of a program that the worker did not spawn. + fn from(error: io::Error) -> Self { + Self { + error, + spawned: false, + } } } -/// Spawns the program of `start` with pipes, in a process group of its own, and returns its -/// execution, which leaves the program's status to the worker, with the worker's own handle of -/// the program. If the program started, but the execution or the handle could not be had, the -/// program and its process group end, and the program's ID goes to `reaping`, so that the worker -/// collects its status once it ended, without waiting for it. -fn spawn(start: &Start, reaping: &mut Vec) -> io::Result<(Execution, Child)> { - let arguments = start - .arguments - .iter() - .map(|argument| CString::new(argument.as_slice())) - .collect::, _>>()?; +/// Spawns the program of `start` with pipes, in a process group of its own, and in `cgroup`, if +/// any, through `operations`, whose spawn enters the root as `prepared` prepared it, which makes +/// the program lead a session of its own, and returns its execution. The program starts in the +/// start's working directory, or the root's `/`, with exactly the start's environment. In a +/// cgroup, the execution settles through the cgroup, and collects the program's status itself. +/// Otherwise, it leaves the status to the worker, and comes with the worker's own handle of the +/// program. If the program started, but the execution or the handle could not be had, the +/// program ends, with its process group or the processes of its cgroup, and the program's ID goes +/// to `reaping`, so that the worker collects its status once it ended, without waiting for it; +/// the failure then tells that the worker spawned the program. +fn spawn( + start: &Start, + prepared: Prepared, + cgroup: Option<&ExecutionCgroup>, + operations: &mut dyn EntryOperations, + reaping: &mut Vec, +) -> Result<(Execution, Option), Unstarted> { + let strings = |items: &[Vec]| { + items + .iter() + .map(|item| CString::new(item.as_slice())) + .collect::, _>>() + .map_err(io::Error::from) + }; + let arguments = strings(&start.arguments)?; + let environment = strings(&start.environment)?; let program = arguments.first().cloned().unwrap_or_default(); let (pipes, ends) = Pipes::new(start.input)?; let stdin = match ends.stdin { Some(stdin) => stdin, None => OwnedFd::from(File::open(NULL_DEVICE)?), }; - let spawned = Command::new(program, arguments) + let mut command = Command::new(program, arguments) .stdin(stdin) .stdout(ends.stdout) .stderr(ends.stderr) .new_process_group() - .spawn()?; + .environment(environment); + if let Some(directory) = &start.working_directory { + let directory = CString::new(directory.as_slice()).map_err(io::Error::from)?; + command = command.working_directory(directory); + } + if let Some(cgroup) = cgroup { + command = command.cgroup(cgroup.placement()?); + } + let spawned = operations.spawn(command, prepared)?; let id = spawned.id(); - let options = Options::new().process_group().caller_reaps(); - let execution = Execution::new(spawned, pipes, options).inspect_err(|error| { - // The execution ended the program and its process group, and the program's status is - // still the worker's, unless `ESRCH` tells that it is gone already. - if error.raw_os_error() != Some(Errno::ESRCH as i32) { - reaping.push(id); - } - })?; + let options = match cgroup { + Some(cgroup) => Options::new().cgroup(cgroup, Scope::Execution), + None => Options::new().process_group().caller_reaps(), + }; + let unstarted = |error| Unstarted { + error, + spawned: true, + }; + let execution = Execution::new(spawned, pipes, options) + .inspect_err(|error| { + // The execution ended the program, and the program's status is still the worker's, + // unless `ESRCH` tells that it is gone already. + if error.raw_os_error() != Some(Errno::ESRCH as i32) { + reaping.push(id); + } + }) + .map_err(unstarted)?; + if cgroup.is_some() { + return Ok((execution, None)); + } // Nothing collected the program's status, so its ID still names it. Dropping the execution // after a failure ends the program and its process group. - let program = Child::with_pidfd(id).inspect_err(|_| reaping.push(id))?; - Ok((execution, program)) + let program = Child::with_pidfd(id) + .inspect_err(|_| reaping.push(id)) + .map_err(unstarted)?; + Ok((execution, Some(program))) +} + +/// Removes the cgroup `cgroup` of an execution that the worker dropped, and returns whether it is +/// gone, or false while a process, a zombie included, is left in it. A removal that fails +/// otherwise is reported, and given up. +fn removed(cgroup: &ExecutionCgroup) -> bool { + match cgroup.remove() { + Ok(()) => true, + Err(error) if error.raw_os_error() == Some(Errno::EBUSY as i32) => false, + Err(error) => { + eprintln!( + "runtime-adapter-guest: cannot remove {}: {error}", + cgroup.path().display() + ); + true + } + } } /// Refuses the execution of `runner` at `now`, whose launch failed at stage 0 with `error`: queues @@ -575,6 +837,42 @@ fn refused(key: Key, violation: Violation) -> ServeError { ServeError::Execution(format!("execution {}: {violation}", name(key))) } +/// Returns the `errno` with which the worker refuses a start whose entry `error` refused, at +/// stage 0: `EPERM` for an identity that the entry does not permit, with a capability outside +/// its allowlist, with any capability but user ID 0's, or with `CAP_NET_BIND_SERVICE` or +/// `CAP_NET_RAW` while listeners are published; `ENOENT` for an identity whose account the root +/// lacks in existing-account mode, without the user ID's passwd entry or the group ID's group +/// entry; the error's own for an account database or a root that cannot be read; and `EINVAL` +/// otherwise. +fn refusal(error: &entry::Error) -> Errno { + match error { + entry::Error::Capability(_) + | entry::Error::NonRootCapabilities + | entry::Error::Published(_) => Errno::EPERM, + entry::Error::Account(AccountError::Missing | AccountError::Group) => Errno::ENOENT, + entry::Error::Account(AccountError::Read(_, error)) | entry::Error::Root(error) => { + error.raw_os_error().map_or(Errno::EIO, Errno::from_raw) + } + _ => Errno::EINVAL, + } +} + +/// Returns the identity of `nvx::guest::entry` that a start's `identity` names. +fn entry_identity(identity: &Identity) -> entry::Identity { + let capabilities = (0..64) + .filter(|number| identity.capabilities & 1 << number != 0) + .filter_map(Capability::new) + .collect::(); + let account = match identity.account { + AccountMode::Existing => entry::AccountMode::Existing, + AccountMode::Numeric => entry::AccountMode::Numeric, + }; + entry::Identity::new(identity.uid, identity.gid) + .groups(identity.groups.clone()) + .capabilities(capabilities) + .account(account) +} + /// Reports that the worker cannot collect the status of the program of `binding`'s execution, /// with `error`: the worker no longer waits for the program, and the runner's deadlines bound /// the execution. diff --git a/tests/runtime-adapter/src/worker/executions/tests.rs b/tests/runtime-adapter/src/worker/executions/tests.rs index 85f7a97ac..bc861bf36 100644 --- a/tests/runtime-adapter/src/worker/executions/tests.rs +++ b/tests/runtime-adapter/src/worker/executions/tests.rs @@ -9,17 +9,44 @@ use std::time::{Duration, Instant}; use nix::poll::{self, PollFd, PollFlags, PollTimeout}; use nvx::exec::{Cause, Key, Limits, Outcome, Status, Timers, Window}; +use nvx::guest::entry::{self, Prepared}; use nvx::guest::process::Child; +use nvx::guest::spawn::{Command, Spawned}; use nvx::protocol::nvxs::{RecordParser, RecordType, encode_record}; use super::Executions; -use crate::exec::{GuestMessage, HostMessage, Start}; +use crate::exec::{AccountMode, GuestMessage, HostMessage, Identity, Start}; use crate::frame::{Decoder, Message}; use crate::worker::connection::{Connection, Received}; +use crate::worker::{Entrance, EntryOperations}; /// Longest wait for an execution's terminal outcome. const TIMEOUT: Duration = Duration::from_secs(30); +/// Entry operations that run each program without entering a root, which needs privileges: they +/// prepare each entry into the test's own root, `/`, whose account databases they read in +/// existing-account mode alone, which the [identities](numeric) of these tests do not use. +struct Unentered; + +impl EntryOperations for Unentered { + fn prepare(&mut self, entrance: &Entrance) -> Result { + entrance.entry("/").prepare() + } + + fn spawn(&mut self, command: Command, _prepared: Prepared) -> io::Result { + command.spawn() + } +} + +/// Returns the identity of user ID 0 and group ID 0, with numeric IDs, which the executions of +/// these tests name, as the worker runs them without a workload identity. +fn numeric() -> Identity { + Identity { + account: AccountMode::Numeric, + ..Identity::new(0, 0) + } +} + /// The broker's end of the worker's stream, which takes the worker's records as they arrive, and /// decodes the messages of their DATA. struct Broker { @@ -121,8 +148,11 @@ fn a_cancellation_after_the_timeout_leaves_a_program_that_ended_before_it_its_ow let timers = Timers::new(Duration::from_millis(300), Duration::from_secs(5)).timeout(timeout); let arguments = [&b"/bin/sh"[..], b"-c", b"exit 0"]; let started = Instant::now(); - let start = HostMessage::Start(Start::new(key, limits, timers, arguments)); - executions.deliver(started, start, &mut connection).unwrap(); + let start = Start::new(key, limits, timers, arguments).with_identity(numeric()); + let start = HostMessage::Start(start); + executions + .deliver(started, start, &mut Unentered, &mut connection) + .unwrap(); { let program = executions.attached[&1].program.as_ref(); let pidfd = program.and_then(Child::pidfd).unwrap(); @@ -141,7 +171,7 @@ fn a_cancellation_after_the_timeout_leaves_a_program_that_ended_before_it_its_ow let cancel = HostMessage::Cancel(key, Cause::Cancelled); executions - .deliver(Instant::now(), cancel, &mut connection) + .deliver(Instant::now(), cancel, &mut Unentered, &mut connection) .unwrap(); let outcome = terminal(&mut executions, &mut connection, &mut broker); assert_eq!(outcome.status(), Some(Status::Exited(0))); diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 8c01d6cc4..556bd7fb7 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -10,28 +10,38 @@ //! the guest agent on a tty, which it puts in raw mode, as it serves a VM's control tty. //! //! The worker of a layered root runs its session with fake root operations, which record the -//! worker's calls and fail as a test asks: the tests cover the order of its stages, the host -//! export that it mounts, the mapping table that it applies in order, in parts, which outlives a -//! reset, a table of zero mappings, which does no work, the parts that it refuses, a mapping that -//! fails and latches the root, the start of the sandbox, and the executions and starts that it -//! refuses out of order, each stage that fails, the requests that it serves once the root is -//! ready, its stop, the messages that it refuses, and a broker that closes only its end of the -//! stream, which still receives the worker's replies. The binary's PID 1 of a layered root, and -//! its worker, run on a pseudo-terminal too, on the test's host, whose kernel command line names -//! no layered root, where PID 1 also collects the statuses that real-time signals leave, whatever -//! the names of its children. +//! worker's calls and fail as a test asks: the tests cover the order of its stages, the host export +//! that it mounts, the mapping table that it applies in order, in parts, which outlives a reset, a +//! table of zero mappings, which does no work, the parts that it refuses, a mapping that fails and +//! latches the root, the start of the sandbox, and the executions and starts that it refuses out of +//! order, each stage that fails, the account check of a sandbox's workload identity, the requests +//! that it serves once the root is ready, its stop, the messages that it refuses, and a broker that +//! closes only its end of the stream, which still receives the worker's replies. The binary's PID 1 +//! of a layered root, and its worker, run on a pseudo-terminal too, on the test's host, whose +//! kernel command line names no layered root, where PID 1 also collects the statuses that real-time +//! signals leave, whatever the names of its children. //! //! The worker's executions run behind the test broker too, whose host side drives the engine's //! peers: the tests cover several executions at once, byte-exact, a stalled sink and a full //! stream to the host, which stall only what waits on them, executions that take turns at the //! stream, timeouts, cancellations, and stops, the starts, messages, and probes that the worker //! refuses, generations and resets, the descendants that a program leaves in its process group, -//! and executions on a tty. +//! and executions on a tty. The fake root prepares each execution's entry, as the worker +//! describes it, in a directory of the tests' own that holds account databases, and then spawns +//! the program without entering a root, which needs privileges, in the tests' own root and +//! identity: the tests cover the identity that exactly one source names, the sandbox's or the +//! start's, the entry's inputs, a program's working directory and environment, and the starts +//! that the worker refuses for their identities. Where the tests can create cgroups, each fake +//! root gives the worker a workload cgroup of the test's own, below which the executions run in +//! cgroups of their own, and settle: the tests then also cover a descendant that leaves its +//! program's process group, which the settlement ends, and an execution that does not settle, +//! which keeps its place until the host stops the worker. Elsewhere, the programs run in process +//! groups alone. #![cfg(all(feature = "guest", target_os = "linux"))] use std::collections::BTreeMap; use std::fs::{self, File, OpenOptions}; -use std::io::{self, Read, Write}; +use std::io::{self, BufRead, BufReader, Read, Write}; use std::net::Shutdown; use std::os::fd::OwnedFd; use std::os::unix::fs::OpenOptionsExt; @@ -39,9 +49,9 @@ use std::os::unix::net::UnixStream; use std::os::unix::process::{CommandExt, ExitStatusExt}; use std::path::{Path, PathBuf}; use std::process::{Child, Command, Output, Stdio}; -use std::sync::Arc; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::mpsc::{self, Receiver, Sender}; +use std::sync::{Arc, OnceLock}; use std::thread::{self, JoinHandle}; use std::time::{Duration, Instant}; @@ -54,8 +64,11 @@ use nix::unistd::Pid; use nvx::exec::{ Cause, Key, LaunchFailure, Limits, Outcome, Peer, Pool, Status, Stream, Timers, Window, peer, }; +use nvx::guest::cgroup::Layout; +use nvx::guest::entry::{self, Capabilities, Capability, Prepared}; use nvx::guest::mappings::{Access, Mapping}; use nvx::guest::nvxs::GuestConnectionError; +use nvx::guest::spawn::{self, Spawned}; use nvx::protocol::export::{EXPORT_DIRECTORY, ExportMode, HostExport}; use nvx::protocol::nvxs::{ ErrorCode, Identity, Record, RecordError, RecordParser, RecordType, SessionError, encode_record, @@ -66,14 +79,16 @@ use nvx::protocol::{ }; use nvx::uuid::Uuid; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::exec::{GuestMessage, HostMessage, MAX_CHUNK, Start}; +use runtime_adapter::exec::{self, AccountMode, GuestMessage, HostMessage, MAX_CHUNK, Start}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, Message}; use runtime_adapter::guest::{self, NO_ROOT, RECEIVE_WINDOW, ServeError, Served}; use runtime_adapter::probe::{Facts, Probe}; use runtime_adapter::root::{ JournalPolicy, MapFailure, OverlayMode, Ready, RootConfig, RootFailure, RootStage, encode_maps, }; -use runtime_adapter::worker::{self, MAX_EXECUTIONS, MapError, RootOperations, Worked}; +use runtime_adapter::worker::{ + self, Entrance, EntryOperations, MAX_EXECUTIONS, MapError, RootOperations, Worked, +}; /// Identity of the broker's first epoch. const FIRST: Identity = Identity { @@ -1114,6 +1129,8 @@ enum Call { MountExport(HostExport), /// The mappings were applied, in order. Map(Vec), + /// The entry of an execution, or of the sandbox's workload identity, was prepared. + Prepare(Entrance), /// The probe ran. Probe(Probe), /// The teardown ran. @@ -1140,6 +1157,9 @@ struct FakeRoot { gate: Option>, /// The worker's calls, in order. calls: Vec, + /// The workload cgroup of the test's own, below which the worker runs its executions in + /// cgroups of their own, where the test can create cgroups. + workload: Option, } impl FakeRoot { @@ -1162,8 +1182,11 @@ impl FakeRoot { Self::reading(command_line.as_bytes()) } - /// Returns operations whose kernel command line is `command_line`. + /// Returns operations whose kernel command line is `command_line`, with a workload cgroup of + /// the test's own where the test can create cgroups. fn reading(command_line: &[u8]) -> Self { + let workload = + cgroups().then(|| Workload::new().expect("the test creates a workload cgroup")); Self { command_line: command_line.to_vec(), failing: None, @@ -1172,6 +1195,7 @@ impl FakeRoot { assembly: Duration::ZERO, gate: None, calls: Vec::new(), + workload, } } @@ -1263,6 +1287,10 @@ impl RootOperations for FakeRoot { } } + fn cgroups(&self) -> Option { + self.workload.as_ref().map(|workload| workload.0.clone()) + } + fn probe(&mut self, probe: &Probe) -> Result { self.calls.push(Call::Probe(probe.clone())); if let Some(gate) = &self.gate { @@ -1279,6 +1307,59 @@ impl RootOperations for FakeRoot { } } +impl EntryOperations for FakeRoot { + /// Prepares the entry into the [accounts' root](accounts), whose account databases it reads. + fn prepare(&mut self, entrance: &Entrance) -> Result { + self.calls.push(Call::Prepare(entrance.clone())); + entrance.entry(accounts()).prepare() + } + + /// Spawns `command` without entering a root, which needs privileges: the program runs in the + /// test's own root, as the test's own identity, in the process group that it leads. + fn spawn(&mut self, command: spawn::Command, _prepared: Prepared) -> io::Result { + command.spawn() + } +} + +/// The user and group IDs of the workload's account in the [accounts' root](accounts). +const WORKLOAD: (u32, u32) = (1000, 1000); + +/// The user and group IDs of an account in the [accounts' root](accounts) whose primary group +/// has no group entry. +const ORPHAN: (u32, u32) = (2000, 2000); + +/// The `/etc/passwd` of the [accounts' root](accounts). +const PASSWD: &str = "root:x:0:0:root:/root:/bin/sh\n\ + workload:x:1000:1000:workload:/home/workload:/bin/sh\n\ + orphan:x:2000:2000:orphan:/home/orphan:/bin/sh\n"; + +/// The `/etc/group` of the [accounts' root](accounts). +const GROUP: &str = "root:x:0:\nstaff:x:50:\nworkload:x:1000:\n"; + +/// Returns the directory that stands for the assembled root of every fake root when the worker +/// prepares an entry: one of the tests' own, below the target directory, for the tests' user, +/// with the account databases [`PASSWD`] and [`GROUP`], and nothing else. +fn accounts() -> &'static Path { + static ROOT: OnceLock = OnceLock::new(); + ROOT.get_or_init(|| { + let uid = nix::unistd::getuid(); + let root = Path::new(env!("CARGO_TARGET_TMPDIR")).join(format!("accounts-{uid}")); + let etc = root.join("etc"); + fs::create_dir_all(&etc).unwrap(); + for (name, contents) in [("passwd", PASSWD), ("group", GROUP)] { + let path = etc.join(name); + if fs::read_to_string(&path).is_ok_and(|current| current == contents) { + continue; + } + // Another test process may read the database meanwhile, so it is replaced whole. + let staged = etc.join(format!("{name}.{}", std::process::id())); + fs::write(&staged, contents).unwrap(); + fs::rename(&staged, &path).unwrap(); + } + root + }) +} + /// Returns the facts that a [`FakeRoot`] finds with `probe`. fn found(probe: &Probe) -> Facts { let mut facts = Facts::new(); @@ -1287,6 +1368,66 @@ fn found(probe: &Probe) -> Facts { facts } +/// Returns whether the tests can create cgroups, as where they run as root in a cgroup v2 +/// hierarchy with the `pids` controller. Each fake root then has a [`Workload`] cgroup of its +/// own, below which the worker runs its executions in cgroups, and otherwise none, so that the +/// worker runs each program in a process group alone. +fn cgroups() -> bool { + static CGROUPS: OnceLock = OnceLock::new(); + *CGROUPS.get_or_init(|| Workload::new().is_some()) +} + +/// A workload cgroup of a test's own, in the real hierarchy, which it removes, with whatever is +/// left in it, when it drops. +#[derive(Debug)] +struct Workload(Layout); + +impl Workload { + /// Creates a new workload cgroup below the root of the hierarchy at `/sys/fs/cgroup`, with the + /// test's own cgroup as the agent's, or returns `None` where the test cannot create one, or + /// the hierarchy lacks the `pids` controller, which the cgroups of executions need. + fn new() -> Option { + static CREATED: AtomicU64 = AtomicU64::new(0); + let root = Path::new("/sys/fs/cgroup"); + let controllers = fs::read_to_string(root.join("cgroup.controllers")).ok()?; + if !controllers + .split_whitespace() + .any(|controller| controller == "pids") + { + return None; + } + let own = fs::read_to_string("/proc/self/cgroup").ok()?; + let own = own.lines().find_map(|line| line.strip_prefix("0::"))?; + let agent = root.join(own.trim_start_matches('/')); + let number = CREATED.fetch_add(1, Ordering::Relaxed); + let name = format!("runtime-adapter-test-{}-{number}", std::process::id()); + let layout = Layout::new(agent, root.join(name)).ok()?; + fs::create_dir(layout.workload()).ok()?; + Some(Self(layout)) + } +} + +impl Drop for Workload { + /// Ends whatever is left in the workload cgroup, waits until it is empty, then removes the + /// cgroups of the executions, and the workload cgroup. + fn drop(&mut self) { + let workload = self.0.workload(); + let _ = fs::write(workload.join("cgroup.kill"), b"1"); + let deadline = Instant::now() + TIMEOUT; + while fs::read(workload.join("pids.current")).is_ok_and(|count| count != b"0\n") + && Instant::now() < deadline + { + thread::sleep(Duration::from_millis(10)); + } + for entry in fs::read_dir(workload).into_iter().flatten().flatten() { + if entry.file_type().is_ok_and(|kind| kind.is_dir()) { + let _ = fs::remove_dir(entry.path()); + } + } + let _ = fs::remove_dir(workload); + } +} + /// Returns a layout that the worker assembles: an EROFS image, with the identity that the guest /// expects of it, over an ext4 scratch. fn layout() -> RootLayout { @@ -1349,6 +1490,15 @@ fn announcing(count: u32) -> RootConfig { } } +/// Returns the configuration of a sandbox whose executions run as the workload identity +/// [`WORKLOAD`], whose account the [accounts' root](accounts) holds. +fn sandbox() -> RootConfig { + RootConfig { + workload_identity: Some(WORKLOAD), + ..RootConfig::default() + } +} + /// Returns a mapping table of `count` mappings: the export's entry `NNN` at `/srv/table/NNN`, for /// each index NNN, read-only for the even ones. fn table(count: usize) -> Vec { @@ -1525,6 +1675,10 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { // The boot tokens of the layout that the worker assembles, after which a command line may name // more. let tokens = format!("console=hvc0 nvx_sandbox=1 {}", layout().tokens().join(" ")); + let lacking = RootConfig { + workload_identity: Some((4242, 4242)), + ..RootConfig::default() + }; let cases = [ ( "a configuration of another version", @@ -1668,6 +1822,20 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { "failed to mount the host export hostfs", exported(&config), ), + ( + "a workload identity whose account the root lacks", + FakeRoot::booting(&layout()), + lacking.encode(), + RootStage::Identity, + "cannot enter the root as the workload identity 4242:4242: the account is \ + unavailable: no passwd entry has the user ID", + vec![ + Call::CommandLine, + Call::Bootstrap(lacking.clone()), + Call::Assemble(lacking.clone(), layout()), + Call::Prepare(entrance(entry::Identity::new(4242, 4242))), + ], + ), ]; for (name, root, body, stage, problem, calls) in cases { let (mut broker, working) = start_worker(root); @@ -1689,6 +1857,47 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { } } +/// Returns the entrance of an execution as `identity`, with the default hostname. +fn entrance(identity: entry::Identity) -> Entrance { + Entrance { + identity, + hostname: entry::DEFAULT_HOSTNAME.as_bytes().to_vec(), + } +} + +/// Returns the entrances that the worker prepared through `root`, in order. +fn entrances(root: &FakeRoot) -> Vec { + root.calls + .iter() + .filter_map(|call| match call { + Call::Prepare(entrance) => Some(entrance.clone()), + _ => None, + }) + .collect() +} + +#[test] +fn the_worker_checks_the_account_of_the_sandbox_s_workload_identity_in_the_assembled_root() { + // The check's entry takes the default hostname, as every execution's does. + let (mut broker, working) = start_worker(FakeRoot::booting(&layout())); + broker.attach(FIRST); + let config = sandbox(); + let messages = [Message::RootConfig(config.encode()), Message::Stop]; + let replies = broker.exchange(&messages, OUTER_MAX_PAYLOAD, 2); + assert_eq!(replies, [ready(), Message::Stopped]); + let (root, worked) = joined(working); + assert_eq!(worked.unwrap(), Worked::Stopped); + let checked = entrance(entry::Identity::new(WORKLOAD.0, WORKLOAD.1)); + let expected = [ + Call::CommandLine, + Call::Bootstrap(config.clone()), + Call::Assemble(config, layout()), + Call::Prepare(checked), + Call::Teardown, + ]; + assert_eq!(root.calls, expected); +} + #[test] fn the_worker_reports_a_teardown_that_failed_before_it_confirms_the_stop() { let problem = "failed to unmount the overlay: Device or resource busy (os error 16)"; @@ -2598,17 +2807,25 @@ struct Host { impl Host { /// Starts a worker whose root operations succeed, and returns the host, once the worker - /// assembled its root. + /// assembled its root for a [sandbox](sandbox) whose executions run as its workload identity. fn new() -> Self { - let (broker, working) = start_worker(FakeRoot::booting(&layout())); - Self::ready(broker, working) + Self::configured(FakeRoot::booting(&layout()), &sandbox()) + } + + /// Starts a worker with the root operations `root`, and returns the host, once the worker + /// assembled its root as `config` configures it. + fn configured(root: FakeRoot, config: &RootConfig) -> Self { + let (broker, working) = start_worker(root); + Self::ready(broker, working, config) } /// Attaches the session of the worker that runs in `working` in the first epoch, has it - /// assemble its root, starts the sandbox, of no mapping, and returns the host. - fn ready(mut broker: Broker, working: Working) -> Self { + /// assemble its root as `config` configures it, starts the sandbox, of no mapping, and returns + /// the host. + fn ready(mut broker: Broker, working: Working, config: &RootConfig) -> Self { broker.attach(FIRST); - let replies = broker.exchange(&[configuration(), Message::Start], OUTER_MAX_PAYLOAD, 2); + let messages = [Message::RootConfig(config.encode()), Message::Start]; + let replies = broker.exchange(&messages, OUTER_MAX_PAYLOAD, 2); assert_eq!(replies, [ready(), Message::Started]); Self { broker, @@ -2731,8 +2948,9 @@ impl Drop for Host { } } -/// Checks that the execution `id` ended on its own with `status`, losing no byte, and fenced, as -/// nothing proves that the descendants of a program without a cgroup are gone. +/// Checks that the execution `id` ended on its own with `status`, losing no byte, and settled if +/// its program ran in a cgroup of its own, where the tests can create cgroups: otherwise it is +/// fenced, as nothing proves that the descendants of a program without a cgroup are gone. fn check_lossless(host: &Host, id: u64, status: Status) { let outcome = host.outcome(id).unwrap(); assert_eq!(outcome.status(), Some(status), "{id}"); @@ -2741,7 +2959,7 @@ fn check_lossless(host: &Host, id: u64, status: Status) { for stream in Stream::ALL { assert!(!outcome.is_truncated(stream), "{id}: {stream:?}"); } - assert!(!outcome.is_reusable(), "{id}"); + assert_eq!(outcome.is_reusable(), cgroups(), "{id}"); let peer = &host.executions[&id].peer; assert!(peer.high_water() <= peer.memory_bound(), "{id}"); } @@ -2835,7 +3053,7 @@ fn the_worker_refuses_probes_while_it_runs_executions() { // runs: a probe that ran then would hold back the execution's timeout. let (root, gate) = FakeRoot::booting(&layout()).gated(); let (broker, working) = start_worker(root); - let mut host = Host::ready(broker, working); + let mut host = Host::ready(broker, working, &sandbox()); let timeout = Duration::from_millis(500); let timed = Start::new( Key::new(1, 1), @@ -3168,11 +3386,342 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { host.run_until(|host| host.executions[&2].output[0].ends_with(b"\n")); let second = host.executions[&2].group(); reaped(second); - let (_, worked) = host.stop(); + // The fake root lives until the check, as its workload cgroup, if any, ends what is left in it + // once it drops. + let (_root, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); gone(second); } +/// Returns the program's process ID, or that of another process, from the line `line` of the +/// standard output of the execution `id`, and the directory of the program's cgroup from the next +/// line, which `/proc/self/cgroup` gave. +fn process_and_cgroup(host: &Host, id: u64, line: usize) -> (u32, PathBuf) { + let output = String::from_utf8(host.executions[&id].output[0].clone()).unwrap(); + let mut lines = output.lines().skip(line); + let process = lines.next().unwrap().parse().unwrap(); + let cgroup = lines.next().unwrap().strip_prefix("0::/").unwrap(); + (process, Path::new("/sys/fs/cgroup").join(cgroup)) +} + +#[test] +fn an_execution_s_settlement_ends_a_descendant_that_left_its_process_group() { + // Only an execution's cgroup reaches a descendant in a session of its own. + if !cgroups() { + return; + } + // The program starts a descendant in a session of its own, which holds the program's output + // open once it wrote its process ID, and prints the descendant's process ID and its own + // cgroup, then ends. The execution's settlement kills the descendant, so that the output ends + // at once, losing nothing, and the execution settles. + let mut host = Host::new(); + let script = "exec 3>&1; \ + descendant=$(setsid sh -c 'echo $$; exec sleep 60 >&3 3>&-' &); \ + echo \"$descendant\"; cat /proc/self/cgroup; exit 3"; + let started = Instant::now(); + host.start(start_of(1, shell(script)), Vec::new()); + host.run_until(|host| host.outcome(1).is_some()); + assert!(started.elapsed() < SETTLEMENT, "{:?}", started.elapsed()); + check_lossless(&host, 1, Status::Exited(3)); + let (descendant, cgroup) = process_and_cgroup(&host, 1, 0); + reaped(descendant); + // The worker removed the execution's cgroup before it sent the outcome, which freed the + // execution's place: it runs its limit of executions at once again. + assert!(!cgroup.exists(), "{}", cgroup.display()); + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + for id in 2..=most + 1 { + host.start(start_of(id, shell("exec sleep 60")), Vec::new()); + } + host.run_until(|host| (2..=most + 1).all(|id| host.executions[&id].peer.has_started())); + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); +} + +/// A process with a child that it never waits for, so that the child, once it ended, stays a +/// zombie, and occupies its cgroup, until the process ends. Both run in a process group that the +/// process leads, which the test kills when it drops, so that neither outlives a test that fails; +/// the system's collector of orphans then collects the child's status. +struct Holder { + /// The process. + process: Child, + /// The process ID of its child. + child: u32, +} + +impl Holder { + /// Starts the process and its child. + fn new() -> Self { + let mut process = Command::new("/bin/sh") + .args(["-c", "sleep 600 & echo $!; exec sleep 600"]) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .process_group(0) + .spawn() + .unwrap(); + let mut line = String::new(); + BufReader::new(process.stdout.take().unwrap()) + .read_line(&mut line) + .unwrap(); + let child = line.trim().parse().unwrap(); + Self { process, child } + } +} + +impl Drop for Holder { + fn drop(&mut self) { + // Until its status is collected, the process keeps its ID, which is the group's. + let group = Pid::from_raw(i32::try_from(self.process.id()).unwrap()); + let _ = signal::killpg(group, Signal::SIGKILL); + let _ = self.process.wait(); + } +} + +#[test] +fn an_execution_that_does_not_settle_keeps_its_place_until_the_host_stops_the_worker() { + // A zombie whose parent is outside the cgroup keeps a cgroup occupied, which needs cgroups. + if !cgroups() { + return; + } + // The program prints its process ID and its cgroup, and waits until the test signals it, + // once the test moved a holder's child into the cgroup. The settlement kills the child, which + // stays a zombie, so that the execution does not settle within its settlement time. + let holder = Holder::new(); + let mut host = Host::new(); + let waiting = "trap 'exit 0' USR1; echo $$; cat /proc/self/cgroup; \ + while :; do sleep 0.01; done"; + let settlement = Duration::from_secs(1); + let start = Start::new( + Key::new(1, 1), + shared_limits(), + Timers::new(GRACE, settlement), + shell(waiting), + ); + host.start(start, Vec::new()); + host.run_until(|host| { + let output = &host.executions[&1].output[0]; + output.iter().filter(|&&byte| byte == b'\n').count() == 2 + }); + let (program, cgroup) = process_and_cgroup(&host, 1, 0); + fs::write(cgroup.join("cgroup.procs"), holder.child.to_string()).unwrap(); + let program = Pid::from_raw(i32::try_from(program).unwrap()); + signal::kill(program, Signal::SIGUSR1).unwrap(); + host.run_until(|host| host.outcome(1).is_some()); + let outcome = host.outcome(1).unwrap(); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + for stream in Stream::ALL { + assert!(!outcome.is_truncated(stream), "{stream:?}"); + } + assert!(!outcome.is_reusable()); + assert!(cgroup.exists(), "{}", cgroup.display()); + + // The execution keeps its place: the worker runs one execution fewer than its limit, and + // refuses the next start. It still does once the cgroup is empty, after the holder ended. + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + fill_but_one(&mut host, 2); + drop(holder); + let deadline = Instant::now() + TIMEOUT; + while fs::read(cgroup.join("pids.current")).unwrap() != b"0\n" { + assert!(Instant::now() < deadline, "the cgroup stays occupied"); + thread::sleep(Duration::from_millis(10)); + } + fill_but_one(&mut host, 2 + most); + + // The stop ends the executions, and removes every cgroup, the fenced one's included. The fake + // root lives until the check, as its workload cgroup removes what is left in it once it drops. + let (_root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + assert!(!cgroup.exists(), "{}", cgroup.display()); +} + +/// Starts one execution fewer than the worker's limit, from the ID `first` on, while another +/// execution keeps its place: checks that they run, and that the worker refuses the next start, +/// then cancels them, and checks that they settled. +fn fill_but_one(host: &mut Host, first: u64) { + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + let ids = first..first + most - 1; + for id in ids.clone() { + host.start(start_of(id, shell("exec sleep 60")), Vec::new()); + } + host.run_until(|host| { + ids.clone() + .all(|id| host.executions[&id].peer.has_started()) + }); + let refused = ids.end; + host.start(start_of(refused, shell("exit 0")), Vec::new()); + host.run_until(|host| host.outcome(refused).is_some()); + let again = LaunchFailure::new(0, Errno::EAGAIN as i32); + assert_eq!(host.outcome(refused).unwrap().launch_failure(), Some(again)); + let now = Instant::now(); + for id in ids.clone() { + host.execution(id).peer.cancel(now); + } + host.run_until(|host| ids.clone().all(|id| host.outcome(id).is_some())); + for id in ids { + let outcome = host.outcome(id).unwrap(); + assert_eq!(outcome.cause(), Some(Cause::Cancelled), "{id}"); + assert!(outcome.is_reusable(), "{id}"); + } +} + +#[test] +fn an_execution_runs_in_its_working_directory_with_exactly_its_environment() { + // In a sandbox with a workload identity, the worker prepares each execution's entry as it. + // Without privileges, the program runs in the test's own root, so that its working directory + // is one of the test's host. The first program prints its environment, which none of the + // worker's own joins, and the second its working directory. + let mut host = Host::new(); + let environment = ["GREETING=hello world", "EMPTY=", "TWICE=a=b"]; + let directory = fs::canonicalize(accounts()).unwrap(); + let directory = directory.to_str().unwrap(); + let printing = start_of(1, vec![b"/usr/bin/env".to_vec()]).with_environment(environment); + host.start(printing, Vec::new()); + let moved = start_of(2, shell("pwd -P")).with_working_directory(directory); + host.start(moved, Vec::new()); + host.run_until(|host| (1..=2).all(|id| host.outcome(id).is_some())); + for id in 1..=2 { + check_lossless(&host, id, Status::Exited(0)); + } + let printed = environment.map(|entry| format!("{entry}\n")).concat(); + assert_eq!(host.executions[&1].output[0], printed.as_bytes()); + assert_eq!( + host.executions[&2].output[0], + format!("{directory}\n").as_bytes() + ); + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + // The configuration's check, then each execution. + let workload = entrance(entry::Identity::new(WORKLOAD.0, WORKLOAD.1)); + assert_eq!( + entrances(&root), + [workload.clone(), workload.clone(), workload] + ); +} + +#[test] +fn an_execution_enters_the_root_as_the_identity_that_its_start_names() { + // In a sandbox without a workload identity, each start names an identity, with its + // supplementary groups, its capabilities, and its account mode, which the worker prepares an + // entry with, and the default hostname. + let mut host = Host::configured(FakeRoot::booting(&layout()), &RootConfig::default()); + let bit = |capability: Capability| 1 << capability.number(); + let identities = [ + exec::Identity { + groups: vec![50], + capabilities: bit(Capability::CHOWN) | bit(Capability::KILL), + ..exec::Identity::new(0, 0) + }, + exec::Identity { + groups: vec![50, WORKLOAD.1], + ..exec::Identity::new(WORKLOAD.0, WORKLOAD.1) + }, + exec::Identity { + account: AccountMode::Numeric, + ..exec::Identity::new(4242, 4243) + }, + ]; + for (id, identity) in (1..).zip(&identities) { + let start = start_of(id, shell("exit 0")).with_identity(identity.clone()); + host.start(start, Vec::new()); + } + host.run_until(|host| (1..=3).all(|id| host.outcome(id).is_some())); + for id in 1..=3 { + check_lossless(&host, id, Status::Exited(0)); + } + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + let capabilities = Capabilities::EMPTY + .with(Capability::CHOWN) + .with(Capability::KILL); + let expected = [ + entry::Identity::new(0, 0) + .groups([50]) + .capabilities(capabilities), + entry::Identity::new(WORKLOAD.0, WORKLOAD.1).groups([50, WORKLOAD.1]), + entry::Identity::new(4242, 4243).account(entry::AccountMode::Numeric), + ] + .map(entrance); + assert_eq!(entrances(&root), expected); +} + +#[test] +fn the_worker_refuses_at_stage_0_a_start_whose_identity_breaks_the_rules() { + // Each refusal's terminal record tells its `errno`, and the refusal takes no place. + fn refused(host: &mut Host, cases: Vec<(Option, Errno)>) { + let ids = 1..=u64::try_from(cases.len()).unwrap(); + for (id, (identity, _)) in ids.clone().zip(&cases) { + let mut start = start_of(id, shell("exit 0")); + start.identity = identity.clone(); + host.start(start, Vec::new()); + } + host.run_until(|host| ids.clone().all(|id| host.outcome(id).is_some())); + for (id, (identity, error)) in ids.zip(&cases) { + let failure = LaunchFailure::new(0, *error as i32); + let outcome = host.outcome(id).unwrap(); + assert_eq!(outcome.launch_failure(), Some(failure), "{identity:?}"); + } + } + + // In a sandbox with a workload identity, a start names none. + let mut host = Host::new(); + let workload = exec::Identity::new(WORKLOAD.0, WORKLOAD.1); + refused(&mut host, vec![(Some(workload), Errno::EINVAL)]); + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + + // In a sandbox without one, a start names its own, which the entry checks: its + // capabilities, its IDs, and its account in existing-account mode. + let mut host = Host::configured(FakeRoot::booting(&layout()), &RootConfig::default()); + let root = |capability: Capability| exec::Identity { + capabilities: 1 << capability.number(), + ..exec::Identity::new(0, 0) + }; + let cases = vec![ + (None, Errno::EINVAL), + ( + Some(exec::Identity { + capabilities: 1 << 21, + ..exec::Identity::new(0, 0) + }), + Errno::EPERM, + ), + ( + Some(exec::Identity { + capabilities: 1 << Capability::CHOWN.number(), + ..exec::Identity::new(WORKLOAD.0, WORKLOAD.1) + }), + Errno::EPERM, + ), + (Some(root(Capability::NET_BIND_SERVICE)), Errno::EPERM), + (Some(root(Capability::NET_RAW)), Errno::EPERM), + (Some(exec::Identity::new(4242, 4242)), Errno::ENOENT), + (Some(exec::Identity::new(ORPHAN.0, ORPHAN.1)), Errno::ENOENT), + (Some(exec::Identity::new(WORKLOAD.0, 50)), Errno::EINVAL), + ( + Some(exec::Identity { + account: AccountMode::Numeric, + ..exec::Identity::new(u32::MAX, 0) + }), + Errno::EINVAL, + ), + ]; + let first = u64::try_from(cases.len()).unwrap() + 1; + refused(&mut host, cases); + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + let ids = first..first + most; + for id in ids.clone() { + let start = start_of(id, shell("exec sleep 60")) + .with_identity(exec::Identity::new(WORKLOAD.0, WORKLOAD.1)); + host.start(start, Vec::new()); + } + host.run_until(|host| { + ids.clone() + .all(|id| host.executions[&id].peer.has_started()) + }); + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); +} + #[test] fn the_worker_collects_the_status_of_a_program_that_a_real_time_signal_ended() { // `nix` decodes no real-time signal in a status, so that the worker learns that the program @@ -3210,7 +3759,7 @@ fn an_execution_that_starts_with_the_root_counts_its_timeout_from_its_own_start( stopping: false, }; host.send(&[ - configuration(), + Message::RootConfig(sandbox().encode()), Message::Start, HostMessage::Start(start).encode(), ]); @@ -3310,7 +3859,7 @@ fn the_worker_runs_executions_on_a_tty() { let worked = worker::work_on_terminal(&path, &mut root); (root, worked) }); - let mut host = Host::ready(broker, working); + let mut host = Host::ready(broker, working, &sandbox()); let script = "seq 1 100000; seq 1 1000 >&2"; host.start(start_of(1, shell(script)), Vec::new()); host.run_until(|host| host.outcome(1).is_some());