From e83119bf8da1f0fa73abf29e6a394c9e1d17db1b Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 00:22:50 -0700 Subject: [PATCH 1/6] tests: settle the runtime adapter's executions through cgroups The worker ran each program in a process group of its own, which it killed before it collected the program's status. Nothing proved that a descendant that left the group was gone, so every outcome was fenced, and the worker could admit a new execution while such a descendant still ran. Each execution now gets a cgroup of its own below the workload cgroup, which the worker prepares with ExecutionCgroup::prepare, and which the program joins as it spawns. The execution settles through that cgroup alone, so the other executions keep running, and an execution whose program ends on its own is reusable. An execution keeps its place among the worker's limit until its cgroup is empty and removed. One that does not settle keeps its place, and its cgroup, until the host stops the worker, which ends and removes every cgroup that is left. The root's operations name the cgroups that the bootstrap prepared. A fake root names none where the in-process tests cannot create cgroups, and the programs keep their process groups there, as before. Where the tests can, each fake root has a workload cgroup of its own, and new tests check a descendant in a session of its own, which the settlement ends, and an execution that does not settle, which keeps its place until the stop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 95 ++++-- tests/runtime-adapter/src/worker.rs | 86 +++-- .../runtime-adapter/src/worker/executions.rs | 320 +++++++++++++----- tests/runtime-adapter/tests/guest_session.rs | 259 +++++++++++++- 4 files changed, 615 insertions(+), 145 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 1fd5644c8..fb1173945 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -522,28 +522,48 @@ credit that it needs, never keeps the others' output from the host. runs until it ends, detached from the host, which receives nothing more about it. - The worker admits at most `worker::MAX_EXECUTIONS`, 8, executions at once, - abandoned ones included, and those whose programs' statuses it has not - collected, and none that would hold more than + abandoned ones included, and those that keep their places once they ended, + as the items below describe, and none that would hold more than `worker::MAX_EXECUTION_MEMORY`, 1 MiB: its runner's buffers, a chunk of each output stream and the input's window. It refuses a start beyond those, or - one whose program it cannot spawn, with a terminal record whose launch failed - at stage 0, with `EAGAIN`, `ENOMEM`, or the spawn's `errno`. + one whose cgroup it cannot prepare, or whose program it cannot spawn, with a + terminal record whose launch failed at stage 0, with `EAGAIN`, `ENOMEM`, or + the `errno` of the preparation or the spawn. - Each program runs in the worker's own root and working directories, with its environment, not in the layered root, with its standard input from the host - or the null device, and leads a process group of its own, which its - execution kills as a whole. -- The worker collects the status of every program itself, without waiting for - it, and learns that a program ended from a pidfd of its own. Before it - collects the status, it kills what is left of the program's process group, - as NVX's settlement through a cgroup kills what is left in the cgroup: until - then, the program, a zombie, keeps its process ID, which is the group's, from - any other process. So a descendant that stayed in the group neither outlives - the program nor holds its output open. The program runs without a cgroup, - though, so nothing proves that a descendant that left the group is gone, and - each outcome is fenced: once the program ended, a stream that such a process - may still hold has the settlement time and the grace to close, after which - the runner gives up on it, while a stream that no process holds any more - loses nothing, however slowly the host takes it. + or the null device, and leads a process group of its own. +- Each execution runs in a cgroup of its own below the workload cgroup of the + root configuration, `execution-N`, where N counts the cgroups that the worker + prepared. The worker prepares it with + `nvx::guest::cgroup::ExecutionCgroup::prepare`, and the program joins it as + it spawns, before it runs, so that its descendants start in it, however they + leave its process group or session. The execution settles through that + cgroup alone, which leaves the other executions running: once the program + ended, as its status, which the execution collects itself, tells, the + execution kills what is left in the cgroup, and settles once the cgroup is + empty. So no descendant outlives the program or holds its output open, and + an execution whose program ends on its own loses nothing, however slowly the + host takes its output, and is reusable. +- An execution keeps its place among `worker::MAX_EXECUTIONS` until its cgroup + is empty and the worker removed it, which the worker does once the runner + returned the terminal outcome of an execution that settled. One that did not + settle, whose outcome is fenced, keeps its place and its cgroup until the + host stops the worker. +- Without cgroups, as where the in-process tests cannot create them, the + root's operations name none, and the program's process group stands in for + its cgroup: its execution kills the group as a whole. The worker collects the + status of every program itself then, without waiting for it, and learns that + a program ended from a pidfd of its own. Before it collects the status, it + kills what is left of the program's process group, as the settlement through + a cgroup kills what is left in the cgroup: until then, the program, a + zombie, keeps its process ID, which is the group's, from any other process. + So a descendant that stayed in the group neither outlives the program nor + holds its output open. Nothing proves that a descendant that left the group + is gone, though, so each outcome is fenced: once the program ended, a stream + that such a process may still hold has the settlement time and the grace to + close, after which the runner gives up on it, while a stream that no process + holds any more loses nothing, however slowly the host takes it. Such an + execution keeps its place until the worker collected its program's status. - Before it hands an execution the host's cancellation, the worker gives the execution a turn, as the loop does, which collects the status of a program that ended, and reports the program's end to the runner. So the end of a @@ -554,11 +574,13 @@ credit that it needs, never keeps the others' output from the host. - The worker forgets an execution once its runner returned the terminal outcome, which may come before the program's end, as once the runner gave up on a program that a kill did not end in time: forgetting it kills the - program and its process group again, and the worker collects the program's - status once it ended. On a stop, or once its session ended otherwise, the - worker drops every execution, which kills the process group of each program - whose status it has not collected, and waits up to a second in all for those - programs to end. + program, and what is left in its cgroup or its process group, again, and the + worker collects the program's status once it ended. On a stop, or once its + session ended otherwise, the worker drops every execution, which kills the + program of each one that has not ended, with what is left in its cgroup or + its process group, and what is left in the cgroups of the executions that did + not settle, and waits up to a second in all for those programs to end and + those cgroups to empty, removing each cgroup that emptied. The worker's `entries` probe runs two more commands of the guest agent: `runtime-adapter-guest entries ROOT CGROUP`, its helper, which runs the probe's @@ -1094,8 +1116,14 @@ the binary exited, so that none of them outlives the test or holds its output open, which tests check too. `tests/guest_session.rs` also runs the worker's executions behind its test -broker, whose host side starts the sandbox, then drives the engine's peers, -with frame-counted credit and one output pool: +broker, whose host side starts the sandbox, then drives the engine's peers, with +frame-counted credit and one output pool. Where the tests can create cgroups, as +root in a cgroup v2 hierarchy with the `pids` controller, the fake root gives +the worker a workload cgroup of the test's own, below which each execution runs +in a cgroup of its own, so that an execution whose program ends on its own +settles, and is reusable. Elsewhere, the fake root names no cgroups, and each +program runs in a process group alone, so that such an execution is fenced. The +tests cover: - three executions at once, each byte-exact: standard output beyond a MiB, 1.5 MiB of input through `cat`, and both streams of a program that fails; @@ -1125,9 +1153,20 @@ with frame-counted credit and one output pool: statuses, and a program whose status the worker collects once it ended, after its execution's runner gave up on it; - programs that end while a descendant holds their output: the worker ends - the descendant, so that the output ends at once, losing nothing, and a stop - once the worker collected such a program's status leaves no process of its - group running; + the descendant, through the execution's cgroup or the program's process + group, so that the output ends at once, losing nothing, and a stop once the + worker collected such a program's status leaves no process of its group + running; +- with cgroups, a descendant in a session of its own, which holds a program's + output once the program ended, and which the execution's settlement ends, so + that the output ends at once, losing nothing, and the execution settles; + the worker removed the execution's cgroup before it sent the outcome, and + runs its limit of executions at once again; +- with cgroups, an execution that does not settle within its settlement time, + because a zombie whose parent is outside its cgroup occupies the cgroup: it + keeps its place, so that the worker runs one execution fewer than its limit, + even once the cgroup emptied, until the host stops the worker, which then + removes the cgroup; - a program that a real-time signal ends, whose status the worker collects; - an execution whose start comes in the payload of the root's configuration and the sandbox's start, whose timeout counts from its own start, however diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index f6ecf540c..de7bedfc1 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -114,22 +114,40 @@ //! session abandons every execution: the worker drops it, and it runs until it ends, detached //! from the host, which receives nothing more about it. //! - The worker admits at most [`MAX_EXECUTIONS`] executions at once, abandoned ones included, -//! and those whose programs' statuses it has not collected, and none that would hold more than -//! [`MAX_EXECUTION_MEMORY`] bytes. It refuses a start beyond those, or one whose program it -//! cannot spawn, with a terminal record whose launch failed at stage 0, with `EAGAIN`, -//! `ENOMEM`, or the spawn's `errno`. +//! and those that keep their places once they ended, as the items below describe, and none +//! that would hold more than [`MAX_EXECUTION_MEMORY`] bytes. It refuses a start beyond those, +//! or one whose cgroup it cannot prepare, or whose program it cannot spawn, with a terminal +//! record whose launch failed at stage 0, with `EAGAIN`, `ENOMEM`, or the `errno` of the +//! preparation or the spawn. //! - Each program runs in the worker's root and working directories, with its environment, and -//! leads a process group of its own, which its execution kills as a whole. -//! - The worker collects the status of every program itself, without waiting for it, and learns -//! that a program ended from a pidfd of its own. Before it collects the status, it kills what -//! is left of the program's process group, as NVX's settlement through a cgroup kills what is -//! left in the cgroup: until then, the program, a zombie, keeps its process ID, which is the -//! group's, from any other process. So a descendant that stayed in the group neither outlives -//! the program nor holds its output open. The program runs without a cgroup, though, so nothing -//! proves that a descendant that left the group is gone, and each outcome is fenced: once the -//! program ended, a stream that such a process may still hold has the settlement time and the -//! grace to close, after which the runner gives up on it, while a stream that no process holds -//! any more loses nothing, however slowly the host takes it. +//! leads a process group of its own. +//! - Each execution runs in a cgroup of its own below the workload cgroup, `execution-N`, where N +//! counts the cgroups that the worker prepared. The worker prepares it with +//! `nvx::guest::cgroup::ExecutionCgroup::prepare`, and the program joins it as it spawns, +//! before it runs, so that its descendants start in it, however they leave its process group or +//! session. The execution settles through that cgroup alone, which leaves the other executions +//! running: once the program ended, as its status, which the execution collects itself, +//! tells, the execution kills what is left in the cgroup, and settles once the cgroup is empty. +//! So no descendant outlives the program or holds its output open, and an execution whose +//! program ends on its own loses nothing, however slowly the host takes its output, and is +//! reusable. +//! - An execution keeps its place among [`MAX_EXECUTIONS`] until its cgroup is empty and the +//! worker removed it, which the worker does once the runner returned the terminal outcome of an +//! execution that settled. One that did not settle, whose outcome is fenced, keeps its place +//! and its cgroup until the host stops the worker. +//! - Without cgroups, as where the in-process tests cannot create them, the [root's +//! operations](RootOperations::cgroups) name none, and the program's process group stands in +//! for its cgroup: its execution kills the group as a whole. The worker collects the status of +//! every program itself then, without waiting for it, and learns that a program ended from a +//! pidfd of its own. Before it collects the status, it kills what is left of the program's +//! process group, as the settlement through a cgroup kills what is left in the cgroup: until +//! then, the program, a zombie, keeps its process ID, which is the group's, from any other +//! process. So a descendant that stayed in the group neither outlives the program nor holds its +//! output open. Nothing proves that a descendant that left the group is gone, though, so each +//! outcome is fenced: once the program ended, a stream that such a process may still hold has +//! the settlement time and the grace to close, after which the runner gives up on it, while a +//! stream that no process holds any more loses nothing, however slowly the host takes it. Such +//! an execution keeps its place until the worker collected its program's status. //! - Before it hands an execution the host's cancellation, the worker gives the execution a turn, //! as the loop does, which collects the status of a program that ended, and reports the //! program's end to the runner. So the end of a program that ended before its timeout while the @@ -138,11 +156,13 @@ //! timeout's instant. //! - The worker forgets an execution once its runner returned the terminal outcome, which may //! come before the program's end, as once the runner gave up on a program that a kill did not -//! end in time: forgetting it kills the program and its process group again, and the worker -//! collects the program's status once it ended. On a stop, or once its session ended -//! otherwise, the worker drops every execution, which kills the process group of each program -//! whose status it has not collected, and waits up to a second in all for those programs to -//! end. +//! end in time: forgetting it kills the program, and what is left in its cgroup or its process +//! group, again, and the worker collects the program's status once it ended. On a stop, or +//! once its session ended otherwise, the worker drops every execution, which kills the program +//! of each one that has not ended, with what is left in its cgroup or its process group, and +//! what is left in the cgroups of the executions that did not settle, and waits up to a second +//! in all for those programs to end and those cgroups to empty, removing each cgroup that +//! emptied. mod connection; pub mod entries; @@ -200,7 +220,9 @@ pub const SESSION_STATUS: u8 = 1; pub const EXECUTING: &str = "the worker runs executions, and probes nothing until they end"; /// The most executions that the worker runs at once, those that the host abandoned included, and -/// those whose programs the worker forgot before it collected their statuses. +/// those that ended but keep their places: with cgroups, those whose cgroups the worker has not +/// removed, as those of the executions that did not settle, and without them, those whose +/// programs the worker forgot before it collected their statuses. pub const MAX_EXECUTIONS: usize = 8; /// The most bytes that one execution may hold in the worker: what its runner holds, a chunk of @@ -242,6 +264,12 @@ pub trait RootOperations { /// Fails with a [`MapError`] that counts the mappings before the one that failed. fn map(&mut self, mappings: &[Mapping]) -> Result<(), MapError>; + /// Returns the cgroups in which the worker runs its executions once the root is ready: the + /// layout that the bootstrap prepared, below whose workload cgroup each execution gets a + /// cgroup of its own. Without them, as in tests that cannot create cgroups, each program runs + /// in a process group alone. + fn cgroups(&self) -> Option; + /// Runs `probe` on the assembled root, and returns what it found. fn probe(&mut self, probe: &Probe) -> Result; @@ -337,7 +365,7 @@ pub fn work( message, )?, None => { - let (step, ready) = configure_root(operations, message)?; + let (step, ready) = configure_root(operations, &mut executions, message)?; order = ready; step } @@ -436,15 +464,18 @@ fn wait( } /// Handles `message` before the root is ready: a root configuration, whose root it assembles -/// through `operations`, and which it answers with its readiness, or a `Stop`. Returns the step, -/// with the worker's order once the root is ready. +/// through `operations`, after which `executions` run in the cgroups that the operations prepared, +/// and which it answers with its readiness, or a `Stop`. Returns the step, with the worker's order +/// once the root is ready. fn configure_root( operations: &mut dyn RootOperations, + executions: &mut Executions, message: Message, ) -> Result<(Step, Option), ServeError> { match message { Message::RootConfig(body) => Ok(match assemble(operations, &body) { Ok(config) => { + executions.run_in(operations.cgroups()); let ready = Ready::new(GUEST_BUILD).expect("this crate's version is a build"); let step = Step::Continue(vec![Message::Ready(ready.encode())]); (step, Some(Order::new(config.mapping_count))) @@ -750,6 +781,13 @@ impl RootOperations for LibraryRoot { Ok(()) } + /// Returns the cgroups that the bootstrap prepared: those of the root configuration. + fn cgroups(&self) -> Option { + self.bootstrap + .as_ref() + .map(|bootstrap| bootstrap.layout().clone()) + } + /// Runs one of the [`probes`], of the [`mapping`] probes, which inspect the root's mappings /// and map from the host export, or of the [`entries`] probes, which enter it. fn probe(&mut self, probe: &Probe) -> Result { diff --git a/tests/runtime-adapter/src/worker/executions.rs b/tests/runtime-adapter/src/worker/executions.rs index dfae13982..627a31735 100644 --- a/tests/runtime-adapter/src/worker/executions.rs +++ b/tests/runtime-adapter/src/worker/executions.rs @@ -1,8 +1,10 @@ //! The worker's executions, as the worker's module documentation describes them: the table that //! reaches each one by its ID, the generations that the host started, their admission, the -//! refusal of a start, the loop's turns at every execution, the collection of each program's -//! status, before which the worker ends what is left of the program's process group, and that of -//! the statuses of the programs that the worker forgot. +//! refusal of a start, the loop's turns at every execution, the cgroup of each execution, through +//! which it settles, and which keeps its place until the worker removes it, and, without +//! cgroups, the collection of each program's status, before which the worker ends what is left +//! of the program's process group, and that of the statuses of the programs that the worker +//! forgot. use std::collections::{BTreeMap, VecDeque}; use std::ffi::CString; @@ -17,6 +19,7 @@ use nix::poll::{self, PollFd, PollFlags, PollTimeout}; use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, WaitPidFlag, WaitStatus}; use nvx::exec::{Key, LaunchFailure, Runner, Settlement, Stream, Violation, runner}; +use nvx::guest::cgroup::{ExecutionCgroup, Layout, Scope}; use nvx::guest::exec::binding::{Binding, Event}; use nvx::guest::exec::{Execution, Options, Pipes, Source}; use nvx::guest::process::Child; @@ -56,6 +59,15 @@ pub(super) struct Executions { /// When the worker stops waiting for the programs once it dropped every execution: a while /// after it first did. cleared: Option, + /// The cgroups in which the executions run, once the root is ready: the layout below whose + /// workload cgroup each execution gets a cgroup of its own. Without them, each program runs + /// in a process group alone. + cgroups: Option, + /// How many cgroups of executions the worker prepared, which numbers the next one. + prepared: u64, + /// The cgroups of the executions that did not settle, each of which keeps its place among + /// [`MAX_EXECUTIONS`] until the worker drops every execution. + fenced: Vec, /// The ID from which the next turn serves the executions that the host reaches, before /// those of lower IDs: the ID after that of the execution that the last turn served first, /// so that the executions take turns at being served first. @@ -73,18 +85,25 @@ pub(super) struct Executions { buffer: Vec, } -/// One of the worker's executions: the binding of its runner to its program, which leaves the -/// program's status to the worker, and the worker's own handle of the program. +/// One of the worker's executions: the binding of its runner to its program, and either the +/// execution's cgroup, through which it settles, or, without cgroups, the worker's own handle of +/// the program, whose status the binding leaves to the worker. struct Run { /// The binding of the execution's runner to its program. binding: Binding, - /// The program, through a pidfd of the worker's own, which polls readable once the program - /// ended, until the worker collected its status. + /// Without cgroups, the program, through a pidfd of the worker's own, which polls readable + /// once the program ended, until the worker collected its status. program: Option, + /// The execution's cgroup, if the worker runs it in one. + cgroup: Option, + /// Whether the runner's terminal outcome tells that the execution settled, so that its + /// cgroup is empty, and the worker may remove it. + settled: bool, } impl Executions { - /// Returns no executions. + /// Returns no executions, which run in process groups alone until + /// [`run_in`](Self::run_in) names their cgroups. pub(super) fn new() -> Self { Self { attached: BTreeMap::new(), @@ -92,6 +111,9 @@ impl Executions { reaping: Vec::new(), reaped: None, cleared: None, + cgroups: None, + prepared: 0, + fenced: Vec::new(), turn: 0, owed: None, generations: BTreeMap::new(), @@ -100,6 +122,12 @@ impl Executions { } } + /// Runs the executions that start from now on in `cgroups`: each in a cgroup of its own, + /// below the workload cgroup, or, without cgroups, in a process group alone. + pub(super) fn run_in(&mut self, cgroups: Option) { + self.cgroups = cgroups; + } + /// Hands `message` from the host to its execution at `now`, and queues on `connection` the /// terminal record of a start that the worker refuses. Before a cancellation, it gives the /// execution a turn at `now`, and queues its messages on `connection`: the end of a program @@ -142,8 +170,9 @@ impl Executions { /// Takes a turn at `now`: collects the status of each program that ended, takes every step /// that the executions can take, queues their messages for the host on `connection`, - /// forgets each execution once it returned its terminal outcome, and collects the statuses - /// of the programs that it forgot, of those that ended. + /// forgets each execution once it returned its terminal outcome, which removes the cgroup of + /// one that settled, and collects the statuses of the programs that it forgot, of those that + /// ended. /// /// The executions that the host reaches take turns at being served first: a turn starts with /// the execution after the one that the last turn started with, in the order of their IDs, @@ -162,7 +191,6 @@ impl Executions { let Self { attached, detached, - reaping, turn, owed, buffer, @@ -184,17 +212,32 @@ impl Executions { *turn = first.wrapping_add(1); } *owed = more.then_some(now); - attached.retain(|_, run| running(run, reaping)); for run in detached.iter_mut() { let key = run.binding.runner().key(); while let Some(event) = run.binding.next(now, buffer) { - // A detached runner returns only its terminal outcome, which no host takes. - if !matches!(event, Event::Message(_)) { - report(key, &event); + match event { + // A detached runner returns only its terminal outcome, which no host takes. + Event::Message(runner::Message::Terminal(outcome)) => { + run.settled = outcome.is_reusable(); + } + Event::Message(_) => {} + event => report(key, &event), } } } - detached.retain(|run| running(run, reaping)); + let finished = attached + .iter() + .filter(|(_, run)| run.binding.is_finished()) + .map(|(&id, _)| id) + .collect::>(); + let mut forgotten = finished + .into_iter() + .filter_map(|id| attached.remove(&id)) + .collect::>(); + forgotten.extend(detached.extract_if(.., |run| run.binding.is_finished())); + for run in forgotten { + self.forget(run); + } self.reap(now); Ok(()) } @@ -208,29 +251,41 @@ impl Executions { } /// Returns whether the worker runs no execution: none that the host reaches, and none that it - /// abandoned. It may still collect the statuses of programs that it forgot. + /// abandoned. It may still collect the statuses of programs that it forgot, and hold the + /// cgroups of executions that did not settle. pub(super) fn is_empty(&self) -> bool { self.attached.is_empty() && self.detached.is_empty() } - /// Drops every execution, which ends the program and its process group of each one whose - /// program's status the worker has not collected, and waits for those programs to end, so - /// that the worker collects their statuses, at most [`REAP_WAIT`] after it first dropped - /// every execution: a later call, as once the executions drop after a stop, waits no longer. - /// The worker ended the process groups of the others when it collected their programs' - /// statuses. + /// Drops every execution, which ends the program and its process group, or the processes of + /// its cgroup, of each one that has not ended, and waits until the programs ended and their + /// cgroups emptied, so that the worker collects the programs' statuses and removes the + /// cgroups, those of the executions that did not settle included, whose processes it ends + /// too, at most [`REAP_WAIT`] after it first dropped every execution: a later call, as once + /// the executions drop after a stop, waits no longer. Without cgroups, the worker ended the + /// process groups of the other programs when it collected their statuses. pub(super) fn clear(&mut self) { - let Self { - attached, - detached, - reaping, - .. - } = self; - for run in std::mem::take(attached) + let runs = std::mem::take(&mut self.attached) .into_values() - .chain(detached.drain(..)) - { - forget(&run, reaping); + .chain(self.detached.drain(..)) + .collect::>(); + for run in runs { + let Run { + binding, cgroup, .. + } = run; + if binding.execution().status().is_none() { + self.reaping.push(binding.id()); + } + self.fenced.extend(cgroup); + } + // An execution that settled gave its cgroup back, which its drop then leaves alone. + for cgroup in &self.fenced { + if let Err(error) = cgroup.kill(Scope::Execution) { + eprintln!( + "runtime-adapter-guest: cannot end the processes of {}: {error}", + cgroup.path().display() + ); + } } let deadline = *self .cleared @@ -238,7 +293,8 @@ impl Executions { loop { let now = Instant::now(); self.reap(now); - if self.reaping.is_empty() || now >= deadline { + self.fenced.retain(|cgroup| !removed(cgroup)); + if (self.reaping.is_empty() && self.fenced.is_empty()) || now >= deadline { return; } thread::sleep(REAP_INTERVAL); @@ -292,6 +348,51 @@ impl Executions { self.attached.values().chain(&self.detached) } + /// Returns how many of the places of [`MAX_EXECUTIONS`] the executions take: one for each + /// execution that the worker runs, and, with cgroups, one for each cgroup of an execution that + /// did not settle, or, without them, one for each program that the worker forgot before it + /// collected its status. + fn places(&self) -> usize { + let held = if self.cgroups.is_some() { + self.fenced.len() + } else { + self.reaping.len() + }; + self.attached.len() + self.detached.len() + held + } + + /// Forgets `run`, whose runner returned its terminal outcome. Unless the worker collected + /// the program's status, the program's ID goes to `reaping`, so that the worker collects the + /// status once the program ended. The worker removes the cgroup of an execution that settled, + /// which frees its place, and keeps that of one that did not, or that it cannot remove, which + /// keeps its place until the worker drops every execution. The binding drops last, which ends + /// a program that has not ended, and its process group, or the processes of its cgroup unless + /// the execution settled: one that settled gave its cgroup back, which its drop leaves alone. + fn forget(&mut self, run: Run) { + let Run { + binding, + cgroup, + settled, + .. + } = run; + if binding.execution().status().is_none() { + self.reaping.push(binding.id()); + } + let Some(cgroup) = cgroup else { + return; + }; + if settled { + match cgroup.remove() { + Ok(()) => return, + Err(error) => eprintln!( + "runtime-adapter-guest: execution {}: cannot remove its cgroup: {error}", + name(binding.runner().key()) + ), + } + } + self.fenced.push(cgroup); + } + /// Collects at `now` the statuses of the programs that the worker forgot, of those that /// ended. fn reap(&mut self, now: Instant) { @@ -339,25 +440,56 @@ impl Executions { self.detached.push(abandoned); } let runner = Runner::new(key, start.limits, start.timers, now); - let executions = self.attached.len() + self.detached.len() + self.reaping.len(); - let admitted = if executions >= MAX_EXECUTIONS { + let admitted = if self.places() >= MAX_EXECUTIONS { Err(Errno::EAGAIN as i32) } else if runner.memory_bound() > MAX_EXECUTION_MEMORY { Err(Errno::ENOMEM as i32) } else { - spawn(&start, &mut self.reaping).map_err(|error| error.raw_os_error().unwrap_or(0)) + self.launch(&start) + .map_err(|error| error.raw_os_error().unwrap_or(0)) }; match admitted { - Ok((execution, program)) => { + Ok((execution, program, cgroup)) => { let binding = Binding::new(execution, runner); - let program = Some(program); - self.attached.insert(key.id(), Run { binding, program }); + let run = Run { + binding, + program, + cgroup, + settled: false, + }; + self.attached.insert(key.id(), run); Ok(()) } Err(error) => refuse(now, runner, error, connection), } } + /// Starts the program of `start`: with cgroups, in a cgroup of its own that the worker + /// prepares below the workload cgroup, and otherwise with the worker's own handle of the + /// program. The worker removes the cgroup of a program that did not start, unless a program + /// that started holds it, as once its execution could not be had, which keeps its place. + fn launch( + &mut self, + start: &Start, + ) -> io::Result<(Execution, Option, Option)> { + let Some(cgroups) = &self.cgroups else { + let (execution, program) = spawn(start, None, &mut self.reaping)?; + return Ok((execution, program, None)); + }; + self.prepared += 1; + let name = format!("execution-{}", self.prepared); + let cgroup = ExecutionCgroup::prepare(cgroups, Some(&name), &[])?; + match spawn(start, Some(&cgroup), &mut self.reaping) { + Ok((execution, program)) => Ok((execution, program, Some(cgroup))), + Err(error) => { + if cgroup.remove().is_err() { + self.fenced.push(cgroup); + } + Err(error) + } + } + } + /// Remembers that the host started `key`, the latest generation of its ID, and forgets the /// ID that it heard of first once it remembers more than [`MAX_HISTORY`]. fn remember(&mut self, key: Key) { @@ -386,13 +518,16 @@ impl Drop for Executions { impl Run { /// Collects the program's status without blocking once the program ended, and hands it to - /// the binding. First, it kills what is left of the process group that the program leads, as - /// the binding's settlement through a cgroup kills what is left in the cgroup, so that no - /// descendant that stayed in the group outlives the program, or holds its output open: until - /// its status is collected, the program, a zombie, keeps its process ID, which is the - /// group's, from any other process, so that the kill reaches that group alone. + /// the binding, if the worker collects it: without cgroups. First, it kills what is left of + /// the process group that the program leads, as the binding's settlement through a cgroup + /// kills what is left in the cgroup, so that no descendant that stayed in the group outlives + /// the program, or holds its output open: until its status is collected, the program, a + /// zombie, keeps its process ID, which is the group's, from any other process, so that the + /// kill reaches that group alone. fn collect(&mut self) { - let Self { binding, program } = self; + let Self { + binding, program, .. + } = self; let Some(child) = program else { return; }; @@ -447,10 +582,11 @@ impl Run { } /// Serves the execution in a turn at `now`: takes every step that it can take, and queues its - /// messages for the host on `connection`. Each read of its program's output goes into - /// `buffer`, until it read [`SHARE`] bytes in the turn, or while the connection is - /// backlogged: an empty buffer then lets it read no output, and return none. Returns whether - /// it read its whole share, and so may have more output to read in the next turn. + /// messages for the host on `connection`, and notes whether its terminal outcome tells that + /// it settled. Each read of its program's output goes into `buffer`, until it read [`SHARE`] + /// bytes in the turn, or while the connection is backlogged: an empty buffer then lets it + /// read no output, and return none. Returns whether it read its whole share, and so may have + /// more output to read in the next turn. fn serve( &mut self, now: Instant, @@ -471,8 +607,10 @@ impl Run { }; match event { Event::Message(message) => { - if let runner::Message::Output { length: bytes, .. } = message { - read += bytes; + match message { + runner::Message::Output { length: bytes, .. } => read += bytes, + runner::Message::Terminal(outcome) => self.settled = outcome.is_reusable(), + _ => {} } if let Some(message) = GuestMessage::from_runner(key, message, lent) { connection.send(&message.encode())?; @@ -485,32 +623,18 @@ impl Run { } } -/// Returns whether the runner of `run` has not returned its terminal outcome, and otherwise -/// [forgets](forget) the execution. -fn running(run: &Run, reaping: &mut Vec) -> bool { - if run.binding.is_finished() { - forget(run, reaping); - return false; - } - true -} - -/// Adds the process ID of the program of `run` to `reaping`, unless the worker collected the -/// program's status: the worker drops the execution next, whose binding kills the program and its -/// process group, which the program's ID still names, and the worker collects the program's -/// status once the program ended. -fn forget(run: &Run, reaping: &mut Vec) { - if run.binding.execution().status().is_none() { - reaping.push(run.binding.id()); - } -} - -/// Spawns the program of `start` with pipes, in a process group of its own, and returns its -/// execution, which leaves the program's status to the worker, with the worker's own handle of -/// the program. If the program started, but the execution or the handle could not be had, the -/// program and its process group end, and the program's ID goes to `reaping`, so that the worker -/// collects its status once it ended, without waiting for it. -fn spawn(start: &Start, reaping: &mut Vec) -> io::Result<(Execution, Child)> { +/// Spawns the program of `start` with pipes, in a process group of its own, and in `cgroup`, if +/// any, and returns its execution. In a cgroup, the execution settles through the cgroup, and +/// collects the program's status itself. Otherwise, it leaves the status to the worker, and comes +/// with the worker's own handle of the program. If the program started, but the execution or the +/// handle could not be had, the program ends, with its process group or the processes of its +/// cgroup, and the program's ID goes to `reaping`, so that the worker collects its status once it +/// ended, without waiting for it. +fn spawn( + start: &Start, + cgroup: Option<&ExecutionCgroup>, + reaping: &mut Vec, +) -> io::Result<(Execution, Option)> { let arguments = start .arguments .iter() @@ -522,25 +646,51 @@ fn spawn(start: &Start, reaping: &mut Vec) -> io::Result<(Execution, Child) Some(stdin) => stdin, None => OwnedFd::from(File::open(NULL_DEVICE)?), }; - let spawned = Command::new(program, arguments) + let mut command = Command::new(program, arguments) .stdin(stdin) .stdout(ends.stdout) .stderr(ends.stderr) - .new_process_group() - .spawn()?; + .new_process_group(); + if let Some(cgroup) = cgroup { + command = command.cgroup(cgroup.placement()?); + } + let spawned = command.spawn()?; let id = spawned.id(); - let options = Options::new().process_group().caller_reaps(); + let options = match cgroup { + Some(cgroup) => Options::new().cgroup(cgroup, Scope::Execution), + None => Options::new().process_group().caller_reaps(), + }; let execution = Execution::new(spawned, pipes, options).inspect_err(|error| { - // The execution ended the program and its process group, and the program's status is - // still the worker's, unless `ESRCH` tells that it is gone already. + // The execution ended the program, and the program's status is still the worker's, + // unless `ESRCH` tells that it is gone already. if error.raw_os_error() != Some(Errno::ESRCH as i32) { reaping.push(id); } })?; + if cgroup.is_some() { + return Ok((execution, None)); + } // Nothing collected the program's status, so its ID still names it. Dropping the execution // after a failure ends the program and its process group. let program = Child::with_pidfd(id).inspect_err(|_| reaping.push(id))?; - Ok((execution, program)) + Ok((execution, Some(program))) +} + +/// Removes the cgroup `cgroup` of an execution that the worker dropped, and returns whether it is +/// gone, or false while a process, a zombie included, is left in it. A removal that fails +/// otherwise is reported, and given up. +fn removed(cgroup: &ExecutionCgroup) -> bool { + match cgroup.remove() { + Ok(()) => true, + Err(error) if error.raw_os_error() == Some(Errno::EBUSY as i32) => false, + Err(error) => { + eprintln!( + "runtime-adapter-guest: cannot remove {}: {error}", + cgroup.path().display() + ); + true + } + } } /// Refuses the execution of `runner` at `now`, whose launch failed at stage 0 with `error`: queues diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 8c01d6cc4..499ce58e4 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -26,12 +26,16 @@ //! stream to the host, which stall only what waits on them, executions that take turns at the //! stream, timeouts, cancellations, and stops, the starts, messages, and probes that the worker //! refuses, generations and resets, the descendants that a program leaves in its process group, -//! and executions on a tty. +//! and executions on a tty. Where the tests can create cgroups, each fake root gives the worker a +//! workload cgroup of the test's own, below which the executions run in cgroups of their own, and +//! settle: the tests then also cover a descendant that leaves its program's process group, which +//! the settlement ends, and an execution that does not settle, which keeps its place until the +//! host stops the worker. Elsewhere, the programs run in process groups alone. #![cfg(all(feature = "guest", target_os = "linux"))] use std::collections::BTreeMap; use std::fs::{self, File, OpenOptions}; -use std::io::{self, Read, Write}; +use std::io::{self, BufRead, BufReader, Read, Write}; use std::net::Shutdown; use std::os::fd::OwnedFd; use std::os::unix::fs::OpenOptionsExt; @@ -39,9 +43,9 @@ use std::os::unix::net::UnixStream; use std::os::unix::process::{CommandExt, ExitStatusExt}; use std::path::{Path, PathBuf}; use std::process::{Child, Command, Output, Stdio}; -use std::sync::Arc; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::mpsc::{self, Receiver, Sender}; +use std::sync::{Arc, OnceLock}; use std::thread::{self, JoinHandle}; use std::time::{Duration, Instant}; @@ -54,6 +58,7 @@ use nix::unistd::Pid; use nvx::exec::{ Cause, Key, LaunchFailure, Limits, Outcome, Peer, Pool, Status, Stream, Timers, Window, peer, }; +use nvx::guest::cgroup::Layout; use nvx::guest::mappings::{Access, Mapping}; use nvx::guest::nvxs::GuestConnectionError; use nvx::protocol::export::{EXPORT_DIRECTORY, ExportMode, HostExport}; @@ -1140,6 +1145,9 @@ struct FakeRoot { gate: Option>, /// The worker's calls, in order. calls: Vec, + /// The workload cgroup of the test's own, below which the worker runs its executions in + /// cgroups of their own, where the test can create cgroups. + workload: Option, } impl FakeRoot { @@ -1162,8 +1170,11 @@ impl FakeRoot { Self::reading(command_line.as_bytes()) } - /// Returns operations whose kernel command line is `command_line`. + /// Returns operations whose kernel command line is `command_line`, with a workload cgroup of + /// the test's own where the test can create cgroups. fn reading(command_line: &[u8]) -> Self { + let workload = + cgroups().then(|| Workload::new().expect("the test creates a workload cgroup")); Self { command_line: command_line.to_vec(), failing: None, @@ -1172,6 +1183,7 @@ impl FakeRoot { assembly: Duration::ZERO, gate: None, calls: Vec::new(), + workload, } } @@ -1263,6 +1275,10 @@ impl RootOperations for FakeRoot { } } + fn cgroups(&self) -> Option { + self.workload.as_ref().map(|workload| workload.0.clone()) + } + fn probe(&mut self, probe: &Probe) -> Result { self.calls.push(Call::Probe(probe.clone())); if let Some(gate) = &self.gate { @@ -1287,6 +1303,66 @@ fn found(probe: &Probe) -> Facts { facts } +/// Returns whether the tests can create cgroups, as where they run as root in a cgroup v2 +/// hierarchy with the `pids` controller. Each fake root then has a [`Workload`] cgroup of its +/// own, below which the worker runs its executions in cgroups, and otherwise none, so that the +/// worker runs each program in a process group alone. +fn cgroups() -> bool { + static CGROUPS: OnceLock = OnceLock::new(); + *CGROUPS.get_or_init(|| Workload::new().is_some()) +} + +/// A workload cgroup of a test's own, in the real hierarchy, which it removes, with whatever is +/// left in it, when it drops. +#[derive(Debug)] +struct Workload(Layout); + +impl Workload { + /// Creates a new workload cgroup below the root of the hierarchy at `/sys/fs/cgroup`, with the + /// test's own cgroup as the agent's, or returns `None` where the test cannot create one, or + /// the hierarchy lacks the `pids` controller, which the cgroups of executions need. + fn new() -> Option { + static CREATED: AtomicU64 = AtomicU64::new(0); + let root = Path::new("/sys/fs/cgroup"); + let controllers = fs::read_to_string(root.join("cgroup.controllers")).ok()?; + if !controllers + .split_whitespace() + .any(|controller| controller == "pids") + { + return None; + } + let own = fs::read_to_string("/proc/self/cgroup").ok()?; + let own = own.lines().find_map(|line| line.strip_prefix("0::"))?; + let agent = root.join(own.trim_start_matches('/')); + let number = CREATED.fetch_add(1, Ordering::Relaxed); + let name = format!("runtime-adapter-test-{}-{number}", std::process::id()); + let layout = Layout::new(agent, root.join(name)).ok()?; + fs::create_dir(layout.workload()).ok()?; + Some(Self(layout)) + } +} + +impl Drop for Workload { + /// Ends whatever is left in the workload cgroup, waits until it is empty, then removes the + /// cgroups of the executions, and the workload cgroup. + fn drop(&mut self) { + let workload = self.0.workload(); + let _ = fs::write(workload.join("cgroup.kill"), b"1"); + let deadline = Instant::now() + TIMEOUT; + while fs::read(workload.join("pids.current")).is_ok_and(|count| count != b"0\n") + && Instant::now() < deadline + { + thread::sleep(Duration::from_millis(10)); + } + for entry in fs::read_dir(workload).into_iter().flatten().flatten() { + if entry.file_type().is_ok_and(|kind| kind.is_dir()) { + let _ = fs::remove_dir(entry.path()); + } + } + let _ = fs::remove_dir(workload); + } +} + /// Returns a layout that the worker assembles: an EROFS image, with the identity that the guest /// expects of it, over an ext4 scratch. fn layout() -> RootLayout { @@ -2731,8 +2807,9 @@ impl Drop for Host { } } -/// Checks that the execution `id` ended on its own with `status`, losing no byte, and fenced, as -/// nothing proves that the descendants of a program without a cgroup are gone. +/// Checks that the execution `id` ended on its own with `status`, losing no byte, and settled if +/// its program ran in a cgroup of its own, where the tests can create cgroups: otherwise it is +/// fenced, as nothing proves that the descendants of a program without a cgroup are gone. fn check_lossless(host: &Host, id: u64, status: Status) { let outcome = host.outcome(id).unwrap(); assert_eq!(outcome.status(), Some(status), "{id}"); @@ -2741,7 +2818,7 @@ fn check_lossless(host: &Host, id: u64, status: Status) { for stream in Stream::ALL { assert!(!outcome.is_truncated(stream), "{id}: {stream:?}"); } - assert!(!outcome.is_reusable(), "{id}"); + assert_eq!(outcome.is_reusable(), cgroups(), "{id}"); let peer = &host.executions[&id].peer; assert!(peer.high_water() <= peer.memory_bound(), "{id}"); } @@ -3173,6 +3250,172 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { gone(second); } +/// Returns the program's process ID, or that of another process, from the line `line` of the +/// standard output of the execution `id`, and the directory of the program's cgroup from the next +/// line, which `/proc/self/cgroup` gave. +fn process_and_cgroup(host: &Host, id: u64, line: usize) -> (u32, PathBuf) { + let output = String::from_utf8(host.executions[&id].output[0].clone()).unwrap(); + let mut lines = output.lines().skip(line); + let process = lines.next().unwrap().parse().unwrap(); + let cgroup = lines.next().unwrap().strip_prefix("0::/").unwrap(); + (process, Path::new("/sys/fs/cgroup").join(cgroup)) +} + +#[test] +fn an_execution_s_settlement_ends_a_descendant_that_left_its_process_group() { + // Only an execution's cgroup reaches a descendant in a session of its own. + if !cgroups() { + return; + } + // The program starts a descendant in a session of its own, which holds the program's output + // open once it wrote its process ID, and prints the descendant's process ID and its own + // cgroup, then ends. The execution's settlement kills the descendant, so that the output ends + // at once, losing nothing, and the execution settles. + let mut host = Host::new(); + let script = "exec 3>&1; \ + descendant=$(setsid sh -c 'echo $$; exec sleep 60 >&3 3>&-' &); \ + echo \"$descendant\"; cat /proc/self/cgroup; exit 3"; + let started = Instant::now(); + host.start(start_of(1, shell(script)), Vec::new()); + host.run_until(|host| host.outcome(1).is_some()); + assert!(started.elapsed() < SETTLEMENT, "{:?}", started.elapsed()); + check_lossless(&host, 1, Status::Exited(3)); + let (descendant, cgroup) = process_and_cgroup(&host, 1, 0); + reaped(descendant); + // The worker removed the execution's cgroup before it sent the outcome, which freed the + // execution's place: it runs its limit of executions at once again. + assert!(!cgroup.exists(), "{}", cgroup.display()); + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + for id in 2..=most + 1 { + host.start(start_of(id, shell("exec sleep 60")), Vec::new()); + } + host.run_until(|host| (2..=most + 1).all(|id| host.executions[&id].peer.has_started())); + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); +} + +/// A process with a child that it never waits for, so that the child, once it ended, stays a +/// zombie, and occupies its cgroup, until the process ends. The test kills the process when it +/// drops, which leaves the child to the system's collector of orphans. +struct Holder { + /// The process. + process: Child, + /// The process ID of its child. + child: u32, +} + +impl Holder { + /// Starts the process and its child. + fn new() -> Self { + let mut process = Command::new("/bin/sh") + .args(["-c", "sleep 600 & echo $!; exec sleep 600"]) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .spawn() + .unwrap(); + let mut line = String::new(); + BufReader::new(process.stdout.take().unwrap()) + .read_line(&mut line) + .unwrap(); + let child = line.trim().parse().unwrap(); + Self { process, child } + } +} + +impl Drop for Holder { + fn drop(&mut self) { + let _ = self.process.kill(); + let _ = self.process.wait(); + } +} + +#[test] +fn an_execution_that_does_not_settle_keeps_its_place_until_the_host_stops_the_worker() { + // A zombie whose parent is outside the cgroup keeps a cgroup occupied, which needs cgroups. + if !cgroups() { + return; + } + // The program prints its process ID and its cgroup, and waits until the test signals it, + // once the test moved a holder's child into the cgroup. The settlement kills the child, which + // stays a zombie, so that the execution does not settle within its settlement time. + let holder = Holder::new(); + let mut host = Host::new(); + let waiting = "trap 'exit 0' USR1; echo $$; cat /proc/self/cgroup; \ + while :; do sleep 0.01; done"; + let settlement = Duration::from_secs(1); + let start = Start::new( + Key::new(1, 1), + shared_limits(), + Timers::new(GRACE, settlement), + shell(waiting), + ); + host.start(start, Vec::new()); + host.run_until(|host| { + let output = &host.executions[&1].output[0]; + output.iter().filter(|&&byte| byte == b'\n').count() == 2 + }); + let (program, cgroup) = process_and_cgroup(&host, 1, 0); + fs::write(cgroup.join("cgroup.procs"), holder.child.to_string()).unwrap(); + let program = Pid::from_raw(i32::try_from(program).unwrap()); + signal::kill(program, Signal::SIGUSR1).unwrap(); + host.run_until(|host| host.outcome(1).is_some()); + let outcome = host.outcome(1).unwrap(); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + for stream in Stream::ALL { + assert!(!outcome.is_truncated(stream), "{stream:?}"); + } + assert!(!outcome.is_reusable()); + assert!(cgroup.exists(), "{}", cgroup.display()); + + // The execution keeps its place: the worker runs one execution fewer than its limit, and + // refuses the next start. It still does once the cgroup is empty, after the holder ended. + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + fill_but_one(&mut host, 2); + drop(holder); + let deadline = Instant::now() + TIMEOUT; + while fs::read(cgroup.join("pids.current")).unwrap() != b"0\n" { + assert!(Instant::now() < deadline, "the cgroup stays occupied"); + thread::sleep(Duration::from_millis(10)); + } + fill_but_one(&mut host, 2 + most); + + // The stop ends the executions, and removes every cgroup, the fenced one's included. + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + assert!(!cgroup.exists(), "{}", cgroup.display()); +} + +/// Starts one execution fewer than the worker's limit, from the ID `first` on, while another +/// execution keeps its place: checks that they run, and that the worker refuses the next start, +/// then cancels them, and checks that they settled. +fn fill_but_one(host: &mut Host, first: u64) { + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + let ids = first..first + most - 1; + for id in ids.clone() { + host.start(start_of(id, shell("exec sleep 60")), Vec::new()); + } + host.run_until(|host| { + ids.clone() + .all(|id| host.executions[&id].peer.has_started()) + }); + let refused = ids.end; + host.start(start_of(refused, shell("exit 0")), Vec::new()); + host.run_until(|host| host.outcome(refused).is_some()); + let again = LaunchFailure::new(0, Errno::EAGAIN as i32); + assert_eq!(host.outcome(refused).unwrap().launch_failure(), Some(again)); + let now = Instant::now(); + for id in ids.clone() { + host.execution(id).peer.cancel(now); + } + host.run_until(|host| ids.clone().all(|id| host.outcome(id).is_some())); + for id in ids { + let outcome = host.outcome(id).unwrap(); + assert_eq!(outcome.cause(), Some(Cause::Cancelled), "{id}"); + assert!(outcome.is_reusable(), "{id}"); + } +} + #[test] fn the_worker_collects_the_status_of_a_program_that_a_real_time_signal_ended() { // `nix` decodes no real-time signal in a status, so that the worker learns that the program From ae5bee06162988a25bd8a9af257396cd424832ba Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 00:35:39 -0700 Subject: [PATCH 2/6] tests: keep an execution's cgroup until its program is collected A program that the worker spawned, but whose execution it could not have, may still be joining its cgroup, as its launch has yet to report, and removing the cgroup could then hide the program once it ended, which ExecutionCgroup::remove forbids. Such a start's cgroup now keeps its place, as that of an execution that did not settle, and a stop removes the executions' cgroups only once the worker collected the status of every program that it forgot. The holder in the test of an execution that does not settle now runs in a process group of its own, which its drop kills, so that its child no longer outlives a test that fails before the child joins the cgroup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 8 +- tests/runtime-adapter/src/worker.rs | 7 +- .../runtime-adapter/src/worker/executions.rs | 73 ++++++++++++++----- tests/runtime-adapter/tests/guest_session.rs | 10 ++- 4 files changed, 72 insertions(+), 26 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index fb1173945..6a495a07c 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -548,7 +548,10 @@ credit that it needs, never keeps the others' output from the host. is empty and the worker removed it, which the worker does once the runner returned the terminal outcome of an execution that settled. One that did not settle, whose outcome is fenced, keeps its place and its cgroup until the - host stops the worker. + host stops the worker, as does a start whose program the worker spawned, but + whose execution it could not have: until the worker collected the status of + such a program, the program may still join the cgroup, so that a removal + could hide it once it ended. - Without cgroups, as where the in-process tests cannot create them, the root's operations name none, and the program's process group stands in for its cgroup: its execution kills the group as a whole. The worker collects the @@ -580,7 +583,8 @@ credit that it needs, never keeps the others' output from the host. program of each one that has not ended, with what is left in its cgroup or its process group, and what is left in the cgroups of the executions that did not settle, and waits up to a second in all for those programs to end and - those cgroups to empty, removing each cgroup that emptied. + those cgroups to empty, removing each cgroup that emptied once it collected + the statuses of those programs. The worker's `entries` probe runs two more commands of the guest agent: `runtime-adapter-guest entries ROOT CGROUP`, its helper, which runs the probe's diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index de7bedfc1..dbde923e1 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -134,7 +134,10 @@ //! - An execution keeps its place among [`MAX_EXECUTIONS`] until its cgroup is empty and the //! worker removed it, which the worker does once the runner returned the terminal outcome of an //! execution that settled. One that did not settle, whose outcome is fenced, keeps its place -//! and its cgroup until the host stops the worker. +//! and its cgroup until the host stops the worker, as does a start whose program the worker +//! spawned, but whose execution it could not have: until the worker collected the status of +//! such a program, the program may still join the cgroup, so that a removal could hide it once +//! it ended. //! - Without cgroups, as where the in-process tests cannot create them, the [root's //! operations](RootOperations::cgroups) name none, and the program's process group stands in //! for its cgroup: its execution kills the group as a whole. The worker collects the status of @@ -162,7 +165,7 @@ //! of each one that has not ended, with what is left in its cgroup or its process group, and //! what is left in the cgroups of the executions that did not settle, and waits up to a second //! in all for those programs to end and those cgroups to empty, removing each cgroup that -//! emptied. +//! emptied once it collected the statuses of those programs. mod connection; pub mod entries; diff --git a/tests/runtime-adapter/src/worker/executions.rs b/tests/runtime-adapter/src/worker/executions.rs index 627a31735..63e14d080 100644 --- a/tests/runtime-adapter/src/worker/executions.rs +++ b/tests/runtime-adapter/src/worker/executions.rs @@ -262,8 +262,11 @@ impl Executions { /// cgroups emptied, so that the worker collects the programs' statuses and removes the /// cgroups, those of the executions that did not settle included, whose processes it ends /// too, at most [`REAP_WAIT`] after it first dropped every execution: a later call, as once - /// the executions drop after a stop, waits no longer. Without cgroups, the worker ended the - /// process groups of the other programs when it collected their statuses. + /// the executions drop after a stop, waits no longer. It removes the cgroups once it + /// collected the status of every program that it forgot: until then, such a program may + /// still join its cgroup, as its launch may not have reported, and a removal could hide it + /// once it ended. Without cgroups, the worker ended the process groups of the other programs + /// when it collected their statuses. pub(super) fn clear(&mut self) { let runs = std::mem::take(&mut self.attached) .into_values() @@ -293,7 +296,9 @@ impl Executions { loop { let now = Instant::now(); self.reap(now); - self.fenced.retain(|cgroup| !removed(cgroup)); + if self.reaping.is_empty() { + self.fenced.retain(|cgroup| !removed(cgroup)); + } if (self.reaping.is_empty() && self.fenced.is_empty()) || now >= deadline { return; } @@ -466,14 +471,17 @@ impl Executions { /// Starts the program of `start`: with cgroups, in a cgroup of its own that the worker /// prepares below the workload cgroup, and otherwise with the worker's own handle of the - /// program. The worker removes the cgroup of a program that did not start, unless a program - /// that started holds it, as once its execution could not be had, which keeps its place. + /// program. The worker removes the cgroup of a program that it could not spawn. One that it + /// spawned, but whose execution could not be had, may still join the cgroup until the worker + /// collected its status, so that a removal could hide it once it ended: its cgroup keeps its + /// place, as that of an execution that did not settle. fn launch( &mut self, start: &Start, ) -> io::Result<(Execution, Option, Option)> { let Some(cgroups) = &self.cgroups else { - let (execution, program) = spawn(start, None, &mut self.reaping)?; + let (execution, program) = + spawn(start, None, &mut self.reaping).map_err(|unstarted| unstarted.error)?; return Ok((execution, program, None)); }; self.prepared += 1; @@ -481,8 +489,8 @@ impl Executions { let cgroup = ExecutionCgroup::prepare(cgroups, Some(&name), &[])?; match spawn(start, Some(&cgroup), &mut self.reaping) { Ok((execution, program)) => Ok((execution, program, Some(cgroup))), - Err(error) => { - if cgroup.remove().is_err() { + Err(Unstarted { error, spawned }) => { + if spawned || cgroup.remove().is_err() { self.fenced.push(cgroup); } Err(error) @@ -623,23 +631,42 @@ impl Run { } } +/// Why the worker could not start a program. +struct Unstarted { + /// The error. + error: io::Error, + /// Whether the worker spawned the program before the error. + spawned: bool, +} + +impl From for Unstarted { + /// Returns the failure of a program that the worker did not spawn. + fn from(error: io::Error) -> Self { + Self { + error, + spawned: false, + } + } +} + /// Spawns the program of `start` with pipes, in a process group of its own, and in `cgroup`, if /// any, and returns its execution. In a cgroup, the execution settles through the cgroup, and /// collects the program's status itself. Otherwise, it leaves the status to the worker, and comes /// with the worker's own handle of the program. If the program started, but the execution or the /// handle could not be had, the program ends, with its process group or the processes of its /// cgroup, and the program's ID goes to `reaping`, so that the worker collects its status once it -/// ended, without waiting for it. +/// ended, without waiting for it; the failure then tells that the worker spawned the program. fn spawn( start: &Start, cgroup: Option<&ExecutionCgroup>, reaping: &mut Vec, -) -> io::Result<(Execution, Option)> { +) -> Result<(Execution, Option), Unstarted> { let arguments = start .arguments .iter() .map(|argument| CString::new(argument.as_slice())) - .collect::, _>>()?; + .collect::, _>>() + .map_err(io::Error::from)?; let program = arguments.first().cloned().unwrap_or_default(); let (pipes, ends) = Pipes::new(start.input)?; let stdin = match ends.stdin { @@ -660,19 +687,27 @@ fn spawn( Some(cgroup) => Options::new().cgroup(cgroup, Scope::Execution), None => Options::new().process_group().caller_reaps(), }; - let execution = Execution::new(spawned, pipes, options).inspect_err(|error| { - // The execution ended the program, and the program's status is still the worker's, - // unless `ESRCH` tells that it is gone already. - if error.raw_os_error() != Some(Errno::ESRCH as i32) { - reaping.push(id); - } - })?; + let unstarted = |error| Unstarted { + error, + spawned: true, + }; + let execution = Execution::new(spawned, pipes, options) + .inspect_err(|error| { + // The execution ended the program, and the program's status is still the worker's, + // unless `ESRCH` tells that it is gone already. + if error.raw_os_error() != Some(Errno::ESRCH as i32) { + reaping.push(id); + } + }) + .map_err(unstarted)?; if cgroup.is_some() { return Ok((execution, None)); } // Nothing collected the program's status, so its ID still names it. Dropping the execution // after a failure ends the program and its process group. - let program = Child::with_pidfd(id).inspect_err(|_| reaping.push(id))?; + let program = Child::with_pidfd(id) + .inspect_err(|_| reaping.push(id)) + .map_err(unstarted)?; Ok((execution, Some(program))) } diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 499ce58e4..4ebf4bc0f 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -3295,8 +3295,9 @@ fn an_execution_s_settlement_ends_a_descendant_that_left_its_process_group() { } /// A process with a child that it never waits for, so that the child, once it ended, stays a -/// zombie, and occupies its cgroup, until the process ends. The test kills the process when it -/// drops, which leaves the child to the system's collector of orphans. +/// zombie, and occupies its cgroup, until the process ends. Both run in a process group that the +/// process leads, which the test kills when it drops, so that neither outlives a test that fails; +/// the system's collector of orphans then collects the child's status. struct Holder { /// The process. process: Child, @@ -3311,6 +3312,7 @@ impl Holder { .args(["-c", "sleep 600 & echo $!; exec sleep 600"]) .stdin(Stdio::null()) .stdout(Stdio::piped()) + .process_group(0) .spawn() .unwrap(); let mut line = String::new(); @@ -3324,7 +3326,9 @@ impl Holder { impl Drop for Holder { fn drop(&mut self) { - let _ = self.process.kill(); + // Until its status is collected, the process keeps its ID, which is the group's. + let group = Pid::from_raw(i32::try_from(self.process.id()).unwrap()); + let _ = signal::killpg(group, Signal::SIGKILL); let _ = self.process.wait(); } } From 9539005ffa34a99baabb2f236e37749a2e353977 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 01:18:59 -0700 Subject: [PATCH 3/6] tests: keep the fake root until the checks that follow a stop Once the in-process tests create cgroups, each fake root holds a workload cgroup of the test's own, which ends what is left in it and removes the cgroups of executions when it drops. Two tests discarded the fake root that a stop returns, so that it dropped before their checks: that the stop removed the cgroup of an execution that did not settle, and that it left no process of a program's group running. Those checks then held whatever the worker did. The tests now keep the fake root until their checks are done. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/tests/guest_session.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 4ebf4bc0f..73ede8a60 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -3245,7 +3245,9 @@ fn the_worker_ends_what_is_left_of_the_process_group_of_a_program_that_ended() { host.run_until(|host| host.executions[&2].output[0].ends_with(b"\n")); let second = host.executions[&2].group(); reaped(second); - let (_, worked) = host.stop(); + // The fake root lives until the check, as its workload cgroup, if any, ends what is left in it + // once it drops. + let (_root, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); gone(second); } @@ -3384,8 +3386,9 @@ fn an_execution_that_does_not_settle_keeps_its_place_until_the_host_stops_the_wo } fill_but_one(&mut host, 2 + most); - // The stop ends the executions, and removes every cgroup, the fenced one's included. - let (_, worked) = host.stop(); + // The stop ends the executions, and removes every cgroup, the fenced one's included. The fake + // root lives until the check, as its workload cgroup removes what is left in it once it drops. + let (_root, worked) = host.stop(); assert_eq!(worked.unwrap(), Worked::Stopped); assert!(!cgroup.exists(), "{}", cgroup.display()); } From 80dbaf1dc072be8552e2b1e500e9017c76dd2078 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 01:06:05 -0700 Subject: [PATCH 4/6] tests: run the runtime adapter's executions in the root, under their identity The worker ran each program in its own root and working directory, as its own identity, with its own environment. A start now names the program's identity, working directory and environment, and the root configuration may name a workload identity for the whole sandbox, whose account the worker checks in the assembled root before it answers: a missing account fails the new identity stage, with status 16. Exactly one of them names each execution's identity, as the activation record's optional identity will, so a start that breaks that rule is refused at stage 0. Each program then enters the assembled root through nvx::guest::entry, in new namespaces, with the hostname that the kernel command line configures, as NVX's init agent reads it, the image's own machine ID, no time status, and published listeners. The entry checks the identity, whose refusals the worker reports at stage 0, and reads its account. The fake root of the in-process tests prepares each entry in a directory of the tests' own, with account databases, and spawns the program without entering it, which needs privileges. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 170 ++++++-- tests/runtime-adapter/src/exec.rs | 224 +++++++++- tests/runtime-adapter/src/exec/tests.rs | 130 +++++- tests/runtime-adapter/src/root.rs | 118 +++++- tests/runtime-adapter/src/worker.rs | 192 +++++++-- .../runtime-adapter/src/worker/executions.rs | 184 ++++++-- .../src/worker/executions/tests.rs | 38 +- tests/runtime-adapter/tests/guest_session.rs | 401 ++++++++++++++++-- 8 files changed, 1272 insertions(+), 185 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 6a495a07c..d78f0bca3 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -108,13 +108,19 @@ zeros: | `overlay-mode` | The overlay's mode: `trusted-lowers` or `untrusted-lowers` | no | | `scratch-journal` | Whether an ext4 scratch may have a journal: `allowed` or `forbidden` | no | | `mapping-count` | The number of mappings of the host's [mapping table](#ready-mapping-tables-and-start), at most 4,096 | no | +| `workload-uid` | The user ID of the sandbox's workload identity, as which every execution runs | with `workload-gid` | +| `workload-gid` | The group ID of the sandbox's workload identity | with `workload-uid` | The paths are absolute and canonical guest paths of at most 4,095 bytes, without a NUL or a newline: not `/`, and without an empty, `.`, or `..` component, or a trailing `/`. The runtime directory holds no `,`, `:`, `\`, or `=`, which the overlay's options reserve, and the two cgroups are siblings, -whose parent is not `/`. The optional numbers are positive. Without an -overlay mode or a journal policy, the guest takes the defaults of +whose parent is not `/`. The optional numbers are positive. A workload +identity has both IDs, each below 4,294,967,295, which the kernel reads as +"unchanged", and no supplementary groups or capabilities, as a layered +sandbox's identity does; the worker checks its account once it assembled the +root, and every execution then runs as it, naming no identity of its own. +Without an overlay mode or a journal policy, the guest takes the defaults of `nvx::guest::root`: an overlay that trusts its lowers, and a scratch that may have a journal. Without a bound of the first partition, the guest bounds it at `root::DEFAULT_FIRST_PARTITION_BOUND`, 64 KiB. Without a mapping count, the @@ -126,7 +132,8 @@ caller names others: the runtime directory `/run/runtime-adapter`, the cgroups `/sys/fs/cgroup/agent` and `/sys/fs/cgroup/workload`, a wait of 30 seconds, an agent's `memory.low` of 16 MiB, as NVX's own guest reserves, and a workload's `memory.max` of 128 MiB and `pids.max` of 128, with neither a bound of the -first partition, an overlay mode, nor a journal policy, and no mapping. +first partition, an overlay mode, a journal policy, nor a workload identity, and no +mapping. A stage of the root that fails is a `root::RootFailure`: the stage's code, one byte, then the problem, as UTF-8 text. The guest's worker, and with it the VM, @@ -135,10 +142,11 @@ then ends with the stage's status: | Code | Stage | What failed | Status | | --- | --- | --- | --- | | 1 | `configuration` | Receiving and checking the root configuration | 11 | -| 2 | `layout` | Reading the boot layout of the root and the host export from the kernel command line, and checking that the worker assembles them, with an export if the configuration announces mappings | 12 | +| 2 | `layout` | Reading the boot layout of the root, the executions' hostname, and the host export from the kernel command line, and checking that the worker assembles them, with an export if the configuration announces mappings | 12 | | 3 | `bootstrap` | Bootstrapping the guest for the root: its cgroups and their resource values | 13 | | 4 | `assembly` | Assembling the root, such as from an image of another identity than the boot tokens expect, and mounting the host export that the kernel command line names | 14 | | 5 | `teardown` | Tearing the root down, such as while a process holds it open | 15 | +| 6 | `identity` | Checking the account of the sandbox's workload identity in the assembled root, in existing-account mode, such as one that the image lacks | 16 | ### Ready, mapping tables, and Start @@ -272,11 +280,26 @@ the key and a stream; the input's window, then the output's, each a unit, 0 for bytes or 1 for frames, then its size, a `u32`; and whether the streams share one output pool, 0 or 1. It then holds the worker's timers, in milliseconds, each a `u64`: the grace, the settlement time, and whether a -timeout follows, 0 or 1, then the timeout, or 0. Last come whether the host +timeout follows, 0 or 1, then the timeout, or 0. Then come whether the host sends the program's standard input, 0 or 1, and the program's arguments: their count, a `u16`, then each argument's length, a `u32`, and its bytes. The first argument is the program's path, which is not searched for in `PATH`, and no -argument holds a NUL byte. +argument holds a NUL byte. The start ends with how the program enters the +worker's root: + +- whether it names the program's identity, `exec::Identity`, 0 or 1, and if + it does, the user ID and the group ID, each a `u32`, the account mode, + `exec::AccountMode`, 0 for an existing account or 1 for numeric IDs, the + capabilities, a `u64` whose bit N stands for the capability numbered N, then + the supplementary groups: their count, a `u16`, then each group, a `u32`. + Exactly one source names an execution's identity: its start, or the + sandbox's workload identity, which the root configuration names; +- whether it names the program's working directory, 0 or 1, and if it does, + the directory's length, a `u32`, and its bytes: an absolute path in the + root, without a NUL byte. Otherwise the program starts in the root's `/`; +- the program's whole environment: the count of its entries, a `u16`, then + each entry's length, a `u32`, and its bytes, `NAME=VALUE`, with a name that + is not empty, and without a NUL byte. An outcome holds the program's status: 0 for none, 1 for an exit, or 2 for a signal, then the exit's code or the signal's number, an `i32`, or 0; the cause, @@ -405,12 +428,15 @@ then: 1. attaches the session; 2. receives the host's root configuration, and checks it; 3. reads the boot layout of the root from the kernel command line, with - `nvx::protocol::root::RootLayout::parse`, and the host export, if the - command line names one, with `nvx::protocol::export::HostExport::parse`, - then bootstraps the guest's cgroups with `nvx::guest::root::bootstrap`, - which moves the guest's processes into the agent's cgroup, assembles the - root with `nvx::guest::root::assemble`, and mounts the export out of the - root with `Root::mount_export`; + `nvx::protocol::root::RootLayout::parse`, the executions' hostname, and the + host export, if the command line names one, with + `nvx::protocol::export::HostExport::parse`, then bootstraps the guest's + cgroups with `nvx::guest::root::bootstrap`, which moves the guest's processes + into the agent's cgroup, assembles the root with + `nvx::guest::root::assemble`, and mounts the export out of the root with + `Root::mount_export`. If the configuration names a workload identity, it then + checks the identity's account in the assembled root, in existing-account + mode, with `nvx::guest::entry`; 4. answers `Ready`, with the guest agent's build, or `RootFailed` with the stage that failed, and then ends; 5. applies the host's [mapping table](#the-mapping-table-and-the-start) @@ -430,22 +456,27 @@ Before the root configuration, the worker accepts only the configuration, or a layers of a GPT disk, whose first partition holds at most the bound of the configuration, over an ext4 scratch or a tmpfs in RAM, in the overlay mode of the configuration. At the layout stage, it refuses a configuration that -configures a scratch for an upper in RAM, bounds a first partition for an -EROFS image, or announces mappings while the kernel command line names no -host export, from which they come, and a command line whose tokens of the -export break their rules, as `HostExport::parse` checks them. An export that -fails to mount fails the assembly. It ends with 0 once the host stopped it, -after a teardown that succeeded, with the stage's status, from 11 to 15, once -a stage failed, and with 1 once its session failed or the broker closed the -tty, each but the first after a diagnostic on standard error. Once the broker -closed the stream at a record boundary, which may close only the broker's end, -the worker drops its executions, and writes what it queued, as once the host -stopped it. Before it exits, it closes what its probes hold and the host -export's root, then unmounts a root that it still holds, as -`nvx::guest::root::Root` does when it drops, ignoring failures. A teardown -that failed leaves the root mounted until the VM ends. The worker runs the -root's operations through `worker::RootOperations`, so that tests run it -against a fake root. +configures a scratch for an upper in RAM, bounds a first partition for an EROFS +image, or announces mappings while the kernel command line names no host export, +from which they come, and a command line whose tokens of the export break their +rules, as `HostExport::parse` checks them. An export that fails to mount fails +the assembly. The executions' hostname is the value of the kernel command line's +first `nvx_hostname=` parameter, which NVX's sandbox plans configure, and which +NVX's init agent reads too, or `nvx-sandbox`, +`nvx::guest::entry::DEFAULT_HOSTNAME`, without one, or with an empty one; a +hostname of more than 64 bytes fails the layout stage. A workload identity whose +account the assembled root lacks fails the identity stage. The worker ends with +0 once the host stopped it, after a teardown that succeeded, with the stage's +status, from 11 to 16, once a stage failed, and with 1 once its session failed +or the broker closed the tty, each but the first after a diagnostic on standard +error. Once the broker closed the stream at a record boundary, which may close +only the broker's end, the worker drops its executions, and writes what it +queued, as once the host stopped it. Before it exits, it closes what its probes +hold and the host export's root, then unmounts a root that it still holds, as +`nvx::guest::root::Root` does when it drops, ignoring failures. A teardown that +failed leaves the root mounted until the VM ends. The worker runs the root's +operations through `worker::RootOperations`, and enters its executions through +`worker::EntryOperations`, so that tests run it against a fake root. #### The mapping table and the start @@ -525,13 +556,34 @@ credit that it needs, never keeps the others' output from the host. abandoned ones included, and those that keep their places once they ended, as the items below describe, and none that would hold more than `worker::MAX_EXECUTION_MEMORY`, 1 MiB: its runner's buffers, a chunk of each - output stream and the input's window. It refuses a start beyond those, or - one whose cgroup it cannot prepare, or whose program it cannot spawn, with a - terminal record whose launch failed at stage 0, with `EAGAIN`, `ENOMEM`, or - the `errno` of the preparation or the spawn. -- Each program runs in the worker's own root and working directories, with its - environment, not in the layered root, with its standard input from the host - or the null device, and leads a process group of its own. + output stream and the input's window. It refuses a start beyond those, one + whose identity or entry it refuses, as the next item describes, or one whose + cgroup it cannot prepare, or whose program it cannot spawn, with a terminal + record whose launch failed at stage 0, with `EAGAIN`, `ENOMEM`, the `errno` + of the refusal, or that of the preparation or the spawn. +- Each program enters the layered root through `nvx::guest::entry`, as a + `worker::Entrance` describes it: in new mount, PID, and UTS namespaces, with + the root moved over `/`, the configured hostname, the image's own machine ID, + and no time status, so that the entry binds nothing into the root, and with + published listeners, the entry's default, until the adapter tells whether + its listeners are published. It runs as the identity that exactly one source + names: the sandbox's workload identity, with no supplementary groups or + capabilities, or, in a sandbox without one, the identity that its start + names, with its supplementary groups, its capabilities, and its account + mode. A start that names an identity in a sandbox with a workload identity, + or none in a sandbox without one, is refused with `EINVAL`. Before the worker + prepares the execution's cgroup, the entry checks the identity, and reads its + account in existing-account mode: it refuses with `EPERM` a capability + outside its allowlist, any capability of a user ID other than 0, and + `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while listeners are published, with + `ENOENT` an identity without the user ID's passwd entry or the group ID's + group entry, with the error of an account database or a root that it cannot + read, and with `EINVAL` otherwise. The program starts in the working + directory of its start, or in the root's `/`, with exactly the environment + of its start, none of the worker's own, and with its standard input from the + host or the null device. It leads a session of its own, without a + controlling terminal, and the process group that the session creates, and, + as PID 1 of its PID namespace, its end ends every process of the namespace. - Each execution runs in a cgroup of its own below the workload cgroup of the root configuration, `execution-N`, where N counts the cgroups that the worker prepared. The worker prepares it with @@ -1010,7 +1062,8 @@ that decode, and those that don't. In the guest-only configuration, the unit tests of `worker::probes` cover what the probes read: the mount table with its escapes, the kernel's records of a process, `os-release`, and the guest's memory, and the probes and arguments that the worker refuses. Those of -`worker::executions` hand an execution the host's cancellation once its +`worker::executions`, which spawn each program without entering a root, as a +fake root does, hand an execution the host's cancellation once its timeout's instant passed, after its program ended before it, as a worker that did not run meanwhile would, and check that the program's end stays its own. Those of `worker` run PID 1's collection alone, in a run of the test binary of @@ -1088,16 +1141,20 @@ byte value, and that `init` fails at once on a path that is no tty. test broker: a root that the worker assembles, probes, and tears down on a stop, across a reset of the session; an image over RAM, in the overlay mode of the configuration; a GPT disk, with the configuration's bound of the first -partition; the host export that it mounts, the mapping table that it applies -in order, in parts, across a reset of the session, and the start; a table of -zero mappings, which does no work, with or without an export; the parts that -do not continue the table, which it refuses as a whole, and a mapping that -the root refuses, which latches the root, after either of which it refuses -the later parts and the start; the executions and starts that it refuses out -of order, before the start and after it; the failure of each stage, the tokens -of a host export that break their rules, mappings without an export, and an -export that fails to mount among them, which the worker reports before it ends -with the stage's status, without the operations of the later stages; a teardown +partition; the host export that it mounts, the mapping table that it applies in +order, in parts, across a reset of the session, and the start; a table of zero +mappings, which does no work, with or without an export; the parts that do not +continue the table, which it refuses as a whole, and a mapping that the root +refuses, which latches the root, after either of which it refuses the later +parts and the start; the executions and starts that it refuses out of order, +before the start and after it; the failure of each stage, the tokens of a host +export that break their rules, mappings without an export, an export that fails +to mount, a hostname of more than 64 bytes, and a workload identity whose +account the root lacks among them, which the worker reports before it ends with +the stage's status, without the operations of the later stages; the account +check of a sandbox's workload identity, once the root is assembled, with the +hostname of the kernel command line, the default one without it or with an empty +one, and the first of two; a teardown that fails; before the root, the configuration and the `Stop` that it accepts, and the messages that it refuses; a second configuration and a broker that closes the stream, which leave the root that it assembled to its operations; @@ -1121,7 +1178,12 @@ open, which tests check too. `tests/guest_session.rs` also runs the worker's executions behind its test broker, whose host side starts the sandbox, then drives the engine's peers, with -frame-counted credit and one output pool. Where the tests can create cgroups, as +frame-counted credit and one output pool. The fake root prepares each +execution's entry, as the worker describes it, in a directory of the tests' own +whose account databases hold the accounts that the tests name, and then spawns +the program without entering a root, which needs privileges, in the tests' own +root and identity. Unless a test says otherwise, the sandbox has a workload +identity, as which every execution runs. Where the tests can create cgroups, as root in a cgroup v2 hierarchy with the `pids` controller, the fake root gives the worker a workload cgroup of the test's own, below which each execution runs in a cgroup of its own, so that an execution whose program ends on its own @@ -1172,6 +1234,20 @@ tests cover: even once the cgroup emptied, until the host stops the worker, which then removes the cgroup; - a program that a real-time signal ends, whose status the worker collects; +- a program that runs in the working directory of its start, with exactly the + environment of its start; +- the identities that starts name in a sandbox without a workload identity, + with their supplementary groups, capabilities, and account modes, and the + configured hostname, all of which the worker's entry takes; +- the starts that the worker refuses at stage 0 for their identities: one that + names an identity in a sandbox with a workload identity, or none in a + sandbox without one, with `EINVAL`; a capability outside the entry's + allowlist, one of a user ID other than 0, and `CAP_NET_BIND_SERVICE` and + `CAP_NET_RAW` while listeners are published, with `EPERM`; an identity + without a passwd entry, or whose primary group has no group entry, with + `ENOENT`; and one whose passwd entry has another primary group, or whose + user ID the kernel reads as "unchanged", with `EINVAL`. None of them takes a + place, so the worker then runs its limit of executions at once; - an execution whose start comes in the payload of the root's configuration and the sandbox's start, whose timeout counts from its own start, however long the root's assembly takes; diff --git a/tests/runtime-adapter/src/exec.rs b/tests/runtime-adapter/src/exec.rs index f17d97d85..083061453 100644 --- a/tests/runtime-adapter/src/exec.rs +++ b/tests/runtime-adapter/src/exec.rs @@ -33,6 +33,19 @@ //! each argument's length, a `u32`, and its bytes. The first argument is the program's path, and //! no argument holds a NUL byte. //! +//! The start ends with how the program enters the guest's root: +//! +//! - whether it names the program's [`Identity`], 0 or 1, and if it does, the user ID and the +//! group ID, each a `u32`, the [`AccountMode`], 0 for an existing account or 1 for numeric IDs, +//! the capabilities, a `u64` whose bit N stands for the capability numbered N, then the +//! supplementary groups: their count, a `u16`, then each group, a `u32`; +//! - whether it names the program's working directory, 0 or 1, and if it does, the directory's +//! length, a `u32`, and its bytes: an absolute path without a NUL byte. Otherwise the program +//! starts in the root's `/`; +//! - the program's whole environment: the count of its entries, a `u16`, then each entry's +//! length, a `u32`, and its bytes, `NAME=VALUE`, with a name that is not empty, and without a +//! NUL byte. +//! //! An outcome, after the key, holds the program's status: 0 for none, 1 for an exit, or 2 for a //! signal, then the exit's code or the signal's number, an `i32`, or 0; the cause, or 0 for none; //! whether the launch failed, 0 or 1, then its stage, a `u32`, and its `errno`, an `i32`, or 0 for @@ -82,11 +95,20 @@ pub struct Start { pub input: bool, /// The program's path, which is also its first argument, then its other arguments. pub arguments: Vec>, + /// The identity as which the program runs, if the start names one. Exactly one source names + /// it: the start, or the workload identity of the guest's whole sandbox. + pub identity: Option, + /// The program's working directory, an absolute path in the guest's root, if the program + /// does not start in the root's `/`. + pub working_directory: Option>, + /// The program's whole environment, as `NAME=VALUE` entries. + pub environment: Vec>, } impl Start { /// Returns the start of the execution `key`, with `limits` and the guest's `timers`, of the - /// program that `arguments` names, without standard input. + /// program that `arguments` names, without standard input, without an identity, in the + /// root's `/`, and with an empty environment. pub fn new>>( key: Key, limits: Limits, @@ -99,6 +121,9 @@ impl Start { timers, input: false, arguments: arguments.into_iter().map(Into::into).collect(), + identity: None, + working_directory: None, + environment: Vec::new(), } } @@ -108,6 +133,27 @@ impl Start { self } + /// Names the identity as which the program runs. + pub fn with_identity(mut self, identity: Identity) -> Self { + self.identity = Some(identity); + self + } + + /// Names the program's working directory. + pub fn with_working_directory(mut self, directory: impl Into>) -> Self { + self.working_directory = Some(directory.into()); + self + } + + /// Gives the program the environment of the `NAME=VALUE` entries `environment`. + pub fn with_environment>>( + mut self, + environment: impl IntoIterator, + ) -> Self { + self.environment = environment.into_iter().map(Into::into).collect(); + self + } + /// Returns the start's encoding, the body of an `ExecStart`. pub fn encode(&self) -> Vec { let mut body = key_body(self.key); @@ -129,18 +175,44 @@ impl Start { put_u64(&mut body, timeout.map_or(0, milliseconds)); body.push(u8::from(self.input)); // A start with more arguments than `u16` counts exceeds the framing's limit too. - let count = u16::try_from(self.arguments.len()).unwrap_or(u16::MAX); - body.extend_from_slice(&count.to_le_bytes()); + put_count(&mut body, self.arguments.len()); for argument in &self.arguments { - put_u32(&mut body, u32::try_from(argument.len()).unwrap_or(u32::MAX)); - body.extend_from_slice(argument); + put_bytes(&mut body, argument); + } + match &self.identity { + None => body.push(0), + Some(identity) => { + body.push(1); + put_u32(&mut body, identity.uid); + put_u32(&mut body, identity.gid); + body.push(identity.account.code()); + put_u64(&mut body, identity.capabilities); + put_count(&mut body, identity.groups.len()); + for group in &identity.groups { + put_u32(&mut body, *group); + } + } + } + match &self.working_directory { + None => body.push(0), + Some(directory) => { + body.push(1); + put_bytes(&mut body, directory); + } + } + put_count(&mut body, self.environment.len()); + for entry in &self.environment { + put_bytes(&mut body, entry); } body } /// Decodes the body of an `ExecStart`, and checks it: its limits, of which none is empty and - /// whose chunk is at most [`MAX_CHUNK`] bytes, its flags, and its arguments, of which there is - /// at least one, the program's path, which is not empty, and none of which holds a NUL byte. + /// whose chunk is at most [`MAX_CHUNK`] bytes, its flags, its arguments, of which there is + /// at least one, the program's path, which is not empty, and none of which holds a NUL byte, + /// its identity's account mode, its working directory, which is absolute and holds no NUL + /// byte, and its environment, whose entries are `NAME=VALUE`, with a name that is not empty, + /// and hold no NUL byte. /// /// Fails with a [`BodyError`] that names the first problem. pub fn decode(body: &[u8]) -> Result { @@ -159,13 +231,64 @@ impl Start { let count = reader.u16()?; let mut arguments = Vec::with_capacity(usize::from(count).min(body.len())); for _ in 0..count { - let length = reader.u32()?; - let argument = reader.bytes(usize::try_from(length).unwrap_or(usize::MAX))?; + let argument = reader.counted()?; if argument.contains(&0) { return Err(refused("has an argument with a NUL byte".to_owned())); } arguments.push(argument.to_vec()); } + let identity = if flag(reader.u8()?).map_err(refused)? { + let uid = reader.u32()?; + let gid = reader.u32()?; + let account = AccountMode::from_code(reader.u8()?).map_err(refused)?; + let capabilities = reader.u64()?; + let count = reader.u16()?; + let mut groups = Vec::with_capacity(usize::from(count).min(body.len())); + for _ in 0..count { + groups.push(reader.u32()?); + } + Some(Identity { + uid, + gid, + groups, + capabilities, + account, + }) + } else { + None + }; + let working_directory = if flag(reader.u8()?).map_err(refused)? { + let directory = reader.counted()?; + if directory.first() != Some(&b'/') { + return Err(refused( + "names a working directory that is not absolute".to_owned(), + )); + } + if directory.contains(&0) { + return Err(refused( + "has a working directory with a NUL byte".to_owned(), + )); + } + Some(directory.to_vec()) + } else { + None + }; + let count = reader.u16()?; + let mut environment = Vec::with_capacity(usize::from(count).min(body.len())); + for _ in 0..count { + let entry = reader.counted()?; + if entry.contains(&0) { + return Err(refused( + "has an environment entry with a NUL byte".to_owned(), + )); + } + if entry.iter().position(|&byte| byte == b'=').unwrap_or(0) == 0 { + return Err(refused( + "has an environment entry that is not NAME=VALUE".to_owned(), + )); + } + environment.push(entry.to_vec()); + } reader.finish()?; if chunk > MAX_CHUNK { return Err(refused(format!( @@ -196,10 +319,74 @@ impl Start { timers, input: with_input, arguments, + identity, + working_directory, + environment, }) } } +/// The identity as which an execution's program runs, which a start names unless the guest's +/// sandbox has a workload identity: its IDs, its capabilities, and how the guest checks its +/// account, as the guest's entry into its root takes them. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Identity { + /// The user ID. + pub uid: u32, + /// The group ID. + pub gid: u32, + /// The supplementary groups. + pub groups: Vec, + /// The capabilities, which only user ID 0 may hold: bit N for the capability numbered N. + pub capabilities: u64, + /// How the guest checks the identity's account. + pub account: AccountMode, +} + +impl Identity { + /// Returns the identity of user ID `uid` and group ID `gid`, without supplementary groups or + /// capabilities, whose account the guest checks in [existing](AccountMode::Existing) mode. + pub fn new(uid: u32, gid: u32) -> Self { + Self { + uid, + gid, + groups: Vec::new(), + capabilities: 0, + account: AccountMode::Existing, + } + } +} + +/// How the guest checks the account of an execution's [`Identity`]. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum AccountMode { + /// The root's `/etc/passwd` holds exactly one entry for the user ID, whose primary group is + /// the group ID, which the root's `/etc/group` holds. + #[default] + Existing, + /// The IDs are used as given, without an account. + Numeric, +} + +impl AccountMode { + /// Returns the mode's code in a start: 0 for an existing account, 1 for numeric IDs. + fn code(self) -> u8 { + match self { + Self::Existing => 0, + Self::Numeric => 1, + } + } + + /// Returns the mode that `code` encodes, or the problem. + fn from_code(code: u8) -> Result { + match code { + 0 => Ok(Self::Existing), + 1 => Ok(Self::Numeric), + _ => Err(format!("names the unknown account mode {code}")), + } + } +} + /// A message of the host's side of an execution, its [`Peer`], to the guest's. #[derive(Clone, Debug, PartialEq, Eq)] pub enum HostMessage<'a> { @@ -418,6 +605,12 @@ impl<'a> Reader<'a> { Ok(taken) } + /// Takes the next bytes, after their length, a `u32`. + fn counted(&mut self) -> Result<&'a [u8], BodyError> { + let length = self.u32()?; + self.bytes(usize::try_from(length).unwrap_or(usize::MAX)) + } + /// Takes the next `N` bytes. fn array(&mut self) -> Result<[u8; N], BodyError> { Ok(self.bytes(N)?.try_into().expect("N bytes")) @@ -755,5 +948,18 @@ fn put_u64(body: &mut Vec, value: u64) { body.extend_from_slice(&value.to_le_bytes()); } +/// Appends the count `count` of a start's items to `body`, as a `u16`. A start with more items +/// than a `u16` counts exceeds the framing's limit too. +fn put_count(body: &mut Vec, count: usize) { + let count = u16::try_from(count).unwrap_or(u16::MAX); + body.extend_from_slice(&count.to_le_bytes()); +} + +/// Appends `bytes` to `body`, after their length, a `u32`. +fn put_bytes(body: &mut Vec, bytes: &[u8]) { + put_u32(body, u32::try_from(bytes.len()).unwrap_or(u32::MAX)); + body.extend_from_slice(bytes); +} + #[cfg(test)] mod tests; diff --git a/tests/runtime-adapter/src/exec/tests.rs b/tests/runtime-adapter/src/exec/tests.rs index 2e3492aac..db112acfb 100644 --- a/tests/runtime-adapter/src/exec/tests.rs +++ b/tests/runtime-adapter/src/exec/tests.rs @@ -8,7 +8,7 @@ use nvx::exec::{ Timers, Window, peer, runner, }; -use super::{GuestMessage, HostMessage, KEY_LEN, MAX_CHUNK, Start, name}; +use super::{AccountMode, GuestMessage, HostMessage, Identity, KEY_LEN, MAX_CHUNK, Start, name}; use crate::frame::{MAX_BODY, Message}; /// The key of most executions in these tests. @@ -27,7 +27,18 @@ fn timers() -> Timers { Timers::new(Duration::from_secs(1), Duration::from_secs(5)) } -/// Returns starts with every unit, flag, and kind of argument. +/// Returns an identity with every field: supplementary groups, capabilities, and numeric IDs. +fn numeric() -> Identity { + Identity { + groups: vec![0, 50, u32::MAX], + capabilities: 1 | 1 << 13 | 1 << 63, + account: AccountMode::Numeric, + ..Identity::new(4242, u32::MAX) + } +} + +/// Returns starts with every unit, flag, and kind of argument, of identity, of working +/// directory, and of environment. fn starts() -> Vec { let bytes = Limits::new(1, Window::bytes(1), Window::bytes(u32::MAX)).unwrap(); let largest = Limits::new(MAX_CHUNK, Window::frames(u32::MAX), Window::frames(1)).unwrap(); @@ -39,14 +50,20 @@ fn starts() -> Vec { bytes, Timers::new(Duration::ZERO, Duration::from_millis(u64::MAX)).timeout(Duration::ZERO), [b"/x".to_vec(), Vec::new(), (1..=u8::MAX).collect()], - ), + ) + .with_identity(numeric()) + .with_working_directory("/") + .with_environment(["A=", "B==", "PATH=/usr/bin:/bin"]), Start::new( Key::new(0, 0), largest, timers().timeout(Duration::from_millis(1500)), ["/bin/true"], ) - .with_input(), + .with_input() + .with_identity(Identity::new(0, 0)) + .with_working_directory(std::iter::once(b'/').chain(1..=u8::MAX).collect::>()) + .with_environment([(1..=u8::MAX).collect::>()]), ] } @@ -97,16 +114,56 @@ fn a_start_s_layout_is_its_key_limits_timers_input_and_arguments() { expected.extend_from_slice(b"/p"); expected.extend_from_slice(&1u32.to_le_bytes()); expected.extend_from_slice(b"a"); + // No identity, no working directory, and an empty environment. + expected.extend_from_slice(&[0, 0, 0, 0]); assert_eq!(start.encode(), expected); } +#[test] +fn a_start_s_layout_ends_with_its_identity_working_directory_and_environment() { + let start = Start::new(KEY, shared(), timers(), ["/p"]) + .with_identity(Identity { + groups: vec![50, 60], + capabilities: 1 << 13 | 1, + account: AccountMode::Numeric, + ..Identity::new(1000, 100) + }) + .with_working_directory("/w") + .with_environment(["A=1", "B="]); + let head = Start::new(KEY, shared(), timers(), ["/p"]).encode(); + let mut expected = head[..head.len() - 4].to_vec(); + expected.push(1); + expected.extend_from_slice(&1000u32.to_le_bytes()); + expected.extend_from_slice(&100u32.to_le_bytes()); + expected.push(1); + expected.extend_from_slice(&0x2001u64.to_le_bytes()); + expected.extend_from_slice(&2u16.to_le_bytes()); + expected.extend_from_slice(&50u32.to_le_bytes()); + expected.extend_from_slice(&60u32.to_le_bytes()); + expected.push(1); + expected.extend_from_slice(&2u32.to_le_bytes()); + expected.extend_from_slice(b"/w"); + expected.extend_from_slice(&2u16.to_le_bytes()); + expected.extend_from_slice(&3u32.to_le_bytes()); + expected.extend_from_slice(b"A=1"); + expected.extend_from_slice(&2u32.to_le_bytes()); + expected.extend_from_slice(b"B="); + assert_eq!(start.encode(), expected); + // An existing account is 0. + let existing = Start::new(KEY, shared(), timers(), ["/p"]).with_identity(Identity::new(0, 0)); + assert_eq!(existing.encode()[head.len() - 4 + 9], 0); +} + #[test] fn starts_that_break_the_rules_are_refused() { let valid = Start::new(KEY, shared(), timers(), ["/bin/sh", "-c", "exit"]).encode(); + let full = starts()[2].encode(); // Every start that ends early, and one with a byte too many. - for length in 0..valid.len() { - let problem = Start::decode(&valid[..length]).unwrap_err(); - assert_eq!(problem.problem(), "an execution's start ends early"); + for body in [&valid, &full] { + for length in 0..body.len() { + let problem = Start::decode(&body[..length]).unwrap_err(); + assert_eq!(problem.problem(), "an execution's start ends early"); + } } let mut longer = valid.clone(); longer.push(0); @@ -154,6 +211,7 @@ fn starts_that_break_the_rules_are_refused() { // Arguments: none, an empty program, a NUL byte, and a length beyond the body. let mut none = valid[..57].to_vec(); none.extend_from_slice(&0u16.to_le_bytes()); + none.extend_from_slice(&[0, 0, 0, 0]); assert_eq!( Start::decode(&none).unwrap_err().problem(), refused("names no program") @@ -174,6 +232,64 @@ fn starts_that_break_the_rules_are_refused() { Start::decode(&beyond).unwrap_err().problem(), "an execution's start ends early" ); + // The flags of the identity and the working directory, and the account mode, which the + // identity's IDs precede. + let head = &valid[..valid.len() - 4]; + let ending = |tail: &[u8]| { + let body = [head, tail].concat(); + Start::decode(&body).unwrap_err().problem().to_owned() + }; + assert_eq!(ending(&[2, 0, 0, 0]), refused("has the flag 2")); + assert_eq!(ending(&[0, 3, 0, 0]), refused("has the flag 3")); + let mut identity = vec![1]; + identity.extend_from_slice(&[0; 8]); + identity.push(2); + identity.extend_from_slice(&[0; 10]); + identity.extend_from_slice(&[0, 0, 0]); + assert_eq!( + ending(&identity), + refused("names the unknown account mode 2") + ); + // Working directories that are not absolute, or hold a NUL byte, and environment entries + // that are not `NAME=VALUE`, or hold a NUL byte. + let start = || Start::new(KEY, shared(), timers(), ["/bin/true"]); + let cases = [ + ( + start().with_working_directory("relative"), + "names a working directory that is not absolute", + ), + ( + start().with_working_directory(""), + "names a working directory that is not absolute", + ), + ( + start().with_working_directory("/a\0b"), + "has a working directory with a NUL byte", + ), + ( + start().with_environment(["A=1", "NAME"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + start().with_environment(["=value"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + start().with_environment([""]), + "has an environment entry that is not NAME=VALUE", + ), + ( + start().with_environment(["A=b\0c"]), + "has an environment entry with a NUL byte", + ), + ]; + for (start, problem) in cases { + assert_eq!( + Start::decode(&start.encode()).unwrap_err().problem(), + refused(problem), + "{start:?}" + ); + } } /// Returns outcomes with every status, cause, launch failure, and flag. diff --git a/tests/runtime-adapter/src/root.rs b/tests/runtime-adapter/src/root.rs index 41dcb762d..8eaffb7d5 100644 --- a/tests/runtime-adapter/src/root.rs +++ b/tests/runtime-adapter/src/root.rs @@ -7,8 +7,8 @@ //! the rest that the root needs, which only the adapter defines, after the host attaches: the //! runtime directory, the cgroups, their resource values, the sizes that the block devices must //! have, how long the guest waits for them, the most bytes that the first partition of a GPT disk -//! may hold, how the guest assembles the overlay over the upper, and how many mappings the host's -//! mapping table holds. +//! may hold, how the guest assembles the overlay over the upper, how many mappings the host's +//! mapping table holds, and the workload identity of the whole sandbox, if it has one. //! //! A root configuration is UTF-8 text: one `NAME=VALUE` line per field, each ended by a newline, //! in any order, and each at most once. Numbers are decimal, without a sign or leading zeros: @@ -29,12 +29,17 @@ //! | `overlay-mode` | The overlay's [mode](OverlayMode): `trusted-lowers`, the guest's default, or `untrusted-lowers` | no | //! | `scratch-journal` | Whether an ext4 scratch may have a journal, a [`JournalPolicy`]: `allowed`, the guest's default, or `forbidden` | no | //! | `mapping-count` | The number of mappings of the host's mapping table, at most [`MAX_HOST_MAPPINGS`] | no | +//! | `workload-uid` | The user ID of the sandbox's workload identity, as which every execution runs | with `workload-gid` | +//! | `workload-gid` | The group ID of the sandbox's workload identity | with `workload-uid` | //! //! The paths are absolute and canonical guest paths: not `/`, without an empty, `.`, or `..` //! component, or a trailing `/`. The runtime directory holds no `,`, `:`, `\`, or `=`, which the //! overlay's options reserve, and the two cgroups have the same parent. The optional numbers are -//! positive. [`RootConfig::decode`] refuses every other configuration, which the guest reports as -//! a failure of [`RootStage::Configuration`]. +//! positive. A workload identity has both IDs, each below 4,294,967,295, which the kernel reads +//! as "unchanged", and no supplementary groups or capabilities; the guest checks its account in +//! the assembled root, a failure of [`RootStage::Identity`], and an execution then names no +//! identity of its own. [`RootConfig::decode`] refuses every other configuration, which the guest +//! reports as a failure of [`RootStage::Configuration`]. //! //! A [`RootFailure`] is a stage's code, one byte, then the problem as UTF-8 text. //! @@ -120,8 +125,12 @@ const OVERLAY_MODE: &str = "overlay-mode"; const SCRATCH_JOURNAL: &str = "scratch-journal"; /// See [`VERSION`]. const MAPPING_COUNT: &str = "mapping-count"; +/// See [`VERSION`]. +const WORKLOAD_UID: &str = "workload-uid"; +/// See [`VERSION`]. +const WORKLOAD_GID: &str = "workload-gid"; /// Every field, in the order in which [`RootConfig::encode`] writes them. -const FIELDS: [&str; 14] = [ +const FIELDS: [&str; 16] = [ VERSION, RUNTIME, AGENT_CGROUP, @@ -136,6 +145,8 @@ const FIELDS: [&str; 14] = [ OVERLAY_MODE, SCRATCH_JOURNAL, MAPPING_COUNT, + WORKLOAD_UID, + WORKLOAD_GID, ]; /// The overlay mode that a configuration names, which the guest's assembly applies: the @@ -230,6 +241,9 @@ pub struct RootConfig { /// The number of mappings of the host's mapping table, at most [`MAX_HOST_MAPPINGS`], or 0 /// for none. pub mapping_count: u32, + /// The workload identity of the whole sandbox, its user ID and group ID, if it has one: every + /// execution runs as it, and names no identity of its own. + pub workload_identity: Option<(u32, u32)>, } impl Default for RootConfig { @@ -237,7 +251,7 @@ impl Default for RootConfig { /// [`DEFAULT_WORKLOAD_CGROUP`], [`DEFAULT_DEVICE_TIMEOUT`], [`DEFAULT_AGENT_MEMORY_LOW`], /// [`DEFAULT_WORKLOAD_MEMORY_MAX`], and [`DEFAULT_WORKLOAD_PIDS_MAX`], with no sizes, the /// guest's default bound of the first partition, overlay mode, and journal policy, and no - /// mapping. + /// mapping or workload identity. fn default() -> Self { Self { runtime: DEFAULT_RUNTIME.to_owned(), @@ -253,6 +267,7 @@ impl Default for RootConfig { overlay_mode: None, scratch_journal: None, mapping_count: 0, + workload_identity: None, } } } @@ -297,6 +312,12 @@ impl RootConfig { if self.mapping_count > 0 { fields.push((MAPPING_COUNT, self.mapping_count.to_string())); } + if let Some((uid, gid)) = self.workload_identity { + fields.extend([ + (WORKLOAD_UID, uid.to_string()), + (WORKLOAD_GID, gid.to_string()), + ]); + } fields } @@ -412,6 +433,26 @@ impl RootConfig { "announces {mapping_count} mappings, more than {MAX_HOST_MAPPINGS}" ))); } + let id = |name: &str| -> Result, BodyError> { + let Some(value) = number(name)? else { + return Ok(None); + }; + match u32::try_from(value) { + Ok(id) if id < u32::MAX => Ok(Some(id)), + _ => Err(refused(format!( + "has a {name} above 4294967294, which no identity has: {value}" + ))), + } + }; + let workload_identity = match (id(WORKLOAD_UID)?, id(WORKLOAD_GID)?) { + (Some(uid), Some(gid)) => Some((uid, gid)), + (None, None) => None, + _ => { + return Err(refused(format!( + "names a workload identity without both {WORKLOAD_UID} and {WORKLOAD_GID}" + ))); + } + }; Ok(Self { runtime, agent_cgroup, @@ -427,6 +468,7 @@ impl RootConfig { scratch_journal, // The count is at most `MAX_HOST_MAPPINGS`, so it fits a `u32`. mapping_count: mapping_count as u32, + workload_identity, }) } } @@ -478,7 +520,7 @@ pub enum RootStage { /// Receiving and checking the root configuration. Configuration, /// Reading the boot layout of the root from the kernel command line, and checking that the - /// guest can assemble it. + /// guest can assemble it, and the hostname of the executions there. Layout, /// Bootstrapping the guest for the root: its cgroups, and user namespaces. Bootstrap, @@ -486,19 +528,24 @@ pub enum RootStage { Assembly, /// Tearing the root down. Teardown, + /// Checking the account of the sandbox's workload identity in the assembled root, in + /// existing-account mode, once the root is assembled. + Identity, } impl RootStage { - /// Every stage, in order. - pub const ALL: [Self; 5] = [ + /// Every stage, in the order of their codes. + pub const ALL: [Self; 6] = [ Self::Configuration, Self::Layout, Self::Bootstrap, Self::Assembly, Self::Teardown, + Self::Identity, ]; - /// Returns the stage's code in a [`RootFailure`]'s body: its position, from 1. + /// Returns the stage's code in a [`RootFailure`]'s body: its position in [`ALL`](Self::ALL), + /// from 1. pub const fn code(self) -> u8 { match self { Self::Configuration => 1, @@ -506,6 +553,7 @@ impl RootStage { Self::Bootstrap => 3, Self::Assembly => 4, Self::Teardown => 5, + Self::Identity => 6, } } @@ -514,8 +562,8 @@ impl RootStage { Self::ALL.into_iter().find(|stage| stage.code() == code) } - /// Returns the stage's name: `configuration`, `layout`, `bootstrap`, `assembly`, or - /// `teardown`. + /// Returns the stage's name: `configuration`, `layout`, `bootstrap`, `assembly`, + /// `teardown`, or `identity`. pub const fn as_str(self) -> &'static str { match self { Self::Configuration => "configuration", @@ -523,6 +571,7 @@ impl RootStage { Self::Bootstrap => "bootstrap", Self::Assembly => "assembly", Self::Teardown => "teardown", + Self::Identity => "identity", } } @@ -532,8 +581,8 @@ impl RootStage { } /// Returns the status with which the guest's worker, and with it the VM, ends once this stage - /// failed: [`FAILURE_STATUS_BASE`] plus its code, from 11 for the configuration to 15 for - /// the teardown. + /// failed: [`FAILURE_STATUS_BASE`] plus its code, from 11 for the configuration to 16 for + /// the identity. pub const fn exit_status(self) -> u8 { FAILURE_STATUS_BASE + self.code() } @@ -834,6 +883,7 @@ mod tests { overlay_mode: Some(OverlayMode::UntrustedLowers), scratch_journal: Some(JournalPolicy::Forbidden), mapping_count: 3, + workload_identity: Some((1000, 100)), ..RootConfig::default() } } @@ -855,7 +905,9 @@ mod tests { first-partition-bound=4096\n\ overlay-mode=untrusted-lowers\n\ scratch-journal=forbidden\n\ - mapping-count=3\n" + mapping-count=3\n\ + workload-uid=1000\n\ + workload-gid=100\n" ); // Without a mapping, the configuration names no count. let unmapped = String::from_utf8(RootConfig::default().encode()).unwrap(); @@ -873,7 +925,11 @@ mod tests { workload_pids_max: None, ..RootConfig::default() }; - for config in [full(), minimal, RootConfig::default()] { + let largest = RootConfig { + workload_identity: Some((u32::MAX - 1, u32::MAX - 1)), + ..RootConfig::default() + }; + for config in [full(), minimal, largest, RootConfig::default()] { assert_eq!(RootConfig::decode(&config.encode()), Ok(config)); } // The fields may come in any order. @@ -1058,6 +1114,30 @@ mod tests { format!("{base}mapping-count=18446744073709551615\n").into_bytes(), "announces 18446744073709551615 mappings, more than 4096", ), + ( + format!("{base}workload-uid=0\nworkload-gid=1000\n").into_bytes(), + "has a workload-uid that is not a positive decimal number: \"0\"", + ), + ( + format!("{base}workload-uid=1000\nworkload-gid=0\n").into_bytes(), + "has a workload-gid that is not a positive decimal number: \"0\"", + ), + ( + format!("{base}workload-uid=4294967295\nworkload-gid=1000\n").into_bytes(), + "has a workload-uid above 4294967294, which no identity has: 4294967295", + ), + ( + format!("{base}workload-uid=1000\nworkload-gid=4294967296\n").into_bytes(), + "has a workload-gid above 4294967294", + ), + ( + format!("{base}workload-uid=1000\n").into_bytes(), + "names a workload identity without both workload-uid and workload-gid", + ), + ( + format!("{base}workload-gid=1000\n").into_bytes(), + "names a workload identity without both workload-uid and workload-gid", + ), ]; for (body, problem) in cases { let error = RootConfig::decode(&body).unwrap_err(); @@ -1095,6 +1175,7 @@ mod tests { (RootStage::Bootstrap, 3, "bootstrap", 13), (RootStage::Assembly, 4, "assembly", 14), (RootStage::Teardown, 5, "teardown", 15), + (RootStage::Identity, 6, "identity", 16), ]; for (stage, code, name, status) in expected { assert_eq!(stage.code(), code); @@ -1104,8 +1185,9 @@ mod tests { assert_eq!(RootStage::from_name(name), Some(stage)); assert_eq!(stage.exit_status(), status); } + assert_eq!(RootStage::ALL.map(RootStage::code), [1, 2, 3, 4, 5, 6]); assert_eq!(RootStage::from_code(0), None); - assert_eq!(RootStage::from_code(6), None); + assert_eq!(RootStage::from_code(7), None); assert_eq!(RootStage::from_name("mapping"), None); } @@ -1131,7 +1213,7 @@ mod tests { for (body, problem) in [ (&b""[..], "names no stage"), (b"\x00problem", "unknown stage 0"), - (b"\x06problem", "unknown stage 6"), + (b"\x07problem", "unknown stage 7"), (b"\x01\xffproblem", "not UTF-8 text"), ] { let error = RootFailure::decode(body).unwrap_err(); diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index dbde923e1..92a9c081c 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -21,9 +21,11 @@ //! 1. It opens the tty and attaches the session. //! 2. It receives the host's root configuration, a [`RootConfig`], and checks it. //! 3. It reads the boot layout of the root from the kernel command line with -//! `nvx::protocol::root::RootLayout::parse`, and the host export, if the command line names -//! one, with `nvx::protocol::export::HostExport::parse`, then bootstraps the guest, assembles -//! the root, and mounts the export out of the root. +//! `nvx::protocol::root::RootLayout::parse`, the executions' hostname, and the host export, if +//! the command line names one, with `nvx::protocol::export::HostExport::parse`, then bootstraps +//! the guest, assembles the root, and mounts the export out of the root. If the configuration +//! names a workload identity for the whole sandbox, it then checks the identity's account in the +//! assembled root, in existing-account mode, through `nvx::guest::entry`. //! 4. It answers `Ready`, with the guest agent's build, [`GUEST_BUILD`], or `RootFailed` with the //! stage that failed, and then ends. //! 5. It applies the host's mapping table beneath the root, from the export, in the parts that @@ -44,11 +46,15 @@ //! configures a scratch for a root over RAM, bounds a first partition for an EROFS image, or //! announces mappings while the kernel command line names no host export, from which they come, //! and a command line whose tokens of the export break their rules. An export that fails to mount -//! fails the assembly. +//! fails the assembly. The executions' hostname is the value of the kernel command line's first +//! `nvx_hostname=` parameter, which NVX's sandbox plans configure, and which NVX's init agent reads +//! too, or `nvx-sandbox`, `nvx::guest::entry::DEFAULT_HOSTNAME`, without one, or with an empty one. +//! A hostname of more than 64 bytes fails the layout stage. A workload identity whose account the +//! assembled root lacks fails the identity stage. //! //! The worker, and with it the VM, ends with 0 once the host stopped it, after a teardown that //! succeeded, with the stage's [exit status](RootStage::exit_status) once a stage failed, from 11 -//! for the configuration to 15 for the teardown, and with [`SESSION_STATUS`] once its session +//! for the configuration to 16 for the identity, and with [`SESSION_STATUS`] once its session //! failed or the broker closed it. Once the broker closed the stream at a record boundary, which //! may close only the broker's end, the worker drops its executions, and writes what it queued, //! as once the host stopped it. @@ -116,11 +122,28 @@ //! - The worker admits at most [`MAX_EXECUTIONS`] executions at once, abandoned ones included, //! and those that keep their places once they ended, as the items below describe, and none //! that would hold more than [`MAX_EXECUTION_MEMORY`] bytes. It refuses a start beyond those, -//! or one whose cgroup it cannot prepare, or whose program it cannot spawn, with a terminal -//! record whose launch failed at stage 0, with `EAGAIN`, `ENOMEM`, or the `errno` of the -//! preparation or the spawn. -//! - Each program runs in the worker's root and working directories, with its environment, and -//! leads a process group of its own. +//! one whose identity or entry it refuses, as the next item describes, or one whose cgroup it +//! cannot prepare, or whose program it cannot spawn, with a terminal record whose launch failed +//! at stage 0, with `EAGAIN`, `ENOMEM`, the `errno` of the refusal, or that of the preparation +//! or the spawn. +//! - Each program enters the assembled root through `nvx::guest::entry`, as an [`Entrance`] +//! describes it: in new mount, PID, and UTS namespaces, with the root moved over `/`, the +//! configured hostname, the image's own machine ID, and no time status, so that the entry binds +//! nothing into the root, and with published listeners, the entry's default. It runs as the +//! identity that exactly one source names: the sandbox's workload identity, which the root +//! configuration names, with no supplementary groups or capabilities, or, in a sandbox without +//! one, the identity that its start names, with its supplementary groups, its capabilities, and +//! its account mode. A start that names an identity in a sandbox with a workload identity, or +//! none in a sandbox without one, is refused with `EINVAL`. Before the worker prepares the +//! execution's cgroup, the entry checks the identity, and reads its account in +//! existing-account mode: it refuses with `EPERM` a capability outside its allowlist, any +//! capability of a user ID other than 0, and `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while +//! listeners are published, with `ENOENT` an identity without the user ID's passwd entry or +//! the group ID's group entry, with the error of an account database or a root that it cannot +//! read, and with `EINVAL` otherwise. The program starts in the working directory of its start, +//! or in the root's `/`, with exactly the environment of its start, and leads a session of its +//! own, without a controlling terminal, and the process group that the session creates. As PID +//! 1 of its PID namespace, its end ends every process of the namespace. //! - Each execution runs in a cgroup of its own below the workload cgroup, `execution-N`, where N //! counts the cgroups that the worker prepared. The worker prepares it with //! `nvx::guest::cgroup::ExecutionCgroup::prepare`, and the program joins it as it spawns, @@ -191,12 +214,14 @@ use nix::sys::wait::{self, Id, WaitPidFlag, WaitStatus}; use nix::unistd::Pid; use nvx::gpt::Expected; use nvx::guest::cgroup::Layout; +use nvx::guest::entry::{self, Entry, Identity, Prepared}; use nvx::guest::mappings::{Mapping, SourceRoot}; use nvx::guest::nvxs::GuestConnectionError; use nvx::guest::process::Child; use nvx::guest::root::{ self, Assembly, Bootstrap, Device, Image, Journal, Resources, Root, Upper, UpperKind, }; +use nvx::guest::spawn::{self, Spawned}; use nvx::protocol::export::HostExport; use nvx::protocol::root::{ DISTRO_MMIO_BASE, ImageFormat, RootLayout, RootUpper, SCRATCH_MMIO_BASE, @@ -212,7 +237,7 @@ use crate::root::{ RootStage, }; use connection::{Connection, Received}; -use executions::Executions; +use executions::{Executions, Sandbox}; use order::Order; /// The status with which the worker ends once its session failed or the broker closed it, and @@ -241,10 +266,49 @@ pub const MAX_HISTORY: usize = 1024; /// The kernel command line, from which the worker reads the boot layout of the root. const COMMAND_LINE: &str = "/proc/cmdline"; +/// The kernel command line's parameter that names the executions' hostname, as NVX's init agent +/// reads it. +const HOSTNAME_PARAMETER: &[u8] = b"nvx_hostname="; + +/// How one execution enters the assembled root: the inputs of its `nvx::guest::entry::Entry` that +/// the worker chooses. The entry keeps its defaults for the rest: new mount, PID, and UTS +/// namespaces, with the root moved over `/`, the image's own machine ID and no time status, which +/// leave `Entry::bind_runtime` nothing to bind, published listeners, which refuse +/// `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW`, and every device node. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Entrance { + /// The identity: the sandbox's workload identity, or the one that the execution's start names. + pub identity: Identity, + /// The hostname that the kernel command line configures. + pub hostname: Vec, +} + +impl Entrance { + /// Returns the entry into the root at `root`. + pub fn entry(&self, root: impl Into) -> Entry { + Entry::new(root, self.identity.clone()).hostname(self.hostname.clone()) + } +} + +/// The operations through which the executions enter the assembled root: those of +/// `nvx::guest::entry` in a VM, in [`LibraryRoot`], and a fake's in tests, which cannot enter a +/// root without privileges. +pub trait EntryOperations { + /// Prepares the entry of an execution into the assembled root, as `entrance` describes it, + /// with `Entry::prepare`: checks its inputs, and reads and checks the identity's account in + /// existing-account mode. + fn prepare(&mut self, entrance: &Entrance) -> Result; + + /// Spawns `command`, whose child enters the execution that `prepared` prepared. A fake that + /// cannot enter a root spawns the command in its own root instead, as its own identity. + fn spawn(&mut self, command: spawn::Command, prepared: Prepared) -> io::Result; +} + /// The operations of a layered root that the worker runs: those of `nvx::guest::root` in a VM, in /// [`LibraryRoot`], and a fake's in tests. Each fails with the problem, as text, and -/// [`map`](Self::map) with the mapping that failed too. -pub trait RootOperations { +/// [`map`](Self::map) with the mapping that failed too, but the [entry's](EntryOperations), which +/// fail as `nvx::guest::entry` does. +pub trait RootOperations: EntryOperations { /// Returns the kernel command line, from which the worker reads the boot layout of the root, /// and the host export. fn command_line(&mut self) -> Result, String>; @@ -468,8 +532,9 @@ fn wait( /// Handles `message` before the root is ready: a root configuration, whose root it assembles /// through `operations`, after which `executions` run in the cgroups that the operations prepared, -/// and which it answers with its readiness, or a `Stop`. Returns the step, with the worker's order -/// once the root is ready. +/// as the sandbox that the configuration and the kernel command line describe, and which it answers +/// with its readiness, or a `Stop`. Returns the step, with the worker's order once the root is +/// ready. fn configure_root( operations: &mut dyn RootOperations, executions: &mut Executions, @@ -477,8 +542,8 @@ fn configure_root( ) -> Result<(Step, Option), ServeError> { match message { Message::RootConfig(body) => Ok(match assemble(operations, &body) { - Ok(config) => { - executions.run_in(operations.cgroups()); + Ok((config, sandbox)) => { + executions.run_in(operations.cgroups(), sandbox); let ready = Ready::new(GUEST_BUILD).expect("this crate's version is a build"); let step = Step::Continue(vec![Message::Ready(ready.encode())]); (step, Some(Order::new(config.mapping_count))) @@ -494,8 +559,8 @@ fn configure_root( } /// Handles `message` at `now` once the root is ready, through `operations`, in the worker's -/// `order`, and hands a message about an execution to `executions`, which queue what they answer -/// on `connection`. +/// `order`, and hands a message about an execution to `executions`, which enter the root through +/// `operations`, and queue what they answer on `connection`. fn serve_root( operations: &mut dyn RootOperations, order: &mut Order, @@ -510,7 +575,7 @@ fn serve_root( return Err(ServeError::Unexpected(message.kind())); } let decoded = decoded.map_err(|error| ServeError::Execution(error.problem().to_owned()))?; - executions.deliver(now, decoded, connection)?; + executions.deliver(now, decoded, operations, connection)?; return Ok(Step::Continue(Vec::new())); } Ok(match message { @@ -545,14 +610,19 @@ fn serve_root( }) } -/// Checks the root configuration `body`, reads the boot layout of the root and the host export, -/// bootstraps the guest, assembles the root, and mounts the export out of it, if the kernel -/// command line names one, through `operations`. Returns the configuration, or the failure of the -/// stage that failed: the export's mount ends the assembly. -fn assemble(operations: &mut dyn RootOperations, body: &[u8]) -> Result { +/// Checks the root configuration `body`, reads the boot layout of the root, the executions' +/// hostname, and the host export, bootstraps the guest, assembles the root, mounts the export out +/// of it, if the kernel command line names one, and checks the account of the sandbox's workload +/// identity there, if the configuration names one, through `operations`. Returns the configuration, +/// with the sandbox in which the executions run, or the failure of the stage that failed: the +/// export's mount ends the assembly. +fn assemble( + operations: &mut dyn RootOperations, + body: &[u8], +) -> Result<(RootConfig, Sandbox), RootFailure> { let config = RootConfig::decode(body) .map_err(|error| RootFailure::new(RootStage::Configuration, error.problem()))?; - let (layout, export) = layout(operations, &config) + let (layout, export, hostname) = layout(operations, &config) .map_err(|problem| RootFailure::new(RootStage::Layout, problem))?; operations .bootstrap(&config) @@ -565,18 +635,34 @@ fn assemble(operations: &mut dyn RootOperations, body: &[u8]) -> Result Result<(RootLayout, Option), String> { +) -> Result<(RootLayout, Option, Vec), String> { let command_line = operations.command_line()?; let layout = RootLayout::parse(&command_line) .map_err(|error| format!("the kernel command line has invalid boot tokens: {error}"))? @@ -607,7 +693,29 @@ fn layout( line names no host export to map from" )); } - Ok((layout, export)) + Ok((layout, export, hostname(&command_line)?)) +} + +/// Returns the hostname that the kernel command line `command_line` configures, as NVX's init +/// agent reads it: the value of its first `nvx_hostname=` parameter, or +/// [`DEFAULT_HOSTNAME`](entry::DEFAULT_HOSTNAME) if it has none, or an empty one. A hostname of +/// more than [`MAX_HOSTNAME_BYTES`](entry::MAX_HOSTNAME_BYTES), or with a NUL, is refused. +fn hostname(command_line: &[u8]) -> Result, String> { + let named = command_line + .split(u8::is_ascii_whitespace) + .find_map(|parameter| parameter.strip_prefix(HOSTNAME_PARAMETER)); + match named { + None | Some([]) => Ok(entry::DEFAULT_HOSTNAME.as_bytes().to_vec()), + Some(hostname) if hostname.len() > entry::MAX_HOSTNAME_BYTES || hostname.contains(&0) => { + Err(format!( + "the kernel command line names a hostname that is longer than {} bytes, or \ + holds a NUL: {:?}", + entry::MAX_HOSTNAME_BYTES, + String::from_utf8_lossy(hostname) + )) + } + Some(hostname) => Ok(hostname.to_vec()), + } } /// Runs the probe `body` through `operations`, and returns the worker's answer: a report of what @@ -841,6 +949,22 @@ impl RootOperations for LibraryRoot { } } +impl EntryOperations for LibraryRoot { + /// Runs `Entry::prepare` on the assembled root. + fn prepare(&mut self, entrance: &Entrance) -> Result { + let Some(root) = &self.root else { + let missing = io::Error::new(io::ErrorKind::NotFound, "the worker holds no root"); + return Err(entry::Error::Root(missing)); + }; + entrance.entry(root.path()).prepare() + } + + /// Spawns `command`, whose child enters the execution with `Command::enter`. + fn spawn(&mut self, command: spawn::Command, prepared: Prepared) -> io::Result { + command.enter(prepared).spawn() + } +} + /// Runs PID 1 of a VM with a layered root, as the [module documentation](self) describes, with /// the worker on the tty `tty`, and returns the worker's status, or [`SESSION_STATUS`] if a child /// cannot start or the worker's status cannot be collected. diff --git a/tests/runtime-adapter/src/worker/executions.rs b/tests/runtime-adapter/src/worker/executions.rs index 63e14d080..fbabd9e5a 100644 --- a/tests/runtime-adapter/src/worker/executions.rs +++ b/tests/runtime-adapter/src/worker/executions.rs @@ -20,14 +20,15 @@ use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, WaitPidFlag, WaitStatus}; use nvx::exec::{Key, LaunchFailure, Runner, Settlement, Stream, Violation, runner}; use nvx::guest::cgroup::{ExecutionCgroup, Layout, Scope}; +use nvx::guest::entry::{self, AccountError, Capabilities, Capability, Prepared}; use nvx::guest::exec::binding::{Binding, Event}; use nvx::guest::exec::{Execution, Options, Pipes, Source}; use nvx::guest::process::Child; use nvx::guest::spawn::Command; use super::connection::{BACKLOG, Connection}; -use super::{MAX_EXECUTION_MEMORY, MAX_EXECUTIONS, MAX_HISTORY}; -use crate::exec::{GuestMessage, HostMessage, MAX_CHUNK, Start, name}; +use super::{Entrance, EntryOperations, MAX_EXECUTION_MEMORY, MAX_EXECUTIONS, MAX_HISTORY}; +use crate::exec::{AccountMode, GuestMessage, HostMessage, Identity, MAX_CHUNK, Start, name}; use crate::guest::ServeError; /// The device that a program without the host's input reads. @@ -63,6 +64,8 @@ pub(super) struct Executions { /// workload cgroup each execution gets a cgroup of its own. Without them, each program runs /// in a process group alone. cgroups: Option, + /// The sandbox in which the executions run. + sandbox: Sandbox, /// How many cgroups of executions the worker prepared, which numbers the next one. prepared: u64, /// The cgroups of the executions that did not settle, each of which keeps its place among @@ -85,6 +88,17 @@ pub(super) struct Executions { buffer: Vec, } +/// The sandbox in which the executions run once the root is ready: what every execution's +/// [`Entrance`] shares. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(super) struct Sandbox { + /// The executions' hostname, which the kernel command line configures. + pub(super) hostname: Vec, + /// The sandbox's workload identity, its user ID and group ID, if the root configuration + /// names one, as which every execution runs. + pub(super) identity: Option<(u32, u32)>, +} + /// One of the worker's executions: the binding of its runner to its program, and either the /// execution's cgroup, through which it settles, or, without cgroups, the worker's own handle of /// the program, whose status the binding leaves to the worker. @@ -102,8 +116,9 @@ struct Run { } impl Executions { - /// Returns no executions, which run in process groups alone until - /// [`run_in`](Self::run_in) names their cgroups. + /// Returns no executions, which run in process groups alone, with the default hostname and + /// without a workload identity, until [`run_in`](Self::run_in) names their cgroups and their + /// sandbox. pub(super) fn new() -> Self { Self { attached: BTreeMap::new(), @@ -112,6 +127,10 @@ impl Executions { reaped: None, cleared: None, cgroups: None, + sandbox: Sandbox { + hostname: entry::DEFAULT_HOSTNAME.as_bytes().to_vec(), + identity: None, + }, prepared: 0, fenced: Vec::new(), turn: 0, @@ -122,27 +141,31 @@ impl Executions { } } - /// Runs the executions that start from now on in `cgroups`: each in a cgroup of its own, - /// below the workload cgroup, or, without cgroups, in a process group alone. - pub(super) fn run_in(&mut self, cgroups: Option) { + /// Runs the executions that start from now on in `cgroups`, and in `sandbox`: each in a + /// cgroup of its own, below the workload cgroup, or, without cgroups, in a process group + /// alone. + pub(super) fn run_in(&mut self, cgroups: Option, sandbox: Sandbox) { self.cgroups = cgroups; + self.sandbox = sandbox; } /// Hands `message` from the host to its execution at `now`, and queues on `connection` the - /// terminal record of a start that the worker refuses. Before a cancellation, it gives the - /// execution a turn at `now`, and queues its messages on `connection`: the end of a program - /// that ended meanwhile, even before its timeout, while the worker did not run, reaches the - /// runner first, so that it stays the program's own, without the cause of the cancellation, - /// or of a timeout that `now` reached. + /// terminal record of a start that the worker refuses. A start's program enters the root + /// through `operations`. Before a cancellation, it gives the execution a turn at `now`, and + /// queues its messages on `connection`: the end of a program that ended meanwhile, even + /// before its timeout, while the worker did not run, reaches the runner first, so that it + /// stays the program's own, without the cause of the cancellation, or of a timeout that + /// `now` reached. pub(super) fn deliver( &mut self, now: Instant, message: HostMessage<'_>, + operations: &mut dyn EntryOperations, connection: &mut Connection, ) -> Result<(), ServeError> { let key = message.key(); if let HostMessage::Start(start) = message { - return self.start(now, start, connection); + return self.start(now, start, operations, connection); } // A message of an execution that the worker no longer runs is late. let Self { @@ -412,13 +435,15 @@ impl Executions { self.reaped = Some(now); } - /// Starts the execution of `start` at `now`, or refuses it with a terminal record that it - /// queues on `connection`. A start of an earlier generation than the latest that the host - /// started of its ID is stale, and one of the same generation fails. + /// Starts the execution of `start` at `now`, whose program enters the root through + /// `operations`, or refuses it with a terminal record that it queues on `connection`. A start + /// of an earlier generation than the latest that the host started of its ID is stale, and one + /// of the same generation fails. fn start( &mut self, now: Instant, start: Start, + operations: &mut dyn EntryOperations, connection: &mut Connection, ) -> Result<(), ServeError> { let key = start.key; @@ -450,7 +475,7 @@ impl Executions { } else if runner.memory_bound() > MAX_EXECUTION_MEMORY { Err(Errno::ENOMEM as i32) } else { - self.launch(&start) + self.launch(&start, operations) .map_err(|error| error.raw_os_error().unwrap_or(0)) }; match admitted { @@ -469,25 +494,39 @@ impl Executions { } } - /// Starts the program of `start`: with cgroups, in a cgroup of its own that the worker - /// prepares below the workload cgroup, and otherwise with the worker's own handle of the - /// program. The worker removes the cgroup of a program that it could not spawn. One that it - /// spawned, but whose execution could not be had, may still join the cgroup until the worker - /// collected its status, so that a removal could hide it once it ended: its cgroup keeps its - /// place, as that of an execution that did not settle. + /// Starts the program of `start`, which enters the root through `operations` as its + /// [entrance](Self::entrance) describes, once the entry prepared its execution: with cgroups, + /// in a cgroup of its own that the worker prepares below the workload cgroup, and otherwise + /// with the worker's own handle of the program. A start whose entry the root's entry refuses + /// fails with the `errno` that [`refusal`] returns, before the worker prepares a cgroup. The + /// worker removes the cgroup of a program that it could not spawn. One that it spawned, but + /// whose execution could not be had, may still join the cgroup until the worker collected its + /// status, so that a removal could hide it once it ended: its cgroup keeps its place, as that + /// of an execution that did not settle. fn launch( &mut self, start: &Start, + operations: &mut dyn EntryOperations, ) -> io::Result<(Execution, Option, Option)> { + let entrance = self.entrance(start)?; + let prepared = operations + .prepare(&entrance) + .map_err(|error| io::Error::from(refusal(&error)))?; let Some(cgroups) = &self.cgroups else { - let (execution, program) = - spawn(start, None, &mut self.reaping).map_err(|unstarted| unstarted.error)?; + let (execution, program) = spawn(start, prepared, None, operations, &mut self.reaping) + .map_err(|unstarted| unstarted.error)?; return Ok((execution, program, None)); }; self.prepared += 1; let name = format!("execution-{}", self.prepared); let cgroup = ExecutionCgroup::prepare(cgroups, Some(&name), &[])?; - match spawn(start, Some(&cgroup), &mut self.reaping) { + match spawn( + start, + prepared, + Some(&cgroup), + operations, + &mut self.reaping, + ) { Ok((execution, program)) => Ok((execution, program, Some(cgroup))), Err(Unstarted { error, spawned }) => { if spawned || cgroup.remove().is_err() { @@ -498,6 +537,22 @@ impl Executions { } } + /// Returns how the program of `start` enters the root: with the sandbox's hostname, as the + /// identity that exactly one source names, the sandbox's workload identity, if it has one, or + /// otherwise the identity that the start names. A start that names an identity in a sandbox + /// with a workload identity, or none in a sandbox without one, is `EINVAL`. + fn entrance(&self, start: &Start) -> io::Result { + let identity = match (self.sandbox.identity, &start.identity) { + (Some((uid, gid)), None) => entry::Identity::new(uid, gid), + (None, Some(identity)) => entry_identity(identity), + _ => return Err(Errno::EINVAL.into()), + }; + Ok(Entrance { + identity, + hostname: self.sandbox.hostname.clone(), + }) + } + /// Remembers that the host started `key`, the latest generation of its ID, and forgets the /// ID that it heard of first once it remembers more than [`MAX_HISTORY`]. fn remember(&mut self, key: Key) { @@ -650,23 +705,31 @@ impl From for Unstarted { } /// Spawns the program of `start` with pipes, in a process group of its own, and in `cgroup`, if -/// any, and returns its execution. In a cgroup, the execution settles through the cgroup, and -/// collects the program's status itself. Otherwise, it leaves the status to the worker, and comes -/// with the worker's own handle of the program. If the program started, but the execution or the -/// handle could not be had, the program ends, with its process group or the processes of its -/// cgroup, and the program's ID goes to `reaping`, so that the worker collects its status once it -/// ended, without waiting for it; the failure then tells that the worker spawned the program. +/// any, through `operations`, whose spawn enters the root as `prepared` prepared it, which makes +/// the program lead a session of its own, and returns its execution. The program starts in the +/// start's working directory, or the root's `/`, with exactly the start's environment. In a +/// cgroup, the execution settles through the cgroup, and collects the program's status itself. +/// Otherwise, it leaves the status to the worker, and comes with the worker's own handle of the +/// program. If the program started, but the execution or the handle could not be had, the +/// program ends, with its process group or the processes of its cgroup, and the program's ID goes +/// to `reaping`, so that the worker collects its status once it ended, without waiting for it; +/// the failure then tells that the worker spawned the program. fn spawn( start: &Start, + prepared: Prepared, cgroup: Option<&ExecutionCgroup>, + operations: &mut dyn EntryOperations, reaping: &mut Vec, ) -> Result<(Execution, Option), Unstarted> { - let arguments = start - .arguments - .iter() - .map(|argument| CString::new(argument.as_slice())) - .collect::, _>>() - .map_err(io::Error::from)?; + let strings = |items: &[Vec]| { + items + .iter() + .map(|item| CString::new(item.as_slice())) + .collect::, _>>() + .map_err(io::Error::from) + }; + let arguments = strings(&start.arguments)?; + let environment = strings(&start.environment)?; let program = arguments.first().cloned().unwrap_or_default(); let (pipes, ends) = Pipes::new(start.input)?; let stdin = match ends.stdin { @@ -677,11 +740,16 @@ fn spawn( .stdin(stdin) .stdout(ends.stdout) .stderr(ends.stderr) - .new_process_group(); + .new_process_group() + .environment(environment); + if let Some(directory) = &start.working_directory { + let directory = CString::new(directory.as_slice()).map_err(io::Error::from)?; + command = command.working_directory(directory); + } if let Some(cgroup) = cgroup { command = command.cgroup(cgroup.placement()?); } - let spawned = command.spawn()?; + let spawned = operations.spawn(command, prepared)?; let id = spawned.id(); let options = match cgroup { Some(cgroup) => Options::new().cgroup(cgroup, Scope::Execution), @@ -760,6 +828,42 @@ fn refused(key: Key, violation: Violation) -> ServeError { ServeError::Execution(format!("execution {}: {violation}", name(key))) } +/// Returns the `errno` with which the worker refuses a start whose entry `error` refused, at +/// stage 0: `EPERM` for an identity that the entry does not permit, with a capability outside +/// its allowlist, with any capability but user ID 0's, or with `CAP_NET_BIND_SERVICE` or +/// `CAP_NET_RAW` while listeners are published; `ENOENT` for an identity whose account the root +/// lacks in existing-account mode, without the user ID's passwd entry or the group ID's group +/// entry; the error's own for an account database or a root that cannot be read; and `EINVAL` +/// otherwise. +fn refusal(error: &entry::Error) -> Errno { + match error { + entry::Error::Capability(_) + | entry::Error::NonRootCapabilities + | entry::Error::Published(_) => Errno::EPERM, + entry::Error::Account(AccountError::Missing | AccountError::Group) => Errno::ENOENT, + entry::Error::Account(AccountError::Read(_, error)) | entry::Error::Root(error) => { + error.raw_os_error().map_or(Errno::EIO, Errno::from_raw) + } + _ => Errno::EINVAL, + } +} + +/// Returns the identity of `nvx::guest::entry` that a start's `identity` names. +fn entry_identity(identity: &Identity) -> entry::Identity { + let capabilities = (0..64) + .filter(|number| identity.capabilities & 1 << number != 0) + .filter_map(Capability::new) + .collect::(); + let account = match identity.account { + AccountMode::Existing => entry::AccountMode::Existing, + AccountMode::Numeric => entry::AccountMode::Numeric, + }; + entry::Identity::new(identity.uid, identity.gid) + .groups(identity.groups.clone()) + .capabilities(capabilities) + .account(account) +} + /// Reports that the worker cannot collect the status of the program of `binding`'s execution, /// with `error`: the worker no longer waits for the program, and the runner's deadlines bound /// the execution. diff --git a/tests/runtime-adapter/src/worker/executions/tests.rs b/tests/runtime-adapter/src/worker/executions/tests.rs index 85f7a97ac..bc861bf36 100644 --- a/tests/runtime-adapter/src/worker/executions/tests.rs +++ b/tests/runtime-adapter/src/worker/executions/tests.rs @@ -9,17 +9,44 @@ use std::time::{Duration, Instant}; use nix::poll::{self, PollFd, PollFlags, PollTimeout}; use nvx::exec::{Cause, Key, Limits, Outcome, Status, Timers, Window}; +use nvx::guest::entry::{self, Prepared}; use nvx::guest::process::Child; +use nvx::guest::spawn::{Command, Spawned}; use nvx::protocol::nvxs::{RecordParser, RecordType, encode_record}; use super::Executions; -use crate::exec::{GuestMessage, HostMessage, Start}; +use crate::exec::{AccountMode, GuestMessage, HostMessage, Identity, Start}; use crate::frame::{Decoder, Message}; use crate::worker::connection::{Connection, Received}; +use crate::worker::{Entrance, EntryOperations}; /// Longest wait for an execution's terminal outcome. const TIMEOUT: Duration = Duration::from_secs(30); +/// Entry operations that run each program without entering a root, which needs privileges: they +/// prepare each entry into the test's own root, `/`, whose account databases they read in +/// existing-account mode alone, which the [identities](numeric) of these tests do not use. +struct Unentered; + +impl EntryOperations for Unentered { + fn prepare(&mut self, entrance: &Entrance) -> Result { + entrance.entry("/").prepare() + } + + fn spawn(&mut self, command: Command, _prepared: Prepared) -> io::Result { + command.spawn() + } +} + +/// Returns the identity of user ID 0 and group ID 0, with numeric IDs, which the executions of +/// these tests name, as the worker runs them without a workload identity. +fn numeric() -> Identity { + Identity { + account: AccountMode::Numeric, + ..Identity::new(0, 0) + } +} + /// The broker's end of the worker's stream, which takes the worker's records as they arrive, and /// decodes the messages of their DATA. struct Broker { @@ -121,8 +148,11 @@ fn a_cancellation_after_the_timeout_leaves_a_program_that_ended_before_it_its_ow let timers = Timers::new(Duration::from_millis(300), Duration::from_secs(5)).timeout(timeout); let arguments = [&b"/bin/sh"[..], b"-c", b"exit 0"]; let started = Instant::now(); - let start = HostMessage::Start(Start::new(key, limits, timers, arguments)); - executions.deliver(started, start, &mut connection).unwrap(); + let start = Start::new(key, limits, timers, arguments).with_identity(numeric()); + let start = HostMessage::Start(start); + executions + .deliver(started, start, &mut Unentered, &mut connection) + .unwrap(); { let program = executions.attached[&1].program.as_ref(); let pidfd = program.and_then(Child::pidfd).unwrap(); @@ -141,7 +171,7 @@ fn a_cancellation_after_the_timeout_leaves_a_program_that_ended_before_it_its_ow let cancel = HostMessage::Cancel(key, Cause::Cancelled); executions - .deliver(Instant::now(), cancel, &mut connection) + .deliver(Instant::now(), cancel, &mut Unentered, &mut connection) .unwrap(); let outcome = terminal(&mut executions, &mut connection, &mut broker); assert_eq!(outcome.status(), Some(Status::Exited(0))); diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 73ede8a60..027dea32c 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -10,27 +10,34 @@ //! the guest agent on a tty, which it puts in raw mode, as it serves a VM's control tty. //! //! The worker of a layered root runs its session with fake root operations, which record the -//! worker's calls and fail as a test asks: the tests cover the order of its stages, the host -//! export that it mounts, the mapping table that it applies in order, in parts, which outlives a -//! reset, a table of zero mappings, which does no work, the parts that it refuses, a mapping that -//! fails and latches the root, the start of the sandbox, and the executions and starts that it -//! refuses out of order, each stage that fails, the requests that it serves once the root is -//! ready, its stop, the messages that it refuses, and a broker that closes only its end of the -//! stream, which still receives the worker's replies. The binary's PID 1 of a layered root, and -//! its worker, run on a pseudo-terminal too, on the test's host, whose kernel command line names -//! no layered root, where PID 1 also collects the statuses that real-time signals leave, whatever -//! the names of its children. +//! worker's calls and fail as a test asks: the tests cover the order of its stages, the host export +//! that it mounts, the mapping table that it applies in order, in parts, which outlives a reset, a +//! table of zero mappings, which does no work, the parts that it refuses, a mapping that fails and +//! latches the root, the start of the sandbox, and the executions and starts that it refuses out of +//! order, each stage that fails, the hostname that the kernel command line configures, the account +//! check of a sandbox's workload identity, the requests that it serves once the root is ready, its +//! stop, the messages that it refuses, and a broker that closes only its end of the stream, which +//! still receives the worker's replies. The binary's PID 1 of a layered root, and its worker, run +//! on a pseudo-terminal too, on the test's host, whose kernel command line names no layered root, +//! where PID 1 also collects the statuses that real-time signals leave, whatever the names of its +//! children. //! //! The worker's executions run behind the test broker too, whose host side drives the engine's //! peers: the tests cover several executions at once, byte-exact, a stalled sink and a full //! stream to the host, which stall only what waits on them, executions that take turns at the //! stream, timeouts, cancellations, and stops, the starts, messages, and probes that the worker //! refuses, generations and resets, the descendants that a program leaves in its process group, -//! and executions on a tty. Where the tests can create cgroups, each fake root gives the worker a -//! workload cgroup of the test's own, below which the executions run in cgroups of their own, and -//! settle: the tests then also cover a descendant that leaves its program's process group, which -//! the settlement ends, and an execution that does not settle, which keeps its place until the -//! host stops the worker. Elsewhere, the programs run in process groups alone. +//! and executions on a tty. The fake root prepares each execution's entry, as the worker +//! describes it, in a directory of the tests' own that holds account databases, and then spawns +//! the program without entering a root, which needs privileges, in the tests' own root and +//! identity: the tests cover the identity that exactly one source names, the sandbox's or the +//! start's, the entry's inputs, a program's working directory and environment, and the starts +//! that the worker refuses for their identities. Where the tests can create cgroups, each fake +//! root gives the worker a workload cgroup of the test's own, below which the executions run in +//! cgroups of their own, and settle: the tests then also cover a descendant that leaves its +//! program's process group, which the settlement ends, and an execution that does not settle, +//! which keeps its place until the host stops the worker. Elsewhere, the programs run in process +//! groups alone. #![cfg(all(feature = "guest", target_os = "linux"))] use std::collections::BTreeMap; @@ -59,8 +66,10 @@ use nvx::exec::{ Cause, Key, LaunchFailure, Limits, Outcome, Peer, Pool, Status, Stream, Timers, Window, peer, }; use nvx::guest::cgroup::Layout; +use nvx::guest::entry::{self, Capabilities, Capability, Prepared}; use nvx::guest::mappings::{Access, Mapping}; use nvx::guest::nvxs::GuestConnectionError; +use nvx::guest::spawn::{self, Spawned}; use nvx::protocol::export::{EXPORT_DIRECTORY, ExportMode, HostExport}; use nvx::protocol::nvxs::{ ErrorCode, Identity, Record, RecordError, RecordParser, RecordType, SessionError, encode_record, @@ -71,14 +80,16 @@ use nvx::protocol::{ }; use nvx::uuid::Uuid; use runtime_adapter::GUEST_BUILD; -use runtime_adapter::exec::{GuestMessage, HostMessage, MAX_CHUNK, Start}; +use runtime_adapter::exec::{self, AccountMode, GuestMessage, HostMessage, MAX_CHUNK, Start}; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, Message}; use runtime_adapter::guest::{self, NO_ROOT, RECEIVE_WINDOW, ServeError, Served}; use runtime_adapter::probe::{Facts, Probe}; use runtime_adapter::root::{ JournalPolicy, MapFailure, OverlayMode, Ready, RootConfig, RootFailure, RootStage, encode_maps, }; -use runtime_adapter::worker::{self, MAX_EXECUTIONS, MapError, RootOperations, Worked}; +use runtime_adapter::worker::{ + self, Entrance, EntryOperations, MAX_EXECUTIONS, MapError, RootOperations, Worked, +}; /// Identity of the broker's first epoch. const FIRST: Identity = Identity { @@ -1119,6 +1130,8 @@ enum Call { MountExport(HostExport), /// The mappings were applied, in order. Map(Vec), + /// The entry of an execution, or of the sandbox's workload identity, was prepared. + Prepare(Entrance), /// The probe ran. Probe(Probe), /// The teardown ran. @@ -1295,6 +1308,59 @@ impl RootOperations for FakeRoot { } } +impl EntryOperations for FakeRoot { + /// Prepares the entry into the [accounts' root](accounts), whose account databases it reads. + fn prepare(&mut self, entrance: &Entrance) -> Result { + self.calls.push(Call::Prepare(entrance.clone())); + entrance.entry(accounts()).prepare() + } + + /// Spawns `command` without entering a root, which needs privileges: the program runs in the + /// test's own root, as the test's own identity, in the process group that it leads. + fn spawn(&mut self, command: spawn::Command, _prepared: Prepared) -> io::Result { + command.spawn() + } +} + +/// The user and group IDs of the workload's account in the [accounts' root](accounts). +const WORKLOAD: (u32, u32) = (1000, 1000); + +/// The user and group IDs of an account in the [accounts' root](accounts) whose primary group +/// has no group entry. +const ORPHAN: (u32, u32) = (2000, 2000); + +/// The `/etc/passwd` of the [accounts' root](accounts). +const PASSWD: &str = "root:x:0:0:root:/root:/bin/sh\n\ + workload:x:1000:1000:workload:/home/workload:/bin/sh\n\ + orphan:x:2000:2000:orphan:/home/orphan:/bin/sh\n"; + +/// The `/etc/group` of the [accounts' root](accounts). +const GROUP: &str = "root:x:0:\nstaff:x:50:\nworkload:x:1000:\n"; + +/// Returns the directory that stands for the assembled root of every fake root when the worker +/// prepares an entry: one of the tests' own, below the target directory, for the tests' user, +/// with the account databases [`PASSWD`] and [`GROUP`], and nothing else. +fn accounts() -> &'static Path { + static ROOT: OnceLock = OnceLock::new(); + ROOT.get_or_init(|| { + let uid = nix::unistd::getuid(); + let root = Path::new(env!("CARGO_TARGET_TMPDIR")).join(format!("accounts-{uid}")); + let etc = root.join("etc"); + fs::create_dir_all(&etc).unwrap(); + for (name, contents) in [("passwd", PASSWD), ("group", GROUP)] { + let path = etc.join(name); + if fs::read_to_string(&path).is_ok_and(|current| current == contents) { + continue; + } + // Another test process may read the database meanwhile, so it is replaced whole. + let staged = etc.join(format!("{name}.{}", std::process::id())); + fs::write(&staged, contents).unwrap(); + fs::rename(&staged, &path).unwrap(); + } + root + }) +} + /// Returns the facts that a [`FakeRoot`] finds with `probe`. fn found(probe: &Probe) -> Facts { let mut facts = Facts::new(); @@ -1425,6 +1491,15 @@ fn announcing(count: u32) -> RootConfig { } } +/// Returns the configuration of a sandbox whose executions run as the workload identity +/// [`WORKLOAD`], whose account the [accounts' root](accounts) holds. +fn sandbox() -> RootConfig { + RootConfig { + workload_identity: Some(WORKLOAD), + ..RootConfig::default() + } +} + /// Returns a mapping table of `count` mappings: the export's entry `NNN` at `/srv/table/NNN`, for /// each index NNN, read-only for the even ones. fn table(count: usize) -> Vec { @@ -1477,6 +1552,14 @@ fn exported(config: &RootConfig) -> Vec { ] } +/// Returns the kernel command line of `layout` that names the executions' hostname `hostname`. +fn naming_hostname(layout: &RootLayout, hostname: &str) -> String { + format!( + "console=hvc0 nvx_sandbox=1 {} nvx_hostname={hostname}", + layout.tokens().join(" ") + ) +} + /// Returns the probe that `text` names. fn probe(text: &str) -> Probe { Probe::parse(text).unwrap() @@ -1601,6 +1684,11 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { // The boot tokens of the layout that the worker assembles, after which a command line may name // more. let tokens = format!("console=hvc0 nvx_sandbox=1 {}", layout().tokens().join(" ")); + let lacking = RootConfig { + workload_identity: Some((4242, 4242)), + ..RootConfig::default() + }; + let long = naming_hostname(&layout(), &"h".repeat(65)); let cases = [ ( "a configuration of another version", @@ -1710,6 +1798,14 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { host export to map from", read(), ), + ( + "a hostname longer than 64 bytes", + FakeRoot::reading(long.as_bytes()), + config.encode(), + RootStage::Layout, + "the kernel command line names a hostname that is longer than 64 bytes", + read(), + ), ( "a bootstrap that failed", FakeRoot::failing(Operation::Bootstrap, "failed to create the agent's cgroup"), @@ -1744,6 +1840,20 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { "failed to mount the host export hostfs", exported(&config), ), + ( + "a workload identity whose account the root lacks", + FakeRoot::booting(&layout()), + lacking.encode(), + RootStage::Identity, + "cannot enter the root as the workload identity 4242:4242: the account is \ + unavailable: no passwd entry has the user ID", + vec![ + Call::CommandLine, + Call::Bootstrap(lacking.clone()), + Call::Assemble(lacking.clone(), layout()), + Call::Prepare(entrance(entry::Identity::new(4242, 4242))), + ], + ), ]; for (name, root, body, stage, problem, calls) in cases { let (mut broker, working) = start_worker(root); @@ -1765,6 +1875,75 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { } } +/// Returns the entrance of an execution as `identity`, with the default hostname. +fn entrance(identity: entry::Identity) -> Entrance { + Entrance { + identity, + hostname: entry::DEFAULT_HOSTNAME.as_bytes().to_vec(), + } +} + +/// Returns the entrances that the worker prepared through `root`, in order. +fn entrances(root: &FakeRoot) -> Vec { + root.calls + .iter() + .filter_map(|call| match call { + Call::Prepare(entrance) => Some(entrance.clone()), + _ => None, + }) + .collect() +} + +#[test] +fn the_worker_checks_the_account_of_the_sandbox_s_workload_identity_in_the_assembled_root() { + // The kernel command line names the executions' hostname, which the check's entry takes + // too, as every execution's does. + let root = FakeRoot::reading(naming_hostname(&layout(), "example").as_bytes()); + let (mut broker, working) = start_worker(root); + broker.attach(FIRST); + let config = sandbox(); + let messages = [Message::RootConfig(config.encode()), Message::Stop]; + let replies = broker.exchange(&messages, OUTER_MAX_PAYLOAD, 2); + assert_eq!(replies, [ready(), Message::Stopped]); + let (root, worked) = joined(working); + assert_eq!(worked.unwrap(), Worked::Stopped); + let checked = Entrance { + identity: entry::Identity::new(WORKLOAD.0, WORKLOAD.1), + hostname: b"example".to_vec(), + }; + let expected = [ + Call::CommandLine, + Call::Bootstrap(config.clone()), + Call::Assemble(config, layout()), + Call::Prepare(checked), + Call::Teardown, + ]; + assert_eq!(root.calls, expected); + + // An empty hostname is the default one, and only the first names it. + for (command_line, hostname) in [ + (naming_hostname(&layout(), ""), entry::DEFAULT_HOSTNAME), + ( + format!( + "{} nvx_hostname=second", + naming_hostname(&layout(), "first") + ), + "first", + ), + ] { + let mut host = Host::configured(FakeRoot::reading(command_line.as_bytes()), &sandbox()); + host.start(start_of(1, shell("exit 0")), Vec::new()); + host.run_until(|host| host.outcome(1).is_some()); + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + let entrance = Entrance { + identity: entry::Identity::new(WORKLOAD.0, WORKLOAD.1), + hostname: hostname.as_bytes().to_vec(), + }; + assert_eq!(entrances(&root), [entrance.clone(), entrance], "{hostname}"); + } +} + #[test] fn the_worker_reports_a_teardown_that_failed_before_it_confirms_the_stop() { let problem = "failed to unmount the overlay: Device or resource busy (os error 16)"; @@ -2674,17 +2853,25 @@ struct Host { impl Host { /// Starts a worker whose root operations succeed, and returns the host, once the worker - /// assembled its root. + /// assembled its root for a [sandbox](sandbox) whose executions run as its workload identity. fn new() -> Self { - let (broker, working) = start_worker(FakeRoot::booting(&layout())); - Self::ready(broker, working) + Self::configured(FakeRoot::booting(&layout()), &sandbox()) + } + + /// Starts a worker with the root operations `root`, and returns the host, once the worker + /// assembled its root as `config` configures it. + fn configured(root: FakeRoot, config: &RootConfig) -> Self { + let (broker, working) = start_worker(root); + Self::ready(broker, working, config) } /// Attaches the session of the worker that runs in `working` in the first epoch, has it - /// assemble its root, starts the sandbox, of no mapping, and returns the host. - fn ready(mut broker: Broker, working: Working) -> Self { + /// assemble its root as `config` configures it, starts the sandbox, of no mapping, and returns + /// the host. + fn ready(mut broker: Broker, working: Working, config: &RootConfig) -> Self { broker.attach(FIRST); - let replies = broker.exchange(&[configuration(), Message::Start], OUTER_MAX_PAYLOAD, 2); + let messages = [Message::RootConfig(config.encode()), Message::Start]; + let replies = broker.exchange(&messages, OUTER_MAX_PAYLOAD, 2); assert_eq!(replies, [ready(), Message::Started]); Self { broker, @@ -2912,7 +3099,7 @@ fn the_worker_refuses_probes_while_it_runs_executions() { // runs: a probe that ran then would hold back the execution's timeout. let (root, gate) = FakeRoot::booting(&layout()).gated(); let (broker, working) = start_worker(root); - let mut host = Host::ready(broker, working); + let mut host = Host::ready(broker, working, &sandbox()); let timeout = Duration::from_millis(500); let timed = Start::new( Key::new(1, 1), @@ -3423,6 +3610,168 @@ fn fill_but_one(host: &mut Host, first: u64) { } } +#[test] +fn an_execution_runs_in_its_working_directory_with_exactly_its_environment() { + // In a sandbox with a workload identity, the worker prepares each execution's entry as it. + // Without privileges, the program runs in the test's own root, so that its working directory + // is one of the test's host. The first program prints its environment, which none of the + // worker's own joins, and the second its working directory. + let mut host = Host::new(); + let environment = ["GREETING=hello world", "EMPTY=", "TWICE=a=b"]; + let directory = fs::canonicalize(accounts()).unwrap(); + let directory = directory.to_str().unwrap(); + let printing = start_of(1, vec![b"/usr/bin/env".to_vec()]).with_environment(environment); + host.start(printing, Vec::new()); + let moved = start_of(2, shell("pwd -P")).with_working_directory(directory); + host.start(moved, Vec::new()); + host.run_until(|host| (1..=2).all(|id| host.outcome(id).is_some())); + for id in 1..=2 { + check_lossless(&host, id, Status::Exited(0)); + } + let printed = environment.map(|entry| format!("{entry}\n")).concat(); + assert_eq!(host.executions[&1].output[0], printed.as_bytes()); + assert_eq!( + host.executions[&2].output[0], + format!("{directory}\n").as_bytes() + ); + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + // The configuration's check, then each execution. + let workload = entrance(entry::Identity::new(WORKLOAD.0, WORKLOAD.1)); + assert_eq!( + entrances(&root), + [workload.clone(), workload.clone(), workload] + ); +} + +#[test] +fn an_execution_enters_the_root_as_the_identity_that_its_start_names() { + // In a sandbox without a workload identity, each start names an identity, with its + // supplementary groups, its capabilities, and its account mode, which the worker prepares an + // entry with, and the hostname that the kernel command line configures. + let root = FakeRoot::reading(naming_hostname(&layout(), "example").as_bytes()); + let mut host = Host::configured(root, &RootConfig::default()); + let bit = |capability: Capability| 1 << capability.number(); + let identities = [ + exec::Identity { + groups: vec![50], + capabilities: bit(Capability::CHOWN) | bit(Capability::KILL), + ..exec::Identity::new(0, 0) + }, + exec::Identity { + groups: vec![50, WORKLOAD.1], + ..exec::Identity::new(WORKLOAD.0, WORKLOAD.1) + }, + exec::Identity { + account: AccountMode::Numeric, + ..exec::Identity::new(4242, 4243) + }, + ]; + for (id, identity) in (1..).zip(&identities) { + let start = start_of(id, shell("exit 0")).with_identity(identity.clone()); + host.start(start, Vec::new()); + } + host.run_until(|host| (1..=3).all(|id| host.outcome(id).is_some())); + for id in 1..=3 { + check_lossless(&host, id, Status::Exited(0)); + } + let (root, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + let capabilities = Capabilities::EMPTY + .with(Capability::CHOWN) + .with(Capability::KILL); + let expected = [ + entry::Identity::new(0, 0) + .groups([50]) + .capabilities(capabilities), + entry::Identity::new(WORKLOAD.0, WORKLOAD.1).groups([50, WORKLOAD.1]), + entry::Identity::new(4242, 4243).account(entry::AccountMode::Numeric), + ] + .map(|identity| Entrance { + identity, + hostname: b"example".to_vec(), + }); + assert_eq!(entrances(&root), expected); +} + +#[test] +fn the_worker_refuses_at_stage_0_a_start_whose_identity_breaks_the_rules() { + // Each refusal's terminal record tells its `errno`, and the refusal takes no place. + fn refused(host: &mut Host, cases: Vec<(Option, Errno)>) { + let ids = 1..=u64::try_from(cases.len()).unwrap(); + for (id, (identity, _)) in ids.clone().zip(&cases) { + let mut start = start_of(id, shell("exit 0")); + start.identity = identity.clone(); + host.start(start, Vec::new()); + } + host.run_until(|host| ids.clone().all(|id| host.outcome(id).is_some())); + for (id, (identity, error)) in ids.zip(&cases) { + let failure = LaunchFailure::new(0, *error as i32); + let outcome = host.outcome(id).unwrap(); + assert_eq!(outcome.launch_failure(), Some(failure), "{identity:?}"); + } + } + + // In a sandbox with a workload identity, a start names none. + let mut host = Host::new(); + let workload = exec::Identity::new(WORKLOAD.0, WORKLOAD.1); + refused(&mut host, vec![(Some(workload), Errno::EINVAL)]); + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); + + // In a sandbox without one, a start names its own, which the entry checks: its + // capabilities, its IDs, and its account in existing-account mode. + let mut host = Host::configured(FakeRoot::booting(&layout()), &RootConfig::default()); + let root = |capability: Capability| exec::Identity { + capabilities: 1 << capability.number(), + ..exec::Identity::new(0, 0) + }; + let cases = vec![ + (None, Errno::EINVAL), + ( + Some(exec::Identity { + capabilities: 1 << 21, + ..exec::Identity::new(0, 0) + }), + Errno::EPERM, + ), + ( + Some(exec::Identity { + capabilities: 1 << Capability::CHOWN.number(), + ..exec::Identity::new(WORKLOAD.0, WORKLOAD.1) + }), + Errno::EPERM, + ), + (Some(root(Capability::NET_BIND_SERVICE)), Errno::EPERM), + (Some(root(Capability::NET_RAW)), Errno::EPERM), + (Some(exec::Identity::new(4242, 4242)), Errno::ENOENT), + (Some(exec::Identity::new(ORPHAN.0, ORPHAN.1)), Errno::ENOENT), + (Some(exec::Identity::new(WORKLOAD.0, 50)), Errno::EINVAL), + ( + Some(exec::Identity { + account: AccountMode::Numeric, + ..exec::Identity::new(u32::MAX, 0) + }), + Errno::EINVAL, + ), + ]; + let first = u64::try_from(cases.len()).unwrap() + 1; + refused(&mut host, cases); + let most = u64::try_from(MAX_EXECUTIONS).unwrap(); + let ids = first..first + most; + for id in ids.clone() { + let start = start_of(id, shell("exec sleep 60")) + .with_identity(exec::Identity::new(WORKLOAD.0, WORKLOAD.1)); + host.start(start, Vec::new()); + } + host.run_until(|host| { + ids.clone() + .all(|id| host.executions[&id].peer.has_started()) + }); + let (_, worked) = host.stop(); + assert_eq!(worked.unwrap(), Worked::Stopped); +} + #[test] fn the_worker_collects_the_status_of_a_program_that_a_real_time_signal_ended() { // `nix` decodes no real-time signal in a status, so that the worker learns that the program @@ -3460,7 +3809,7 @@ fn an_execution_that_starts_with_the_root_counts_its_timeout_from_its_own_start( stopping: false, }; host.send(&[ - configuration(), + Message::RootConfig(sandbox().encode()), Message::Start, HostMessage::Start(start).encode(), ]); @@ -3560,7 +3909,7 @@ fn the_worker_runs_executions_on_a_tty() { let worked = worker::work_on_terminal(&path, &mut root); (root, worked) }); - let mut host = Host::ready(broker, working); + let mut host = Host::ready(broker, working, &sandbox()); let script = "seq 1 100000; seq 1 1000 >&2"; host.start(start_of(1, shell(script)), Vec::new()); host.run_until(|host| host.outcome(1).is_some()); From cdc93ad690d6615274c532630f4800b055ed875b Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sat, 24 Oct 2026 10:01:14 -0700 Subject: [PATCH 5/6] tests: give the runtime adapter's executions the entry's default hostname The worker took the executions' hostname from the kernel command line's nvx_hostname= parameter. That parameter belongs to NVX's own boot contract, not to the runtime adapter: an adapter's launch need not carry it, and the adapter configures everything else through its root configuration. The executions now take nvx::guest::entry's default hostname, nvx-sandbox, which the worker's Sandbox::new names in one place, so that a sandbox's own hostname can later be set there. The worker no longer reads the parameter, and its layout stage no longer refuses a long one. No root-configuration field, frame field, or host option names a hostname. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/README.md | 102 ++++++++--------- tests/runtime-adapter/src/root.rs | 2 +- tests/runtime-adapter/src/worker.rs | 108 ++++++------------ .../runtime-adapter/src/worker/executions.rs | 19 ++- tests/runtime-adapter/tests/guest_session.rs | 74 ++---------- 5 files changed, 113 insertions(+), 192 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index d78f0bca3..03f84aedd 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -142,7 +142,7 @@ then ends with the stage's status: | Code | Stage | What failed | Status | | --- | --- | --- | --- | | 1 | `configuration` | Receiving and checking the root configuration | 11 | -| 2 | `layout` | Reading the boot layout of the root, the executions' hostname, and the host export from the kernel command line, and checking that the worker assembles them, with an export if the configuration announces mappings | 12 | +| 2 | `layout` | Reading the boot layout of the root and the host export from the kernel command line, and checking that the worker assembles them, with an export if the configuration announces mappings | 12 | | 3 | `bootstrap` | Bootstrapping the guest for the root: its cgroups and their resource values | 13 | | 4 | `assembly` | Assembling the root, such as from an image of another identity than the boot tokens expect, and mounting the host export that the kernel command line names | 14 | | 5 | `teardown` | Tearing the root down, such as while a process holds it open | 15 | @@ -428,15 +428,14 @@ then: 1. attaches the session; 2. receives the host's root configuration, and checks it; 3. reads the boot layout of the root from the kernel command line, with - `nvx::protocol::root::RootLayout::parse`, the executions' hostname, and the - host export, if the command line names one, with - `nvx::protocol::export::HostExport::parse`, then bootstraps the guest's - cgroups with `nvx::guest::root::bootstrap`, which moves the guest's processes - into the agent's cgroup, assembles the root with - `nvx::guest::root::assemble`, and mounts the export out of the root with - `Root::mount_export`. If the configuration names a workload identity, it then - checks the identity's account in the assembled root, in existing-account - mode, with `nvx::guest::entry`; + `nvx::protocol::root::RootLayout::parse`, and the host export, if the + command line names one, with `nvx::protocol::export::HostExport::parse`, + then bootstraps the guest's cgroups with `nvx::guest::root::bootstrap`, + which moves the guest's processes into the agent's cgroup, assembles the + root with `nvx::guest::root::assemble`, and mounts the export out of the + root with `Root::mount_export`. If the configuration names a workload + identity, it then checks the identity's account in the assembled root, in + existing-account mode, with `nvx::guest::entry`; 4. answers `Ready`, with the guest agent's build, or `RootFailed` with the stage that failed, and then ends; 5. applies the host's [mapping table](#the-mapping-table-and-the-start) @@ -460,22 +459,18 @@ configures a scratch for an upper in RAM, bounds a first partition for an EROFS image, or announces mappings while the kernel command line names no host export, from which they come, and a command line whose tokens of the export break their rules, as `HostExport::parse` checks them. An export that fails to mount fails -the assembly. The executions' hostname is the value of the kernel command line's -first `nvx_hostname=` parameter, which NVX's sandbox plans configure, and which -NVX's init agent reads too, or `nvx-sandbox`, -`nvx::guest::entry::DEFAULT_HOSTNAME`, without one, or with an empty one; a -hostname of more than 64 bytes fails the layout stage. A workload identity whose -account the assembled root lacks fails the identity stage. The worker ends with -0 once the host stopped it, after a teardown that succeeded, with the stage's -status, from 11 to 16, once a stage failed, and with 1 once its session failed -or the broker closed the tty, each but the first after a diagnostic on standard -error. Once the broker closed the stream at a record boundary, which may close -only the broker's end, the worker drops its executions, and writes what it -queued, as once the host stopped it. Before it exits, it closes what its probes -hold and the host export's root, then unmounts a root that it still holds, as -`nvx::guest::root::Root` does when it drops, ignoring failures. A teardown that -failed leaves the root mounted until the VM ends. The worker runs the root's -operations through `worker::RootOperations`, and enters its executions through +the assembly. A workload identity whose account the assembled root lacks fails +the identity stage. The worker ends with 0 once the host stopped it, after a +teardown that succeeded, with the stage's status, from 11 to 16, once a stage +failed, and with 1 once its session failed or the broker closed the tty, each +but the first after a diagnostic on standard error. Once the broker closed the +stream at a record boundary, which may close only the broker's end, the worker +drops its executions, and writes what it queued, as once the host stopped it. +Before it exits, it closes what its probes hold and the host export's root, then +unmounts a root that it still holds, as `nvx::guest::root::Root` does when it +drops, ignoring failures. A teardown that failed leaves the root mounted until +the VM ends. The worker runs the root's operations through +`worker::RootOperations`, and enters its executions through `worker::EntryOperations`, so that tests run it against a fake root. #### The mapping table and the start @@ -563,27 +558,28 @@ credit that it needs, never keeps the others' output from the host. of the refusal, or that of the preparation or the spawn. - Each program enters the layered root through `nvx::guest::entry`, as a `worker::Entrance` describes it: in new mount, PID, and UTS namespaces, with - the root moved over `/`, the configured hostname, the image's own machine ID, - and no time status, so that the entry binds nothing into the root, and with - published listeners, the entry's default, until the adapter tells whether - its listeners are published. It runs as the identity that exactly one source - names: the sandbox's workload identity, with no supplementary groups or - capabilities, or, in a sandbox without one, the identity that its start - names, with its supplementary groups, its capabilities, and its account - mode. A start that names an identity in a sandbox with a workload identity, - or none in a sandbox without one, is refused with `EINVAL`. Before the worker - prepares the execution's cgroup, the entry checks the identity, and reads its - account in existing-account mode: it refuses with `EPERM` a capability - outside its allowlist, any capability of a user ID other than 0, and - `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while listeners are published, with - `ENOENT` an identity without the user ID's passwd entry or the group ID's - group entry, with the error of an account database or a root that it cannot - read, and with `EINVAL` otherwise. The program starts in the working - directory of its start, or in the root's `/`, with exactly the environment - of its start, none of the worker's own, and with its standard input from the - host or the null device. It leads a session of its own, without a - controlling terminal, and the process group that the session creates, and, - as PID 1 of its PID namespace, its end ends every process of the namespace. + the root moved over `/`, the entry's default hostname, `nvx-sandbox`, the + image's own machine ID, and no time status, so that the entry binds nothing + into the root, and with published listeners, the entry's default, until the + adapter tells whether its listeners are published. It runs as the identity + that exactly one source names: the sandbox's workload identity, with no + supplementary groups or capabilities, or, in a sandbox without one, the + identity that its start names, with its supplementary groups, its + capabilities, and its account mode. A start that names an identity in a + sandbox with a workload identity, or none in a sandbox without one, is refused + with `EINVAL`. Before the worker prepares the execution's cgroup, the entry + checks the identity, and reads its account in existing-account mode: it + refuses with `EPERM` a capability outside its allowlist, any capability of a + user ID other than 0, and `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while + listeners are published, with `ENOENT` an identity without the user ID's + passwd entry or the group ID's group entry, with the error of an account + database or a root that it cannot read, and with `EINVAL` otherwise. The + program starts in the working directory of its start, or in the root's `/`, + with exactly the environment of its start, none of the worker's own, and with + its standard input from the host or the null device. It leads a session of its + own, without a controlling terminal, and the process group that the session + creates, and, as PID 1 of its PID namespace, its end ends every process of the + namespace. - Each execution runs in a cgroup of its own below the workload cgroup of the root configuration, `execution-N`, where N counts the cgroups that the worker prepared. The worker prepares it with @@ -1149,12 +1145,10 @@ refuses, which latches the root, after either of which it refuses the later parts and the start; the executions and starts that it refuses out of order, before the start and after it; the failure of each stage, the tokens of a host export that break their rules, mappings without an export, an export that fails -to mount, a hostname of more than 64 bytes, and a workload identity whose -account the root lacks among them, which the worker reports before it ends with -the stage's status, without the operations of the later stages; the account -check of a sandbox's workload identity, once the root is assembled, with the -hostname of the kernel command line, the default one without it or with an empty -one, and the first of two; a teardown +to mount, and a workload identity whose account the root lacks among them, which +the worker reports before it ends with the stage's status, without the +operations of the later stages; the account check of a sandbox's workload +identity, once the root is assembled, with the default hostname; a teardown that fails; before the root, the configuration and the `Stop` that it accepts, and the messages that it refuses; a second configuration and a broker that closes the stream, which leave the root that it assembled to its operations; @@ -1238,7 +1232,7 @@ tests cover: environment of its start; - the identities that starts name in a sandbox without a workload identity, with their supplementary groups, capabilities, and account modes, and the - configured hostname, all of which the worker's entry takes; + default hostname, all of which the worker's entry takes; - the starts that the worker refuses at stage 0 for their identities: one that names an identity in a sandbox with a workload identity, or none in a sandbox without one, with `EINVAL`; a capability outside the entry's diff --git a/tests/runtime-adapter/src/root.rs b/tests/runtime-adapter/src/root.rs index 8eaffb7d5..3435d5056 100644 --- a/tests/runtime-adapter/src/root.rs +++ b/tests/runtime-adapter/src/root.rs @@ -520,7 +520,7 @@ pub enum RootStage { /// Receiving and checking the root configuration. Configuration, /// Reading the boot layout of the root from the kernel command line, and checking that the - /// guest can assemble it, and the hostname of the executions there. + /// guest can assemble it. Layout, /// Bootstrapping the guest for the root: its cgroups, and user namespaces. Bootstrap, diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index 92a9c081c..c2db2b503 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -21,11 +21,11 @@ //! 1. It opens the tty and attaches the session. //! 2. It receives the host's root configuration, a [`RootConfig`], and checks it. //! 3. It reads the boot layout of the root from the kernel command line with -//! `nvx::protocol::root::RootLayout::parse`, the executions' hostname, and the host export, if -//! the command line names one, with `nvx::protocol::export::HostExport::parse`, then bootstraps -//! the guest, assembles the root, and mounts the export out of the root. If the configuration -//! names a workload identity for the whole sandbox, it then checks the identity's account in the -//! assembled root, in existing-account mode, through `nvx::guest::entry`. +//! `nvx::protocol::root::RootLayout::parse`, and the host export, if the command line names one, +//! with `nvx::protocol::export::HostExport::parse`, then bootstraps the guest, assembles the +//! root, and mounts the export out of the root. If the configuration names a workload identity +//! for the whole sandbox, it then checks the identity's account in the assembled root, in +//! existing-account mode, through `nvx::guest::entry`. //! 4. It answers `Ready`, with the guest agent's build, [`GUEST_BUILD`], or `RootFailed` with the //! stage that failed, and then ends. //! 5. It applies the host's mapping table beneath the root, from the export, in the parts that @@ -46,11 +46,8 @@ //! configures a scratch for a root over RAM, bounds a first partition for an EROFS image, or //! announces mappings while the kernel command line names no host export, from which they come, //! and a command line whose tokens of the export break their rules. An export that fails to mount -//! fails the assembly. The executions' hostname is the value of the kernel command line's first -//! `nvx_hostname=` parameter, which NVX's sandbox plans configure, and which NVX's init agent reads -//! too, or `nvx-sandbox`, `nvx::guest::entry::DEFAULT_HOSTNAME`, without one, or with an empty one. -//! A hostname of more than 64 bytes fails the layout stage. A workload identity whose account the -//! assembled root lacks fails the identity stage. +//! fails the assembly. A workload identity whose account the assembled root lacks fails the +//! identity stage. //! //! The worker, and with it the VM, ends with 0 once the host stopped it, after a teardown that //! succeeded, with the stage's [exit status](RootStage::exit_status) once a stage failed, from 11 @@ -127,23 +124,23 @@ //! at stage 0, with `EAGAIN`, `ENOMEM`, the `errno` of the refusal, or that of the preparation //! or the spawn. //! - Each program enters the assembled root through `nvx::guest::entry`, as an [`Entrance`] -//! describes it: in new mount, PID, and UTS namespaces, with the root moved over `/`, the -//! configured hostname, the image's own machine ID, and no time status, so that the entry binds -//! nothing into the root, and with published listeners, the entry's default. It runs as the -//! identity that exactly one source names: the sandbox's workload identity, which the root +//! describes it: in new mount, PID, and UTS namespaces, with the root moved over `/`, the entry's +//! default hostname, `nvx-sandbox`, the image's own machine ID, and no time status, so that the +//! entry binds nothing into the root, and with published listeners, the entry's default. It runs +//! as the identity that exactly one source names: the sandbox's workload identity, which the root //! configuration names, with no supplementary groups or capabilities, or, in a sandbox without //! one, the identity that its start names, with its supplementary groups, its capabilities, and //! its account mode. A start that names an identity in a sandbox with a workload identity, or //! none in a sandbox without one, is refused with `EINVAL`. Before the worker prepares the -//! execution's cgroup, the entry checks the identity, and reads its account in -//! existing-account mode: it refuses with `EPERM` a capability outside its allowlist, any -//! capability of a user ID other than 0, and `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while -//! listeners are published, with `ENOENT` an identity without the user ID's passwd entry or -//! the group ID's group entry, with the error of an account database or a root that it cannot -//! read, and with `EINVAL` otherwise. The program starts in the working directory of its start, -//! or in the root's `/`, with exactly the environment of its start, and leads a session of its -//! own, without a controlling terminal, and the process group that the session creates. As PID -//! 1 of its PID namespace, its end ends every process of the namespace. +//! execution's cgroup, the entry checks the identity, and reads its account in existing-account +//! mode: it refuses with `EPERM` a capability outside its allowlist, any capability of a user ID +//! other than 0, and `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` while listeners are published, with +//! `ENOENT` an identity without the user ID's passwd entry or the group ID's group entry, with +//! the error of an account database or a root that it cannot read, and with `EINVAL` otherwise. +//! The program starts in the working directory of its start, or in the root's `/`, with exactly +//! the environment of its start, and leads a session of its own, without a controlling terminal, +//! and the process group that the session creates. As PID 1 of its PID namespace, its end ends +//! every process of the namespace. //! - Each execution runs in a cgroup of its own below the workload cgroup, `execution-N`, where N //! counts the cgroups that the worker prepared. The worker prepares it with //! `nvx::guest::cgroup::ExecutionCgroup::prepare`, and the program joins it as it spawns, @@ -266,10 +263,6 @@ pub const MAX_HISTORY: usize = 1024; /// The kernel command line, from which the worker reads the boot layout of the root. const COMMAND_LINE: &str = "/proc/cmdline"; -/// The kernel command line's parameter that names the executions' hostname, as NVX's init agent -/// reads it. -const HOSTNAME_PARAMETER: &[u8] = b"nvx_hostname="; - /// How one execution enters the assembled root: the inputs of its `nvx::guest::entry::Entry` that /// the worker chooses. The entry keeps its defaults for the rest: new mount, PID, and UTS /// namespaces, with the root moved over `/`, the image's own machine ID and no time status, which @@ -279,7 +272,7 @@ const HOSTNAME_PARAMETER: &[u8] = b"nvx_hostname="; pub struct Entrance { /// The identity: the sandbox's workload identity, or the one that the execution's start names. pub identity: Identity, - /// The hostname that the kernel command line configures. + /// The hostname, the sandbox's: the entry's default. pub hostname: Vec, } @@ -610,19 +603,19 @@ fn serve_root( }) } -/// Checks the root configuration `body`, reads the boot layout of the root, the executions' -/// hostname, and the host export, bootstraps the guest, assembles the root, mounts the export out -/// of it, if the kernel command line names one, and checks the account of the sandbox's workload -/// identity there, if the configuration names one, through `operations`. Returns the configuration, -/// with the sandbox in which the executions run, or the failure of the stage that failed: the -/// export's mount ends the assembly. +/// Checks the root configuration `body`, reads the boot layout of the root and the host export, +/// bootstraps the guest, assembles the root, mounts the export out of it, if the kernel command +/// line names one, and checks the account of the sandbox's workload identity there, if the +/// configuration names one, through `operations`. Returns the configuration, with the sandbox in +/// which the executions run, or the failure of the stage that failed: the export's mount ends the +/// assembly. fn assemble( operations: &mut dyn RootOperations, body: &[u8], ) -> Result<(RootConfig, Sandbox), RootFailure> { let config = RootConfig::decode(body) .map_err(|error| RootFailure::new(RootStage::Configuration, error.problem()))?; - let (layout, export, hostname) = layout(operations, &config) + let (layout, export) = layout(operations, &config) .map_err(|problem| RootFailure::new(RootStage::Layout, problem))?; operations .bootstrap(&config) @@ -635,10 +628,11 @@ fn assemble( .mount_export(export) .map_err(|problem| RootFailure::new(RootStage::Assembly, problem))?; } - if let Some((uid, gid)) = config.workload_identity { + let sandbox = Sandbox::new(config.workload_identity); + if let Some((uid, gid)) = sandbox.identity { let entrance = Entrance { identity: Identity::new(uid, gid), - hostname: hostname.clone(), + hostname: sandbox.hostname.clone(), }; operations.prepare(&entrance).map_err(|error| { RootFailure::new( @@ -647,22 +641,18 @@ fn assemble( ) })?; } - let sandbox = Sandbox { - hostname, - identity: config.workload_identity, - }; Ok((config, sandbox)) } -/// Reads the boot layout of the root, the executions' hostname, and the host export, if any, from -/// the kernel command line, through `operations`, and checks that the worker assembles them as -/// `config` configures it: an EROFS image or a GPT disk, over an ext4 scratch, or over RAM if -/// `config` configures no scratch, with a bound of the first partition for a GPT disk alone, and -/// with a host export, the source of the mapping table, if `config` announces mappings. +/// Reads the boot layout of the root, and the host export, if any, from the kernel command line, +/// through `operations`, and checks that the worker assembles them as `config` configures it: an +/// EROFS image or a GPT disk, over an ext4 scratch, or over RAM if `config` configures no scratch, +/// with a bound of the first partition for a GPT disk alone, and with a host export, the source of +/// the mapping table, if `config` announces mappings. fn layout( operations: &mut dyn RootOperations, config: &RootConfig, -) -> Result<(RootLayout, Option, Vec), String> { +) -> Result<(RootLayout, Option), String> { let command_line = operations.command_line()?; let layout = RootLayout::parse(&command_line) .map_err(|error| format!("the kernel command line has invalid boot tokens: {error}"))? @@ -693,29 +683,7 @@ fn layout( line names no host export to map from" )); } - Ok((layout, export, hostname(&command_line)?)) -} - -/// Returns the hostname that the kernel command line `command_line` configures, as NVX's init -/// agent reads it: the value of its first `nvx_hostname=` parameter, or -/// [`DEFAULT_HOSTNAME`](entry::DEFAULT_HOSTNAME) if it has none, or an empty one. A hostname of -/// more than [`MAX_HOSTNAME_BYTES`](entry::MAX_HOSTNAME_BYTES), or with a NUL, is refused. -fn hostname(command_line: &[u8]) -> Result, String> { - let named = command_line - .split(u8::is_ascii_whitespace) - .find_map(|parameter| parameter.strip_prefix(HOSTNAME_PARAMETER)); - match named { - None | Some([]) => Ok(entry::DEFAULT_HOSTNAME.as_bytes().to_vec()), - Some(hostname) if hostname.len() > entry::MAX_HOSTNAME_BYTES || hostname.contains(&0) => { - Err(format!( - "the kernel command line names a hostname that is longer than {} bytes, or \ - holds a NUL: {:?}", - entry::MAX_HOSTNAME_BYTES, - String::from_utf8_lossy(hostname) - )) - } - Some(hostname) => Ok(hostname.to_vec()), - } + Ok((layout, export)) } /// Runs the probe `body` through `operations`, and returns the worker's answer: a report of what diff --git a/tests/runtime-adapter/src/worker/executions.rs b/tests/runtime-adapter/src/worker/executions.rs index fbabd9e5a..a92782a98 100644 --- a/tests/runtime-adapter/src/worker/executions.rs +++ b/tests/runtime-adapter/src/worker/executions.rs @@ -92,13 +92,25 @@ pub(super) struct Executions { /// [`Entrance`] shares. #[derive(Clone, Debug, PartialEq, Eq)] pub(super) struct Sandbox { - /// The executions' hostname, which the kernel command line configures. + /// The executions' hostname. pub(super) hostname: Vec, /// The sandbox's workload identity, its user ID and group ID, if the root configuration /// names one, as which every execution runs. pub(super) identity: Option<(u32, u32)>, } +impl Sandbox { + /// Returns the sandbox whose workload identity is `identity`, if it has one. Its executions + /// take the entry's default hostname, [`DEFAULT_HOSTNAME`](entry::DEFAULT_HOSTNAME), as + /// nothing configures another: this is the one place that names their hostname. + pub(super) fn new(identity: Option<(u32, u32)>) -> Self { + Self { + hostname: entry::DEFAULT_HOSTNAME.as_bytes().to_vec(), + identity, + } + } +} + /// One of the worker's executions: the binding of its runner to its program, and either the /// execution's cgroup, through which it settles, or, without cgroups, the worker's own handle of /// the program, whose status the binding leaves to the worker. @@ -127,10 +139,7 @@ impl Executions { reaped: None, cleared: None, cgroups: None, - sandbox: Sandbox { - hostname: entry::DEFAULT_HOSTNAME.as_bytes().to_vec(), - identity: None, - }, + sandbox: Sandbox::new(None), prepared: 0, fenced: Vec::new(), turn: 0, diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 027dea32c..556bd7fb7 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -14,13 +14,12 @@ //! that it mounts, the mapping table that it applies in order, in parts, which outlives a reset, a //! table of zero mappings, which does no work, the parts that it refuses, a mapping that fails and //! latches the root, the start of the sandbox, and the executions and starts that it refuses out of -//! order, each stage that fails, the hostname that the kernel command line configures, the account -//! check of a sandbox's workload identity, the requests that it serves once the root is ready, its -//! stop, the messages that it refuses, and a broker that closes only its end of the stream, which -//! still receives the worker's replies. The binary's PID 1 of a layered root, and its worker, run -//! on a pseudo-terminal too, on the test's host, whose kernel command line names no layered root, -//! where PID 1 also collects the statuses that real-time signals leave, whatever the names of its -//! children. +//! order, each stage that fails, the account check of a sandbox's workload identity, the requests +//! that it serves once the root is ready, its stop, the messages that it refuses, and a broker that +//! closes only its end of the stream, which still receives the worker's replies. The binary's PID 1 +//! of a layered root, and its worker, run on a pseudo-terminal too, on the test's host, whose +//! kernel command line names no layered root, where PID 1 also collects the statuses that real-time +//! signals leave, whatever the names of its children. //! //! The worker's executions run behind the test broker too, whose host side drives the engine's //! peers: the tests cover several executions at once, byte-exact, a stalled sink and a full @@ -1552,14 +1551,6 @@ fn exported(config: &RootConfig) -> Vec { ] } -/// Returns the kernel command line of `layout` that names the executions' hostname `hostname`. -fn naming_hostname(layout: &RootLayout, hostname: &str) -> String { - format!( - "console=hvc0 nvx_sandbox=1 {} nvx_hostname={hostname}", - layout.tokens().join(" ") - ) -} - /// Returns the probe that `text` names. fn probe(text: &str) -> Probe { Probe::parse(text).unwrap() @@ -1688,7 +1679,6 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { workload_identity: Some((4242, 4242)), ..RootConfig::default() }; - let long = naming_hostname(&layout(), &"h".repeat(65)); let cases = [ ( "a configuration of another version", @@ -1798,14 +1788,6 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { host export to map from", read(), ), - ( - "a hostname longer than 64 bytes", - FakeRoot::reading(long.as_bytes()), - config.encode(), - RootStage::Layout, - "the kernel command line names a hostname that is longer than 64 bytes", - read(), - ), ( "a bootstrap that failed", FakeRoot::failing(Operation::Bootstrap, "failed to create the agent's cgroup"), @@ -1896,10 +1878,8 @@ fn entrances(root: &FakeRoot) -> Vec { #[test] fn the_worker_checks_the_account_of_the_sandbox_s_workload_identity_in_the_assembled_root() { - // The kernel command line names the executions' hostname, which the check's entry takes - // too, as every execution's does. - let root = FakeRoot::reading(naming_hostname(&layout(), "example").as_bytes()); - let (mut broker, working) = start_worker(root); + // The check's entry takes the default hostname, as every execution's does. + let (mut broker, working) = start_worker(FakeRoot::booting(&layout())); broker.attach(FIRST); let config = sandbox(); let messages = [Message::RootConfig(config.encode()), Message::Stop]; @@ -1907,10 +1887,7 @@ fn the_worker_checks_the_account_of_the_sandbox_s_workload_identity_in_the_assem assert_eq!(replies, [ready(), Message::Stopped]); let (root, worked) = joined(working); assert_eq!(worked.unwrap(), Worked::Stopped); - let checked = Entrance { - identity: entry::Identity::new(WORKLOAD.0, WORKLOAD.1), - hostname: b"example".to_vec(), - }; + let checked = entrance(entry::Identity::new(WORKLOAD.0, WORKLOAD.1)); let expected = [ Call::CommandLine, Call::Bootstrap(config.clone()), @@ -1919,29 +1896,6 @@ fn the_worker_checks_the_account_of_the_sandbox_s_workload_identity_in_the_assem Call::Teardown, ]; assert_eq!(root.calls, expected); - - // An empty hostname is the default one, and only the first names it. - for (command_line, hostname) in [ - (naming_hostname(&layout(), ""), entry::DEFAULT_HOSTNAME), - ( - format!( - "{} nvx_hostname=second", - naming_hostname(&layout(), "first") - ), - "first", - ), - ] { - let mut host = Host::configured(FakeRoot::reading(command_line.as_bytes()), &sandbox()); - host.start(start_of(1, shell("exit 0")), Vec::new()); - host.run_until(|host| host.outcome(1).is_some()); - let (root, worked) = host.stop(); - assert_eq!(worked.unwrap(), Worked::Stopped); - let entrance = Entrance { - identity: entry::Identity::new(WORKLOAD.0, WORKLOAD.1), - hostname: hostname.as_bytes().to_vec(), - }; - assert_eq!(entrances(&root), [entrance.clone(), entrance], "{hostname}"); - } } #[test] @@ -3648,9 +3602,8 @@ fn an_execution_runs_in_its_working_directory_with_exactly_its_environment() { fn an_execution_enters_the_root_as_the_identity_that_its_start_names() { // In a sandbox without a workload identity, each start names an identity, with its // supplementary groups, its capabilities, and its account mode, which the worker prepares an - // entry with, and the hostname that the kernel command line configures. - let root = FakeRoot::reading(naming_hostname(&layout(), "example").as_bytes()); - let mut host = Host::configured(root, &RootConfig::default()); + // entry with, and the default hostname. + let mut host = Host::configured(FakeRoot::booting(&layout()), &RootConfig::default()); let bit = |capability: Capability| 1 << capability.number(); let identities = [ exec::Identity { @@ -3687,10 +3640,7 @@ fn an_execution_enters_the_root_as_the_identity_that_its_start_names() { entry::Identity::new(WORKLOAD.0, WORKLOAD.1).groups([50, WORKLOAD.1]), entry::Identity::new(4242, 4243).account(entry::AccountMode::Numeric), ] - .map(|identity| Entrance { - identity, - hostname: b"example".to_vec(), - }); + .map(entrance); assert_eq!(entrances(&root), expected); } From 6fca426f4a6a09d372e579da882a0ed6749137c7 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 26 Oct 2026 14:29:02 -0700 Subject: [PATCH 6/6] tests: count the identity stage among the worker's exit statuses The worker exits with 16 once the root's identity stage fails, as its stages' statuses now run from 11 for the configuration to 16 for the identity, but the guest agent's usage still gave them as 11 to 15. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90 --- tests/runtime-adapter/src/bin/runtime-adapter-guest.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs b/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs index da8206ffd..761c4c460 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-guest.rs @@ -18,7 +18,7 @@ //! - `runtime-adapter-guest worker TTY [SIBLING]` runs the worker on the tty TTY: it assembles the //! layered root that the host configures, and serves the host program until the host stops it. //! SIBLING is the process ID of PID 1's unrelated child, which its probes check. It exits with 0 -//! once the host stopped it, with 11 to 15 once a stage of the root failed, and with 1 once its +//! once the host stopped it, with 11 to 16 once a stage of the root failed, and with 1 once its //! session failed or the broker closed the tty, each but the first after a diagnostic on //! standard error. //! - `runtime-adapter-guest idle` sleeps until a signal ends it.