diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index fecb00b1d..01f8f12aa 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -22,11 +22,12 @@ stages and plans with NVX's typed plan for an adapter's own guest, whose staged bindings a launch checks before it hands OpenVMM the checked files, and which the guest agent's worker assembles through `nvx::guest::root`, in a process of its own rather than PID 1, as the host program's root configuration asks. The -host program may share host directories and files with the root, which it -exports with NVX's `filesystem::plan`, and which the worker maps beneath the -root from the host's mapping table, before the host starts the sandbox. The host -program then checks the root from inside the VM with the worker's probes, and -runs executions in it, several at once over one session, in a +plan also holds the VM's [network](#networks), if any. The host program may +share host directories and files with the root, which it exports with NVX's +`filesystem::plan`, and which the worker maps beneath the root from the host's +mapping table, before the host starts the sandbox. The host program then checks +the root from inside the VM with the worker's probes, and runs executions in +it, several at once over one session, in a [table of scenarios](#layered-scenarios), to which each new check adds a row. ## Configurations @@ -860,15 +861,15 @@ tokens of the root alone: the image's, such as `nvx_layer=distro,0xd0003000,erofs:UUID`, and the upper's, such as `nvx_scratch=0xd0006000,ext4`. OpenVMM's arguments are the plan's, which attach the [checked files](#the-checked-handoff) of the image and the scratch -as block devices, then those of the export of the launch's -[shares](#shares), if any, then the VM in one process, NVX's default memory, -the kernel, the initramfs, that command line, no virtio console, the control -console with the launch capability on standard input, and the outcome report. -They name no workload identity, and no managed lifecycle: OpenVMM only checks -the arguments of the managed lifecycle and labels its outcome report with it, -and only NVX's init agent reads its boot token. A VM without a layered root -keeps the managed lifecycle, with which NVX's `/init` starts the managed agent, -which the guest agent replaces. +as block devices, and the VM's [network](#networks), if any, then those of the +export of the launch's [shares](#shares), if any, then the VM in one process, +NVX's default memory, the kernel, the initramfs, that command line, no virtio +console, the control console with the launch capability on standard input, +and the outcome report. They name no workload identity, and no managed +lifecycle: OpenVMM only checks the arguments of the managed lifecycle and +labels its outcome report with it, and only NVX's init agent reads its boot +token. A VM without a layered root keeps the managed lifecycle, with which +NVX's `/init` starts the managed agent, which the guest agent replaces. The boot tokens carry only what NVX's boot contract defines. The launch sends the rest in the [root configuration](#root-configurations), once it attaches: @@ -889,11 +890,12 @@ runtime records, in `runtime-adapter.json`, before it spawns the VMM: `mode` is `layered`, or `blockless` for a VM without a layered root, whose `plan`, `bindings`, and `rootConfiguration` are `null`. `plan` is the persisted -form of the VM's `SandboxPlan`. `bindings` holds the bindings that the launch -checked, which the store [stages](#staged-images) for the launch's images: the -image's, and the scratch image's, `null` for an upper in RAM. -`rootConfiguration` is the root configuration that the launch sends, whether -or not the guest accepts it. +form of the VM's `SandboxPlan`, whose `network` keeps the VM's +[network](#networks), or is `null` without one. `bindings` holds the bindings +that the launch checked, which the store [stages](#staged-images) for the +launch's images: the image's, and the scratch image's, `null` for an upper in +RAM. `rootConfiguration` is the root configuration that the launch sends, +whether or not the guest accepts it. ### The checked handoff @@ -950,6 +952,42 @@ planned. Once the guest is ready, the launch sends the table, `LayeredLaunch::table`, in the [host's order](#the-hosts-order): each mapping before those inside its target. +### Networks + +A layered launch may give the VM a network: an +`nvx::host::sandbox::SandboxNetwork`, the guest's address and the policy that +OpenVMM enforces there. `layered::read_network` reads it from a file, in the +serde form of a plan's network, as NVX's plan persists it, such as: + +```json +{"guestNetwork": "10.0.0.2/24", "egress": "deny", + "allow": {"name": "egress.allow", + "rules": [{"to": [{"cidr": "192.0.2.0/24", "except": []}], + "ports": [{"protocol": "tcp", "port": 443, "endPort": null}]}]}, + "deny": {"name": "egress.deny", "rules": []}, + "ingress": "deny", "hostLoopback": "deny", "hostLoopbackForwards": [], + "proxy": null} +``` + +The form requires every member and refuses any other. Reading it checks the +network as the plan does: the guest's address must leave a gateway, and the +policy must lower to OpenVMM's arguments. A launch reads its network before +anything else, so it refuses one that the plan refuses before it needs NVX's +artifacts or creates the store. + +The launch puts the network on the adapter's plan, with +`SandboxPlan::with_network`, which renders it after the block devices, through +NVX's one lowering of a policy, `nvx::host::network::openvmm_arguments`: the +portable profile at the guest's address, the defaults of egress and ingress, +the host-loopback access, the egress rules, lowered into OpenVMM's native +ones, the host-loopback forwards, and the proxy of a proxy-only network, which +the guest reaches at its gateway, at the URL that `SandboxPlan::proxy_url` +gives. A policy that lets nothing through attaches no network device. OpenVMM +appends the guest's network settings to the kernel command line, from which +NVX's `/init` configures the network device before it hands the VM to the +guest agent. The launch's [record](#layered-roots) keeps the network in its +plan. + ### The step hook For tests, `launch` takes a step hook, `hook::StepHook`, from the variable @@ -1003,7 +1041,7 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from ends its VM. LAYERED, - `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [--workload-identity UID:GID] [SHARES]`, + `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [--workload-identity UID:GID] [--network FILE] [SHARES]`, gives the VM a [layered root](#layered-roots). Its image is the file PATH, an EROFS file system, `erofs`, or a GPT disk, `gpt`, of which the guest expects the identity UUID: the file system's UUID, or the disk's GUID. Its upper is @@ -1015,11 +1053,14 @@ the directory that `NVX_ARTIFACTS_DIR` names, and the guest agent's build from takes, with the guest's defaults unless given. A workload identity, the user ID UID and the group ID GID, two numbers from 1 to 4294967294, is the identity as which every execution of the sandbox runs: the guest checks its account in - the assembled root, without which the root's `identity` stage fails. Before it - spawns the VM, the launch checks the bindings that STORE stages for the image - and the scratch image, and fails if STORE stages none for either, or one no - longer holds, as [the checked handoff](#the-checked-handoff) describes. It - records the bindings that it checked. + the assembled root, without which the root's `identity` stage fails. The VM + has the [network](#networks) that FILE holds, if given, which the launch + reads before anything else: one that the plan refuses fails the launch before + it needs NVX's artifacts or creates the store. Before it spawns the VM, the + launch checks the bindings that STORE stages for the image and the scratch + image, and fails if STORE stages none for either, or one no longer holds, as + [the checked handoff](#the-checked-handoff) describes. It records the + bindings that it checked. SHARES, `--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]... [--share-root DIR]...`, @@ -1409,11 +1450,19 @@ Windows: canonical guest paths, targets that take or cover the adapter's or the container root's reserved targets, a read-only share inside a read-write one, and a hidden path that does not exist inside a share; +- the networks that a launch reads: the OpenVMM arguments of each element of a + policy, which NVX's lowering renders after the plan's block devices, a policy + that lets nothing through, which attaches no network device, the proxy's + URL, and the record, which keeps the network as the file gave it; and the + files that it refuses, each for the plan's reason; - `runtime-adapter-host stage`, which records the staged images, a staging that fails, which leaves them, and a staging that replaces them; - `runtime-adapter-host launch` with shares that it cannot export, which it refuses with status 1 before it creates the store, with artifacts that boot nothing; +- `runtime-adapter-host launch --network`, which refuses a network that the + plan refuses, or that it cannot read, with status 1 before it needs NVX's + artifacts or creates the store, and passes one that the plan takes; - the command lines that the host program refuses, each with status 2, those of `exec` and of `--workload-identity` included. diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 6af8e7017..c446c9a1d 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -21,23 +21,28 @@ //! //! LAYERED, `--distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) //! [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY] -//! [--first-partition-bound BYTES] [--workload-identity UID:GID] [SHARES]`, gives the VM a -//! layered root, which the guest agent's worker assembles; see `runtime_adapter::layered` and -//! `runtime_adapter::worker`. Its image is the raw file PATH, an EROFS filesystem, `erofs`, or a -//! GPT disk, `gpt`, of which the guest expects the identity UUID: the file system's UUID, or the -//! disk's GUID. Its upper is the ext4 scratch image PATH, or RAM. The guest assembles the root -//! below its runtime directory DIR, `/run/runtime-adapter` by default, in the overlay mode MODE, -//! `trusted-lowers` or `untrusted-lowers`, checks a scratch's journal against the policy POLICY, -//! `allowed` or `forbidden`, and refuses a first partition of a GPT disk larger than BYTES, with -//! the guest's defaults, `trusted-lowers`, `allowed`, and 64 KiB, unless given. A workload -//! identity, the user ID UID and the group ID GID, two numbers from 1 to 4294967294, is the -//! identity as which every execution of the sandbox runs: the guest checks its account in the -//! assembled root, without which the root's `identity` stage fails. Before it spawns the VM, the -//! launch checks the bindings that STORE stages for the image and the scratch image, from their -//! metadata alone, and fails if STORE stages none for either, or one no longer holds, as for an -//! image that changed after it was staged. It hands OpenVMM the files that the checks opened, -//! which it holds until the guest has assembled the root, or until it has torn down a launch that -//! failed, and records the bindings. +//! [--first-partition-bound BYTES] [--workload-identity UID:GID] [--network FILE] [SHARES]`, +//! gives the VM a layered root, which the guest agent's worker assembles; see +//! `runtime_adapter::layered` and `runtime_adapter::worker`. Its image is the raw file PATH, an +//! EROFS filesystem, `erofs`, or a GPT disk, `gpt`, of which the guest expects the identity UUID: +//! the file system's UUID, or the disk's GUID. Its upper is the ext4 scratch image PATH, or RAM. +//! The guest assembles the root below its runtime directory DIR, `/run/runtime-adapter` by +//! default, in the overlay mode MODE, `trusted-lowers` or `untrusted-lowers`, checks a scratch's +//! journal against the policy POLICY, `allowed` or `forbidden`, and refuses a first partition of +//! a GPT disk larger than BYTES, with the guest's defaults, `trusted-lowers`, `allowed`, and 64 +//! KiB, unless given. A workload identity, the user ID UID and the group ID GID, two numbers from +//! 1 to 4294967294, is the identity as which every execution of the sandbox runs: the guest +//! checks its account in the assembled root, without which the root's `identity` stage fails. The +//! VM has a network if FILE names one: an `nvx::host::sandbox::SandboxNetwork`, the guest's +//! address and the policy that OpenVMM enforces there, in the serde form of a plan's network, +//! which the launch reads before anything else, so that it refuses a network that the plan +//! refuses before it needs NVX's artifacts or the store, and puts on the adapter's plan, which +//! renders it; see `runtime_adapter::layered::read_network`. Before it spawns the VM, the launch +//! checks the bindings that STORE stages for the image and the scratch image, from their metadata +//! alone, and fails if STORE stages none for either, or one no longer holds, as for an image that +//! changed after it was staged. It hands OpenVMM the files that the checks opened, which it holds +//! until the guest has assembled the root, or until it has torn down a launch that failed, and +//! records the plan, with its network, and the bindings. //! //! SHARES, `--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]... //! [--share-root DIR]...`, shares each host directory or regular file HOST with the root, at the @@ -171,7 +176,7 @@ use runtime_adapter::frame::{MAX_BODY, Message}; use runtime_adapter::hook::{Step, StepHook}; use runtime_adapter::host::{self, Client, ClientError, Executions, OrderError}; use runtime_adapter::initramfs; -use runtime_adapter::layered::{LaunchRecord, LayeredLaunch, Share, Shares, Staged}; +use runtime_adapter::layered::{LaunchRecord, LayeredLaunch, Share, Shares, Staged, read_network}; use runtime_adapter::probe::{Probe, ProbeReply, escape}; use runtime_adapter::root::{JournalPolicy, OverlayMode, RootFailure}; use runtime_adapter::runtime::Runtime; @@ -224,7 +229,7 @@ const USAGE: &str = concat!( " --upper (scratch:PATH | ram) [--runtime-dir DIR]\n", " [--overlay-mode MODE] [--scratch-journal POLICY]", " [--first-partition-bound BYTES]\n", - " [--workload-identity UID:GID]\n", + " [--workload-identity UID:GID] [--network FILE]\n", " [--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]...", " [--share-root DIR]...]]\n", " runtime-adapter-host (reattach | stop) STORE [--guest-build BUILD]\n", @@ -376,6 +381,8 @@ struct Layered { workload_identity: Option<(u32, u32)>, /// The host paths that the launch shares with the root. shares: Shares, + /// The file of the VM's network, if it has one. + network: Option, } /// The options of a command. @@ -405,6 +412,8 @@ struct Options { workload_identity: Option<(u32, u32)>, /// `--share`, `--deny`, `--reserved`, and `--share-root`, in order. shares: Shares, + /// `--network`. + network: Option, } fn main() -> ExitCode { @@ -556,6 +565,7 @@ fn parse(arguments: Vec) -> Result { first_partition_bound: options.first_partition_bound, workload_identity: options.workload_identity, shares: options.shares, + network: options.network, })) } (None, Some(_), _) => { @@ -573,6 +583,7 @@ fn parse(arguments: Vec) -> Result { ), ("--workload-identity", options.workload_identity.is_some()), ("--share", !options.shares.shares.is_empty()), + ("--network", options.network.is_some()), ]; if let Some((option, _)) = layered.iter().find(|(_, given)| *given) { return Err(format!("{option} needs --distro")); @@ -701,6 +712,9 @@ fn parse_option( ("launch", "--deny") => options.shares.denied.push(PathBuf::from(value)), ("launch", "--reserved") => options.shares.reserved.push(text(value)?), ("launch", "--share-root") => options.shares.roots.push(PathBuf::from(value)), + ("launch", "--network") if options.network.is_none() => { + options.network = Some(PathBuf::from(value)); + } (_, "--guest-build") if name != "stage" && options.guest_build.is_none() => { options.guest_build = Some(text(value)?); } @@ -873,6 +887,13 @@ fn launch( layered: Option<&Layered>, hook: Option<&StepHook>, ) -> Result { + // A network that the plan refuses fails the launch before it needs NVX's artifacts or the + // store. + let network = layered + .and_then(|layered| layered.network.as_deref()) + .map(read_network) + .transpose() + .map_err(|error| describe(&error))?; let hypervisor = match hypervisor { Some(hypervisor) => hypervisor, None => Hypervisor::from_env_or_default().map_err(|error| describe(&error))?, @@ -899,6 +920,7 @@ fn launch( if let Some(export) = layered.shares.plan()? { launch.share(export)?; } + launch.plan = launch.plan.with_network(network); Ok(launch) }) .transpose() diff --git a/tests/runtime-adapter/src/layered.rs b/tests/runtime-adapter/src/layered.rs index 8d69338fc..b052613e0 100644 --- a/tests/runtime-adapter/src/layered.rs +++ b/tests/runtime-adapter/src/layered.rs @@ -30,8 +30,17 @@ //! export, out of the root. The root configuration announces how many mappings the export's table //! holds, which the host sends once the guest is ready; see [`LayeredLaunch::table`]. //! +//! A launch may give the VM a network: a `SandboxNetwork`, the guest's address and the policy that +//! OpenVMM enforces there, which [`read_network`] reads from a file in the serde form of a plan's +//! network, and which the launch puts on the plan with `SandboxPlan::with_network`. The plan +//! renders its OpenVMM arguments through NVX's one lowering of a policy, +//! `nvx::host::network::openvmm_arguments`, as it renders those of NVX's sandbox profile. OpenVMM +//! appends the guest's network settings to the kernel command line itself, and NVX's `/init` +//! configures the network device from them before it hands the VM to the guest agent. +//! //! The host records what it launched in the sandbox's store, beside NVX's runtime records, as a -//! [`LaunchRecord`], with the bindings that the launch checked. +//! [`LaunchRecord`], with the bindings that the launch checked, and the plan, which keeps the +//! network. use std::ffi::{OsStr, OsString}; use std::fs; @@ -47,7 +56,8 @@ use nvx::host::launch::DEFAULT_MEMORY_MIB; use nvx::host::sandbox::distro::{DISTRO_POLICY_REVISION, DistroBinding}; use nvx::host::sandbox::scratch::{CLAIM_RELEASE_WAIT, ScratchBinding}; use nvx::host::sandbox::{ - LayerRole, PreparedLaunch, SandboxDistro, SandboxLayer, SandboxPlan, SandboxUpper, + LayerRole, PreparedLaunch, SandboxDistro, SandboxLayer, SandboxNetwork, SandboxPlan, + SandboxUpper, }; use nvx::protocol::HostMap; use nvx::protocol::root::{ImageFormat, RootImage}; @@ -78,7 +88,8 @@ const SECTOR_SIZE: u64 = 512; /// shares with the root, if any. #[derive(Clone, Debug, PartialEq, Eq)] pub struct LayeredLaunch { - /// The adapter's plan of the VM's block devices and of the boot tokens of its root. + /// The adapter's plan of the VM's block devices, of its network, if it has one, and of the boot + /// tokens of its root. pub plan: SandboxPlan, /// The root configuration that the host sends the guest agent once it attaches. pub config: RootConfig, @@ -94,7 +105,8 @@ pub struct LayeredLaunch { impl LayeredLaunch { /// Returns the launch of the image at `distro`, in `format`, of which the guest expects the /// identity `identity`, over the ext4 scratch image at `scratch`, or over RAM without one, - /// without bindings, and without shares. + /// without bindings, without shares, and without a network, which `SandboxPlan::with_network` + /// puts on the plan. /// /// The root configuration takes its defaults, with the runtime directory `runtime` instead, if /// any, as it is, so that the guest checks it, and with the size of each image, if it is a @@ -227,13 +239,15 @@ impl LayeredLaunch { /// Returns OpenVMM's arguments for the VM of `prepared`, the launch that /// [`prepare`](Self::prepare) prepared: its arguments, which attach the checked files as the - /// plan's block devices, those of the [export](Self::export) of the launch's shares, if any, - /// which `ExportPlan::openvmm_arguments` renders, then the VM in one process on `hypervisor`, - /// NVX's default memory, the kernel at `kernel`, the initramfs at `initramfs`, its kernel - /// command line, which is the [launch's](Self::kernel_command_line), no virtio console, the - /// control console listening on `endpoint` with the launch capability on standard input, and - /// the outcome report at `report`. OpenVMM appends the boot tokens with which the guest mounts - /// the export to the kernel command line. + /// plan's block devices, and the plan's network device, with the policy that NVX's lowering + /// renders, if the plan has a network that lets anything through, those of the + /// [export](Self::export) of the launch's shares, if any, which `ExportPlan::openvmm_arguments` + /// renders, then the VM in one process on `hypervisor`, NVX's default memory, the kernel at + /// `kernel`, the initramfs at `initramfs`, its kernel command line, which is the + /// [launch's](Self::kernel_command_line), no virtio console, the control console listening on + /// `endpoint` with the launch capability on standard input, and the outcome report at `report`. + /// OpenVMM appends the boot tokens with which the guest mounts the export to the kernel command + /// line. /// /// Unlike those of NVX's sandbox start, they name no workload identity, which the adapter's /// plan does not hold, and no managed lifecycle: OpenVMM only checks the arguments of the @@ -445,6 +459,27 @@ fn authorize(roots: &[PathBuf], path: &Path) -> Result<()> { ))) } +/// Reads the network of a launch from the file at `path`: one `SandboxNetwork`, in the serde form +/// of a plan's network, as NVX's plan persists it, which `SandboxPlan::with_network` puts on the +/// adapter's plan. Without a network, a launch has no network device. +/// +/// The form requires every member and refuses any other, and reading it checks the network as +/// `SandboxNetwork::new` does: the guest's address must leave a gateway, and the policy must lower +/// to OpenVMM's arguments, as [`LayeredLaunch::openvmm_arguments`] renders them. So a launch that +/// reads its network first refuses one that the plan refuses before it launches anything. +/// +/// Fails with [`ErrorKind::Io`](nvx::ErrorKind::Io) if the file cannot be read, and with +/// [`ErrorKind::Unsupported`](nvx::ErrorKind::Unsupported), whose cause says why, if the file does +/// not hold a network in that form, or the plan refuses it. +pub fn read_network(path: &Path) -> Result { + let bytes = fs::read(path).map_err(|error| { + Error::io(format!("cannot read the network {}", path.display())).with_source(error) + })?; + serde_json::from_slice(&bytes).map_err(|error| { + Error::unsupported(format!("the network {} is refused", path.display())).with_source(error) + }) +} + /// Returns the adapter's plan of the image at `distro`, in `format`, of which the guest expects /// the identity `identity`, over the ext4 scratch image at `scratch`, or over RAM without one, /// checked against the host with `SandboxPlan::validated`. @@ -793,11 +828,12 @@ fn write_record(path: &Path, record: &Value) -> Result<()> { /// /// The `mode` is `layered`, or `blockless` for a VM without a layered root, whose `plan`, /// `bindings`, and `rootConfiguration` are `null`. The `plan` is the persisted form of the VM's -/// `SandboxPlan`. The `bindings` are those that the launch checked, which the store stages for the -/// plan's images, as [`Staged::bindings`] finds them, in the serde forms of NVX's `DistroBinding` -/// and `ScratchBinding`; the scratch image's is `null` for an upper in RAM, which has no scratch -/// image. The `rootConfiguration` is the encoding of the root configuration that the host sends, -/// as it sends it, whether or not the guest accepts it. +/// `SandboxPlan`, whose `network` keeps the VM's network, in the form that [`read_network`] +/// reads, or `null` without one. The `bindings` are those that the launch checked, which the store +/// stages for the plan's images, as [`Staged::bindings`] finds them, in the serde forms of NVX's +/// `DistroBinding` and `ScratchBinding`; the scratch image's is `null` for an upper in RAM, which +/// has no scratch image. The `rootConfiguration` is the encoding of the root configuration that +/// the host sends, as it sends it, whether or not the guest accepts it. #[derive(Clone, Copy, Debug)] pub struct LaunchRecord<'a> { /// The VM's kernel command line. diff --git a/tests/runtime-adapter/src/lib.rs b/tests/runtime-adapter/src/lib.rs index a8a775e0c..73881c937 100644 --- a/tests/runtime-adapter/src/lib.rs +++ b/tests/runtime-adapter/src/lib.rs @@ -27,18 +27,19 @@ //! `runtime-adapter-guest init` serves the VM's control tty. //! //! The host program also launches VMs with a layered root, of an image over an upper, which it -//! stages, plans with NVX's typed plan for an adapter's own guest, and hands OpenVMM as the files -//! that its checks of the staged bindings opened, with the host directories and files that it -//! shares with the root, which NVX's `filesystem::plan` exports, in the `layered` module. Its step -//! hook, in the `hook` module, lets tests pause a launch, or end it as a crash would, at a named -//! step. In those VMs, the guest agent's PID 1 starts a worker, in the `worker` module, which -//! assembles the root through `nvx::guest::root` outside PID 1, as the host's root configuration -//! of the [`root`] module asks, applies the host's mapping table from the host export, and runs -//! the [`probe`]s with which the host checks the root. The host runs that order, in the `host` -//! module: it checks that the guest agent that assembled the root is of the build that it ships, -//! sends the table, then starts the sandbox, and tears down a VM whose order failed. Once the host -//! started the sandbox, the worker also runs several [`exec`]utions at once through NVX's -//! execution engine and its guest binding, from one loop that never blocks. +//! stages, plans with NVX's typed plan for an adapter's own guest, which also renders the VM's +//! network, if any, and hands OpenVMM as the files that its checks of the staged bindings opened, +//! with the host directories and files that it shares with the root, which NVX's `filesystem::plan` +//! exports, in the `layered` module. Its step hook, in the `hook` module, lets tests pause a +//! launch, or end it as a crash would, at a named step. In those VMs, the guest agent's PID 1 +//! starts a worker, in the `worker` module, which assembles the root through `nvx::guest::root` +//! outside PID 1, as the host's root configuration of the [`root`] module asks, applies the host's +//! mapping table from the host export, and runs the [`probe`]s with which the host checks the root. +//! The host runs that order, in the `host` module: it checks that the guest agent that assembled +//! the root is of the build that it ships, sends the table, then starts the sandbox, and tears down +//! a VM whose order failed. Once the host started the sandbox, the worker also runs several +//! [`exec`]utions at once through NVX's execution engine and its guest binding, from one loop that +//! never blocks. pub mod exec; pub mod frame; diff --git a/tests/runtime-adapter/tests/layered_launch.rs b/tests/runtime-adapter/tests/layered_launch.rs index 209b8f8a2..85c6e8f15 100644 --- a/tests/runtime-adapter/tests/layered_launch.rs +++ b/tests/runtime-adapter/tests/layered_launch.rs @@ -1,11 +1,12 @@ //! The host program stages the images of a layered root, plans a VM with a layered root with NVX's -//! typed plan for an adapter's own guest, checks the staged bindings of its images and hands -//! OpenVMM the checked files, plans the export of the host paths that it shares with the root -//! inside the directories that the adapter authorizes, attaches it, and verifies it before the -//! launch, composes OpenVMM's arguments without NVX's managed lifecycle, records what it launches, -//! with the bindings that it checked, and refuses options that do not make a command, shares that -//! it cannot export before it touches the store, executions whose starts the guest would not take, -//! and more executions than the guest runs at once. The tests need no hypervisor. +//! typed plan for an adapter's own guest, with the network that it reads, checks the staged +//! bindings of its images and hands OpenVMM the checked files, plans the export of the host paths +//! that it shares with the root inside the directories that the adapter authorizes, attaches it, +//! and verifies it before the launch, composes OpenVMM's arguments without NVX's managed lifecycle, +//! records what it launches, with the bindings that it checked, and refuses options that do not +//! make a command, networks that the plan refuses before it launches anything, shares that it +//! cannot export before it touches the store, executions whose starts the guest would not take, and +//! more executions than the guest runs at once. The tests need no hypervisor. #![cfg(feature = "host")] use std::ffi::OsString; @@ -14,11 +15,12 @@ use std::io::Write; #[cfg(target_os = "linux")] use std::os::fd::AsRawFd; use std::path::{Path, PathBuf}; -use std::process::Command; +use std::process::{Command, Output}; use std::time::{Duration, Instant}; use nvx::host::artifacts::Artifacts; use nvx::host::filesystem::GUEST_EXPORT; +use nvx::host::network; use nvx::host::sandbox::SandboxPlan; use nvx::host::sandbox::distro::DistroFormat; use nvx::host::sandbox::scratch::CLAIM_RELEASE_WAIT; @@ -30,7 +32,7 @@ use runtime_adapter::exec::MAX_EXECUTIONS; use runtime_adapter::hook::STEP_HOOK; use runtime_adapter::layered::{ Bindings, LaunchRecord, LayeredLaunch, RECORD_NAME, SANDBOX_TOKEN, STAGED_NAME, Share, Shares, - Staged, + Staged, read_network, }; use runtime_adapter::root::RootConfig; use serde_json::{Value, json}; @@ -177,6 +179,56 @@ fn assert_refused(result: nvx::Result, problem: &str) { assert!(error.to_string().contains(problem), "{problem}: {error}"); } +/// Returns the serde form of an open network of a guest at `10.0.0.2/24`, from which the network +/// cases change one element at a time: egress allowed, without rules, ingress and host-loopback +/// access denied, no forwards, and no proxy. +fn open_network() -> Value { + json!({ + "guestNetwork": "10.0.0.2/24", + "egress": "allow", + "allow": {"name": "egress.allow", "rules": []}, + "deny": {"name": "egress.deny", "rules": []}, + "ingress": "deny", + "hostLoopback": "deny", + "hostLoopbackForwards": [], + "proxy": null, + }) +} + +/// Returns the serde form of the open network, changed by `edit`. +fn network_with(edit: impl FnOnce(&mut Value)) -> Value { + let mut network = open_network(); + edit(&mut network); + network +} + +/// Returns the OpenVMM arguments of a network of a guest at `guest`: the portable profile, with +/// the defaults `egress` and `ingress`, the host-loopback access `loopback`, or none for OpenVMM's +/// own, from the guest alone, then `more`. +fn portable<'a>( + guest: &'a str, + egress: &'a str, + ingress: &'a str, + loopback: Option<&'a str>, + more: &[&'a str], +) -> Vec<&'a str> { + let mut arguments = vec![ + "--net", + guest, + "--network-profile", + "portable", + "--network-egress", + egress, + "--network-ingress", + ingress, + ]; + if let Some(loopback) = loopback { + arguments.extend(["--host-loopback", loopback]); + } + arguments.extend_from_slice(more); + arguments +} + #[test] fn a_layered_launch_plans_its_blocks_and_boot_tokens_with_the_adapters_plan() { let directory = directory("layered-plan"); @@ -1049,6 +1101,460 @@ fn the_host_program_refuses_shares_before_it_touches_the_store() { } } +#[test] +fn a_launch_renders_each_element_of_its_network_through_the_plan() { + let directory = directory("layered-network"); + let distro = image(directory.join("distro.erofs"), 512); + let scratch = image(directory.join("scratch.ext4"), 512); + let (kernel, initramfs) = (directory.join("vmlinux"), directory.join("initramfs")); + let report = directory.join("outcome.json"); + let without = staged_launch(&distro, Some(&scratch)); + // OpenVMM's arguments for `launch`, prepared anew, after the plan's block devices, which name + // the checked files and come first. + let arguments = |launch: &LayeredLaunch| { + let prepared = launch.prepare().unwrap(); + let mut arguments = launch.openvmm_arguments( + &prepared, + Hypervisor::Kvm, + &kernel, + &initramfs, + "/run/control.sock", + &report, + ); + let distro_file = checked(prepared.distro().file(), &distro); + let scratch_file = checked(prepared.scratch().unwrap().file(), &scratch); + let blocks: [OsString; 6] = [ + "--machine".into(), + "microvm".into(), + "--microvm-sandbox-block".into(), + format!("distro:file:{distro_file},ro").into(), + "--microvm-sandbox-block".into(), + format!("scratch:file:{scratch_file}").into(), + ]; + assert_eq!(arguments[..blocks.len()], blocks); + arguments.split_off(blocks.len()) + }; + let options = arguments(&without); + let cases: [(&str, Value, Vec<&str>, Option<&str>); 13] = [ + ( + "open", + open_network(), + portable("10.0.0.2/24", "allow", "deny", Some("deny"), &[]), + None, + ), + ( + "guest-network", + network_with(|network| network["guestNetwork"] = json!("192.168.7.6/30")), + portable("192.168.7.6/30", "allow", "deny", Some("deny"), &[]), + None, + ), + // A policy that lets nothing through attaches no network device. + ( + "closed", + network_with(|network| network["egress"] = json!("deny")), + Vec::new(), + None, + ), + ( + "allow", + network_with(|network| { + network["egress"] = json!("deny"); + network["allow"]["rules"] = json!([{ + "to": [{"cidr": "192.0.2.0/24", "except": ["192.0.2.128/25"]}], + "ports": [{"protocol": "tcp", "port": 443, "endPort": null}], + }]); + }), + portable( + "10.0.0.2/24", + "deny", + "deny", + Some("deny"), + &["--network-egress-allow", "192.0.2.0/25:tcp:443"], + ), + None, + ), + ( + "allow-everywhere", + network_with(|network| { + network["egress"] = json!("deny"); + network["allow"]["rules"] = json!([{ + "to": [], + "ports": [{"protocol": "any", "port": 8000, "endPort": 8010}], + }]); + }), + portable( + "10.0.0.2/24", + "deny", + "deny", + Some("deny"), + &[ + "--network-egress-allow", + "0.0.0.0/0:tcp:8000-8010", + "--network-egress-allow", + "0.0.0.0/0:udp:8000-8010", + "--network-egress-allow", + "::/0:tcp:8000-8010", + "--network-egress-allow", + "::/0:udp:8000-8010", + ], + ), + None, + ), + ( + "deny", + network_with(|network| { + network["deny"]["rules"] = json!([ + { + "to": [{"cidr": "198.51.100.0/24", "except": []}], + "ports": [{"protocol": "udp", "port": 53, "endPort": null}], + }, + { + "to": [{"cidr": "2001:db8::/32", "except": []}], + "ports": [{"protocol": "icmp", "port": null, "endPort": null}], + }, + ]); + }), + portable( + "10.0.0.2/24", + "allow", + "deny", + Some("deny"), + &[ + "--network-egress-deny", + "198.51.100.0/24:udp:53", + "--network-egress-deny", + "2001:db8::/32:icmp", + ], + ), + None, + ), + ( + "ingress", + network_with(|network| { + network["egress"] = json!("deny"); + network["ingress"] = json!("allow"); + }), + portable("10.0.0.2/24", "deny", "allow", Some("deny"), &[]), + None, + ), + ( + "host-loopback-outbound", + network_with(|network| network["hostLoopback"] = json!("outbound")), + portable("10.0.0.2/24", "allow", "deny", None, &[]), + None, + ), + ( + "host-loopback-allowed", + network_with(|network| { + network["egress"] = json!("deny"); + network["hostLoopback"] = json!("allow"); + }), + portable("10.0.0.2/24", "deny", "deny", Some("allow"), &[]), + None, + ), + ( + "forwards", + network_with(|network| { + network["hostLoopback"] = json!("allow"); + network["hostLoopbackForwards"] = json!([ + {"protocol": "tcp", "hostPort": 18080, "guestPort": 8080}, + {"protocol": "udp", "hostPort": 18081, "guestPort": 8081}, + ]); + }), + portable( + "10.0.0.2/24", + "allow", + "deny", + Some("allow"), + &[ + "--host-loopback-forward", + "tcp:18080:8080", + "--host-loopback-forward", + "udp:18081:8081", + ], + ), + None, + ), + // A proxy-only network reaches its proxy at the gateway, which its URL names. + ( + "proxy", + network_with(|network| { + network["egress"] = json!("deny"); + network["proxy"] = json!({"port": 3128, "scheme": null}); + }), + portable( + "10.0.0.2/24", + "deny", + "deny", + Some("deny"), + &["--network-proxy", "10.0.0.1:3128"], + ), + Some("10.0.0.1:3128"), + ), + ( + "proxy-http", + network_with(|network| { + network["egress"] = json!("deny"); + network["proxy"] = json!({"port": 3128, "scheme": "http"}); + }), + portable( + "10.0.0.2/24", + "deny", + "deny", + Some("deny"), + &["--network-proxy", "10.0.0.1:3128"], + ), + Some("http://10.0.0.1:3128"), + ), + ( + "proxy-https", + network_with(|network| { + network["guestNetwork"] = json!("192.168.7.6/30"); + network["egress"] = json!("deny"); + network["proxy"] = json!({"port": 8443, "scheme": "https"}); + }), + portable( + "192.168.7.6/30", + "deny", + "deny", + Some("deny"), + &["--network-proxy", "192.168.7.5:8443"], + ), + Some("https://192.168.7.5:8443"), + ), + ]; + for (name, form, rendered, proxy_url) in cases { + let path = directory.join(format!("{name}.json")); + fs::write(&path, form.to_string()).unwrap(); + let network = read_network(&path).unwrap(); + let mut launch = without.clone(); + launch.plan = launch.plan.with_network(Some(network.clone())); + // NVX's lowering renders the network, as it does for NVX's own sandbox profile, and the + // plan puts it after the block devices, before the launch's own options. + assert_eq!( + network::openvmm_arguments(Some(network.policy()), network.guest_network()).unwrap(), + rendered, + "{name}" + ); + let expected = rendered + .iter() + .copied() + .map(OsString::from) + .chain(options.iter().cloned()) + .collect::>(); + assert_eq!(arguments(&launch), expected, "{name}"); + assert_eq!(launch.plan.proxy_url().as_deref(), proxy_url, "{name}"); + // The network adds no boot token: OpenVMM appends the guest's network settings itself. + assert_eq!( + launch.kernel_command_line().unwrap(), + without.kernel_command_line().unwrap() + ); + + // The record keeps the network in the plan, as the file gave it. + let record = LaunchRecord { + kernel_command_line: "nvx_sandbox=1", + layered: Some(&launch), + }; + let json = record.to_json().unwrap(); + assert_eq!(json["plan"]["network"], form, "{name}"); + let plan: SandboxPlan = serde_json::from_value(json["plan"].clone()).unwrap(); + assert_eq!(plan, launch.plan, "{name}"); + assert_eq!(plan.proxy_url().as_deref(), proxy_url, "{name}"); + } + // Without a network, the record says so. + let record = LaunchRecord { + kernel_command_line: "nvx_sandbox=1", + layered: Some(&without), + }; + assert_eq!(record.to_json().unwrap()["plan"]["network"], Value::Null); +} + +#[test] +fn a_launch_reads_only_the_networks_that_the_plan_takes() { + let directory = directory("layered-network-refused"); + let missing = directory.join("missing.json"); + let error = read_network(&missing).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + assert_eq!( + error.message(), + format!("cannot read the network {}", missing.display()) + ); + + // Each refusal names its reason in its cause: the forms that the plan does not read, then the + // networks that it refuses. + let mut cases = vec![ + ("{".to_owned(), "EOF while parsing".to_owned()), + ("null".to_owned(), "invalid type: null".to_owned()), + ]; + let mut change = |problem: &str, edit: &dyn Fn(&mut Value)| { + cases.push((network_with(edit).to_string(), problem.to_owned())); + }; + change("unknown field `extra`", &|network| { + network["extra"] = json!(null); + }); + change("missing field `proxy`", &|network| { + network.as_object_mut().unwrap().remove("proxy"); + }); + change("unknown variant `open`", &|network| { + network["egress"] = json!("open"); + }); + change("unknown variant `both`", &|network| { + network["hostLoopback"] = json!("both"); + }); + change("invalid value: integer `65536`", &|network| { + network["hostLoopback"] = json!("allow"); + network["hostLoopbackForwards"] = + json!([{"protocol": "tcp", "hostPort": 65536, "guestPort": 8080}]); + }); + for guest_network in ["10.0.0.1/24", "10.0.0.0/24", "10.0.0.2/31", "10.0.0.2"] { + change( + &format!( + "guest network '{guest_network}' must be an IPv4 address with a /1 to /30 prefix" + ), + &|network| network["guestNetwork"] = json!(guest_network), + ); + } + change( + "egress.allow[0].to[0].cidr is not a valid IPv4 or IPv6 CIDR: 192.0.2.0/33", + &|network| { + network["allow"]["rules"] = + json!([{"to": [{"cidr": "192.0.2.0/33", "except": []}], "ports": []}]); + }, + ); + change( + "egress.deny[0].to[0].except[0] must be contained in 198.51.100.0/24", + &|network| { + network["deny"]["rules"] = json!([{ + "to": [{"cidr": "198.51.100.0/24", "except": ["192.0.2.0/25"]}], + "ports": [], + }]); + }, + ); + change( + "egress.allow[0].ports[0].endPort 80 cannot be below port 443", + &|network| { + network["allow"]["rules"] = + json!([{"to": [], "ports": [{"protocol": "tcp", "port": 443, "endPort": 80}]}]); + }, + ); + change( + "egress.deny[0].ports[0].port is not supported with icmp", + &|network| { + network["deny"]["rules"] = + json!([{"to": [], "ports": [{"protocol": "icmp", "port": 8, "endPort": null}]}]); + }, + ); + // Hosts that never merge, one more than OpenVMM takes of one action. + change("egress.allow emits at most 256 native rules", &|network| { + let hosts = (0..=network::MAX_RULES) + .map(|index| { + let host = 2 * index + 1; + json!({"cidr": format!("10.1.{}.{}", host / 256, host % 256), "except": []}) + }) + .collect::>(); + network["egress"] = json!("deny"); + network["allow"]["rules"] = json!([{"to": hosts, "ports": []}]); + }); + change( + "host-loopback forwards need host-loopback access allowed", + &|network| { + network["hostLoopbackForwards"] = + json!([{"protocol": "tcp", "hostPort": 18080, "guestPort": 8080}]); + }, + ); + change( + "host-loopback forwards bind tcp host port 18080 more than once", + &|network| { + network["hostLoopback"] = json!("allow"); + network["hostLoopbackForwards"] = json!([ + {"protocol": "tcp", "hostPort": 18080, "guestPort": 8080}, + {"protocol": "tcp", "hostPort": 18080, "guestPort": 8081}, + ]); + }, + ); + change( + "host-loopback UDP forwards need egress allowed by default for their replies", + &|network| { + network["egress"] = json!("deny"); + network["hostLoopback"] = json!("allow"); + network["hostLoopbackForwards"] = + json!([{"protocol": "udp", "hostPort": 18081, "guestPort": 8081}]); + }, + ); + change( + "host-loopback access needs a guest network prefix of /29 or shorter", + &|network| { + network["guestNetwork"] = json!("10.0.0.2/30"); + network["hostLoopback"] = json!("allow"); + }, + ); + // A proxy is the guest's one way out. + let proxy = |network: &mut Value| { + network["egress"] = json!("deny"); + network["proxy"] = json!({"port": 3128, "scheme": "http"}); + }; + change( + "a network proxy needs egress denied by default", + &|network| { + proxy(network); + network["egress"] = json!("allow"); + }, + ); + change("a network proxy takes no egress.allow rules", &|network| { + proxy(network); + network["allow"]["rules"] = json!([{"to": [], "ports": []}]); + }); + change("a network proxy needs ingress denied", &|network| { + proxy(network); + network["ingress"] = json!("allow"); + }); + change( + "a network proxy needs host-loopback access denied", + &|network| { + proxy(network); + network["hostLoopback"] = json!("outbound"); + }, + ); + change( + "a network proxy takes no host-loopback forwards", + &|network| { + proxy(network); + network["hostLoopbackForwards"] = + json!([{"protocol": "tcp", "hostPort": 18080, "guestPort": 8080}]); + }, + ); + change("a network proxy needs a port other than 53", &|network| { + proxy(network); + network["proxy"]["port"] = json!(53); + }); + change( + "a network proxy needs a port between 1 and 65535", + &|network| { + proxy(network); + network["proxy"]["port"] = json!(0); + }, + ); + change( + "network proxy scheme \"ftp\" must be http or https", + &|network| { + proxy(network); + network["proxy"]["scheme"] = json!("ftp"); + }, + ); + let path = directory.join("network.json"); + for (contents, problem) in cases { + fs::write(&path, &contents).unwrap(); + let error = read_network(&path).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Unsupported, "{problem}: {error}"); + assert_eq!( + error.message(), + format!("the network {} is refused", path.display()) + ); + let cause = std::error::Error::source(&error).unwrap().to_string(); + assert!(cause.contains(&problem), "{problem}: {cause}"); + } +} + #[test] fn the_host_program_stages_images_in_the_store() { let directory = directory("layered-stage-command"); @@ -1097,11 +1603,63 @@ fn the_host_program_stages_images_in_the_store() { assert_eq!(staged.scratch, None); } +#[test] +fn the_host_program_refuses_a_network_before_it_launches() { + let directory = directory("layered-network-command"); + let distro = image(directory.join("distro.erofs"), 1024); + let store = directory.join("store"); + // Without NVX's artifacts, a launch fails once it has read its network. + let launch = |network: &Path| -> Output { + let mut image = OsString::from("erofs:"); + image.push(&distro); + Command::new(env!("CARGO_BIN_EXE_runtime-adapter-host")) + .arg("launch") + .arg(&store) + .args(["--guest-agent", "agent", "--hypervisor", "kvm", "--distro"]) + .arg(image) + .args(["--distro-identity", &identity().to_string()]) + .args(["--upper", "ram", "--network"]) + .arg(network) + .env_remove(Artifacts::ENV_DIR) + .output() + .unwrap() + }; + let failed = |output: Output| { + assert_eq!(output.status.code(), Some(1), "{output:?}"); + // Nothing was launched: the store was never created. + assert!(!store.exists(), "{output:?}"); + String::from_utf8_lossy(&output.stderr).into_owned() + }; + + // A network that the plan refuses, and one that cannot be read, fail the launch first. + let network = directory.join("network.json"); + let proxy = network_with(|network| network["proxy"] = json!({"port": 3128, "scheme": "http"})); + fs::write(&network, proxy.to_string()).unwrap(); + let stderr = failed(launch(&network)); + let refused = format!( + "the network {} is refused: a network proxy needs egress denied by default", + network.display() + ); + assert!(stderr.contains(&refused), "{stderr}"); + let missing = directory.join("missing.json"); + let stderr = failed(launch(&missing)); + let unreadable = format!("cannot read the network {}", missing.display()); + assert!(stderr.contains(&unreadable), "{stderr}"); + + // A network that the plan takes passes, and the launch goes on to need NVX's artifacts. + fs::write(&network, open_network().to_string()).unwrap(); + let stderr = failed(launch(&network)); + assert!( + stderr.contains(&format!("set {} to NVX's artifacts", Artifacts::ENV_DIR)), + "{stderr}" + ); +} + #[test] fn the_host_program_refuses_options_that_do_not_make_a_command() { let launch = ["launch", "store", "--guest-agent", "agent"]; let uuid = "0a490905-41be-ed5f-984e-63eb7e5e20b1"; - let cases: [(&[&str], &str); 34] = [ + let cases: [(&[&str], &str); 37] = [ ( &["--distro", "erofs:image", "--upper", "ram"], "--distro needs --distro-identity", @@ -1157,6 +1715,23 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { &["--first-partition-bound", "4096"], "--first-partition-bound needs --distro", ), + (&["--network", "network.json"], "--network needs --distro"), + ( + &[ + "--distro", + "erofs:image", + "--distro-identity", + uuid, + "--upper", + "ram", + "--network", + "network.json", + "--network", + "other.json", + ], + "unexpected option \"--network\"", + ), + (&["--network"], "\"--network\" needs a value"), ( &[ "--distro", @@ -1227,7 +1802,7 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { .map(|(options, problem)| ([&launch[..], options].concat(), *problem)) .collect::>(); let stage = ["stage", "store", "--distro", "erofs:image"]; - let stage_cases: [(&[&str], &str); 9] = [ + let stage_cases: [(&[&str], &str); 10] = [ (&[], "stage needs --distro-identity"), ( &["--distro-identity", uuid, "--upper", "ram"], @@ -1257,6 +1832,10 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { &["--share-root", "sources"], "unexpected option \"--share-root\" of stage", ), + ( + &["--network", "network.json"], + "unexpected option \"--network\" of stage", + ), (&["image"], "unexpected argument \"image\" of stage"), ]; commands.extend( @@ -1289,6 +1868,10 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { vec!["reattach", "store", "--share", "a:/b"], "unexpected option \"--share\" of reattach", ), + ( + vec!["stop", "store", "--network", "network.json"], + "unexpected option \"--network\" of stop", + ), ]); // Each execution of `exec` has options of its own, before `--` and its program, and its // arguments end at a lone `;`. The program refuses a start that the guest would not decode,