Repository navigation
runtime adapter: settle executions through cgroups and run them in a VM #805
Description
Parent: #822 (#816). Follow-up to #784; refs #624 and #545. Needs: nothing. Unblocks:
- runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's PRs 10 and 11;
- runtime adapter: the consumer-contract scenarios C2, C3, C6 and C7 on real VMs #859;
- in the snapshot/restore workstream (Snapshot/restore workstream: start sandboxes from captured templates through the shared host and guest libraries #833): nvx-rs: restore clones from a template through Activate, Ready and Start #844's PR 5, with nvx-rs: the restricted control state and Activate after a restore, in the guest library and the test adapter's framing #843's VM case "no workload before Ready"; runtime adapter: template and clone operations #847's PR 2.
Goal. The test adapter's executions run in VMs, each:
- in a cgroup of its own;
- in the root that the worker assembled;
- under an identity that the host names;
- settled before its place is reused.
The host program runs them over the session.
Today (dev d04d9fb):
- Executions: the worker runs up to
worker::MAX_EXECUTIONS(8) at once, throughnvx::execand its guest binding. - No cgroup:
- Each program leads a process group, which the worker kills before it collects the program's status (
Options::new().process_group().caller_reaps(),tests/runtime-adapter/src/worker/executions.rs:513-545). - Nothing proves that a descendant that left the group is gone. So every outcome is fenced, and the worker may readmit while such a descendant still runs.
- Each program leads a process group, which the worker kills before it collects the program's status (
- Not in the assembled root: each program runs in the worker's own root and working directory, under the worker's identity.
- No host command: only in-process tests drive executions, through the test broker of
tests/guest_session.rs.runtime-adapter-hosthas nothing that runs one. - Cgroups: the worker already bootstraps the agent and workload cgroups from the root configuration (
nvx::guest::root::bootstrap).
Stacked PRs. Each PR builds on the one before it, lands with its own tests, and updates the adapter's README and the worker's module docs.
PR 1 of 3: a cgroup per execution. Needs nothing.
- Each execution gets a cgroup below the workload cgroup, with
ExecutionCgroup::prepare, and joins it at spawn (Command::cgroup(cgroup.placement()?)). - It settles through that cgroup with
Options::cgroup(&cgroup, Scope::Execution), which leaves the other executions running. - An execution keeps its place among
worker::MAX_EXECUTIONSuntil its cgroup is empty and removed. One that fails to settle keeps it until the host stops the VM. - The in-process tests keep process groups where they can't create cgroups.
- Tests, in-process, with cgroups where the runner allows them:
- a descendant that leaves the process group, which settlement ends;
- a settlement that fails, and keeps its place.
PR 2 of 3: executions in the assembled root, under their identity. Needs PR 1.
-
ExecStartgains:- an optional identity: UID, GID, supplementary groups, capabilities and account mode;
- a working directory;
- an environment.
-
The root configuration gains an optional workload identity for the whole sandbox: a nonzero UID and GID, with no groups or capabilities.
- The worker checks its account in the assembled root, in existing-account mode, before it answers the root configuration.
- A missing account fails a new root stage,
identity, with status 16.
-
Exactly one source names each execution's identity:
- in a sandbox with a workload identity, an execution names none, and runs as it;
- in a sandbox without one, an execution names its own.
Either violation is refused at stage 0. This mirrors the activation record's optional identity (nvx-rs: a canonical activation record for platform clones #838): an adapter's clones use one model or the other.
-
The worker enters each execution through
nvx::guest::entry:- new namespaces, with the root moved over
/; - the configured hostname;
- the image's own machine ID and no time status, so
Entry::bind_runtimebinds nothing.
Entry::checkrefuses, at stage 0, an identity outside the entry's allowlist. Until runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's PR 10 says whether listeners are published, executions enter as published, the entry's default, which refusesNET_BIND_SERVICEandNET_RAW. - new namespaces, with the root moved over
-
Tests:
- the encodings, and each refusal;
- in
guest_session.rs, with a fake root: the entry's inputs; the account check of a sandbox-wide identity.
PR 3 of 3: the host's command, and executions in VMs. Needs PR 2.
- A host-side driver runs one
nvx::exec::Peerper execution over the client. It comes from the test broker's host side (tests/guest_session.rs:2081). runtime-adapter-host exec STORE [--identity ...] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]...runs one or more executions over one session, and prints their outputs and outcomes.- Tests, in VMs on KVM, MSHV and WHP:
- output beyond 1 MiB;
- several executions at once;
- a descendant that leaves the process group, which its execution's settlement ends;
- an execution under a non-root identity, and one as UID 0 with an allowed capability;
- a sandbox-wide identity, which executions run as;
- a sandbox-wide identity whose account the image lacks, which fails before the guest answers.
Not in scope:
- probes while executions run: the worker keeps answering a probe with
ProbeFailedandworker::EXECUTING; - the consumer-contract scenarios on real VMs (runtime adapter: the consumer-contract scenarios C2, C3, C6 and C7 on real VMs #859).
Done when:
- the three PRs are merged;
- the adapter's checks pass in every configuration, as its README lists them;
- the in-process tests still run where they can't create cgroups.
Activity
Temporarily integrate #805 PRs 1-3