Repository navigation
tests: settle the runtime adapter's executions through cgroups - #862
Enrique Saurez (esaurez) wants to merge 13 commits into
Conversation
…uest The runtime adapter's host program gains `stage`, which verifies an image with DistroBinding::verify, hashing it only when a manifest is beside it, binds a scratch image with ScratchBinding::bind, and records both bindings in the sandbox's store. Staging is where hashing belongs: a launch never hashes. A layered launch now plans its root with SandboxPlan::adapter, which holds none of the sandbox profile's settings, passes nvx_sandbox=1 as its own boot token, and drops --microvm-lifecycle managed, which OpenVMM only checks and labels its report with, and whose token only NVX's init agent reads, which the guest agent replaces. The image's identity is required, since a binding names it, and the launch record moves to format 2, with the bindings that the store stages for the launch's images. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
There was a problem hiding this comment.
🟡 Changes recommended
Fixture cleanup can mask a worker shutdown regression, so the cgroup-removal assertion needs to run before the fixture is dropped.
1 open issue
What changed in this PR
This PR implements the first part of #805 by giving each runtime-adapter execution its own cgroup, allowing settlement to account for descendants that leave the program’s process group.
Changes:
- Creates per-execution cgroups and retains execution slots until cleanup, keeping unsettled executions fenced until shutdown.
- Connects root bootstrap cgroups to the worker while preserving the process-group fallback.
- Adds cgroup lifecycle tests and documents settlement and admission behavior.
| File | Description |
|---|---|
tests/runtime-adapter/tests/guest_session.rs |
Adds cgroup fixtures and descendant-settlement and shutdown tests. |
tests/runtime-adapter/src/worker/executions.rs |
Implements cgroup placement, settlement, slot retention, and cleanup. |
tests/runtime-adapter/src/worker.rs |
Supplies bootstrap cgroups to executions and documents their lifecycle. |
tests/runtime-adapter/README.md |
Describes cgroup behavior, fallback behavior, and test coverage. |
Review effort: Balanced
… checked files A layered launch of runtime-adapter-host now checks the bindings that the sandbox's store stages for its image and scratch image before it spawns anything, with DistroBinding::check and ScratchBinding::check, which read no content and claim the scratch image for the sandbox. A binding that the store does not stage, or that no longer holds, fails the launch before it begins. LayeredLaunch::prepare returns NVX's PreparedLaunch of the checked files. OpenVMM's arguments name those files rather than their paths, and the VmmCommand inherits what PreparedLaunch::inherited names. The launch holds the checked files until the guest has assembled the root, or until it has torn down a launch that failed. Every scenario of vm_layered.rs and vm_restart.rs now stages its images, then launches them, and three scenarios check that unstaged images, an image changed in place, and a replaced scratch image are refused before spawn. The restart scenarios boot NVX's Ubuntu EROFS image over a copy of the scratch template, whose fixtures move to tests/common. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
…ce the checked images runtime-adapter-host launch takes a test-only step hook from RUNTIME_ADAPTER_STEP_HOOK. At a named step of the launch (checked, spawned, ready, or commit), STEP:pause:PATH creates PATH with the step's name and waits until it is gone, and STEP:exit exits at once with status 5, without unwinding, as a crash would end the launch there. A value that names no step or action is a usage error. The hook lives in the hook module, for later scenarios to reuse. vm_steps.rs pauses a launch after its checks while files of zeros try to take the staged image's and scratch image's paths. On Linux they do, and the guest still assembles the checked originals, since OpenVMM opens the inherited descriptors. On Windows the checked files keep the replacements, deletions, and writes of the image out until the guest is ready. It also ends a launch at its checks, which leaves nothing of a launch and releases the scratch image's claim, and pauses a launch at each step. The VM tests' sandbox can start a command, wait for its pause, and finish it, and ends a command that a failed test left paused. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
A launch paused at `spawned` or `ready` spent its startup deadline while it waited, so a pause longer than that deadline resumed into a timeout and ended a healthy VM. The step hook now returns how long it paused, which the launch adds to its deadline, so that a test may pause a launch for as long as it needs. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
The adapter's guest answered a root configuration with an empty RootReady, and its framing had no message to hand the guest a mapping table or to tell it to start the sandbox's workloads. RootReady, kind 7, becomes Ready, whose body carries the guest agent's build, so that the host can check it against the build that it ships. The build is length-prefixed, so that later fields can follow it. New kinds carry a part of the host's mapping table as a MAPS payload of nvx::protocol (Maps, 22), the guest's answers to it (Mapped, 23, and MapFailed, 24, with how many mappings the guest applied and the problem), and the start of the sandbox's workloads (Start, 25, and Started, 26). The root configuration gains the number of mappings of the table, mapping-count, at most 4,096. The worker answers Ready with the crate's version, which moves to the crate root as GUEST_BUILD so that the guest side can report it. Applying the table, and the order of Ready, the table, and Start, come in later changes. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9d3891be-9b19-4f82-96d4-fe7320eb1022
A guest that maps host paths into a layered root from the host's mapping table needs the host export, the aggregate virtio-fs device that OpenVMM attaches for those paths, mounted outside the root. NVX's shell init mounts it for its own agent, but a guest built on nvx::guest had no way to mount it as the source of Root::map. - nvx::protocol::export: HostExport::parse, a strict parser of the virtfs_dir=, virtfs_tag=, virtfs_mode=, and virtfs_aggregate=1 tokens that OpenVMM appends for the export: all four or none, each once and in its form, with the checks of the shell init's mount_host_export, and no nvx_share= token. EXPORT_DIRECTORY is the export's guest directory, which nvx::host::filesystem::GUEST_EXPORT now names, so the host's plan and the guest's parser share it. tests/fixtures/export-tokens.json holds the parser's cases. - Root::mount_export mounts the export nosuid,nodev, read-only or read-write, at hostfs/root in the runtime directory, below a directory that only root may enter, owns the mount, unmounts it after the mappings at teardown, and returns its SourceRoot for Root::map. A root mounts one export; a failure latches the root, and an assembly refuses a runtime directory where an export is mounted already. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d31a6be-a0f8-4952-8667-0ebdc440d08d
Root::mount_export mounts the host export below hostfs in the runtime directory, a directory that only root may enter, so that a workload reaches host paths only through their mappings. A hostfs that was there already kept its owner and mode, so one that another user owned, or that its group or others could enter, would expose the export's children directly. Accept a hostfs that is there already only if it is private to the process's effective user: owned by that user, with no permission for its group or others, as the one that an earlier root leaves. Refuse any other, and leave it unchanged. NVX's shell init refuses any hostfs that is there already, since it mounts the export once per boot, while a root may be assembled again in a runtime directory where an earlier root left hostfs. Part of #822 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d31a6be-a0f8-4952-8667-0ebdc440d08d
… worker The runtime adapter's worker answered Ready once it assembled its root, then ran the host's executions at once: it had no host export to map from, took no mapping table, and nothing told it when the sandbox's workloads could start, so an execution could run on a root whose mappings were missing. After the assembly, the worker now mounts the host export that the kernel command line names, with nvx::protocol::export::HostExport::parse and Root::mount_export, then answers Ready. It applies the parts of the host's mapping table in order, with Root::map, from the export, and answers each with Mapped or MapFailed. A part that does not decode or does not continue the table, whose length the root configuration's mapping count announces, is refused as a whole; a mapping that the root refuses or fails to apply latches the root, and the worker refuses every later part. Start, which the worker confirms with Started, comes once every announced mapping is applied, and only then does the worker run executions: an execution before the start, a premature or second Start, and a part after the start end its session. A table of zero mappings does no work. The layout stage refuses a mapping count without an export, and export tokens that break their rules; an export that fails to mount fails the assembly. The root operations gain mount_export and map, which the tests' fake root records. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 04632931-537b-49f9-acc9-b12024ad6f3a
…rogram The runtime adapter's host program sent the root configuration, took any Ready as the guest's readiness, whatever the guest agent's build, and went on to its exchanges: it shared no host path with the root, sent no mapping table, and never started the sandbox, so the worker's order had no host side, and a guest agent of another build than the host ships went on serving. launch now takes shares: --share HOST:TARGET[:ro], each a host directory or file at a guest path in the root; --deny PATH, host paths hidden inside them; --reserved TARGET, guest trees that the adapter keeps beside the container root's; and --share-root DIR, the host directories that the adapter authorizes. layered::Shares::plan refuses a share outside those directories, once links are resolved, before it plans the export with nvx::host::filesystem::plan, and checks the export's directories and files again. The launch refuses what it cannot export before it touches the store. LayeredLaunch::share attaches the export, whose table the root configuration announces, LayeredLaunch::prepare checks it with ExportPlan::verify, and OpenVMM's arguments export it, as ExportPlan::openvmm_arguments renders them. Once attached, host::ready sends the root configuration and checks that the Ready names the build that the host ships; a guest agent of another build gets nothing more, and the launch fails, which ends the VM. host::start_sandbox then sends the table, each part once the guest applied the one before, then Start, and waits for Started; the launch commits only after. A part that the guest fails ends the order: the host sends no more of the table, nor Start, but Stop, which tears the root down, then waits for the VM to end, and exits with 6 once it ended with a success and a complete teardown. The launch prints build, table, mapped, and sandbox. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 04632931-537b-49f9-acc9-b12024ad6f3a
The runtime adapter's mapping probes wrote their own sources below /tmp in the guest and mapped them through the root themselves, so no layered scenario mapped a host path: every launch shared nothing, and its guest applied an empty table. The probes now inspect what the host's mapping table put beneath the root, and map from the host export. map [NAME=PATH...] reports the export's mount, with its options, and the mode and owner of the directory above it; the mounts below the root, in order; and for each path, its mount options, what it shows, its entries or contents, whether a write is allowed, and what the overlay's upper holds there. map-checks DIR FILE maps the export's directory DIR and file FILE through mappers of the adapter's own, as it mapped its own sources before. map-refused is gone: a failed mapping of the table now latches the root. The layered scenarios share host directories and a file with the root through the launch's --share options: - the table over each image and upper, in the plan's order, whose writes reach the writable host sources alone, and which hides a denied path; - a failed mapping, which blocks Start: the launch stops the VM, which tears the root down and leaves the sources as they were; - a read-only share inside a writable one, shares at and over a tree that the adapter reserves, and a share outside the host directories that it authorizes, which the launch refuses before it spawns the VM. Without shares, the image over the scratch checks that zero mappings do no work: the guest mounts no export, and nothing is mounted below the root. Part of #822. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 04632931-537b-49f9-acc9-b12024ad6f3a
a40048e to
20d5cdc
Compare
| if binding.execution().status().is_none() { | ||
| self.reaping.push(binding.id()); | ||
| } | ||
| drop(binding); |
There was a problem hiding this comment.
Why does binding need to be drop manually? Are we required to release its resources before the next stament? Could we rely on automatic drop semantics instead?
There was a problem hiding this comment.
No, the explicit drop isn't required. Nothing after it depends on the binding being gone, so the automatic drop at the end of scope behaves the same. The explicit drops only marked where a program that hasn't ended gets killed.
Your 21fd36f already relies on automatic drop. It removes drop(binding); from clear() (line 282 of the version you commented on) and let key = binding.runner().key(); drop(binding); from forget() (lines 386-387), names binding.runner().key() in forget()'s diagnostic instead, and updates forget()'s doc to say that the binding drops last. I checked that the new order is equivalent:
clear(): eachbindingstill drops at the end of its own loop iteration. So every program that hasn't ended, with its process group or its cgroup's processes, is still killed before thecgroup.killloop overfencedand the reaping wait.self.fenced.extend(cgroup)only stores the handle.forget(): the binding now drops after the cgroup's removal. That's safe becausesettledcomes from a reusable outcome, whichBinding::settlereports only after it finds the cgroup unoccupied and callsExecution::release_cgroup(). By then no process or zombie is left in the cgroup, whichExecutionCgroup::removechecks anyway, and the binding's drop no longer touches it. An unsettled execution's cgroup only goes tofenced, and the drop still kills its processes before the stop'sclear()kills and removes what's left.
So your head needs nothing more from me. With 5a071a5 checked out, rustfmt and Clippy in the guest-only configuration pass. I'll leave the thread open for you to resolve.
The worker ran each program in a process group of its own, which it killed before it collected the program's status. Nothing proved that a descendant that left the group was gone, so every outcome was fenced, and the worker could admit a new execution while such a descendant still ran. Each execution now gets a cgroup of its own below the workload cgroup, which the worker prepares with ExecutionCgroup::prepare, and which the program joins as it spawns. The execution settles through that cgroup alone, so the other executions keep running, and an execution whose program ends on its own is reusable. An execution keeps its place among the worker's limit until its cgroup is empty and removed. One that does not settle keeps its place, and its cgroup, until the host stops the worker, which ends and removes every cgroup that is left. The root's operations name the cgroups that the bootstrap prepared. A fake root names none where the in-process tests cannot create cgroups, and the programs keep their process groups there, as before. Where the tests can, each fake root has a workload cgroup of its own, and new tests check a descendant in a session of its own, which the settlement ends, and an execution that does not settle, which keeps its place until the stop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90
A program that the worker spawned, but whose execution it could not have, may still be joining its cgroup, as its launch has yet to report, and removing the cgroup could then hide the program once it ended, which ExecutionCgroup::remove forbids. Such a start's cgroup now keeps its place, as that of an execution that did not settle, and a stop removes the executions' cgroups only once the worker collected the status of every program that it forgot. The holder in the test of an execution that does not settle now runs in a process group of its own, which its drop kills, so that its child no longer outlives a test that fails before the child joins the cgroup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90
Once the in-process tests create cgroups, each fake root holds a workload cgroup of the test's own, which ends what is left in it and removes the cgroups of executions when it drops. Two tests discarded the fake root that a stop returns, so that it dropped before their checks: that the stop removed the cgroup of an execution that did not settle, and that it left no process of a program's group running. Those checks then held whatever the worker did. The tests now keep the fake root until their checks are done. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90
20d5cdc to
5a071a5
Compare
4edd9d5 to
9f0c445
Compare

Stacked on #878, PR 8 of #822 and the last of its chain #863 → #867 → #869 → #865 → #864 → #872 → #875 → #878; this PR retargets to
devonce they merge.Part of #805: PR 1 of 3, a cgroup per execution.
What changed
The runtime adapter's worker ran each program in a process group of its own, which it killed before it collected the program's status. Nothing proved that a descendant that left the group was gone, so every outcome was fenced, and the worker could admit a new execution while such a descendant still ran.
execution-N, prepared withExecutionCgroup::prepare, which the program joins as it spawns (Command::cgroup(cgroup.placement()?)).Options::cgroup(&cgroup, Scope::Execution), which leaves the other executions running. An execution whose program ends on its own is now reusable.worker::MAX_EXECUTIONSuntil its cgroup is empty and the worker removed it. One that does not settle keeps its place and its cgroup until the host stops the worker, which ends and removes every cgroup that is left.RootOperations::cgroupsnames the cgroups that the bootstrap prepared, in which the worker runs its executions once the root is ready; under runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's order, it runs none before the host started the sandbox. Where the in-process tests cannot create cgroups, the fake root names none, and the programs keep their process groups, as before.Tests
In
tests/guest_session.rs, where the runner lets the tests create cgroups (root, in a cgroup v2 hierarchy with thepidscontroller), each fake root gets a workload cgroup of its own, and every execution test runs with cgroups, its outcomes reusable. Elsewhere, the tests run with process groups, as before. New tests, with cgroups:Validation
This branch is rebased onto #878 (4edd9d5). Its first commit's conflicts with #822, in the worker's handling of the root (#822's order of the root configuration, the mapping table and the sandbox's start), the in-process tests' fake root (its export's mount and its mappings), and the README, are resolved keeping both sides:
RootOperationskeeps #822'smount_exportandmapbesidecgroups, and the worker gives its executions the cgroups once it answersReady.git range-diffshows only that context for the first commit (9079629 → a9ed650), and the other two are unchanged (d046b70 → 9fc2635, a40048e → 20d5cdc). At 20d5cdc:check-nvx-rsin its three configurations, with rustfmt, Clippy with warnings denied, the build, the tests, andcargo +1.89 check(the MSRV): protocol-only, host-only, and the guest-onlyx86_64-unknown-linux-muslconfiguration; andscripts/check_markdown_links.py.guest_sessiontests; then 10 runs ofguest_sessionas root, with cgroups, and 10 unprivileged, with process groups, and 3 runs of the library's tests, all passing, with no cgroup left behind.host_session, all passing.v0.1.0-dev.36f8a6202435artifacts: on WHP, locally, all 35 layered scenarios, the 3 restart tests and the 3 step tests pass; on MSHV, on a Linux test host, as root, with OpenVMM's--cpu-profile host, so do all of them,first_partition_over_its_bound_fails_the_assemblyunder a short temporary directory, since its control socket's path exceedsSUN_LENunder that host's long one. KVM is left to CI. This PR adds no VM scenario; PR 3 of runtime adapter: settle executions through cgroups and run them in a VM #805 adds those that run executions.