Skip to content

tests: settle the runtime adapter's executions through cgroups - #862

Open
Enrique Saurez (esaurez) wants to merge 13 commits into
esaurez/822-pr08-mapping-tables-vms-20261023from
esaurez/805-pr1-execution-cgroups-20261023
Open

Enrique Saurez (esaurez) wants to merge 13 commits into
esaurez/822-pr08-mapping-tables-vms-20261023from
esaurez/805-pr1-execution-cgroups-20261023

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #878, PR 8 of #822 and the last of its chain #863 → #867 → #869 → #865 → #864 → #872 → #875 → #878; this PR retargets to dev once they merge.

Part of #805: PR 1 of 3, a cgroup per execution.

What changed

The runtime adapter's worker ran each program in a process group of its own, which it killed before it collected the program's status. Nothing proved that a descendant that left the group was gone, so every outcome was fenced, and the worker could admit a new execution while such a descendant still ran.

  • Each execution gets a cgroup of its own below the workload cgroup, execution-N, prepared with ExecutionCgroup::prepare, which the program joins as it spawns (Command::cgroup(cgroup.placement()?)).
  • It settles through that cgroup with Options::cgroup(&cgroup, Scope::Execution), which leaves the other executions running. An execution whose program ends on its own is now reusable.
  • An execution keeps its place among worker::MAX_EXECUTIONS until its cgroup is empty and the worker removed it. One that does not settle keeps its place and its cgroup until the host stops the worker, which ends and removes every cgroup that is left.
  • RootOperations::cgroups names the cgroups that the bootstrap prepared, in which the worker runs its executions once the root is ready; under runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's order, it runs none before the host started the sandbox. Where the in-process tests cannot create cgroups, the fake root names none, and the programs keep their process groups, as before.
  • The adapter's README and the worker's module docs describe the cgroups, the places, and the process-group fallback.

Tests

In tests/guest_session.rs, where the runner lets the tests create cgroups (root, in a cgroup v2 hierarchy with the pids controller), each fake root gets a workload cgroup of its own, and every execution test runs with cgroups, its outcomes reusable. Elsewhere, the tests run with process groups, as before. New tests, with cgroups:

  • a descendant in a session of its own, which holds the program's output once the program ended, and which the settlement ends: the output ends at once, losing nothing, the worker removed the cgroup before it sent the outcome, and it runs its limit of executions again;
  • an execution that does not settle, because a zombie whose parent is outside its cgroup occupies the cgroup: it keeps its place, even once the cgroup emptied, until the host stops the worker, which removes the cgroup.

Validation

This branch is rebased onto #878 (4edd9d5). Its first commit's conflicts with #822, in the worker's handling of the root (#822's order of the root configuration, the mapping table and the sandbox's start), the in-process tests' fake root (its export's mount and its mappings), and the README, are resolved keeping both sides: RootOperations keeps #822's mount_export and map beside cgroups, and the worker gives its executions the cgroups once it answers Ready. git range-diff shows only that context for the first commit (9079629 → a9ed650), and the other two are unchanged (d046b70 → 9fc2635, a40048e → 20d5cdc). At 20d5cdc:

  • Linux, on a cgroup v2 host: the adapter's steps of check-nvx-rs in its three configurations, with rustfmt, Clippy with warnings denied, the build, the tests, and cargo +1.89 check (the MSRV): protocol-only, host-only, and the guest-only x86_64-unknown-linux-musl configuration; and scripts/check_markdown_links.py.
  • The guest-only tests as root, 46 library tests and 44 guest_session tests; then 10 runs of guest_session as root, with cgroups, and 10 unprivileged, with process groups, and 3 runs of the library's tests, all passing, with no cgroup left behind.
  • Windows: rustfmt, Clippy and the tests of the protocol-only and host-only configurations, and 10 runs of host_session, all passing.
  • Each commit of runtime adapter: settle executions through cgroups and run them in a VM #805's three PRs passes Clippy with warnings denied in the protocol-only, host-only and guest-only configurations.
  • The adapter's VM suites with a release build of the guest agent of tests: run the runtime adapter's executions in VMs through its host program #870's head, which holds this PR, and the v0.1.0-dev.36f8a6202435 artifacts: on WHP, locally, all 35 layered scenarios, the 3 restart tests and the 3 step tests pass; on MSHV, on a Linux test host, as root, with OpenVMM's --cpu-profile host, so do all of them, first_partition_over_its_bound_fails_the_assembly under a short temporary directory, since its control socket's path exceeds SUN_LEN under that host's long one. KVM is left to CI. This PR adds no VM scenario; PR 3 of runtime adapter: settle executions through cgroups and run them in a VM #805 adds those that run executions.

…uest

The runtime adapter's host program gains `stage`, which verifies an image
with DistroBinding::verify, hashing it only when a manifest is beside it,
binds a scratch image with ScratchBinding::bind, and records both bindings
in the sandbox's store. Staging is where hashing belongs: a launch never
hashes.

A layered launch now plans its root with SandboxPlan::adapter, which holds
none of the sandbox profile's settings, passes nvx_sandbox=1 as its own
boot token, and drops --microvm-lifecycle managed, which OpenVMM only
checks and labels its report with, and whose token only NVX's init agent
reads, which the guest agent replaces. The image's identity is required,
since a binding names it, and the launch record moves to format 2, with the
bindings that the store stages for the launch's images.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
Copilot AI balanced review requested due to automatic review settings October 24, 2026 07:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fixture cleanup can mask a worker shutdown regression, so the cgroup-removal assertion needs to run before the fixture is dropped.

1 open issue
What changed in this PR

This PR implements the first part of #805 by giving each runtime-adapter execution its own cgroup, allowing settlement to account for descendants that leave the program’s process group.

Changes:

  • Creates per-execution cgroups and retains execution slots until cleanup, keeping unsettled executions fenced until shutdown.
  • Connects root bootstrap cgroups to the worker while preserving the process-group fallback.
  • Adds cgroup lifecycle tests and documents settlement and admission behavior.
File Description
tests/​runtime-adapter/​tests/​guest_session.rs Adds cgroup fixtures and descendant-settlement and shutdown tests.
tests/​runtime-adapter/​src/​worker/​executions.rs Implements cgroup placement, settlement, slot retention, and cleanup.
tests/​runtime-adapter/​src/​worker.rs Supplies bootstrap cgroups to executions and documents their lifecycle.
tests/​runtime-adapter/​README.md Describes cgroup behavior, fallback behavior, and test coverage.

Review effort: Balanced

Comment thread tests/runtime-adapter/tests/guest_session.rs Outdated
… checked files

A layered launch of runtime-adapter-host now checks the bindings that the
sandbox's store stages for its image and scratch image before it spawns
anything, with DistroBinding::check and ScratchBinding::check, which read
no content and claim the scratch image for the sandbox. A binding that the
store does not stage, or that no longer holds, fails the launch before it
begins.

LayeredLaunch::prepare returns NVX's PreparedLaunch of the checked files.
OpenVMM's arguments name those files rather than their paths, and the
VmmCommand inherits what PreparedLaunch::inherited names. The launch holds
the checked files until the guest has assembled the root, or until it has
torn down a launch that failed.

Every scenario of vm_layered.rs and vm_restart.rs now stages its images,
then launches them, and three scenarios check that unstaged images, an
image changed in place, and a replaced scratch image are refused before
spawn. The restart scenarios boot NVX's Ubuntu EROFS image over a copy of
the scratch template, whose fixtures move to tests/common.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
Copilot AI balanced review requested due to automatic review settings October 24, 2026 07:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged cgroup settlement and teardown warrant human confirmation, with VM validation still pending in CI.

1 open issue

Review effort: Balanced

…ce the checked images

runtime-adapter-host launch takes a test-only step hook from
RUNTIME_ADAPTER_STEP_HOOK. At a named step of the launch (checked,
spawned, ready, or commit), STEP:pause:PATH creates PATH with the step's
name and waits until it is gone, and STEP:exit exits at once with status
5, without unwinding, as a crash would end the launch there. A value that
names no step or action is a usage error. The hook lives in the hook
module, for later scenarios to reuse.

vm_steps.rs pauses a launch after its checks while files of zeros try to
take the staged image's and scratch image's paths. On Linux they do, and
the guest still assembles the checked originals, since OpenVMM opens the
inherited descriptors. On Windows the checked files keep the replacements,
deletions, and writes of the image out until the guest is ready. It also
ends a launch at its checks, which leaves nothing of a launch and releases
the scratch image's claim, and pauses a launch at each step.

The VM tests' sandbox can start a command, wait for its pause, and finish
it, and ends a command that a failed test left paused.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

No blocking defect was identified, but asynchronous cgroup settlement and shutdown sequencing warrant final human review.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Enrique Saurez and others added 7 commits October 24, 2026 01:27
A launch paused at `spawned` or `ready` spent its startup deadline while
it waited, so a pause longer than that deadline resumed into a timeout
and ended a healthy VM. The step hook now returns how long it paused,
which the launch adds to its deadline, so that a test may pause a
launch for as long as it needs.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 95934b67-b8c3-419d-ab53-05fe1b79af13
The adapter's guest answered a root configuration with an empty RootReady,
and its framing had no message to hand the guest a mapping table or to tell
it to start the sandbox's workloads.

RootReady, kind 7, becomes Ready, whose body carries the guest agent's build,
so that the host can check it against the build that it ships. The build is
length-prefixed, so that later fields can follow it. New kinds carry a part
of the host's mapping table as a MAPS payload of nvx::protocol (Maps, 22),
the guest's answers to it (Mapped, 23, and MapFailed, 24, with how many
mappings the guest applied and the problem), and the start of the sandbox's
workloads (Start, 25, and Started, 26). The root configuration gains the
number of mappings of the table, mapping-count, at most 4,096.

The worker answers Ready with the crate's version, which moves to the crate
root as GUEST_BUILD so that the guest side can report it. Applying the table,
and the order of Ready, the table, and Start, come in later changes.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9d3891be-9b19-4f82-96d4-fe7320eb1022
A guest that maps host paths into a layered root from the host's mapping
table needs the host export, the aggregate virtio-fs device that OpenVMM
attaches for those paths, mounted outside the root. NVX's shell init
mounts it for its own agent, but a guest built on nvx::guest had no way
to mount it as the source of Root::map.

- nvx::protocol::export: HostExport::parse, a strict parser of the
  virtfs_dir=, virtfs_tag=, virtfs_mode=, and virtfs_aggregate=1 tokens
  that OpenVMM appends for the export: all four or none, each once and
  in its form, with the checks of the shell init's mount_host_export,
  and no nvx_share= token. EXPORT_DIRECTORY is the export's guest
  directory, which nvx::host::filesystem::GUEST_EXPORT now names, so
  the host's plan and the guest's parser share it.
  tests/fixtures/export-tokens.json holds the parser's cases.
- Root::mount_export mounts the export nosuid,nodev, read-only or
  read-write, at hostfs/root in the runtime directory, below a
  directory that only root may enter, owns the mount, unmounts it after
  the mappings at teardown, and returns its SourceRoot for Root::map.
  A root mounts one export; a failure latches the root, and an assembly
  refuses a runtime directory where an export is mounted already.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3d31a6be-a0f8-4952-8667-0ebdc440d08d
Root::mount_export mounts the host export below hostfs in the runtime
directory, a directory that only root may enter, so that a workload reaches
host paths only through their mappings. A hostfs that was there already kept
its owner and mode, so one that another user owned, or that its group or
others could enter, would expose the export's children directly.

Accept a hostfs that is there already only if it is private to the
process's effective user: owned by that user, with no permission for its
group or others, as the one that an earlier root leaves. Refuse any other,
and leave it unchanged. NVX's shell init refuses any hostfs that is there
already, since it mounts the export once per boot, while a root may be
assembled again in a runtime directory where an earlier root left hostfs.

Part of #822

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3d31a6be-a0f8-4952-8667-0ebdc440d08d
… worker

The runtime adapter's worker answered Ready once it assembled its root, then
ran the host's executions at once: it had no host export to map from, took
no mapping table, and nothing told it when the sandbox's workloads could
start, so an execution could run on a root whose mappings were missing.

After the assembly, the worker now mounts the host export that the kernel
command line names, with nvx::protocol::export::HostExport::parse and
Root::mount_export, then answers Ready. It applies the parts of the host's
mapping table in order, with Root::map, from the export, and answers each
with Mapped or MapFailed. A part that does not decode or does not continue
the table, whose length the root configuration's mapping count announces,
is refused as a whole; a mapping that the root refuses or fails to apply
latches the root, and the worker refuses every later part. Start, which the
worker confirms with Started, comes once every announced mapping is applied,
and only then does the worker run executions: an execution before the
start, a premature or second Start, and a part after the start end its
session. A table of zero mappings does no work.

The layout stage refuses a mapping count without an export, and export
tokens that break their rules; an export that fails to mount fails the
assembly. The root operations gain mount_export and map, which the tests'
fake root records.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 04632931-537b-49f9-acc9-b12024ad6f3a
…rogram

The runtime adapter's host program sent the root configuration, took any
Ready as the guest's readiness, whatever the guest agent's build, and went
on to its exchanges: it shared no host path with the root, sent no mapping
table, and never started the sandbox, so the worker's order had no host
side, and a guest agent of another build than the host ships went on
serving.

launch now takes shares: --share HOST:TARGET[:ro], each a host directory
or file at a guest path in the root; --deny PATH, host paths hidden inside
them; --reserved TARGET, guest trees that the adapter keeps beside the
container root's; and --share-root DIR, the host directories that the
adapter authorizes. layered::Shares::plan refuses a share outside those
directories, once links are resolved, before it plans the export with
nvx::host::filesystem::plan, and checks the export's directories and files
again. The launch refuses what it cannot export before it touches the
store. LayeredLaunch::share attaches the export, whose table the root
configuration announces, LayeredLaunch::prepare checks it with
ExportPlan::verify, and OpenVMM's arguments export it, as
ExportPlan::openvmm_arguments renders them.

Once attached, host::ready sends the root configuration and checks that
the Ready names the build that the host ships; a guest agent of another
build gets nothing more, and the launch fails, which ends the VM.
host::start_sandbox then sends the table, each part once the guest applied
the one before, then Start, and waits for Started; the launch commits only
after. A part that the guest fails ends the order: the host sends no more
of the table, nor Start, but Stop, which tears the root down, then waits
for the VM to end, and exits with 6 once it ended with a success and a
complete teardown. The launch prints build, table, mapped, and sandbox.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 04632931-537b-49f9-acc9-b12024ad6f3a
The runtime adapter's mapping probes wrote their own sources below /tmp in
the guest and mapped them through the root themselves, so no layered
scenario mapped a host path: every launch shared nothing, and its guest
applied an empty table.

The probes now inspect what the host's mapping table put beneath the root,
and map from the host export. map [NAME=PATH...] reports the export's
mount, with its options, and the mode and owner of the directory above it;
the mounts below the root, in order; and for each path, its mount options,
what it shows, its entries or contents, whether a write is allowed, and
what the overlay's upper holds there. map-checks DIR FILE maps the
export's directory DIR and file FILE through mappers of the adapter's own,
as it mapped its own sources before. map-refused is gone: a failed mapping
of the table now latches the root.

The layered scenarios share host directories and a file with the root
through the launch's --share options:

- the table over each image and upper, in the plan's order, whose writes
  reach the writable host sources alone, and which hides a denied path;
- a failed mapping, which blocks Start: the launch stops the VM, which
  tears the root down and leaves the sources as they were;
- a read-only share inside a writable one, shares at and over a tree that
  the adapter reserves, and a share outside the host directories that it
  authorizes, which the launch refuses before it spawns the VM.

Without shares, the image over the scratch checks that zero mappings do no
work: the guest mounts no export, and nothing is mounted below the root.

Part of #822.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 04632931-537b-49f9-acc9-b12024ad6f3a
@esaurez
Enrique Saurez (esaurez) changed the base branch from dev to esaurez/822-pr08-mapping-tables-vms-20261023 October 24, 2026 20:04
@esaurez
Enrique Saurez (esaurez) force-pushed the esaurez/805-pr1-execution-cgroups-20261023 branch from a40048e to 20d5cdc Compare October 24, 2026 20:04
@esaurez
Enrique Saurez (esaurez) requested a balanced review from Copilot October 24, 2026 20:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged cgroup placement, deferred process reaping, and shutdown cleanup warrant final human review despite no identified blocking defects.

No open issues 🎉

Review effort: Balanced

if binding.execution().status().is_none() {
self.reaping.push(binding.id());
}
drop(binding);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does binding need to be drop manually? Are we required to release its resources before the next stament? Could we rely on automatic drop semantics instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the explicit drop isn't required. Nothing after it depends on the binding being gone, so the automatic drop at the end of scope behaves the same. The explicit drops only marked where a program that hasn't ended gets killed.

Your 21fd36f already relies on automatic drop. It removes drop(binding); from clear() (line 282 of the version you commented on) and let key = binding.runner().key(); drop(binding); from forget() (lines 386-387), names binding.runner().key() in forget()'s diagnostic instead, and updates forget()'s doc to say that the binding drops last. I checked that the new order is equivalent:

  • clear(): each binding still drops at the end of its own loop iteration. So every program that hasn't ended, with its process group or its cgroup's processes, is still killed before the cgroup.kill loop over fenced and the reaping wait. self.fenced.extend(cgroup) only stores the handle.
  • forget(): the binding now drops after the cgroup's removal. That's safe because settled comes from a reusable outcome, which Binding::settle reports only after it finds the cgroup unoccupied and calls Execution::release_cgroup(). By then no process or zombie is left in the cgroup, which ExecutionCgroup::remove checks anyway, and the binding's drop no longer touches it. An unsettled execution's cgroup only goes to fenced, and the drop still kills its processes before the stop's clear() kills and removes what's left.

So your head needs nothing more from me. With 5a071a5 checked out, rustfmt and Clippy in the guest-only configuration pass. I'll leave the thread open for you to resolve.

@ppenna
Pedro Henrique Penna (ppenna) changed the base branch from esaurez/822-pr08-mapping-tables-vms-20261023 to dev October 25, 2026 13:20
@ppenna
Pedro Henrique Penna (ppenna) changed the base branch from dev to esaurez/822-pr08-mapping-tables-vms-20261023 October 25, 2026 13:20
@ppenna Pedro Henrique Penna (ppenna) removed their assignment Oct 25, 2026
Enrique Saurez and others added 3 commits October 25, 2026 08:37
The worker ran each program in a process group of its own, which it
killed before it collected the program's status. Nothing proved that a
descendant that left the group was gone, so every outcome was fenced,
and the worker could admit a new execution while such a descendant
still ran.

Each execution now gets a cgroup of its own below the workload cgroup,
which the worker prepares with ExecutionCgroup::prepare, and which the
program joins as it spawns. The execution settles through that cgroup
alone, so the other executions keep running, and an execution whose
program ends on its own is reusable. An execution keeps its place among
the worker's limit until its cgroup is empty and removed. One that does
not settle keeps its place, and its cgroup, until the host stops the
worker, which ends and removes every cgroup that is left.

The root's operations name the cgroups that the bootstrap prepared. A
fake root names none where the in-process tests cannot create cgroups,
and the programs keep their process groups there, as before. Where the
tests can, each fake root has a workload cgroup of its own, and new
tests check a descendant in a session of its own, which the settlement
ends, and an execution that does not settle, which keeps its place
until the stop.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90
A program that the worker spawned, but whose execution it could not
have, may still be joining its cgroup, as its launch has yet to report,
and removing the cgroup could then hide the program once it ended, which
ExecutionCgroup::remove forbids. Such a start's cgroup now keeps its
place, as that of an execution that did not settle, and a stop removes
the executions' cgroups only once the worker collected the status of
every program that it forgot.

The holder in the test of an execution that does not settle now runs in
a process group of its own, which its drop kills, so that its child no
longer outlives a test that fails before the child joins the cgroup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90
Once the in-process tests create cgroups, each fake root holds a
workload cgroup of the test's own, which ends what is left in it and
removes the cgroups of executions when it drops. Two tests discarded the
fake root that a stop returns, so that it dropped before their checks:
that the stop removed the cgroup of an execution that did not settle,
and that it left no process of a program's group running. Those checks
then held whatever the worker did. The tests now keep the fake root
until their checks are done.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a7ad408a-da3b-45c4-aa02-5d25300b1c90
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the esaurez/805-pr1-execution-cgroups-20261023 branch from 20d5cdc to 5a071a5 Compare October 25, 2026 13:37
@esaurez
Enrique Saurez (esaurez) force-pushed the esaurez/822-pr08-mapping-tables-vms-20261023 branch from 4edd9d5 to 9f0c445 Compare October 26, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants