Skip to content

runtime adapter: settle executions through cgroups and run them in a VM #805

Closed

Description

Parent: #822 (#816). Follow-up to #784; refs #624 and #545. Needs: nothing. Unblocks:

Goal. The test adapter's executions run in VMs, each:

  • in a cgroup of its own;
  • in the root that the worker assembled;
  • under an identity that the host names;
  • settled before its place is reused.

The host program runs them over the session.

Today (dev d04d9fb):

  • Executions: the worker runs up to worker::MAX_EXECUTIONS (8) at once, through nvx::exec and its guest binding.
  • No cgroup:
    • Each program leads a process group, which the worker kills before it collects the program's status (Options::new().process_group().caller_reaps(), tests/runtime-adapter/src/worker/executions.rs:513-545).
    • Nothing proves that a descendant that left the group is gone. So every outcome is fenced, and the worker may readmit while such a descendant still runs.
  • Not in the assembled root: each program runs in the worker's own root and working directory, under the worker's identity.
  • No host command: only in-process tests drive executions, through the test broker of tests/guest_session.rs. runtime-adapter-host has nothing that runs one.
  • Cgroups: the worker already bootstraps the agent and workload cgroups from the root configuration (nvx::guest::root::bootstrap).

Stacked PRs. Each PR builds on the one before it, lands with its own tests, and updates the adapter's README and the worker's module docs.

PR 1 of 3: a cgroup per execution. Needs nothing.

  • Each execution gets a cgroup below the workload cgroup, with ExecutionCgroup::prepare, and joins it at spawn (Command::cgroup(cgroup.placement()?)).
  • It settles through that cgroup with Options::cgroup(&cgroup, Scope::Execution), which leaves the other executions running.
  • An execution keeps its place among worker::MAX_EXECUTIONS until its cgroup is empty and removed. One that fails to settle keeps it until the host stops the VM.
  • The in-process tests keep process groups where they can't create cgroups.
  • Tests, in-process, with cgroups where the runner allows them:
    • a descendant that leaves the process group, which settlement ends;
    • a settlement that fails, and keeps its place.

PR 2 of 3: executions in the assembled root, under their identity. Needs PR 1.

  • ExecStart gains:

    • an optional identity: UID, GID, supplementary groups, capabilities and account mode;
    • a working directory;
    • an environment.
  • The root configuration gains an optional workload identity for the whole sandbox: a nonzero UID and GID, with no groups or capabilities.

    • The worker checks its account in the assembled root, in existing-account mode, before it answers the root configuration.
    • A missing account fails a new root stage, identity, with status 16.
  • Exactly one source names each execution's identity:

    • in a sandbox with a workload identity, an execution names none, and runs as it;
    • in a sandbox without one, an execution names its own.

    Either violation is refused at stage 0. This mirrors the activation record's optional identity (nvx-rs: a canonical activation record for platform clones #838): an adapter's clones use one model or the other.

  • The worker enters each execution through nvx::guest::entry:

    • new namespaces, with the root moved over /;
    • the configured hostname;
    • the image's own machine ID and no time status, so Entry::bind_runtime binds nothing.

    Entry::check refuses, at stage 0, an identity outside the entry's allowlist. Until runtime adapter: run cold sandboxes end to end on KVM, MSHV and WHP #822's PR 10 says whether listeners are published, executions enter as published, the entry's default, which refuses NET_BIND_SERVICE and NET_RAW.

  • Tests:

    • the encodings, and each refusal;
    • in guest_session.rs, with a fake root: the entry's inputs; the account check of a sandbox-wide identity.

PR 3 of 3: the host's command, and executions in VMs. Needs PR 2.

  • A host-side driver runs one nvx::exec::Peer per execution over the client. It comes from the test broker's host side (tests/guest_session.rs:2081).
  • runtime-adapter-host exec STORE [--identity ...] [--cwd DIR] [--env NAME=VALUE]... -- PROGRAM [ARG]... runs one or more executions over one session, and prints their outputs and outcomes.
  • Tests, in VMs on KVM, MSHV and WHP:
    • output beyond 1 MiB;
    • several executions at once;
    • a descendant that leaves the process group, which its execution's settlement ends;
    • an execution under a non-root identity, and one as UID 0 with an allowed capability;
    • a sandbox-wide identity, which executions run as;
    • a sandbox-wide identity whose account the image lacks, which fails before the guest answers.

Not in scope:

Done when:

  • the three PRs are merged;
  • the adapter's checks pass in every configuration, as its README lists them;
  • the in-process tests still run where they can't create cgroups.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions