diff --git a/docs/designs/infra/runtime/compass-elastic-session-runtime/microvm-v3-egress-in-guest.md b/docs/designs/infra/runtime/compass-elastic-session-runtime/microvm-v3-egress-in-guest.md new file mode 100644 index 00000000..209fa26f --- /dev/null +++ b/docs/designs/infra/runtime/compass-elastic-session-runtime/microvm-v3-egress-in-guest.md @@ -0,0 +1,580 @@ +# microVM Runner V3 — egress-in-guest + +Status: PROPOSED — details the V3 milestone under the frozen parent +[microvm-runner.md](./microvm-runner.md) (its Plan § V3, +microvm-runner.md:492-506) and its frozen networking decision D6 +(microvm-runner.md:733-750). + +Ledger impact: none. V3 fills behavior the parent's own D6 already ratified +(the in-guest nft arm, guestd-as-root, agent-without-NET_ADMIN); nothing here +is a new cross-cutting decision, and `docs/designs/DECISIONS.md` is untouched. + +## Problem / Intent + +V2b built the gate but not the arm: guestd's `Provision` transitions +`ready → provisioned` and opens the exec gate, but a non-empty `nft_script` is +a hard `CodeUnimplemented` error ("nft egress arm is V3", +`go/internal/guestd/supervisor.go:146-149`), and `MicroVMRuntime.Start` sends +`ProvisionRequest` with only `default_exec_uid` + `base_env` — no script +(`go/internal/runtime/microvm_lifecycle.go:304-310`). So today a microVM +session boots with **open egress**: passt does no filtering (D6, +microvm-runner.md:739-741), and nothing arms nft inside the guest. + +V3 closes that: deliver `EgressPolicy.NftScript()` (consumed unchanged, +`go/internal/runtime/egress.go:71-107`) to guestd, run it **as guest root +before the exec gate opens**, fail the boot when arming fails, and make the +host's arm path backend-correct while leaving the podman path byte-identical +(parent § V3, microvm-runner.md:494-496). The proto surface already exists — +`ProvisionRequest.nft_script = 1` was seeded for exactly this +(`proto/compass/v1/guest_control.proto:177-180`) — so V3 is a behavior fill, +not a wire change (§(e)). + +## Approach + +Each subsection resolves one fork the parent's V3 plan leaves to detailing. +The load-bearing arm-routing fork is (a)-(c); every resolution is also listed +in `## Open Questions` for the pre-freeze batch, and the body designs against +the recommended option. + +### (a) How `NftScript()` reaches the backend: `ContainerSpec.Egress` + +Today the policy stops at the `AgentRuntime` layer: `AgentSpec.Egress` +(`go/internal/runtime/agent.go:40-42`) is consumed only by +`AgentRuntime.armEgress`, which execs the script into the running container +(`agent.go:303-309`). `ContainerSpec` — the only thing a `ContainerRuntime` +backend ever sees (`go/internal/runtime/podman.go:88-114`) — carries no egress +field. The V2b record already recorded this exact gap as V3's inheritance: +"the intended data path is `ContainerSpec` growing an egress field captured at +Create and delivered by Start's Provision call" +(microvm-v2b-guest-supervisor-exec.md:214-226). + +**Resolution: `ContainerSpec` grows `Egress EgressPolicy`.** + +- `AgentRuntime.createAndStart` sets it from `spec.Egress` when assembling the + `ContainerSpec` (`agent.go:262-272`). +- `PodmanCLI` **ignores** the field entirely: `createArgs` is untouched, so the + podman argv — and the whole podman path — stays byte-identical. Podman keeps + arming via the post-start `armEgress` exec as before ((c)). +- `MicroVMRuntime.Create` records `spec.Egress.NftScript()` on the + `microvmSession` (beside the `uid`/`env` it already records "for the + Provision RPC Start issues", `microvm_lifecycle.go:96-98,164-172`). + +The alternative — a distinct host→guest arm call after Start — is rejected in +(b); a distinct *host-side* delivery channel (e.g. a file in the workspace +share) is rejected outright: the script is host-assembled trusted input for a +guest-root shell, and the vsock control channel is the only surface with the +right trust direction (parent §(c), microvm-runner.md:144-151). + +### (b) Who arms: `MicroVMRuntime.Start`, intrinsically — the load-bearing fork + +Podman's shape is a **post-start root-capable exec**: `AgentRuntime.provision` +runs `armEgress` first among the post-start steps, exec-ing +`sh -c NftScript()` as the image default user (uid 1000) holding the +container's `CAP_NET_ADMIN` grant (`agent.go:288-309`, the grant at +`agent.go:266`). The microVM backend cannot reuse that seam verbatim: the +guest supervisor refuses uid-0 execs before any spawn +(`guest_control.proto:99-102`, `supervisor.go:428-441`), every spawned child +gets an empty capability set (`supervisor.go:55-61`), and `spec.CapAdd` is +ignored — CAP_NET_ADMIN is never granted to the workload boundary +(`microvm_lifecycle.go:138-143`; D6, microvm-runner.md:741-743). The arm must +run as **guest root inside guestd**, before the gate opens. + +Two candidate owners for issuing the arm: + +- **Option 1 (recommended): arming is intrinsic to `Start`.** + `MicroVMRuntime.Start` already issues the `Provision` RPC as its final + transactional step (`microvm_lifecycle.go:269-310`); V3 adds + `NftScript: session.nftScript` to that same request. One RPC provisions + *and* arms; the gate opens only when both succeed. + - Pro: the `ContainerRuntime` contract identity holds — on podman, + `Start` then `Exec` works with no intermediate call, and the V2b contract + suite asserts exactly that identity on both backends + (contract_microvm_test.go:5-9, microvm_lifecycle_test.go's + Create→Start→Exec shape). Any design where the gate opens *outside* Start + breaks that: an `Exec` right after `Start` would be gate-refused on + microVM only. + - Pro: fail-closed teardown comes for free — Start's existing + tear-down-on-any-failure posture (`microvm_lifecycle.go:286-295`) makes a + failed arm tear the VM down with no new code path (§(d)). + - Con: the arm's *timing* moves relative to podman (during Start, not during + `AgentRuntime.provision`). This is strictly earlier, so "armed before any + exec" is preserved with margin; flagged against the parent's phrasing in + OQ-1. +- **Option 2 (rejected): a distinct arm call between Start and the first + exec** — e.g. `Provision` split into a gate RPC and an arm RPC, or the arm + riding a second host call. This keeps the podman *sequence* (start, then + arm) but leaves a window where the gate design must choose between (i) gate + open before arm — violating the parent's "Only after a successful arm does + the supervisor accept exec requests" (microvm-runner.md:152-155) — or (ii) + gate closed until the extra call — breaking the Start→Exec contract identity + above. Both horns lose; rejected. + +### (c) What the host `armEgress` does on the microVM backend: probe-and-skip + +With (b), the arm has already happened by the time `AgentRuntime.provision` +runs. `armEgress`'s exec — nil `User`, so the session-default uid 1000 +(`podman.go:119-121`, `microvm_lifecycle.go:432-438`) with an empty capability +set on this backend (`supervisor.go:55-61`) — would not just be redundant, it +would **fail** (nft as a capability-less uid), failing every microVM provision. +It +must not run on this backend. Three candidates: + +- **Option A (rejected): grow `ContainerRuntime` with an + `ArmEgress(ctx, id, EgressPolicy) error` verb** (podman impl = today's exec + moved verbatim; microVM impl = no-op). Clean in the abstract, but it + violates the interface's freeze discipline — the surface was deliberately + completed at S1 so "no interface change lands after S1" + (`podman.go:379-388`, the `Resize` reservation) — and it touches every fake + and the shared contract suite for a verb one backend no-ops. Largest blast + radius of the three. +- **Option B (rejected): hoist arming into both backends' `Start`** (podman + gains a session table to carry the spec from Create to Start, and issues the + same exec from inside `PodmanCLI.Start`). This deletes `armEgress` entirely + and makes always-armed a property of every backend — but `PodmanCLI` is + stateless today and the podman path is under an explicit + changed-nothing constraint (parent § V3 "the podman path unchanged", + microvm-runner.md:496). Restructuring podman internals to satisfy a microVM + milestone is the opposite of small blast radius; rejected. +- **Option C (recommended): a backend capability probe in + `AgentRuntime.provision`.** `MicroVMRuntime` gains one exported marker + method, `EgressArmedInGuest() bool` (returns true), NOT on the + `ContainerRuntime` interface. `AgentRuntime.provision` type-asserts an + unexported single-method interface and skips `armEgress` when the backend + self-arms: + + ```go + // in agent.go + type inGuestEgressArmer interface{ EgressArmedInGuest() bool } + + func (r *AgentRuntime) provision(ctx context.Context, id ContainerID, spec AgentSpec) error { + if armer, ok := r.runtime.(inGuestEgressArmer); !ok || !armer.EgressArmedInGuest() { + if err := r.armEgress(ctx, id, spec.Egress); err != nil { + return err + } + } + // installCredentials, ensureCheckoutDir unchanged (agent.go:294-297) + ``` + + `PodmanCLI` does not implement the probe, so the assertion fails and the + existing `armEgress` exec runs byte-identically (`agent.go:300-309`); fakes + don't implement it either, so every existing hermetic test (e.g. + `TestEgressIsArmedAsRootNotTheAgentUser`, agent_test.go:213-216) keeps + passing unchanged. The frozen interface is untouched. Smallest blast radius: + one marker method + one guarded call site. + +### (d) The guest arm: guestd runs the script as root, fail-closed + +`Provision` (`supervisor.go:141-169`) replaces its `CodeUnimplemented` branch +with the real arm. Handler order (all before the `stateProvisioned` +transition): + +1. Validate `default_exec_uid` non-zero (existing, `supervisor.go:150-153`). +2. Check state: still refuse when not `stateReady` + (`supervisor.go:161-163`) — this also guarantees the arm's preconditions: + `stateReady` means net bringup completed, so the guest IP, default route, + and `/etc/resolv.conf` exist (`go/internal/guestd/net.go:24-34`), which the + script requires (it parses resolv.conf for the DNS carve-out and resolves + allowlisted hosts via `getent`, `egress.go:71-79,109-115`; D6, + microvm-runner.md:747-748). +3. **Arm** when `nft_script` is non-empty: spawn `/bin/sh -c