feat(vmm,init): per-child writable layer in guest RAM; bakes boot the rootfs read-only

[jrimmer]

This is option A from #317, implemented.

Why A, and not the role signal

The constraints I confirmed while investigating leave only one enforceable property:

• Firecracker reopens the drive's path_on_host and is_read_only from the binary vmstate, and there is no drive_overrides to override either. The host cannot give a child its own rootfs.
• The kernel command line is frozen in that same vmstate, so the guest cannot be told at boot which role it plays.

Since a child cannot be handed a different rootfs, the only thing the host can guarantee is that the shared one is never written. That is what this does, and it needs no role signal — which matters, because the alternative is a boot-time protocol on the path where a wrong answer means a child writes the shared ext4, i.e. exactly the corruption being fixed. A's failure mode is bounded writable space; B's is silent corruption.

I also corrected my own first framing of A: it does not lose the warm caches. The writable upper is guest memory, and memory.bin is what a snapshot captures — the property the existing /tmp tmpfs comment already relies on ("tmpfs state survives BRANCH because it lives in guest RAM, which is what memory.bin captures"). This change is that comment generalised to the rest of the filesystem, so the writable state a bake warms is present in every child.

What changes

Host. BootConfig::ext4_overlay opens the drive read-only (rootfs_read_only: true) and boots root=/dev/vda ro init=/forkd-init.sh. Both halves are load-bearing: the host fd and the guest mount. forkd snapshot and the daemon's create_snapshot use it; ext4_rw and the CLI's --rw stay for single-writer use.

Guest. /forkd-init.sh builds the writable layer in RAM:

plain tmpfs   /run /dev/shm /tmp          scratch that owns nothing
overlay       /etc /root /home /opt       image content as the lower layer,
              /srv /usr/local /var        writes in the tmpfs upper

Overlay rather than bare tmpfs for those trees, so a preinstalled toolchain or a cache warmed at bake time stays visible instead of being masked. /etc is on the list because the existing DNS fix writes resolv.conf. Paths the image does not have are skipped (a read-only root cannot be mkdir'd into), and a write outside the set fails with EROFS — loud, rather than silently mutating the shared base.

Side effect worth noting: because the writable state is memory, the rootfs a tag ships is now the pristine base, so tags baked from the same image can share a rootfs sidecar.

The two trade-offs, deliberately chosen

• Guest RAM bounds writable space. The tmpfs is capped at 2 GiB (forkd.rw_size=<size> on the cmdline via BootConfig::with_rw_size), so a job that outgrows it fails its own write with ENOSPC rather than OOMing the guest. Counterpart: mem_size_mib now has to cover the job's writable footprint, not just its working set.
• A write outside the provided set is an error. That is the honest failure for a read-only root, but it is a behaviour change for any job that writes somewhere unexpected — say /usr/bin during a build step.

Evidence, and where it is thin

cargo fmt --check and cargo clippy --all-targets --all-features -D warnings clean; cargo test -p forkd-vmm 57, -p forkd-controller 114, -p forkd-cli 53, all passing. Two unit tests cover the config (read-only on both sides, no rw token, init= present, cmdline size hint).

The gap is boot-level, and I want to be straight about it: the unit tests prove the arguments are what I intend, not that a guest boots, mounts ro, and keeps the base byte-identical under two concurrent children. That needs a real bake on a KVM host, and I am running it on ours now — I will post the transcript here (guest mounts, base hash before/after, two children writing the same paths) rather than leave it asserted. Until that lands, treat the guest-side change as unverified.

Existing tags keep the read-write drive their vmstate froze and need re-baking to get the layer.

[jrimmer]

Boot-level evidence, from our KVM host

Baked a throwaway tag from a small image carrying the new /forkd-init.sh, then restored two children concurrently (per_child_netns=true, since shared-TAP spawns are single-child by design) and had both write 20 MB to the same guest path at the same time. The tag's rootfs was hashed before and after.

base sha256 before : 2ed42680b7b31638a520afc160ed0c2d56278e41bcd28f103163ae1bf882cd43
children           : ['sb-6aa48936-000000000000000d', 'sb-6aa48936-000000000000000e']
--- child A
MOUNT_/: /dev/root / ext4 ro,relatime 0 0
OVERLAY_MOUNTS: 7
f211e953068458fe4541ace30b484c11320385e466c92f8919146a9378e884c7  /root/probe.bin
FIRST_BYTES: AAAAAAAA
--- child B
MOUNT_/: /dev/root / ext4 ro,relatime 0 0
OVERLAY_MOUNTS: 7
cf27f98c0bf605e7177937c772e8b4ec94134a57b08614915bb7f25918900ce7  /root/probe.bin
FIRST_BYTES: BBBBBBBB
base sha256 after  : 2ed42680b7b31638a520afc160ed0c2d56278e41bcd28f103163ae1bf882cd43
BASE UNCHANGED

Reading it:

• / is ro in the guest, and all seven overlays (/etc /root /home /opt /srv /usr/local /var) mounted.
• Both children wrote the same path, got different payload hashes, and each read back its own bytes — the cross-contamination that made concurrent restores unsafe is gone.
• The shared base is byte-identical across the whole run. That is the property the fix is for: the file every child opens was never written.
• The guest kernel has overlayfs built in (nodev overlay in /proc/filesystems, kernel 6.1.141) — I checked that before relying on it.

Guest RAM was 2 GiB with the default 2 GiB writable cap, so this run wrote 20 MB twice into RAM comfortably; the cap is what a job hits first if it outgrows the budget, and with_rw_size is the knob.

The tag and the loom-in image are throwaway (overlay-test) and the two children are cleaned up; nothing else on the host changed. Worth knowing if you try this: multi-child restores need per-child netns, so scripts/netns-setup.sh N has to cover concurrency you intend to use — our pool had claimed all twelve.

[jrimmer]

Gap in this PR: the writable set is a fixed path list, and the build workspace is not in it

Found while writing up what CI jobs need to know, with a live restore of a tag baked by this branch:

ls /workspace: ls: cannot access '/workspace': No such file or directory
mkdir /workspace  rc=1  mkdir: cannot create directory '/workspace': Read-only file system
write /root  rc=0
write /tmp   rc=0
write /      rc=1  Read-only file system

The workspace is /workspace in our runner, and it is not in the image and not on the overlay list. Under a read-only root the mount point has to exist already, and a top-level path that the image does not ship cannot be created — so a job fails at checkout, before it runs anything.

That generalises: a fixed list of seven trees only covers write paths we happened to think of, and images differ — the Elixir image we use also carries kaniko for image pushes, which writes /kaniko. Every such path is a runtime EROFS failure that depends on what the image does.

Three ways out, ranked:

1. Overlay / itself (lowerdir /, tmpfs upper) and pivot_root into it. Then there is no path list and no mount-point problem — every path is writable, and the base is still never written. This is the general fix and what I would rather do, but it belongs in the guest init carefully (the merged root has to be a mount point, and PID 1 has to re-exec), so I did not want to bolt it onto this draft untested.
2. The image ships the paths. scripts/build-rootfs.sh creates them and the init overlays them. Keeps the current shape, but it turns "what can a job write" into a per-image build convention.
3. The workspace moves under an overlaid tree, e.g. /var/workspace. Fixes our runner only, and any other absolute path a job uses still breaks.

I would like your read on 1 before I spend the effort: it is the version that makes a read-only root invisible to jobs, which seems like the property we actually want. Until then this PR should not merge, and I would not re-bake any tag with it.

[jrimmer]

One more gap this PR does not cover: volumes.

VolumeSpec.read_only defaults to false (#[serde(default)] on a bool), so an attached volume is writable by default — and like the rootfs, its host path is frozen into the vmstate. That makes it a second shared-mutable surface: every child restored from the tag opens the same writable image file, which is the corruption this PR exists to remove, just on /dev/vdb instead of /dev/vda.

The init would need the same treatment for volumes: mount read-only and put the writable layer for guest_path on the tmpfs upper, rather than trusting the host-side flag. Not an issue for us — none of our tags attach a volume — but it should be handled here rather than discovered later.

[jrimmer]

Withdrawing this. The approach is wrong at the platform level, not just incomplete, and I would rather say so than keep patching it.

What we learned by building and booting it:

• A writable rootfs is not a CI problem. Our own consumers — interactive dev sandboxes, per-PR environments, in-sandbox agents (Shelley/MCP/ACP), plus the build runner — all need somewhere to write. Two interactive sandboxes of one image share that rootfs today, so a developer's npm install can be corrupted by someone else's session with no CI involved. The corruption is platform-wide; CI just made it frequent.
• The fixed path list does not survive contact with real images — /workspace for our runner, /kaniko for the image-builder tag — and a read-only root cannot create a mount point it does not already ship.
• RAM is the wrong medium for it. One shared 2 GiB budget, copy-up charging whole files, and a suspend that dumps the writable layer into memory.bin — exactly what long-lived interactive sandboxes cannot afford.
• Volumes are the same defect on a different device, and that one is still open.

The model that fits every consumer is: immutable shared rootfs + a per-sandbox writable volume. forkd already has the second half — VolumeSpec, and suspend/resume documented as workspace-backed only — so what is missing is not storage, it is letting a restored child be given its own drive paths. Both the rootfs and the volume path are frozen into the vmstate, which is what drive_overrides would fix.

I am filing a follow-up issue with the full evaluation and the ordered plan. The guest-init work here is not wasted: the overlay-upper mechanism becomes the writable layer with the volume as its upper instead of tmpfs, and stays as the no-volume fallback.