Skip to content

fix(controller): detect and warn on a shared writable rootfs - #318

Closed
jrimmer wants to merge 2 commits into
deeplethe:devfrom
jrimmer:fix/shared-rw-rootfs-guard
Closed

jrimmer wants to merge 2 commits into
deeplethe:devfrom
jrimmer:fix/shared-rw-rootfs-guard

Conversation

@jrimmer

@jrimmer jrimmer commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Detection and an opt-in guard for the defect in #317 — the bridge, not the per-child writable layer, because the layer turns out to hinge on a question I would rather you answer than assume.

What I found that changes the options in #317

I proposed a host-side per-child reflink there. That option does not exist: Firecracker serialises the drive's path_on_host and is_read_only into the vmstate and reopens both verbatim at restore, and there is no way to override either at load time. I searched upstream v1.12 for drive_overrides and there is no such field, so the host cannot re-point a child's rootfs, and forkd's JSON path rewriting cannot reach it because the vmstate is binary. Host-side per-child copies would need a Firecracker feature first.

That leaves the guest side, and the guest side has the same shape of problem: the boot path cannot tell a child from a parent either — the kernel command line is frozen in the vmstate too. So a per-child layer needs a mechanism for a restored guest to learn it is a child (and then mount the shared base read-only with its own upper). That is a design decision, not an implementation detail, which is why #317 asks rather than assumes.

What lands here

The part every option needs, and that is safe on its own.

  • Snapshot.rootfs_read_only: Option<bool> — purely additive, serde defaulted and skipped when absent, so existing snapshot.json files stay byte-identical (the same pattern as volumes / parent_tag / rootfs). The CLI bake records the flag it actually booted with.
  • Snapshot::rootfs_is_read_only() — the recorded flag wins; for pre-flag snapshots an .ext4 rootfs is inferred writable, because that is exactly the convention the boot path already uses to choose BootConfig::ext4_rw (rw_flag || extension == "ext4"). Anything else returns None, which callers must read as "cannot vouch for it" rather than "safe".
  • The controller warns on any restore that would add a second writer to a writable-rootfs snapshot, naming the live holders and both remedies (re-bake read-only, or one child at a time). FORKD_REFUSE_SHARED_RW=1 upgrades that to a 409.

Why the refusal is opt-in

Refusing rejects fork -n N on a writable snapshot — and that is the flow from-image prints as its next step, with rw hardcoded, while restore_many_with is built to share the inode across n children. Making that a hard error by default is a behaviour change that belongs to you, not to me. Set the flag and the current "one writer at a time" discipline becomes an enforced invariant; leave it unset and operators get the signal without a broken workflow.

Verification

cargo fmt --check and cargo clippy --all-targets --all-features -D warnings clean; cargo test -p forkd-vmm --lib 55 passed, -p forkd-controller --lib 114 passed. A unit test covers the flag resolution (recorded wins both ways; .ext4 inferred; non-ext4 and no-rootfs stay unknown).

I have not written a test that exercises the warning through create_sandbox — building an AppState for it is heavy, and I would rather add it in the shape you would want than guess. Deployed on our own host with FORKD_REFUSE_SHARED_RW=1 as the interim.

Firecracker serialises the rootfs drive's `path_on_host` and
`is_read_only` into the vmstate and reopens both verbatim at restore.
Neither can be overridden at load time: the API has no `drive_overrides`
(searched upstream v1.12 — no such field exists) and the vmstate is a
binary blob, not JSON, so forkd's JSON path rewriting cannot reach it.

A snapshot baked with a writable rootfs therefore gives every restored
child the SAME ext4 opened read-write. Two concurrent children are two
guest kernels writing one filesystem with no coordinator: `/usr/bin/uname`
picks up another file's bytes (still a plausible ELF, so an
"it executes" check passes), directory entries return EBADMSG, and the
damage surfaces as a random compile failure rather than a sandbox error.

This is the bridge, not the destination. Per-child writable layers need
the guest to know it is a child, which is a design decision (issue deeplethe#317)
because the boot path cannot tell it: the command line is frozen in the
vmstate too. What lands here is the part every option needs and that is
safe on its own:

- `Snapshot.rootfs_read_only: Option<bool>` — purely additive, serde
  defaulted and skipped when absent, so existing snapshot.json files
  stay byte-identical. The CLI bake records it.
- `Snapshot::rootfs_is_read_only()` resolves the recorded flag, and for
  pre-flag snapshots infers an `.ext4` rootfs as writable (the same
  convention the boot path uses to pick `ext4_rw`), leaving anything
  else unknown rather than asserting it is safe.
- The controller warns on a restore that would add a second writer to a
  writable-rootfs snapshot, naming the live holders and both remedies.
  `FORKD_REFUSE_SHARED_RW=1` turns that into a 409.

The refusal is opt-in on purpose: it rejects `fork -n N` on a writable
snapshot, which is the flow `from-image` prints as its next step, and
that behaviour change should be the operator's call rather than a
surprise in a patch release.

Signed-off-by: jrimmer <jason@rimmer.net>
… the flag it is already honouring

Signed-off-by: jrimmer <jason@rimmer.net>
@jrimmer

jrimmer commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing this: #319 supersedes it, and keeping it would be actively wrong.

This warned — or refused, under FORKD_REFUSE_SHARED_RW=1 — when a restore would add a second writer to a writable-rootfs snapshot. With #319 the guest never writes the shared base at all (/ is mounted ro, writes go to a tmpfs/overlay layer in guest RAM), so concurrency is safe and the refusal would be a false positive blocking exactly the parallelism the fix restores.

The rootfs_read_only field it added to snapshot.json is therefore no longer needed for safety either; I dropped it from #319 rather than carry an unused field. The branch stays on my fork if you ever want the drive flag recorded for diagnostics.

@jrimmer jrimmer closed this Sep 11, 2026
@jrimmer
jrimmer deleted the fix/shared-rw-rootfs-guard branch September 11, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant