Conversation
Firecracker serialises the rootfs drive's `path_on_host` and `is_read_only` into the vmstate and reopens both verbatim at restore. Neither can be overridden at load time: the API has no `drive_overrides` (searched upstream v1.12 — no such field exists) and the vmstate is a binary blob, not JSON, so forkd's JSON path rewriting cannot reach it. A snapshot baked with a writable rootfs therefore gives every restored child the SAME ext4 opened read-write. Two concurrent children are two guest kernels writing one filesystem with no coordinator: `/usr/bin/uname` picks up another file's bytes (still a plausible ELF, so an "it executes" check passes), directory entries return EBADMSG, and the damage surfaces as a random compile failure rather than a sandbox error. This is the bridge, not the destination. Per-child writable layers need the guest to know it is a child, which is a design decision (issue deeplethe#317) because the boot path cannot tell it: the command line is frozen in the vmstate too. What lands here is the part every option needs and that is safe on its own: - `Snapshot.rootfs_read_only: Option<bool>` — purely additive, serde defaulted and skipped when absent, so existing snapshot.json files stay byte-identical. The CLI bake records it. - `Snapshot::rootfs_is_read_only()` resolves the recorded flag, and for pre-flag snapshots infers an `.ext4` rootfs as writable (the same convention the boot path uses to pick `ext4_rw`), leaving anything else unknown rather than asserting it is safe. - The controller warns on a restore that would add a second writer to a writable-rootfs snapshot, naming the live holders and both remedies. `FORKD_REFUSE_SHARED_RW=1` turns that into a 409. The refusal is opt-in on purpose: it rejects `fork -n N` on a writable snapshot, which is the flow `from-image` prints as its next step, and that behaviour change should be the operator's call rather than a surprise in a patch release. Signed-off-by: jrimmer <jason@rimmer.net>
… the flag it is already honouring Signed-off-by: jrimmer <jason@rimmer.net>
Contributor
Author
|
Closing this: #319 supersedes it, and keeping it would be actively wrong. This warned — or refused, under The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detection and an opt-in guard for the defect in #317 — the bridge, not the per-child writable layer, because the layer turns out to hinge on a question I would rather you answer than assume.
What I found that changes the options in #317
I proposed a host-side per-child reflink there. That option does not exist: Firecracker serialises the drive's
path_on_hostandis_read_onlyinto the vmstate and reopens both verbatim at restore, and there is no way to override either at load time. I searched upstream v1.12 fordrive_overridesand there is no such field, so the host cannot re-point a child's rootfs, and forkd's JSON path rewriting cannot reach it because the vmstate is binary. Host-side per-child copies would need a Firecracker feature first.That leaves the guest side, and the guest side has the same shape of problem: the boot path cannot tell a child from a parent either — the kernel command line is frozen in the vmstate too. So a per-child layer needs a mechanism for a restored guest to learn it is a child (and then mount the shared base read-only with its own upper). That is a design decision, not an implementation detail, which is why #317 asks rather than assumes.
What lands here
The part every option needs, and that is safe on its own.
Snapshot.rootfs_read_only: Option<bool>— purely additive,serdedefaulted and skipped when absent, so existingsnapshot.jsonfiles stay byte-identical (the same pattern asvolumes/parent_tag/rootfs). The CLI bake records the flag it actually booted with.Snapshot::rootfs_is_read_only()— the recorded flag wins; for pre-flag snapshots an.ext4rootfs is inferred writable, because that is exactly the convention the boot path already uses to chooseBootConfig::ext4_rw(rw_flag || extension == "ext4"). Anything else returnsNone, which callers must read as "cannot vouch for it" rather than "safe".FORKD_REFUSE_SHARED_RW=1upgrades that to a409.Why the refusal is opt-in
Refusing rejects
fork -n Non a writable snapshot — and that is the flowfrom-imageprints as its next step, withrwhardcoded, whilerestore_many_withis built to share the inode across n children. Making that a hard error by default is a behaviour change that belongs to you, not to me. Set the flag and the current "one writer at a time" discipline becomes an enforced invariant; leave it unset and operators get the signal without a broken workflow.Verification
cargo fmt --checkandcargo clippy --all-targets --all-features -D warningsclean;cargo test -p forkd-vmm --lib55 passed,-p forkd-controller --lib114 passed. A unit test covers the flag resolution (recorded wins both ways;.ext4inferred; non-ext4 and no-rootfs stay unknown).I have not written a test that exercises the warning through
create_sandbox— building anAppStatefor it is heavy, and I would rather add it in the shape you would want than guess. Deployed on our own host withFORKD_REFUSE_SHARED_RW=1as the interim.