fix(vmm): reap the Firecracker child on every failed boot - #315
Merged
WaylandYang merged 1 commit intoSep 14, 2026
Merged
Conversation
`Vm::boot` spawned Firecracker and then drove `/boot-source`, `/drives/rootfs`, `/entropy` and `/actions` with a bare `std::process::Child` in hand. `Child` has no kill-on-drop, so any `?` in that window — `wait_for_sock` timing out, or an API call failing after the process was up — returned an error and abandoned a *live* Firecracker. The orphan keeps the sandbox's tap device and rootfs fd open, and the tap name is frozen into the vmstate, so every later spawn of that snapshot dies inside Firecracker with `Open tap device failed` before it can create its API socket. Callers see `socket .../child-N.sock never appeared` and the snapshot stays unusable until the orphan is killed by hand — the permanent-failure mode in deeplethe#301. Own the child in a `PendingFirecracker` guard that reaps it on every early return and hands it to the `Vm` only on success. `restore_many_with` already wrapped its children immediately, so `boot` was the only window in the crate that could orphan a process with no handle left. Two unit tests cover the drop path (the pid must be gone, not just untracked) and the take path (the handed-over child must still be live). Signed-off-by: jrimmer <jason@rimmer.net>
WaylandYang
marked this pull request as ready for review
September 14, 2026 04:56
WaylandYang
approved these changes
Sep 14, 2026
Contributor
There was a problem hiding this comment.
Reviewed: the guard closes the only spawn→Vm window in Vm::boot that could abandon a live Firecracker, and take() is the single hand-over. Verified on Linux (Ubuntu 22.04, stable Rust in Docker): fmt, clippy -D warnings, and cargo test --all all green, both on its own and merged with #316. Not exercised against a live KVM boot failure, because the test host has no KVM. Thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant to #301 (the orphan reap/gate remainder you left open).
The window
Vm::bootspawned Firecracker and then drove/boot-source,/drives/rootfs,/entropyand/actionswith a barestd::process::Childin hand.Childhas no kill-on-drop, so any?between the spawn and theOk(Vm { .. })at the end —wait_for_socktiming out, or any API call failing after the process was up — returned an error and abandoned a live Firecracker.restore_many_withwraps its children in aVmimmediately, sobootwas the only place in the crate that could orphan a process with no handle left.Why it is permanent
The orphan keeps the sandbox's tap device and rootfs fd open, and the tap name is frozen into the vmstate. Every later spawn of that snapshot therefore dies inside Firecracker with
Open tap device failedbefore it can create its API socket, and the caller sees:for good. The snapshot stays unusable until the orphan is killed by hand — a failed boot converts into a permanently failing tag. In our deployment this is what made a single bad boot look like an exhausted netns allocator, because the symptom (every subsequent spawn failing at socket wait) is identical.
Fix
The child is owned by a
PendingFirecrackerguard that kills and reaps it on every early return and hands it to theVmonly on success.take()is the single hand-over point; the guard mirrors the existingWorkDirGuardidiom in the test module.Two unit tests: the drop path (the owned pid must actually be gone, not merely untracked) and the take path (the handed-over child must still be live).
Verification, and the gap
cargo fmt --checkandcargo clippy --all-targets --all-features -D warningsclean;cargo test -p forkd-vmm pending_firecrackerpasses.The gap: this is unit-level only. I have not run a live bake on a KVM host to watch a boot fail mid-window and confirm no orphan survives, and given your point on
#[ignore]d tests I would rather say so than imply otherwise. I can run that on our KVM host if you want it before review.