Skip to content

fix(vmm): reap the Firecracker child on every failed boot - #315

Merged
WaylandYang merged 1 commit into
deeplethe:devfrom
jrimmer:fix/vm-boot-orphan-firecracker
Sep 14, 2026
Merged

WaylandYang merged 1 commit into
deeplethe:devfrom
jrimmer:fix/vm-boot-orphan-firecracker

Conversation

@jrimmer

@jrimmer jrimmer commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Relevant to #301 (the orphan reap/gate remainder you left open).

The window

Vm::boot spawned Firecracker and then drove /boot-source, /drives/rootfs, /entropy and /actions with a bare std::process::Child in hand. Child has no kill-on-drop, so any ? between the spawn and the Ok(Vm { .. }) at the end — wait_for_sock timing out, or any API call failing after the process was up — returned an error and abandoned a live Firecracker.

restore_many_with wraps its children in a Vm immediately, so boot was the only place in the crate that could orphan a process with no handle left.

Why it is permanent

The orphan keeps the sandbox's tap device and rootfs fd open, and the tap name is frozen into the vmstate. Every later spawn of that snapshot therefore dies inside Firecracker with Open tap device failed before it can create its API socket, and the caller sees:

restore_many: socket .../child-N.sock never appeared

for good. The snapshot stays unusable until the orphan is killed by hand — a failed boot converts into a permanently failing tag. In our deployment this is what made a single bad boot look like an exhausted netns allocator, because the symptom (every subsequent spawn failing at socket wait) is identical.

Fix

The child is owned by a PendingFirecracker guard that kills and reaps it on every early return and hands it to the Vm only on success. take() is the single hand-over point; the guard mirrors the existing WorkDirGuard idiom in the test module.

Two unit tests: the drop path (the owned pid must actually be gone, not merely untracked) and the take path (the handed-over child must still be live).

Verification, and the gap

cargo fmt --check and cargo clippy --all-targets --all-features -D warnings clean; cargo test -p forkd-vmm pending_firecracker passes.

The gap: this is unit-level only. I have not run a live bake on a KVM host to watch a boot fail mid-window and confirm no orphan survives, and given your point on #[ignore]d tests I would rather say so than imply otherwise. I can run that on our KVM host if you want it before review.

`Vm::boot` spawned Firecracker and then drove `/boot-source`,
`/drives/rootfs`, `/entropy` and `/actions` with a bare
`std::process::Child` in hand. `Child` has no kill-on-drop, so any `?`
in that window — `wait_for_sock` timing out, or an API call failing
after the process was up — returned an error and abandoned a *live*
Firecracker.

The orphan keeps the sandbox's tap device and rootfs fd open, and the
tap name is frozen into the vmstate, so every later spawn of that
snapshot dies inside Firecracker with `Open tap device failed` before
it can create its API socket. Callers see
`socket .../child-N.sock never appeared` and the snapshot stays
unusable until the orphan is killed by hand — the permanent-failure
mode in deeplethe#301.

Own the child in a `PendingFirecracker` guard that reaps it on every
early return and hands it to the `Vm` only on success. `restore_many_with`
already wrapped its children immediately, so `boot` was the only window
in the crate that could orphan a process with no handle left.

Two unit tests cover the drop path (the pid must be gone, not just
untracked) and the take path (the handed-over child must still be live).

Signed-off-by: jrimmer <jason@rimmer.net>
@WaylandYang
WaylandYang marked this pull request as ready for review September 14, 2026 04:56

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: the guard closes the only spawn→Vm window in Vm::boot that could abandon a live Firecracker, and take() is the single hand-over. Verified on Linux (Ubuntu 22.04, stable Rust in Docker): fmt, clippy -D warnings, and cargo test --all all green, both on its own and merged with #316. Not exercised against a live KVM boot failure, because the test host has no KVM. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants