Add Sandbox::recreate to recover unrecoverable sandboxes by ludfjig · Pull Request #1751 · hyperlight-dev/hyperlight

ludfjig · 2026-08-21T00:48:59Z

When a restore fails to remap a sandbox's base memory, the sandbox becomes Unrecoverable and rejects every operation, including restore(). The only way out is to drop it and build a new sandbox, which loses its configuration, host functions, and interrupt handles.

Add Sandbox::recreate(&mut self, snapshot), which replaces the sandbox's VM and memory with new ones built from snapshot.

The sandbox keeps its configuration, host functions, page table root finder, guest log level, and interrupt handles. Existing interrupt handles control the new VM.
The new VM is built before the old one is dropped. A failed recreate leaves the sandbox unchanged, so it can be retried.
Recreation accepts any status. restore() stays the cheaper choice for ready and poisoned sandboxes.
Like restore(), recreation removes regions added with map_region() or map_file_cow().

On Windows the new VM needs its own surrogate process while the old VM still holds one. Recreation fails with HYPERLIGHT_MAX_SURROGATES=0. When all surrogates are in use it waits, as creating any sandbox does. With HYPERLIGHT_MAX_SURROGATES=1 that wait never ends. The rustdoc documents this.

Closes #1747

ludfjig · 2026-08-21T00:49:21Z

@syntactically first draft at #1747

Copilot

Pull request overview

Adds a public MultiUseSandbox::recreate() API in hyperlight-host to rebuild an unrecoverable sandbox from scratch using its existing configuration and the snapshot that triggered the unrecoverable restore failure (addressing #1747’s “replace this sandbox with a new one with the same configuration” request).

Changes:

Track additional lifecycle state needed for recreation (store the failed-restore snapshot and sandbox config when transitioning to Unrecoverable).
Add MultiUseSandbox::recreate(self) -> Result<Self> plus tests validating what state/config is preserved vs not preserved.
Update errors, changelog, and test recipes to cover the new recreation flow.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Thread sandbox configuration into `MultiUseSandbox::from_uninit` so it can be reused for recreation.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Introduce `SandboxLifecycle`, store recreation snapshot/config, and implement `MultiUseSandbox::recreate()` with accompanying tests.
src/hyperlight_host/src/sandbox/host_funcs.rs	Add an internal constructor to rebuild `HostFunctions` from a `FunctionRegistry` during recreation.
src/hyperlight_host/src/error.rs	Add recreation-specific error variants and adjust unrecoverable messaging.
Justfile	Ensure the new unrecoverable-mapping test is exercised in crashdump/coverage recipes.
CHANGELOG.md	Document the new `MultiUseSandbox::recreate()` API.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

syntactically

I left a few comments inline.

I'm still trying to work out what the best API here is. It feels to me like it is a smaller/less breaking change to move this into restore() and add a new way to opt-out of the implicit recreate, but the possibility of recreation itself failing (which is probably pretty likely if we've recently failed to restore, since the relevant mapping failures seem likely to be resource exhaustion) is a little difficult to handle ergonomically. It seems inescapable to me that client code, if it is going to handle this locally, will need to do something like

while should_keep_trying() {
  match sb.<restore/recreate/whatever>(snapshot) {
    Ok(()) => break,
    Err(HyperlightError::UnrecoverableSandbox) => {
      // maybe try to free up resources somehow, try a different snapshot, etc
      continue;
    },
    Err(e) => return e,
}

which is just kind of nasty, whether you have one function for restore/recreate or multiple.

(The other option for clients I guess would be to just back out all the way to some top level retry when they get an unrecoverable error. That probably doesn't benefit at all from having recreate functionality at all, though...)

syntactically

Seems reasonable to me!

recreate() replaces a sandbox's VM and memory with new ones built from a snapshot. It recovers unrecoverable sandboxes. A failed recreate leaves the sandbox unchanged, so it can be retried. The sandbox keeps its configuration, host functions, page table root finder, guest log level, and interrupt handles. Existing interrupt handles control the new VM. Sandbox stores its configuration, which replaces the separate guest log level field. Closes hyperlight-dev#1747 Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-11-12T00:29:48Z

Benchmark Results

Measured commit: 3bf937b03211
Baseline commit: e364f7d5bb81

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`771.20 ns` (➖ 1.02x slower)
`vec_bytes`	`592.40 ns` (➖ 1.01x slower)
		`374.42 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`520.62 ns` (➖ 1.02x faster)
`65536`	`141.78 ns` (➖ 1.06x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.31 ms` (➖ 1.00x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`7.68 ns` (➖ 1.02x faster)	`7.69 ns` (➖ 1.00x slower)	`7.67 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`97.27 µs` (➖ 1.00x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.08 µs` (➖ 1.00x slower)	`6.96 µs` (➖ 1.04x faster)
`65536`	`1.92 µs` (➖ 1.08x faster)	`2.05 µs` (➖ 1.05x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`28.21 µs` (➖ 1.04x slower)
`65536`	`6.18 µs` (➖ 1.02x faster)	`6.38 µs` (➖ 1.01x slower)
`8192`	`1.07 µs` (➖ 1.01x slower)	`1.04 µs` (➖ 1.01x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`798.04 ns` (➖ 1.05x slower)
`vec_bytes`	`589.82 ns` (➖ 1.03x slower)
		`728.32 µs` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`535.21 ns` (➖ 1.07x faster)
`65536`	`188.91 ns` (➖ 1.14x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.71 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`7.21 ns` (➖ 1.11x faster)	`7.42 ns` (➖ 1.08x faster)	`7.49 ns` (➖ 1.07x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`55.48 µs` (➖ 1.04x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`8.02 µs` (➖ 1.11x faster)	`7.89 µs` (➖ 1.12x faster)
`65536`	`2.26 µs` (➖ 1.09x faster)	`2.22 µs` (➖ 1.13x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`30.43 µs` (➖ 1.16x faster)
`65536`	`7.57 µs` (➖ 1.10x faster)	`7.39 µs` (➖ 1.14x faster)
`8192`	`738.11 ns` (➖ 1.13x faster)	`982.92 ns` (➖ 1.15x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`933.30 ns` (➖ 1.00x faster)
`vec_bytes`	`721.44 ns` (➖ 1.02x slower)
		`317.68 µs` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`693.68 ns` (➖ 1.03x slower)
`65536`	`184.63 ns` (➖ 1.08x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`67.96 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.12 ns` (➖ 1.02x slower)	`10.00 ns` (➖ 1.01x slower)	`10.04 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`108.49 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.26 µs` (➖ 1.01x slower)	`9.24 µs` (➖ 1.02x slower)
`65536`	`2.58 µs` (➖ 1.05x slower)	`2.63 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`34.78 µs` (➖ 1.04x faster)
`65536`	`8.09 µs` (➖ 1.08x faster)	`8.09 µs` (➖ 1.00x faster)
`8192`	`1.24 µs` (➖ 1.01x faster)	`1.27 µs` (➖ 1.01x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`912.97 ns` (➖ 1.03x faster)
`vec_bytes`	`656.33 ns` (➖ 1.00x faster)
		`654.07 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`682.91 ns` (➖ 1.04x slower)
`65536`	`233.13 ns` (➖ 1.05x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`92.15 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`8.68 ns` (➖ 1.00x faster)	`8.80 ns` (➖ 1.01x slower)	`8.68 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`55.05 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.91 µs` (➖ 1.00x faster)	`7.87 µs` (➖ 1.01x faster)
`65536`	`2.25 µs` (➖ 1.01x faster)	`2.28 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`39.56 µs` (➖ 1.01x slower)
`65536`	`7.43 µs` (➖ 1.01x faster)	`7.50 µs` (➖ 1.06x faster)
`8192`	`905.14 ns` (➖ 1.02x faster)	`1.01 µs` (➖ 1.10x slower)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.22 µs` (➖ 1.04x slower)
`vec_bytes`	`775.11 ns` (➖ 1.02x faster)
		`2.29 ms` (➖ 1.06x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`788.48 ns` (➖ 1.00x slower)
`65536`	`237.13 ns` (➖ 1.03x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.81 ms` (➖ 1.06x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.19 ns` (➖ 1.02x slower)	`10.04 ns` (➖ 1.00x faster)	`10.12 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`969.10 µs` (➖ 1.03x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.34 µs` (➖ 1.03x faster)	`9.22 µs` (➖ 1.03x faster)
`65536`	`2.37 µs` (➖ 1.00x slower)	`2.44 µs` (➖ 1.05x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`38.48 µs` (➖ 1.18x faster)
`65536`	`8.84 µs` (➖ 1.00x faster)	`9.22 µs` (➖ 1.01x slower)
`8192`	`1.27 µs` (➖ 1.13x faster)	`1.28 µs` (➖ 1.08x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.23 µs` (➖ 1.00x faster)
`vec_bytes`	`789.46 ns` (➖ 1.01x faster)
		`3.04 ms` (➖ 1.03x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`756.10 ns` (➖ 1.09x faster)
`65536`	`214.17 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.18 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.76 ns` (➖ 1.04x faster)	`10.07 ns` (➖ 1.04x faster)	`9.80 ns` (➖ 1.22x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`680.43 µs` (➖ 1.00x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.71 µs` (➖ 1.00x slower)	`7.45 µs` (➖ 1.03x faster)
`65536`	`2.32 µs` (➖ 1.01x slower)	`2.26 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`40.17 µs` (➖ 1.05x faster)
`65536`	`8.39 µs` (➖ 1.01x faster)	`8.75 µs` (➖ 1.04x slower)
`8192`	`1.04 µs` (➖ 1.14x faster)	`1.11 µs` (➖ 1.03x slower)

_{Reported by cargo ci bench-report --candidate run:43123163833 --config-file bench_report.toml.}

ludfjig added the C-enhancement For PRs adding features, improving functionality, docs, tests, etc. label Aug 21, 2026

ludfjig changed the title ~~Add method for recreate unrecoverable sandbox~~ Add method for recreating unrecoverable sandbox Aug 21, 2026

ludfjig marked this pull request as ready for review August 21, 2026 17:06

Copilot AI lite review requested due to automatic review settings August 21, 2026 17:07

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners August 21, 2026 17:07

Copilot started reviewing on behalf of ludfjig August 21, 2026 17:10

View session

Copilot AI reviewed Aug 21, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/error.rs Resolved

Comment thread src/hyperlight_host/src/sandbox/initialized_multi_use.rs Resolved

ludfjig force-pushed the recover_unrecoverable branch from 9e79d10 to f806823 Compare August 21, 2026 20:21

syntactically reviewed Aug 24, 2026

View reviewed changes

ludfjig force-pushed the recover_unrecoverable branch from f806823 to d37d029 Compare October 19, 2026 20:54

ludfjig changed the title ~~Add method for recreating unrecoverable sandbox~~ Add Sandbox::recreate to recover unrecoverable sandboxes Oct 19, 2026

This comment has been minimized.

Sign in to view

ludfjig force-pushed the recover_unrecoverable branch from a3f0ebe to 6d37d09 Compare October 21, 2026 23:18

This comment has been minimized.

Sign in to view

jsturtevant reviewed Oct 29, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/initialized.rs Resolved

jsturtevant previously approved these changes Oct 29, 2026

View reviewed changes

ludfjig dismissed jsturtevant’s stale review via 11f17c2 November 2, 2026 19:24

ludfjig force-pushed the recover_unrecoverable branch 2 times, most recently from 11f17c2 to bae38dc Compare November 2, 2026 19:51

This comment has been minimized.

Sign in to view

syntactically previously approved these changes Nov 4, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs Resolved

ludfjig dismissed syntactically’s stale review via 3fe149e November 9, 2026 17:26

ludfjig force-pushed the recover_unrecoverable branch from bae38dc to 3fe149e Compare November 9, 2026 17:26

This comment has been minimized.

Sign in to view

ludfjig force-pushed the recover_unrecoverable branch from 3fe149e to 3bf937b Compare November 11, 2026 23:52

jsturtevant approved these changes Nov 12, 2026

View reviewed changes

ludfjig merged commit 01df991 into hyperlight-dev:main Nov 12, 2026
62 checks passed

Conversation

ludfjig commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ludfjig commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

syntactically left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

syntactically left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

hyperlight-gh-bot Bot commented Nov 12, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

ludfjig commented Aug 21, 2026 •

edited

Loading