feat!: replace shared-memory init data with virtqueue buffers by andreiltd · Pull Request #2030 · hyperlight-dev/hyperlight

andreiltd · 2026-11-06T09:03:05Z

Remove the peb backed init-data region and APIs. Use retained virtqueue ByteChunks for startup data. After removing the init section PEB no longer contains any data so it can be removed as well.

Copilot

🔵 Needs a closer look

Guest entry ABI and persisted snapshot layout changes across x86_64 and aarch64 require validation of rebuilt guests and initialization, restore, fork, and OCI round trips on supported hypervisors before approval.

No open issues 🎉

What changed in this PR

This PR routes guest startup data through existing virtqueue buffers and removes the PEB-backed initialization region. This is a breaking change requiring rebuilt guests and regenerated snapshots.

Changes:

Remove initialization-data APIs and simplify host and guest startup arguments.
Update snapshots to ABI 10 and config schema v6, with legacy formats rejected.
Add retained-buffer snapshot tests and document startup-data migration.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Remove the legacy user-memory test function.
`src/tests/rust_guests/simpleguest/src/bin/simpleguest-initial-data.rs`	Add a guest that retains startup buffers.
`src/tests/rust_guests/dummyguest/src/main.rs`	Validate the simplified entry arguments.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Set snapshot goldens to v10.0.
`src/hyperlight_host/tests/integration_test.rs`	Remove assertions for deleted region variants.
`src/hyperlight_host/src/sandbox/uninitialized.rs`	Remove blob wrappers and accept `GuestBinary` directly.
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Initialize guests without a PEB address.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pin the updated snapshot contract.
`src/hyperlight_host/src/sandbox/snapshot/mod.rs`	Build snapshots without PEB or init-data regions.
`src/hyperlight_host/src/sandbox/snapshot/file/mod.rs`	Load schema v6 and reject older schemas.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Define config v6 and ABI 10.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Remove init-data fields from serialized configuration.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Test retained startup buffers and format compatibility.
`src/hyperlight_host/src/sandbox/initialized.rs`	Update initialization and snapshot tests.
`src/hyperlight_host/src/sandbox/builder.rs`	Remove the initialization-data builder option.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Update the memory-layout fixture.
`src/hyperlight_host/src/mem/mgr.rs`	Update snapshot and layout test constructors.
`src/hyperlight_host/src/mem/memory_region.rs`	Remove PEB and init-data region definitions.
`src/hyperlight_host/src/mem/layout.rs`	Simplify initial memory to guest code and update layout tests.
`src/hyperlight_host/src/mem/exe.rs`	Update tests for `Snapshot::from_binary`.
`src/hyperlight_host/src/hypervisor/mod.rs`	Update the guest initialization test.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Adjust x86-64 startup registers.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Adjust AArch64 startup registers.
`src/hyperlight_guest/src/lib.rs`	Remove the guest-handle module export.
`src/hyperlight_guest/src/guest_handle/handle.rs`	Delete the PEB-backed guest handle.
`src/hyperlight_guest_bin/src/lib.rs`	Remove PEB initialization and simplify random seeding.
`src/hyperlight_guest_bin/src/host_comm.rs`	Remove the legacy user-memory reader.
`src/hyperlight_guest_bin/src/arch/amd64/init.rs`	Update entry arguments and stack-pivot register.
`src/hyperlight_guest_bin/src/arch/aarch64/mod.rs`	Update entry arguments and stack-pivot register.
`src/hyperlight_common/src/mem.rs`	Delete shared PEB structures.
`src/hyperlight_common/src/lib.rs`	Remove the deleted memory module export.
`docs/snapshot-versioning.md`	Document ABI 10 and config v6 compatibility.
`docs/snapshot-oci-format.md`	Reference config schema v6.
`docs/how-to-build-a-hyperlight-guest-binary.md`	Explain startup-buffer retention and pool sizing.
`CHANGELOG.md`	Record breaking APIs and migration requirements.

_{Review effort: Balanced}

ludfjig

LGTM, with 1 nit: build-rust-guests-non-pie runs cargo build without --bin, so it also builds this binary. Only simpleguest is copied to non_pie. Adding --bin simpleguest to that recipe avoids an extra build in each guest CI job.

Remove the peb backed init-data region and APIs. Use retained virtqueue ByteChunks for startup data. After removing the init section PEB no longer contains any data so it can be removed as well. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

syntactically

Very glad to see this finally gone!

syntactically · 2026-11-09T14:07:35Z

 macro still happen automatically. Your `hyperlight_main` only needs to do
 whatever extra setup the macros do not cover.

+### Startup data


I think this section is not very clear right now: it reads to me like it is mentioning 3 different approaches to init data (call host function in hyperlight_main returning vec, call guest function returning vec , call guest function with vec)) but doesn't really contrast them.

andreiltd · 2026-11-09T16:52:07Z

Hey @syntactically, I moved the instruction cache fixes here: #2039

hyperlight-gh-bot · 2026-11-09T17:38:14Z

Benchmark Results

Measured commit: 9b9389c7e5bb
Baseline commit: 6c8cf87bad1f

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`788.41 ns` (➖ 1.01x slower)
`vec_bytes`	`591.88 ns` (➖ 1.01x slower)
		`373.84 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`557.95 ns` (➖ 1.06x slower)
`65536`	`155.35 ns` (➖ 1.08x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.07 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`7.67 ns` (➖ 1.00x faster)	`8.23 ns` (➖ 1.07x slower)	`7.68 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`99.29 µs` (➖ 1.03x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.21 µs` (➖ 1.21x faster)	`7.36 µs` (➖ 1.01x slower)
`65536`	`1.93 µs` (➖ 1.10x faster)	`2.04 µs` (➖ 1.06x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`28.33 µs` (➖ 1.03x slower)
`65536`	`6.29 µs` (➖ 1.01x faster)	`6.16 µs` (➖ 1.05x faster)
`8192`	`1.04 µs` (➖ 1.02x faster)	`1.10 µs` (➖ 1.06x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`678.98 ns` (➖ 1.01x slower)
`vec_bytes`	`516.77 ns` (➖ 1.01x slower)
		`668.36 µs` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`495.38 ns` (➖ 1.00x slower)
`65536`	`148.28 ns` (➖ 1.05x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.29 ms` (➖ 1.03x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`6.77 ns` (➖ 1.00x slower)	`6.77 ns` (➖ 1.00x slower)	`6.77 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`48.34 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.52 µs` (➖ 1.00x faster)	`7.47 µs` (➖ 1.00x faster)
`65536`	`2.12 µs` (➖ 1.01x slower)	`2.09 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`28.93 µs` (➖ 1.03x faster)
`65536`	`7.00 µs` (➖ 1.03x faster)	`6.99 µs` (➖ 1.01x faster)
`8192`	`727.12 ns` (➖ 1.03x slower)	`713.88 ns` (➖ 1.01x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`944.88 ns` (➖ 1.01x faster)
`vec_bytes`	`708.07 ns` (➖ 1.03x faster)
		`266.60 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`734.05 ns` (➖ 1.00x slower)
`65536`	`194.08 ns` (➖ 1.08x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`60.20 ms` (➖ 1.06x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.51 ns` (➖ 1.00x faster)	`9.55 ns` (➖ 1.01x faster)	`9.63 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`105.15 µs` (➖ 1.00x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`10.01 µs` (➖ 1.06x slower)	`10.96 µs` (➖ 1.00x faster)
`65536`	`2.27 µs` (➖ 1.00x slower)	`2.39 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`35.95 µs` (➖ 1.06x faster)
`65536`	`8.30 µs` (➖ 1.21x faster)	`8.19 µs` (➖ 1.17x faster)
`8192`	`1.27 µs` (➖ 1.01x faster)	`1.25 µs` (➖ 1.05x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`934.42 ns` (➖ 1.01x slower)
`vec_bytes`	`646.26 ns` (➖ 1.01x faster)
		`696.06 µs` (➖ 1.13x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`674.00 ns` (➖ 1.02x slower)
`65536`	`277.17 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`58.16 ms` (➖ 1.34x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`8.70 ns` (➖ 1.16x faster)	`8.68 ns` (➖ 1.00x slower)	`8.74 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`55.18 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.95 µs` (➖ 1.00x faster)	`7.96 µs` (➖ 1.04x faster)
`65536`	`2.34 µs` (➖ 1.03x slower)	`2.28 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`41.55 µs` (➖ 1.05x slower)
`65536`	`7.48 µs` (➖ 1.00x slower)	`7.47 µs` (➖ 1.01x faster)
`8192`	`899.64 ns` (➖ 1.01x faster)	`906.77 ns` (➖ 1.01x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.24 µs` (➖ 1.06x slower)
`vec_bytes`	`769.40 ns` (➖ 1.00x slower)
		`2.15 ms` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`790.15 ns` (➖ 1.00x faster)
`65536`	`230.54 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.01 ms` (➖ 1.05x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.75 ns` (➖ 1.01x slower)	`10.77 ns` (➖ 1.02x slower)	`10.78 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`845.55 µs` (➖ 1.13x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`8.60 µs` (➖ 1.05x faster)	`8.65 µs` (➖ 1.01x slower)
`65536`	`2.52 µs` (➖ 1.09x faster)	`2.55 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`39.38 µs` (➖ 1.19x faster)
`65536`	`8.62 µs` (➖ 1.13x faster)	`9.32 µs` (➖ 1.04x slower)
`8192`	`1.37 µs` (➖ 1.06x faster)	`1.38 µs` (➖ 1.01x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.25 µs` (➖ 1.05x slower)
`vec_bytes`	`790.99 ns` (➖ 1.04x slower)
		`2.95 ms` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`734.31 ns` (➖ 1.02x faster)
`65536`	`209.58 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.97 ms` (➖ 1.01x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.61 ns` (➖ 1.03x faster)	`10.18 ns` (➖ 1.03x slower)	`9.56 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`663.75 µs` (➖ 1.00x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.44 µs` (➖ 1.03x faster)	`7.77 µs` (➖ 1.04x slower)
`65536`	`2.42 µs` (➖ 1.04x slower)	`2.30 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`42.42 µs` (➖ 1.16x faster)
`65536`	`8.30 µs` (➖ 1.02x faster)	`8.21 µs` (➖ 1.03x faster)
`8192`	`1.07 µs` (➖ 1.01x slower)	`1.21 µs` (➖ 1.13x slower)

_{Reported by cargo ci bench-report --candidate run:42660062741 --config-file bench_report.toml.}

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd · 2026-11-12T19:25:23Z

Just to be clear, #2039 needs to land first. For some reason the changes in this PR triggered that bug.

andreiltd requested review from danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 6, 2026 09:03

Copilot AI balanced review requested due to automatic review settings November 6, 2026 09:03

Copilot started reviewing on behalf of andreiltd November 6, 2026 09:03

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

andreiltd force-pushed the remove-init-data branch from 11f0a16 to e9a13c4 Compare November 6, 2026 09:07

This comment has been minimized.

Sign in to view

andreiltd added C-refactor For PRs that restructure or remove code without adding new functionality. S-needs-goldens Regenerate snapshot golden fixtures labels Nov 6, 2026

andreiltd mentioned this pull request Nov 6, 2026

Move the executable heap flag into the PEB and make it a runtime setting #2026

Open

ludfjig previously approved these changes Nov 6, 2026

View reviewed changes

andreiltd dismissed ludfjig’s stale review via 05b0780 November 6, 2026 19:53

andreiltd force-pushed the remove-init-data branch from e9a13c4 to 05b0780 Compare November 6, 2026 19:53

This comment has been minimized.

Sign in to view

Merge branch 'main' into remove-init-data

9b9389c

This comment has been minimized.

Sign in to view

andreiltd force-pushed the remove-init-data branch from 0244efc to e29c4ba Compare November 9, 2026 13:36

This comment has been minimized.

Sign in to view

andreiltd force-pushed the remove-init-data branch from e29c4ba to f90efe7 Compare November 9, 2026 14:27

syntactically reviewed Nov 9, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

andreiltd force-pushed the remove-init-data branch from f90efe7 to 9b9389c Compare November 9, 2026 16:58

fix: improve init data docs

3cfcae6

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

jsturtevant approved these changes Nov 11, 2026

View reviewed changes

Conversation

andreiltd commented Nov 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This comment has been minimized.

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

syntactically left a comment

Choose a reason for hiding this comment

Uh oh!

syntactically Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

andreiltd commented Nov 9, 2026

Uh oh!

hyperlight-gh-bot Bot commented Nov 9, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

andreiltd commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

andreiltd commented Nov 12, 2026 •

edited

Loading