Repository navigation
Remove the legacy heap region from snapshots - #1968
Merged
Merged
Conversation
ludfjig
added
A-perf
ludfjig
changed the title
This comment has been minimized.
This comment has been minimized.
ludfjig
force-pushed
the
ll/4-unbounded-heap
branch
from
October 21, 2026 19:27
1642d27 to
f27c1ef
Compare
This comment has been minimized.
This comment has been minimized.
ludfjig
force-pushed
the
ll/4-unbounded-heap
branch
from
October 21, 2026 21:02
f27c1ef to
8b0f611
Compare
This comment has been minimized.
This comment has been minimized.
ludfjig
marked this pull request as ready for review
October 23, 2026 21:25
ludfjig
requested review from
andreiltd,
danbugs,
dblnz,
devigned,
jprendes,
jsturtevant,
simongdavies,
squillace and
syntactically
as code owners
October 23, 2026 21:25
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
3 open issues
Ordering::Relaxedto publishHEAP_EXECUTABLEis risky because the value is consumed from… · NewOrdering::Relaxedto publishHEAP_EXECUTABLEis risky because the value is consumed from… · Newinit_data_offsethere is misleading: this value is the offset after pushing the… · New
What changed in this PR
Removes the legacy fixed-size guest heap region from snapshot memory and switches the guest to mapping heap pages from scratch on first touch, reducing snapshot size and improving performance.
Changes:
- Removes heap-size configuration and updates tests/fixtures/benchmarks to size by scratch instead.
- Bumps snapshot ABI to 8 and config schema media type to v5, updating goldens and versioning docs.
- Updates guest fault handling to map heap pages from scratch on demand and propagates heap executability via a PEB flag.
| File | Description |
|---|---|
| src/tests/rust_guests/simpleguest/src/main.rs | Updates guest OOM/exhaustion behavior to reflect scratch-backed heap semantics. |
| src/hyperlight_testing/src/lib.rs | Renames benchmark/test size constants from heap sizes to scratch sizes and adjusts types. |
| src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs | Bumps goldens version to v8.0 for the new snapshot ABI/schema. |
| src/hyperlight_host/tests/snapshot_goldens/fixtures.rs | Removes heap sizing from golden snapshot config fixture. |
| src/hyperlight_host/tests/sandbox_host_tests.rs | Updates leak-related tests to constrain scratch rather than heap. |
| src/hyperlight_host/tests/integration_test.rs | Updates integration tests to assert scratch exhaustion aborts with MallocFailed. |
| src/hyperlight_host/src/sandbox/uninitialized.rs | Updates sandbox/snapshot tests to remove heap sizing and validate scratch-only layout behavior. |
| src/hyperlight_host/src/sandbox/snapshot/tripwires.rs | Bumps expected ABI/schema and updates invariant commentary around layout offsets. |
| src/hyperlight_host/src/sandbox/snapshot/file_tests.rs | Updates snapshot file tests for schema v5 and ABI 8; removes heap layout assertions. |
| src/hyperlight_host/src/sandbox/snapshot/file/mod.rs | Updates loader to accept config v5 only and removes heap from persisted layout reconstruction. |
| src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs | Adds config v5 media type and bumps snapshot ABI version to 8. |
| src/hyperlight_host/src/sandbox/snapshot/file/config.rs | Removes heap_size from config schema/layout and updates schema pins/validation. |
| src/hyperlight_host/src/sandbox/initialized.rs | Removes heap override propagation from snapshots and updates tests to size by scratch. |
| src/hyperlight_host/src/sandbox/config.rs | Removes heap size configuration API; clarifies scratch semantics and updates tests. |
| src/hyperlight_host/src/sandbox/builder.rs | Removes heap_size builder APIs; updates docs to show scratch sizing. |
| src/hyperlight_host/src/mem/mgr.rs | Updates memory manager tests to use scratch sizing constants (no heap sizing). |
| src/hyperlight_host/src/mem/layout.rs | Removes heap region from snapshot memory layout; updates offsets/PEB contents accordingly. |
| src/hyperlight_host/src/hypervisor/mod.rs | Adjusts initialise test parameters now that heap region assumptions are gone. |
| src/hyperlight_host/examples/map-file-cow-test/main.rs | Removes heap sizing from example; relies on scratch sizing only. |
| src/hyperlight_host/benches/benchmarks.rs | Re-keys benchmark configurations to scratch sizes and adds heap_first_touch benchmark. |
| src/hyperlight_guest_bin/src/lib.rs | Claims a fixed virtual heap range and reads heap executability from the PEB. |
| src/hyperlight_guest_bin/src/arch/amd64/layout.rs | Adds a static assert to ensure heap range doesn’t overlap processor control. |
| src/hyperlight_guest_bin/src/arch/amd64/exception/handle.rs | Maps heap pages from scratch on first touch; supports executable heap via PEB flag. |
| src/hyperlight_guest_bin/src/arch/aarch64/exception/handle.rs | Same as amd64: scratch-backed heap faults with executable flag support. |
| src/hyperlight_guest/src/layout.rs | Exposes heap virtual range constants and asserts no overlap with transport/stack. |
| src/hyperlight_guest/src/arch/amd64/layout.rs | Defines 512GiB-aligned heap virtual range for amd64. |
| src/hyperlight_guest/src/arch/aarch64/layout.rs | Defines 512GiB-aligned heap virtual range for aarch64. |
| src/hyperlight_common/src/mem.rs | Replaces PEB heap region field with a heap executability flag. |
| fuzz/fuzz_targets/host_call.rs | Removes heap sizing from fuzz target configuration. |
| docs/snapshot-versioning.md | Documents ABI 8/config v5 and notes heap region removal from snapshot memory. |
| docs/snapshot-oci-format.md | Updates config media type reference to v5. |
| docs/hyperlight-execution-details.md | Updates lifecycle docs to describe scratch-backed stack+heap mapping. |
| CHANGELOG.md | Records breaking changes: ABI/schema bump and removal of heap sizing APIs. |
Review effort: Lite
ludfjig
force-pushed
the
ll/4-unbounded-heap
branch
2 times, most recently
from
October 30, 2026 19:14
53bbd23 to
360b772
Compare
This comment has been minimized.
This comment has been minimized.
ludfjig
force-pushed
the
ll/4-unbounded-heap
branch
from
October 30, 2026 22:13
360b772 to
40f3bec
Compare
This comment has been minimized.
This comment has been minimized.
dummyguest halts only when it receives its expected PEB address and seed. Otherwise it reads 0x8000 to force an exit. The test passed a dummy PEB address and seed, so it relied on 0x8000 being mapped, which held only because that address fell inside the guest heap region. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
A not-present fault in the stack range always maps a fresh scratch page, so the page table walk in virt_to_phys is wasted work. Check the range first and walk only for faults outside it. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
ludfjig
force-pushed
the
ll/4-unbounded-heap
branch
from
October 30, 2026 23:16
40f3bec to
a7dc630
Compare
The guest heap is a fixed 512 GiB virtual range. Guest init claims the whole range with talc, and the page fault handler maps each heap page from scratch on first touch, the same way it maps stack pages. Heap memory is bounded by scratch rather than a fixed heap size, and untouched heap pages cost nothing. The host publishes the executable heap policy in the scratch-top metadata, and the PEB drops the heap address and size. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
The guest maps heap pages from scratch on first touch, so the host no longer reserves a zero-filled heap region in snapshot memory. This removes SandboxConfiguration::set_heap_size, DEFAULT_HEAP_SIZE, and SandboxBuilder::heap_size and get_heap_size. Scratch is the only guest memory setting. Snapshots no longer carry up to hundreds of MiB of zero pages, which speeds up sandbox creation and snapshot capture, save, and load. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
Snapshot memory no longer contains a heap region, and the PEB drops the heap address and size. The scratch-top metadata carries the executable heap policy. The config schema drops the heap size. This is a hard break. Loaders reject config v4 and older, and the goldens move to v9.0. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
Tests that sized the heap now size scratch. Allocation succeeds beyond scratch, and the guest aborts with MallocFailed when a write needs a page and scratch is full, so heap exhaustion tests expect that abort. FillHeapAndCauseException is removed because the heap can no longer run out before scratch does. The shared sandbox sizes in hyperlight-testing become scratch sizes. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
Benchmark sandbox sizes set scratch, which is the only guest memory setting. The small, medium, and large variants use 8, 64, and 256 MiB of scratch. heap_first_touch measures writes to heap the guest has not touched since the last restore. Each variant writes half its scratch. The default size is skipped because fixed reservations take most of its scratch. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
ludfjig
force-pushed
the
ll/4-unbounded-heap
branch
from
October 30, 2026 23:33
a7dc630 to
94f50e6
Compare
Benchmark ResultsMeasured commit: kvm / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
kvm / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / amd (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / intel (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
Reported by |
andreiltd
reviewed
Nov 3, 2026
syntactically
approved these changes
Nov 4, 2026
syntactically
left a comment
syntactically
left a comment
Member
There was a problem hiding this comment.
Very glad to see the old heap region go! Two very minor comments/questions.
danbugs
added a commit
that referenced
this pull request
Nov 4, 2026
#1968 removed SandboxBuilder::heap_size (and set_heap_size/get_heap_size), but the scratch_allocated_bytes_counts_written_pages test added by #2019 still called it, so main stopped compiling its tests once both landed — which is why clippy and miri fail on every PR. Drop the .heap_size(8 << 20) line; the test already sizes scratch with .scratch_size(16 << 20). Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Nov 4, 2026
Prepares the v0.19.0 release per `docs/how-to-make-releases.md` (the Cargo.toml version and CHANGELOG steps). The tag and GitHub release are cut after this merges. - Bump the workspace version `0.18.0` → `0.19.0`, and the internal `hyperlight-*` dependency version requirements with it. - Roll the CHANGELOG `[Prerelease]` section into `[v0.19.0]`, add a fresh empty `[Prerelease]`, and update the compare-link footer (`[v0.19.0]` added; `[Prerelease]` now compares from `v0.19.0`). - Sync the root `Cargo.lock` and `src/tests/rust_guests/Cargo.lock` to the new versions (workspace-only; no other dependencies touched). - Drop a stray `.heap_size(8 << 20)` call in the `scratch_allocated_bytes` test: #1968 removed `SandboxBuilder::heap_size`, but #2019's test (which landed separately) still called it, so main stopped compiling its tests once both merged. Folding the one-line fix in here so the release compiles. --------- Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
to unikraft/unikraft
that referenced
this pull request
Nov 5, 2026
Hyperlight 0.19 removes the guest heap region from sandbox memory (hyperlight-dev/hyperlight#1968). The PEB holds only init_data, and the memory the guest allocates from is scratch. This kernel read guest_heap past the end of the PEB, found no heap, and crashed at boot: "Boot stack alloc failed". The heap region served only as the boot stack: with paging and HEAP_BASE, the frame allocator and the kernel heap already come from scratch, the heap as frames mapped at HEAP_BASE, which a snapshot captures. The boot stack, which stays the "init" thread's for good, is needed before the heap is up, and memory taken straight from scratch (hl_scratch_alloc_pages(), used through the scratch window) is not captured by a snapshot and is zeroed by a restore. So the 1 MiB boot stack is a page-aligned array in .bss, which is snapshot memory as the heap region was, and is pre-faulted for CoW as before. A canary in the stack's lowest word crashes the guest if the stack is run down to it. It is checked at four points. The halt that ends boot checks it, so an overflow during boot fails the boot. Each later entry from the host (every guest function call, and the checkpoint before a snapshot) checks it, catching later use of the stack by the init thread (a native main() without LIBUKBOOT_MAINTHREAD that outlives the boot halt) or a stray write, so such a guest fails a snapshot rather than being captured. The end of a call's entry, before its result, checks it for the call's own work, so a call that ran the stack down fails rather than returning. A shutdown, usually run by the init thread, checks it before reporting the exit, so a call in which the stack was run down and the guest then shut down fails rather than ending cleanly. A snapshot of a guest with no call since its last snapshot or restore enters no guest code: it captures memory the boot halt or a checkpoint has already checked. At the boot halt, in a shutdown and at the start and end of an ordinary call's entry the report ("Boot stack canary overwritten (stack overflow?)") goes the console's usual way, to the output the host collects. During a checkpoint the host refuses a host call, the console's HostWriteErr and HostWrite included, and would end the run before the report reached it, so for any entry but an ordinary call the transport is marked lost first (hl_transport_abandon(), which transport_lost() and a failed checkpoint now use too) and the report goes out through DebugPrint. Each check crashes through hyperlight_boot_stack_crash(), which prints the report and aborts directly: UK_CRASH, under LIBUKDEBUG_CRASH_ACTION_REBOOT, ends in uk_pm_sysrestart(), the shutdown, whose check would fail again and crash again, for good. Drop guest_heap from struct hyperlight_peb to match HyperlightPEB, and the FREE region and its bookkeeping with it. Nothing reads the PEB any more, so the pointer kept to it goes too; its address is still checked. Without paging there is nothing left to allocate from, so PLAT_HYPERLIGHT selects LIBUKPAGING, which also gives the kernel heap its LIBUKBOOT_HEAP_BASE once a boot allocator is chosen; memory.c refuses a build without one (LIBUKBOOT_INITALLOC). pagetable64.S goes: it only defined bpt_unmap_mrd, which plat/common reads only in the ukplat_mem_init() it builds without paging. Since it reads nothing from the PEB and registers no free region, this kernel still boots on a host that maps the heap region (hyperlight main before hyperlight-dev/hyperlight#1968), leaving that region unused. Signed-off-by: danbugs <danilochiarlone@gmail.com>
NopeCharon
pushed a commit
to NopeCharon/hyperlight-unikraft
that referenced
this pull request
Nov 5, 2026
Bump the kernel (plat-hyperlight-v2 9f414729 -> 3b7cba88), rebuild the kernels and the native-kernel fixture, and stop sizing the heap: - Hyperlight 0.19 gives the guest no heap region (hyperlight-dev/hyperlight#1968), and the kernel took its 1 MiB boot stack from it: on 0.19 the guest crashed at boot ("Boot stack alloc failed"). The boot stack is in the kernel's .bss now, which a snapshot keeps as it kept the heap, with a canary in its lowest word. The halt that ends boot checks it, so an overflow during boot fails the boot. Each later entry from the host (every guest function call, and the checkpoint before a snapshot) checks it too, catching later use of the stack by the init thread (a native main() without LIBUKBOOT_MAINTHREAD that outlives the boot halt) or a stray write, so such a guest fails a snapshot rather than being captured; a snapshot with no call since the last snapshot or restore enters no guest code, and captures memory the boot halt or a checkpoint has already checked. The end of a call's entry, before its result, checks it for the call's own work, so a call that ran the stack down fails rather than returning. A shutdown checks it before reporting the exit, so a call in which the stack was run down and the guest then shut down fails rather than ending cleanly. The report goes to the output hluk collects, or through DebugPrint for an entry that may be a checkpoint, during which the host refuses a host call; the crash bypasses UK_CRASH, whose reboot action would re-enter the shutdown's check. - The kernel's PEB matches Hyperlight 0.19's, init_data alone. - hluk no longer sizes the heap: HEAP_SIZE and its two set_heap_size calls go, as nothing uses the region now, and the SNAPSHOT_CONTRACT doc in build.rs no longer lists a heap among what a snapshot depends on. The hyperlight main this tree builds against still maps one, at its default size, unused. - plat/hyperlight selects paging, the only way it has to allocate, so the elfloader and native-kernel defconfigs no longer set LIBUKPAGING and LIBUKFALLOC themselves; the resolved .config and the kernels built from it are unchanged by that. It also refuses a build without a boot allocator, which its heap needs, and drops pagetable64.S, whose only symbol nothing reads with paging on. The CHANGELOG notes these for native-kernel builders. The kernel boots on hyperlight 0.19.0 and on the hyperlight main this tree builds against (6c17f7cd). Signed-off-by: danbugs <danilochiarlone@gmail.com>
NopeCharon
pushed a commit
to NopeCharon/hyperlight-unikraft
that referenced
this pull request
Nov 5, 2026
hyperlight-host and hyperlight-common move to the 0.19.0 release, which has scratch resize (hyperlight-dev/hyperlight#1947) and scratch_allocated_bytes (hyperlight-dev/hyperlight#2019), so the [patch.crates-io] git pin goes from the crate, the clickhouse integration and the pptx-gen demo, and the crate can be published to crates.io again. The three lockfiles take 0.19.0 from crates.io; in re-resolving them cargo also moves some crates' windows-sys dependency to versions already in the locks (mostly 0.61.2). 0.19 drops the guest heap region (hyperlight-dev/hyperlight#1968), and with it SandboxConfiguration::set_heap_size, which the previous commit stopped calling. The CHANGELOG's 0.19 entry says what the re-exported hyperlight_host 0.19 changes, that a native kernel built against an earlier plat/hyperlight must be rebuilt, and, folding in the previous commit's note, what plat/hyperlight now requires and checks; the transport arena entry's own rebuild note goes. SNAPSHOT_CONTRACT stays 8: it was bumped for the transport arena after v0.20.1 and has not been released. Dev builds since then carry it, and their snapshots are refused all the same: the kernel change (the previous commit) rolls the key's other half, k<hash>, the embedded kernel's SHA-256, and Hyperlight's own snapshot ABI 9 refuses them besides. Closes hyperlight-dev#233 Signed-off-by: danbugs <danilochiarlone@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove the fixed-size guest heap region from sandbox memory. It held only zeros, and the first write to each page copied it into scratch anyway. The guest now maps each heap page from scratch on first touch, as it does for stack pages. Snapshots no longer carry the zero-filled region, and the heap is bounded only by scratch.
Each first write to a heap page no longer copies a zero page out of the snapshot, which speeds up all guest workloads that allocate.
Please review commit by commit.
Closes #1697
details
talc. The page fault handler maps a fresh scratch page on each heap page's first touch, so untouched pages cost nothing.MallocFailed.executable_heapstays a host feature. The guest sets the heap's execute permission in its own page tables, as the host did before. The hypervisor never enforced it, so a malicious guest can circumvent it. We could consider removing the feature, since it adds little security. See Consider removingexecutable_heaphost cargo feature #2003.Alternatives considered
ZeroInitpage-table entries for the heap. Rejected for performance, and because it needs a fixed heap size again. The marker exists only on 4 KiB leaf entries, so all of the heap's page tables exist up front, 1/512 of the heap size. Every snapshot captures them and every restore copies them into scratch. First touch is also slower, because the fault handler walks the page tables to find the marker instead of checking a range.Breaking changes
SandboxConfiguration::set_heap_size,SandboxConfiguration::DEFAULT_HEAP_SIZE,SandboxBuilder::heap_sizeandSandboxBuilder::get_heap_sizeare removed. Size scratch for the guest's heap use.MallocFailedon the write that needs a page. Allocations no longer return null, and Rust guests no longer panic withmemory allocation of N bytes failed.HyperlightPEB::guest_heapis removed.Benchmarks
Compared to the PR below. MSHV runs Linux 6.6.137.mshv2-5 and 6.6.100.mshv1-2. KVM runs Linux 6.17.
Each size keeps the PR below's scratch and drops its heap of 128 KiB, 8, 64 or 256 MiB. This PR then redefines the sizes as 8, 64 and 256 MiB of scratch.
heap_first_touchis new. It writes half of 8, 64 or 256 MiB of scratch, and the PR below gets a heap as large as scratch.guest_calls/heap_first_touch/mediumguest_calls/heap_first_touch/largeguest_functions_with_large_parameters/guest_call_with_large_parameterssandboxes/create_initialized/largesandboxes/sandbox_from_snapshot/largesnapshots/create/largesnapshot_files/save_snapshot/largesnapshot_files/cold_start_via_snapshot/large6.6.137 populates the old heap region on first touch and 6.6.100 at sandbox creation, so first touch gains more on 6.6.137 and creation gains more on 6.6.100.
Regressions (not really when compared to main before this stack!)
snapshots/create/defaultsnapshot_files/cold_start_via_snapshot_unverified/defaultWHP and aarch64 were not benchmarked.