Expose how much scratch a sandbox has allocated by danbugs · Pull Request #2019 · hyperlight-dev/hyperlight

danbugs · 2026-11-03T21:21:00Z

Adds Sandbox::scratch_allocated_bytes(), which returns how far the guest's scratch bump allocator has moved since the sandbox was last restored or snapshotted.

Every page a guest writes after a restore is taken from scratch through that allocator, whether it's a copy-on-write copy of a snapshot page, a fresh page, or a page table. So this is how much memory the sandbox has dirtied. The host already reads the allocator on restore. This makes it public, so a host such as fiberd can charge each sandbox for what it wrote, the same way on KVM, mshv, WHP and HVF.

SandboxMemoryManager::used_scratch_end_gpa() reads the allocator, held to the transport arena since the guest controls it. install_captured_snapshot now uses it too.
Sandbox::scratch_allocated_bytes() returns the bytes between the first free scratch page and that end. It is 0 after a restore or snapshot and capped at the scratch the guest can allocate.
A unit test covers growth with writes, determinism across restores, and the reset at restore and at snapshot.
CHANGELOG entry under Added.

Copilot

🟢 Approval recommended

No active review issues are reported for the new guest scratch allocation accounting API.

No open issues 🎉

What changed in this PR

Exposes Sandbox::scratch_allocated_bytes() so hosts can track guest scratch allocation consistently across hypervisors.

Changes:

Adds a public allocation counter with documented limits for guest-controlled values.
Centralizes the bounded allocator read and reuses it during snapshot installation.
Adds tests for allocation growth, deterministic counts across restores, and resets after restore and snapshot.

File	Description
`src/hyperlight_host/src/sandbox/initialized.rs`	Adds the public API, usage documentation, and allocation-counter tests.
`src/hyperlight_host/src/mem/mgr.rs`	Reads and bounds the scratch allocator position and calculates allocated bytes.
`CHANGELOG.md`	Documents the allocation counter and its host accounting use case.

_{Review effort: Balanced}

hyperlight-gh-bot · 2026-11-03T22:34:01Z

Benchmark Results

Measured commit: e76e510c1ee5
Baseline commit: 35dd23664883

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`374.22 µs` (➖ 1.00x slower)
`byte_chunks`		`774.56 ns` (➖ 1.03x faster)
`vec_bytes`		`583.29 ns` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`142.48 ns` (➖ 1.01x faster)
`262144`	`524.61 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.16 ms` (➖ 1.09x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.68 ns` (➖ 1.00x faster)	`7.68 ns` (➖ 1.00x faster)	`7.70 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`101.67 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.93 µs` (➖ 1.07x faster)	`2.03 µs` (➖ 1.06x slower)
`262144`	`7.06 µs` (➖ 1.03x faster)	`7.79 µs` (➖ 1.07x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.04 µs` (➖ 1.02x faster)	`1.04 µs` (➖ 1.01x slower)
`65536`	`6.17 µs` (➖ 1.03x faster)	`6.19 µs` (➖ 1.04x faster)
`262144`	`27.09 µs` (➖ 1.00x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`670.61 µs` (➖ 1.01x faster)
`byte_chunks`		`664.41 ns` (➖ 1.03x faster)
`vec_bytes`		`515.96 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`153.70 ns` (➖ 1.01x faster)
`262144`	`494.32 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.81 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`6.76 ns` (➖ 1.00x faster)	`6.82 ns` (➖ 1.01x slower)	`6.77 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`50.61 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.13 µs` (➖ 1.01x slower)	`2.13 µs` (➖ 1.00x faster)
`262144`	`7.54 µs` (➖ 1.01x slower)	`7.50 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`712.22 ns` (➖ 1.01x slower)	`707.22 ns` (➖ 1.03x faster)
`65536`	`7.00 µs` (➖ 1.01x faster)	`6.98 µs` (➖ 1.01x faster)
`262144`	`29.69 µs` (➖ 1.00x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`302.82 µs` (➖ 1.02x faster)
`byte_chunks`		`970.51 ns` (➖ 1.02x slower)
`vec_bytes`		`709.76 ns` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`193.72 ns` (➖ 1.04x faster)
`262144`	`765.96 ns` (➖ 1.05x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`64.36 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.53 ns` (➖ 1.05x faster)	`9.61 ns` (➖ 1.01x faster)	`9.61 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`107.53 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.27 µs` (➖ 1.00x slower)	`2.45 µs` (➖ 1.01x slower)
`262144`	`9.64 µs` (➖ 1.02x faster)	`10.37 µs` (➖ 1.06x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.23 µs` (➖ 1.04x faster)	`1.22 µs` (➖ 1.05x faster)
`65536`	`8.45 µs` (➖ 1.17x faster)	`8.53 µs` (➖ 1.13x faster)
`262144`	`36.20 µs` (➖ 1.06x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`671.97 µs` (➖ 1.01x faster)
`byte_chunks`		`952.31 ns` (➖ 1.06x faster)
`vec_bytes`		`658.39 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`257.48 ns` (➖ 1.03x faster)
`262144`	`670.20 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`75.44 ms` (➖ 1.08x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.80 ns` (➖ 1.00x slower)	`8.73 ns` (➖ 1.01x faster)	`8.75 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`56.97 µs` (➖ 1.04x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.30 µs` (➖ 1.00x faster)	`2.31 µs` (➖ 1.01x faster)
`262144`	`7.94 µs` (➖ 1.02x faster)	`8.00 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`900.54 ns` (➖ 1.02x faster)	`895.48 ns` (➖ 1.02x faster)
`65536`	`7.49 µs` (➖ 1.01x faster)	`7.51 µs` (➖ 1.00x faster)
`262144`	`39.39 µs` (➖ 1.01x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`1.90 ms` (➖ 1.01x slower)
`byte_chunks`		`1.16 µs` (➖ 1.02x slower)
`vec_bytes`		`754.80 ns` (➖ 1.10x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`238.77 ns` (➖ 1.04x slower)
`262144`	`1.03 µs` (➖ 1.30x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`10.08 ms` (➖ 1.00x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`11.06 ns` (➖ 1.04x slower)	`11.09 ns` (➖ 1.05x slower)	`11.40 ns` (➖ 1.08x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`595.55 µs` (➖ 1.03x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.49 µs` (➖ 1.09x faster)	`2.53 µs` (➖ 1.01x faster)
`262144`	`8.56 µs` (➖ 1.11x faster)	`8.56 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.40 µs` (➖ 1.01x faster)	`1.35 µs` (➖ 1.01x slower)
`65536`	`8.69 µs` (➖ 1.06x faster)	`8.63 µs` (➖ 1.02x faster)
`262144`	`38.04 µs` (➖ 1.19x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.95 ms` (➖ 1.03x faster)
`byte_chunks`		`1.18 µs` (➖ 1.03x faster)
`vec_bytes`		`760.55 ns` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`213.93 ns` (➖ 1.01x slower)
`262144`	`743.13 ns` (➖ 1.04x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.51 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.04 ns` (➖ 1.00x faster)	`9.89 ns` (➖ 1.01x slower)	`11.10 ns` (➖ 1.15x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`745.41 µs` (➖ 1.12x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.48 µs` (➖ 1.08x slower)	`2.34 µs` (➖ 1.05x slower)
`262144`	`7.75 µs` (➖ 1.02x slower)	`7.70 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.07 µs` (➖ 1.02x slower)	`1.07 µs` (➖ 1.09x faster)
`65536`	`8.88 µs` (➖ 1.06x slower)	`8.24 µs` (➖ 1.01x faster)
`262144`	`44.66 µs` (➖ 1.21x faster)

_{Reported by cargo ci bench-report --candidate run:41721001092 --config-file bench_report.toml.}

ludfjig

Are there any alternatives we can consider where we can trust the value that comes out of here? Maybe related to dirty-bit tracking done by the hypervisor (I understand that it would be a bigger change though). But overall looks good to me!

Signed-off-by: danbugs <danilochiarlone@gmail.com>

#1968 removed SandboxBuilder::heap_size (and set_heap_size/get_heap_size), but the scratch_allocated_bytes_counts_written_pages test added by #2019 still called it, so main stopped compiling its tests once both landed — which is why clippy and miri fail on every PR. Drop the .heap_size(8 << 20) line; the test already sizes scratch with .scratch_size(16 << 20). Signed-off-by: danbugs <danilochiarlone@gmail.com>

Prepares the v0.19.0 release per `docs/how-to-make-releases.md` (the Cargo.toml version and CHANGELOG steps). The tag and GitHub release are cut after this merges. - Bump the workspace version `0.18.0` → `0.19.0`, and the internal `hyperlight-*` dependency version requirements with it. - Roll the CHANGELOG `[Prerelease]` section into `[v0.19.0]`, add a fresh empty `[Prerelease]`, and update the compare-link footer (`[v0.19.0]` added; `[Prerelease]` now compares from `v0.19.0`). - Sync the root `Cargo.lock` and `src/tests/rust_guests/Cargo.lock` to the new versions (workspace-only; no other dependencies touched). - Drop a stray `.heap_size(8 << 20)` call in the `scratch_allocated_bytes` test: #1968 removed `SandboxBuilder::heap_size`, but #2019's test (which landed separately) still called it, so main stopped compiling its tests once both merged. Folding the one-line fix in here so the release compiles. --------- Signed-off-by: danbugs <danilochiarlone@gmail.com>

hyperlight-host and hyperlight-common move to the 0.19.0 release, which has scratch resize (hyperlight-dev/hyperlight#1947) and scratch_allocated_bytes (hyperlight-dev/hyperlight#2019), so the [patch.crates-io] git pin goes from the crate, the clickhouse integration and the pptx-gen demo, and the crate can be published to crates.io again. The three lockfiles take 0.19.0 from crates.io; in re-resolving them cargo also moves some crates' windows-sys dependency to versions already in the locks (mostly 0.61.2). 0.19 drops the guest heap region (hyperlight-dev/hyperlight#1968), and with it SandboxConfiguration::set_heap_size, which the previous commit stopped calling. The CHANGELOG's 0.19 entry says what the re-exported hyperlight_host 0.19 changes, that a native kernel built against an earlier plat/hyperlight must be rebuilt, and, folding in the previous commit's note, what plat/hyperlight now requires and checks; the transport arena entry's own rebuild note goes. SNAPSHOT_CONTRACT stays 8: it was bumped for the transport arena after v0.20.1 and has not been released. Dev builds since then carry it, and their snapshots are refused all the same: the kernel change (the previous commit) rolls the key's other half, k<hash>, the embedded kernel's SHA-256, and Hyperlight's own snapshot ABI 9 refuses them besides. Closes hyperlight-dev#233 Signed-off-by: danbugs <danilochiarlone@gmail.com>

syntactically · 2026-11-09T14:01:57Z

so a host such as fiberd can charge each sandbox for what it wrote, the same way on KVM, mshv, WHP and HVF.

@danbugs If you're suggesting billing based on the allocator watermark to others, I will caution that it probably isn't a good idea, since it isn't necessarily trustworthy---it would be pretty easy to get undercharged. That said, it's difficult to efficiently calculate actually-used memory efficiently on all platforms.

Maybe related to dirty-bit tracking done by the hypervisor

@ludfjig We try hard to eliminate the expensive exit-on-first-write that's needed to allow the HV to track things. The tradeoff is that the host can't trust the tracking information, indeed. On host platform configurations where the physical memory of the scratch region is lazily populated, you could get some information at coarser granularity out of that, but it isn't (/can't be efficiently) the case everywhere.

danbugs requested review from andreiltd, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 3, 2026 21:21

Copilot AI balanced review requested due to automatic review settings November 3, 2026 21:21

danbugs added the C-enhancement For PRs adding features, improving functionality, docs, tests, etc. label Nov 3, 2026

Copilot started reviewing on behalf of danbugs November 3, 2026 21:24

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

danbugs enabled auto-merge (squash) November 3, 2026 21:27

jsturtevant reviewed Nov 3, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/initialized.rs Resolved

ludfjig previously approved these changes Nov 4, 2026

View reviewed changes

Comment thread CHANGELOG.md Resolved

Comment thread src/hyperlight_host/src/mem/mgr.rs Resolved

Comment thread src/hyperlight_host/src/sandbox/initialized.rs Resolved

Comment thread src/hyperlight_host/src/sandbox/initialized.rs Resolved

danbugs disabled auto-merge November 4, 2026 00:03

danbugs dismissed ludfjig’s stale review via c1e0cf6 November 4, 2026 00:16

ludfjig approved these changes Nov 4, 2026

View reviewed changes

danbugs enabled auto-merge (squash) November 4, 2026 00:25

danbugs added 2 commits November 4, 2026 00:41

Read the scratch allocator in one place

1aac749

Signed-off-by: danbugs <danilochiarlone@gmail.com>

Expose how much scratch a sandbox has allocated

e833eb4

Signed-off-by: danbugs <danilochiarlone@gmail.com>

danbugs force-pushed the scratch-allocated-bytes branch from c1e0cf6 to e833eb4 Compare November 4, 2026 00:42

danbugs merged commit 6c17f7c into main Nov 4, 2026
65 checks passed

danbugs deleted the scratch-allocated-bytes branch November 4, 2026 01:31

danbugs mentioned this pull request Nov 4, 2026

chore: prepare release v0.19.0 #2024

Merged

Conversation

danbugs commented Nov 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

hyperlight-gh-bot Bot commented Nov 3, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

syntactically commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

syntactically commented Nov 9, 2026 •

edited

Loading