fix: avoid apple hypercalls and synchronize snapshot instruction caches by andreiltd · Pull Request #2039 · hyperlight-dev/hyperlight

andreiltd · 2026-11-09T16:49:53Z

HVF handles some hvc #0 calls internally based on x0. Guest register values can select these calls, causing the tlb flush trampoline to continue past its hvc and fault.

Copilot

🔵 Needs a closer look

The HVF trampoline and instruction-cache changes require Apple Silicon and Windows runtime validation because this review could inspect code and API contracts but could not exercise those platform-specific paths.

No open issues 🎉

What changed in this PR

This PR addresses HVF initialization faults caused by guest register values selecting Apple hypercalls. It also prepares host-written snapshot memory for guest execution by synchronizing instruction caches.

Changes:

Uses a nonzero HVC identifier for the HVF TLB-flush trampoline and tests Apple hypercall register values.
Synchronizes instruction caches on macOS and Windows when preparing snapshot memory.
Documents both fixes in the changelog.

File	Description
`src/hyperlight_host/src/mem/shared_mem.rs`	Adds instruction-cache synchronization to snapshot memory preparation.
`src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs`	Updates the trampoline HVC identifier and adds a regression test.
`CHANGELOG.md`	Records the HVF initialization and instruction-cache fixes.

_{Review effort: Balanced}

jsturtevant

LGTM, @syntactically for another set of eyes on ARM

HVF handles some hvc #0 calls internally based on x0. Guest register values can select these calls, causing the tlb flush trampoline to continue past its hvc and fault. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

hyperlight-gh-bot · 2026-11-12T20:30:25Z

Benchmark Results

Measured commit: c2a773a9a7da
Baseline commit: 1afa48e03fc1

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`755.19 ns` (➖ 1.01x faster)
`vec_bytes`	`584.16 ns` (➖ 1.01x faster)
		`389.04 µs` (➖ 1.04x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`524.59 ns` (➖ 1.01x faster)
`65536`	`142.86 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.32 ms` (➖ 1.00x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`8.96 ns` (➖ 1.17x slower)	`7.70 ns` (➖ 1.00x slower)	`7.68 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`100.00 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.03 µs` (➖ 1.03x faster)	`7.14 µs` (➖ 1.02x faster)
`65536`	`1.92 µs` (➖ 1.09x faster)	`2.04 µs` (➖ 1.06x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`28.18 µs` (➖ 1.05x slower)
`65536`	`6.17 µs` (➖ 1.05x faster)	`6.24 µs` (➖ 1.04x faster)
`8192`	`1.03 µs` (➖ 1.01x slower)	`1.04 µs` (➖ 1.03x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`689.78 ns` (➖ 1.02x faster)
`vec_bytes`	`535.15 ns` (➖ 1.03x slower)
		`692.08 µs` (➖ 1.04x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`504.20 ns` (➖ 1.00x slower)
`65536`	`174.22 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.36 ms` (➖ 1.07x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`6.88 ns` (➖ 1.01x faster)	`6.88 ns` (➖ 1.00x faster)	`6.87 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`50.33 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.64 µs` (➖ 1.00x slower)	`7.63 µs` (➖ 1.00x slower)
`65536`	`2.17 µs` (➖ 1.01x slower)	`2.17 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`30.76 µs` (➖ 1.03x slower)
`65536`	`7.14 µs` (➖ 1.02x faster)	`7.30 µs` (➖ 1.00x faster)
`8192`	`720.21 ns` (➖ 1.01x slower)	`740.53 ns` (➖ 1.01x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`961.57 ns` (➖ 1.01x slower)
`vec_bytes`	`731.27 ns` (➖ 1.01x slower)
		`270.26 µs` (➖ 1.23x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`730.83 ns` (➖ 1.00x faster)
`65536`	`198.52 ns` (➖ 1.04x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`59.84 ms` (➖ 1.04x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.24 ns` (➖ 1.07x slower)	`9.55 ns` (➖ 1.00x faster)	`9.62 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`105.27 µs` (➖ 1.04x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.64 µs` (➖ 1.05x faster)	`11.41 µs` (➖ 1.02x slower)
`65536`	`2.22 µs` (➖ 1.00x faster)	`2.48 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`36.19 µs` (➖ 1.05x faster)
`65536`	`8.52 µs` (➖ 1.18x faster)	`8.36 µs` (➖ 1.28x faster)
`8192`	`1.31 µs` (➖ 1.07x slower)	`1.39 µs` (➖ 1.05x slower)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`939.87 ns` (➖ 1.06x faster)
`vec_bytes`	`684.84 ns` (➖ 1.03x faster)
		`1.33 ms` (➖ 1.31x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`688.87 ns` (➖ 1.06x faster)
`65536`	`286.61 ns` (➖ 1.04x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`97.56 ms` (➖ 1.07x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`8.82 ns` (➖ 1.05x faster)	`8.83 ns` (➖ 1.05x faster)	`8.82 ns` (➖ 1.04x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`57.45 µs` (➖ 1.12x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`8.02 µs` (➖ 1.02x faster)	`8.17 µs` (➖ 1.00x slower)
`65536`	`2.31 µs` (➖ 1.01x faster)	`2.30 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`41.85 µs` (➖ 1.09x faster)
`65536`	`7.67 µs` (➖ 1.04x faster)	`7.59 µs` (➖ 1.06x faster)
`8192`	`918.63 ns` (➖ 1.04x faster)	`925.76 ns` (➖ 1.03x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.20 µs` (➖ 1.02x faster)
`vec_bytes`	`777.67 ns` (➖ 1.01x faster)
		`2.02 ms` (➖ 1.03x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`784.29 ns` (➖ 1.01x faster)
`65536`	`229.33 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.96 ms` (➖ 1.08x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.12 ns` (➖ 1.01x slower)	`10.24 ns` (➖ 1.03x faster)	`9.97 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`805.13 µs` (➖ 1.03x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.01 µs` (➖ 1.03x faster)	`9.04 µs` (➖ 1.03x faster)
`65536`	`2.33 µs` (➖ 1.01x faster)	`2.31 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`40.16 µs` (➖ 1.18x faster)
`65536`	`8.94 µs` (➖ 1.00x slower)	`8.99 µs` (➖ 1.06x faster)
`8192`	`1.25 µs` (➖ 1.03x faster)	`1.27 µs` (➖ 1.01x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.28 µs` (➖ 1.02x slower)
`vec_bytes`	`788.77 ns` (➖ 1.02x faster)
		`3.04 ms` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`747.16 ns` (➖ 1.09x faster)
`65536`	`213.21 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.72 ms` (➖ 1.00x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.73 ns` (➖ 1.00x faster)	`9.89 ns` (➖ 1.02x faster)	`9.67 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`681.03 µs` (➖ 1.06x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.77 µs` (➖ 1.01x slower)	`7.74 µs` (➖ 1.01x slower)
`65536`	`2.33 µs` (➖ 1.00x slower)	`2.29 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`43.58 µs` (➖ 1.02x slower)
`65536`	`8.38 µs` (➖ 1.01x faster)	`8.92 µs` (➖ 1.06x slower)
`8192`	`1.06 µs` (➖ 1.14x faster)	`1.12 µs` (➖ 1.05x slower)

_{Reported by cargo ci bench-report --candidate run:43288165605 --config-file bench_report.toml.}

ludfjig

lgtm to me as well but i'm not too familiar with this

Copilot AI balanced review requested due to automatic review settings November 9, 2026 16:49

andreiltd requested review from danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 9, 2026 16:49

andreiltd added the C-bug For PRs that fix bugs label Nov 9, 2026

andreiltd mentioned this pull request Nov 9, 2026

feat!: replace shared-memory init data with virtqueue buffers #2030

Open

Copilot started reviewing on behalf of andreiltd November 9, 2026 16:54

View session

Copilot AI reviewed Nov 9, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

jsturtevant reviewed Nov 11, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/mem/shared_mem.rs Resolved

jsturtevant previously approved these changes Nov 11, 2026

View reviewed changes

andreiltd dismissed jsturtevant’s stale review via c2d292e November 12, 2026 16:10

andreiltd force-pushed the fix/hvf-snapshot-icache branch from ec02e9b to c2d292e Compare November 12, 2026 16:10

andreiltd force-pushed the fix/hvf-snapshot-icache branch from c2d292e to c2a773a Compare November 12, 2026 19:14

ludfjig reviewed Nov 12, 2026

View reviewed changes

jsturtevant approved these changes Nov 12, 2026

View reviewed changes

jsturtevant enabled auto-merge (squash) November 12, 2026 21:56

Conversation

andreiltd commented Nov 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This comment has been minimized.

Uh oh!

jsturtevant left a comment

Choose a reason for hiding this comment

Uh oh!

hyperlight-gh-bot Bot commented Nov 12, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants