feat(aarch64): add GDB debugging for Linux MSHV by cshung · Pull Request #2052 · hyperlight-dev/hyperlight

cshung · 2026-11-11T20:15:20Z

ARM64 Linux MSHV GDB

MSHV guests support register and memory access, two hardware breakpoints,
BRK software breakpoints, and hardware single-step.
Guest exception vectors forward debug stops through HVC.
Debugger register access uses the saved exception context and active stack pointer.
Replacement-VM reset preserves debugger configuration.

The hypercall parser validates message type and payload length.
Native ARM64 MSHV GDB tests run in debug CI.
The default-driver build retains non-debug KVM use and rejects KVM debugging.
KVM debugger support is independent in #2016.

Commit layers

main
`-- D: MSHV backend, #1626
    `-- E: cherry-pick of shared GDB support A, #2050
        `-- F: MSHV-specific GDB support, this PR

Each layer is one signed commit. A and E have identical zero-context patch IDs.
WHP and KVM debugger support have independent branches.

MSHV GDB-only comparison.

The graph and update rules are in
docs/gdb-commit-layers.md.
The full PR diff includes unmerged prerequisites #1626 and #2050.

Validation

Native Azure ARM64 Linux/MSHV validation before this rebase:

Formatting, standard Clippy, MSHV-only GDB Clippy and host build pass.
The full just test recipe passes with 1,076 successful test executions.
All 26 debugger, backend and recovery tests pass with default and MSHV-only
GDB features.
All seven real GDB client tests pass, including bootstrap stack preservation,
software and hardware breakpoints, single-step, code writes and snapshots.
Debug and release Rust and C guest builds pass.
Native regression failures demonstrate rejected-resume handling, unbacked MMIO,
listener setup, bootstrap stack corruption, stale exception-context selection
and the extra instruction after an MMIO step.
MSHV backend code and stress limits are unchanged.
Windows formatting, Clippy, build and the full test recipe pass after the
rebase. Linux and ARM64 cross-Clippy pass on all six layers.
The shared ARM64 test constructor passes the configuration required by
upstream Sandbox::recreate.
Shared support passes Windows x64 Clippy, WHP ARM64 cross-Clippy, Linux
just clippyw, and Linux x64 and ARM64 KVM Clippy.

Native ARM64 runtime tests were not repeated for this rebase.

The dedicated MSHV validation VM is deallocated.

Copilot

🟡 Changes recommended

Unresolved critical debugging defects can strand the GDB listener, expose or modify the wrong ARM64 register context, and corrupt live bootstrap stack data.

6 open issues

What changed in this PR

This PR adds ARM64 Linux/MSHV guest debugging to Hyperlight. Its full scope includes the MSHV backend and shared debugger prerequisites from #1626 and #2050.

Changes:

Implements MSHV register access, breakpoints, single-step, exception forwarding, and debugger state preservation across VM reset.
Shares debugger handling across architectures, including page-sized memory access and snapshot recovery.
Adds native ARM64 MSHV CI, bounded test concurrency, debugger tests, and documentation.

File	Description
`typos.toml`	Allows an upstream MSHV identifier spelling.
`src/hyperlight_host/tests/sandbox_host_tests.rs`	Bounds ARM64 MSHV test concurrency.
`src/hyperlight_host/tests/integration_test.rs`	Reduces concurrent MSHV partitions in stress tests.
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Initializes guest debug vectors during setup.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Exposes backing ranges for debugger cache flushing.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Accepts Arm CPU implementers in Linux tests.
`src/hyperlight_host/src/sandbox/initialized.rs`	Guards snapshots against breakpoints and tests recovery.
`src/hyperlight_host/src/mem/mgr.rs`	Exposes host memory ranges to ARM64 debugging.
`src/hyperlight_host/src/mem/memory_region.rs`	Decodes ARM64 MSHV memory access flags.
`src/hyperlight_host/src/mem/exe.rs`	Carries debug-vector metadata.
`src/hyperlight_host/src/mem/elf.rs`	Locates and relocates guest debug vectors.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/x86_64.rs`	Adapts the debugger translation interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs`	Adapts the debugger translation interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/mshv/aarch64.rs`	Implements the ARM64 MSHV backend and debugger.
`src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs`	Adds ARM64 debug exit and setup error support.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Adapts the debugger translation interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs`	Explicitly rejects unsupported KVM debugging.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Uses shared debugger handling and breakpoint storage.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Integrates ARM64 debug stops and cancellation handling.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/debug.rs`	Provides shared requests, memory access, and breakpoint handling.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Connects MSHV, debug vectors, and initial stops.
`src/hyperlight_host/src/hypervisor/gdb/target.rs`	Supports ARM64 register packets and target operations.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Adds page-table translation and instruction-cache flushing.
`src/hyperlight_host/src/hypervisor/gdb/event_loop.rs`	Handles ARM64 initial-stop notifications.
`src/hyperlight_host/src/hypervisor/gdb/arch/x86_64.rs`	Isolates x86 breakpoint and stop classification.
`src/hyperlight_host/src/hypervisor/gdb/arch/aarch64.rs`	Defines ARM64 BRK encoding and alignment checks.
`src/hyperlight_host/src/hypervisor/gdb/arch.rs`	Selects architecture-specific debugger helpers.
`src/hyperlight_host/examples/guest-debugging/main.rs`	Adds ARM64 real-client debugger tests.
`src/hyperlight_host/build.rs`	Enables debug-build GDB support for ARM64 Linux/MSHV.
`src/hyperlight_guest_bin/src/arch/aarch64/exception/entry.rs`	Forwards synchronous exceptions through HVC.
`src/hyperlight_common/src/arch/aarch64/exn.rs`	Defines the shared debug HVC immediate.
`Justfile`	Supports nextest selection and broader debugger test filters.
`docs/how-to-debug-a-hyperlight-guest.md`	Documents MSHV debugging and snapshot constraints.
`docs/gdb-commit-layers.md`	Documents shared and backend-specific commit dependencies.
`CHANGELOG.md`	Records backend, debugging, and recovery support.
`.github/workflows/ValidatePullRequest.yml`	Adds ARM64 MSHV debug and release jobs.
`.github/workflows/dep_build_test.yml`	Configures native MSHV runners and debugger validation.
`.config/nextest.toml`	Limits concurrent partition-heavy tests.

_{Review effort: Balanced}

MSHV provides partition creation, memory mappings, MMIO dispatch, batched register access and replacement-VM reset on ARM64. Native CI uses tagged hierarchical-virtualization runners and bounded scheduling for the partition limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Architecture adapters, software breakpoints, page-sized memory access and recovery handling are shared by backend layers. Guest debug vectors and common tests support ARM64. Backend layers control platform enablement. The commit-layer document defines dependencies and shared-update validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

MSHV forwards guest debug exceptions through intercepted HVCs. Register access uses the saved exception context and active stack pointer. Two hardware breakpoints and single-step survive replacement-VM reset. Native ARM64 GDB tests run in debug CI. Non-debug KVM use is available with default features. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

hyperlight-gh-bot · 2026-11-12T20:32:46Z

Benchmark Results

Measured commit: 19159c6cb618
Baseline commit: 01df99173865

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`759.80 ns` (➖ 1.00x faster)
`vec_bytes`	`588.87 ns` (➖ 1.01x faster)
		`381.28 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`634.54 ns` (➖ 1.21x slower)
`65536`	`143.88 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.27 ms` (➖ 1.01x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`7.75 ns` (➖ 1.01x slower)	`7.78 ns` (➖ 1.01x slower)	`8.13 ns` (➖ 1.06x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`99.85 µs` (➖ 1.02x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`6.99 µs` (➖ 1.03x faster)	`7.24 µs` (➖ 1.03x slower)
`65536`	`1.94 µs` (➖ 1.07x faster)	`2.06 µs` (➖ 1.08x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`28.20 µs` (➖ 1.05x slower)
`65536`	`6.16 µs` (➖ 1.01x faster)	`6.11 µs` (➖ 1.04x faster)
`8192`	`1.03 µs` (➖ 1.02x slower)	`1.01 µs` (➖ 1.02x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`787.86 ns` (➖ 1.01x slower)
`vec_bytes`	`571.89 ns` (➖ 1.04x faster)
		`709.76 µs` (➖ 1.04x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`558.46 ns` (➖ 1.03x faster)
`65536`	`193.82 ns` (➖ 1.04x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.25 ms` (➖ 1.00x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`7.64 ns` (➖ 1.01x faster)	`7.63 ns` (➖ 1.01x faster)	`7.65 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`52.87 µs` (➖ 1.03x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`8.40 µs` (➖ 1.03x faster)	`8.40 µs` (➖ 1.03x faster)
`65536`	`2.36 µs` (➖ 1.02x faster)	`2.36 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`32.55 µs` (➖ 1.01x faster)
`65536`	`7.89 µs` (➖ 1.04x faster)	`7.87 µs` (➖ 1.02x faster)
`8192`	`791.32 ns` (➖ 1.03x faster)	`829.03 ns` (➖ 1.01x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`930.97 ns` (➖ 1.02x faster)
`vec_bytes`	`895.75 ns` (➖ 1.25x slower)
		`314.76 µs` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`669.29 ns` (➖ 1.01x faster)
`65536`	`183.94 ns` (➖ 1.03x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`69.00 ms` (➖ 1.00x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.87 ns` (➖ 1.00x faster)	`9.87 ns` (➖ 1.00x faster)	`9.88 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`112.81 µs` (➖ 1.05x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`11.98 µs` (➖ 1.30x slower)	`9.31 µs` (➖ 1.00x slower)
`65536`	`2.45 µs` (➖ 1.02x faster)	`2.64 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`34.87 µs` (➖ 1.01x faster)
`65536`	`8.01 µs` (➖ 1.02x faster)	`8.03 µs` (➖ 1.01x faster)
`8192`	`1.27 µs` (➖ 1.01x slower)	`1.27 µs` (➖ 1.01x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`936.90 ns` (➖ 1.01x faster)
`vec_bytes`	`665.94 ns` (➖ 1.02x faster)
		`679.12 µs` (➖ 1.14x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`685.01 ns` (➖ 1.02x slower)
`65536`	`260.42 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`83.76 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`8.73 ns` (➖ 1.00x faster)	`8.69 ns` (➖ 1.00x faster)	`8.76 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`55.94 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.98 µs` (➖ 1.00x faster)	`8.07 µs` (➖ 1.01x slower)
`65536`	`2.32 µs` (➖ 1.01x slower)	`2.29 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`38.59 µs` (➖ 1.03x faster)
`65536`	`7.49 µs` (➖ 1.00x faster)	`7.51 µs` (➖ 1.01x faster)
`8192`	`903.17 ns` (➖ 1.01x faster)	`920.51 ns` (➖ 1.02x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.18 µs` (➖ 1.01x faster)
`vec_bytes`	`784.11 ns` (➖ 1.03x faster)
		`1.96 ms` (➖ 1.05x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`808.17 ns` (➖ 1.05x faster)
`65536`	`234.05 ns` (➖ 1.02x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.14 ms` (➖ 1.05x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.02 ns` (➖ 1.00x slower)	`9.99 ns` (➖ 1.02x faster)	`10.16 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`815.61 µs` (➖ 1.04x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.34 µs` (➖ 1.01x slower)	`9.26 µs` (➖ 1.02x faster)
`65536`	`2.34 µs` (➖ 1.00x slower)	`2.33 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`39.28 µs` (➖ 1.01x faster)
`65536`	`9.02 µs` (➖ 1.03x slower)	`9.43 µs` (➖ 1.07x slower)
`8192`	`1.31 µs` (➖ 1.02x faster)	`1.33 µs` (➖ 1.04x slower)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.27 µs` (➖ 1.04x slower)
`vec_bytes`	`839.05 ns` (➖ 1.07x slower)
		`3.05 ms` (➖ 1.06x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`738.56 ns` (➖ 1.06x faster)
`65536`	`217.79 ns` (➖ 1.04x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.99 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.00 ns` (➖ 1.00x faster)	`10.07 ns` (➖ 1.02x faster)	`9.62 ns` (➖ 1.12x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`675.36 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.70 µs` (➖ 1.01x slower)	`7.46 µs` (➖ 1.03x faster)
`65536`	`2.32 µs` (➖ 1.01x slower)	`2.27 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`42.42 µs` (➖ 1.07x slower)
`65536`	`8.32 µs` (➖ 1.02x faster)	`8.72 µs` (➖ 1.02x slower)
`8192`	`1.06 µs` (➖ 1.12x faster)	`1.13 µs` (➖ 1.05x slower)

_{Reported by cargo ci bench-report --candidate run:43277991763 --config-file bench_report.toml.}

cshung requested review from danbugs and devigned as code owners November 11, 2026 20:15

cshung added the C-enhancement For PRs adding features, improving functionality, docs, tests, etc. label Nov 11, 2026

cshung requested review from andreiltd, dblnz, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 11, 2026 20:15

Copilot AI balanced review requested due to automatic review settings November 11, 2026 20:15

cshung mentioned this pull request Nov 11, 2026

Implement MSHV aarch64 backend #1626

Open

cshung added the S-needs-review PR is ready for (re-)review label Nov 11, 2026

Copilot started reviewing on behalf of cshung November 11, 2026 20:19

View session

Copilot AI reviewed Nov 11, 2026

View reviewed changes

cshung force-pushed the cshung/mshv-aarch64-gdb branch 2 times, most recently from d38fe01 to 46beba7 Compare November 12, 2026 00:12

This comment has been minimized.

Sign in to view

cshung and others added 3 commits November 12, 2026 10:01

cshung force-pushed the cshung/mshv-aarch64-gdb branch from 46beba7 to 19159c6 Compare November 12, 2026 18:15

Conversation

cshung commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ARM64 Linux MSHV GDB

Commit layers

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

hyperlight-gh-bot Bot commented Nov 12, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cshung commented Nov 11, 2026 •

edited

Loading