linux-arm64 nested KVM: guest wedges the vCPU (enter/exit spin) under some workloads #2025

Summary

On nested arm64 KVM (a Linux guest running under Apple's Hypervisor.framework via Tart on Apple-silicon Macs — our self-hosted arm64 CI runners), certain guest workloads wedge the vCPU: the hyperlight VMM process pegs a full CPU core spinning in the KVM vCPU run loop, and the guest never makes progress. The same workloads run fine on macOS arm64 (HVF) and Windows arm64 (WHP); only the linux-arm64 (KVM, nested) backend is affected.

Observed with the Node/V8 http-express demo and the smolagents/mcp suites in hyperlight-unikraft's arm64 nightly; http-flask (CPython) and compute-heavy jobs (verl, bench, clickhouse) are fine.

Evidence (live capture on a wedged runner VM)

With http-express wedged, the single-threaded VMM process:

• is in state R (running), ~100% of one core (CPU jiffies/2s ≈ 200);
• has its kernel stack consistently in kvm_arch_vcpu_ioctl_run (+ kvm_arch_vcpu_ctxsync_fp) — i.e. spinning entering/exiting the guest;
• makes zero network host-calls and never reaches listen() on the server port, so it wedges during guest execution, before/independent of any I/O.

A concurrent http-flask run captured in the same trace completed its accept/recv/send host-calls normally.

Environment

• Host: Apple M4, macOS 26.x; guest: Ubuntu 24.04 Tart VM booted --nested, exposing /dev/kvm.
• hyperlight-host KVM aarch64 backend; guest is a Unikraft unikernel running the workload.

Impact

Fails the affected jobs in hyperlight-unikraft's arm64 nightly (previously as 90-minute job-timeout hangs that also starved co-tenant jobs; a fast-fail harness change has since bounded them).

Next diagnostic

KVM event tracing (ftrace/perf kvm:*) on the runner VM to identify the exit storm. Candidates: KVM_EXIT_ARM_NISV cache-maintenance retries from V8 self-modifying/JIT code, a stage-2 fault loop, or FP/SIMD trapping under nested virt. Happy to run this on the runner.

Repro

Run the http-express demo under hluk on a nested-arm64-KVM runner (Apple-silicon Tart --nested): the VMM pegs a core and the server never answers.