Expose one CPU feature set to MSHV and WHP guests

CHANGELOG.md

@@ -4,6 +4,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Prerelease] - Unreleased
 
+### Changed
+* **Breaking:** Expose one processor feature set to MSHV and WHP guests. Leave out speculation control, TSX, CET, AMX, and other stateful features Hyperlight guests do not use.
+* Run WHP guests in their own speculation domain.
+
+### Fixed
+* Fix restore on WHP not rewinding the guest TSC on hosts with `TSC_ADJUST` support.
+
 ## [v0.19.0] - 2026-11-04
 
 ### Added

docs/README.md

@@ -45,3 +45,5 @@ This project is composed internally of several components, depicted in the below
 * [Hyperlight Surrogate Development Notes](./hyperlight-surrogate-development-notes.md)
 * [Debugging Hyperlight](./debugging-hyperlight.md)
 * [Signal Handling in Hyperlight](./signal-handlers-development-notes.md)
+* [MSR state across restore](./msr.md)
+* [Guest CPU features](./guest-cpu.md)

docs/guest-cpu.md

@@ -0,0 +1,35 @@
+# Guest CPU features
+
+Hyperlight exposes only the CPU features its guests need. Each extra feature
+adds guest state that restore must reset.
+
+## MSHV and WHP
+
+MSHV and WHP guests see one set of CPU features, limited by the host. Both
+backends pass the list in `partition_features.rs` to Hyper-V at partition
+creation.
+
+The list holds only features of the 21H2 compatibility version. The MSHV
+driver creates every partition at that version. Hyper-V hides newer features
+from such partitions.
+
+The list keeps instruction-set features such as SSE4.2, BMI2, and AES. They
+hold no state. It also keeps AVX through AVX-512. Restore resets their state
+with the rest of the XSAVE area. It leaves out:
+
+| Features | Reason |
+| --- | --- |
+| Speculation control: IBRS, STIBP, IBPB, SSBD, PSFD, BHI_DIS, VIRT_SPEC_CTRL | Guest `SPEC_CTRL` state. A guest is one trust domain. |
+| TSX | `TSX_CTRL` state. |
+| CET | Shadow-stack state and CET MSRs. |
+| `TSC_ADJUST` | A write to `TSC` or `TSC_ADJUST` shifts both registers equally. Restoring both would undo the TSC rewind. |
+| `TSC_DEADLINE`, WAITPKG, MPERF/APERF, RDPRU | Timer, wait, and counter state. RDPRU reads MPERF/APERF. |
+| MPX, XFD, AMX | `BNDCFGS` and `XFD` MSR state. AMX needs XFD. |
+| Supervisor XSAVE state | Used only by CET and PASID. |
+| Hyper-V enlightenments | Guests do not use them. |
+
+## KVM
+
+KVM guests get the CPUID that KVM supports, minus VMX, SVM, x2APIC, and CET.
+KVM's MSR filter denies every MSR the sandbox does not declare. See
+[MSR state across restore](./msr.md).

docs/msr.md

@@ -37,7 +37,7 @@ The backend owns discovery because register mappings and capabilities differ.
 Sorting, deduplication, baseline capture, snapshot validation, and the fallback
 to the destination baseline are shared in `MsrResetState`.
 
-Every reset entry must represent guest-writable retained state the host can read
+Every reset entry must represent guest-writable state the host can read
 and write. On the Hyper-V backends the candidate table is derived from the
 Hyper-V source and must be audited when that source, register mappings, or
 feature exposure changes. VM creation probes each entry with a host read and a
@@ -125,49 +125,27 @@ poisons the sandbox.
 ## Hyper-V backends
 
 MSHV and WHP cannot filter guest MSR access, so restore scrubs a broad set of
-retained MSRs to the destination baseline before writing the saved core and
-declared-MSR values. The scrub set is built from:
+MSRs that hold guest state to the destination baseline before writing the saved
+core and declared-MSR values. The scrub set is built from:
 
-* Retained-state candidates the backend maps and can read, including state
-  reachable only through the VP register API such as active SSP.
+* Candidates the backend maps and can read, including state reachable only
+  through the VP register API such as active SSP.
 * MTRRs required by the virtual CPU's `MTRRCAP`.
 
 The candidate table is not an intercept list. Hyper-V also intercepts read-only,
-command, and host-derived MSRs that retain no guest-controlled value. An entry
+command, and host-derived MSRs that keep no guest-controlled value. An entry
 belongs in the table only when guest execution can leave state that affects
-later execution.
+later execution. The table ignores the host's MSR index list. That list does
+not show which MSRs keep guest state.
+
+Guest CPU features limit which MSRs MSHV and WHP guests can reach. See
+[Guest CPU features](./guest-cpu.md).
 
 A future partition-scrub hypercall will reset all MSRs directly and replace the
 candidate table. WHP has kernel support. MSHV support is in progress. The saved
 set stays the core plus the declared MSRs across that migration, so restore
 behavior does not change.
 
-### MSHV
-
-MSHV enables the processor features supported by the host unless a partition
-feature mask disables them. Its guest-visible MSR surface therefore varies by
-host CPU.
-
-MSHV maps `IA32_XSS` through `MSR_IA32_REGISTER_U_XSS`. It maps `IA32_MPERF`
-and `IA32_APERF` through the per-VP `MCount` and `ACount` registers. TSX,
-WAITPKG, CET, XFD, MPX, and deadline-timer state enter the reset set when the
-host exposes and maps them.
-
-The host's enumerated MSR index list does not identify retained state, so it
-does not define the reset set.
-
-On capable Intel hosts MSHV can expose ENQCMD and PASID. PASID is a supervisor
-XSAVE component, so XSAVE restore owns it.
-
-### WHP
-
-WHP's default feature banks disable speculation control, experimental
-`DEBUGCTL` bits, and performance monitoring. Its supported feature mask omits
-ENQCMD and defines no FRED feature, so WHP does not expose PASID or FRED.
-
-WHP maps supported MSRs to `WHV_REGISTER_NAME` values. The same mapping is used
-for snapshot reads and restore writes.
-
 ### MTRRs
 
 MSHV and WHP read `IA32_MTRRCAP` during VM creation. The reset set includes
@@ -178,21 +156,12 @@ clear.
 VM creation fails if `VCNT` exceeds the supported maximum of 16 or required
 MTRRs cannot be read.
 
-### TSC
-
-Hyper-V stores `TSC` and `TSC_ADJUST` independently. While time runs, it
-preserves `TSC - TSC_ADJUST`: writing `TSC` also changes `TSC_ADJUST`, while
-writing `TSC_ADJUST` changes the internal TSC offset.
-
-Restore writes `TSC` before `TSC_ADJUST`. The two writes remove the guest's
-delta without freezing partition time. KVM also restores `TSC` so all backends
-use the same guest-time semantics.
-
-## Retained-state inventory
+## Guest-state inventory
 
-These MSRs are reset when supported by the selected backend and host.
+These MSRs hold guest-writable state. A backend resets each one the host
+exposes. MSHV and WHP guests lack the features behind many of them.
 
-| MSR (index) | Retained state |
+| MSR (index) | State |
 | --- | --- |
 | SYSENTER CS, ESP, EIP (`0x174`-`0x176`) | System-call entry state. |
 | STAR, LSTAR, CSTAR, SFMASK (`0xC000_0081`-`0xC000_0084`) | System-call target state. |
@@ -243,12 +212,5 @@ CPU. It is a regression tool, not a complete inventory of vendor MSRs.
   MSHV gains kernel support. WHP already has it.
 * Exercise MSHV and WHP on more CPU models. Their reachable MSR surfaces depend
   on host features.
-* Disable optional stateful features the guest does not need, so the guest
-  cannot retain that state. This shrinks the reset set toward the always-present
-  core (SYSENTER, syscall targets, KERNEL_GS_BASE, PAT, DEBUGCTL, TSC, MTRRs),
-  which every host can read. The reset set then becomes fixed, so the discovery
-  probe that drops host-unsupported entries becomes unnecessary, and the audit
-  surface shrinks. Do not add a minimum feature set unless the goal is to bar
-  some host CPUs from running Hyperlight.
 * Extend the inventory when Hyperlight enables new CPU features such as
   perfmon, FRED, or nested virtualization.

src/hyperlight_host/src/hypervisor/regs/x86_64/msrs.rs

@@ -183,7 +183,6 @@ const NON_MTRR_RESETTABLE_MSRS: &[u32] = &[
     // MSHV maps XSS through its U_XSS alias.
     MSR_XSS,
     MSR_TSC,
-    // Hyper-V stores TSC_ADJUST independently from TSC.
     MSR_TSC_ADJUST,
     // Host probing omits TSC_AUX when RDTSCP is unavailable.
     MSR_TSC_AUX,

src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs

@@ -38,6 +38,10 @@ pub(crate) mod whp;
 #[cfg(feature = "hw-interrupts")]
 pub(crate) mod x86_64;
 
+/// Processor features shared by MSHV and WHP partitions
+#[cfg(all(target_arch = "x86_64", any(mshv3, target_os = "windows")))]
+pub(crate) mod partition_features;
+
 static AVAILABLE_HYPERVISOR: OnceLock<Option<HypervisorType>> = OnceLock::new();
 
 /// Returns which type of hypervisor is available, if any

src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs

@@ -13,8 +13,9 @@ use mshv_bindings::LapicState;
 #[cfg(gdb)]
 use mshv_bindings::{DebugRegisters, hv_message_type_HVMSG_X64_EXCEPTION_INTERCEPT};
 use mshv_bindings::{
-    FloatingPointUnit, HV_X64_REGISTER_CLASS_IP, SpecialRegisters, StandardRegisters, XSave, Xcrs,
-    hv_message_type, hv_message_type_HVMSG_GPA_INTERCEPT, hv_message_type_HVMSG_UNMAPPED_GPA,
+    FloatingPointUnit, HV_X64_REGISTER_CLASS_IP, MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES,
+    SpecialRegisters, StandardRegisters, XSave, Xcrs, hv_message_type,
+    hv_message_type_HVMSG_GPA_INTERCEPT, hv_message_type_HVMSG_UNMAPPED_GPA,
     hv_message_type_HVMSG_X64_HALT, hv_message_type_HVMSG_X64_IO_PORT_INTERCEPT,
     hv_partition_property_code_HV_PARTITION_PROPERTY_SYNTHETIC_PROC_FEATURES,
     hv_partition_synthetic_processor_features, hv_register_assoc,
@@ -44,6 +45,7 @@ use crate::hypervisor::regs::{
 };
 #[cfg(test)]
 use crate::hypervisor::virtual_machine::XSAVE_BUFFER_SIZE;
+use crate::hypervisor::virtual_machine::partition_features::{PROCESSOR_FEATURES, XSAVE_FEATURES};
 #[cfg(feature = "hw-interrupts")]
 use crate::hypervisor::virtual_machine::x86_64::hw_interrupts::TimerThread;
 use crate::hypervisor::virtual_machine::{
@@ -171,14 +173,20 @@ impl MshvVm {
     pub(crate) fn new() -> std::result::Result<Self, CreateVmError> {
         let mshv = MSHV.as_ref().map_err(|e| e.clone())?;
 
+        // The kernel takes masks of disabled features.
         #[allow(unused_mut)]
-        let mut pr: mshv_create_partition_v2 = Default::default();
-        // The default has no partition flags. Hardware interrupts add only the
-        // LAPIC required by request_virtual_interrupt.
+        let mut pr = mshv_create_partition_v2 {
+            pt_flags: 1u64 << MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES,
+            pt_num_cpu_fbanks: PROCESSOR_FEATURES.len() as u16,
+            pt_cpu_fbanks: PROCESSOR_FEATURES.map(|bank| !bank),
+            pt_disabled_xsave: !XSAVE_FEATURES,
+            ..Default::default()
+        };
+        // Hardware interrupts add the LAPIC required by request_virtual_interrupt.
         #[cfg(feature = "hw-interrupts")]
         {
             use mshv_bindings::MSHV_PT_BIT_LAPIC;
-            pr.pt_flags = 1u64 << MSHV_PT_BIT_LAPIC;
+            pr.pt_flags |= 1u64 << MSHV_PT_BIT_LAPIC;
         }
         // It's important to use create_vm_with_args() (not create_vm()),
         // because create_vm() sets up a SynIC partition by default.

src/hyperlight_host/src/hypervisor/virtual_machine/partition_features.rs

@@ -0,0 +1,112 @@
+// SPDX-License-Identifier: Apache-2.0
+// Copyright 2025 The Hyperlight Authors.
+
+//! Processor features that MSHV and WHP guests see. See `docs/guest-cpu.md`.
+//!
+//! Bit positions follow `hv_partition_processor_features` and
+//! `hv_partition_processor_xsave_features` in Linux `include/hyperv/hvhdk.h`. They match
+//! `WHV_PROCESSOR_FEATURES_BANKS` and `WHV_PROCESSOR_XSAVE_FEATURES`. Every bit is part of the
+//! 21H2 compatibility version and of the mask WHP accepts.
+
+const fn bits(list: &[u32]) -> u64 {
+    let mut mask = 0;
+    let mut i = 0;
+    while i < list.len() {
+        mask |= 1 << list[i];
+        i += 1;
+    }
+    mask
+}
+
+const BANK0: &[u32] = &[
+    0,  // SSE3
+    1,  // LAHF/SAHF
+    2,  // SSSE3
+    3,  // SSE4.1
+    4,  // SSE4.2
+    5,  // SSE4A
+    6,  // XOP
+    7,  // POPCNT
+    8,  // CMPXCHG16B
+    9,  // ALTMOVCR8
+    10, // LZCNT
+    11, // MisalignSSE
+    12, // MMX extensions
+    13, // 3DNow!
+    14, // Extended 3DNow!
+    15, // 1 GB pages
+    16, // AES
+    17, // PCLMULQDQ
+    18, // PCID
+    19, // FMA4
+    20, // F16C
+    21, // RDRAND
+    22, // RDFSBASE/WRFSBASE
+    23, // SMEP
+    24, // Enhanced fast strings
+    25, // BMI1
+    26, // BMI2
+    29, // MOVBE
+    30, // NPIEP1
+    31, // Deprecated x87 FPU CS/DS
+    32, // RDSEED
+    33, // ADX
+    34, // PREFETCHW
+    35, // SMAP
+    38, // RDTSCP
+    39, // CLFLUSHOPT
+    43, // INVPCID
+    47, // Unrestricted guest
+    49, // Fast short REP MOV
+    51, // RDCL_NO
+    54, // SSB_NO
+    55, // RSBA_NO
+    57, // RDPID
+    58, // UMIP
+    59, // MDS_NO
+    60, // MD_CLEAR
+    61, // TAA_NO
+];
+
+const BANK1: &[u32] = &[
+    1,  // Invariant TSC
+    5,  // MBEC
+    10, // VMX exception inject
+    28, // FB_CLEAR
+    29, // BTC_NO
+    48, // RFDS_NO
+    49, // RFDS_CLEAR
+];
+
+const XSAVE: &[u32] = &[
+    0,  // XSAVE
+    1,  // XSAVEOPT
+    2,  // AVX
+    3,  // AVX2
+    4,  // FMA
+    6,  // AVX512F
+    7,  // AVX512DQ
+    8,  // AVX512CD
+    9,  // AVX512BW
+    10, // AVX512VL
+    11, // Compacted XSAVE
+    14, // AVX512_BITALG
+    15, // AVX512_IFMA
+    16, // AVX512_VBMI
+    17, // AVX512_VBMI2
+    18, // AVX512_VNNI
+    19, // GFNI
+    20, // VAES
+    21, // AVX512_VPOPCNTDQ
+    22, // VPCLMULQDQ
+    23, // AVX512_BF16
+    24, // AVX512_VP2INTERSECT
+    25, // AVX512_FP16
+    30, // AVX_VNNI
+];
+
+/// Enabled processor feature banks 0 and 1.
+pub(crate) const PROCESSOR_FEATURES: [u64; 2] = [bits(BANK0), bits(BANK1)];
+
+/// Enabled processor XSAVE features.
+pub(crate) const XSAVE_FEATURES: u64 = bits(XSAVE);