ci: OpenVMM's KVP IC test boots Windows Server 2022, whose COM activation race can fail GetIpInfo with 0x80080005 (flaky OpenVMM vmm-tests / Windows / WHP) #1035

Summary

The dev push run for c938ab2 (#978) failed in OpenVMM vmm-tests / Windows / WHP / Shard 1 of 2 because OpenVMM's multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2022_x64_kvp_ic failed with KVP error: 80080005. The test boots Windows Server 2022 Datacenter Core with a NIC and exchanges KVP messages with it. The guest answered its last request, the NIC's IP information (GetIpInfo), with 0x80080005 (CO_E_SERVER_EXEC_FAILURE, "Server execution failed") 120 s after the test began to wait for the KVP IC, which is COM's activation timeout.

#978 did not cause this: it changes only nvx-rs/, tests/runtime-adapter/, and documentation, not the openvmm gitlink, the test, or NVX's selection of the VMM tests. Since 2026-10-27, the test passed in 216 WHP shard jobs and failed only in this one. Upstream OpenVMM sees the same intermittent failure in its own CI and traced it to a COM activation race in Windows Server 2022 that can lose WMI's server-registration notification; microsoft/openvmm#4803 moves the test to Windows Server 2025, which has the fix. This is a flaky test caused by the guest image, not an NVX or OpenVMM defect.

Incident

• Workflow: CI run 40950662563, attempt 1, push to dev.
• Head: c938ab20da4709a881cf1eff8c94b4083641e106, the merge of nvx-rs: per-clone network policies on deny-all templates #978. Push runs on dev queue rather than cancel each other, so this run waited from 20:37 to 00:23 UTC and was not superseded.
• OpenVMM: c32c57d129a36fc8c82308d6659f8d7ab5955474, the head of feature/microvm/dev.
• Failed job: OpenVMM vmm-tests / Windows / WHP / Shard 1 of 2 on azure-windows-3, step Run OpenVMM tests on Windows, which runs python scripts\nvx.py test-openvmm --backend whp --shard "1/2". The shard's other 14 tests passed, as did shard 2 and the KVM and MSHV legs.
• Downstream: Required status check failed with openvmm-vmm-tests-windows: expected success, got failure, and Persist performance baseline and Publish development release were skipped.

Decisive output:

[petri] 2026-10-30T00:26:07.538110Z  INFO petri::vm::openvmm::runtime: Waiting for KVP IC
[petri] 2026-10-30T00:28:07.753721Z ERROR petri::tracing: test failed error=KVP error: 80080005
…
    FAIL [ 140.820s] ( 4/15) vmm_tests::tests multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2022_x64_kvp_ic
     Summary [ 228.508s] 15 tests run: 14 passed, 1 failed, 424 skipped

The job's diagnostics artifact, openvmm-vmm-tests-Windows-whp-shard-1-40950662563-1, holds the test's logs, screenshots, and the VMM's inspection at the failure (failure_inspect_vmm.log).

History

The WHP partition has put the test in shard 1 since it was created in d5343ae (2026-10-23). I checked every OpenVMM vmm-tests / Windows job of the last 300 CI runs, from 2026-10-27 03:08 UTC to this run, reruns included. Of the 223 Shard 1 of 2 jobs, 216 passed, 6 were cancelled, and this one failed. No job of shard 2 failed.

Upstream sees the same failure: microsoft/openvmm#4564 investigated it in run 36779609946, where GetIpInfo "returned 0x80080005 after COM's ~120-second activation deadline", and microsoft/openvmm#4803 cites run 40792200712.

Root cause

Permalinks point to OpenVMM c32c57d, which NVX pins.

1. The test. kvp_ic boots windows_datacenter_core_2022_x64 with a NIC, waits for the KVP IC, sets a key in the external pool and enumerates it back, and then asks the guest for the NIC's IP information (ic.rs#L61-L69). It is OpenVMM's only VMM test of the KVP IC.

2. The error is the guest's status. The host's KVP IC turns a response whose status is not SUCCESS into a RequestError (kvp.rs#L377-L383), whose message is KVP error: {status:x} (kvp.rs#L680-L682). 0x80080005 is CO_E_SERVER_EXEC_FAILURE, which COM returns when the server that it activates does not register within the activation timeout of 120 s.

3. GetIpInfo failed. The test failed 120.2 s after it began to wait for the KVP IC. At the failure, the KVP channel was idle (state: ready), and its rings had carried 15456 bytes (0x3c60) to the guest and 5448 (0x1548) back, exactly the test's exchange with full replies to the Set and the first Enumerate and replies without a body to the rest. A VMBus packet is a 16-byte descriptor, the payload padded to 8 bytes, and an 8-byte trailer; an IC payload is an 8-byte pipe header, the 20-byte IC header, and the message, 2580 bytes for Set and Enumerate (KvpMessage) and 7428 for GetIpInfo (KvpMessage2).

   Message	To the guest	To the host
   Version negotiation, 2 framework and 3 KVP versions offered	80	72
   Set test_key	2632	2632
   Enumerate index 0	2632	2632
   Enumerate index 1, NO_MORE_ITEMS as the test expects	2632	56
   GetIpInfo, 0x80080005	7480	56
   Total	15456	5448

4. The guest image. Upstream traced the status to Windows Server 2022: "this race can lose WMI's server-registration notification, causing GET_IP_INFO to fail after 120 seconds with 0x80080005" (vmm_tests: Fix kvp test flakiness by updating windows version openvmm#4803). Windows Server 2025 has the fix, and the test passes on it upstream, in 44.3 s on x64-windows-intel and 36.3 s on x64-windows-amd.

5. Not NVX. NVX runs the test unchanged on WHP from the exact names of OPENVMM_WHP_TESTS (ci.py#L708-L710). Neither ic.rs nor the host's kvp.rs has changed since August 2026, and no commit of feature/microvm/dev touches them.

Impact

• c938ab2 has a red required check, and neither a performance baseline nor a development release.
• The next dev push run, 40950709799 for 6fb8b24 (nvx-rs: aggregate exports for clones #992), contains c938ab2. It skipped the OpenVMM VMM tests, passed, persisted the performance baseline, and published a development release, so dev lost no coverage beyond c938ab2.
• On a pull request, the failure turns Required status check red until someone reruns the shard.

Suggested fix

Run the test on Windows Server 2025, as microsoft/openvmm#4803 does upstream:

• Port vmm_tests: Fix kvp test flakiness by updating windows version openvmm#4803 to feature/microvm/dev and pin it in NVX.
• The test's name holds its image, so it becomes multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic. Rename it in OPENVMM_WHP_TESTS and replan openvmm_test_shards.json, with an estimate of its duration until CI's reports measure it: upstream, it takes 1.55 to 1.83 times as long as on Server 2022.
• The KVM and MSHV filters select every test whose name contains openvmm and leave out Windows guests only by windows_datacenter_core_2022_x64 (ci.py#L685, ci.py#L700), so the renamed test would start to run on them. Upstream, it fails there at once, since those jobs do not stage the Server 2025 image (failed to resolve artifacts: … GEN2_WINDOWS_DATA_CENTER_CORE2025_X64). Leave out every Windows Server Datacenter Core guest instead.
• Neither retry the test nor leave it out on WHP: a retry would hide a guest that sometimes fails, and leaving it out would drop the only test of the KVP IC.

Related work

• vmm_tests: Fix kvp test flakiness by updating windows version openvmm#4803 (open, approved) moves the test to Windows Server 2025.
• Capture WMI wait chains when Windows KVP IP lookup stalls openvmm#4564 (closed) collected diagnostics for the same failure upstream.