Skip to content

runtime-adapter: let each connection to a guest listener open or fail - #1034

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/nvx-ci-failure-investigation-674b9465
Oct 30, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/nvx-ci-failure-investigation-674b9465

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Fixes #1033.

reach_guest, with which the runtime adapter's VM scenarios reach a listener of the guest that OpenVMM publishes on host loopback, connected with a 1 s timeout. Until OpenVMM publishes the listener, nothing listens at its port, and Windows retries such a connection every 0.5 s for about 2 s, so the retry at 1.0 s lands on the timeout: when OpenVMM published the port between 0.5 s and 1.0 s into an attempt, that retry could open a connection just as connect_timeout gave up on it and closed it. OpenVMM relayed that connection to the guest, whose nc -l took it for its one client, and the host's later attempts found no listener. That failed vm_layered allowed_host_loopback_publishes_the_listeners on Windows/WHP in CI run 40950585891, with listener 0 and left: false, right: true.

Changes

  1. runtime-adapter: let each connection to a guest listener open or fail
    • reach_guest connects with TcpStream::connect(address) rather than TcpStream::connect_timeout(&address, Duration::from_secs(1)), in vm_layered.rs and in the copy that vm_clone.rs has had since nvx-rs: per-clone network policies on deny-all templates #978. On loopback, a connection opens at once where something listens, and fails at once on Linux and after about 2 s on Windows where nothing does, so an attempt stays bounded and never leaves a connection behind.
    • The documentation of each reach_guest says why its attempts run to their end.
    • What the scenarios check is unchanged: the host must exchange a line each way with a published listener, and never reach an unpublished one, which it keeps trying for the listener's whole time.

Validation

Mechanism. On a 20-thread Windows 11 Enterprise workstation, without a hypervisor, a replica of reach_guest ran against a listener that appears 0–3 s after the client starts and serves one client, as nc -l does, 64 trials at a time:

Connection Trials Failures with the incident's signature
connect_timeout of 1 s, as on dev 1,920 19 (1.0%)
TcpStream::connect, as here 1,280 0

Each failure's listener appeared 0.5–1.0 s into one of the client's attempts, where Windows' retry at 1.0 s meets the timeout. #1033 holds the replica, the measurements of Windows' retries, and the CI history.

Gates. On the workstation, the runtime adapter's steps of check-nvx-rs that cover this change: fmt; Clippy with warnings denied in the protocol-only and host-only configurations; and, in the host-only configuration, the build, the tests, the documentation with warnings denied, and the 1.89 MSRV check. All pass. Without --include-ignored, the tests skip the VM scenarios, as in CI's check-nvx-rs.

Not covered. The VM scenarios did not run here: the workstation has no OpenVMM build or runtime adapter guest agent. This PR's NVX microVM tests jobs run them on WHP, KVM, and MSHV. On Linux, which fails a loopback connection to a closed port at once, the change leaves the attempts as they were.

reach_guest connected to a listener of the guest that OpenVMM
publishes on host loopback with a 1 s timeout. Until OpenVMM publishes
the listener, nothing listens at its port, and Windows answers the
reset by sending the SYN again 0.5 s, 1.0 s, 1.5 s, and 2.0 s later
before it fails the connection, so the attempts timed out instead, and
the retry at 1.0 s landed on the timeout. When OpenVMM published the
port between 0.5 s and 1.0 s into an attempt, that retry could
complete the handshake just as connect_timeout gave up and closed its
socket. OpenVMM accepted the connection all the same and relayed it to
the guest, whose nc -l took it for its one client and ended with
"listener 0", and the host's later attempts found no listener. That
failed vm_layered allowed_host_loopback_publishes_the_listeners on
Windows/WHP in CI run 40950585891, as #1033 describes.

Each attempt now runs until it opens or fails, with TcpStream::connect.
On loopback, a connection opens at once where something listens, and
fails at once on Linux and after about 2 s on Windows where nothing
does, so an attempt stays bounded and never leaves a connection
behind. The copy of reach_guest in vm_clone.rs, which serves the
listener checks of clones, gets the same change.

On a Windows workstation, a replica of reach_guest against a listener
that serves one client and appears at a random time failed with the
incident's signature in 19 of 1,920 trials, and in none of 1,280 with
TcpStream::connect.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings October 30, 2026 00:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the VM connection retry changes.

No open issues 🎉

What changed in this PR

Addresses #1033’s Windows connection race in the runtime adapter’s VM scenarios by letting each connection attempt finish before retrying, rather than abandoning a connection the guest may accept.

Changes:

  • Replace the one-second connect_timeout with TcpStream::connect in both reach_guest helpers.
  • Document why connection attempts must complete, while preserving the existing listener checks.
File Description
tests/​runtime-adapter/​tests/​vm_layered.rs Update layered VM listener connections and document the Windows retry race.
tests/​runtime-adapter/​tests/​vm_clone.rs Apply the same connection change and documentation to clone listener checks.

Review effort: Max

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants