Repository navigation
runtime-adapter: let each connection to a guest listener open or fail - #1034
Merged
Pedro Henrique Penna (ppenna) merged 1 commit intoOct 30, 2026
Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
Pedro Henrique Penna (ppenna) merged 1 commit into
Conversation
reach_guest connected to a listener of the guest that OpenVMM publishes on host loopback with a 1 s timeout. Until OpenVMM publishes the listener, nothing listens at its port, and Windows answers the reset by sending the SYN again 0.5 s, 1.0 s, 1.5 s, and 2.0 s later before it fails the connection, so the attempts timed out instead, and the retry at 1.0 s landed on the timeout. When OpenVMM published the port between 0.5 s and 1.0 s into an attempt, that retry could complete the handshake just as connect_timeout gave up and closed its socket. OpenVMM accepted the connection all the same and relayed it to the guest, whose nc -l took it for its one client and ended with "listener 0", and the host's later attempts found no listener. That failed vm_layered allowed_host_loopback_publishes_the_listeners on Windows/WHP in CI run 40950585891, as #1033 describes. Each attempt now runs until it opens or fails, with TcpStream::connect. On loopback, a connection opens at once where something listens, and fails at once on Linux and after about 2 s on Windows where nothing does, so an attempt stays bounded and never leaves a connection behind. The copy of reach_guest in vm_clone.rs, which serves the listener checks of clones, gets the same change. On a Windows workstation, a replica of reach_guest against a listener that serves one client and appears at a random time failed with the incident's signature in 19 of 1,920 trials, and in none of 1,280 with TcpStream::connect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the VM connection retry changes.
No open issues 🎉
What changed in this PR
Addresses #1033’s Windows connection race in the runtime adapter’s VM scenarios by letting each connection attempt finish before retrying, rather than abandoning a connection the guest may accept.
Changes:
- Replace the one-second
connect_timeoutwithTcpStream::connectin bothreach_guesthelpers. - Document why connection attempts must complete, while preserving the existing listener checks.
| File | Description |
|---|---|
tests/runtime-adapter/tests/vm_layered.rs |
Update layered VM listener connections and document the Windows retry race. |
tests/runtime-adapter/tests/vm_clone.rs |
Apply the same connection change and documentation to clone listener checks. |
Review effort: Max
Pedro Henrique Penna (ppenna)
deleted the
agents/nvx-ci-failure-investigation-674b9465
branch
October 30, 2026 02:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1033.
reach_guest, with which the runtime adapter's VM scenarios reach a listener of the guest that OpenVMM publishes on host loopback, connected with a 1 s timeout. Until OpenVMM publishes the listener, nothing listens at its port, and Windows retries such a connection every 0.5 s for about 2 s, so the retry at 1.0 s lands on the timeout: when OpenVMM published the port between 0.5 s and 1.0 s into an attempt, that retry could open a connection just asconnect_timeoutgave up on it and closed it. OpenVMM relayed that connection to the guest, whosenc -ltook it for its one client, and the host's later attempts found no listener. That failedvm_layered allowed_host_loopback_publishes_the_listenerson Windows/WHP in CI run 40950585891, withlistener 0andleft: false,right: true.Changes
runtime-adapter: let each connection to a guest listener open or failreach_guestconnects withTcpStream::connect(address)rather thanTcpStream::connect_timeout(&address, Duration::from_secs(1)), invm_layered.rsand in the copy thatvm_clone.rshas had since nvx-rs: per-clone network policies on deny-all templates #978. On loopback, a connection opens at once where something listens, and fails at once on Linux and after about 2 s on Windows where nothing does, so an attempt stays bounded and never leaves a connection behind.reach_guestsays why its attempts run to their end.Validation
Mechanism. On a 20-thread Windows 11 Enterprise workstation, without a hypervisor, a replica of
reach_guestran against a listener that appears 0–3 s after the client starts and serves one client, asnc -ldoes, 64 trials at a time:connect_timeoutof 1 s, as ondevTcpStream::connect, as hereEach failure's listener appeared 0.5–1.0 s into one of the client's attempts, where Windows' retry at 1.0 s meets the timeout. #1033 holds the replica, the measurements of Windows' retries, and the CI history.
Gates. On the workstation, the runtime adapter's steps of
check-nvx-rsthat cover this change:fmt; Clippy with warnings denied in the protocol-only and host-only configurations; and, in the host-only configuration, the build, the tests, the documentation with warnings denied, and the 1.89 MSRV check. All pass. Without--include-ignored, the tests skip the VM scenarios, as in CI'scheck-nvx-rs.Not covered. The VM scenarios did not run here: the workstation has no OpenVMM build or runtime adapter guest agent. This PR's
NVX microVM testsjobs run them on WHP, KVM, and MSHV. On Linux, which fails a loopback connection to a closed port at once, the change leaves the attempts as they were.