Repository navigation
nvx-rs, runtime-adapter: run a clone's restore sequence to its sandbox's start - #950
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The readiness-event read accepts a valid prefix with trailing data, allowing clone activation without verifying that the endpoint closes with no extra bytes under the same deadline.
1 open issue
What changed in this PR
This PR completes the host-side clone restore sequence for #844, building on #949’s launch support to activate a clone, start its sandbox, and commit the launch.
Changes:
- Adds restore lifecycle types, deadlines, activation checks, and bounded retries for lost sessions.
- Reads restore-ready events with peer-process checks on Linux and Windows.
- Adds runtime-adapter framing, fake-VMM and session tests, and lifecycle documentation.
| File | Description |
|---|---|
| tests/runtime-adapter/tests/host_session.rs | Tests clone activation framing, retries, and failures. |
| tests/runtime-adapter/src/lib.rs | Documents the host’s clone order. |
| tests/runtime-adapter/src/host/order_error.rs | Documents activation-related order errors. |
| tests/runtime-adapter/src/host.rs | Adds activation requests and CloneOrder. |
| tests/runtime-adapter/README.md | Documents clone framing and test coverage. |
| nvx-rs/src/host/template/mod.rs | Exports and documents restore APIs. |
| nvx-rs/src/host/template/clone/tests/restore.rs | Tests restore sequencing, deadlines, and refusals. |
| nvx-rs/src/host/template/clone/tests/launch.rs | Extends fake-VMM behavior and crash-recovery tests. |
| nvx-rs/src/host/template/clone/tests.rs | Registers restore tests. |
| nvx-rs/src/host/template/clone/restore.rs | Implements activation, start, and commit transitions. |
| nvx-rs/src/host/template/clone/launch.rs | Records launch timing and activation-record location. |
| nvx-rs/src/host/template/clone.rs | Exports restore lifecycle types. |
| nvx-rs/src/host/sys/socket.rs | Adds bounded socket readability polling. |
| nvx-rs/src/host/sys/mod.rs | Exposes the readability helper internally. |
| nvx-rs/src/host/supervisor/mod.rs | Documents restore lifecycle and recovery behavior. |
| nvx-rs/src/host/platform/windows.rs | Adds overlapped restore-ready reads and peer checks. |
| nvx-rs/src/host/platform/other.rs | Adds an unsupported-platform receive stub. |
| nvx-rs/src/host/platform/linux.rs | Adds nonblocking restore-ready reads and peer checks. |
| nvx-rs/README.md | Documents the complete host restore sequence. |
Review effort: Balanced
aa44693 to
49adcfe
Compare
281d8d0 to
2c8d550
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The coupled restore, authentication, recovery, and Windows overlapped-I/O changes need platform review or integrated KVM/MSHV/WHP validation because current tests rely on fake VMMs and transports.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
49adcfe to
42cf6b4
Compare
2c8d550 to
7ff6e1f
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-platform restore lifecycle and Windows overlapped-pipe handling have only fake-layer coverage, leaving real KVM/MSHV/WHP integration unverified until PR 5 and requiring human assessment.
No open issues 🎉
Review effort: Balanced
42cf6b4 to
caa6a0b
Compare
7ff6e1f to
cc3cefc
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The restore handshake combines Windows overlapped I/O with crash-sensitive VM lifecycle ordering, but coverage uses fake VMM/control paths and real KVM, MSHV, and WHP validation is deferred to PR 5.
No open issues 🎉
Review effort: Balanced
caa6a0b to
8fe2446
Compare
cc3cefc to
40ee8e4
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Fake-VMM validation leaves real restore, reconnect, commit/recovery, and Windows pipe-handshake behavior unverified across KVM, MSHV, and WHP, requiring maintainer confirmation and platform tests deferred to PR 5.
No open issues 🎉
Review effort: Balanced
8fe2446 to
34638df
Compare
40ee8e4 to
a4893fd
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-platform restore sequencing and Windows pipe cancellation affect VM lifecycle safety, but coverage relies on fake VMMs and transports while real KVM/MSHV/WHP validation is deferred, requiring platform-maintainer review.
No open issues 🎉
Review effort: Balanced
34638df to
492a35e
Compare
a4893fd to
933dc80
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Human review is needed because cross-platform restore ordering and Windows overlapped-pipe handling have only fake-VMM/control coverage, while real clone-restore and exact-head KVM/MSHV/WHP validation remain deferred.
No open issues 🎉
Review effort: Balanced
933dc80 to
a53a0fa
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The restore handoff and Windows pipe cancellation rely on fake-based tests, while real clone restore/Start validation across KVM, MSHV, and WHP remains deferred and needs human confirmation.
No open issues 🎉
Review effort: Balanced
8c12d92 to
b48e295
Compare
a53a0fa to
d6f9b68
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The restore lifecycle spans authentication, Windows overlapped I/O, and crash recovery, while fake-only validation leaves KVM/MSHV/WHP integration deferred and platform contracts needing human review.
No open issues 🎉
Review effort: Balanced
A clone that ClonePlan::launch started now runs its restore sequence through the adapter's framing of its control session, a new CloneControl trait: - Restoring::activate waits for OpenVMM's restore-ready event, reading it as soon as it comes, since OpenVMM on Windows waits until the host has read it, and refuses another process's connection and other bytes. Only then does it connect to the clone's control endpoint, whose console bounds the host's authentication from the connection on. It attaches with the launch's capability, sends the record, read again from clone-activation.bin and byte-equal to the launch's, in the adapter's Activate as the session's first request, and checks that Ready reports the record's transaction ID. - All of it runs by a restore-to-Ready deadline from the launch, 60 seconds by default. A session that the adapter lost is replaced by a new one that sends the same record, in at most three sessions; nothing else is retried. - Activated::start starts the sandbox by a Ready-to-Start deadline from the check of Ready, 30 seconds by default, and never sends the start again. Started::commit commits the launch, which writes its runtime record. - A failure ends the launch as Starting::abandon does. A crash before the commit leaves the journal, whose recovery ends the VMM while the lease lasts until the release; a crash after it leaves a VM that reattaches with its lease. The default deadlines keep a wide margin on WHP's CI runs, the slowest backend: restores within a third of a second, and 25 parallel layered launches of the test adapter within 19 seconds. The platform layer's ReadyListener gains receive(), which checks the peer's process on both platforms, and reads with an overlapped pipe on Windows. Tests, with a fake VMM and a fake control: the sequence through the commit; Ready of another transaction, a cold boot's, or none; a lost session replaced with the same record first; a changed record and a diverged Ready, which end the retries; failures that aren't retried; the limit of sessions; a clone that ended after a lost session; both deadlines; a start that's never resent; a VMM that exits, or sends anything but its event; and crashes of the caller around Ready and the start, before and after the commit, each with the lease state that recovery or reattach leaves. Two of the launch tests asked their fake VMM to exit before they opened its process, which no longer opens once the VMM has exited; a shared helper now opens it first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0
The host client sends a clone's activation record with Client::activate, and refuses an Activate from the guest, which only the host sends. host::activate runs the first step of a clone's order in place of host::ready: it sends the record as the session's first request, and checks the build in Ready, which also reports the clone's activation. host::CloneOrder implements nvx's CloneControl with these steps and host::start_sandbox. It takes a session for lost once a reset, the end of the broker's stream, or a failure of the transport ended it. OpenVMM's broker also ends the stream to refuse an attachment, so such a refusal is retried too, within the sequence's limit of sessions. Any other failure is final. Tests, with the fake broker: the session's bootstrap, the record as the first request, Ready with and without an activation, then the table and the start; a session that a later reset, the end of the stream, or a failed transport ended, whose successor sends the same record first; and the final failures: an ERROR refusal, deadlines, another build, a mismatched or undecodable answer, and a record beyond the framing's limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0
OpenVMM writes the restore-ready event once, and then closes its end of the endpoint, so the wait reads the event's bytes and nothing past them, and refuses those that differ from the event, beside a connection of another process than the clone's VMM. The documentation of the restore sequence and the crate's README said that it refused "other bytes than the event", which reads as if it refused bytes past the event too; they now say what it does. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0
b48e295 to
70f3b68
Compare
d6f9b68 to
426403c
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The restore handshake and Windows overlapped-pipe lifecycle are validated only with fakes, leaving real clone restores on KVM, MSHV, and WHP requiring human confirmation before integration.
No open issues 🎉
Review effort: Balanced
Stacked on #949; this PR retargets when its parent merges.
Chain: #939 → #940 → #941 → #888 → #949 → this PR → #951, on
dev; #879, #881, #885, #886, #877, #932, #884, #887, #933, #880, #936, and #937 have merged.Part of #844 (PR 3 of 5: the restore sequence).
What and why
#949 launches a clone as OpenVMM's restore of its leased generation's snapshot and returns a
Restoring. This PR runs the clone's restore sequence from there to its sandbox's start and the launch's commit, through the adapter's framing of the clone's control session, and frames that sequence in the test adapter.The restore sequence
Restoring::activateruns the sequence through aCloneControl, a new trait for the adapter's framing of the control session, by a restore-to-Ready deadline that runs from the launch,RESTORE_TO_READY_TIMEOUT, 60 seconds by default:clone-activation.bin, which must still be byte for byte the launch's, in the adapter'sActivate, as the session's first request.ReadyActivation::ofreturns it. A Ready that reports none, as a cold boot's does, or another transaction, fails the sequence.A session that the adapter lost is replaced by a new one that sends the same record first, in at most
MAX_ACTIVATION_SESSIONS, three, sessions; nothing else is retried.Activated::startthen starts the sandbox by a Ready-to-Start deadline that runs from the check of Ready,READY_TO_START_TIMEOUT, 30 seconds by default, and never sends the start again.Started::commitcommits the launch, which writes its runtime record. The host sends nothing but the record before Ready, so the guest's mappings and start always follow it.A failure ends the launch as
Starting::abandondoes. A crash before the commit leaves the journal, whose recovery ends the VMM, while the lease lasts until the release; a crash after it leaves a VM that reattaches with its lease.The default deadlines keep a wide margin on WHP's CI runs, the slowest backend: restores within a third of a second, and 25 parallel layered launches of the test adapter within 19 seconds.
The platform layer's
ReadyListenergainsreceive(), which checks the peer's process on both platforms, and reads with an overlapped pipe on Windows.The test adapter's framing
The host client sends a clone's activation record with
Client::activate, and refuses anActivatefrom the guest, which only the host sends.host::activateruns the first step of a clone's order in place ofhost::ready: it sends the record as the session's first request, and checks the build in Ready, which also reports the clone's activation.host::CloneOrderimplementsCloneControlwith these steps andhost::start_sandbox. It takes a session for lost once a reset, the end of the broker's stream, or a failure of the transport ended it. OpenVMM's broker also ends the stream to refuse an attachment, so such a refusal is retried too, within the sequence's limit of sessions. Any other failure is final.Not in this PR
Validation
At this head, 426403c, with OpenVMM at d2338ea2, per commit: on Windows, fmt and Clippy with
-D warningsfor nvx-rs in CI's three configurations (--all-targets --all-features,--lib, and--lib --no-default-features),tests/host_modules.rs, the tests of the platform, template, store, sandbox, and supervisor modules, and the test adapter's fmt, Clippy, and host tests; and for the x86_64 Linux target, Clippy with-D warningsin the same three configurations, the aarch64-apple-darwin check, and the test adapter's Linux host and musl guest Clippy.The gates, every step of check-nvx-rs on Windows and on Linux x86_64 with the host tests as root on Linux, and the VM suites on WHP and MSHV, run at the final heads of the stack once its reviews are clean.
Tests added, with a fake VMM and a fake control: the sequence through the commit; a Ready of another transaction, a cold boot's, or none; a lost session replaced with the same record first; a changed record and a diverged Ready, which end the retries; failures that are not retried; the limit of sessions; a clone that ended after a lost session; both deadlines; a start that is never resent; a VMM that exits, or sends anything but its event; and crashes of the caller around Ready and the start, before and after the commit, each with the lease state that recovery or reattach leaves. In the test adapter, with the fake broker: the session's bootstrap, the record as the first request, after which the host waits for Ready, Ready with and without an activation, then the table and the start; a session that a later reset, the end of the stream, or a failed transport ended, whose successor sends the same record first; and the final failures.
No VM: real restores come with PR 5's VM tests.