Repository navigation
runtime-adapter: activate a clone through Activate in the framing - #941
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Oversized Activate headers produce the wrong session status, and synchronous application/device waits can exceed the activation deadline; both need fixes and regression tests.
2 open issues
What changed in this PR
This PR completes the guest-side activation framing for #843, connecting NVX’s clone activation library to the runtime adapter. Repaired clones can now accept Activate, report activation in Ready, and serve workloads instead of exiting immediately.
Changes:
- Adds
Activateframing and activation metadata toReady. - Resumes captured sessions after repair, applies clone configuration, and handles deadlines, retries, and failure shutdown.
- Updates template preparation, documentation, and framing/session tests.
| File | Description |
|---|---|
tests/runtime-adapter/tests/host_session.rs |
Moves the unknown-kind test to 28. |
tests/runtime-adapter/tests/guest_session.rs |
Adds clone activation and session lifecycle tests. |
tests/runtime-adapter/src/worker/connection.rs |
Supports resumed sessions and bounded output draining. |
tests/runtime-adapter/src/worker/capture.rs |
Completes the template handshake and serves repaired clones. |
tests/runtime-adapter/src/worker/activation.rs |
Implements activation handling and failure-status selection. |
tests/runtime-adapter/src/worker.rs |
Integrates clone phases, deadlines, and shutdown handling. |
tests/runtime-adapter/src/template.rs |
Retires status 31 and updates lifecycle documentation. |
tests/runtime-adapter/src/root.rs |
Encodes and decodes activation metadata in Ready. |
tests/runtime-adapter/src/lib.rs |
Documents the clone activation flow. |
tests/runtime-adapter/src/guest.rs |
Adds time-limited terminal draining. |
tests/runtime-adapter/src/frame.rs |
Adds message kind 27 and its body limit. |
tests/runtime-adapter/src/bin/runtime-adapter-guest.rs |
Wires clone serving into the guest worker. |
tests/runtime-adapter/README.md |
Documents framing, lifecycle, statuses, and tests. |
nvx-rs/src/protocol/status.rs |
Clarifies protocol-failure documentation after Ready. |
nvx-rs/src/guest/clone/activation/tests.rs |
Uses the public resource-based constructor. |
nvx-rs/src/guest/clone/activation/apply/tests.rs |
Updates the test adapter’s activation construction. |
nvx-rs/src/guest/clone/activation.rs |
Exposes Activation::with_resources. |
nvx-rs/README.md |
Documents resource-based activation construction. |
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Deadline-limited device polling drops the final partial interval, potentially rejecting a device before the deadline and reporting ACTIVATION_ROOT instead of ACTIVATION_DEADLINE.
1 open issue
2 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The NVXS snapshot/restore and tty lifecycle interactions require human review and real-VM validation, since socket-based fakes cannot establish their runtime ordering.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
A clone that a restore of a template's snapshot starts now continues past its repair to Ready, in the adapter's own framing: - The template's worker bootstraps the guest for every clone's root before its capture, with the template's own root configuration, the default one: its cgroups and their resource values, which the snapshot holds. A bootstrap that fails ends the template with 13. - Before the capture, the worker also takes the reset with which the broker answers its attachment, and acknowledges it, so that the snapshot holds the session active. A clone then reads no reset but the one that the restore queues: the restore's broker would finish a reset that the template's broker began to send, and a second reset would fail the clone's activation. - Once it repaired a clone, the worker resumes that session on the same tty, without attaching again, acknowledges the reset that the restore queued, and runs the clone's activation, nvx::guest::clone::Activation, whose deadline ends 120 seconds after the repair's acknowledgement. - The framing has a new message, Activate, of kind 27, whose body is a clone's activation record, at most nvx::activation::MAX_RECORD_BYTES. The worker applies the accepted record with nvx::guest::clone::apply, with the inputs of the root's assembly that the template's configuration gives, and answers Ready, which now reports the activation after the build, as ReadyActivation encodes it. - The record's workload identity, if it has one, becomes the sandbox's identity, as a root configuration's does, so that executions name none. The worker then serves the host's order as after a cold boot. Until the clone is ready, its session takes nothing but the Activate. Each failure of the activation ends the clone with the status of its class, 50 to 58, which worker::activation::failure_status selects, and a broker that closes the stream before the clone is ready ends it with status 1. Either way, the worker waits for its stream, and then its tty, to take what it wrote for at most a second each, and keeps that status whatever the stream does, so that a stream that takes nothing, or fails, neither keeps the clone from ending nor changes its status. A retry of the same Activate gets the same Ready. The template's status 31, which ended a repaired clone at once, is gone, and a VM that boots cold refuses an Activate. Unknown kinds in the framing's tests move from 27 to 28. nvx-rs gains Activation::with_resources, which builds an activation from the resources and the acknowledgement's instant that a repair reports, for the adapter's session tests, which run without a restore. The documentation of status 51 now covers an Activate that changes the record once the clone is ready. Part of #843 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a2f342b-b46d-4a26-a11e-a996e3c1e925
An Activate whose header announces more than MAX_ACTIVATE_BODY bytes, the most that a record has, failed the framing, which ended a clone's session with status 1, although nvx::protocol::status gives a record that breaks a limit ACTIVATION_RECORD (52). - nvx::guest::clone::ActivationInput gets TooLong(length): an Activate whose record the adapter's framing refused unread. The activation takes it as an Activate whose record has that many bytes, which ActivationRecord::decode refuses with ActivationError::TooLong: in the transport bootstrap, as any Activate, with ActivationCause::Request, and from then on with ActivationCause::Record, whether the clone is ready or not, after the deadline's check, as every input. - The test adapter's worker hands such a refusal of a clone's framing, which still comes before the worker takes any byte of the body, to the clone's activation, and ends with the failure's status, through the bounded drain of UNACTIVATED_DRAIN, as for any failed activation. The worker of a VM that boots cold still ends its session on it. Part of #843 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a2f342b-b46d-4a26-a11e-a996e3c1e925
A clone's adapter applies the accepted record synchronously, and the root's assembly waited for each of its devices as long as the template's timeout allows, up to 600 s each in the test adapter, while the activation's deadline, 120 s there, passed: Activation::complete then refused the late application, but the clone failed only once the waits ended. - nvx::guest::clone::apply takes the activation's deadline, until which the assembly waits for each device at the latest, through the crate's root::assemble_until; root::assemble and root::Assembly are unchanged. A device that has not appeared by then fails the assembly as at its timeout, so the application ends by the deadline, past which complete fails the activation as expired, whatever the result. - The activation's example passes activation.deadline(), and its agent queues its acknowledgement and Ready, so that a host that reads nothing holds up neither the agent nor the deadline. - The test adapter's RootOperations::apply takes the deadline, which a clone's worker passes from its activation. Part of #843 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a2f342b-b46d-4a26-a11e-a996e3c1e925
An assembly with a deadline capped each device's wait at the time left when the wait began, then polled a fixed number of times, so a slow poll or a sleep that overran could carry it past the deadline. The polling now takes the deadline itself: it sleeps no longer than until the deadline, and polls no more once the deadline passed, however long a poll or a sleep took. Part of #843 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a2f342b-b46d-4a26-a11e-a996e3c1e925
A clone's worker completed the activation as soon as the application of the Activate's record returned, and answered Ready before it took what the host had sent meanwhile: a reset or a request that came before Ready was taken once the clone was ready, as a later session's reset or as a request in the clone's order. The worker now keeps the application's result until its session took everything that it received, which it passes the activation first: a retry of the Activate, byte for byte, joins the application, and Ready answers it too, and anything else, a reset included, fails the activation, as before the application. A host therefore waits for Ready before it sends the mapping table or the start. The example of nvx's Activation does the same, and the tests' fake root holds a clone's application at its gate, so that the host's input comes while the application runs. Part of #843 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a2f342b-b46d-4a26-a11e-a996e3c1e925
When a clone's deadline came before the end of an assembly's timeout, the polling for a device counted the polls that whole intervals before the deadline allowed, and dropped the last part of an interval: with less than an interval left, it looked once, and failed the assembly, before the deadline, with the root's status. The polling now rounds the count up when the deadline comes first, so that it looks once more, at the deadline, after a shorter sleep, and a device that is still missing then fails the activation with the deadline's status. Part of #843 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a2f342b-b46d-4a26-a11e-a996e3c1e925
d912f46 to
ea69821
Compare
d3809fb to
2747742
Compare

Stacked on #940; this PR retargets when its parent merges.
Chain: #933 → #880 → #936 → #937 → #939 → #940 → this PR, on
dev; #879, #881, #885, #886, #877, #932, #884, and #887 have merged.Part of #843 (PR 3 of 3: Activate in the test adapter's framing).
What and why
With #937, the runtime adapter's guest repairs each clone that a restore of its template's snapshot starts, then ends it at once with status 31, since nothing activates a clone yet. With #939's
Activationand #940'sapply, the guest now continues past the repair to Ready, in the adapter's own framing.The framing
Activate: the host's activation of a clone, which the guest answers withReadynvx::activation::ActivationRecord::encodewrites it, at mostnvx::activation::MAX_RECORD_BYTES(frame::MAX_ACTIVATE_BODY)Ready(kind 7) reports its activation after the guest agent's build, asnvx::activation::ReadyActivation::encodewrites it:1, then the transaction ID of the record's restore.root::Ready::with_activationadds it, androot::Ready::activationreturns it, so a host checksReady::decode(body)?.activation() == Some(ReadyActivation::of(&record)).Readyreports no activation, androot::Ready::decoderefuses bytes after the build that are not an activation.Activate, before and after Ready, as it refuses any message that it does not take in its state.Activatethat announces more thanframe::MAX_ACTIVATE_BODYbytes before it reads any byte of the body. A clone's activation takes that refusal as anActivatewhose record has that many bytes, which fails it with 52 once the session is restricted, whether the clone is ready or not.A clone's session
Activate. The activation decodes and checks the record, and the worker applies it withnvx::guest::clone::apply, with the inputs of the root's assembly that the template's own configuration gives (worker::capture::root_template), and the activation's deadline, past which the application waits for no device. Once it took what the session received while the application ran, it completes the activation, which opens the fence of workload admission, and answersReady, to theActivateand to each byte-equal retry that came meanwhile, which joins the application.Until the clone is ready, the session takes nothing but that
Activateand its byte-equal retries: anything else that came beforeReady, while the application ran included, a reset among them, fails the activation, so a host waits forReadybefore it sends the mapping table or the start. A byte-equal retry gets the sameReadywithout a second application, also after a later reset. The activation's deadline,worker::activation::ACTIVATION_TIMEOUT, is 120 seconds from the repair's acknowledgement of the restore.How a clone ends
worker::activation::failure_statusselects the status of an activation's failure, withnvx::protocol::status's constants:ACTIVATION_DEADLINEACTIVATION_PROTOCOLActivateof another transaction or recordACTIVATION_RECORDActivate's header announces more bytes than a record hasACTIVATION_RESOURCESACTIVATION_ROOTACTIVATION_WORKLOADACTIVATION_ORDERworker::Worked::Abandoned).worker::UNACTIVATED_DRAIN, a second each. It keeps the status whatever the stream does, so a stream that takes nothing, or fails, neither keeps the clone from ending nor changes its status.Changes to #937's template path
Changes to #939's and #940's APIs
The review's fixes, below, change two APIs of this PR's parents, which stay as they are:
nvx::guest::clone::apply(record, template, deadline)takes the activation's deadline,Option<Instant>. The assembly waits for each device as long as the template's timeout allows, but never past the deadline: it checks the time before each poll, and sleeps no longer than until the deadline, at which it polls a last time if the deadline comes first, so that the application ends by then, past whichActivation::completefails the activation as expired.nvx::guest::root::assemblekeeps its behavior.nvx::guest::clone::ActivationInput::TooLong(length)is anActivatewhose header announcedlengthbytes, more than a record has, which the adapter's framing refused unread. It fails the activation as a record that long does, withActivationError::TooLong, or, in the transport bootstrap, as a request.Activationexample passes the activation what the session received while the application ran before it completes the activation, and answers each retry that joined the application withReady.Review fixes
Activatetoo long to read fails a clone with 52, rather than ending its session with 1.Readyfails the activation rather than reaching the ready clone.Elsewhere
nvx::guest::clone::Activation::with_resourcesbuilds an activation from the resources and the instant of the acknowledgement that a repair reports. The session tests use it, since they run without a restore.Activation::newcalls it.nvx::protocol::status::ACTIVATION_PROTOCOLnow covers anActivatethat changes the record after Ready.Validation
This PR was rebased onto #937's new head, 5683164, without conflicts and with its patches unchanged. At its new head, ea69821, with the OpenVMM submodule at its gitlink, d2338ea2:
Linux x86_64:
cargo fmt --check; Clippy with-D warnings,--all-targets --all-features,--lib, and--lib --no-default-features;cargo test --all-features(1800 passed),cargo test(877 passed), and the--lib --no-default-featuresbuild and test (260 passed);tests/host_modules.rs(22 passed); the bundled-artifacts test (19 passed); rustdoc with-D warnings; the MSRV (1.89) check; the aarch64-apple-darwin check.nvx.py check-guest-binaries(musl Clippy and tests, 982 passed),build-guest-binaries,test-guest-binaries(33 passed),check-static-elf.tests/runtime-adapter: fmt, Clippy, build, tests, and MSRV in its protocol-only (47 passed), host-only (99 passed), and musl guest (136 passed) configurations.scripts/check_markdown_links.py./dev/null: the privileged capture test, the strict cold boot that stops at its violation, and the tests ofguest::cloneandguest::reseed(75 passed); the runtime adapter's library and guest-session tests (128 passed).Windows (x86_64-pc-windows-msvc): nvx-rs fmt, Clippy (all features,
--lib, no default features, each feature alone, and all features for the musl target), tests (all features 1118 passed, default 896, no default features 260),tests/host_modules.rs, rustdoc, MSRV;tests/runtime-adapterfmt, Clippy, build, tests (protocol-only 47 passed, host 100), and MSRV in its protocol-only and host configurations, and Clippy and MSRV in its musl guest configuration; the Markdown links. The adapter's musl rustdoc with-D warnings, which CI doesn't run, fails with the same five broken links inexec.rsandworker/mapping.rsas at #937's head; this PR doesn't touch them.The runtime adapter's VM suites, in the CI run below, with OpenVMM at d2338ea2 and the guest agent built from ea69821: 58 of 58 passed on KVM, on MSHV, and on WHP (
vm_layered43,vm_template9,vm_restart3,vm_steps3). No suite restores a clone yet, so the activation runs in no VM here.Per commit, on Linux x86_64 and on Windows: fmt and Clippy with
-D warningsfor nvx-rs in CI's three configurations and--lib,tests/host_modules.rs, and the runtime adapter's fmt and Clippy in its protocol-only, host-only, and musl guest configurations; on Linux, also the aarch64-apple-darwin check.CI:
ci.ymlrun 40619378275, dispatched on ea69821, since this chain's PRs don't targetdevand get no PR CI, passed: 36 jobs succeeded, among them nvx-rs and the CLI on Linux and Windows, the guest artifacts, OpenVMM's binaries, unit tests, and vmm-tests, and the NVX microVM tests, with the runtime adapter's VM suites, and the platform jobs on KVM, MSHV, and WHP; the 6 jobs that only pull requests ordevrun were skipped.Tests added:
Activate's kind, body, and limit, and a kind for every code from 1 to 27.Ready: a clone's activation after the build, for builds of any length, and the bytes after a build that are not an activation, whichReady::decoderefuses.Activatetoo long to read fails as a request in the transport bootstrap, and from then on as a record that long, ready or not; an assembly waits for its devices until its deadline at the latest, also when a poll or a sleep overruns it, and finds a device that appears in the last, shorter, interval before the deadline; and the application hands the assembly the activation's deadline.tests/guest_session.rs): the cold worker refusesActivate; and a clone:Activate, whose retries get the sameReady;Activatebefore it is ready;Activatethat is not a retry once it is ready;Activatelonger than a record, before it takes the body, whether it is ready or not;Ready;Readyafter one application;Left to #844's PR 5, as the issue says: the VM tests on KVM, MSHV, and WHP, of no host input before the acknowledgement and no workload before Ready.