nvx-rs: attribute a clone's failures, record them, and reattach it by esaurez · Pull Request #951 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-28T03:24:20Z

Stacked on #950; this PR retargets when its parent merges.
Chain: #936 → #937 → #939 → #940 → #941 → #888 → #949 → #950 → this PR, on dev; #879, #881, #885, #886, #877, #932, #884, #887, #933, and #880 have merged.

Part of #844 (PR 4 of 5: failures, teardown, and reattach).

What and why

#950 runs a clone's restore sequence to its sandbox's start and the launch's commit. A failure there ends the launch with an error that says what failed, but not whose failure it is, and nothing of it outlives the caller. This PR attributes every failure of a clone, from its plan to its commit, records it in the clone's sandbox with the teardown that followed, and lets a caller that restarted take its clones over.

Attribution

A clone's plan, launch, and restore sequence fail with a CloneFailure, whose Evidence attributes the failure to an Outcome, at a Stage, once the clone got as far as its Progress:

the request: a denied authorization; a stale binding; invalid targets, format, settings, Activate, or mappings; an image, a scratch, or a root that the template cannot serve; and a supplied file of the wrong identity, length, or geometry, which the host's staging takes instead once the caller proves it, CloneFailure::with_staging_proven;
the host: an endpoint that cannot be bound, resource exhaustion, a host that cannot restore the template, another runtime, or a failure of its own operations;
unknown: a lost status, a timeout, a reset, a forced kill, or a broken protocol;
the template: a malformed generation, or a defect of its capture or repair that a diagnosis attributes to it. This is the only outcome that quarantines the generation, through its tombstone, CloneFailure::quarantine;
the repair stage, for a status of the clone's repair alone.

OpenVMM's outcome report gives the status with which the guest ended the clone's VM, which status_evidence attributes: a status of the repair at any point, and one of the activation before the start was sent. It prevails over what the host or the caller saw of the same end, but not over evidence of the template, which shows the provenance that a status alone leaves to establish.

A VMM that exits before its restore-ready event, as OpenVMM does once it cannot restore the snapshot, leaves its fatal error in its log. The failure's message keeps that error, and its time ABI code attributes the failure to the host when it says that the host cannot present the template's CPU profile, TSC, or LAPIC timer. The wait for the event gives a VMM that closed the endpoint without sending it ten seconds to exit.

Records and teardown

From the launch on, a failure ends the clone's VMM through its verified identity. The sandbox keeps two more files, which a release removes, as a removal of the sandbox does, and which a cold launch never writes:

clone-progress.log, the clone's progress log: a first line that names the launch and the boot of the host, then a line for each step from the spawn to the commit, without a sync. The launch writes the spawn before it returns, since a teardown of the launch removes its journal, which records the spawn too;
clone-failure.json, the record of the failure, JSON in format FAILURE_FORMAT. It is written before the teardown, with the teardown pending, so that a crash of the caller during the teardown leaves it for a reattach to finish, and again once the teardown ended. It holds the stage, the evidence and the status, the original outcome, the transaction, the generation, the progress, and the teardown: no VM, an exit or a termination that the host verified, or an unresolved one.

An unresolved teardown keeps the launch's journal or runtime record, and with it the lease. A clone whose sandbox records a failure launches no more under its lease. A release ends the reservation of the lock that acquired the lease before it removes anything (#949), so one that fails after it removed the failure record leaves no launch to begin. A launch that spawned its VMM but could not record it in its journal records the failure as of the spawn, and then ends the VMM through its handle.

A step of the caller's own between Ready and the commit, such as one that maps a directory of the host into the clone, fails the clone with the caller's evidence through Activated::fail or Started::fail. These record the failure and end the launch, as the sequence's own failures do.

Reattach

reattach_clone takes a clone over once the caller of its launch restarted:

A VMM of the clone's launch that runs under the caller's runtime identity is checked in a new session. The clone answers the byte-equal Activate of clone-activation.bin with the Ready that it answered once, and the reattach checks the transaction ID that it reports. ReattachedClone::Live returns the VM with the session.
A clone of another runtime identity, or one that fails the check, is ended through its verified identity, and fails at the reattach stage.
It records the failure of a launch that a crash interrupted, and of a VM that exited, which it classifies by one reading of its outcome report, before it ends the launch or forgets the VM. It finishes a failure that a crash left pending, and resolves the teardown of an earlier one with what the VMM left as it ended.
A record of a failure that it cannot read, malformed or of another format, still tells that the clone launched and failed, though not how. The reattach takes it as the record of an earlier failure. If the store holds the journal or the runtime record of the clone's launch, which the reattach may remove, it first replaces the record with its own failure at the reattach stage, with the read error and the host's evidence, its teardown pending, which keeps how far the clone got for a reattach after a crash. It then ends a running VMM through its verified identity and, whatever it found of the VM, records the failure with its teardown, over whose evidence a status of the guest's in OpenVMM's outcome report prevails. It never returns the clone live, ended, or indeterminate.
A progress log that is damaged, missing, or written during another boot of the host leaves how far the clone got uncertain, so a status that a workload may have given is no evidence of the activation.
It reads nothing of the template store, so a tombstone or a retirement of the generation leaves the clone its lease.
Its termination bound applies to a VMM whose launch committed. A launch that a crash interrupted ends as LockedStore::recover ends it, within the store's own wait for the VMM's termination.

CloneRecords::read reads what a sandbox keeps of a clone, for the operations that list, inspect, or remove clones.

Not in this PR

Clones in VMs through the test adapter (PR 5).
GPT clones and the fresh-scratch check (nvx-rs: activate a clone's root for GPT images and either upper, with its export, network settings and workload view #845); exports and network templates (nvx-rs: exports, MAPS and network templates for clones #846).

Validation

At this head, 6ea96a3b, with OpenVMM at d2338ea2, per commit: on Windows, fmt and Clippy with -D warnings for nvx-rs in CI's three configurations (--all-targets --all-features, --lib, and --lib --no-default-features), tests/host_modules.rs, the tests of the platform, template, store, sandbox, and supervisor modules, and the test adapter's fmt, Clippy, and host tests; and for the x86_64 Linux target, Clippy with -D warnings in the same three configurations, the aarch64-apple-darwin check, and the test adapter's Linux host and musl guest Clippy.

The gates, every step of check-nvx-rs on Windows and on Linux x86_64 with the host tests as root on Linux, and the test adapter's VM suites, vm_layered, vm_template, vm_restart, and vm_steps, on Windows (WHP) and Linux (MSHV), run at the final heads of the stack once its reviews are clean.

Tests added, with the fake VMM and the fake control:

the restore-ready listener, which reads a client that closed before its connection was accepted to its end, on both platforms;
the attribution table, row by row: the repair and activation statuses, the precedence of a status over what was seen, but not over the template's evidence, OpenVMM's fatal error and its codes, and resource exhaustion;
the teardowns, and the records: progress logs that a crash cut short or that another launch wrote, failure records of another format, and the line of OpenVMM's fatal error read from the end of its log;
the request's refusals: unauthorized images, malformed generations, and paths that no longer name their supplied files;
failures recorded before their teardown and again once it ended, and launches that cannot be ended, which keep their records and their lease until a reattach ends them;
reattaches: a committed clone in a new session; a clone that the reattach may not keep; launches that a crash interrupted; VMs that exited, classified by one reading of their reports; earlier failures finished; records of a failure that cannot be read, with a live VM, one under another runtime identity, VMs that exited with a status that is evidence, one that is not, or no report, launches that a crash interrupted before and after the recovery, an unbegun clone beside an earlier VM's report, and an indeterminate VM; and progress logs that a reboot or a crash may have cut short;
a caller that fails a ready or started clone with its own evidence;
a launch that its caller abandons before the clone's activation, whose progress log records the spawn, so that a reattach reads the outcome report of the clone's VM and records the failure that the guest's status gives.

These tests run no VM: clones' real restores come with PR 5's VM tests.

Copilot

🟡 Changes recommended

Lease release can remove the retry-blocking failure marker before revoking the reservation, allowing another launch under the same lease if cleanup fails.

3 open issues

What changed in this PR

This PR extends NVX’s clone restore lifecycle with attributed failures, persistent failure and teardown records, and reattachment after the caller restarts. It implements the failure-handling and recovery portion of #844; real-VM integration remains for the next PR.

Changes:

Adds typed failure attribution across clone planning, launch, activation, and commit.
Records clone progress and teardown outcomes, with cleanup when the lease is released.
Adds clone reattachment and fake-process tests for failures, recovery, and restore-ready connections.

File	Description
nvx-rs/src/host/template/store/leases/tests.rs	Tests release cleanup of progress and failure records.
nvx-rs/src/host/template/store/leases.rs	Documents clone records removed during release.
nvx-rs/src/host/template/store.rs	Exposes directory validation for failure attribution.
nvx-rs/src/host/template/mod.rs	Exports and documents failure and reattachment APIs.
nvx-rs/src/host/template/clone/tests/restore.rs	Tests attribution, teardown records, and caller-reported failures.
nvx-rs/src/host/template/clone/tests.rs	Tests planning failures and their evidence.
nvx-rs/src/host/template/clone/record.rs	Attributes activation-record construction errors.
nvx-rs/src/host/template/clone/reattach.rs	Implements clone reattachment and failure recovery.
nvx-rs/src/host/template/clone/launch.rs	Tracks launch progress and records failures.
nvx-rs/src/host/supervisor/tests.rs	Adapts tests to reattachment reporting callbacks.
nvx-rs/src/host/supervisor/mod.rs	Adds callbacks before teardown and runtime-record removal.
nvx-rs/src/host/store/tests.rs	Extends clone-file cleanup coverage.
nvx-rs/src/host/store/mod.rs	Registers progress and failure files with the sandbox store.
nvx-rs/src/host/store/lifecycle/tests.rs	Tests removal and validation of the new records.
nvx-rs/src/host/sandbox/scratch.rs	Exposes binding version for attribution checks.
nvx-rs/src/host/sandbox/distro.rs	Exposes binding version for attribution checks.
nvx-rs/src/host/platform/windows.rs	Tests restore-ready clients that disconnect before acceptance.
nvx-rs/src/host/platform/linux.rs	Tests restore-ready clients that disconnect before acceptance.
nvx-rs/README.md	Documents failure attribution, records, teardown, and reattachment.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

An unreadable failure record is treated as absent, allowing a clone with unresolved teardown to become usable again under the same lease instead of preserving the error or safely terminating the VM.

1 open issue

Do not treat unreadable failure records as absent · New

3 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The reattach wildcard branch discards clone-failure.json read errors and records a generic run-stage exit failure, losing diagnostics that should be preserved at Stage::Reattach.

1 open issue

Preserve clone-failure diagnostics for committed VMM exits · New

1 resolved since last review

Do not treat unreadable failure records as absent

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Cross-platform VMM termination and crash-recovery changes need human review because tests use fakes and real clone restore/reattach validation is deferred to PR 5.

No open issues 🎉

1 resolved since last review

Preserve clone-failure diagnostics for committed VMM exits

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The spawn state is not durably recorded before Restoring is returned, so early teardown can cause reattachment to skip failure reporting and incorrectly report a failed guest repair as Ended.

1 open issue

If Restoring is abandoned or dropped before activate(), teardown can remove the journal before… · New

_{Review effort: Balanced}

A clone's plan, launch, and restore sequence now fail with a CloneFailure, whose Evidence attributes the failure to an Outcome: the request, the host's staging once the caller proves it, the host, unknown, the template, which alone quarantines the leased generation through its tombstone, or the repair stage. OpenVMM's outcome report gives the status with which the guest ended the clone's VM, which prevails over what the host saw: a status of the repair at any point, and one of the activation before the start was sent. A VMM that exits before its restore-ready event leaves OpenVMM's fatal error in its log, whose time ABI code may attribute the failure to a host that cannot present the template's CPU or time; the wait for the event gives a VMM that closed the endpoint without the event ten seconds to exit. From the launch on, a failure ends the clone's VMM through its verified identity, and the sandbox records it in clone-failure.json, before the teardown, with the teardown pending, and again once the teardown ended, with its stage, evidence, status, original outcome, transaction, generation, progress, and teardown. The clone's progress log, clone-progress.log, whose first line names the launch and the host's boot, records each step from the spawn to the commit without a sync, in one write for each step from the restore-ready event on. A teardown that is unresolved keeps the launch's journal or runtime record, and with it the lease, and a clone whose sandbox records a failure launches no more under its lease. A release removes both files, and a cold launch writes neither. reattach_clone takes a clone over once the caller of its launch restarted: it checks a committed clone in a new session, with the byte-equal Activate of clone-activation.bin, which the clone answers with its cached Ready; ends a clone of another runtime identity, or one that fails the check, through its verified identity; records the failure of a launch that a crash interrupted, and of a VM that exited, before it ends the launch or forgets the VM; classifies a VM that exited by one reading of its outcome report; and finishes a failure that a crash left pending, or resolves the teardown of an earlier one. A progress log that is damaged, missing, or written during another boot leaves how far the clone got uncertain, so that a status that a workload may have given is no evidence of the activation. It reads nothing of the template store, so a tombstone or a retirement of the generation leaves the clone its lease. CloneRecords::read reads what a sandbox keeps of a clone. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0

A step of the caller's own between a clone's Ready and the commit of its launch, such as one that maps a directory of the host into the clone, may fail. Activated::fail and Started::fail fail the clone with the caller's evidence, at the start and the commit stages, through the recorded teardown that the restore sequence's own failures take: the failure is recorded before the teardown and once it ended. Abandoning a launch records nothing, as before. A status of the guest's that is evidence prevails over what the host or the caller saw of the same end, but no longer over evidence of the template, a check of the generation or a diagnosis that attributes a defect to it, which shows the provenance that the status alone leaves to establish. The record holds the status beside it, for a failure's record, the finish of a pending one, and the resolution of an unresolved one alike. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0

A launch whose VMM was spawned, but whose identity it could not record in its journal, recorded the clone's failure as if no VMM had been spawned: the launch then ends the VMM through its handle, and removes the journal, so nothing else tells it. The failure now records the spawn, and Progress::Spawned says that the journal records the VMM unless the launch could not record it. The documentation of reattach_clone says that its bound applies to a VMM whose launch committed: a launch that a crash interrupted ends as LockedStore::recover ends it, within the wait that the store's recovery allows the VMM's termination. Evidence::Protocol says, as the restore sequence now does, that the bytes that it refuses at the restore-ready endpoint are those that differ from the event. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0

A reattach took a record of the clone's failure that it could not read for none, so a clone that had failed came back live if its VM still ran and answered, ended without a failure if its VM had exited, and indeterminate if its VM could not be examined. The record still tells that the clone launched, and failed: the reattach now fails such a clone at its own stage, with the error of the record's read, the host's evidence, once it ended a VMM of the clone that runs through its verified identity, or with the teardown unresolved for one that it cannot examine, which a later reattach resolves. The failure that it records replaces the record that cannot be read. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0

A reattach that found a record of the clone's failure that it could not read failed the clone with the error of the read only where it would otherwise have kept the clone, found it ended, or found its VM indeterminate. Elsewhere the error was lost: before it ended a launch that a crash of its caller interrupted, or cleared the runtime record of a VM that exited with a status of the guest's that is evidence, or without a report of the guest's own end, the reattach recorded a new failure in place of the record, at the stage of the clone's progress, as it did for a VM that runs under another runtime identity. The reattach now takes such a record as the record of an earlier failure that cannot tell how the clone failed. If the store holds the journal or the runtime record of the clone's launch, which the reattach may remove, it first replaces the record with its own failure at its own stage, with the error of the record's read and the host's evidence, its teardown pending, which keeps how far the clone got for a reattach after a crash: before the restore-ready event, only the journal holds the spawn. It then ends a VMM of the clone that still runs through its verified identity, and records the failure with the teardown that it found or made, over whose evidence a status of the guest's in OpenVMM's outcome report prevails. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0

…launch returns A clone's launch noted the spawn of its VMM without writing it in the progress log, which recorded it with the next step, since the launch's journal records the spawn too. A launch that its caller abandoned, or dropped, before the clone's activation removed the journal as it ended, and nothing recorded the spawn any more: a later reattach then took the clone for one whose VMM never spawned, read no outcome report of its VM, and found it ended, even when the guest had ended the VM with a status that is evidence of a failure, such as one of its repair. The launch now appends the spawn to the progress log before it returns. A launch that cannot write it fails at the launch stage, which ends its VMM and records the failure, as the restore sequence does when it cannot record a step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0

Enrique Saurez (esaurez) self-assigned this Oct 28, 2026

Enrique Saurez (esaurez) requested a balanced review from Copilot October 28, 2026 03:24

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 28, 2026 03:24

View session

This was referenced Oct 28, 2026

nvx-rs: plan the clones of templates, and check their block geometry #888

Open

nvx-rs: build a clone's activation record and launch its restore #949

Open

nvx-rs, runtime-adapter: run a clone's restore sequence to its sandbox's start #950

Open