[Enrique Saurez (esaurez)]

Stacked on #881; this PR retargets to dev when the pull requests below it merge. The chain on dev is #883 → #876 → #871 → #873 → #874 → #879 → #881 → #885 → #886, after #863, #864, #865, #866, #867, #868, #869, #870, #872, #875, #878, and #882, which have merged.

A clone restores a generation of a template from the store's payload, so the generation must stay in its store as long as a clone may still restore it, or run from it. And a clone must not restore a capture outside the restore window of its time contract, which OpenVMM refuses only once the restore has begun. This adds both to nvx::host::template: the eligibility of a capture at a host's time, apart from its compatibility, and the leases of a store's generations, with their clone bindings, tombstones, and retirement. It changes no behavior: nothing leases a generation yet. Restore, #844, builds on it.

Part of #839: the third of its four pull requests. The first, #879, adds the compatibility key, and the second, #881, the store; the fourth adds the supervisor seam and the lock order.

What changed

Eligibility (template/eligibility.rs). TimeContract::eligibility(&HostTime), and Generation::eligibility, decide whether a clone may restore a capture at a host's time, as OpenVMM's restore decides it (vmm_core/virt/src/time_abi/downtime.rs, select_downtime, and openvmm_defs/src/time_abi.rs, vm_time_downtime_ns):

• the downtime is measured from the capture's anchor by the host's monotonic clock if the host, its boot, and its clock are those of the capture, and by UTC otherwise: after a reboot, on another host, or by another clock;
• a negative downtime is refused (Eligibility::Early), as on a host whose UTC is behind the capturing host's;
• so is one that would advance the clone's VM time, by the downtime and the interval from the cut of the VM's time to the anchor, by more than MAX_DOWNTIME_NS, 30 days (Eligibility::Expired).

HostTime holds the host's identity, its boot's, its monotonic clock and that clock's reading, and its UTC. HostTime::now samples them as OpenVMM samples them at a capture (vmm_core/virt/src/time_abi/host.rs): the machine ID or machine GUID, the boot ID, and UTC between two readings of CLOCK_BOOTTIME or the interrupt time, whose midpoint it keeps. Its fields are public, so that callers can check eligibility at other times, such as with a margin before a window closes.

Leases (template/store/leases.rs). Store::acquire leases a generation to the clone of one sandbox, in one transaction under the store's lock and then the sandbox's lifecycle lock. It checks that the store indexes the generation, that no tombstone quarantines it, that it is eligible at the host's time, and that the sandbox is provisioned, records no VM and no launch, and holds no lease. It then writes the lease's record in the store, leases/<L>.json, and the same record as the sandbox's clone binding, clone.json, beside its launch journal, before any VMM of the clone exists. It resolves the links of the sandbox's path before it locks the sandbox, and the lease, the binding, and the returned lock all name the sandbox by that path. So the removal or change of a link, or of a directory that a .. component passed through, can neither hide a live binding from a retirement nor redirect it. It returns the lease with the sandbox's lock, under which the caller launches the clone, and releases the store's lock.

A lease never expires. Store::release ends it only once the sandbox records no VM and no launch, so once the clone's VM has exited, as the sandbox's store verifies through the VM's recorded ProcessRef, also after a reattach. It removes the binding, and then the store's record. The caller takes the sandbox's lock through the lease's path, Lease::sandbox, as acquire does: a release refuses any other lock before it removes anything, since a link in the lock's path could change and redirect the removal to another sandbox. Store::select selects the latest generation of a key that is neither quarantined nor ineligible, and Store::leases lists a generation's leases. Lease::template returns the TemplateReference that a clone's activation record holds.

Tombstones and retirement. Store::tombstone quarantines a generation, tombstones/<G>.json: it takes no new lease, and selection skips it, while its leases keep working. Store::retire tombstones a generation, and deletes it once its leases have drained: its payload, then its entry in the index, then its tombstone, so that an interrupted retirement leaves a quarantined generation, which retiring it again deletes. It also removes the records of leases that no clone binding names any more, which an interrupted release or a removal of a sandbox leaves, and the temporary files of interrupted record replacements. The store's records and directories have the checks of #881's: private to the current user, no links or reparse points, and records in STORE_FORMAT 1, refused in any other. Every reader of a tombstone checks it, so a tombstone of another store or of another format fails a listing, a selection, an acquisition, or a retirement, rather than quarantining a generation or being removed. An entry that is not the store's fails with ForeignFiles.

Durability. Each transaction makes its steps durable in order, so that the order holds through a crash of the host, even when a sandbox and the store are on different filesystems:

• acquire makes the lease's record durable before it writes the binding, and the binding durable before it returns. If it cannot make the binding's removal durable after a failure, it keeps the record, and its error says that the sandbox may still hold the lease;
• release makes the binding's removal durable before it removes the record;
• a removal of a sandbox, LockedStore::remove, removes its binding first, and makes the binding's absence durable before anything else goes. Its configuration record still goes last, so a removal that fails to sync leaves a provisioned sandbox, which can be removed again;
• retire makes the tombstone durable before it deletes anything. It makes a binding's absence durable before it removes the record of that binding's lease, since a release that crashed or failed its sync may not have: in the sandbox's directory, and in the directory that holds the sandbox's entry, since a sandbox that was removed and provisioned again at its path is a new directory. Each deletion is durable before the next, and a retirement that finds its generation already unindexed makes the index durable before it removes the tombstone.

A new sync_dir, beside the stores' atomic writes (store/files.rs), syncs a directory on Unix. A process also syncs the store's directory, and then the store's parent, the first time that it uses each of leases/ and tombstones/, which the store never removes, in case the process that created them, or the store, did not finish that sync, so that no binding names a record of a store whose own entry a crash could lose. Windows has no portable way to flush a directory, so there a rename or a removal is as durable as the filesystem makes it, as OpenVMM's snapshot publication already assumes (openvmm_helpers/src/snapshot/fs.rs, sync_directory).

Concurrent removals. A release and a retirement, in other processes, may remove one lease's record at once, and the readers that take no lock may meet a record as another transaction removes it. On Windows, a file system without POSIX deletion keeps a removed file under its name while other handles, such as a scanner's, hold it, and refuses every open of it until then. Win32 reports that refusal as a denied access, as it reports a file whose access control list refuses the open. So the store takes a refused file for gone only if platform::is_deleted confirms it: that function opens the file again with the native API, relative to its directory, and only the status STATUS_DELETE_PENDING, or no file at all, counts as gone. Any other refusal fails the read or the removal, so a record that refuses access never counts as missing.

A removal of a sandbox removes its binding once, and takes a binding whose deletion is pending for removed. It deletes the binding with POSIX semantics where the filesystem supports them, so that its name goes at once. Elsewhere, a binding that a reader holds lingers, as a removed lock file does, until the reader closes it: the removal leaves it, and its Lingering::wait waits for it, rather than reporting it as a foreign entry.

The sandbox store (store/mod.rs, store/lifecycle.rs) counts the clone binding among its files, CLONE_BINDING_NAME in FILE_NAMES, so that a removal of a sandbox removes it, first, once, and durably, as above. The removal opens the sandbox's directory once. Through that handle, it checks that the directory holds the lock file whose lock it holds, and refuses otherwise, then that the sandbox records no VM and no launch, and then removes the binding and syncs the directory. So a link in the sandbox's path that changes meanwhile cannot redirect the binding's removal to another sandbox, whose VM may run. The rest of the removal goes through the path, as before, under the store's trust model: only the user or an administrator may change the directories above a sandbox's. The sandbox store also gains SandboxStore::binding_path.

Platform (platform/). host_id and monotonic_ns, on Linux and Windows, for HostTime::now. is_deleted, which on Windows tells a file whose deletion is pending from one that refuses access, through NtOpenFile; windows-sys gains its Wdk_Foundation and Wdk_Storage_FileSystem features for it. And Directory, a directory held open for a removal's operations on its entries: fstatat, unlinkat, and fsync on Unix, and, on Windows, native opens relative to the directory, which is_deleted now shares, and a deletion through a handle.

New public API to review

• In nvx::host::template: MAX_DOWNTIME_NS; HostTime; DowntimeSource; Downtime; Eligibility; TimeContract::eligibility; Generation::eligibility; LeaseId; Lease; Acquired; Retirement; LEASES_NAME; TOMBSTONES_NAME; and Store::select, acquire, release, leases, tombstone, tombstones, and retire.
• In nvx::host::store: CLONE_BINDING_NAME, which FILE_NAMES now includes, and SandboxStore::binding_path.
• Three error codes, in nvx::ErrorCode, the README, and doc/usage.md:
  • template-ineligible (kind Unavailable): the generation's restore window excludes the host's time;
  • template-tombstoned (kind Unavailable): a tombstone quarantines the generation;
  • sandbox-leased (kind Other): the sandbox holds a lease already.

The documentation of foreign-files, which #881 began to use for the template store, now covers it too.

Assumptions to review

• A lease names its sandbox by the path of its directory with its links resolved. That path must be valid Unicode, and, on Windows, a local path that names the directory without the verbatim prefix. A sandbox that holds a lease must not move, and is removed only through LockedStore::remove. For a sandbox removed otherwise, a retirement makes its binding's absence durable only while the directories that held the sandbox remain.
• A release needs the sandbox's lock taken through the lease's path, as Lease::sandbox returns it; the lock that acquire returns is one.
• The caller samples the host's time with HostTime::now, and passes it to acquire and select; OpenVMM checks the window again as it restores.
• A crash after acquire leaves the sandbox's lease, which release ends, since the sandbox then records no launch; the fourth pull request ties a lease to the launch that it reserves.
• select, leases, tombstones, and Lease::of read without the store's lock, so what they return may change before the caller acts on it; acquire checks again under the lock.
• Durability costs a clone's start two directory syncs on Unix, one for the lease's record and one for the binding, besides the syncs of the two files that the atomic writes already make. A process adds two syncs, of the store's directory and of its parent, the first time that it leases from a store. A removal of a sandbox costs one directory sync on Unix, and, on both platforms, a check of its lock file's identity: two stat calls. On Windows, it relies on the filesystem, as above, and also opens the sandbox's directory once. While a binding whose deletion is pending lingers, each scan of the directory checks it once more.

Validation

At c2aa75a, the head, restacked on #881's 867090f, on Windows and on Linux x86_64:

• cargo fmt --check; cargo clippy with --all-targets --all-features and --lib --no-default-features, -D warnings; cargo test with all, default, and no features; cargo doc with -D warnings, also of private items on Windows; cargo +1.89 check;
• tests/openvmm_vectors.rs with --ignored, against the OpenVMM submodule at its pin, a12d916;
• the runtime adapter's protocol-only and host-only configurations, with their MSRV checks, and, on Linux, its guest-only configuration;
• the checks of aci_edge_sandboxes, which builds on nvx: cargo fmt --check, cargo clippy with all features and without default features, cargo test with all and default features, the agent's bundling, cargo doc, and its MSRV check, and, on Linux, its macOS check;
• the tests of the bundled artifacts of nvx, against a synthetic bundle;
• the template and sandbox-store tests repeatedly: twice more on Windows and three times on Linux;
• scripts/test_nvx_tools.py's check that doc/usage.md documents every error code;
• on Linux, also the macOS check, and the guest binaries' check, build, and tests;
• at each of its commits, on both hosts: cargo fmt --check, and cargo clippy with --all-targets --all-features and --lib --no-default-features, -D warnings, and, on Linux, the macOS check.

The eligibility tests check clock fixtures within one boot and across a reboot, on another host, and by another clock, the bounds of the window, the cut of the VM's time, and the host's own time. The lease tests cover acquisitions, releases, tombstones, and retirements, their refusals, interrupted releases and retirements, records of other formats and stores, foreign entries, and links. They also check the order in which each transaction makes its steps durable, and the failed syncs, and that a removal of a sandbox makes its binding's absence durable before anything else goes, and keeps the sandbox provisioned if that sync fails. They check that a lease names its sandbox by its resolved path, also through a .. component or a link that goes later, that a link that changes during an acquisition does not redirect its binding, and that a release refuses a lock taken through a link, also once the link names another sandbox. The sandbox store's tests check that a removal through a path whose link changes between its checks and the binding's removal removes only the binding of the sandbox whose lock it holds, and that a removal through a path that has come to name another sandbox's directory refuses, and removes nothing. On Windows, they check that a removal takes a binding whose deletion is pending for removed, waits while a reader holds it, and finishes once the reader closes it. Each of these regressions fails when its fix is reverted. With several processes, they cover acquisitions that race to lease one sandbox, a tombstone that races acquisitions, and a retirement that races acquisitions and releases. On Windows, they also leave records pending deletion, as a file system without POSIX deletion does while a scanner holds them, and check that readers and a release take them for gone. They also check that records whose access control lists refuse every open are not taken for gone: a retirement then fails, and keeps the generation, and a tombstone still quarantines its generation. The race of a retirement and releases also ran 320 times on Windows, in eight parallel copies, with no failure; before the fix of concurrent removals, about one run in a hundred failed.

Every check passed on both hosts. One earlier run on Linux failed guest::cgroup::tests::settlement_collects_statuses_between_its_checks_until_the_deadline, a timing test of guest code that this change does not touch, which expects two collections within 50 ms. It passed 30 reruns on that host, and the whole suite passed again.

A final review by a separate model found issues in three rounds, which bb8fc3d addressed:

• The lease's record and the clone binding were each written atomically, but not durably in order. A crash of the host could then keep a binding on one filesystem and lose, on another, the record that keeps its generation, which a retirement would then delete. The transactions now make each step durable before the next, as described above.
• Three retry paths skipped a sync. A retirement trusted a binding's absence that a release whose sync failed, or that it raced, had not made durable. A retirement that found its generation already unindexed removed the tombstone without making the index durable. A directory of records whose creation failed to sync its entry was never synced again. Each now syncs.
• A retirement synced the parent of a link to a directory, rather than the directory itself, when the path of a removed sandbox ran through that link. It now follows the links of the sandbox's path to the directory from which the sandbox went, and fails, keeping the generation, if a link leads nowhere.

Copilot's review then found four issues, and the Windows tests a race, which 7c6f75b addresses:

• A sandbox that was removed and provisioned again at its path is a new directory. A retirement synced that directory, rather than the one that held the removed sandbox's entry, so a crash of the host could bring back the old binding. A retirement now syncs both.
• A new store's entry in its parent was never made durable, so a crash of the host could lose the store, and with it the record of a lease whose binding survived. Each process's first use of a directory of records now syncs the store's parent too.
• tombstones(), and so select(), took every entry named as a tombstone for one. The retirement of an unindexed generation also removed its tombstone without reading it. Both now check the tombstone, as acquire does.
• On Windows, a retirement failed with a denied access if it read a lease's record while a release was removing it. It now takes a record whose deletion is pending for gone, as described above.

The review of that fix found that it also took a record whose access control list refuses every open for gone, so a retirement could delete a leased generation. 1190e11 tells the two apart by the native status of the open.

Copilot's second review found that a lease recorded the path through which its sandbox was leased. On Unix, that path could pass through a directory by a .. component, or, on either platform, through a link. Once that directory or link went, the path named nothing, and a retirement took the live lease for unbound. b23cb64 resolves the links of the sandbox's path before it locks the sandbox. The lease, the binding, and the returned lock all name the sandbox by the resolved path, so a link that changes during or after the acquisition cannot redirect the binding either, as the final review of the fix required.

Copilot's third review found that a release took the sandbox's lock through any path. If a link in that path changed after the release checked that no VM runs, the release could remove another sandbox's binding, whose lock it did not hold. 43f5272 refuses, before it removes anything, a lock that does not name the sandbox by its lease's path.

Copilot's fourth review found that a removal of a sandbox removed its binding without making that durable. A retirement then synced only the directories that remained, so if the sandbox's parent had been removed and created again, a crash of the host could bring the binding back after the retirement had deleted its generation. a7e68ad makes a removal remove the binding first, and sync the sandbox's directory before anything else goes. Syncing every ancestor in the retirement instead would need read access to each, which hosts with unreadable ancestors do not grant.

Copilot's fifth review found two more issues in the removal of a sandbox, which 780ab5f addresses:

• A removal checked that the sandbox recorded no VM and no launch, and then removed its binding, through the sandbox's path. If a link in that path changed between the checks and the removal, the removal could remove the binding of another sandbox, whose VM may run, and a retirement could then delete the generation that its clone still needs. The removal now makes its checks and the binding's removal through one handle on the directory, after checking through it that the directory holds its lock, as described above.
• A removal removed the binding twice: first, and again with the store's other files. On a Windows filesystem without POSIX deletion, a reader that held the binding left the first removal pending, so the second failed with a denied access and aborted the removal. The binding now goes once, and one whose deletion is pending lingers until its readers close it.

780ab5f also makes the lease tests retry the recoveries of closed launches, as the fourth pull request's tests do: on Linux, a process that a concurrent test forks holds a copy of the closed log until it executes its program. A final review of 780ab5f by a separate model found no issues.

The restack on #874, through #879 and #881, resolved two conflicts, and changed no code. In the README's crate tree, it keeps dev's text and adds this pull request's leases. In the Windows tests of the sandbox store, dev added a test of provisions that wait for a removed directory where this pull request adds its test of a binding whose deletion is pending: both tests stay, and share dev's helper, which now also deletes directories. The race of a retirement and releases ran 320 times again at 4a8b534, in eight parallel copies, with no failure. A final review of the restack by a separate model found no issues.

A second restack, on #874's fa45e48, which drops the configuration identity of activation records, through #879 and #881, replayed this pull request's commits without conflicts, and changed no code. The race of a retirement and releases ran 320 times again at c2aa75a, in eight parallel copies, with no failure. On Linux, aci_edge_sandboxes' test crashed_vms_fall_back_to_provisioned, whose fake OpenVMM exits one second after it starts, failed once at this head while the host was loaded, because the start had not completed by then. The step passed on its rerun, and the test in 40 further runs, at this head and at #881's; this pull request adds nothing to a start's path. A final review of it by a separate model found no issues.

NVX CI does not run until this pull request targets dev.

[Copilot]

🟡 Changes recommended

Unresolved crash-durability gaps and tombstone validation bypasses undermine the store’s documented guarantees.

4 open issues

• Durably unlink sandbox binding before reprovisioning · New
• Sync store parent before publishing bindings · New
• Validate tombstone records before excluding generations · New
• Validate tombstone during already-unindexed retirement · New

What changed in this PR

Adds eligibility checks and persistent leases to the Rust host library’s template store, preparing it for clone restoration. This is the third stage of #839; runtime integration remains for later PRs.

Changes:

• Adds restore-window eligibility checks and host identity/time sampling on Linux and Windows.
• Adds clone bindings, lease acquisition and release, tombstones, retirement, and ordered durability steps.
• Adds concurrency and failure-recovery tests, public API documentation, and error codes.

File	Description
nvx-rs/​src/​lib.rs	Documents template leases in the crate overview.
nvx-rs/​src/​host/​template/​store/​tests.rs	Shares fixtures with lease tests.
nvx-rs/​src/​host/​template/​store/​leases/​tests.rs	Tests lease lifecycle, durability, validation, and concurrency.
nvx-rs/​src/​host/​template/​store/​leases.rs	Implements leases, clone bindings, tombstones, and retirement.
nvx-rs/​src/​host/​template/​store.rs	Exports lease APIs and documents store guarantees.
nvx-rs/​src/​host/​template/​mod.rs	Exports eligibility and lease types.
nvx-rs/​src/​host/​template/​eligibility/​tests.rs	Tests clock selection and restore-window boundaries.
nvx-rs/​src/​host/​template/​eligibility.rs	Implements host-time sampling and eligibility checks.
nvx-rs/​src/​host/​store/​mod.rs	Registers clone bindings as sandbox-owned files.
nvx-rs/​src/​host/​store/​files.rs	Adds directory synchronization.
nvx-rs/​src/​host/​platform/​windows.rs	Reads machine identity and interrupt time.
nvx-rs/​src/​host/​platform/​other.rs	Adds unsupported-platform stubs.
nvx-rs/​src/​host/​platform/​mod.rs	Adds host identity parsing and tests.
nvx-rs/​src/​host/​platform/​linux.rs	Reads machine identity and boot-time clock.
nvx-rs/​src/​host/​mod.rs	Documents template leasing.
nvx-rs/​src/​error.rs	Adds eligibility, quarantine, and lease error codes.
nvx-rs/​README.md	Documents new APIs and errors.
nvx-rs/​Cargo.toml	Enables Windows interrupt-time API support.
doc/​usage.md	Lists the new error codes.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-process crash-consistency protocol requires human review, and a confirmed lease-retirement correctness issue remains unresolved.

1 open issue

• Reject unresolved parent paths before recording leases · New

4 resolved since last review

• Sync store parent before publishing bindings
• Durably unlink sandbox binding before reprovisioning
• Validate tombstone during already-unindexed retirement
• Validate tombstone records before excluding generations

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Release can follow a retargeted symlink and remove a running clone’s lease without holding that sandbox’s lock.

1 open issue

• Reject symlinked lock paths before release cleanup · New

1 resolved since last review

• Reject unresolved parent paths before recording leases

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Retirement can delete a generation before an old binding’s disappearance is durable when a sandbox ancestor directory has been removed and recreated.

1 open issue

• Persist removal when sandbox parent is replaced · New

1 resolved since last review

• Reject symlinked lock paths before release cleanup

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Sandbox removal still has a path-retargeting race that can permit premature retirement and incomplete handling of Windows delete-pending bindings.

2 open issues

• Pin sandbox path before removal lock and unlink · New
• Avoid removing clone.json twice during sandbox cleanup · New

1 resolved since last review

• Persist removal when sandbox parent is replaced

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-filesystem crash consistency, concurrent retirement, and native Windows deletion semantics warrant final human review despite no new confirmed defects.

No open issues 🎉

2 resolved since last review

• Pin sandbox path before removal lock and unlink
• Avoid removing clone.json twice during sandbox cleanup

_{Review effort: Balanced}