From e46043e3be138a4c4459647107cd0694ec5ce191 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 25 Oct 2026 18:17:37 -0700 Subject: [PATCH 01/12] nvx-rs: attribute a clone's failures, record them, and reattach it A clone's plan, launch, and restore sequence now fail with a CloneFailure, whose Evidence attributes the failure to an Outcome: the request, the host's staging once the caller proves it, the host, unknown, the template, which alone quarantines the leased generation through its tombstone, or the repair stage. OpenVMM's outcome report gives the status with which the guest ended the clone's VM, which prevails over what the host saw: a status of the repair at any point, and one of the activation before the start was sent. A VMM that exits before its restore-ready event leaves OpenVMM's fatal error in its log, whose time ABI code may attribute the failure to a host that cannot present the template's CPU or time; the wait for the event gives a VMM that closed the endpoint without the event ten seconds to exit. From the launch on, a failure ends the clone's VMM through its verified identity, and the sandbox records it in clone-failure.json, before the teardown, with the teardown pending, and again once the teardown ended, with its stage, evidence, status, original outcome, transaction, generation, progress, and teardown. The clone's progress log, clone-progress.log, whose first line names the launch and the host's boot, records each step from the spawn to the commit without a sync, in one write for each step from the restore-ready event on. A teardown that is unresolved keeps the launch's journal or runtime record, and with it the lease, and a clone whose sandbox records a failure launches no more under its lease. A release removes both files, and a cold launch writes neither. reattach_clone takes a clone over once the caller of its launch restarted: it checks a committed clone in a new session, with the byte-equal Activate of clone-activation.bin, which the clone answers with its cached Ready; ends a clone of another runtime identity, or one that fails the check, through its verified identity; records the failure of a launch that a crash interrupted, and of a VM that exited, before it ends the launch or forgets the VM; classifies a VM that exited by one reading of its outcome report; and finishes a failure that a crash left pending, or resolves the teardown of an earlier one. A progress log that is damaged, missing, or written during another boot leaves how far the clone got uncertain, so that a status that a workload may have given is no evidence of the activation. It reads nothing of the template store, so a tombstone or a retirement of the generation leaves the clone its lease. CloneRecords::read reads what a sandbox keeps of a clone. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/README.md | 71 +- nvx-rs/src/host/platform/linux.rs | 39 + nvx-rs/src/host/platform/windows.rs | 39 +- nvx-rs/src/host/sandbox/distro.rs | 6 + nvx-rs/src/host/sandbox/scratch.rs | 6 + nvx-rs/src/host/store/lifecycle/tests.rs | 10 +- nvx-rs/src/host/store/mod.rs | 38 +- nvx-rs/src/host/store/regular.rs | 27 +- nvx-rs/src/host/store/tests.rs | 2 + nvx-rs/src/host/supervisor/mod.rs | 117 +- nvx-rs/src/host/supervisor/tests.rs | 6 +- nvx-rs/src/host/template/clone.rs | 702 ++++++--- nvx-rs/src/host/template/clone/failure.rs | 1231 ++++++++++++++++ nvx-rs/src/host/template/clone/launch.rs | 433 ++++-- nvx-rs/src/host/template/clone/reattach.rs | 499 +++++++ nvx-rs/src/host/template/clone/record.rs | 35 +- nvx-rs/src/host/template/clone/records.rs | 1268 +++++++++++++++++ nvx-rs/src/host/template/clone/restore.rs | 526 ++++--- nvx-rs/src/host/template/clone/tests.rs | 455 +++++- .../src/host/template/clone/tests/launch.rs | 1082 +++++++++++++- .../src/host/template/clone/tests/restore.rs | 394 ++++- nvx-rs/src/host/template/mod.rs | 18 +- nvx-rs/src/host/template/store.rs | 5 +- nvx-rs/src/host/template/store/leases.rs | 7 +- .../src/host/template/store/leases/tests.rs | 8 +- 25 files changed, 6361 insertions(+), 663 deletions(-) create mode 100644 nvx-rs/src/host/template/clone/failure.rs create mode 100644 nvx-rs/src/host/template/clone/reattach.rs create mode 100644 nvx-rs/src/host/template/clone/records.rs diff --git a/nvx-rs/README.md b/nvx-rs/README.md index df06835a6..4bf58e062 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -678,11 +678,12 @@ store's names is refused. `ClonePlan::launch` then launches the clone as the launch that its lease reserves, once it has checked that the sandbox's clone binding records the -lease, and returns a `Restoring`: it writes the clone's activation record in its -canonical encoding, `clone-activation.bin`, from which a caller resends it -unchanged; it listens for the VMM's restore-ready event, at the socket -`restore.sock` on Linux, and at a named pipe of a random name on Windows; and it -runs `supervisor::launch` with OpenVMM's restore of the clone's devices, +lease, and that the sandbox records no failure of the clone, which fails once +under its lease, and returns a `Restoring`: it writes the clone's activation +record in its canonical encoding, `clone-activation.bin`, from which a caller +resends it unchanged; it listens for the VMM's restore-ready event, at the +socket `restore.sock` on Linux, and at a named pipe of a random name on Windows; +and it runs `supervisor::launch` with OpenVMM's restore of the clone's devices, `--restore-snapshot` with the generation's snapshot in place, which no clone copies or changes, so that clones of one generation restore it at once, `--restore-memory` with the memory target, and `--restore-ready-path`, under the @@ -728,6 +729,66 @@ before the commit leaves the launch's journal, whose recovery ends the VMM, while the lease lasts until the release, and a crash after it leaves a VM that reattaches, and keeps its lease while it runs. +A failure of a clone is a `CloneFailure`, whose `Evidence` attributes it to an +`Outcome`, at a `Stage`, once the clone got as far as its `Progress`: the +request, for a denied authorization, a stale binding, invalid targets, format, +settings, Activate, or mappings, or an image, a scratch, or a root that the +template cannot serve, and for a supplied file of the wrong identity, length, or +geometry, which the host's staging takes instead once the caller proves it, +`CloneFailure::with_staging_proven`; the host, for an endpoint that cannot be +bound, resource exhaustion, a host that cannot restore the template, another +runtime, or a failure of its own operations; unknown, for a lost status, a +timeout, a reset, a forced kill, or a broken protocol; the template, for a +malformed generation, which alone quarantines the generation, through its +tombstone, `CloneFailure::quarantine`; and the repair stage, for a status of the +clone's repair alone. + +OpenVMM's outcome report gives the status with which the guest ended the clone's +VM, which `status_evidence` attributes, and which prevails over what the host +saw: a status of the repair at any point, and one of the activation before the +start was sent. + +A VMM that exits before its restore-ready event, as OpenVMM does once it cannot +restore the snapshot, after it closed the event's endpoint, which the sequence +then gives ten seconds to exit, leaves the line with which OpenVMM reported its +fatal error in its log, which the failure's message keeps, and whose time ABI +code, if it says that the host cannot present the template's CPU profile, TSC, +or LAPIC timer, attributes the failure to the host. + +From the launch on, a failure ends the clone's VMM through its verified +identity, and the sandbox keeps the clone's progress log, `clone-progress.log`, +which records each step from the spawn to the commit, after a first line that +names the launch and the boot of the host, a few lines that it appends without a +sync, and the record of its failure, `clone-failure.json`, JSON in format +`FAILURE_FORMAT`, which it writes before the teardown, with the teardown +pending, so that a crash of the caller during the teardown leaves it for a +reattach to finish, and again once the teardown ended, with the stage, the +evidence and the status, the original outcome, the transaction, the generation, +the progress, and the teardown: no VM, an exit or a termination that the host +verified, or an unresolved one, which keeps the launch's journal or runtime +record, and the lease; a release removes both, and a sandbox that boots cold +writes neither. + +`reattach_clone` takes a clone over after the caller of its launch restarted: it +reattaches by the identity of its VMM's process, checks a committed clone in a +new session, in which the clone answers the byte-equal Activate of +`clone-activation.bin` with the Ready that it answered once, and ends a clone +that another runtime identity launched, or that fails the check, through its +verified identity; it records the failure of a launch that a crash interrupted, +but not the journal that a commit may leave beside its runtime record, +classifies a VM that exited by its outcome report, records a failure before it +ends a VMM or clears the runtime record of one that exited, finishes a failure +that a crash left pending as its teardown would have recorded it, keeping its +transaction, resolves the teardown of an earlier failure with what the VMM left +as it ended, its outcome report and OpenVMM's fatal error, attributes a status +of the guest's as one that a workload may have given once the clone's progress +log is lost, damaged, or was written during another boot of the host, whose +crash may have lost its last lines, and reads nothing of the template store, so +that a tombstone or a retirement of the generation leaves the clone its lease. + +`CloneRecords::read` reads what the sandbox keeps of a clone, for the operations +that list, inspect, or remove clones. + `Store::tombstone` quarantines a generation, which takes no new lease while its leases keep working, and `Store::retire` deletes it once its leases have drained, crash-safely, removing the records of leases that no clone binding diff --git a/nvx-rs/src/host/platform/linux.rs b/nvx-rs/src/host/platform/linux.rs index eae6c4477..06d11afe2 100644 --- a/nvx-rs/src/host/platform/linux.rs +++ b/nvx-rs/src/host/platform/linux.rs @@ -1110,4 +1110,43 @@ mod tests { assert!(elapsed < Duration::from_secs(1), "{elapsed:?}"); assert!(process.wait(Instant::now()).unwrap().unwrap()); } + + #[test] + fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() { + let pid = std::process::id(); + let receive_all = |listener: &mut ReadyListener| { + let mut received = Vec::new(); + let deadline = Instant::now() + Duration::from_secs(10); + while listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + { + assert!(Instant::now() < deadline, "the endpoint did not close"); + } + received + }; + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("ready.sock"); + // A client that sent nothing, as a VMM that fails as it starts, and one that sent its + // event, each closed before the listener accepted their connection: what they sent + // remains to read, and then the end of the connection. + for sent in [&b""[..], b"the event\n"] { + let (endpoint, mut listener) = listen_ready(&path).unwrap(); + let mut client = UnixStream::connect(&endpoint).unwrap(); + std::io::Write::write_all(&mut client, sent).unwrap(); + drop(client); + assert_eq!(receive_all(&mut listener), sent); + } + // A client that closes once the listener accepted its connection. + let (endpoint, mut listener) = listen_ready(&path).unwrap(); + let client = UnixStream::connect(&endpoint).unwrap(); + let mut received = Vec::new(); + assert!( + listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + ); + drop(client); + assert_eq!(receive_all(&mut listener), b""); + } } diff --git a/nvx-rs/src/host/platform/windows.rs b/nvx-rs/src/host/platform/windows.rs index f249d9343..56c051bd2 100644 --- a/nvx-rs/src/host/platform/windows.rs +++ b/nvx-rs/src/host/platform/windows.rs @@ -2137,7 +2137,7 @@ pub(crate) fn interrupted() -> bool { #[cfg(test)] mod tests { - use std::io::Read; + use std::io::{Read, Write}; use super::*; @@ -2165,6 +2165,43 @@ mod tests { end_process(false, 7, || panic!("terminated"), || Ok(true)).unwrap(); } + #[test] + fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() { + let pid = std::process::id(); + let receive_all = |listener: &mut ReadyListener| { + let mut received = Vec::new(); + let deadline = Instant::now() + Duration::from_secs(10); + while listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + { + assert!(Instant::now() < deadline, "the endpoint did not close"); + } + received + }; + // A client that sent nothing, as a VMM that fails as it starts, and one that sent its + // event, each closed before the listener accepted their connection: what they sent + // remains to read, and then the end of the pipe. + for sent in [&b""[..], b"the event\n"] { + let (endpoint, mut listener) = listen_ready(Path::new("unused")).unwrap(); + let mut client = fs::OpenOptions::new().write(true).open(&endpoint).unwrap(); + client.write_all(sent).unwrap(); + drop(client); + assert_eq!(receive_all(&mut listener), sent); + } + // A client that closes once the listener accepted its connection. + let (endpoint, mut listener) = listen_ready(Path::new("unused")).unwrap(); + let client = fs::OpenOptions::new().write(true).open(&endpoint).unwrap(); + let mut received = Vec::new(); + assert!( + listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + ); + drop(client); + assert_eq!(receive_all(&mut listener), b""); + } + #[test] fn windows_refuses_to_terminate_a_process_that_exited_with_access_denied() { let mut child = std::process::Command::new("ping") diff --git a/nvx-rs/src/host/sandbox/distro.rs b/nvx-rs/src/host/sandbox/distro.rs index 0774bddc1..a3dbe7663 100644 --- a/nvx-rs/src/host/sandbox/distro.rs +++ b/nvx-rs/src/host/sandbox/distro.rs @@ -337,6 +337,12 @@ impl DistroBinding { self.policy } + /// Returns the binding's version, which a [check](Self::check) requires to be + /// [`BINDING_VERSION`]. + pub(crate) fn version(&self) -> u32 { + self.version + } + /// Returns the binding of `layer`, with the seals of its image and manifest and `policy`. fn new( layer: &SandboxLayer, diff --git a/nvx-rs/src/host/sandbox/scratch.rs b/nvx-rs/src/host/sandbox/scratch.rs index ccea75636..d69cfe4e5 100644 --- a/nvx-rs/src/host/sandbox/scratch.rs +++ b/nvx-rs/src/host/sandbox/scratch.rs @@ -257,6 +257,12 @@ impl ScratchBinding { self.length } + /// Returns the binding's version, which a [check](Self::check) requires to be + /// [`SCRATCH_BINDING_VERSION`]. + pub(crate) fn version(&self) -> u32 { + self.version + } + /// Returns the binding of the image that `seal` seals. fn of(seal: &FileSeal) -> Self { Self { diff --git a/nvx-rs/src/host/store/lifecycle/tests.rs b/nvx-rs/src/host/store/lifecycle/tests.rs index ee248e2f6..9b09876a8 100644 --- a/nvx-rs/src/host/store/lifecycle/tests.rs +++ b/nvx-rs/src/host/store/lifecycle/tests.rs @@ -11,8 +11,8 @@ use std::time::Duration; use super::*; use crate::host::store::{ - CAPABILITY_NAME, CLONE_ACTIVATION_NAME, CLONE_SCRATCH_NAME, CONSOLE_NAME, LOG_NAME, - OUTCOME_NAME, RESTORE_READY_NAME, SOCKET_NAME, + CAPABILITY_NAME, CLONE_ACTIVATION_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, + CLONE_SCRATCH_NAME, CONSOLE_NAME, LOG_NAME, OUTCOME_NAME, RESTORE_READY_NAME, SOCKET_NAME, }; use crate::host::test_support::released; @@ -326,6 +326,8 @@ fn a_removal_removes_the_store_the_callers_files_and_the_directory() { CLONE_BINDING_NAME, CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, BOOT, ".runtime.json.12345.tmp", @@ -417,6 +419,8 @@ fn write_clone_files(dir: &Path) { CLONE_BINDING_NAME, CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ] { fs::write(dir.join(name), b"{}").unwrap(); @@ -536,6 +540,8 @@ fn linked_entries(more: &[&str]) -> Vec { let mut names = vec![ CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, + CLONE_FAILURE_NAME, + CLONE_PROGRESS_NAME, CLONE_SCRATCH_NAME, CONFIG, LOCK_NAME, diff --git a/nvx-rs/src/host/store/mod.rs b/nvx-rs/src/host/store/mod.rs index d6a66e3c8..45833869c 100644 --- a/nvx-rs/src/host/store/mod.rs +++ b/nvx-rs/src/host/store/mod.rs @@ -19,6 +19,8 @@ //! clone-scratch.raw the file of the scratch device of a clone whose upper is in RAM, a //! sparse file that the clone's plan creates //! clone-activation.bin the activation record of a clone, which its launch writes +//! clone-progress.log the progress log of a clone's launch and restore sequence +//! clone-failure.json the record of a clone's failure //! restore.sock the endpoint at which the launch of a clone awaits the restore-ready //! event of its VMM, on Linux //! ... the caller's configuration record and other files, which it declares @@ -316,19 +318,34 @@ pub const CLONE_ACTIVATION_NAME: &str = "clone-activation.bin"; /// random. [`template::Store::release`](crate::host::template::Store::release) removes it once the /// clone's VM has exited, as a [removal](self#removal) of the sandbox does. pub const RESTORE_READY_NAME: &str = "restore.sock"; +/// Name of the progress log of a clone's launch, which the launch, +/// [`template::ClonePlan::launch`](crate::host::template::ClonePlan::launch), creates, and to which +/// it and the clone's restore sequence append a line for each step that the clone reaches, without +/// syncing it, so that a reattach tells how far the clone got once the caller of its launch +/// crashed, and which [`template::Store::release`](crate::host::template::Store::release) removes, +/// as a [removal](self#removal) of the sandbox does. No other launch writes it. +pub const CLONE_PROGRESS_NAME: &str = "clone-progress.log"; +/// Name of the record of a clone's failure, which a failure of the clone from its launch on +/// writes before its teardown, and again once its teardown is known, as +/// [`template::CloneFailure`](crate::host::template::CloneFailure) describes, and which +/// [`template::Store::release`](crate::host::template::Store::release) removes, as a +/// [removal](self#removal) of the sandbox does. +pub const CLONE_FAILURE_NAME: &str = "clone-failure.json"; /// The files that the clone of a sandbox keeps of its launch, which a release removes once the /// clone's VM has exited, as a [removal](self#removal) of the sandbox does, before the clone /// binding. -const CLONE_FILE_NAMES: [&str; 3] = [ +const CLONE_FILE_NAMES: [&str; 5] = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, RESTORE_READY_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, ]; /// Every file that the store keeps in a sandbox's directory. Replacing a record leaves a /// temporary file named `.NAME.NUMBER.tmp` beside it only if the replacement was interrupted, and /// a VMM that is killed while it writes its outcome report leaves the file in which it stages it; /// see [the report staging file](self#the-report-staging-file). -pub const FILE_NAMES: [&str; 12] = [ +pub const FILE_NAMES: [&str; 14] = [ LOCK_NAME, JOURNAL_NAME, RUNTIME_NAME, @@ -341,6 +358,8 @@ pub const FILE_NAMES: [&str; 12] = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, RESTORE_READY_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, ]; /// Interval at which a lock that another holder has is tried again: the sandbox's lock, by a @@ -550,6 +569,16 @@ impl SandboxStore { self.path(RESTORE_READY_NAME) } + /// Returns the path of the progress log of a clone's launch, [`CLONE_PROGRESS_NAME`]. + pub fn clone_progress_path(&self) -> PathBuf { + self.path(CLONE_PROGRESS_NAME) + } + + /// Returns the path of the record of a clone's failure, [`CLONE_FAILURE_NAME`]. + pub fn clone_failure_path(&self) -> PathBuf { + self.path(CLONE_FAILURE_NAME) + } + /// Reads the runtime record, which names the sandbox's VM from the commit of its launch until /// its runtime is cleared. pub fn runtime(&self) -> Result> { @@ -807,8 +836,9 @@ impl LockedStore { /// Removes what the clone of the sandbox kept of its launch in the sandbox's directory, whose /// VM has exited, through `directory`, the handle of the directory that /// [`open_dir`](Self::open_dir) returned: the file of its scratch device, - /// [`CLONE_SCRATCH_NAME`], its activation record, [`CLONE_ACTIVATION_NAME`], and the socket of - /// its restore-ready event, [`RESTORE_READY_NAME`]. A release, as a [removal](self#removal) + /// [`CLONE_SCRATCH_NAME`], its activation record, [`CLONE_ACTIVATION_NAME`], the socket of its + /// restore-ready event, [`RESTORE_READY_NAME`], its progress log, [`CLONE_PROGRESS_NAME`], and + /// the record of its failure, [`CLONE_FAILURE_NAME`]. A release, as a [removal](self#removal) /// does, removes them before the clone binding, whose removal, once durable, makes theirs /// durable too, since all are entries of the sandbox's directory. A file whose deletion is /// pending, as a Windows file system without POSIX deletion leaves a removed file that another diff --git a/nvx-rs/src/host/store/regular.rs b/nvx-rs/src/host/store/regular.rs index e85f2845b..da503f91b 100644 --- a/nvx-rs/src/host/store/regular.rs +++ b/nvx-rs/src/host/store/regular.rs @@ -2,6 +2,7 @@ //! binding: regular files that are not links, which they read within a bound, and the identity of //! a file that a template module holds open. +use std::fmt; use std::fs::{File, OpenOptions}; use std::io::{self, Read}; use std::path::Path; @@ -9,9 +10,29 @@ use std::path::Path; use crate::host::seal::FileSeal; use crate::host::{filesystem, platform}; +/// Why a read of this module refused a file for what the file is, as [`is_refused`] tells. +#[derive(Debug)] +struct Refused(String); + +impl fmt::Display for Refused { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&self.0) + } +} + +impl std::error::Error for Refused {} + +/// Returns whether `error` is one with which a read of this module refused a file for what the +/// file is, rather than an operation of the host's that failed: a link, or a file that is not a +/// regular file, which [`open_regular`] and [`open_regular_in`] refuse, and a file that holds +/// more than its bound, which [`read_at_most`] refuses. +pub(crate) fn is_refused(error: &io::Error) -> bool { + error.get_ref().is_some_and(|inner| inner.is::()) +} + /// Returns the error of a file that is a link, or not a regular file. fn not_regular() -> io::Error { - io::Error::other("it is a link, or not a regular file") + io::Error::other(Refused("it is a link, or not a regular file".to_owned())) } /// Returns whether `error`, with which an open or a removal of the file at `path` failed, means @@ -99,9 +120,9 @@ pub(crate) fn read_at_most(file: File, limit: u64, what: &str) -> io::Result limit { - return Err(io::Error::other(format!( + return Err(io::Error::other(Refused(format!( "it holds more than the {limit} bytes that {what} may hold" - ))); + )))); } Ok(bytes) } diff --git a/nvx-rs/src/host/store/tests.rs b/nvx-rs/src/host/store/tests.rs index ef11598a0..12ddc19ea 100644 --- a/nvx-rs/src/host/store/tests.rs +++ b/nvx-rs/src/host/store/tests.rs @@ -410,6 +410,8 @@ fn the_files_that_a_clone_kept_of_its_launch_go_and_nothing_else() { for path in [ store.clone_scratch_path(), store.clone_activation_path(), + store.clone_progress_path(), + store.clone_failure_path(), store.restore_ready_path(), store.log_path(), store.binding_path(), diff --git a/nvx-rs/src/host/supervisor/mod.rs b/nvx-rs/src/host/supervisor/mod.rs index 5d1b81005..95a5dbac0 100644 --- a/nvx-rs/src/host/supervisor/mod.rs +++ b/nvx-rs/src/host/supervisor/mod.rs @@ -130,7 +130,10 @@ //! session, and checks its Ready, by a deadline that runs from the launch; //! [`Activated::start`](crate::host::template::Activated::start) starts its sandbox; and //! [`Started::commit`](crate::host::template::Started::commit) commits the launch, as -//! [`Starting::commit`] does. A failure ends the launch, as [`Starting::abandon`] does. +//! [`Starting::commit`] does. A failure ends the launch, as [`Starting::abandon`] does, through +//! the VMM's verified identity, and the sandbox records it until the release, with the evidence +//! that attributes it, as a [`CloneFailure`](crate::host::template::CloneFailure) describes: a +//! launch that cannot be ended keeps its journal, and the lease. //! 4. Once the clone's VM has exited, as a stop, a termination, or a reattach verifies, //! [`Store::release`](crate::host::template::Store::release) ends the lease, under the //! sandbox's lock, taken through the lease's path, @@ -174,6 +177,12 @@ //! - During a release, the binding goes first, durably, and leaves the record of the lease, which //! a retirement removes. //! +//! The caller of a clone's launch that restarts takes the clone over with +//! [`reattach_clone`](crate::host::template::reattach_clone), which reattaches as [`reattach`] +//! does, records the failure of a launch that it ended, checks the activation of a clone whose +//! launch committed in a new session, and ends a clone that another runtime identity launched, or +//! that fails the check, through its verified identity. +//! //! The evidence of the VM's exit decides each outcome. A record of an earlier boot, or of a process //! ID that another process now has, names a VM that has exited: reattach clears it, and the lease //! may be released. A VM that cannot be examined is indeterminate, and keeps its records, and so @@ -222,31 +231,75 @@ pub fn launch<'s, 'h>( capability: &[u8; CAPABILITY_LEN], command: impl FnOnce(&str) -> Result>, ) -> Result> { + launch_reporting(locked, reserved, runtime, capability, command, |_, _| {}) + .map_err(|failure| failure.error) +} + +/// How a launch of [`launch_reporting`] failed. +#[derive(Debug)] +pub(crate) struct LaunchFailure { + /// The error of the step that failed, which says why the launch could not be ended if it + /// could not. + pub(crate) error: Error, + /// Whether the launch spawned its VMM, which it then ended through the VMM's handle. + pub(crate) spawned: bool, + /// Why the launch could not be ended, whose journal then remains for + /// [`LockedStore::recover`], or `None` once it ended. + pub(crate) unended: Option, +} + +/// Launches a VMM as [`launch`] does, and tells how a launch that fails ended, as a +/// [`LaunchFailure`]. A launch that began, and fails, calls `failing` with the step's error, and +/// whether it spawned the VMM, before it ends the launch, so that the caller may record the +/// failure, as a clone's launch does, before anything of the launch ends. +pub(crate) fn launch_reporting<'s, 'h>( + locked: &'s mut LockedStore, + reserved: &LaunchId, + runtime: &RuntimeIdentity, + capability: &[u8; CAPABILITY_LEN], + command: impl FnOnce(&str) -> Result>, + failing: impl FnOnce(&Error, bool), +) -> std::result::Result, LaunchFailure> { let store = SandboxStore::clone(locked); - let mut launch = locked.begin_launch(reserved, runtime, capability)?; + // A launch that does not begin leaves nothing to end. + let mut launch = locked + .begin_launch(reserved, runtime, capability) + .map_err(|error| LaunchFailure { + error, + spawned: false, + unended: None, + })?; let command = match command(launch.endpoint()) { Ok(command) => command, - Err(error) => return Err(abandon_after(launch, error)), + Err(error) => { + failing(&error, false); + return Err(abandon_after(launch, error, false)); + } }; let log = match launch.take_log() { Ok(log) => log, - Err(error) => return Err(abandon_after(launch, error)), + Err(error) => { + failing(&error, false); + return Err(abandon_after(launch, error, false)); + } }; let child = match spawn(&command, capability, log) { Ok(child) => child, Err(error) => { let error = Error::io(format!("cannot launch {}", command.program.display())) .with_source(error); - return Err(abandon_after(launch, error)); + failing(&error, false); + return Err(abandon_after(launch, error, false)); } }; let process = match launch.record_process(&child) { Ok(process) => process, Err(error) => { + failing(&error, true); // An unrecorded VMM holds the claim of its launch, so only its handle can end it, // before the launch ends. child.kill_and_wait(); - return Err(abandon_after(launch, error)); + return Err(abandon_after(launch, error, true)); } }; Ok(Starting { @@ -259,12 +312,19 @@ pub fn launch<'s, 'h>( }) } -/// Ends `launch`, which failed with `error`, and returns `error`, which says why the launch could -/// not be ended if it could not: its journal then remains for [`LockedStore::recover`]. -fn abandon_after(launch: Launch<'_>, error: Error) -> Error { +/// Ends `launch`, which failed with `error`, after it `spawned` a VMM or not, and returns how it +/// failed, whose error says why the launch could not be ended if it could not: its journal then +/// remains for [`LockedStore::recover`]. +fn abandon_after(launch: Launch<'_>, error: Error, spawned: bool) -> LaunchFailure { let reason = match launch.abandon() { Ok(Recovery::Indeterminate(reason)) => reason, - Ok(_) => return error, + Ok(_) => { + return LaunchFailure { + error, + spawned, + unended: None, + }; + } Err(failure) => failure.to_string(), }; let (kind, message, source) = error.into_parts(); @@ -272,9 +332,13 @@ fn abandon_after(launch: Launch<'_>, error: Error) -> Error { kind, format!("{message}; the launch could not be ended, and remains for recovery: {reason}"), ); - match source { - Some(source) => error.with_source(source), - None => error, + LaunchFailure { + error: match source { + Some(source) => error.with_source(source), + None => error, + }, + spawned, + unended: Some(reason), } } @@ -367,7 +431,14 @@ impl<'s> Starting<'s> { /// /// Fails if the runtime record cannot be written. The VMM is then ended through its handle, /// and the journal that remains shows [`LockedStore::recover`] that it exited. - pub fn commit(mut self) -> Result { + pub fn commit(self) -> Result { + self.commit_reporting(|_| {}) + } + + /// Commits the launch as [`commit`](Self::commit) does, and calls `failing` with the error of + /// a commit that fails, before it ends the VMM, so that the caller may record the failure + /// first, as a clone's launch does. + pub(crate) fn commit_reporting(mut self, failing: impl FnOnce(&Error)) -> Result { let StartingInner { launch, child, @@ -383,6 +454,7 @@ impl<'s> Starting<'s> { }) } Err(error) => { + failing(&error); // No runtime record names the VM, so its VMM must not run on. child.kill_and_wait(); Err(error) @@ -752,15 +824,27 @@ pub enum Reattached { /// can. Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for an invalid /// `runtime`, and as [`LockedStore::recover`] does for inconsistent records. pub fn reattach(locked: &mut LockedStore, runtime: &RuntimeIdentity) -> Result { - reattach_with(locked, runtime, Process::open) + reattach_with(locked, runtime, Process::open, |_| {}) +} + +/// Reattaches as [`reattach`] does, and calls `exited` with the runtime record of a VMM that it +/// found exited, before it clears the record, so that the caller may record what ended the VM +/// while the sandbox still names it, as a clone's reattach does. +pub(crate) fn reattach_reporting( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + exited: impl FnOnce(&RuntimeRecord), +) -> Result { + reattach_with(locked, runtime, Process::open, exited) } -/// Reattaches as [`reattach`] does, holding the recorded VMM with `open`, which is +/// Reattaches as [`reattach_reporting`] does, holding the recorded VMM with `open`, which is /// [`Process::open`] outside tests. fn reattach_with( locked: &mut LockedStore, runtime: &RuntimeIdentity, open: impl FnOnce(&ProcessRef) -> io::Result>, + exited: impl FnOnce(&RuntimeRecord), ) -> Result { runtime.check()?; let recovery = locked.recover()?; @@ -779,6 +863,7 @@ fn reattach_with( let process = match open(record.process()) { Ok(Some(process)) => process, Ok(None) => { + exited(&record); locked.clear_runtime_of(Some(&record))?; return Ok(Reattached::Exited { record: Some(record), diff --git a/nvx-rs/src/host/supervisor/tests.rs b/nvx-rs/src/host/supervisor/tests.rs index 4a71a08a1..bd8397f8a 100644 --- a/nvx-rs/src/host/supervisor/tests.rs +++ b/nvx-rs/src/host/supervisor/tests.rs @@ -460,7 +460,8 @@ fn a_vmm_that_cannot_be_examined_is_indeterminate_and_keeps_its_record() { let hidden = |_: &ProcessRef| -> io::Result> { Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) }; - let Reattached::Indeterminate(reason) = reattach_with(&mut locked, &runtime(), hidden).unwrap() + let Reattached::Indeterminate(reason) = + reattach_with(&mut locked, &runtime(), hidden, |_| {}).unwrap() else { panic!("a VMM that cannot be examined was classified"); }; @@ -623,7 +624,8 @@ fn a_clone_whose_vmm_cannot_be_examined_keeps_its_lease() { let hidden = |_: &ProcessRef| -> io::Result> { Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) }; - let Reattached::Indeterminate(reason) = reattach_with(&mut locked, &runtime(), hidden).unwrap() + let Reattached::Indeterminate(reason) = + reattach_with(&mut locked, &runtime(), hidden, |_| {}).unwrap() else { panic!("a VMM that cannot be examined was classified"); }; diff --git a/nvx-rs/src/host/template/clone.rs b/nvx-rs/src/host/template/clone.rs index a99d49872..fa0cfe003 100644 --- a/nvx-rs/src/host/template/clone.rs +++ b/nvx-rs/src/host/template/clone.rs @@ -7,11 +7,19 @@ #[cfg(all(test, any(target_os = "linux", windows)))] mod tests; +mod failure; mod launch; +mod reattach; mod record; +mod records; mod restore; +pub use self::failure::{ + CloneFailure, Evidence, FailureRecord, Outcome, Progress, Stage, Teardown, status_evidence, +}; pub use self::launch::Restoring; +pub use self::reattach::{ReattachedClone, reattach_clone}; +pub use self::records::{CloneRecords, FAILURE_FORMAT}; pub use self::restore::{ Activated, CloneControl, MAX_ACTIVATION_SESSIONS, READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, RestoreDeadlines, Started, @@ -23,18 +31,21 @@ use std::io; use std::path::{Path, PathBuf}; use std::time::Instant; +use self::failure::{Attribute, Fault, host_evidence}; use super::key::TemplateKey; use super::manifest::BlockGeometry; use super::plan::MEMORY_BLOCK_SIZE; use super::store::{Acquired, Generation, Lease, Store}; use crate::activation::{ActivationRecord, Journal, OverlayMode, WorkloadView}; use crate::contract; -use crate::error::{Error, ErrorKind, Result}; +use crate::error::{Error, ErrorCode, ErrorKind, Result}; use crate::host::filesystem; use crate::host::instance::generate_capability; use crate::host::paths; use crate::host::platform; -use crate::host::sandbox::distro::{self, CheckedDistro, DistroBinding, DistroFormat}; +use crate::host::sandbox::distro::{ + self, CheckedDistro, DistroBinding, DistroFormat, PolicyResult, +}; use crate::host::sandbox::plan::bound_image; use crate::host::sandbox::scratch::{self, CheckedScratch, ScratchBinding}; use crate::host::sandbox::{ @@ -43,7 +54,7 @@ use crate::host::sandbox::{ }; use crate::host::store::LockedStore; use crate::host::store::files::{create_new, remove_if_present}; -use crate::host::store::regular::{identity_of, open_regular, was_removed}; +use crate::host::store::regular::{identity_of, is_refused, open_regular, was_removed}; use crate::protocol::CAPABILITY_LEN; /// Bytes in a mebibyte, the unit in which messages give the size of memory blocks. @@ -286,59 +297,94 @@ impl ClonePlan { /// snapshot, [`Store::snapshot_dir`], then checks the clone's files, as the [type](Self) /// describes. /// - /// Fails, and holds nothing, keeping no file that it created, checking in this order: + /// Fails with a [`CloneFailure`] at [`Stage::Plan`], and holds nothing, keeping no file that + /// it created. The failure's record holds the evidence of the check that failed, as below, no + /// transaction, and no VM, [`Teardown::NoVm`], and the plan records it nowhere: the caller + /// holds it, and releases the lease. The evidence of a failure of the host's own operations is + /// [`Evidence::ResourceExhaustion`] if an I/O error among the error's sources says that the + /// host ran out of storage, memory, or descriptors, and [`Evidence::HostFailure`] otherwise. + /// The plan checks in this order: /// /// - with [`ErrorKind::Unsupported`] for a request that no clone takes: a plan of the sandbox - /// profile, with workload settings and its shares; a plan whose image is not the format and - /// identity that the binding names, as for a binding whose identity is not a UUID in its - /// canonical text; an image that is a GPT disk, of which this build restores no clone, since - /// a clone's guest assembles its root only from an EROFS layer; an image or a scratch image - /// whose path is relative, which OpenVMM would resolve against the directory in which it - /// runs, holds a comma or a semicolon, which OpenVMM's block device options would split, or - /// whose name ends in `.vhd`, `.vhdx`, `.vmgs`, or `.iso`, which OpenVMM would not attach as - /// a raw image; and an upper on a scratch image without the image's binding, or one in RAM - /// with a binding; + /// profile, with workload settings and its shares, [`Evidence::InvalidSettings`]; a plan + /// whose image is not the format and identity that the binding names, as for a binding + /// whose identity is not a UUID in its canonical text, and an image that is a GPT disk, of + /// which this build restores no clone, since a clone's guest assembles its root only from an + /// EROFS layer, [`Evidence::InvalidFormat`]; an image or a scratch image whose path is + /// relative, which OpenVMM would resolve against the directory in which it runs, holds a + /// comma or a semicolon, which OpenVMM's block device options would split, or whose name + /// ends in `.vhd`, `.vhdx`, `.vmgs`, or `.iso`, which OpenVMM would not attach as a raw + /// image, and an upper on a scratch image without the image's binding, or one in RAM with a + /// binding, [`Evidence::InvalidSettings`]; /// - with [`ErrorKind::Other`] for a lease of another store, and a lock that is not the one - /// that acquired the lease, before the launch that it reserves began; - /// - with [`ErrorKind::NotFound`] if the store no longer indexes the leased generation, with - /// [`ErrorKind::Io`] if it indexes it with another key than the lease names, and as - /// [`Store::index`], [`Store::read_key`], and [`Store::snapshot_dir`] fail; + /// that acquired the lease, before the launch that it reserves began, + /// [`Evidence::HostFailure`]; + /// - with [`ErrorKind::NotFound`] if the store no longer indexes the leased generation, and as + /// [`Store::index`] fails, the host's evidence; with [`ErrorKind::Io`] if it indexes it with + /// another key than the lease names, [`Evidence::MalformedGeneration`]; and as + /// [`Store::read_key`] and [`Store::snapshot_dir`] fail, once the store's directory of + /// generations passed their check, whose failure is the host's: the host's evidence for an + /// I/O error, [`Evidence::IncompatibleHost`] for a key of another encoding version than this + /// build reads, with [`ErrorCode::FormatUnsupported`], which a build that reads it may + /// serve, and [`Evidence::MalformedGeneration`] for a generation whose payload, key, or + /// snapshot is missing, is not private to the current user, or is malformed, as a key that + /// [`Store::read_key`] refuses for what it is: a link, a file that is not a regular file, or + /// one that holds more than a key may; /// - with [`ErrorKind::Unsupported`] for a request that the template cannot serve: a template /// whose key names other contracts than this build speaks, the boot and control contracts - /// of the runtime that captured it included; a plan with a network for a template without a - /// NIC, without one for a template with a NIC, or with another guest network than the - /// template's NIC, whose address OpenVMM restores from the snapshot; and a memory target - /// that is not whole blocks of [`MEMORY_BLOCK_SIZE`] and of the template's memory blocks, - /// from the template's RAM to its RAM capacity; + /// of the runtime that captured it included, [`Evidence::IncompatibleHost`]; a plan with a + /// network for a template without a NIC, without one for a template with a NIC, or with + /// another guest network than the template's NIC, whose address OpenVMM restores from the + /// snapshot, [`Evidence::InvalidSettings`]; and a memory target that is not whole blocks of + /// [`MEMORY_BLOCK_SIZE`] and of the template's memory blocks, from the template's RAM to its + /// RAM capacity, [`Evidence::InvalidTargets`]; /// - with [`ErrorKind::Io`] for a template whose command line names the DNS server of its /// guest's resolver in another form than OpenVMM writes: more than once, without a NIC, or - /// other than as one IP address in its canonical spelling; + /// other than as one IP address in its canonical spelling, [`Evidence::MalformedGeneration`]; /// - with [`ErrorKind::Unsupported`] for settings that no activation record holds, as /// [`Configuration::check`] finds them, such as a hostname that is empty, longer than /// [`MAX_HOSTNAME_BYTES`](crate::activation::MAX_HOSTNAME_BYTES), or holds a NUL, and a - /// workload identity whose UID or GID is 0, and with [`ErrorKind::Other`] if the - /// transaction ID of the clone's restore cannot be drawn; + /// workload identity whose UID or GID is 0, [`Evidence::InvalidActivate`], and with + /// [`ErrorKind::Other`] if the transaction ID of the clone's restore cannot be drawn, the + /// host's evidence; /// - with [`ErrorKind::Unavailable`] for a link or a directory under a name that the sandbox's - /// store owns, and with [`ErrorKind::Io`] if the store's files cannot be listed or examined; - /// - as [`DistroBinding::check`] and [`CheckedDistro::geometry`] fail, with - /// [`ErrorKind::Unsupported`] for an image of another geometry than the template's - /// placeholder, its length included, with [`ErrorKind::Unavailable`] for an image that is - /// not, at its path, the regular file that the check found, of the current user or an - /// administrator, and that no one else may change, and with [`ErrorKind::Unsupported`] for - /// an image that is a file of the sandbox's store; + /// store owns, and with [`ErrorKind::Io`] if the store's files cannot be listed or examined, + /// the host's evidence; + /// - as [`DistroBinding::check`] and [`CheckedDistro::geometry`] fail: with + /// [`ErrorKind::Unavailable`] for a binding that holds a verdict of another revision of the + /// distro policy than the request accepts, [`Evidence::StaleBinding`], and with + /// [`ErrorKind::Unsupported`] for one whose verdict rejects the image, + /// [`Evidence::AuthorizationDenied`]; with [`ErrorKind::Unavailable`] for an image that + /// changed, was replaced, or went since its binding was made, or whose manifest did, and for + /// an image that is not, at its path, the regular file that the check found, of the current + /// user or an administrator, and that no one else may change, + /// [`Evidence::SuppliedIdentity`], and with [`ErrorKind::Io`] if the image cannot be opened + /// at its path otherwise, or identified or examined once open, the host's evidence; with + /// [`ErrorKind::Unsupported`] for an image of another length than the template's + /// placeholder, [`Evidence::SuppliedLength`], or of other block sizes, or of a geometry that + /// OpenVMM refuses, [`Evidence::SuppliedGeometry`]; and with + /// [`ErrorKind::Unsupported`] for an image that is a file of the sandbox's store, + /// [`Evidence::InvalidSettings`]; /// - for an upper in RAM, with [`ErrorKind::Unsupported`] for a sandbox whose path holds a - /// comma or a semicolon, or whose filesystem presents another geometry than the template's - /// scratch, and with [`ErrorKind::Io`] if the sparse file cannot be created, sized, or - /// checked; + /// comma or a semicolon, [`Evidence::HostFailure`], or whose filesystem presents another + /// geometry than the template's scratch, [`Evidence::IncompatibleHost`], and with + /// [`ErrorKind::Io`] if the sparse file cannot be created, sized, or checked, the host's + /// evidence; /// - for an upper on a scratch image, with [`ErrorKind::Io`] if the image's location cannot be - /// compared with the template store's directory, or its names cannot be counted, as - /// [`ScratchBinding::check`] and [`CheckedScratch::geometry`] fail, with - /// [`ErrorKind::Unsupported`] for an image of another geometry than the template's scratch, - /// its length included, with [`ErrorKind::Unavailable`] for an image that is not, at its - /// path, the regular file that the check found, of the current user or an administrator, - /// and that no one else may change, and with [`ErrorKind::Unsupported`] for an image that - /// is a file of the sandbox's store, or that is in the template store's directory, as the - /// files of its snapshots are, or has another name. + /// compared with the template store's directory, or its names cannot be counted, the host's + /// evidence, and as [`ScratchBinding::check`] and [`CheckedScratch::geometry`] fail: with + /// [`ErrorKind::TimedOut`] and [`ErrorCode::ScratchClaimed`] for an image that another + /// launch still claims, [`Evidence::UnusableScratch`]; with [`ErrorKind::Unavailable`] for an + /// image that changed, was replaced, or went since its binding was made, or that is not, at + /// its path, the regular file that the check found, of the current user or an administrator, + /// and that no one else may change, [`Evidence::SuppliedIdentity`], and with + /// [`ErrorKind::Io`] if the image cannot be opened at its path otherwise, or identified or + /// examined once open, the host's evidence; with [`ErrorKind::Unsupported`] for an image of + /// another length than the template's scratch, [`Evidence::SuppliedLength`], or of other + /// block sizes, [`Evidence::SuppliedGeometry`]; and with [`ErrorKind::Unsupported`] for an + /// image that is a file of the sandbox's store, or that is in the template store's + /// directory, as the files of its snapshots are, or has another name, or whose path is not + /// Unicode, [`Evidence::InvalidSettings`]. /// /// [`Configuration::check`]: crate::activation::Configuration::check pub fn new( @@ -346,7 +392,7 @@ impl ClonePlan { acquired: &mut Acquired, request: &CloneRequest, deadline: Instant, - ) -> Result { + ) -> Result { Self::new_with(store, acquired, request, deadline, Access::CHECKED) } @@ -357,43 +403,78 @@ impl ClonePlan { request: &CloneRequest, deadline: Instant, access: Access<'_>, - ) -> Result { + ) -> Result { + let lease = acquired.lease.clone(); + Self::plan(store, acquired, request, deadline, access).map_err(|fault| { + let record = FailureRecord::new( + &lease, + None, + (Stage::Plan, Progress::Planned), + fault.evidence, + None, + Teardown::NoVm, + &fault.error, + ); + CloneFailure::new(fault.error, record) + }) + } + + /// Plans a clone as [`new`](Self::new) does, reaching its files through `access`, and + /// returns the fault of the check that failed. + fn plan( + store: &Store, + acquired: &mut Acquired, + request: &CloneRequest, + deadline: Instant, + access: Access<'_>, + ) -> Result { check_request(request)?; let Acquired { lease, locked } = acquired; - check_lease(store, lease, locked)?; + check_lease(store, lease, locked).attribute(Evidence::HostFailure)?; let generation = store - .index()? + .index() + .on_host()? .get(lease.generation()) .cloned() .ok_or_else(|| { - Error::new( - ErrorKind::NotFound, - format!( - "the template store {} holds no generation {}, which the lease {} names", - store.dir().display(), - lease.generation(), - lease.id() + Fault::new( + Evidence::HostFailure, + Error::new( + ErrorKind::NotFound, + format!( + "the template store {} holds no generation {}, which the lease {} \ + names", + store.dir().display(), + lease.generation(), + lease.id() + ), ), ) })?; if generation.key() != lease.key() { - return Err(Error::io(format!( - "the generation {} of the template store {} has the key {}, not the key {} that \ - the lease {} names", - generation.generation(), - store.dir().display(), - generation.key(), - lease.key(), - lease.id() - ))); + return Err(Fault::new( + Evidence::MalformedGeneration, + Error::io(format!( + "the generation {} of the template store {} has the key {}, not the key {} \ + that the lease {} names", + generation.generation(), + store.dir().display(), + generation.key(), + lease.key(), + lease.id() + )), + )); } - let key = store.read_key(&generation)?; - let snapshot = store.snapshot_dir(&generation)?; + // A store whose directory of generations is not private fails every generation: the + // host's, not the generation's. + store.check_generations_dir().on_host()?; + let key = store.read_key(&generation).map_err(generation_fault)?; + let snapshot = store.snapshot_dir(&generation).map_err(generation_fault)?; check_template(request, &key, &generation)?; let activation = record::build(request, lease, &key)?; - let owned = StoreFiles::of(locked)?; + let owned = StoreFiles::of(locked).on_host()?; let image = check_image(request, &key, &owned, access)?; - let capability = generate_capability()?; + let capability = generate_capability().on_host()?; let upper = match (request.adapter.upper().scratch_path(), &request.scratch) { (Some(path), Some(binding)) => CloneUpper::Scratch(check_scratch( path, @@ -530,41 +611,80 @@ fn devices_of(request: &CloneRequest, upper: &CloneUpper) -> SandboxPlan { /// Refuses a request that no clone takes, as [`ClonePlan::new`] describes, without reading or /// opening anything. -fn check_request(request: &CloneRequest) -> Result<()> { +fn check_request(request: &CloneRequest) -> Result<(), Fault> { let adapter = &request.adapter; if adapter.settings().is_some() { - return Err(Error::unsupported( - "a clone restores an adapter's plan, not a plan of the sandbox profile, whose \ - workload settings no template holds", + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported( + "a clone restores an adapter's plan, not a plan of the sandbox profile, whose \ + workload settings no template holds", + ), )); } let distro = adapter.distro(); let binding = &request.binding; - if bound_image(binding)? != distro.image() { - return Err(Error::unsupported(format!( - "the image {} is not the {} layer with identity {} that its distro binding names", - distro.path().display(), - binding.format(), - binding.identity() - ))); + if bound_image(binding).attribute(Evidence::InvalidFormat)? != distro.image() { + return Err(Fault::new( + Evidence::InvalidFormat, + Error::unsupported(format!( + "the image {} is not the {} layer with identity {} that its distro binding names", + distro.path().display(), + binding.format(), + binding.identity() + )), + )); } if matches!(binding.format(), DistroFormat::Gpt) { - return Err(record::gpt_refused()); + return Err(Fault::new(Evidence::InvalidFormat, record::gpt_refused())); } - check_disk_path(distro.path(), "image")?; + check_disk_path(distro.path(), "image").attribute(Evidence::InvalidSettings)?; match (adapter.upper().scratch_path(), &request.scratch) { - (Some(path), Some(_)) => check_disk_path(path, "scratch image"), + (Some(path), Some(_)) => { + check_disk_path(path, "scratch image").attribute(Evidence::InvalidSettings) + } (None, None) => Ok(()), - (Some(path), None) => Err(Error::unsupported(format!( - "the upper of a clone on the scratch image {} needs the image's scratch binding", - path.display() - ))), - (None, Some(_)) => Err(Error::unsupported( - "the upper of a clone in RAM takes no scratch binding", + (Some(path), None) => Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the upper of a clone on the scratch image {} needs the image's scratch binding", + path.display() + )), + )), + (None, Some(_)) => Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported("the upper of a clone in RAM takes no scratch binding"), )), } } +/// Returns the fault of `error`, with which the template store's read of the key or of the +/// snapshot of the leased generation failed, once the store's directory of generations passed its +/// check: the host's, which cannot restore the template, for a key of another encoding version +/// than this build reads, as [`TemplateKey::from_bytes`] refuses it with +/// [`ErrorCode::FormatUnsupported`], since a build that reads it may serve the generation; the +/// generation's, which is malformed, for a payload, a key, or a snapshot that is missing, that is +/// not private to the current user, or that is not one that the store writes, such as a key that +/// the read refuses for what it is, as [`is_refused`] tells: a link, a file that is not a regular +/// file, or one that holds more than a key may; and the host's for any other I/O error among the +/// error's sources. +fn generation_fault(error: Error) -> Fault { + if error.code() == ErrorCode::FormatUnsupported { + return Fault::new(Evidence::IncompatibleHost, error); + } + let mut source = std::error::Error::source(&error); + while let Some(cause) = source { + if let Some(io) = cause.downcast_ref::() + && io.kind() != io::ErrorKind::NotFound + && !is_refused(io) + { + return Fault::new(host_evidence(&error), error); + } + source = cause.source(); + } + Fault::new(Evidence::MalformedGeneration, error) +} + /// Refuses the path of the file of a clone's block device, which messages call `what`, that /// OpenVMM would not open as the plan checks it: a relative one, which it would resolve against the /// directory in which it runs, one with a comma or a semicolon, which its block device options @@ -612,7 +732,7 @@ fn check_template( request: &CloneRequest, key: &TemplateKey, generation: &Generation, -) -> Result<()> { +) -> Result<(), Fault> { let name = generation.generation(); // The key's contracts are those through which its clones are restored; the runtime's are // those through which this build boots and drives the guest that the template holds. @@ -622,28 +742,37 @@ fn check_template( || runtime.control_contract != contract::CONTROL_CONTRACT_REVISION || runtime.control_session != contract::CONTROL_SESSION_PROTOCOL_VERSION { - return Err(Error::unsupported(format!( - "the template's generation {name} speaks other contracts than this build, which \ - restores none of its clones" - ))); + return Err(Fault::new( + Evidence::IncompatibleHost, + Error::unsupported(format!( + "the template's generation {name} speaks other contracts than this build, which \ + restores none of its clones" + )), + )); } // A network that OpenVMM attaches no device for is no NIC, as a capture class's. let attached = match request.adapter.network() { None => false, - Some(network) => !network.openvmm_arguments()?.is_empty(), + Some(network) => !network + .openvmm_arguments() + .attribute(Evidence::InvalidSettings)? + .is_empty(), }; if attached != key.has_network() { - return Err(Error::unsupported(if attached { - format!( - "the adapter's plan of a clone attaches a network, but the template's generation \ - {name} has no NIC" - ) - } else { - format!( - "the template's generation {name} has a NIC, which its clones restore with the \ - network of the adapter's plan, but the plan has none" - ) - })); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(if attached { + format!( + "the adapter's plan of a clone attaches a network, but the template's \ + generation {name} has no NIC" + ) + } else { + format!( + "the template's generation {name} has a NIC, which its clones restore with \ + the network of the adapter's plan, but the plan has none" + ) + }), + )); } // OpenVMM restores the guest's address on the NIC from the snapshot, and a clone's launch // drops the plan's, so the plan must name the template's. @@ -651,11 +780,14 @@ fn check_template( && let Some(template) = key.guest_network() && network.guest_network() != template { - return Err(Error::unsupported(format!( - "the adapter's plan of a clone gives its guest the network {}, but the template's \ - generation {name} restores its NIC on {template}, from its snapshot", - network.guest_network() - ))); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the adapter's plan of a clone gives its guest the network {}, but the template's \ + generation {name} restores its NIC on {template}, from its snapshot", + network.guest_network() + )), + )); } let memory = request.memory; let block = key.memory_block_size(); @@ -664,15 +796,18 @@ fn check_template( || !memory.is_multiple_of(MEMORY_BLOCK_SIZE) || !memory.is_multiple_of(block) { - return Err(Error::unsupported(format!( - "a clone of the template's generation {name} cannot restore with {memory} bytes of \ - RAM: its memory target must be whole blocks of {} MiB, and of the template's memory \ - blocks of {block} bytes, from the template's {} bytes of RAM to its capacity of {} \ - bytes", - MEMORY_BLOCK_SIZE / MIB, - key.memory(), - key.memory_capacity() - ))); + return Err(Fault::new( + Evidence::InvalidTargets, + Error::unsupported(format!( + "a clone of the template's generation {name} cannot restore with {memory} bytes \ + of RAM: its memory target must be whole blocks of {} MiB, and of the template's \ + memory blocks of {block} bytes, from the template's {} bytes of RAM to its \ + capacity of {} bytes", + MEMORY_BLOCK_SIZE / MIB, + key.memory(), + key.memory_capacity() + )), + )); } Ok(()) } @@ -686,29 +821,54 @@ fn check_image( key: &TemplateKey, owned: &StoreFiles, access: Access<'_>, -) -> Result { +) -> Result { let path = request.adapter.distro().path(); let shown = path.display(); - let refused = |problem: String| Error::unavailable(format!("the image {shown} {problem}")); - let text = path - .to_str() - .ok_or_else(|| refused("has a path that is not valid Unicode".to_owned()))?; + let text = path.to_str().ok_or_else(|| { + Fault::new( + Evidence::InvalidSettings, + Error::unavailable(format!( + "the image {shown} has a path that is not valid Unicode" + )), + ) + })?; let binding = &request.binding; let layer = match binding.format() { DistroFormat::Erofs => SandboxLayer::new(LayerRole::Distro, text, binding.identity()), DistroFormat::Gpt => SandboxLayer::gpt(LayerRole::Distro, text, binding.identity()), }; - let checked = binding.check_with(&layer, request.policy_revision, access.distro)?; + let policy = binding.policy(); + let checked = binding + .check_with(&layer, request.policy_revision, access.distro) + .map_err(|error| { + // The check refuses a stale binding, then one whose verdict rejects the image, before + // it opens anything. + let evidence = if binding.version() != distro::BINDING_VERSION + || policy.revision != request.policy_revision + { + Evidence::StaleBinding + } else if policy.result != PolicyResult::Accepted { + Evidence::AuthorizationDenied + } else { + binding_evidence(&error) + }; + Fault::new(evidence, error) + })?; + let geometry = checked.geometry().map_err(supplied_geometry_fault)?; check_geometry( - checked.geometry()?, + geometry, key.distro(), &format!("the image {shown}"), "placeholder", + Mismatch::SUPPLIED, )?; - let identity = identity_of(checked.file()) - .map_err(|error| refused(format!("cannot be identified: {error}")))?; - check_at_path(path, identity, refused)?; - owned.check(identity, "image", path)?; + let identity = identity_of(checked.file()).map_err(|error| { + Fault::host(Error::io(format!("the image {shown} cannot be identified")).with_source(error)) + })?; + check_at_path(path, identity, &format!("the image {shown}"))?; + owned + .check(identity, "image", path) + .attribute(Evidence::InvalidSettings)?; Ok(checked) } @@ -725,21 +885,38 @@ fn check_scratch( (owned, store): (&StoreFiles, &Store), deadline: Instant, access: Access<'_>, -) -> Result { +) -> Result { let shown = path.display(); - let checked = binding.check_with(path, deadline, access.scratch)?; + let checked = binding + .check_with(path, deadline, access.scratch) + .map_err(|error| { + let evidence = if binding.version() != scratch::SCRATCH_BINDING_VERSION { + Evidence::StaleBinding + } else if error.code() == ErrorCode::ScratchClaimed { + // Another launch holds the image, which is not fresh for this clone. + Evidence::UnusableScratch + } else { + binding_evidence(&error) + }; + Fault::new(evidence, error) + })?; + let geometry = checked.geometry().map_err(supplied_geometry_fault)?; check_geometry( - checked.geometry()?, + geometry, key.scratch(), &format!("the scratch image {shown}"), "scratch", + Mismatch::SUPPLIED, )?; - let refused = - |problem: String| Error::unavailable(format!("the scratch image {shown} {problem}")); - let identity = identity_of(checked.file()) - .map_err(|error| refused(format!("cannot be identified: {error}")))?; - check_at_path(path, identity, refused)?; - owned.check(identity, "scratch image", path)?; + let identity = identity_of(checked.file()).map_err(|error| { + Fault::host( + Error::io(format!("the scratch image {shown} cannot be identified")).with_source(error), + ) + })?; + check_at_path(path, identity, &format!("the scratch image {shown}"))?; + owned + .check(identity, "scratch image", path) + .attribute(Evidence::InvalidSettings)?; check_outside_store(&checked, store, path)?; check_single_name(&checked, path)?; Ok(checked) @@ -754,26 +931,33 @@ fn check_scratch( /// file into the store's directory, so one outside it stays outside it. The location resolves the /// aliases that the file system makes, links and junctions on the way and directories held open, /// but not a mount or a network share that exposes the store's directories elsewhere, which -/// [`ClonePlan`] leaves to the system's administration. -fn check_outside_store(checked: &CheckedScratch, store: &Store, path: &Path) -> Result<()> { +/// [`ClonePlan`] leaves to the system's administration. The request supplied the image, so its +/// refusal is invalid settings, while the failure to locate it is the host's. +fn check_outside_store(checked: &CheckedScratch, store: &Store, path: &Path) -> Result<(), Fault> { let failed = |error: io::Error| { - Error::io(format!( - "cannot compare the location of the scratch image {} with the template store in {}", - path.display(), - store.dir().display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot compare the location of the scratch image {} with the template store in \ + {}", + path.display(), + store.dir().display() + )) + .with_source(error), + ) }; let location = location_of(checked.file()).map_err(failed)?; if is_within(&location, store.dir()).map_err(failed)? { - return Err(Error::unsupported(format!( - "the scratch image {} is {}, in the directory of the template store {}, where its \ - captures write the snapshots that every clone of a generation restores in place: a \ - clone's scratch image must be outside it", - path.display(), - location.display(), - store.dir().display() - ))); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the scratch image {} is {}, in the directory of the template store {}, where its \ + captures write the snapshots that every clone of a generation restores in \ + place: a clone's scratch image must be outside it", + path.display(), + location.display(), + store.dir().display() + )), + )); } Ok(()) } @@ -825,26 +1009,54 @@ fn location_of(file: &File) -> io::Result { /// Refuses the clone's scratch image at `path`, which `checked` holds open, unless it has a single /// name: another name, a hard link, could make it a file of a snapshot that the clone's guest then /// writes, of one of the store's generations or captures, or of another store, which the image's -/// location does not show. -fn check_single_name(checked: &CheckedScratch, path: &Path) -> Result<()> { +/// location does not show. The request supplied the image, so its refusal is invalid settings, +/// while the failure to count its names is the host's. +fn check_single_name(checked: &CheckedScratch, path: &Path) -> Result<(), Fault> { let links = platform::link_count(checked.file()).map_err(|error| { - Error::io(format!( - "cannot count the names of the scratch image {}", - path.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot count the names of the scratch image {}", + path.display() + )) + .with_source(error), + ) })?; if links > 1 { - return Err(Error::unsupported(format!( - "the scratch image {} has {links} names, but a clone's scratch image must have a \ - single name: another, a hard link, could make it a file of a snapshot, which every \ - clone of its generation restores in place", - path.display() - ))); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the scratch image {} has {links} names, but a clone's scratch image must have a \ + single name: another, a hard link, could make it a file of a snapshot, which \ + every clone of its generation restores in place", + path.display() + )), + )); } Ok(()) } +/// Returns the evidence of `error`, with which the check of the binding of a file that the request +/// supplied failed, past the binding's own version and verdict: a file that changed, was replaced, +/// or went since its binding was made, a supplied file of the wrong identity; a path that the +/// check refuses, invalid settings; and any other failure, the host's. +fn binding_evidence(error: &Error) -> Evidence { + match error.kind() { + ErrorKind::Unavailable => Evidence::SuppliedIdentity, + ErrorKind::Unsupported => Evidence::InvalidSettings, + _ => host_evidence(error), + } +} + +/// Returns the fault of `error`, with which the geometry of a file that the request supplied +/// cannot be read: a geometry that OpenVMM refuses, a supplied file of the wrong geometry, and any +/// other failure, the host's. +fn supplied_geometry_fault(error: Error) -> Fault { + match error.kind() { + ErrorKind::Unsupported => Fault::new(Evidence::SuppliedGeometry, error), + _ => Fault::host(error), + } +} + /// The regular files that the store of a sandbox owns, with their identities, which no image or /// scratch image of a clone may be. struct StoreFiles(Vec<(PathBuf, (u64, u128))>); @@ -928,34 +1140,53 @@ impl StoreFiles { /// the capture of a template checks its own, as [`ClonePlan`] describes. Returns its path, once /// the file is closed. It removes the file if any step fails, its creation included, which can /// fail once it made the file, as a file that cannot be made sparse does. -fn create_scratch(locked: &LockedStore, key: &TemplateKey, access: Access<'_>) -> Result { +fn create_scratch( + locked: &LockedStore, + key: &TemplateKey, + access: Access<'_>, +) -> Result { let path = locked.clone_scratch_path(); let shown = path.display(); - reject_disk_path(&path, &shown.to_string())?; + reject_disk_path(&path, &shown.to_string()).attribute(Evidence::HostFailure)?; let failed = |error: io::Error| { - Error::io(format!("cannot create the {SPARSE_WHAT} {shown}")).with_source(error) + Fault::host( + Error::io(format!("cannot create the {SPARSE_WHAT} {shown}")).with_source(error), + ) }; let created = (|| { let file = (access.create)(&path).map_err(&failed)?; let scratch = key.scratch(); file.set_len(scratch.length).map_err(&failed)?; + let geometry = + block_geometry(&file, &path, SPARSE_WHAT).map_err(|error| match error.kind() { + // The sandbox's filesystem presents a geometry that OpenVMM refuses. + ErrorKind::Unsupported => Fault::new(Evidence::IncompatibleHost, error), + _ => Fault::host(error), + })?; check_geometry( - block_geometry(&file, &path, SPARSE_WHAT)?, + geometry, scratch, &format!("the {SPARSE_WHAT} {shown}, in the sandbox's filesystem,"), "scratch", + Mismatch::HOST, )?; let identity = identity_of(&file).map_err(&failed)?; drop(file); - check_at_path(&path, identity, |problem| { - Error::io(format!("the {SPARSE_WHAT} {shown} {problem}")) + // The host made the file, so what its path shows of it is the host's. + check_at_path(&path, identity, &format!("the {SPARSE_WHAT} {shown}")).map_err(|fault| { + let (_, message, source) = fault.error.into_parts(); + let error = Error::io(message); + Fault::host(match source { + Some(source) => error.with_source(source), + None => error, + }) }) })(); match created { Ok(()) => Ok(path), - Err(error) => { + Err(fault) => { let _ = remove_if_present(&path); - Err(error) + Err(fault) } } } @@ -973,54 +1204,119 @@ fn create_sparse(path: &Path) -> io::Result { Ok(file) } +/// The evidence of a file of a clone's block device whose geometry is not the one that the +/// template's device presented. +#[derive(Clone, Copy)] +struct Mismatch { + /// The evidence of another length. + length: Evidence, + /// The evidence of other block sizes. + blocks: Evidence, +} + +impl Mismatch { + /// A file that the request supplied. + const SUPPLIED: Self = Self { + length: Evidence::SuppliedLength, + blocks: Evidence::SuppliedGeometry, + }; + + /// The sparse file of an upper in RAM, which the plan sizes as the template's scratch, and + /// whose block sizes the sandbox's filesystem presents: the host's. + const HOST: Self = Self { + length: Evidence::IncompatibleHost, + blocks: Evidence::IncompatibleHost, + }; +} + /// Refuses `found`, the geometry of the file of a clone's block device, which messages call -/// `what`, unless it is `expected`, the geometry that the template's `device` presented. +/// `what`, unless it is `expected`, the geometry that the template's `device` presented, with the +/// evidence that `mismatch` gives. fn check_geometry( found: BlockGeometry, expected: BlockGeometry, what: &str, device: &str, -) -> Result<()> { + mismatch: Mismatch, +) -> Result<(), Fault> { if found.length != expected.length { - return Err(Error::unsupported(format!( - "{what} is {} bytes long, not the {} bytes of the template's {device}", - found.length, expected.length - ))); + return Err(Fault::new( + mismatch.length, + Error::unsupported(format!( + "{what} is {} bytes long, not the {} bytes of the template's {device}", + found.length, expected.length + )), + )); } if found != expected { - return Err(Error::unsupported(format!( - "{what} presents logical blocks of {} bytes and physical blocks of {} bytes, but the \ - template's {device} presented {} and {} bytes", - found.logical_block_size, - found.physical_block_size, - expected.logical_block_size, - expected.physical_block_size - ))); + return Err(Fault::new( + mismatch.blocks, + Error::unsupported(format!( + "{what} presents logical blocks of {} bytes and physical blocks of {} bytes, but \ + the template's {device} presented {} and {} bytes", + found.logical_block_size, + found.physical_block_size, + expected.logical_block_size, + expected.physical_block_size + )), + )); } Ok(()) } -/// Checks the file at `path` as OpenVMM will open it, by its path: a regular file, not a link, -/// nor on Windows any other reparse point, whose identity is `expected`, the identity of the file -/// that a check found, and that no one but the current user, or an administrator, may change. -/// Fails with the error that `refused` makes of the problem that it found. -fn check_at_path( - path: &Path, - expected: (u64, u128), - refused: impl Fn(String) -> Error, -) -> Result<()> { - let file = open_regular(path).map_err(|error| refused(format!("cannot be used: {error}")))?; - let identity = - identity_of(&file).map_err(|error| refused(format!("cannot be identified: {error}")))?; +/// Checks the file at `path`, which messages call `what`, as OpenVMM will open it, by its path: a +/// regular file, not a link, nor on Windows any other reparse point, whose identity is +/// `expected`, the identity of the file that a check found, and that no one but the current user, +/// or an administrator, may change. +/// +/// Fails with [`ErrorKind::Unavailable`] and [`Evidence::SuppliedIdentity`] for a path that no +/// longer names that file: one whose file went, a component of which is no longer a directory, +/// that the current user may no longer open, or that names a link, a file that is not regular, or +/// another file, and for a file that others may change; and with [`ErrorKind::Io`] and the host's +/// evidence if the file cannot be opened at its path otherwise, as when the host runs out of +/// descriptors, or if the file that it opened cannot be identified or examined. +fn check_at_path(path: &Path, expected: (u64, u128), what: &str) -> Result<(), Fault> { + let refused = |problem: String| { + Fault::new( + Evidence::SuppliedIdentity, + Error::unavailable(format!("{what} {problem}")), + ) + }; + let failed = |action: &str, error: io::Error| { + Fault::host(Error::io(format!("{what} cannot be {action}")).with_source(error)) + }; + let file = open_regular(path).map_err(|error| { + if path_refused(&error) { + refused(format!("cannot be used: {error}")) + } else { + failed("opened", error) + } + })?; + let identity = identity_of(&file).map_err(|error| failed("identified", error))?; if identity != expected { return Err(refused( "changed while it was checked: its path names another file".to_owned(), )); } - if let Some(problem) = platform::foreign_writer_problem(&file) - .map_err(|error| refused(format!("cannot be checked: {error}")))? + if let Some(problem) = + platform::foreign_writer_problem(&file).map_err(|error| failed("checked", error))? { return Err(refused(problem)); } Ok(()) } + +/// Returns whether `error`, with which [`open_regular`] failed to open a file that a request +/// supplied at its path, says that the path no longer names the file that a check found: the +/// file went, a component of the path is no longer a directory, the current user may no longer +/// open it, or the open refused a link or a file that is not regular, a failure that the system +/// does not report. Any other failure is the host's. +fn path_refused(error: &io::Error) -> bool { + error.raw_os_error().is_none() + || matches!( + error.kind(), + io::ErrorKind::NotFound + | io::ErrorKind::NotADirectory + | io::ErrorKind::PermissionDenied + ) +} diff --git a/nvx-rs/src/host/template/clone/failure.rs b/nvx-rs/src/host/template/clone/failure.rs new file mode 100644 index 000000000..1a5bb193e --- /dev/null +++ b/nvx-rs/src/host/template/clone/failure.rs @@ -0,0 +1,1231 @@ +//! The failures of clones, [`CloneFailure`]: the [`Evidence`] of a failure of a clone's plan, +//! launch, restore sequence, or reattach, the [`Outcome`] to which the evidence attributes it, as +//! [the attribution of evidence](Evidence#attribution) describes, the [`Stage`] at which the clone +//! failed, the clone's [`Progress`] by then, and the [`Teardown`] that followed, which the +//! failure's [`FailureRecord`] keeps, and the clone's sandbox records, as [the records of +//! failures](FailureRecord#records) describe. + +use std::fmt; +use std::io; +use std::time::Instant; + +use serde::{Deserialize, Serialize}; + +use crate::activation::{TemplateReference, TransactionId}; +use crate::error::{Error, ErrorCode, ErrorKind, Result}; +use crate::host::report::Report; +use crate::host::store::{LaunchId, Recovery}; +use crate::host::template::{Lease, LeaseId, Store, StoreId}; +use crate::protocol::status; + +/// What the host saw of a failure of a clone, which attributes the failure to an [`Outcome`]. +/// +/// # Attribution +/// +/// The outcome of a failure follows from its evidence alone, as [`outcome`](Self::outcome) gives +/// it: +/// +/// | Evidence | Outcome | Quarantine | +/// | --- | --- | --- | +/// | An authorization that the image's binding denies; a stale image binding; invalid targets, format, settings, Activate, or mappings; an image that the template can't serve; a scratch that isn't ext4 or isn't fresh; a root that does not assemble from them | [`Request`](Outcome::Request) | No | +/// | A supplied file with the wrong identity, length, or geometry | [`Request`](Outcome::Request), or the host's [`Staging`](Outcome::Staging) once the caller proves it | No | +/// | An endpoint or forward that cannot be bound; resource exhaustion; a destination that cannot restore the template, such as an incompatible CPU or TSC; another runtime; another failure of the host's own operations | [`Host`](Outcome::Host) | No | +/// | A lost status; a timeout; a reset; a forced kill; a broken protocol | [`Unknown`](Outcome::Unknown): the host, the protocol, or neither | No | +/// | A malformed or inconsistent generation; a diagnosed capture or repair defect attributable to it | [`Template`](Outcome::Template) | Yes | +/// | A repair status alone, 193, 194, and 195 included | [`Repair`](Outcome::Repair): the repair stage, whose provenance is still to establish | No | +/// +/// Only a failure of the template quarantines the template's generation, through the generation's +/// tombstone, as [`CloneFailure::quarantine`] does: no other evidence shows that the generation, +/// rather than the request, the host, or the session, failed the clone, and a quarantine changes +/// nothing of the generation's payload or snapshot, which its other clones keep restoring. +/// +/// The host sees the guest of a clone through its control session, and through the status with +/// which its VM ends, which OpenVMM's outcome report gives: a clone's restore discards the +/// guest's console, so no diagnostic tells where a status came from. [`status_evidence`] +/// attributes a status alone, as [`nvx::protocol::status`](crate::protocol::status) classes it: +/// +/// - a status of the time ABI or of a clone's repair is evidence of the repair stage, at any point, +/// since the repair's deferred work may end a clone that is ready, and its provenance is still to +/// establish, since a workload may end its VM with any status; +/// - a status of a clone's activation is evidence of its class while no workload can have run, +/// before the start of the clone's sandbox was sent, and none after; +/// - no other status is evidence. +/// +/// A status that is evidence prevails over what the host saw of the same end: the loss of the +/// session, the passing of a deadline, or the VMM's exit only show that the clone ended. +/// +/// The host sees the failures of OpenVMM itself through the line with which OpenVMM reports the +/// error that ended it, which the launch's log keeps, and which leads with the error's time ABI +/// code, a spelling that OpenVMM keeps stable for its callers. A VMM that exits before its +/// restore-ready event with a code that says that the host cannot present the template's CPU +/// profile, TSC, or LAPIC timer failed on an [incompatible host](Self::IncompatibleHost). Any +/// other code, such as one of the snapshot's time contract, keeps what the host saw, since it does +/// not tell a defect of the generation from one of this host's OpenVMM, and the failure's message +/// keeps the line. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Evidence { + /// The verdict of the distro policy that the binding of the clone's image holds rejects the + /// image: the request may not run it. + AuthorizationDenied, + /// The binding of the clone's image is stale: it holds a verdict of another revision of the + /// distro policy than the caller accepts, or it is of another version than this build checks, + /// or the image's manifest changed after it was made. + StaleBinding, + /// The request's targets are invalid: a memory target that is not whole blocks of the + /// template's, from its RAM to its capacity, or resources that the repaired clone does not + /// have, as the guest's + /// [`ACTIVATION_RESOURCES`](crate::protocol::status::ACTIVATION_RESOURCES) tells. + InvalidTargets, + /// The request's image is of a format that no clone restores, a GPT disk, or is not of the + /// format and identity that its binding names. + InvalidFormat, + /// The request's settings are invalid: a plan that no clone takes, a path that OpenVMM would + /// not open as checked, a network that the template's NIC does not match, an image or a + /// scratch image that is a file of the sandbox's store, or settings that the guest cannot + /// apply, as its [`ACTIVATION_WORKLOAD`](crate::protocol::status::ACTIVATION_WORKLOAD) and + /// [`ACTIVATION_NETWORK`](crate::protocol::status::ACTIVATION_NETWORK) tell. + InvalidSettings, + /// The request's Activate is invalid: settings that no activation record holds, or a record + /// that the guest refused, as its + /// [`ACTIVATION_RECORD`](crate::protocol::status::ACTIVATION_RECORD) tells. + InvalidActivate, + /// The request's mappings are invalid: the guest cannot mount the host export that they map, + /// as its [`ACTIVATION_EXPORT`](crate::protocol::status::ACTIVATION_EXPORT) tells. + InvalidMappings, + /// The request's image is one that the template cannot serve, as a diagnostic tells. No + /// clone's restore keeps the guest's diagnostics yet, so the guest's status alone, + /// [`UnusableRoot`](Self::UnusableRoot), tells of such an image. + UnservableImage, + /// The request's scratch image is not ext4, or not fresh: another launch holds it, or, as a + /// diagnostic tells, the guest found a filesystem on it. No clone's restore keeps the guest's + /// diagnostics yet, so the guest's status alone, [`UnusableRoot`](Self::UnusableRoot), tells + /// of a filesystem on it. + UnusableScratch, + /// The clone's root does not assemble from the request's image over its upper, as the guest's + /// [`ACTIVATION_ROOT`](crate::protocol::status::ACTIVATION_ROOT) tells: an image that the + /// template cannot serve, or a scratch that is not ext4 or not fresh, which the status alone + /// does not tell apart. + UnusableRoot, + /// A file that the request supplied is not the one that its binding names: it was replaced, + /// changed, or removed after its binding was made, or its path names another file than the + /// check found, or one that others may change. + SuppliedIdentity, + /// A file that the request supplied has another length than the template's device. + SuppliedLength, + /// A file that the request supplied presents other block sizes than the template's device, or + /// a geometry that OpenVMM refuses. + SuppliedGeometry, + /// An endpoint of the clone's launch cannot be bound: the endpoint of the VMM's restore-ready + /// event, or a control socket whose path a Unix socket's address, or OpenVMM's option + /// `listen=`, cannot hold. + EndpointBind, + /// A forward of the clone's network cannot be bound. No clone's network forwards a port yet. + ForwardBind, + /// The host ran out of a resource, such as its storage, its memory, or its descriptors. + ResourceExhaustion, + /// The host cannot restore the template: this build speaks other contracts than the template, + /// or reads no key of the template's encoding version, or the sandbox's filesystem presents + /// another geometry than the template's scratch, or OpenVMM refused to restore the snapshot + /// with a time ABI code that says that the host cannot present the template's CPU profile, + /// TSC, or LAPIC timer, as [the attribution](Self#attribution) describes. + IncompatibleHost, + /// The clone runs, or would launch, under another runtime identity than the template's or the + /// caller's, which the clone's guest might not speak. + RuntimeMismatch, + /// Another failure of the host's own operations: an I/O error of its files, a store whose + /// records are inconsistent, a clock that cannot hold a deadline, or a caller that drives the + /// clone through another lease or lock than the clone's, or launches it again once it failed. + HostFailure, + /// The clone's status was lost: its VMM ended without an outcome report that tells why, as one + /// that is killed does, or the caller of its launch crashed. + LostStatus, + /// A deadline of the clone's restore passed. + Timeout, + /// The adapter lost the clone's session: its reset, the end of its stream, or a failure of its + /// transport. + Reset, + /// The host's recovery terminated the clone's VMM, whose launch's caller had crashed. + ForcedKill, + /// The clone's session broke its protocol: another process at one of the clone's endpoints, + /// other bytes than OpenVMM's restore-ready event, a refusal of the adapter's framing, a Ready + /// that does not report the record's activation, or a session that broke the order of the + /// activation, as the guest's + /// [`ACTIVATION_PROTOCOL`](crate::protocol::status::ACTIVATION_PROTOCOL) and + /// [`ACTIVATION_ORDER`](crate::protocol::status::ACTIVATION_ORDER) tell. + Protocol, + /// The leased generation is malformed or inconsistent: the store indexes it with another key + /// than the lease names, its key or snapshot is malformed, its key names its guest's DNS + /// server in a form that OpenVMM does not write, or its payload is not a directory of the + /// current user with mode 0700. + MalformedGeneration, + /// A defect of the generation's capture or repair, which a diagnostic attributes to the + /// generation. No clone's restore keeps the guest's diagnostics yet, so a status of the repair + /// alone is [`RepairStatus`](Self::RepairStatus), and no time ABI code of OpenVMM's tells a + /// defect of the generation from one of the host's OpenVMM. + TemplateDefect, + /// A status of the time ABI or of a clone's repair, alone, which + /// [`FailureRecord::status`] holds. + RepairStatus, +} + +impl Evidence { + /// Returns the outcome to which the evidence attributes a failure, as [the + /// attribution](Self#attribution) describes. + pub const fn outcome(self) -> Outcome { + match self { + Self::AuthorizationDenied + | Self::StaleBinding + | Self::InvalidTargets + | Self::InvalidFormat + | Self::InvalidSettings + | Self::InvalidActivate + | Self::InvalidMappings + | Self::UnservableImage + | Self::UnusableScratch + | Self::UnusableRoot + | Self::SuppliedIdentity + | Self::SuppliedLength + | Self::SuppliedGeometry => Outcome::Request, + Self::EndpointBind + | Self::ForwardBind + | Self::ResourceExhaustion + | Self::IncompatibleHost + | Self::RuntimeMismatch + | Self::HostFailure => Outcome::Host, + Self::LostStatus | Self::Timeout | Self::Reset | Self::ForcedKill | Self::Protocol => { + Outcome::Unknown + } + Self::MalformedGeneration | Self::TemplateDefect => Outcome::Template, + Self::RepairStatus => Outcome::Repair, + } + } + + /// Returns whether the evidence is of a file that the request supplied, which the host's + /// staging may have made wrong, as [`CloneFailure::with_staging_proven`] attributes it once + /// the caller proves it. + pub const fn is_supplied_file(self) -> bool { + matches!( + self, + Self::SuppliedIdentity | Self::SuppliedLength | Self::SuppliedGeometry + ) + } + + /// Returns the evidence's name, in lowercase words joined by hyphens, as its record spells it. + pub const fn as_str(self) -> &'static str { + match self { + Self::AuthorizationDenied => "authorization-denied", + Self::StaleBinding => "stale-binding", + Self::InvalidTargets => "invalid-targets", + Self::InvalidFormat => "invalid-format", + Self::InvalidSettings => "invalid-settings", + Self::InvalidActivate => "invalid-activate", + Self::InvalidMappings => "invalid-mappings", + Self::UnservableImage => "unservable-image", + Self::UnusableScratch => "unusable-scratch", + Self::UnusableRoot => "unusable-root", + Self::SuppliedIdentity => "supplied-identity", + Self::SuppliedLength => "supplied-length", + Self::SuppliedGeometry => "supplied-geometry", + Self::EndpointBind => "endpoint-bind", + Self::ForwardBind => "forward-bind", + Self::ResourceExhaustion => "resource-exhaustion", + Self::IncompatibleHost => "incompatible-host", + Self::RuntimeMismatch => "runtime-mismatch", + Self::HostFailure => "host-failure", + Self::LostStatus => "lost-status", + Self::Timeout => "timeout", + Self::Reset => "reset", + Self::ForcedKill => "forced-kill", + Self::Protocol => "protocol", + Self::MalformedGeneration => "malformed-generation", + Self::TemplateDefect => "template-defect", + Self::RepairStatus => "repair-status", + } + } +} + +impl fmt::Display for Evidence { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// What a failure of a clone is attributed to, by its [`Evidence`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Outcome { + /// The request: its image, upper, targets, or settings. + Request, + /// The host's staging of a file that the request supplied, which the caller proved. + Staging, + /// The host. + Host, + /// The host, the protocol, or neither: the evidence shows no provenance. + Unknown, + /// The leased generation of the template, which [`CloneFailure::quarantine`] quarantines. + Template, + /// The repair stage of the clone, whose provenance is still to establish. + Repair, +} + +impl Outcome { + /// Returns whether a failure of this outcome quarantines the leased generation, through its + /// tombstone: only a failure of the [template](Self::Template) does. + pub const fn quarantines(self) -> bool { + matches!(self, Self::Template) + } + + /// Returns the outcome's name, in lowercase, as its record spells it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Request => "request", + Self::Staging => "staging", + Self::Host => "host", + Self::Unknown => "unknown", + Self::Template => "template", + Self::Repair => "repair", + } + } +} + +impl fmt::Display for Outcome { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// The stage of a clone at which it failed. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Stage { + /// The clone's plan, [`ClonePlan::new`](super::ClonePlan::new), before anything of its launch. + Plan, + /// The clone's launch, [`ClonePlan::launch`](super::ClonePlan::launch), until its VMM runs. + Launch, + /// OpenVMM's restore, from the spawn of the VMM until its restore-ready event. + Restore, + /// The clone's repair and activation, from the restore-ready event until the check of the + /// clone's Ready. + Activation, + /// The start of the clone's sandbox, from the check of its Ready until the start's + /// confirmation. + Start, + /// The commit of the clone's launch. + Commit, + /// The run of the clone's VM, from the commit of its launch until the VM exits. + Run, + /// A reattach's check of the clone, [`reattach_clone`](super::reattach_clone). + Reattach, +} + +impl Stage { + /// Returns the stage's name, in lowercase, as its record spells it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Plan => "plan", + Self::Launch => "launch", + Self::Restore => "restore", + Self::Activation => "activation", + Self::Start => "start", + Self::Commit => "commit", + Self::Run => "run", + Self::Reattach => "reattach", + } + } +} + +impl fmt::Display for Stage { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// How far a clone got, in the order of its steps: the latest step that it reached. +/// +/// The steps from the restore-ready event to the start's confirmation are those of the clone's +/// restore sequence, which its progress log, +/// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), records as each is reached; +/// the store's records tell the others: the launch's journal records the spawn of its VMM, and the +/// runtime record its commit. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Progress { + /// The clone's plan: nothing of its launch exists. + Planned, + /// The launch wrote the clone's files, and may have begun its journal: no VMM was spawned. + Launching, + /// The launch's VMM was spawned, and its journal records it. + Spawned, + /// The VMM's restore-ready event was read: the restored VM runs. + Event, + /// A session of the adapter's framing attached: the guest's repair acknowledged the restore, + /// before which OpenVMM reads none of the host's input. + Acknowledged, + /// The Activate of the clone's record was sent. + Activating, + /// The clone's Ready was checked: the clone is activated, with the record's transaction. + Ready, + /// The start of the clone's sandbox was sent. + Starting, + /// The start was confirmed: the clone's sandbox started. + Started, + /// The launch committed: the runtime record names the clone's VM. + Committed, +} + +impl Progress { + /// Returns the stage of a clone that fails once it got this far: the stage of the step that + /// follows. + pub const fn stage(self) -> Stage { + match self { + Self::Planned => Stage::Plan, + Self::Launching => Stage::Launch, + Self::Spawned => Stage::Restore, + Self::Event | Self::Acknowledged | Self::Activating => Stage::Activation, + Self::Ready | Self::Starting => Stage::Start, + Self::Started => Stage::Commit, + Self::Committed => Stage::Run, + } + } + + /// Returns the step's name, in lowercase, as its records spell it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Planned => "planned", + Self::Launching => "launching", + Self::Spawned => "spawned", + Self::Event => "event", + Self::Acknowledged => "acknowledged", + Self::Activating => "activating", + Self::Ready => "ready", + Self::Starting => "starting", + Self::Started => "started", + Self::Committed => "committed", + } + } + + /// Returns whether a workload may have run in the clone once it got this far: once the start + /// of its sandbox was sent. + pub const fn may_have_run(self) -> bool { + matches!(self, Self::Starting | Self::Started | Self::Committed) + } +} + +impl fmt::Display for Progress { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// How the teardown that follows a failure of a clone ended. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Teardown { + /// The clone failed before its VMM was spawned: no VM runs. + NoVm, + /// No VMM of the clone runs: the VMM that the launch spawned has exited, as its verified + /// handle proved, or a launch's claim on its log, which no process holds, proved for the + /// launch, whether it spawned a VMM or not. The store's records of the VMM are cleared. + Exited, + /// The teardown terminated the clone's VMM through its verified identity, or through its + /// handle, a VMM that the launch could not record, and confirmed its exit, and the store's + /// records of it are cleared. + Terminated, + /// Whether the clone's VMM still runs cannot be determined, for the reason that this gives: + /// its identity or its exit is indeterminate. The launch's journal or the runtime record + /// remains, and with it the clone's lease, which a release refuses to end until a reattach + /// resolves the teardown. + Unresolved(String), + /// The teardown had not ended when the failure was recorded: the record that a failure writes + /// before its teardown, so that a crash of the caller during the teardown leaves it, which the + /// record that the failure writes once its teardown ended replaces. A reattach that finds it + /// finishes the failure, as [`reattach_clone`](super::reattach_clone) describes, so no + /// [`CloneFailure`] that a step of a clone returns has it. + Pending, +} + +impl Teardown { + /// Returns whether the teardown is resolved: whether no VM of the clone runs. + pub fn is_resolved(&self) -> bool { + !matches!(self, Self::Unresolved(_) | Self::Pending) + } + + /// Returns the teardown of a launch that [`Starting::abandon`](crate::host::supervisor::Starting::abandon), + /// or a recovery, ended as `ended` tells. + pub(super) fn of(ended: Result) -> Self { + match ended { + Ok(Recovery::Clean | Recovery::Exited) => Self::Exited, + Ok(Recovery::Terminated(_)) => Self::Terminated, + Ok(Recovery::Indeterminate(reason)) => Self::Unresolved(reason), + Err(error) => Self::Unresolved(error.to_string()), + } + } + + /// Returns the teardown's name, in lowercase words joined by hyphens, as its record spells it. + pub const fn as_str(&self) -> &'static str { + match self { + Self::NoVm => "no-vm", + Self::Exited => "exited", + Self::Terminated => "terminated", + Self::Unresolved(_) => "unresolved", + Self::Pending => "pending", + } + } +} + +/// The record of a failure of a clone: what the failure was about, where and how it failed, and +/// how its teardown ended, as [`CloneFailure::record`] returns it. +/// +/// # Records +/// +/// A failure from the clone's launch on is recorded in the clone's sandbox, as +/// [`CLONE_FAILURE_NAME`](crate::host::store::CLONE_FAILURE_NAME): before the teardown that +/// follows it, whose record is [pending](Teardown::Pending) until then, so that a crash of the +/// caller during the teardown leaves the record, which a reattach finishes, and once its teardown +/// is known, so that the record holds the stage, the evidence and the status, the original +/// outcome, the transaction of the clone's restore, the leased generation, the clone's progress, +/// and the teardown's outcome. A failure of the plan, before the launch, leaves no VM and no +/// record: the caller holds the failure, and releases the lease. A launch keeps its lease, and +/// with it the leased generation, until its VM has exited, so a failure that comes after the +/// clone's Ready, such as one of the repair's deferred work, is classified against that +/// generation all the same, and so is the status with which the guest ended a VM whose exit a +/// reattach confirms only later. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct FailureRecord { + /// The identity of the template store that leased the clone's generation. + pub store: StoreId, + /// The identity of the clone's lease. + pub lease: LeaseId, + /// The leased generation, and the identity of its key. + pub template: TemplateReference, + /// The launch that the lease reserves. + pub launch: LaunchId, + /// The transaction ID of the clone's restore, once its plan drew it. + pub transaction: Option, + /// The stage at which the clone failed. + pub stage: Stage, + /// How far the clone got: its acknowledgment and the progress of its activation included. + pub progress: Progress, + /// The evidence of the failure. + pub evidence: Evidence, + /// The status with which the guest ended the clone's VM, as OpenVMM's outcome report gives + /// it, if the report has one. + pub status: Option, + /// The original outcome of the failure, to which its evidence attributes it, whatever the + /// teardown that followed it, and the status with which the guest ended the VM, once the VMM + /// exited. + pub outcome: Outcome, + /// How the teardown that followed the failure ended. + pub teardown: Teardown, + /// The code of the failure's error. + pub code: ErrorCode, + /// The message of the failure's error, followed by the messages of its sources. + pub message: String, +} + +impl FailureRecord { + /// Returns the record of a failure of the clone that holds `lease`, with `transaction`, at + /// the stage and the progress `at`, with `evidence`, the guest's `status`, and `teardown`, + /// whose error is `error`. + pub(super) fn new( + lease: &Lease, + transaction: Option, + (stage, progress): (Stage, Progress), + evidence: Evidence, + status: Option, + teardown: Teardown, + error: &Error, + ) -> Self { + Self { + store: lease.store(), + lease: lease.id(), + template: lease.template(), + launch: lease.launch().clone(), + transaction, + stage, + progress, + evidence, + status, + outcome: evidence.outcome(), + teardown, + code: error.code(), + message: describe(error), + } + } + + /// Returns the error that the record describes, with its code and its message. + pub fn error(&self) -> Error { + Error::new(self.code, self.message.clone()) + } +} + +/// A failure of a clone: the error of the step that failed, with the [`FailureRecord`] that +/// attributes it, as the clone's plan, launch, restore sequence, and reattach return it. +/// +/// It displays as its error, whose [`kind`](Self::kind) and [`code`](Self::code) it has, and +/// converts into it, without the record. A failure from the clone's launch on is recorded in the +/// clone's sandbox as well, as [the records of failures](FailureRecord#records) describe. +#[derive(Debug)] +pub struct CloneFailure { + /// The error of the step that failed. + error: Error, + /// The failure's record. + record: Box, +} + +impl CloneFailure { + /// Returns the failure of `error`, which `record` attributes. + pub(super) fn new(error: Error, record: FailureRecord) -> Self { + Self { + error, + record: Box::new(record), + } + } + + /// Returns the error of the step that failed. + pub fn error(&self) -> &Error { + &self.error + } + + /// Returns the error of the step that failed, without the record. + pub fn into_error(self) -> Error { + self.error + } + + /// Returns the error's kind. + pub fn kind(&self) -> ErrorKind { + self.error.kind() + } + + /// Returns the error's code. + pub fn code(&self) -> ErrorCode { + self.error.code() + } + + /// Returns the error's message. + pub fn message(&self) -> &str { + self.error.message() + } + + /// Returns the failure's record. + pub fn record(&self) -> &FailureRecord { + &self.record + } + + /// Returns the outcome to which the failure is attributed. + pub fn outcome(&self) -> Outcome { + self.record.outcome + } + + /// Returns how the teardown that followed the failure ended. + pub fn teardown(&self) -> &Teardown { + &self.record.teardown + } + + /// Returns the failure attributed to the host's staging, once the caller proved that its + /// staging, rather than the request, made a file that the request supplied wrong: a failure + /// whose evidence is a [supplied file](Evidence::is_supplied_file) with the wrong identity, + /// length, or geometry, as the plan finds it, which records nothing. Any other failure is + /// returned unchanged. + pub fn with_staging_proven(mut self) -> Self { + if self.record.evidence.is_supplied_file() { + self.record.outcome = Outcome::Staging; + } + self + } + + /// Quarantines the leased generation through its tombstone, as + /// [`Store::tombstone`] does, waiting for the store's lock until `deadline`, or for as long as + /// it takes without one, if the failure is the [template's](Outcome::Template), and returns + /// whether it did. A failure of any other outcome quarantines nothing. + /// + /// The tombstone changes nothing of the generation's payload or snapshot: the generation takes + /// no new lease, while the clones that lease it keep running, and a retirement deletes it once + /// they have released it. The store's lock comes before a sandbox's in the + /// [lock order](crate::host::supervisor#the-lock-order), so the caller quarantines once it + /// released the lock of the clone's sandbox, such as the one that an + /// [`Acquired`](crate::host::template::Acquired) holds. + /// + /// Fails with [`ErrorKind::Other`] for a store of another identity than the one that leased + /// the generation, and as [`Store::tombstone`] fails. + pub fn quarantine(&self, store: &Store, deadline: Option) -> Result { + if !self.record.outcome.quarantines() { + return Ok(false); + } + if store.id() != self.record.store { + return Err(Error::other(format!( + "the generation {} that a failure quarantines is one of the template store {}, \ + not of this one, {}", + self.record.template.generation, + self.record.store, + store.id() + ))); + } + store.tombstone(self.record.template.generation, deadline)?; + Ok(true) + } +} + +impl fmt::Display for CloneFailure { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(&self.error, formatter) + } +} + +impl std::error::Error for CloneFailure { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + std::error::Error::source(&self.error) + } +} + +impl From for Error { + fn from(failure: CloneFailure) -> Self { + failure.error + } +} + +/// The error of a step of a clone, with the evidence that attributes it, before the stage at which +/// the clone failed, its progress, and the teardown that follows are known. +#[derive(Debug)] +pub(super) struct Fault { + /// The evidence of the failure. + pub(super) evidence: Evidence, + /// The step's error. + pub(super) error: Error, +} + +impl Fault { + /// Returns the fault of `error`, which `evidence` attributes. + pub(super) fn new(evidence: Evidence, error: Error) -> Self { + Self { evidence, error } + } + + /// Returns the fault of `error`, a failure of the host's own operations, as + /// [`host_evidence`] attributes it. + pub(super) fn host(error: Error) -> Self { + Self::new(host_evidence(&error), error) + } +} + +/// Attributes the error of a step of a clone. +pub(super) trait Attribute { + /// Returns the result with its error attributed by `evidence`. + fn attribute(self, evidence: Evidence) -> std::result::Result; + + /// Returns the result with its error attributed as a failure of the host's own operations, as + /// [`host_evidence`] attributes it. + fn on_host(self) -> std::result::Result; +} + +impl Attribute for Result { + fn attribute(self, evidence: Evidence) -> std::result::Result { + self.map_err(|error| Fault::new(evidence, error)) + } + + fn on_host(self) -> std::result::Result { + self.map_err(Fault::host) + } +} + +/// Returns the evidence of `error`, a failure of the host's own operations: resource exhaustion +/// if an I/O error among its sources says that the host ran out of storage, memory, or +/// descriptors, and a host failure otherwise. +pub(super) fn host_evidence(error: &Error) -> Evidence { + let mut source = std::error::Error::source(error); + while let Some(cause) = source { + if let Some(io) = cause.downcast_ref::() + && exhausted(io) + { + return Evidence::ResourceExhaustion; + } + source = cause.source(); + } + Evidence::HostFailure +} + +/// Returns whether `error` says that the host ran out of a resource. +fn exhausted(error: &io::Error) -> bool { + // Too many open files, in the system or in the process. + #[cfg(unix)] + if [nix::errno::Errno::ENFILE, nix::errno::Errno::EMFILE] + .into_iter() + .any(|errno| error.raw_os_error() == Some(errno as i32)) + { + return true; + } + matches!( + error.kind(), + io::ErrorKind::StorageFull + | io::ErrorKind::QuotaExceeded + | io::ErrorKind::FileTooLarge + | io::ErrorKind::OutOfMemory + ) +} + +/// Returns the message of `error`, followed by the messages of its sources, as the record of a +/// failure keeps it. +pub(super) fn describe(error: &Error) -> String { + let mut text = error.to_string(); + let mut source = std::error::Error::source(error); + while let Some(cause) = source { + text.push_str(": "); + text.push_str(&cause.to_string()); + source = cause.source(); + } + text +} + +/// Returns `error` with `addition` after its message, with its code and its source. +pub(super) fn amended(error: Error, addition: &str) -> Error { + let code = error.code(); + let (_, message, source) = error.into_parts(); + let error = Error::new(code, format!("{message}{addition}")); + match source { + Some(source) => error.with_source(source), + None => error, + } +} + +/// Returns the evidence of `status`, with which the guest ended a clone's VM once the clone got +/// as far as `progress`, if the status alone is evidence, as [the attribution of +/// evidence](Evidence#attribution) describes: a status of the time ABI or of a clone's repair is +/// evidence of the repair stage at any point, a status of a clone's activation is evidence of its +/// class before the start of the clone's sandbox was sent, and no other status is evidence. +pub fn status_evidence(status: u8, progress: Progress) -> Option { + if status::TIME_STATUSES.contains(&status) || status::REPAIR_STATUSES.contains(&status) { + return Some(Evidence::RepairStatus); + } + if progress.may_have_run() { + return None; + } + Some(match status { + status::ACTIVATION_DEADLINE => Evidence::Timeout, + status::ACTIVATION_PROTOCOL | status::ACTIVATION_ORDER => Evidence::Protocol, + status::ACTIVATION_RECORD => Evidence::InvalidActivate, + status::ACTIVATION_RESOURCES => Evidence::InvalidTargets, + status::ACTIVATION_ROOT => Evidence::UnusableRoot, + status::ACTIVATION_WORKLOAD | status::ACTIVATION_NETWORK => Evidence::InvalidSettings, + status::ACTIVATION_EXPORT => Evidence::InvalidMappings, + _ => return None, + }) +} + +/// Returns the guest's status in OpenVMM's outcome report `report` of a clone's VM, and the +/// evidence that the status alone is once the clone got as far as `progress`, as +/// [`status_evidence`] gives it: the status of a report whose category is `guest-exit` or +/// `success`, the guest's own end, and none for any other, such as a failure of the VMM. +pub(super) fn reported( + report: Option<&Report>, + progress: Progress, +) -> (Option, Option) { + let Some(report) = report else { + return (None, None); + }; + let status = match (report.category.as_deref(), report.status_code) { + (Some("guest-exit" | "success"), Some(code)) => u8::try_from(code).ok(), + _ => None, + }; + let evidence = status.and_then(|status| status_evidence(status, progress)); + (evidence, status) +} + +/// The start of the line with which OpenVMM reports, on its standard error, which the launch's log +/// keeps, the error that ended it: the line goes on with the error's first time ABI code in +/// brackets, such as `[E_TSC_RATE_TOLERANCE]`, if the error's chain has one. +pub(super) const FATAL_PREFIX: &str = "fatal error: "; + +/// The time ABI codes with which OpenVMM refuses to restore a snapshot on a host that cannot +/// present the template's CPU or time: a CPU that the template's CPU profile does not serve, a +/// hypervisor that cannot present the profile, as OpenVMM groups the failures of a profile, and a +/// TSC or a LAPIC timer that the snapshot's time contract does not accept. +const INCOMPATIBLE_HOST_CODES: [&str; 13] = [ + "E_PROFILE_HOST_UNKNOWN", + "E_CPU_GENERATION", + "E_PROFILE_UNSUPPORTED", + "E_PROFILE_TIME_BITS", + "E_CPU_SURFACE", + "E_CPU_UNLISTED", + "E_IDENTITY_ROUTING", + "E_TSC_SYNC_UNSUPPORTED", + "E_TSC_SCALING_ACTIVE", + "E_TSC_RATE_UNAVAILABLE", + "E_TSC_RATE_TOLERANCE", + "E_LAPIC_RATE_UNAVAILABLE", + "E_LAPIC_RATE_MISMATCH", +]; + +/// Returns the evidence of `line`, the line with which OpenVMM reported the error that ended a +/// clone's VMM before its restore-ready event, as [the attribution of +/// evidence](Evidence#attribution) describes: [`Evidence::IncompatibleHost`] for an error whose +/// time ABI code says that the host cannot present the template's CPU or time, and none for any +/// other. +pub(super) fn fatal_evidence(line: &str) -> Option { + let code = line.strip_prefix(FATAL_PREFIX)?.strip_prefix('[')?; + let code = &code[..code.find(']')?]; + INCOMPATIBLE_HOST_CODES + .contains(&code) + .then_some(Evidence::IncompatibleHost) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Every evidence, in the order of the module's table. + const ALL: [Evidence; 27] = [ + Evidence::AuthorizationDenied, + Evidence::StaleBinding, + Evidence::InvalidTargets, + Evidence::InvalidFormat, + Evidence::InvalidSettings, + Evidence::InvalidActivate, + Evidence::InvalidMappings, + Evidence::UnservableImage, + Evidence::UnusableScratch, + Evidence::UnusableRoot, + Evidence::SuppliedIdentity, + Evidence::SuppliedLength, + Evidence::SuppliedGeometry, + Evidence::EndpointBind, + Evidence::ForwardBind, + Evidence::ResourceExhaustion, + Evidence::IncompatibleHost, + Evidence::RuntimeMismatch, + Evidence::HostFailure, + Evidence::LostStatus, + Evidence::Timeout, + Evidence::Reset, + Evidence::ForcedKill, + Evidence::Protocol, + Evidence::MalformedGeneration, + Evidence::TemplateDefect, + Evidence::RepairStatus, + ]; + + #[test] + fn each_row_of_the_table_attributes_its_evidence() { + let rows: [(&[Evidence], Outcome); 6] = [ + ( + &[ + Evidence::AuthorizationDenied, + Evidence::StaleBinding, + Evidence::InvalidTargets, + Evidence::InvalidFormat, + Evidence::InvalidSettings, + Evidence::InvalidActivate, + Evidence::InvalidMappings, + Evidence::UnservableImage, + Evidence::UnusableScratch, + Evidence::UnusableRoot, + ], + Outcome::Request, + ), + ( + &[ + Evidence::SuppliedIdentity, + Evidence::SuppliedLength, + Evidence::SuppliedGeometry, + ], + Outcome::Request, + ), + ( + &[ + Evidence::EndpointBind, + Evidence::ForwardBind, + Evidence::ResourceExhaustion, + Evidence::IncompatibleHost, + Evidence::RuntimeMismatch, + Evidence::HostFailure, + ], + Outcome::Host, + ), + ( + &[ + Evidence::LostStatus, + Evidence::Timeout, + Evidence::Reset, + Evidence::ForcedKill, + Evidence::Protocol, + ], + Outcome::Unknown, + ), + ( + &[Evidence::MalformedGeneration, Evidence::TemplateDefect], + Outcome::Template, + ), + (&[Evidence::RepairStatus], Outcome::Repair), + ]; + let mut seen = Vec::new(); + for (evidence, outcome) in rows { + for evidence in evidence { + assert_eq!(evidence.outcome(), outcome, "{evidence}"); + assert_eq!( + evidence.outcome().quarantines(), + outcome == Outcome::Template, + "{evidence}" + ); + assert_eq!( + evidence.is_supplied_file(), + evidence.as_str().starts_with("supplied-"), + "{evidence}" + ); + seen.push(*evidence); + } + } + // The table attributes every evidence once. + assert_eq!(seen, ALL); + // Only the template's outcome quarantines. + for outcome in [ + Outcome::Request, + Outcome::Staging, + Outcome::Host, + Outcome::Unknown, + Outcome::Repair, + ] { + assert!(!outcome.quarantines(), "{outcome}"); + } + assert!(Outcome::Template.quarantines()); + } + + #[test] + fn names_are_those_of_the_records() { + for evidence in ALL { + let json = serde_json::to_string(&evidence).unwrap(); + assert_eq!(json, format!("\"{evidence}\"")); + assert_eq!(serde_json::from_str::(&json).unwrap(), evidence); + } + let progress = [ + Progress::Planned, + Progress::Launching, + Progress::Spawned, + Progress::Event, + Progress::Acknowledged, + Progress::Activating, + Progress::Ready, + Progress::Starting, + Progress::Started, + Progress::Committed, + ]; + // The steps are in order, and spelled as their records spell them. + assert!(progress.windows(2).all(|pair| pair[0] < pair[1])); + for step in progress { + assert_eq!(serde_json::to_string(&step).unwrap(), format!("\"{step}\"")); + } + let stages: Vec = progress.iter().map(|step| step.stage()).collect(); + assert_eq!( + stages, + [ + Stage::Plan, + Stage::Launch, + Stage::Restore, + Stage::Activation, + Stage::Activation, + Stage::Activation, + Stage::Start, + Stage::Start, + Stage::Commit, + Stage::Run, + ] + ); + for stage in [ + Stage::Plan, + Stage::Launch, + Stage::Restore, + Stage::Activation, + Stage::Start, + Stage::Commit, + Stage::Run, + Stage::Reattach, + ] { + assert_eq!( + serde_json::to_string(&stage).unwrap(), + format!("\"{stage}\"") + ); + } + for outcome in [ + Outcome::Request, + Outcome::Staging, + Outcome::Host, + Outcome::Unknown, + Outcome::Template, + Outcome::Repair, + ] { + assert_eq!( + serde_json::to_string(&outcome).unwrap(), + format!("\"{outcome}\"") + ); + } + } + + #[test] + fn a_repair_status_alone_is_evidence_of_the_repair_stage_at_any_point() { + for status in status::TIME_STATUSES.chain(status::REPAIR_STATUSES) { + for progress in [Progress::Spawned, Progress::Ready, Progress::Committed] { + assert_eq!( + status_evidence(status, progress), + Some(Evidence::RepairStatus), + "{status} {progress}" + ); + } + } + } + + #[test] + fn an_activation_status_is_evidence_of_its_class_until_a_workload_may_have_run() { + let classes = [ + (status::ACTIVATION_DEADLINE, Evidence::Timeout), + (status::ACTIVATION_PROTOCOL, Evidence::Protocol), + (status::ACTIVATION_RECORD, Evidence::InvalidActivate), + (status::ACTIVATION_RESOURCES, Evidence::InvalidTargets), + (status::ACTIVATION_ROOT, Evidence::UnusableRoot), + (status::ACTIVATION_WORKLOAD, Evidence::InvalidSettings), + (status::ACTIVATION_EXPORT, Evidence::InvalidMappings), + (status::ACTIVATION_NETWORK, Evidence::InvalidSettings), + (status::ACTIVATION_ORDER, Evidence::Protocol), + ]; + assert_eq!( + classes.map(|(status, _)| status).to_vec(), + status::ACTIVATION_STATUSES.collect::>() + ); + for (status, evidence) in classes { + for progress in [Progress::Spawned, Progress::Activating, Progress::Ready] { + assert_eq!( + status_evidence(status, progress), + Some(evidence), + "{status} {progress}" + ); + } + // Once the start was sent, a workload may end the VM with any status. + for progress in [Progress::Starting, Progress::Started, Progress::Committed] { + assert_eq!( + status_evidence(status, progress), + None, + "{status} {progress}" + ); + } + } + // No other status is evidence: a success, the generic failure, or a workload's. + for status in [0, 1, 2, 3, 39, 46, 49, 59, 137, 192, 196, 255] { + assert_eq!( + status_evidence(status, Progress::Activating), + None, + "{status}" + ); + } + } + + #[test] + fn a_report_gives_the_guest_s_status_and_its_evidence() { + let report = |category: &str, status_code: i64| Report { + category: Some(category.to_owned()), + status_code: Some(status_code), + teardown_complete: Some(true), + }; + assert_eq!(reported(None, Progress::Event), (None, None)); + assert_eq!( + reported(Some(&report("guest-exit", 195)), Progress::Event), + (Some(Evidence::RepairStatus), Some(195)) + ); + assert_eq!( + reported(Some(&report("guest-exit", 54)), Progress::Activating), + (Some(Evidence::UnusableRoot), Some(54)) + ); + assert_eq!( + reported(Some(&report("guest-exit", 54)), Progress::Committed), + (None, Some(54)) + ); + assert_eq!( + reported(Some(&report("success", 0)), Progress::Committed), + (None, Some(0)) + ); + // A failure of the VMM, or of its teardown, has no status of the guest's. + assert_eq!( + reported(Some(&report("vmm-failure", 1)), Progress::Spawned), + (None, None) + ); + assert_eq!( + reported(Some(&report("teardown-failure", 1)), Progress::Spawned), + (None, None) + ); + assert_eq!( + reported(Some(&report("guest-exit", 256)), Progress::Spawned), + (None, None) + ); + } + + #[test] + fn openvmm_s_fatal_error_is_evidence_of_an_incompatible_host_only_for_its_codes() { + for code in INCOMPATIBLE_HOST_CODES { + // OpenVMM leads its fatal error with the first time ABI code of the error's chain. + let line = format!("fatal error: [{code}] failed to launch vm worker"); + assert_eq!( + fatal_evidence(&line), + Some(Evidence::IncompatibleHost), + "{line}" + ); + } + // A code of the snapshot's time contract, or of this OpenVMM, does not tell the + // generation's defect from the host's, and an error without a code tells nothing. + for line in [ + "fatal error: [E_MANIFEST_TIME] failed to launch vm worker", + "fatal error: [E_PROFILE_DIGEST] failed to launch vm worker", + "fatal error: [E_SNAPSHOT_VERSION] failed to restore", + "fatal error: failed to open the snapshot", + "fatal error: [E_TSC_RATE_TOLERANCE failed to launch vm worker", + // A line that is not the fatal error, such as a warning, names no code of it. + "NVX-CPU-PROFILE-FALLBACK: [E_PROFILE_UNSUPPORTED] the host profile serves", + "warning: fatal error: [E_CPU_GENERATION] quoted", + ] { + assert_eq!(fatal_evidence(line), None, "{line}"); + } + } + + #[test] + fn host_failures_that_ran_out_of_a_resource_are_resource_exhaustion() { + let failed = |kind: io::ErrorKind| { + host_evidence(&Error::io("cannot write").with_source(io::Error::from(kind))) + }; + assert_eq!( + failed(io::ErrorKind::StorageFull), + Evidence::ResourceExhaustion + ); + assert_eq!( + failed(io::ErrorKind::OutOfMemory), + Evidence::ResourceExhaustion + ); + assert_eq!( + failed(io::ErrorKind::PermissionDenied), + Evidence::HostFailure + ); + assert_eq!( + host_evidence(&Error::io("no source")), + Evidence::HostFailure + ); + // A cause deeper in the chain counts too. + let nested = Error::io("outer").with_source( + Error::io("inner").with_source(io::Error::from(io::ErrorKind::StorageFull)), + ); + assert_eq!(host_evidence(&nested), Evidence::ResourceExhaustion); + } + + #[test] + fn teardowns_follow_the_end_of_the_launch() { + assert_eq!(Teardown::of(Ok(Recovery::Exited)), Teardown::Exited); + assert_eq!(Teardown::of(Ok(Recovery::Clean)), Teardown::Exited); + assert_eq!( + Teardown::of(Ok(Recovery::Indeterminate("held".to_owned()))), + Teardown::Unresolved("held".to_owned()) + ); + let failed = Teardown::of(Err(Error::io("cannot read"))); + assert!(!failed.is_resolved()); + assert!(!Teardown::Pending.is_resolved()); + assert!(Teardown::NoVm.is_resolved()); + assert!(Teardown::Terminated.is_resolved()); + } +} diff --git a/nvx-rs/src/host/template/clone/launch.rs b/nvx-rs/src/host/template/clone/launch.rs index 2354f622b..860fb2c1d 100644 --- a/nvx-rs/src/host/template/clone/launch.rs +++ b/nvx-rs/src/host/template/clone/launch.rs @@ -4,10 +4,17 @@ use std::ffi::{OsStr, OsString}; use std::fmt; +use std::fs; +use std::io; use std::path::{Path, PathBuf}; use std::time::Instant; -use super::{ClonePlan, CloneUpper, MIB, SPARSE_WHAT, check_geometry}; +use super::failure::{ + Attribute, CloneFailure, Evidence, FailureRecord, Fault, Progress, Stage, Teardown, + host_evidence, +}; +use super::records::{End, ProgressLog, Recorder, Tracker, abandon}; +use super::{ClonePlan, CloneUpper, MIB, Mismatch, SPARSE_WHAT, check_geometry}; use crate::activation::ActivationRecord; use crate::error::{Error, Result}; use crate::host::artifacts; @@ -42,6 +49,8 @@ pub(super) struct Hooks<'a> { pub(super) command: &'a dyn for<'h> Fn(VmmCommand<'h>) -> VmmCommand<'h>, /// Fails the launch at a point, as a test asks. pub(super) interrupt: &'a dyn Fn(Point) -> Result<()>, + /// Ends the launch of a clone that failed after its VMM was spawned. + pub(super) end: End, } impl Hooks<'static> { @@ -49,6 +58,7 @@ impl Hooks<'static> { pub(super) const LAUNCH: Self = Self { command: &unchanged, interrupt: &uninterrupted, + end: abandon, }; } @@ -73,26 +83,38 @@ impl ClonePlan { /// launch does. /// /// The launch first checks, without writing anything, that `acquired` holds the plan's lease, - /// under the lock that acquired it, before the launch that it reserves began, and that the + /// under the lock that acquired it, before the launch that it reserves began, that the /// sandbox's clone binding records the lease: the acquisition made the binding durable, and no - /// VMM of the clone starts without it. For an upper in RAM, it checks that the file of the - /// scratch device that the plan created is still there: a later plan under the same - /// acquisition replaces it, and removes it if that plan fails. It then: + /// VMM of the clone starts without it, and that the sandbox records no failure of the clone: + /// a lease's clone fails once, and the record of its failure lasts until the release, even + /// for a launch that failed before it began, which leaves the lock's reservation. For an upper + /// in RAM, it checks that the file of the scratch device that the plan created is still + /// there: a later plan under the same acquisition replaces it, and removes it if that plan + /// fails. It then: /// - /// 1. Writes the clone's activation record, in its canonical encoding, as the file + /// 1. Creates the clone's progress log, + /// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME) in the sandbox's + /// directory, in place of one that an earlier launch left, which it removes through a + /// handle of the directory that holds the sandbox's lock, once it removed the same way the + /// outcome report that an earlier VM of the sandbox left, so that a report beside the + /// clone's records is the clone's VM's. The launch and the restore sequence append to the + /// log each step that the clone reaches, without syncing it, as + /// [`CloneRecords`](super::CloneRecords) describes. + /// 2. Writes the clone's activation record, in its canonical encoding, as the file /// [`CLONE_ACTIVATION_NAME`](crate::host::store::CLONE_ACTIVATION_NAME) in the sandbox's /// directory, in place of one that an earlier launch left, before the VMM starts: the /// record, with the transaction ID of its restore section, is the one that the clone /// activates with, which the caller, or a caller that takes the sandbox over after a /// crash, reads there again and resends unchanged. - /// 2. Listens at the endpoint of the VMM's restore-ready event, which OpenVMM connects to once + /// 3. Listens at the endpoint of the VMM's restore-ready event, which OpenVMM connects to once /// it has prepared the restore, before the restored VM runs, and which must exist before /// the VMM starts: on Linux, the socket /// [`RESTORE_READY_NAME`](crate::host::store::RESTORE_READY_NAME) in the sandbox's /// directory, in place of one that an earlier launch left; on Windows, a named pipe whose /// name the launch draws at random. - /// 3. Runs [`supervisor::launch`] as the launch that the lease reserves, which begins it with - /// the store's launch journal, starts the VMM, and records its identity. + /// 4. Runs [`supervisor::launch`] as the launch that the lease reserves, which begins it with + /// the store's launch journal, starts the VMM, and records its identity in the journal, + /// which holds the spawn until the progress log records it with the restore-ready event. /// /// The VMM restores the generation's snapshot in place, in the generation's payload, which it /// only reads, and which the launch neither copies nor changes, so any number of clones @@ -132,51 +154,72 @@ impl ClonePlan { /// VMM, if it started, left beside the snapshot. /// /// Like every launch of the sandbox's store, the launch names the sandbox's files by their - /// paths, under its lock: the activation record that it writes, the restore-ready socket at - /// which it listens on Linux, and the journal, capability, and log with which it begins; and - /// the VMM, which runs in the sandbox's directory, opens the files that its arguments name - /// there. So the directories above the sandbox's must be ones that only the current user, or - /// an administrator, may change, as the [store](crate::host::store) requires of them, and as - /// [Block devices by path](crate::host::template::Store#block-devices-by-path) requires of - /// those above the clone's block devices. + /// paths, under its lock: the progress log that it creates, the activation record that it + /// writes, the restore-ready socket at which it listens on Linux, and the journal, capability, + /// and log with which it begins; and the VMM, which runs in the sandbox's directory, opens the + /// files that its arguments name there. So the directories above the sandbox's must be ones + /// that only the current user, or an administrator, may change, as the + /// [store](crate::host::store) requires of them, and as [Block devices by + /// path](crate::host::template::Store#block-devices-by-path) requires of those above the + /// clone's block devices. The launch removes the outcome report of an earlier VM of the + /// sandbox, and the progress log of an earlier launch, through a handle of the directory that + /// holds the sandbox's lock, though, as a release removes the clone's files, and creates its + /// own log at its path, where no file may exist: no change of the path makes the launch remove + /// another sandbox's report or log, and another sandbox's log fails the launch instead, before + /// it writes anything else. /// - /// Fails, and starts no VMM, checking in this order: + /// Fails with a [`CloneFailure`] at [`Stage::Launch`], whose evidence is the host's for a + /// failure of its own operations, as [`ClonePlan::new`] describes it, checking in this order: /// - /// - with [`ErrorKind::Other`](crate::ErrorKind::Other) if `acquired` holds another lease - /// than the plan's, under another lock than the one that acquired it, or after the launch - /// that it reserves began, and if the sandbox's clone binding does not record the plan's - /// lease, and as [`Lease::of`] fails, before anything is written; - /// - with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a runtime identity - /// other than the template's; - /// - for an upper in RAM, with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) if - /// the file of its scratch device, which the plan created, cannot be opened, as when a later - /// plan of the clone replaced it and failed, and with + /// - before anything is written, and so recording nothing, as a plan does: with + /// [`ErrorKind::Other`](crate::ErrorKind::Other) if `acquired` holds another lease than the + /// plan's, under another lock than the one that acquired it, or after the launch that it + /// reserves began, if the sandbox's clone binding does not record the plan's lease, and if + /// the sandbox records a failure of the clone, [`Evidence::HostFailure`], and as + /// [`Lease::of`] fails, or the record of a failure cannot be examined, the host's evidence; + /// with + /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a runtime identity other + /// than the template's, [`Evidence::RuntimeMismatch`]; for an upper in RAM, with + /// [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) if the file of its scratch + /// device, which the plan created, cannot be opened, as when a later plan of the clone + /// replaced it and failed, the host's evidence, and with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if it no longer presents the - /// template's scratch geometry; - /// - on Linux, with [`ErrorKind::Other`](crate::ErrorKind::Other) if the control socket that - /// the lease reserves is not beside the plan's snapshot, as for a plan and a lease that - /// name the template store by different directories; with + /// template's scratch geometry, [`Evidence::IncompatibleHost`]; on Linux, with + /// [`ErrorKind::Other`](crate::ErrorKind::Other) if the control socket that the lease + /// reserves is not beside the plan's snapshot, as for a plan and a lease that name the + /// template store by different directories, [`Evidence::HostFailure`], with /// [`ErrorKind::Io`](crate::ErrorKind::Io) if the generation's payload cannot be resolved - /// or opened, or is not a directory of the current user with mode 0700, as OpenVMM requires - /// of the directory of a control socket; and with + /// or opened, the host's evidence, or is not a directory of the current user with mode + /// 0700, as OpenVMM requires of the directory of a control socket, + /// [`Evidence::MalformedGeneration`], and with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if the path of the control /// socket, with or without the payload's links resolved, or of the restore-ready socket, is /// not shorter than 108 bytes, as a Unix socket's address requires, or is not valid UTF-8, /// as OpenVMM and the host take it, or if the path of the control socket holds a comma or an - /// equals sign, which OpenVMM's option `listen=` cannot take; - /// - with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) for an `openvmm` whose - /// path cannot be resolved; - /// - with [`ErrorKind::Io`](crate::ErrorKind::Io) if the activation record cannot be written, - /// or the endpoint of the restore-ready event cannot be opened; - /// - and as [`supervisor::launch`] fails, which ends the launch. + /// equals sign, which OpenVMM's option `listen=` cannot take, + /// [`Evidence::EndpointBind`]; with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) + /// for an `openvmm` whose path cannot be resolved, the host's evidence; with + /// [`ErrorKind::Other`](crate::ErrorKind::Other) if the sandbox's path no longer names the + /// directory that holds its lock, and if the progress log cannot be created, as when the + /// path names a directory that holds one already, the host's evidence; + /// - once the progress log exists, recording the failure in the sandbox, as the + /// [failures](CloneFailure) of a clone's launch are: with + /// [`ErrorKind::Io`](crate::ErrorKind::Io) if the activation record cannot be written, the + /// host's evidence, or the endpoint of the restore-ready event cannot be opened, + /// [`Evidence::EndpointBind`] unless the host ran out of a resource; and as + /// [`supervisor::launch`] fails, which ends the launch, the host's evidence, with the + /// teardown [`Teardown::NoVm`] before the spawn of the VMM, [`Teardown::Terminated`] once + /// the launch ended the VMM that it spawned through its handle, and + /// [`Teardown::Unresolved`] for a launch that could not be ended, whose journal remains. /// - /// A launch that fails leaves the clone's files that it wrote for the release. + /// A launch that fails leaves the clone's files that it wrote, and the record of its failure, + /// for the release. pub fn launch<'s>( self, acquired: &'s mut Acquired, openvmm: &Path, runtime: &RuntimeIdentity, - ) -> Result> { + ) -> Result, CloneFailure> { self.launch_with(acquired, openvmm, runtime, Hooks::LAUNCH) } @@ -187,12 +230,29 @@ impl ClonePlan { openvmm: &Path, runtime: &RuntimeIdentity, hooks: Hooks<'_>, - ) -> Result> { + ) -> Result, CloneFailure> { // The restore-to-Ready deadline of the clone's restore sequence starts here. let launched = Instant::now(); let Acquired { lease, locked } = acquired; - self.check_launch(lease, locked, runtime)?; - let openvmm = artifacts::absolute(openvmm)?; + let planned = self.lease.clone(); + let transaction = self.activation.restore().map(|restore| restore.transaction); + // Until the launch writes a file, the sandbox may not be the clone's, so its failure is + // recorded nowhere. + let unrecorded = |fault: Fault| { + let record = FailureRecord::new( + &planned, + transaction, + (Stage::Launch, Progress::Planned), + fault.evidence, + None, + Teardown::NoVm, + &fault.error, + ); + CloneFailure::new(fault.error, record) + }; + self.check_launch(lease, locked, runtime) + .map_err(unrecorded)?; + let openvmm = artifacts::absolute(openvmm).on_host().map_err(unrecorded)?; let Self { lease: _, generation: _, @@ -205,18 +265,40 @@ impl ClonePlan { capability, devices, } = self; - let devices = device_arguments(&devices)?; + let devices = device_arguments(&devices) + .attribute(Evidence::InvalidSettings) + .map_err(unrecorded)?; + let log = locked + .open_dir() + .and_then(|directory| ProgressLog::create(&directory, locked.dir(), lease.launch())) + .on_host() + .map_err(unrecorded)?; + let recorder = Recorder::new(lease.clone(), transaction, locked); + let mut tracker = Tracker::new(recorder, log, hooks.end); + let unspawned = + |tracker: &Tracker, fault: Fault| tracker.record(Stage::Launch, fault, Teardown::NoVm); let record = locked.clone_activation_path(); - write_private(&record, &activation.encode())?; + write_private(&record, &activation.encode()) + .on_host() + .map_err(|fault| unspawned(&tracker, fault))?; let (ready, listener) = platform::listen_ready(&locked.restore_ready_path()).map_err(|error| { - Error::io(format!( - "cannot listen for the restore-ready event of the clone in {}", - locked.dir().display() - )) - .with_source(error) + let fault = Fault::host( + Error::io(format!( + "cannot listen for the restore-ready event of the clone in {}", + locked.dir().display() + )) + .with_source(error), + ); + let fault = match fault.evidence { + Evidence::HostFailure => Fault::new(Evidence::EndpointBind, fault.error), + _ => fault, + }; + unspawned(&tracker, fault) })?; - (hooks.interrupt)(Point::Prepared)?; + (hooks.interrupt)(Point::Prepared) + .on_host() + .map_err(|fault| unspawned(&tracker, fault))?; // On Windows, this closes the claim's handle, and keeps the image's guard. let claim = match upper { CloneUpper::Scratch(checked) => Some(checked.into_file()), @@ -228,8 +310,15 @@ impl ClonePlan { let claim = match claim { Some(mut file) => { crate::host::sys::move_above_standard_streams(&mut file).map_err(|error| { - Error::io("cannot pass the claim of the clone's scratch image to its VMM") - .with_source(error) + unspawned( + &tracker, + Fault::host( + Error::io( + "cannot pass the claim of the clone's scratch image to its VMM", + ) + .with_source(error), + ), + ) })?; Some(file) } @@ -245,17 +334,37 @@ impl ClonePlan { }; let dir = locked.dir().to_owned(); let inherit: Vec> = claim.iter().map(Inherited::new).collect(); - let starting = - supervisor::launch(locked, lease.launch(), runtime, &capability, |endpoint| { + let starting = supervisor::launch_reporting( + locked, + lease.launch(), + runtime, + &capability, + |endpoint| { (hooks.interrupt)(Point::Begun)?; let mut command = VmmCommand::new(&openvmm, restore.arguments(endpoint), &dir); command.inherit = inherit; Ok((hooks.command)(command)) - }); + }, + |error, _| tracker.pending(Stage::Launch, host_evidence(error), error), + ); // The VMM holds the claim, if it started. drop(claim); + let starting = match starting { + Ok(starting) => starting, + Err(failure) => { + let teardown = match (failure.unended, failure.spawned) { + (Some(reason), _) => Teardown::Unresolved(reason), + (None, true) => Teardown::Terminated, + (None, false) => Teardown::NoVm, + }; + return Err(tracker.record(Stage::Launch, Fault::host(failure.error), teardown)); + } + }; + // The launch's journal records the spawn, which the progress log records with the next + // step. + tracker.advance(Progress::Spawned); Ok(Restoring { - starting: starting?, + starting, launched, ready, listener, @@ -263,6 +372,7 @@ impl ClonePlan { activation, record, capability, + tracker, }) } @@ -275,37 +385,72 @@ impl ClonePlan { lease: &Lease, locked: &LockedStore, runtime: &RuntimeIdentity, - ) -> Result<()> { + ) -> Result<(), Fault> { let planned = &self.lease; if lease != planned || locked.dir() != planned.sandbox() || locked.reserved() != Some(planned.launch()) { - return Err(Error::other(format!( - "a clone is launched under the lock that acquired the lease of its plan, {}, \ - before the launch that the lease reserves begins, but the lock of the sandbox in \ - {} holds the lease {}, and reserves no launch {} of it", - planned.id(), - locked.dir().display(), - lease.id(), - planned.launch() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "a clone is launched under the lock that acquired the lease of its plan, {}, \ + before the launch that the lease reserves begins, but the lock of the \ + sandbox in {} holds the lease {}, and reserves no launch {} of it", + planned.id(), + locked.dir().display(), + lease.id(), + planned.launch() + )), + )); + } + if Lease::of(locked).on_host()?.as_ref() != Some(planned) { + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the sandbox in {} records no clone binding of the lease {}, which its \ + acquisition makes durable before any VMM of the clone starts", + locked.dir().display(), + planned.id() + )), + )); } - if Lease::of(locked)?.as_ref() != Some(planned) { - return Err(Error::other(format!( - "the sandbox in {} records no clone binding of the lease {}, which its \ - acquisition makes durable before any VMM of the clone starts", - locked.dir().display(), - planned.id() - ))); + // A launch that failed before it began leaves the lock's reservation, but the record of + // its failure, which no launch removes, ends the clone's launches until the release. + let failure = locked.clone_failure_path(); + match fs::symlink_metadata(&failure) { + Ok(_) => { + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the clone of the lease {} failed, as {} records: a lease's clone fails \ + once, and the release of the lease ends it", + planned.id(), + failure.display() + )), + )); + } + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + Err(error) => { + return Err(Fault::host( + Error::io(format!( + "cannot examine the record of a failure of the clone {}", + failure.display() + )) + .with_source(error), + )); + } } if runtime != self.key.runtime() { - return Err(Error::unsupported(format!( - "the template's generation {} restores under the runtime that captured it, {:?}, \ - not under {runtime:?}", - self.generation.generation(), - self.key.runtime() - ))); + return Err(Fault::new( + Evidence::RuntimeMismatch, + Error::unsupported(format!( + "the template's generation {} restores under the runtime that captured it, \ + {:?}, not under {runtime:?}", + self.generation.generation(), + self.key.runtime() + )), + )); } if let CloneUpper::Tmpfs { path } = &self.upper { check_sparse(path, self.key.scratch())?; @@ -330,20 +475,25 @@ impl ClonePlan { /// socket. OpenVMM takes the control socket's path in its option `listen=`, among /// options that commas separate, and whose values follow an equals sign, so the path must hold /// neither. - fn check_sockets(&self, socket: &Path, ready: &Path) -> Result<()> { + fn check_sockets(&self, socket: &Path, ready: &Path) -> Result<(), Fault> { let (Some(payload), Some(name)) = (socket.parent(), socket.file_name()) else { - return Err(Error::other(format!( - "the control socket {} of the clone's VMM names no directory", - socket.display() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the control socket {} of the clone's VMM names no directory", + socket.display() + )), + )); }; let resolve = |path: &Path| { filesystem::canonicalize(path).map_err(|error| { - Error::io(format!( - "cannot resolve the payload of the generation {}", - path.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot resolve the payload of the generation {}", + path.display() + )) + .with_source(error), + ) }) }; let canonical = resolve(payload)?; @@ -353,51 +503,68 @@ impl ClonePlan { None => false, }; if !beside { - return Err(Error::other(format!( - "the control socket {} of the clone's VMM, which its lease reserves, is not beside \ - the snapshot {} of its plan, where OpenVMM requires it", - socket.display(), - self.snapshot.display() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the control socket {} of the clone's VMM, which its lease reserves, is not \ + beside the snapshot {} of its plan, where OpenVMM requires it", + socket.display(), + self.snapshot.display() + )), + )); } let problem = platform::check_store_dir(payload).map_err(|error| { - Error::io(format!( - "cannot open the payload of the generation {}", - payload.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot open the payload of the generation {}", + payload.display() + )) + .with_source(error), + ) })?; if let Some(problem) = problem { - return Err(Error::io(format!( - "the payload of the generation {} {problem}", - payload.display() - ))); + return Err(Fault::new( + Evidence::MalformedGeneration, + Error::io(format!( + "the payload of the generation {} {problem}", + payload.display() + )), + )); } let resolved = canonical.join(name); for path in [socket, &resolved, ready] { if path.as_os_str().len() >= SOCKET_PATH_LIMIT { - return Err(Error::unsupported(format!( - "the path of the socket {} of a clone's launch must be shorter than \ - {SOCKET_PATH_LIMIT} bytes, as a Unix socket's address requires", - path.display() - ))); + return Err(Fault::new( + Evidence::EndpointBind, + Error::unsupported(format!( + "the path of the socket {} of a clone's launch must be shorter than \ + {SOCKET_PATH_LIMIT} bytes, as a Unix socket's address requires", + path.display() + )), + )); } } for path in [socket, &resolved, ready] { if path.to_str().is_none() { - return Err(Error::unsupported(format!( - "the path of the socket {} of a clone's launch must be valid UTF-8, as \ - OpenVMM takes it", - path.display() - ))); + return Err(Fault::new( + Evidence::EndpointBind, + Error::unsupported(format!( + "the path of the socket {} of a clone's launch must be valid UTF-8, as \ + OpenVMM takes it", + path.display() + )), + )); } } if socket.to_string_lossy().contains([',', '=']) { - return Err(Error::unsupported(format!( - "the path of the control socket {} of a clone's VMM must hold no ',' or '=', \ - which end the path in OpenVMM's option `listen=`", - socket.display() - ))); + return Err(Fault::new( + Evidence::EndpointBind, + Error::unsupported(format!( + "the path of the control socket {} of a clone's VMM must hold no ',' or '=', \ + which end the path in OpenVMM's option `listen=`", + socket.display() + )), + )); } Ok(()) } @@ -406,21 +573,29 @@ impl ClonePlan { /// Refuses the file of the `scratch` block device of a clone whose upper is in RAM, at `path`, /// which the clone's plan created, unless it is still a regular file that presents the template's /// scratch geometry, `scratch`: OpenVMM opens it by its path, and a later plan under the same -/// acquisition replaces it, or removes it if that plan fails. -fn check_sparse(path: &Path, scratch: BlockGeometry) -> Result<()> { +/// acquisition replaces it, or removes it if that plan fails. The file is the host's, as is the +/// evidence of its refusal. +fn check_sparse(path: &Path, scratch: BlockGeometry) -> Result<(), Fault> { let shown = path.display(); let file = open_regular(path).map_err(|error| { - Error::unavailable(format!( - "the {SPARSE_WHAT} {shown}, which the clone's plan created, cannot be opened, as when \ - a later plan of the clone replaced it and failed" - )) - .with_source(error) + Fault::host( + Error::unavailable(format!( + "the {SPARSE_WHAT} {shown}, which the clone's plan created, cannot be opened, as \ + when a later plan of the clone replaced it and failed" + )) + .with_source(error), + ) })?; check_geometry( - block_geometry(&file, path, SPARSE_WHAT)?, + block_geometry(&file, path, SPARSE_WHAT).map_err(|error| match error.kind() { + // The sandbox's filesystem presents a geometry that OpenVMM refuses. + crate::ErrorKind::Unsupported => Fault::new(Evidence::IncompatibleHost, error), + _ => Fault::host(error), + })?, scratch, &format!("the {SPARSE_WHAT} {shown}"), "scratch", + Mismatch::HOST, ) } @@ -521,6 +696,8 @@ pub struct Restoring<'s> { pub(super) record: PathBuf, /// The capability of the launch. pub(super) capability: [u8; CAPABILITY_LEN], + /// How far the clone got, and how its failures are recorded. + pub(super) tracker: Tracker, } impl<'s> Restoring<'s> { diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs new file mode 100644 index 000000000..a2995a8b6 --- /dev/null +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -0,0 +1,499 @@ +//! The reattach of a clone, [`reattach_clone`], once the caller of its launch restarted: it finds +//! the clone's VM by the identity of its VMM's process, takes it over if this host may manage it, +//! and checks the clone's activation in a new session, or tells how the clone ended, from the +//! records of its sandbox. + +use std::path::Path; +use std::time::{Duration, Instant}; + +use super::failure::{ + Attribute, CloneFailure, Evidence, Fault, Progress, Stage, Teardown, amended, describe, +}; +use super::records::{Recorder, progress_of, read_failure, read_progress}; +use super::restore::{ActivationSession, CloneControl}; +use crate::activation::{ActivationRecord, MAX_RECORD_BYTES, ReadyActivation}; +use crate::error::{Error, ErrorCode, Result}; +use crate::host::launch::timeout_problem; +use crate::host::process::BootId; +use crate::host::report::Report; +use crate::host::store::regular::{open_regular, read_at_most}; +use crate::host::store::{IdentityPart, LockedStore, Recovery, RuntimeIdentity, RuntimeRecord}; +use crate::host::supervisor::{self, Reattached, StopOutcome, Vmm}; +use crate::host::template::Lease; + +/// What [`reattach_clone`] found of a clone. +#[derive(Debug)] +#[non_exhaustive] +pub enum ReattachedClone { + /// The clone's VMM runs under the caller's runtime identity, and the clone answered the + /// Activate of its record, byte for byte, in a new session of the adapter's framing, with the + /// Ready of the record's activation: the caller holds the VM, and the session, in which it + /// drives the clone. + Live { + /// The clone's VM. + vmm: Box, + /// The session in which the clone answered. + session: S, + }, + /// No VM of the clone runs, and the sandbox records no failure of it: its VM exited once its + /// launch committed, with an outcome report of the guest's own end whose status is no evidence + /// of a failure, or its holder stopped it, or its launch ended, or never began, without a + /// failure, as its progress tells. The caller releases the clone's lease. + Ended { + /// How far the clone got. + progress: Progress, + /// OpenVMM's outcome report of the clone's VM, if its VMM wrote one. + report: Option, + }, + /// The clone failed, as the record of its failure in its sandbox tells, which the reattach + /// found, or wrote: the launch of a caller that crashed, which the reattach ended; a VM that + /// exited once its launch committed, with a status of the guest's that is evidence, or + /// without a report of the guest's own end; a check of the clone that failed; or a VM that + /// runs under another runtime identity. The failure's teardown tells whether a VM of the + /// clone may still run: until it is resolved, the sandbox keeps the store's records of the VM, + /// and with them the clone's lease, for a later reattach. + Failed(CloneFailure), + /// Whether the clone's VMM, whose launch committed, runs cannot be determined, for the reason + /// that this gives, and the sandbox records no failure of the clone: the runtime record + /// remains, and with it the clone's lease, for a later reattach, which may succeed. + Indeterminate(String), +} + +/// Reattaches to the clone of the sandbox of `locked`, under the runtime identity `runtime`, +/// which the caller would launch the clone's template with now, through the adapter's framing, +/// `control`, which must answer in a new session by `deadline`, and ends a VMM of the clone that +/// it may not keep within `bound`. +/// +/// It reads the records of the clone that the sandbox keeps, as [`CloneRecords`] describes, then +/// reattaches to the sandbox's VM, as [`supervisor::reattach`] does, which ends a launch that a +/// crash of its caller interrupted, through the verified identity of its VMM, removes a journal +/// that the runtime record of its launch supersedes, as the launch's commit may leave it, which is +/// no failure, and finds a VMM whose launch committed by the identity of its process, which says +/// which VM it is: +/// +/// - A VMM that runs under `runtime`, whose launch is the one that the clone's lease reserves, is +/// the clone's, which the reattach checks in a new session, as the restore sequence of the +/// clone's launch does, [`Restoring::activate`](super::Restoring::activate), sending the +/// sandbox's activation record, byte for byte, which must still be the canonical encoding of a +/// record of the restore of the lease's generation: the clone answers with the Ready that it +/// answered once, and the reattach checks the transaction ID that it reports, which +/// [`ReattachedClone::Live`] returns with the session. A check that fails, by the evidence that +/// the restore sequence gives it, ends the VMM through its verified identity, as +/// [`Vmm::terminate`] does within `bound`, and the clone fails at [`Stage::Reattach`]. +/// - A VMM that runs under another runtime identity, which might not speak the caller's +/// protocol, is ended the same way, without a word to its guest, and the clone fails at +/// [`Stage::Reattach`] with [`Evidence::RuntimeMismatch`], and +/// [`ErrorCode::RuntimeMismatch`]. +/// - A launch that the reattach ended fails at the stage of its progress, with +/// [`ErrorCode::Exited`], and with [`Evidence::ForcedKill`] for a VMM that the reattach +/// terminated, and [`Evidence::LostStatus`] otherwise, which the error that OpenVMM reported, for +/// a VMM that exited before its restore-ready event, attributes as the restore sequence does. One +/// that it cannot end yet fails the same way with [`Evidence::LostStatus`], whose teardown is +/// unresolved: its journal remains, and the lease with it, for a later reattach. +/// - A VMM whose launch committed, and that exited, fails at [`Stage::Run`] with +/// [`ErrorCode::Exited`] if OpenVMM's outcome report gives a status of the guest's that is +/// evidence, as a status of the clone's repair is at any point, and with +/// [`Evidence::LostStatus`] if no report of the guest's own end tells why it exited, and +/// [`ReattachedClone::Ended`] otherwise. One that cannot be examined is +/// [`ReattachedClone::Indeterminate`]. +/// +/// The evidence of a status of the guest's prevails, as in the restore sequence. A failure is +/// recorded in the sandbox, as the restore sequence records it: before the reattach ends a VMM of +/// the clone, recovers an interrupted launch, or clears the runtime record of a VM that exited, +/// with its teardown [pending](Teardown::Pending), and again once the teardown ended. A clone +/// whose sandbox records a failure already failed for good: the reattach ends a VMM of the clone +/// that still runs, through its verified identity, then finishes a failure whose teardown a crash +/// of its caller left pending, as the failure would have recorded it once its teardown ended, or +/// records the teardown of one whose teardown was unresolved in place of the record's, which +/// keeps the stage, the evidence, and the original outcome, unless what the clone's VMM left as +/// it ended since tells more, as the guest's status, which the record does not hold, and the +/// error that OpenVMM reported, for a VMM that exited before its restore-ready event, may, and +/// returns the failure, with the record's lease, transaction, stage, progress, and code. +/// +/// The reattach reads no record of the template store: the lease holds the clone's generation, +/// and the sandbox's activation record the clone's transaction, so a tombstone, or a retirement, +/// of the generation leaves the clone's reattach, as its run, as it is. It signals no process that +/// it has not verified. The progress log and the record of a failure only inform it: one that +/// cannot be read is none, so that the reattach still ends the clone's VM, and a failure that the +/// reattach records replaces it. A clone that launched, whose progress log, which its launch +/// created before it began, and appends to without a sync, cannot be read, is missing, or was +/// written during another boot of the host, whose crash may have lost its last lines, may have +/// run its workload: a status of the guest's is then attributed as once the start of the clone's +/// sandbox was sent. +/// +/// Fails with [`ErrorKind::Other`](crate::ErrorKind::Other) if the sandbox holds no clone +/// binding, whose VM reattaches as any sandbox's does, with +/// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if `bound` is zero or longer than +/// [`MAX_TIMEOUT`](crate::host::launch::MAX_TIMEOUT), and for an invalid `runtime`, before it +/// writes anything, and as [`Lease::of`] and [`supervisor::reattach`] fail for records that are +/// inconsistent, which leave the clone's VM as it was, and the record of the failure of an +/// interrupted launch, if the reattach wrote it, pending. +/// +/// [`CloneRecords`]: super::CloneRecords +pub fn reattach_clone( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + deadline: Instant, + bound: Duration, +) -> Result> { + reattach_clone_with( + locked, + runtime, + control, + deadline, + bound, + |locked, runtime, exited| supervisor::reattach_reporting(locked, runtime, exited), + ) +} + +/// Reattaches to a clone as [`reattach_clone`] does, reattaching to its VM with `reattach`, which +/// is [`supervisor::reattach_reporting`] outside tests, and which calls its last argument with +/// the runtime record of a VMM that exited before it clears the record. +pub(super) fn reattach_clone_with( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + deadline: Instant, + bound: Duration, + reattach: impl FnOnce( + &mut LockedStore, + &RuntimeIdentity, + &dyn Fn(&RuntimeRecord), + ) -> Result, +) -> Result> { + let started = Instant::now(); + if let Some(problem) = timeout_problem(bound, true) { + return Err(Error::unsupported(format!( + "the bound of a clone's termination {problem}" + ))); + } + // The reattach fails for an invalid runtime identity before it writes anything. + runtime.check()?; + let Some(lease) = Lease::of(locked)? else { + return Err(Error::other(format!( + "the sandbox in {} holds no clone binding, and so no clone to reattach to", + locked.dir().display() + ))); + }; + let launch = lease.launch(); + let journal = locked + .journal()? + .filter(|journal| journal.launch() == launch); + let runtime_record = locked.runtime()?; + let committed = runtime_record + .as_ref() + .filter(|record| record.launch() == launch); + let earlier = read_failure(&locked.clone_failure_path()).unwrap_or(None); + let logged = read_progress(&locked.clone_progress_path(), launch).unwrap_or(None); + // The launch of a clone creates its progress log before it begins, and before it records a + // failure, and appends to it without a sync, so the log of a clone that launched tells how + // far the clone got for sure only if it can be read, and was written during the current boot + // of the host, whose crash would have ended the clone's VM: a crash of the host may have lost + // the log's last lines, and a workload may then have run in the clone. A log that can be read + // tells that the clone launched as well as the store's records do, which a stop of its VM + // clears, and a release alone removes it. + let launched = + journal.is_some() || committed.is_some() || earlier.is_some() || logged.is_some(); + let certain = logged.is_some_and(|logged| logged.is_certain(BootId::current().ok())); + let uncertain = launched && !certain; + let logged = logged.map(|logged| logged.progress); + let progress = progress_of(&lease, journal.as_ref(), committed, logged); + let activation = read_activation(&locked.clone_activation_path(), &lease); + // A failure that the reattach records has the transaction that the clone's activation record + // names, while the record of an earlier failure keeps its own. + let transaction = activation + .as_ref() + .ok() + .and_then(|(_, record)| record.restore().map(|restore| restore.transaction)); + let recorder = Recorder::new(lease.clone(), transaction, locked).uncertain(uncertain); + let fresh = earlier.is_none(); + // A launch that the journal names, which no runtime record supersedes, and whose failure the + // sandbox does not record, was interrupted by a crash of its caller: its failure is recorded + // before the recovery ends it, so that a crash of this caller during the recovery leaves the + // record. A journal that the runtime record of its launch supersedes, which the launch's + // commit may leave, is no failure: the recovery removes it. + if fresh && journal.is_some() && runtime_record.is_none() { + recorder.pending( + progress.stage(), + progress, + (Evidence::LostStatus, None), + &interrupted(progress), + ); + } + // So is the failure of a VM whose launch committed, and which exited, before the reattach + // clears its runtime record, as the outcome report that the recorder reads then tells, which + // the rest of the reattach takes as it found it. + let exiting = |_: &RuntimeRecord| { + if fresh && let Some((fault, status)) = exit_fault(&recorder, progress, true) { + let evidence = (fault.evidence, status); + recorder.pending(progress.stage(), progress, evidence, &fault.error); + } + }; + let reattached = reattach(locked, runtime, &exiting)?; + if let Some(earlier) = earlier { + let teardown = match reattached { + Reattached::Live(vmm) | Reattached::Mismatched { vmm, .. } => { + stopped(vmm.terminate(locked, bound), bound) + } + Reattached::Exited { + recovery: recovery @ (Recovery::Exited | Recovery::Terminated(_)), + .. + } => Teardown::of(Ok(recovery)), + Reattached::Exited { + record: Some(_), .. + } => Teardown::Exited, + // Nothing of the clone remains: a teardown that was not resolved is now. + Reattached::Exited { record: None, .. } => match earlier.teardown { + Teardown::Unresolved(_) | Teardown::Pending => Teardown::Exited, + ref teardown => teardown.clone(), + }, + Reattached::Indeterminate(reason) => Teardown::Unresolved(reason), + }; + let failure = if earlier.teardown == Teardown::Pending { + // The crash of the caller during the teardown left the failure pending, which is + // recorded as the failure would have recorded it once its teardown ended. + let error = unended(earlier.error(), &teardown); + recorder.finish(earlier, error, teardown) + } else { + recorder.resolve(earlier, teardown) + }; + return Ok(ReattachedClone::Failed(failure)); + } + let failed = |fault: Fault, stage: Stage, teardown: Teardown| { + let fault = Fault::new(fault.evidence, unended(fault.error, &teardown)); + Ok(ReattachedClone::Failed( + recorder.record(stage, progress, fault, teardown), + )) + }; + match reattached { + Reattached::Live(vmm) => { + let checked = activation.and_then(|activation| { + check_activation(locked, &lease, &vmm, control, activation, deadline, started) + }); + match checked { + Ok(session) => Ok(ReattachedClone::Live { + vmm: Box::new(vmm), + session, + }), + Err(fault) => { + let evidence = (fault.evidence, None); + recorder.pending(Stage::Reattach, progress, evidence, &fault.error); + let teardown = stopped(vmm.terminate(locked, bound), bound); + failed(fault, Stage::Reattach, teardown) + } + } + } + Reattached::Mismatched { vmm, differences } => { + let fault = Fault::new(Evidence::RuntimeMismatch, mismatched(&differences)); + let evidence = (fault.evidence, None); + recorder.pending(Stage::Reattach, progress, evidence, &fault.error); + let teardown = stopped(vmm.terminate(locked, bound), bound); + failed(fault, Stage::Reattach, teardown) + } + Reattached::Exited { + recovery: recovery @ (Recovery::Exited | Recovery::Terminated(_)), + .. + } => { + let evidence = match recovery { + Recovery::Terminated(_) => Evidence::ForcedKill, + _ => Evidence::LostStatus, + }; + let fault = Fault::new(evidence, interrupted(progress)); + failed(fault, progress.stage(), Teardown::of(Ok(recovery))) + } + Reattached::Exited { record, .. } => { + match exit_fault(&recorder, progress, record.is_some()) { + Some((fault, _)) => failed(fault, progress.stage(), Teardown::Exited), + // A report is of the clone's VM only once its launch began, as in `exit_fault`. + None => Ok(ReattachedClone::Ended { + progress, + report: (progress >= Progress::Spawned) + .then(|| recorder.observed().cloned()) + .flatten(), + }), + } + } + Reattached::Indeterminate(reason) => { + if locked.journal()?.is_none() { + return Ok(ReattachedClone::Indeterminate(reason)); + } + // The launch that a crash of its caller interrupted cannot be ended yet. + let fault = Fault::new(Evidence::LostStatus, interrupted(progress)); + failed(fault, progress.stage(), Teardown::Unresolved(reason)) + } + } +} + +/// Returns the fault of the clone whose failures `recorder` records, which got as far as +/// `progress`, and whose VM exited, as OpenVMM's outcome report, which +/// [`Recorder::observed`] reads, tells, with the guest's status, or `None` if the VM ended +/// without a failure: a status of the guest's that is evidence, as [`Recorder::reported`] gives +/// it, or, for a VM that exited `unheld`, which the reattach found exited, the loss of the clone's +/// status, if no report of the guest's own end tells why. +fn exit_fault( + recorder: &Recorder, + progress: Progress, + unheld: bool, +) -> Option<(Fault, Option)> { + // A report is of the clone's VM only once its launch began, which removes the report of an + // earlier VM of the sandbox. + if progress < Progress::Spawned { + return None; + } + let (evidence, status) = recorder.reported(recorder.observed(), progress); + let evidence = match evidence { + Some(evidence) => evidence, + // Only a VM that exited unheld ends without telling why: its holder's stop ends one + // otherwise. + None if unheld && status.is_none() => Evidence::LostStatus, + None => return None, + }; + Some((Fault::new(evidence, exited(progress, status)), status)) +} + +/// Reads the clone's activation record at `path`, which must be the canonical encoding of a +/// record of the restore of the generation that `lease` holds, as the clone's launch wrote it, +/// and returns its bytes and the record, or the fault, the host's, of one that is not. +fn read_activation( + path: &Path, + lease: &Lease, +) -> std::result::Result<(Vec, ActivationRecord), Fault> { + let bytes = open_regular(path) + .and_then(|file| read_at_most(file, MAX_RECORD_BYTES as u64, "an activation record")) + .map_err(|error| { + Fault::host( + Error::io(format!( + "cannot read the clone's activation record {}", + path.display() + )) + .with_source(error), + ) + })?; + let malformed = |problem: &str| { + Error::io(format!( + "the clone's activation record {} {problem}", + path.display() + )) + }; + let record = ActivationRecord::decode(&bytes) + .map_err(|error| malformed("is malformed").with_source(error)) + .attribute(Evidence::HostFailure)?; + if record.encode() != bytes { + return Err(Fault::new( + Evidence::HostFailure, + malformed("is not in its canonical encoding, in which the clone's launch wrote it"), + )); + } + if !record + .restore() + .is_some_and(|restore| restore.template == lease.template()) + { + return Err(Fault::new( + Evidence::HostFailure, + malformed(&format!( + "does not restore the generation {} that the clone's lease holds", + lease.generation() + )), + )); + } + Ok((bytes, record)) +} + +/// Checks the clone of `lease`, whose VM `vmm` runs in the sandbox of `locked`, in a new session +/// of `control`, with the sandbox's capability and its activation record, `activation`, by +/// `deadline`, as the reattach that `started` describes, and returns the session in which the +/// clone answered. +fn check_activation( + locked: &LockedStore, + lease: &Lease, + vmm: &Vmm, + control: &mut C, + (encoded, record): (Vec, ActivationRecord), + deadline: Instant, + started: Instant, +) -> std::result::Result { + let launch = vmm.record().launch(); + if launch != lease.launch() { + return Err(Fault::new( + Evidence::HostFailure, + Error::io(format!( + "the VM of the sandbox in {} is of the launch {launch}, not of the launch {} that \ + the clone's lease reserves", + locked.dir().display(), + lease.launch() + )), + )); + } + let capability = locked.read_capability().on_host()?; + let path = locked.clone_activation_path(); + let session = ActivationSession { + capability: &capability, + record: &path, + expected: ReadyActivation::of(&record), + encoded, + deadline, + timeout: deadline.saturating_duration_since(started), + since: "its reattach", + }; + session.activate(control, vmm.endpoint(), vmm.process().reference(), None) +} + +/// Returns the teardown of a termination of a clone's VMM within `bound` that ended as `outcome` +/// tells, as [`Vmm::terminate`] returns it. +fn stopped(outcome: Result, bound: Duration) -> Teardown { + match outcome { + Ok(StopOutcome::Forced) => Teardown::Terminated, + Ok(StopOutcome::Graceful) => Teardown::Exited, + Ok(StopOutcome::TimedOut) => Teardown::Unresolved(format!( + "the clone's VMM did not exit within {bound:?} of its termination" + )), + Err(error) => Teardown::Unresolved(describe(&error)), + } +} + +/// Returns `error`, which says so if `teardown` is unresolved. +fn unended(error: Error, teardown: &Teardown) -> Error { + match teardown { + Teardown::Unresolved(reason) => amended( + error, + &format!("; the clone's VMM could not be ended, and remains for a reattach: {reason}"), + ), + _ => error, + } +} + +/// Returns the error of the launch of a clone that got as far as `progress` before a crash of its +/// caller interrupted it. +fn interrupted(progress: Progress) -> Error { + Error::exited(format!( + "the launch of the clone was interrupted at its step {progress}, as a crash of its \ + caller interrupts it" + )) +} + +/// Returns the error of a clone's VM that exited once the clone got as far as `progress`, with +/// the guest's `status`, if OpenVMM's outcome report gives one. +fn exited(progress: Progress, status: Option) -> Error { + let ending = match status { + Some(status) => format!("with the guest's status {status}"), + None => "without an outcome report of the guest's own end".to_owned(), + }; + Error::exited(format!( + "the clone's VM exited at its step {progress}, {ending}" + )) +} + +/// Returns the error of a clone's VM that runs under another runtime identity, which differs in +/// `differences`. +fn mismatched(differences: &[IdentityPart]) -> Error { + let parts: Vec<&str> = differences.iter().map(|part| part.name()).collect(); + Error::new( + ErrorCode::RuntimeMismatch, + format!( + "the clone's VM runs under another runtime identity, which differs in {}, so it is \ + not driven, and is ended through its verified identity", + parts.join(", ") + ), + ) +} diff --git a/nvx-rs/src/host/template/clone/record.rs b/nvx-rs/src/host/template/clone/record.rs index f644a6fc2..9f04ab631 100644 --- a/nvx-rs/src/host/template/clone/record.rs +++ b/nvx-rs/src/host/template/clone/record.rs @@ -12,6 +12,7 @@ use serde::Serialize; use sha2::{Digest, Sha256}; use super::CloneRequest; +use super::failure::{Attribute, Evidence, Fault}; use crate::activation::{ ActivationRecord, Configuration, ExpectedResources, Identity, Image, Network, Publication, RestoreSection, TransactionId, Upper, @@ -53,33 +54,34 @@ const SCRATCH_DOMAIN: &[u8] = b"nvx scratch binding\0"; /// online, and the request's memory target. /// /// Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a request whose image -/// is not an EROFS layer, and for a configuration that no record holds, as -/// [`Configuration::check`] finds it, such as a hostname that is empty or longer than +/// is not an EROFS layer, [`Evidence::InvalidFormat`], and for a configuration that no record +/// holds, as [`Configuration::check`] finds it, such as a hostname that is empty or longer than /// [`MAX_HOSTNAME_BYTES`](crate::activation::MAX_HOSTNAME_BYTES), or a workload identity whose -/// UID or GID is 0; with [`ErrorKind::Io`](crate::ErrorKind::Io) for a key whose command line -/// names its DNS server in a form that OpenVMM does not write, as [`TemplateKey::dns_server`] -/// finds it; and with [`ErrorKind::Other`](crate::ErrorKind::Other) if the transaction ID cannot -/// be drawn. +/// UID or GID is 0, [`Evidence::InvalidActivate`]; with [`ErrorKind::Io`](crate::ErrorKind::Io) +/// for a key whose command line names its DNS server in a form that OpenVMM does not write, as +/// [`TemplateKey::dns_server`] finds it, [`Evidence::MalformedGeneration`]; and with +/// [`ErrorKind::Other`](crate::ErrorKind::Other) if the transaction ID cannot be drawn, or a +/// binding's identity computed, the host's evidence. pub(super) fn build( request: &CloneRequest, lease: &Lease, key: &TemplateKey, -) -> Result { +) -> std::result::Result { let image = match request.binding.format() { DistroFormat::Erofs => Image::Erofs { - binding: distro_identity(&request.binding)?, + binding: distro_identity(&request.binding).on_host()?, uuid: request.adapter.distro().image().identity, }, - DistroFormat::Gpt => return Err(gpt_refused()), + DistroFormat::Gpt => return Err(Fault::new(Evidence::InvalidFormat, gpt_refused())), }; let upper = match &request.scratch { None => Upper::Tmpfs, Some(binding) => Upper::Scratch { - binding: scratch_identity(binding)?, + binding: scratch_identity(binding).on_host()?, journal: request.journal, }, }; - let dns = key.dns_server()?; + let dns = key.dns_server().attribute(Evidence::MalformedGeneration)?; let network = key.network_policy_digest().map(|policy| Network { policy: Identity::from_bytes(policy), dns, @@ -103,13 +105,16 @@ pub(super) fn build( }, }; let restore = RestoreSection { - transaction: transaction()?, + transaction: transaction().on_host()?, template: lease.template(), }; ActivationRecord::new(Some(restore), configuration).map_err(|error| { - Error::unsupported(format!( - "no clone activates with the settings of the request: {error}" - )) + Fault::new( + Evidence::InvalidActivate, + Error::unsupported(format!( + "no clone activates with the settings of the request: {error}" + )), + ) }) } diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs new file mode 100644 index 000000000..b021c8b3d --- /dev/null +++ b/nvx-rs/src/host/template/clone/records.rs @@ -0,0 +1,1268 @@ +//! The records that a clone keeps in its sandbox about its launch, beside its activation record: +//! its progress log, [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), and the +//! record of its failure, [`CLONE_FAILURE_NAME`](crate::host::store::CLONE_FAILURE_NAME), which +//! [`CloneRecords::read`] reads with the clone's other records. +//! +//! # The progress log +//! +//! The launch of a clone creates its progress log, in place of one that an earlier launch left, +//! with a first line that names its format, the launch, and the boot of the host during which the +//! launch writes it: `nvx-clone-progress 1 `. The launch and its restore sequence +//! then append a line for each step of the clone's [`Progress`] as the clone reaches it, from the +//! spawn of its VMM to the commit of its launch, in order, and sync none of them: the log tells a +//! reattach how far the clone got once the caller of its launch crashed, which a crash of the +//! caller alone loses no write of. A crash of the host, which ends the clone's VM, may lose the +//! log's last lines, so how far the clone got is uncertain, once it launched, by a log that was +//! written during another boot, as by one that is damaged or missing: a workload may have run in +//! the clone. A step that another record holds, the spawn, which the launch's journal records, +//! and one that the next step follows at once, the acknowledgment, which the Activate follows, is +//! appended with that next step, in its one write, so that the log takes a write for each step +//! from the restore-ready event on. A reader ignores a last line that a crash cut short. A sandbox +//! that boots cold writes no log. +//! +//! # The failure record +//! +//! The record of a clone's failure is JSON in format [`FAILURE_FORMAT`], which the failure writes +//! before its teardown, with the teardown [pending](Teardown::Pending), and again once its +//! teardown is known, replacing a record atomically, as the sandbox's store replaces its records, +//! and durably: each write syncs the sandbox's directory once the record is in place, so that the +//! record outlasts a crash of the host, as the clone's binding does. Like the store's records, the +//! record, and the directory that a write syncs, are named by their paths: the directories above +//! the sandbox's must be ones that only the current user, or an administrator, may change, as the +//! [store](crate::host::store) requires of them. A record of another format +//! fails rather than being read. The failure of a VMM that +//! exited before its restore-ready event reads the end of the launch's log for the line with +//! which OpenVMM reported the error that ended it, which the record's message keeps, and whose +//! time ABI code may attribute the failure, as [the attribution of +//! evidence](Evidence#attribution) describes. +//! +//! A release of the clone's lease removes both, as it removes the clone's other files, so that +//! the records of a clone last as long as its lease, which a failure whose teardown is unresolved +//! keeps. + +use std::cell::OnceCell; +use std::fs::File; +use std::io::{self, Read, Seek, SeekFrom, Write}; +use std::num::NonZeroU64; +use std::path::{Path, PathBuf}; + +use serde::{Deserialize, Serialize}; + +use super::failure::{ + self, CloneFailure, Evidence, FailureRecord, Fault, Outcome, Progress, Stage, Teardown, + amended, describe, +}; +use crate::activation::{Identity, StoreGeneration, TemplateReference, TransactionId}; +use crate::error::{Error, ErrorCode, Result}; +use crate::host::platform::{self, parse_hex_id}; +use crate::host::process::BootId; +use crate::host::report::{self, Report}; +use crate::host::store::files::{create_new, sync_dir, write_private}; +use crate::host::store::regular::{open_regular, read_at_most, was_removed}; +use crate::host::store::{ + CLONE_PROGRESS_NAME, LaunchId, LaunchJournal, OUTCOME_NAME, Recovery, RuntimeRecord, + SandboxStore, +}; +use crate::host::supervisor::Starting; +use crate::host::template::{KeyIdentity, Lease}; + +/// The format of the record of a clone's failure. +pub const FAILURE_FORMAT: u32 = 1; + +/// The first word of the first line of a clone's progress log. +const PROGRESS_MAGIC: &str = "nvx-clone-progress"; + +/// The format of a clone's progress log, after [`PROGRESS_MAGIC`] in its first line. +const PROGRESS_FORMAT: u32 = 1; + +/// The most bytes of a clone's progress log that a read takes: its first line, and a line for each +/// of its steps, take a few hundred. +const MAX_PROGRESS_BYTES: u64 = 4096; + +/// The most bytes of the record of a clone's failure that a read takes: the message of an error, +/// which it holds, takes at most a few kilobytes. +const MAX_FAILURE_BYTES: u64 = 64 * 1024; + +/// What messages call the progress log of a clone. +const PROGRESS_WHAT: &str = "clone progress log"; + +/// What messages call the record of a clone's failure. +const FAILURE_WHAT: &str = "clone failure record"; + +/// The most bytes at the end of a VMM's log in which a failure looks for the line of OpenVMM's +/// fatal error, which OpenVMM writes as it exits, followed by the error's chain alone. +const FATAL_TAIL_BYTES: u64 = 64 * 1024; + +/// The most characters of the line of OpenVMM's fatal error that the record of a failure keeps. +const MAX_FATAL_CHARS: usize = 1024; + +/// The progress log of a clone's launch, open for writing, with the latest step that the clone +/// reached, and the latest that the log records, as the [module](self#the-progress-log) +/// describes. +#[derive(Debug)] +pub(super) struct ProgressLog { + /// The log, open for writing at its end. + file: File, + /// The log's path. + path: Box, + /// The latest step that the clone reached. + progress: Progress, + /// The latest step that the log records. + logged: Progress, +} + +impl ProgressLog { + /// Creates the progress log of the launch `launch` in the sandbox's directory `dir`, in place + /// of one that an earlier launch left, with its first line, which names the current boot of + /// the host, and returns it, at [`Progress::Launching`]. It first removes the outcome report + /// that an earlier VM of the sandbox left, which no acquisition, clearing of a VM's runtime + /// record, or release removes, so that a report beside the clone's records is the clone's VM's, + /// as [`CloneRecords::read`] takes it. + /// + /// `directory` holds the sandbox's directory open, as the sandbox's lock opened and checked it + /// through `dir`. The earlier report and log go through it, as a release removes the clone's + /// files, so that the removals reach the directory that holds the lock, whatever `dir` names + /// meanwhile. The log is then created at its path, where no file may exist, so that a path that + /// names another sandbox's directory by then fails the creation if that sandbox holds a log, + /// rather than replace it. + pub(super) fn create( + directory: &platform::Directory, + dir: &Path, + launch: &LaunchId, + ) -> Result { + let path = dir.join(CLONE_PROGRESS_NAME); + let failed = |error: io::Error| { + Error::io(format!( + "cannot create the {PROGRESS_WHAT} {}", + path.display() + )) + .with_source(error) + }; + let boot = BootId::current().map_err(failed)?; + for name in [OUTCOME_NAME, CLONE_PROGRESS_NAME] { + directory.remove_file(name).map_err(|error| { + Error::io(format!("cannot remove {}", dir.join(name).display())).with_source(error) + })?; + } + let mut file = create_new(&path).map_err(failed)?; + file.write_all(format!("{PROGRESS_MAGIC} {PROGRESS_FORMAT} {launch} {boot}\n").as_bytes()) + .map_err(failed)?; + Ok(Self { + file, + path: path.into(), + progress: Progress::Launching, + logged: Progress::Launching, + }) + } + + /// Returns the latest step that the clone reached. + pub(super) fn progress(&self) -> Progress { + self.progress + } + + /// Notes that the clone reached `step`, unless it reached it, or a later step, already, + /// without writing it: the log records it with the next step that it writes. It is for a + /// step that another record holds until then, as the launch's journal holds the spawn of its + /// VMM, or that the next step follows at once. + pub(super) fn advance(&mut self, step: Progress) { + self.progress = self.progress.max(step); + } + + /// Records that the clone reached `step`, unless it reached it, or a later step, already: + /// appends the line of each step that the clone reached since the latest that the log + /// records, through `step`, in order, with one write, which nothing syncs. + pub(super) fn reach(&mut self, step: Progress) -> Result<()> { + self.advance(step); + if self.progress <= self.logged { + return Ok(()); + } + let lines: String = LOGGED + .iter() + .filter(|&&logged| self.logged < logged && logged <= self.progress) + .map(|logged| format!("{logged}\n")) + .collect(); + self.file.write_all(lines.as_bytes()).map_err(|error| { + Error::io(format!( + "cannot record that the clone reached its step {} in the {PROGRESS_WHAT} {}", + self.progress, + self.path.display() + )) + .with_source(error) + })?; + self.logged = self.progress; + Ok(()) + } +} + +/// The steps that a progress log records after its first line, in order. +const LOGGED: [Progress; 8] = [ + Progress::Spawned, + Progress::Event, + Progress::Acknowledged, + Progress::Activating, + Progress::Ready, + Progress::Starting, + Progress::Started, + Progress::Committed, +]; + +/// How the launch of a clone that failed is ended: [`Starting::abandon`], outside tests, which +/// replace it to keep a launch that cannot be ended. +pub(super) type End = for<'a> fn(Starting<'a>) -> Result; + +/// Ends `starting` as [`Starting::abandon`] does: how the launch of a clone that failed is ended +/// outside tests. +pub(super) fn abandon(starting: Starting<'_>) -> Result { + starting.abandon() +} + +/// What a clone's failures are recorded with: the clone's lease and the transaction of its +/// restore, the paths, in its sandbox, of the record of its failure, of OpenVMM's outcome report +/// of its VM, and of its VMM's log, whether a workload may have run in the clone whatever its +/// records say, as for a clone whose progress log was damaged, or written during another boot, +/// and the report, once it was read. +#[derive(Debug)] +pub(super) struct Recorder { + /// The clone's lease. + lease: Lease, + /// The transaction ID of the clone's restore. + transaction: Option, + /// The path of the record of the clone's failure. + failure: PathBuf, + /// The path of OpenVMM's outcome report of the clone's VM. + report: PathBuf, + /// The path of the log of the clone's VMM. + log: PathBuf, + /// Whether how far the clone got is uncertain beyond the progress that its records give, so + /// that a workload may have run in it. + uncertain: bool, + /// OpenVMM's outcome report of the clone's VM, as [`observed`](Self::observed) first read it. + observed: OnceCell>, +} + +impl Recorder { + /// Returns the recorder of the failures of the clone of `lease`, with the transaction of its + /// restore, `transaction`, in the sandbox of `sandbox`. + pub(super) fn new( + lease: Lease, + transaction: Option, + sandbox: &SandboxStore, + ) -> Self { + Self { + lease, + transaction, + failure: sandbox.clone_failure_path(), + report: sandbox.outcome_path(), + log: sandbox.log_path(), + uncertain: false, + observed: OnceCell::new(), + } + } + + /// Returns the recorder, for a clone whose progress is uncertain beyond what its records + /// give, if `uncertain`: a status of the guest's is then attributed as once a workload may + /// have run, as [`reported`](Self::reported) does. + pub(super) fn uncertain(self, uncertain: bool) -> Self { + Self { uncertain, ..self } + } + + /// Returns OpenVMM's outcome report of the clone's VM, or `None` if it is missing or + /// malformed, as the recorder first read it, the first time that it needed it: once the VM + /// has ended, whose report is final then, so that a read that fails later, as a read that + /// another process's open of the report refuses may, takes nothing back of what the first + /// one showed. A recorder serves one step, a launch that fails or a reattach, which reads the + /// report only once the clone's VM ended, or its teardown did. + pub(super) fn observed(&self) -> Option<&Report> { + self.observed + .get_or_init(|| report::read(&self.report)) + .as_ref() + } + + /// Returns the guest's status in OpenVMM's outcome report `report` of the clone's VM, and the + /// evidence that the status alone is once the clone got as far as `progress`, as + /// [`failure::reported`] gives them, or as once a workload may have run, if the clone's + /// progress is [uncertain](Self::uncertain). + pub(super) fn reported( + &self, + report: Option<&Report>, + progress: Progress, + ) -> (Option, Option) { + let progress = if self.uncertain { + progress.max(Progress::Starting) + } else { + progress + }; + failure::reported(report, progress) + } + + /// Records the failure of the clone at `stage`, once it got as far as `progress`, with + /// `fault`, whose teardown ended as `teardown` tells, as [the records of + /// failures](FailureRecord#records) describe, attributed as [`attribute`](Self::attribute) + /// does, and returns it. A failure that cannot be recorded says so. + pub(super) fn record( + &self, + stage: Stage, + progress: Progress, + fault: Fault, + teardown: Teardown, + ) -> CloneFailure { + let (evidence, status, error) = self.attribute(progress, fault, &teardown); + let record = FailureRecord::new( + &self.lease, + self.transaction, + (stage, progress), + evidence, + status, + teardown, + &error, + ); + self.write(error, record) + } + + /// Finishes `earlier`, the record of a failure whose teardown a crash of the caller left + /// [pending](Teardown::Pending), once its teardown ended as `teardown` tells, with the + /// failure's `error`, as the failure would have recorded it then: attributed from its + /// evidence, as [`attribute`](Self::attribute) does, keeping its lease, generation, launch, + /// transaction, stage, and progress, and the guest's status that it holds, if the outcome + /// report gives none now. Writes the record in place of the earlier one, and returns the + /// failure. + pub(super) fn finish( + &self, + earlier: FailureRecord, + error: Error, + teardown: Teardown, + ) -> CloneFailure { + let fault = Fault::new(earlier.evidence, error); + let (evidence, status, error) = self.attribute(earlier.progress, fault, &teardown); + let record = FailureRecord { + evidence, + status: status.or(earlier.status), + outcome: evidence.outcome(), + teardown, + code: error.code(), + message: describe(&error), + ..earlier + }; + self.write(error, record) + } + + /// Returns the evidence, the guest's status, and the error of the failure of the clone with + /// `fault`, once it got as far as `progress`, whose teardown ended as `teardown` tells: the + /// loss of a VMM that exited before its restore-ready event takes the line of OpenVMM's fatal + /// error that the VMM's log ends with, as [`diagnosed`](Self::diagnosed) does, and the + /// evidence of a status of the guest's, which OpenVMM's outcome report gives once the VMM has + /// exited, as [`observed`](Self::observed) reads it, prevails over the fault's, as + /// [`reported`](Self::reported) gives it. + fn attribute( + &self, + progress: Progress, + fault: Fault, + teardown: &Teardown, + ) -> (Evidence, Option, Error) { + let fault = self.diagnosed(progress, fault); + let report = match teardown { + Teardown::NoVm => None, + _ => self.observed(), + }; + let (reported, status) = self.reported(report, progress); + (reported.unwrap_or(fault.evidence), status, fault.error) + } + + /// Returns `fault`, of the clone that got as far as `progress`, which is the loss of a VMM + /// that exited before its restore-ready event, with the line with which OpenVMM reported the + /// error that ended the VMM after its message, if the VMM's log ends with one, as + /// [`read_fatal`] reads it, and the message does not hold it yet, as the message of a failure + /// that a reattach resolves may, and with the line's evidence, if it is evidence, as [the + /// attribution of evidence](Evidence#attribution) describes. Returns any other fault as it is. + fn diagnosed(&self, progress: Progress, fault: Fault) -> Fault { + if fault.evidence != Evidence::LostStatus || progress >= Progress::Event { + return fault; + } + let Some(line) = read_fatal(&self.log) else { + return fault; + }; + let evidence = failure::fatal_evidence(&line).unwrap_or(fault.evidence); + let reported = format!("; OpenVMM reported: {line}"); + let error = if describe(&fault.error).contains(&reported) { + fault.error + } else { + amended(fault.error, &reported) + }; + Fault::new(evidence, error) + } + + /// Records the failure of the clone at `stage`, once it got as far as `progress`, with + /// `evidence`, the guest's `status`, if the failure knows it already, and `error`, before the + /// teardown that follows it, whose record is [`Teardown::Pending`], so that a crash of the + /// caller during the teardown leaves the record, which a reattach finishes, as + /// [`reattach_clone`](super::reattach_clone) describes. The record that the failure writes + /// once its teardown ended replaces it. A record that cannot be written now is written then, + /// if it can be. + pub(super) fn pending( + &self, + stage: Stage, + progress: Progress, + (evidence, status): (Evidence, Option), + error: &Error, + ) { + let record = FailureRecord::new( + &self.lease, + self.transaction, + (stage, progress), + evidence, + status, + Teardown::Pending, + error, + ); + let _ = write_failure(&self.failure, &record); + } + + /// Resolves `earlier`, the record of a failure that its teardown left unresolved, with + /// `teardown`, which ended the clone's VMM, keeping its stage, evidence, outcome, code, and + /// message, unless what the VMM left as it ended tells more: the line of OpenVMM's fatal + /// error, for the loss of a VMM that exited before its restore-ready event, as + /// [`diagnosed`](Self::diagnosed) adds it, and the guest's status in OpenVMM's outcome + /// report, which the record does not hold, whose evidence, as [`reported`](Self::reported) + /// gives it, prevails, as in [`record`](Self::record). Writes the record in place of the + /// earlier one, if they differ, and returns the failure. + pub(super) fn resolve(&self, mut earlier: FailureRecord, teardown: Teardown) -> CloneFailure { + let error = earlier.error(); + if earlier.teardown == teardown { + return CloneFailure::new(error, earlier); + } + let fault = self.diagnosed(earlier.progress, Fault::new(earlier.evidence, error)); + let mut evidence = fault.evidence; + if earlier.status.is_none() && teardown != Teardown::NoVm { + let (reported, status) = self.reported(self.observed(), earlier.progress); + evidence = reported.unwrap_or(evidence); + earlier.status = status; + } + if evidence != earlier.evidence { + earlier.evidence = evidence; + earlier.outcome = evidence.outcome(); + } + earlier.message = describe(&fault.error); + earlier.teardown = teardown; + self.write(fault.error, earlier) + } + + /// Writes `record`, the record of the failure of `error`, in place of the sandbox's record, + /// and returns the failure, whose error says so if the record could not be written. + pub(super) fn write(&self, error: Error, record: FailureRecord) -> CloneFailure { + let error = match write_failure(&self.failure, &record) { + Ok(()) => error, + Err(failed) => amended( + error, + &format!("; the failure could not be recorded: {}", describe(&failed)), + ), + }; + CloneFailure::new(error, record) + } +} + +/// What the launch of a clone keeps to record how far the clone got and how it failed: the +/// [`Recorder`] of its failures, its progress log, and how a failure ends the launch. +#[derive(Debug)] +pub(super) struct Tracker { + /// The recorder of the clone's failures. + recorder: Recorder, + /// The clone's progress log. + log: ProgressLog, + /// How a failure ends the launch. + end: End, +} + +impl Tracker { + /// Returns the tracker of the launch of a clone whose failures `recorder` records, with its + /// progress `log`, and the end of a launch that fails, `end`. + pub(super) fn new(recorder: Recorder, log: ProgressLog, end: End) -> Self { + Self { recorder, log, end } + } + + /// Notes that the clone reached `step`, as [`ProgressLog::advance`] does, without writing it. + pub(super) fn advance(&mut self, step: Progress) { + self.log.advance(step); + } + + /// Records that the clone reached `step` in its progress log, as [`ProgressLog::reach`] does, + /// and returns the fault of a failure to record it, the host's. + pub(super) fn reach(&mut self, step: Progress) -> std::result::Result<(), Fault> { + self.log.reach(step).map_err(Fault::host) + } + + /// Records that the clone reached `step` in its progress log if it can: a step that another + /// record holds, such as the commit, which the runtime record holds while the VM runs. + pub(super) fn note(&mut self, step: Progress) { + let _ = self.log.reach(step); + } + + /// Records the failure of the clone at `stage` with `evidence` and `error` before the teardown + /// that follows it, once the clone got as far as its progress, as [`Recorder::pending`] does. + pub(super) fn pending(&self, stage: Stage, evidence: Evidence, error: &Error) { + self.recorder + .pending(stage, self.log.progress(), (evidence, None), error); + } + + /// Ends `starting`, the launch of the clone, which failed at `stage` with `fault`, as `end` + /// does, and records the failure before the teardown, as [`pending`](Self::pending) does, and + /// with the teardown's outcome, as [`record`](Self::record) does. The error of a launch that + /// could not be ended says so, and that its journal remains for recovery. + pub(super) fn fail(&self, starting: Starting<'_>, stage: Stage, fault: Fault) -> CloneFailure { + self.pending(stage, fault.evidence, &fault.error); + let teardown = Teardown::of((self.end)(starting)); + let fault = match &teardown { + Teardown::Unresolved(reason) => Fault::new( + fault.evidence, + amended( + fault.error, + &format!("; the launch could not be ended, and remains for recovery: {reason}"), + ), + ), + _ => fault, + }; + self.record(stage, fault, teardown) + } + + /// Records the failure of the clone at `stage` with `fault`, whose teardown ended as + /// `teardown` tells, once the clone got as far as its progress, as [`Recorder::record`] + /// does, and returns it. + pub(super) fn record(&self, stage: Stage, fault: Fault, teardown: Teardown) -> CloneFailure { + self.recorder + .record(stage, self.log.progress(), fault, teardown) + } +} + +/// What a clone's progress log records. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct Logged { + /// The latest step that the log records. + pub(super) progress: Progress, + /// The boot of the host during which the launch wrote the log, as its first line names it, + /// unless a crash cut that line short. + pub(super) boot: Option, +} + +impl Logged { + /// Returns whether the log tells for sure how far the clone got: whether the launch wrote it + /// during the boot `current`, the current boot of the host, if it is known, so that no crash + /// of the host lost a line of it since, as [the module](self#the-progress-log) describes. + pub(super) fn is_certain(&self, current: Option) -> bool { + self.boot.is_some() && self.boot == current + } +} + +/// Reads the progress log at `path` of the launch `launch`, and returns what it records, or +/// `None` if it does not exist. +/// +/// Fails with [`ErrorKind::Io`](crate::ErrorKind::Io) for a log that cannot be read, is not a +/// private regular file, names another launch or format, or no boot, or records other lines than +/// its steps, in order, besides a last line that a crash cut short. +pub(super) fn read_progress(path: &Path, launch: &LaunchId) -> Result> { + let Some(bytes) = read_private(path, PROGRESS_WHAT, MAX_PROGRESS_BYTES)? else { + return Ok(None); + }; + let malformed = + |problem: String| Error::io(format!("the {PROGRESS_WHAT} {} {problem}", path.display())); + let text = String::from_utf8(bytes).map_err(|_| malformed("is not UTF-8".to_owned()))?; + // A line without its newline is one that a crash cut short. + let mut lines = text + .split_inclusive('\n') + .filter_map(|line| line.strip_suffix('\n')); + let header = format!("{PROGRESS_MAGIC} {PROGRESS_FORMAT} {launch} "); + let boot = match lines.next() { + Some(line) => match line + .strip_prefix(&header) + .and_then(|boot| boot.parse::().ok()) + { + Some(boot) => boot, + None => { + return Err(malformed(format!( + "begins with {line:?}, not with the first line of the launch {launch}, \ + {header:?} followed by the identity of a boot" + ))); + } + }, + // A log that a crash cut short before its first line ended records the launch alone. + None => { + return Ok(Some(Logged { + progress: Progress::Launching, + boot: None, + })); + } + }; + let mut progress = Progress::Launching; + for line in lines { + match LOGGED.iter().find(|step| step.as_str() == line) { + Some(&step) if step > progress => progress = step, + _ => { + return Err(malformed(format!( + "records {line:?} after the step {progress}, which is not a later step" + ))); + } + } + } + Ok(Some(Logged { + progress, + boot: Some(boot), + })) +} + +/// Returns how far the clone of `lease` got, by the latest step that its progress log records, +/// `logged`, and by the store's records of its launch: its `journal`, which records the spawn of +/// its VMM, and its `runtime` record, which records the commit of its launch. +pub(super) fn progress_of( + lease: &Lease, + journal: Option<&LaunchJournal>, + runtime: Option<&RuntimeRecord>, + logged: Option, +) -> Progress { + let launch = lease.launch(); + let recorded = if runtime.is_some_and(|record| record.launch() == launch) { + Some(Progress::Committed) + } else { + journal + .filter(|journal| journal.launch() == launch) + .map(|journal| match journal.process() { + Some(_) => Progress::Spawned, + None => Progress::Launching, + }) + }; + [logged, recorded] + .into_iter() + .flatten() + .max() + .unwrap_or(Progress::Planned) +} + +/// The record of a clone's failure, as JSON. +#[derive(Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct Recorded { + /// [`FAILURE_FORMAT`]. + format: u32, + /// The identity of the template store that leased the clone's generation. + store: String, + /// The identity of the clone's lease. + lease: String, + /// The leased generation. + generation: u64, + /// The identity of the leased generation's key. + key: String, + /// The launch that the lease reserves. + launch: String, + /// The transaction ID of the clone's restore, in hexadecimal, once the plan drew it. + #[serde(default, skip_serializing_if = "Option::is_none")] + transaction: Option, + /// The stage at which the clone failed. + stage: Stage, + /// How far the clone got. + progress: Progress, + /// The evidence of the failure. + evidence: Evidence, + /// The guest's status, if OpenVMM's outcome report has one. + #[serde(default, skip_serializing_if = "Option::is_none")] + status: Option, + /// The original outcome of the failure. + outcome: Outcome, + /// How the teardown ended, as [`Teardown::as_str`] spells it. + teardown: String, + /// Why the teardown is unresolved, if it is. + #[serde(default, skip_serializing_if = "Option::is_none")] + unresolved: Option, + /// The code of the failure's error, as [`ErrorCode::as_str`] spells it. + code: String, + /// The message of the failure's error. + message: String, +} + +impl Recorded { + /// Returns the JSON form of `record`. + fn new(record: &FailureRecord) -> Self { + Self { + format: FAILURE_FORMAT, + store: record.store.to_string(), + lease: record.lease.to_string(), + generation: record.template.generation.get().get(), + key: KeyIdentity::from(record.template.key).to_string(), + launch: record.launch.to_string(), + transaction: record + .transaction + .map(|transaction| transaction.to_string()), + stage: record.stage, + progress: record.progress, + evidence: record.evidence, + status: record.status, + outcome: record.outcome, + teardown: record.teardown.as_str().to_owned(), + unresolved: match &record.teardown { + Teardown::Unresolved(reason) => Some(reason.clone()), + _ => None, + }, + code: record.code.as_str().to_owned(), + message: record.message.clone(), + } + } + + /// Checks the record, and returns it, or says why it is malformed. + fn into_record(self) -> std::result::Result { + let field = + |what: &str, error: Error| format!("its {what} is invalid: {}", error.message()); + let teardown = match (self.teardown.as_str(), self.unresolved) { + ("no-vm", None) => Teardown::NoVm, + ("exited", None) => Teardown::Exited, + ("terminated", None) => Teardown::Terminated, + ("pending", None) => Teardown::Pending, + ("unresolved", Some(reason)) => Teardown::Unresolved(reason), + (teardown, _) => { + return Err(format!( + "its teardown, {teardown:?}, is not one that a failure records, or names a \ + reason only when it is unresolved" + )); + } + }; + let transaction = match self.transaction { + None => None, + Some(digits) => Some(TransactionId::from_bytes( + parse_hex_id(&digits).ok_or_else(|| { + format!("its transaction, {digits:?}, is not 32 hexadecimal digits") + })?, + )), + }; + let key: KeyIdentity = self.key.parse().map_err(|error| field("key", error))?; + // A later build may record a code that this one does not know, which is a failure all + // the same. + let code = ErrorCode::ALL + .iter() + .copied() + .find(|code| code.as_str() == self.code) + .unwrap_or(ErrorCode::Other); + Ok(FailureRecord { + store: self.store.parse().map_err(|error| field("store", error))?, + lease: self.lease.parse().map_err(|error| field("lease", error))?, + template: TemplateReference { + generation: NonZeroU64::new(self.generation) + .map(StoreGeneration::new) + .ok_or("it names store generation 0, which no generation has")?, + key: Identity::from(key), + }, + launch: LaunchId::try_from(self.launch) + .map_err(|problem| format!("its launch is invalid: {problem}"))?, + transaction, + stage: self.stage, + progress: self.progress, + evidence: self.evidence, + status: self.status, + outcome: self.outcome, + teardown, + code, + message: self.message, + }) + } +} + +/// Replaces the record of a clone's failure at `path` with `record`, atomically, and durably: once +/// the record is in place, it syncs the record's directory, as [`sync_dir`] does, and fails if the +/// sync does, though the record is in place then. +pub(super) fn write_failure(path: &Path, record: &FailureRecord) -> Result<()> { + write_failure_with(path, record, &sync_dir) +} + +/// Replaces the record of a clone's failure as [`write_failure`] does, syncing its directory with +/// `sync`, which tests observe, and fail. +fn write_failure_with( + path: &Path, + record: &FailureRecord, + sync: &dyn Fn(&Path) -> Result<()>, +) -> Result<()> { + let mut text = serde_json::to_vec_pretty(&Recorded::new(record)).map_err(|error| { + Error::other(format!("cannot encode the {FAILURE_WHAT}")).with_source(error) + })?; + text.push(b'\n'); + write_private(path, &text)?; + match path.parent() { + Some(dir) if !dir.as_os_str().is_empty() => sync(dir), + _ => sync(Path::new(".")), + } +} + +/// Reads the record of a clone's failure at `path`, or returns `None` if it does not exist. +/// +/// Fails with [`ErrorCode::FormatUnsupported`] for a record of another format than +/// [`FAILURE_FORMAT`], and with [`ErrorKind::Io`](crate::ErrorKind::Io) for one that cannot be +/// read, is not a private regular file, or is malformed. +pub(super) fn read_failure(path: &Path) -> Result> { + let Some(bytes) = read_private(path, FAILURE_WHAT, MAX_FAILURE_BYTES)? else { + return Ok(None); + }; + let malformed = || format!("the {FAILURE_WHAT} {} is malformed", path.display()); + let value: serde_json::Value = serde_json::from_slice(&bytes) + .map_err(|error| Error::io(malformed()).with_source(error))?; + match value.get("format") { + Some(format) if format.as_u64() == Some(FAILURE_FORMAT.into()) => {} + Some(format) => { + return Err(Error::new( + ErrorCode::FormatUnsupported, + format!( + "the {FAILURE_WHAT} {} has the unsupported format {format}; this build reads \ + format {FAILURE_FORMAT} only", + path.display() + ), + )); + } + None => return Err(Error::io(format!("{}: it has no format", malformed()))), + } + let recorded: Recorded = + serde_json::from_value(value).map_err(|error| Error::io(malformed()).with_source(error))?; + recorded + .into_record() + .map(Some) + .map_err(|problem| Error::io(format!("{}: {problem}", malformed()))) +} + +/// Reads the private regular file at `path`, the clone's `description`, of at most `limit` bytes, +/// or returns `None` if it does not exist, or was removed as this opened it, as [`was_removed`] +/// tells. +fn read_private(path: &Path, description: &str, limit: u64) -> Result>> { + let failed = |error: io::Error| { + Error::io(format!("cannot read the {description} {}", path.display())).with_source(error) + }; + let file = match open_regular(path) { + Ok(file) => file, + Err(error) if error.kind() == io::ErrorKind::NotFound || was_removed(path, &error) => { + return Ok(None); + } + Err(error) => return Err(failed(error)), + }; + if let Some(problem) = platform::private_file_problem(&file).map_err(failed)? { + return Err(Error::io(format!( + "the {description} {} {problem}", + path.display() + ))); + } + read_at_most(file, limit, &format!("the {description}")) + .map(Some) + .map_err(failed) +} + +/// Returns the last line with which OpenVMM reported the error that ended it, among the last +/// [`FATAL_TAIL_BYTES`] of the VMM's log at `path`, cut to [`MAX_FATAL_CHARS`], or none if they +/// hold none, or the log cannot be read: the line that leads OpenVMM's fatal error, then the +/// error's time ABI code, if it has one, as [the attribution of evidence](Evidence#attribution) +/// describes. +fn read_fatal(path: &Path) -> Option { + let mut log = open_regular(path).ok()?; + let length = log.metadata().ok()?.len(); + log.seek(SeekFrom::Start(length.saturating_sub(FATAL_TAIL_BYTES))) + .ok()?; + let mut tail = Vec::new(); + log.take(FATAL_TAIL_BYTES).read_to_end(&mut tail).ok()?; + let text = String::from_utf8_lossy(&tail); + let line = text + .lines() + .rev() + .find(|line| line.starts_with(failure::FATAL_PREFIX))?; + Some(line.chars().take(MAX_FATAL_CHARS).collect()) +} + +/// What the sandbox of a clone keeps of it, as [`CloneRecords::read`] reads it: the lease that its +/// clone binding records, the store's records of its launch and of its VM, its progress, the +/// record of its failure, and OpenVMM's outcome report of its VM. +/// +/// The records tell the state of a clone, as an operation that lists, inspects, or removes +/// clones reads it: +/// +/// - a [`journal`](Self::journal) without a [`runtime`](Self::runtime) record: a launch that runs, +/// under the lock of the sandbox, or one that was interrupted, which +/// [`reattach_clone`](super::reattach_clone) ends; +/// - a runtime record: a VM that may run, which a reattach holds again, or a stop ends; +/// - a [`failure`](Self::failure): a clone that failed, whose teardown is unresolved while the +/// journal or the runtime record remains, and [pending](Teardown::Pending) if a crash of the +/// caller interrupted its teardown, which a reattach finishes; +/// - neither a journal nor a runtime record: no VM of the clone runs, and a release ends the +/// lease, which removes the clone's records, before a removal of the sandbox. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct CloneRecords { + /// The lease that the sandbox's clone binding records. + pub lease: Lease, + /// The journal of the clone's launch, while it runs, or once it was interrupted. + pub journal: Option, + /// The runtime record of the clone's VM, from the commit of its launch until a stop, a + /// termination, or a reattach verifies that the VM exited. + pub runtime: Option, + /// How far the clone got, as its progress log and the store's records of its launch tell. + pub progress: Progress, + /// The record of the clone's failure, if it failed from its launch on. + pub failure: Option, + /// OpenVMM's outcome report of the clone's VM, once its VMM wrote one, and none before the + /// clone got past its plan, as [`read`](Self::read) tells. + pub report: Option, +} + +impl CloneRecords { + /// Reads the records of the clone of the sandbox of `sandbox`, or returns `None` if the + /// sandbox holds no clone binding. + /// + /// It takes no lock, as an inspection of the sandbox does not: each record is read whole, as + /// the latest transition that replaced it left it, and the records may come from transitions + /// that ran between the reads. A caller that holds the sandbox's lock reads them at one point + /// of its lifecycle. Fails as the reads of the clone binding, [`Lease::of`], of the store's + /// records, of the progress log, and of the failure record fail; an outcome report that is + /// missing or malformed is none. + /// + /// It reads the outcome report only once the clone got past its plan, as the progress log of + /// its launch, the launch's journal, the runtime record of its VM, or the record of a failure + /// past the plan shows. Before the launch writes the first of those, its progress log, it + /// removes the report that an earlier VM of the sandbox left, which no acquisition, clearing + /// of a VM's runtime record, or release removes, and which is not the clone's. + pub fn read(sandbox: &SandboxStore) -> Result> { + let Some(lease) = Lease::of(sandbox)? else { + return Ok(None); + }; + let journal = sandbox.journal()?; + let runtime = sandbox.runtime()?; + let logged = read_progress(&sandbox.clone_progress_path(), lease.launch())? + .map(|logged| logged.progress); + let failure = read_failure(&sandbox.clone_failure_path())?; + // A failure records how far the clone got, which the store's records of its launch no + // longer tell once the failure ended the launch. + let progress = progress_of(&lease, journal.as_ref(), runtime.as_ref(), logged).max( + failure + .as_ref() + .map_or(Progress::Planned, |failure| failure.progress), + ); + // The launch removed the report of an earlier VM of the sandbox before it wrote the first + // of the clone's records, which a progress past the plan shows. + let report = (progress > Progress::Planned) + .then(|| report::read(&sandbox.outcome_path())) + .flatten(); + Ok(Some(Self { + lease, + journal, + runtime, + progress, + failure, + report, + })) + } +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + use crate::error::ErrorKind; + use crate::host::store::CLONE_FAILURE_NAME; + use crate::host::template::{LeaseId, StoreId}; + + /// Returns a directory private to the current user, as a sandbox's is, whose files are + /// private too, in a temporary directory, which removes it as it drops. + fn private_dir() -> (tempfile::TempDir, PathBuf) { + let directory = tempfile::tempdir().unwrap(); + let private = directory.path().join("sandbox"); + assert_eq!(platform::create_owner_only_dir(&private).unwrap(), None); + (directory, private) + } + + /// Returns the record of a failure of a clone whose teardown ended as `teardown`. + fn failure(teardown: Teardown) -> FailureRecord { + FailureRecord { + store: StoreId::from_bytes([1; 16]), + lease: LeaseId::from_bytes([2; 16]), + template: TemplateReference { + generation: StoreGeneration::new(NonZeroU64::new(7).unwrap()), + key: Identity::from_bytes([3; 32]), + }, + launch: LaunchId::try_from("4".repeat(32)).unwrap(), + transaction: Some(TransactionId::from_bytes([5; 16])), + stage: Stage::Activation, + progress: Progress::Activating, + evidence: Evidence::UnusableRoot, + status: Some(54), + outcome: Outcome::Request, + teardown, + code: ErrorCode::Exited, + message: "the clone's VMM exited".to_owned(), + } + } + + #[test] + fn a_progress_log_records_each_step_once_with_the_steps_that_it_deferred() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_PROGRESS_NAME); + let launch = LaunchId::generate().unwrap(); + let boot = BootId::current().unwrap(); + let logged = |progress| { + Some(Logged { + progress, + boot: Some(boot), + }) + }; + // The log of an earlier launch goes, and so does the report of an earlier VM. + write_private(&path, b"left").unwrap(); + let report = private.join(OUTCOME_NAME); + write_private(&report, b"left").unwrap(); + let directory = platform::Directory::open(&private).unwrap(); + let mut log = ProgressLog::create(&directory, &private, &launch).unwrap(); + assert!(!report.exists()); + assert_eq!( + read_progress(&path, &launch).unwrap(), + logged(Progress::Launching) + ); + // A step that another record holds waits for the next step that the log writes. + log.advance(Progress::Spawned); + assert_eq!(log.progress(), Progress::Spawned); + assert_eq!( + read_progress(&path, &launch).unwrap(), + logged(Progress::Launching) + ); + log.reach(Progress::Event).unwrap(); + log.advance(Progress::Acknowledged); + log.reach(Progress::Activating).unwrap(); + // A step that the clone reached already is not recorded again. + log.reach(Progress::Event).unwrap(); + log.advance(Progress::Spawned); + assert_eq!(log.progress(), Progress::Activating); + assert_eq!( + fs::read_to_string(&path).unwrap(), + format!( + "{PROGRESS_MAGIC} 1 {launch} {boot}\nspawned\nevent\nacknowledged\nactivating\n" + ) + ); + assert_eq!( + read_progress(&path, &launch).unwrap(), + logged(Progress::Activating) + ); + } + + #[test] + fn a_progress_log_replaces_an_earlier_one_only_in_the_directory_that_it_holds_open() { + let (temporary, private) = private_dir(); + let launch = LaunchId::generate().unwrap(); + let directory = platform::Directory::open(&private).unwrap(); + write_private(&private.join(CLONE_PROGRESS_NAME), b"left").unwrap(); + write_private(&private.join(OUTCOME_NAME), b"left").unwrap(); + // The sandbox's path names another sandbox's directory by the time that the log is + // created: the earlier log and report of the directory that the handle holds go, and the + // other sandbox's stay, whose log fails the creation. + let moved = temporary.path().join("moved"); + fs::rename(&private, &moved).unwrap(); + let other = temporary.path().join("other"); + assert_eq!(platform::create_owner_only_dir(&other).unwrap(), None); + for name in [CLONE_PROGRESS_NAME, OUTCOME_NAME] { + write_private(&other.join(name), b"the other's").unwrap(); + } + fs::rename(&other, &private).unwrap(); + let error = ProgressLog::create(&directory, &private, &launch).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + for name in [CLONE_PROGRESS_NAME, OUTCOME_NAME] { + assert!(!moved.join(name).exists(), "{name}"); + assert_eq!( + fs::read(private.join(name)).unwrap(), + b"the other's", + "{name}" + ); + } + } + + #[test] + fn a_progress_log_that_a_crash_cut_short_is_read_and_one_of_another_launch_is_refused() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_PROGRESS_NAME); + let launch = LaunchId::generate().unwrap(); + assert_eq!(read_progress(&path, &launch).unwrap(), None); + let (boot, other) = (BootId::current().unwrap(), BootId::from_bytes([0xee; 16])); + let header = format!("{PROGRESS_MAGIC} 1 {launch} {boot}\n"); + // A last line that a crash cut short is not a step, nor is the first line, without which + // the log names no boot. A log of another boot is read, with its boot, which a crash of + // the host may have cut short. + for (text, progress, written) in [ + (format!("{header}event\nackno"), Progress::Event, Some(boot)), + (header[..10].to_owned(), Progress::Launching, None), + ( + header[..header.len() - 1].to_owned(), + Progress::Launching, + None, + ), + (String::new(), Progress::Launching, None), + ( + format!("{header}committed\n"), + Progress::Committed, + Some(boot), + ), + ( + format!("{PROGRESS_MAGIC} 1 {launch} {other}\nevent\n"), + Progress::Event, + Some(other), + ), + ] { + write_private(&path, text.as_bytes()).unwrap(); + let read = read_progress(&path, &launch).unwrap().unwrap(); + assert_eq!((read.progress, read.boot), (progress, written), "{text:?}"); + // Only a log of the current boot tells for sure how far the clone got. + assert_eq!( + read.is_certain(Some(boot)), + written == Some(boot), + "{text:?}" + ); + assert!(!read.is_certain(None), "{text:?}"); + } + // Another launch's log, another format, a first line without a boot, and steps out of + // their order, or that no log records, are refused. + let other = LaunchId::generate().unwrap(); + for text in [ + format!("{PROGRESS_MAGIC} 1 {other} {boot}\nevent\n"), + format!("{PROGRESS_MAGIC} 2 {launch} {boot}\n"), + format!("{PROGRESS_MAGIC} 1 {launch}\n"), + format!("{PROGRESS_MAGIC} 1 {launch} the-boot\n"), + format!("{header}ready\nevent\n"), + format!("{header}event\nevent\n"), + format!("{header}planned\n"), + format!("{header}launching\n"), + format!("{header}event\nbogus\n"), + ] { + write_private(&path, text.as_bytes()).unwrap(); + let error = read_progress(&path, &launch).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{text:?}: {error}"); + } + } + + #[test] + fn a_failure_record_is_read_as_it_was_written() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + assert_eq!(read_failure(&path).unwrap(), None); + let unresolved = Teardown::Unresolved("its VMM did not exit".to_owned()); + for teardown in [ + Teardown::NoVm, + Teardown::Exited, + Teardown::Terminated, + unresolved, + ] { + let written = failure(teardown); + write_failure(&path, &written).unwrap(); + assert_eq!(read_failure(&path).unwrap(), Some(written)); + } + // Its JSON spells each field as the record's types do. + let value: serde_json::Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + assert_eq!( + value, + serde_json::json!({ + "format": 1, + "store": "01".repeat(16), + "lease": "02".repeat(16), + "generation": 7, + "key": "03".repeat(32), + "launch": "4".repeat(32), + "transaction": "05".repeat(16), + "stage": "activation", + "progress": "activating", + "evidence": "unusable-root", + "status": 54, + "outcome": "request", + "teardown": "unresolved", + "unresolved": "its VMM did not exit", + "code": "exited", + "message": "the clone's VMM exited", + }) + ); + // A code that a later build records, which this one does not know, is still a failure. + let mut later = value.clone(); + later["code"] = "a-later-code".into(); + write_private(&path, &serde_json::to_vec(&later).unwrap()).unwrap(); + assert_eq!(read_failure(&path).unwrap().unwrap().code, ErrorCode::Other); + } + + #[test] + fn a_failure_record_is_in_place_when_its_directory_is_synced() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + // Each write syncs the record's directory once the record is in place, so that the sync + // makes the record's rename durable. + let synced = std::cell::RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced + .borrow_mut() + .push((dir.to_owned(), read_failure(&path).unwrap())); + sync_dir(dir) + }; + let (pending, exited) = (failure(Teardown::Pending), failure(Teardown::Exited)); + write_failure_with(&path, &pending, &sync).unwrap(); + write_failure_with(&path, &exited, &sync).unwrap(); + assert_eq!( + synced.into_inner(), + [ + (private.clone(), Some(pending)), + (private.clone(), Some(exited)) + ] + ); + // A sync that fails fails the write, whose record is in place, though maybe not durably. + let refused = |_: &Path| -> Result<()> { Err(Error::io("the test's sync failed")) }; + let terminated = failure(Teardown::Terminated); + let error = write_failure_with(&path, &terminated, &refused).unwrap_err(); + assert_eq!(error.message(), "the test's sync failed"); + assert_eq!(read_failure(&path).unwrap(), Some(terminated)); + } + + #[test] + fn a_failure_record_of_another_format_or_a_malformed_one_is_refused() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + write_failure(&path, &failure(Teardown::Exited)).unwrap(); + let value: serde_json::Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + let changed = |change: &dyn Fn(&mut serde_json::Value)| { + let mut value = value.clone(); + change(&mut value); + write_private(&path, &serde_json::to_vec(&value).unwrap()).unwrap(); + read_failure(&path).unwrap_err() + }; + let error = changed(&|value| value["format"] = 2.into()); + assert_eq!(error.code(), ErrorCode::FormatUnsupported, "{error}"); + type Change = fn(&mut serde_json::Value); + let malformed: [Change; 6] = [ + |value| value["unresolved"] = "a reason of a resolved teardown".into(), + |value| value["teardown"] = "unresolved".into(), + |value| value["teardown"] = "vanished".into(), + |value| value["generation"] = 0.into(), + |value| value["transaction"] = "05".into(), + |value| value["extra"] = true.into(), + ]; + for change in malformed { + let error = changed(&change); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + } + write_private(&path, b"{").unwrap(); + assert_eq!(read_failure(&path).unwrap_err().kind(), ErrorKind::Io); + } + + #[test] + fn the_line_of_openvmm_s_fatal_error_is_read_from_the_end_of_its_log() { + let (_directory, private) = private_dir(); + let path = private.join("openvmm.log"); + assert_eq!(read_fatal(&path), None); + // OpenVMM's warnings may name a time ABI code too; the line that leads its fatal error, + // which the error's chain follows, is the one that tells why it exited. + let fatal = "fatal error: [E_TSC_RATE_TOLERANCE] failed to launch vm worker"; + let log = format!( + "NVX-CPU-PROFILE-FALLBACK: [E_PROFILE_UNSUPPORTED] a warning\r\n{fatal}\r\n\r\n\ + Caused by:\r\n 0: [E_TSC_RATE_TOLERANCE] the destination's TSC rate\r\n" + ); + fs::write(&path, &log).unwrap(); + assert_eq!(read_fatal(&path).as_deref(), Some(fatal)); + // The line is looked for at the end of a long log, and kept to its first characters. + let long = format!("fatal error: {}", "x".repeat(2 * MAX_FATAL_CHARS)); + let filler = "a line of the VMM's log\n".repeat(2 * FATAL_TAIL_BYTES as usize / 24); + fs::write( + &path, + format!("fatal error: an earlier one\n{filler}{long}\n"), + ) + .unwrap(); + let read = read_fatal(&path).unwrap(); + assert_eq!(read.chars().count(), MAX_FATAL_CHARS); + assert!(long.starts_with(&read)); + // A log without a fatal error, or one that only names it, holds none. + fs::write(&path, "the VMM exited\nwarning: fatal error: quoted\n").unwrap(); + assert_eq!(read_fatal(&path), None); + } +} diff --git a/nvx-rs/src/host/template/clone/restore.rs b/nvx-rs/src/host/template/clone/restore.rs index f2ef00613..d97858049 100644 --- a/nvx-rs/src/host/template/clone/restore.rs +++ b/nvx-rs/src/host/template/clone/restore.rs @@ -10,10 +10,13 @@ use std::path::Path; use std::thread; use std::time::{Duration, Instant}; +use super::failure::{Attribute, CloneFailure, Evidence, Fault, Progress, Stage, host_evidence}; use super::launch::Restoring; +use super::records::Tracker; use crate::activation::{ActivationRecord, MAX_RECORD_BYTES, ReadyActivation}; use crate::error::{Error, Result}; use crate::host::platform::{ReadyListener, Transport}; +use crate::host::process::ProcessRef; use crate::host::sandbox::distro::CheckedDistro; use crate::host::session::{self, SessionError}; use crate::host::store::Recovery; @@ -48,6 +51,11 @@ pub const MAX_ACTIVATION_SESSIONS: usize = 3; /// How often the wait for the restore-ready event checks whether the VMM still runs. const EXIT_CHECK_INTERVAL: Duration = Duration::from_millis(50); +/// How long the wait for the restore-ready event waits for the VMM to exit once it closed the +/// event's endpoint without a byte of the event, as OpenVMM does as it fails to launch its VM +/// worker, which it does after it connected the endpoint, before it reports the error and exits. +const CLOSED_EXIT_WAIT: Duration = Duration::from_secs(10); + /// How long the sequence waits before a session that replaces a lost one, so that the VMM's /// control console notices the end of the lost session, which frees the console's one attachment. const SESSION_RETRY_INTERVAL: Duration = Duration::from_millis(50); @@ -149,6 +157,12 @@ impl<'s> Restoring<'s> { /// identities in it: the transaction ID of the record's restore section, as /// [`ReadyActivation::of`] returns it. /// + /// The sequence records each step that the clone reaches in its progress log, as + /// [`CloneRecords`](super::CloneRecords) describes: the event; its acknowledgment, which the + /// attachment of a session shows, since OpenVMM reads none of the host's input, its + /// authentication included, until the guest's repair acknowledged a restore that gates the + /// input; the Activate; and the Ready. + /// /// Returns the [`Activated`] clone, whose Ready-to-Start deadline, `deadlines.start`, starts /// at the check of its Ready, and whose sandbox [`Activated::start`] starts, after which /// [`Started::commit`] commits the launch, as the cold boot of a sandbox commits its launch @@ -172,30 +186,46 @@ impl<'s> Restoring<'s> { /// # Failures /// /// A step that fails ends the launch, as [`abandon`](Self::abandon) does, which terminates the - /// VMM, and ends the launch once nothing of it runs, and fails with the step's error: + /// VMM through its verified identity, and ends the launch once nothing of it runs, and fails + /// with a [`CloneFailure`], which the sandbox records, at [`Stage::Restore`] until the + /// restore-ready event and at [`Stage::Activation`] after it, with the step's error and the + /// evidence below, unless OpenVMM's outcome report of a VMM that exited gives a status of the + /// guest's that is evidence, as [`status_evidence`](super::status_evidence) tells, which + /// prevails: /// /// - [`ErrorKind::TimedOut`](crate::ErrorKind::TimedOut) once the restore-to-Ready deadline - /// passed; + /// passed, [`Evidence::Timeout`]; /// - [`ErrorKind::Exited`](crate::ErrorKind::Exited) once the VMM exited before its event, or - /// before a connection; + /// before a connection, and once it closed the event's endpoint without a byte of the event, + /// and then exited, as OpenVMM does as it fails to launch its VM worker, within ten seconds + /// of the close, [`Evidence::LostStatus`], and, for a VMM that exited before its + /// event, the error that OpenVMM reported, as its log ends with it, after the error's + /// message, and [`Evidence::IncompatibleHost`] for an error whose time ABI code says that + /// the host cannot present the template's CPU profile, TSC, or LAPIC timer, as [the + /// attribution of evidence](Evidence#attribution) describes; /// - [`ErrorKind::Protocol`](crate::ErrorKind::Protocol) for a connection to the restore-ready - /// endpoint of another process than the VMM, bytes there that differ from the event, a - /// control endpoint that another process serves, a failure of the adapter's framing, a - /// session that it lost in the last of the sessions, and a Ready that reports another - /// activation than the record's, or none; - /// - [`ErrorKind::Io`](crate::ErrorKind::Io) if the endpoints fail otherwise, and if the - /// sandbox's activation record cannot be read, or is no longer the launch's; + /// endpoint of another process than the VMM, bytes there that differ from the event, an + /// endpoint that the VMM closed after a part of the event, or without a byte of it and then + /// ran on for ten seconds, a control endpoint that another process serves, a failure of the + /// adapter's framing, and a Ready that reports another activation than the record's, or + /// none, [`Evidence::Protocol`], and for a session that the adapter lost in the last of the + /// sessions, [`Evidence::Reset`]; + /// - [`ErrorKind::Io`](crate::ErrorKind::Io) if the endpoints fail otherwise, if the sandbox's + /// activation record cannot be read, or is no longer the launch's, and if the progress log + /// cannot be appended to, the host's evidence, as [`ClonePlan::new`](super::ClonePlan::new) + /// describes it; /// - [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a deadline beyond the - /// host's clock. + /// host's clock, [`Evidence::HostFailure`]. /// - /// The error says so if the launch could not be ended, whose journal then remains for + /// The failure's teardown says whether the launch could be ended, and the error says so if it + /// could not: its journal then remains for /// [`LockedStore::recover`](crate::host::store::LockedStore::recover), and the lease with it, /// as [Crashes](crate::host::supervisor#crashes) describes. pub fn activate( self, control: &mut C, deadlines: RestoreDeadlines, - ) -> Result> { + ) -> Result, CloneFailure> { let Self { starting, launched, @@ -205,16 +235,20 @@ impl<'s> Restoring<'s> { activation, record, capability, + mut tracker, } = self; let Some(deadline) = launched.checked_add(deadlines.ready) else { - let error = beyond_clock("restore-to-Ready", deadlines.ready); - return Err(abandoned(starting, error)); + let fault = beyond_clock("restore-to-Ready", deadlines.ready); + return Err(tracker.fail(starting, Stage::Restore, fault)); }; - if let Err(error) = wait_event(&mut listener, &starting, deadline, deadlines.ready) { - return Err(abandoned(starting, error)); + if let Err(fault) = wait_event(&mut listener, &starting, deadline, deadlines.ready) { + return Err(tracker.fail(starting, Stage::Restore, fault)); } // OpenVMM sends one event, and the endpoint takes nothing more. drop(listener); + if let Err(fault) = tracker.reach(Progress::Event) { + return Err(tracker.fail(starting, Stage::Activation, fault)); + } let activation_session = ActivationSession { capability: &capability, record: &record, @@ -222,25 +256,20 @@ impl<'s> Restoring<'s> { expected: ReadyActivation::of(&activation), deadline, timeout: deadlines.ready, + since: "its launch", }; - let mut sessions = 0; - let session = loop { - sessions += 1; - match activation_session.run(control, &starting) { - Ok(session) => break session, - Err(Attempt::Lost(_)) if sessions < MAX_ACTIVATION_SESSIONS => { - let left = deadline.saturating_duration_since(Instant::now()); - thread::sleep(SESSION_RETRY_INTERVAL.min(left)); - } - Err(Attempt::Lost(error) | Attempt::Failed(error)) => { - return Err(abandoned(starting, error)); - } - } + let endpoint = starting.endpoint().to_owned(); + let reference = starting.process().reference().clone(); + let activated = + activation_session.activate(control, &endpoint, &reference, Some(&mut tracker)); + let session = match activated { + Ok(session) => session, + Err(fault) => return Err(tracker.fail(starting, Stage::Activation, fault)), }; let Some(start) = Instant::now().checked_add(deadlines.start) else { drop(session); - let error = beyond_clock("Ready-to-Start", deadlines.start); - return Err(abandoned(starting, error)); + let fault = beyond_clock("Ready-to-Start", deadlines.start); + return Err(tracker.fail(starting, Stage::Start, fault)); }; Ok(Activated { session, @@ -248,6 +277,7 @@ impl<'s> Restoring<'s> { deadline: start, image, activation, + tracker, }) } } @@ -273,6 +303,8 @@ pub struct Activated<'s, S> { image: CheckedDistro, /// The clone's activation record. activation: ActivationRecord, + /// How far the clone got, and how its failures are recorded. + tracker: Tracker, } impl<'s, S> Activated<'s, S> { @@ -303,41 +335,76 @@ impl<'s, S> Activated<'s, S> { /// Starts the clone's sandbox through the adapter's framing, `control`, in the session in /// which the clone answered, by the Ready-to-Start deadline, and returns the [`Started`] - /// clone, whose launch the caller commits. + /// clone, whose launch the caller commits. The progress log records that the start was sent, + /// then that it was confirmed. /// /// The start is not sent again: a guest that started its sandbox refuses another start, so a /// session that fails once the start was sent leaves the start unknown. A failure ends the - /// launch, as [`abandon`](Self::abandon) does, and fails with - /// [`ErrorKind::TimedOut`](crate::ErrorKind::TimedOut) once the deadline passed, and with - /// [`ErrorKind::Protocol`](crate::ErrorKind::Protocol) before, and says so if the launch - /// could not be ended, as [`Restoring::activate`] does. - pub fn start>(self, control: &mut C) -> Result> { + /// launch, as [`abandon`](Self::abandon) does, and fails with a [`CloneFailure`] at + /// [`Stage::Start`], as [`Restoring::activate`] does: with + /// [`ErrorKind::TimedOut`](crate::ErrorKind::TimedOut) once the deadline passed, + /// [`Evidence::Timeout`], and with [`ErrorKind::Protocol`](crate::ErrorKind::Protocol) + /// before, [`Evidence::Reset`] for a session that the adapter lost and + /// [`Evidence::Protocol`] otherwise, unless OpenVMM's outcome report of a VMM that exited + /// gives evidence that prevails; and with the host's evidence if the progress log cannot be + /// appended to. + pub fn start>( + self, + control: &mut C, + ) -> Result, CloneFailure> { let Self { mut session, starting, deadline, image, activation, + mut tracker, } = self; - match control.start(&mut session, deadline) { - Ok(()) => Ok(Started { - session, - starting, - image, - activation, - }), + if let Err(fault) = tracker.reach(Progress::Starting) { + drop(session); + return Err(tracker.fail(starting, Stage::Start, fault)); + } + let started = control.start(&mut session, deadline); + let fault = match started { + Ok(()) => match tracker.reach(Progress::Started) { + Ok(()) => { + return Ok(Started { + session, + starting, + image, + activation, + tracker, + }); + } + Err(fault) => fault, + }, Err(source) => { - drop(session); - let error = if Instant::now() >= deadline { - Error::timed_out( - "the clone's sandbox did not start by its Ready-to-Start deadline", + let (evidence, error) = if Instant::now() >= deadline { + ( + Evidence::Timeout, + Error::timed_out( + "the clone's sandbox did not start by its Ready-to-Start deadline", + ), + ) + } else if control.lost(&source) { + ( + Evidence::Reset, + Error::protocol( + "the clone's control session was lost: cannot start the clone's \ + sandbox", + ), ) } else { - Error::protocol("cannot start the clone's sandbox") + ( + Evidence::Protocol, + Error::protocol("cannot start the clone's sandbox"), + ) }; - Err(abandoned(starting, error.with_source(source))) + Fault::new(evidence, error.with_source(source)) } - } + }; + drop(session); + Err(tracker.fail(starting, Stage::Start, fault)) } /// Ends the launch, as [`Starting::abandon`] does, which terminates its VMM and ends the @@ -380,6 +447,8 @@ pub struct Started<'s, S> { image: CheckedDistro, /// The clone's activation record. activation: ActivationRecord, + /// How far the clone got, and how its failures are recorded. + tracker: Tracker, } impl<'s, S> Started<'s, S> { @@ -404,15 +473,38 @@ impl<'s, S> Started<'s, S> { } /// Commits the launch, as [`Starting::commit`] does, which writes its runtime record, and - /// returns the VM, with the adapter's session. The clone's image closes. + /// returns the VM, with the adapter's session. The clone's image closes, and the progress log + /// records the commit, if it can: the runtime record holds it while the VM runs. /// - /// Fails as [`Starting::commit`] fails, which ends the VMM. - pub fn commit(self) -> Result<(Vmm, S)> { + /// Fails as [`Starting::commit`] fails, which ends the VMM through its handle and keeps the + /// launch's journal, with a [`CloneFailure`] at [`Stage::Commit`], whose evidence is the + /// host's, and whose teardown is [`Teardown::Unresolved`](super::Teardown::Unresolved) until + /// a reattach ends the launch that the journal names, and releases the lease's hold. + pub fn commit(self) -> Result<(Vmm, S), CloneFailure> { let Self { - session, starting, .. + session, + starting, + mut tracker, + .. } = self; - let vmm = starting.commit()?; - Ok((vmm, session)) + let committed = starting.commit_reporting(|error| { + tracker.pending(Stage::Commit, host_evidence(error), error); + }); + match committed { + Ok(vmm) => { + tracker.note(Progress::Committed); + Ok((vmm, session)) + } + Err(error) => { + drop(session); + let teardown = super::Teardown::Unresolved( + "the clone's VMM ended through its handle, and the launch's journal remains \ + until a reattach ends the launch" + .to_owned(), + ); + Err(tracker.record(Stage::Commit, Fault::host(error), teardown)) + } + } } /// Ends the launch, as [`Activated::abandon`] does. @@ -438,68 +530,120 @@ impl fmt::Debug for Started<'_, S> { /// What every session of a clone's activation takes: the launch's capability, the sandbox's /// activation record, and the activation that its Ready must report, by the restore-to-Ready -/// deadline. -struct ActivationSession<'a> { +/// deadline, or by the deadline of a reattach. +pub(super) struct ActivationSession<'a> { /// The capability of the launch. - capability: &'a [u8; CAPABILITY_LEN], + pub(super) capability: &'a [u8; CAPABILITY_LEN], /// The path of the activation record in the sandbox. - record: &'a Path, + pub(super) record: &'a Path, /// The canonical encoding of the launch's activation record. - encoded: Vec, + pub(super) encoded: Vec, /// The activation that the clone's Ready must report. - expected: ReadyActivation, - /// The restore-to-Ready deadline. - deadline: Instant, - /// The restore-to-Ready deadline's duration from the launch. - timeout: Duration, + pub(super) expected: ReadyActivation, + /// The deadline by which the clone must be ready. + pub(super) deadline: Instant, + /// The deadline's duration from its start. + pub(super) timeout: Duration, + /// What starts the deadline, as messages say: `its launch`, or `its reattach`. + pub(super) since: &'static str, } /// How a session of a clone's activation failed. enum Attempt { /// The adapter lost the session, after which another session may find the clone ready. - Lost(Error), + Lost(Fault), /// Any other failure, which is final. - Failed(Error), + Failed(Fault), } impl ActivationSession<'_> { - /// Activates the clone of `starting` in a new session of `control`, as - /// [`Restoring::activate`] describes, and returns the session once the clone is ready. + /// Activates the clone whose VMM, `reference`, serves its control endpoint at `endpoint`, in + /// sessions of `control`, as [`Restoring::activate`] describes: in a new session after a + /// session that the adapter lost, in [`MAX_ACTIVATION_SESSIONS`] at most, and returns the + /// session in which the clone answered with its Ready. A launch records the clone's progress + /// in its `tracker`. + pub(super) fn activate( + &self, + control: &mut C, + endpoint: &str, + reference: &ProcessRef, + mut tracker: Option<&mut Tracker>, + ) -> std::result::Result { + let mut sessions = 0; + loop { + sessions += 1; + match self.run(control, endpoint, reference, tracker.as_deref_mut()) { + Ok(session) => return Ok(session), + Err(Attempt::Lost(_)) if sessions < MAX_ACTIVATION_SESSIONS => { + let left = self.deadline.saturating_duration_since(Instant::now()); + thread::sleep(SESSION_RETRY_INTERVAL.min(left)); + } + Err(Attempt::Lost(fault) | Attempt::Failed(fault)) => return Err(fault), + } + } + } + + /// Activates the clone in a new session of `control`, as [`activate`](Self::activate) + /// describes, records its progress in `tracker`, if it has one, and returns the session once + /// the clone is ready. fn run( &self, control: &mut C, - starting: &Starting<'_>, + endpoint: &str, + reference: &ProcessRef, + mut tracker: Option<&mut Tracker>, ) -> std::result::Result { let record = self.read_record().map_err(Attempt::Failed)?; - let transport = self.connect(starting).map_err(Attempt::Failed)?; + let transport = self.connect(endpoint, reference).map_err(Attempt::Failed)?; let attached = control.attach(transport, self.capability, self.deadline); let mut session = attached.map_err(|error| self.failed(control, error, "attach its control session"))?; + if let Some(tracker) = &mut tracker { + // OpenVMM authenticated the session, which it does only once the guest acknowledged + // a restore that gates the host's input. The Activate follows at once, with which + // the progress log records the acknowledgment. + tracker.advance(Progress::Acknowledged); + tracker + .reach(Progress::Activating) + .map_err(Attempt::Failed)?; + } let reported = control .activate(&mut session, &record, self.deadline) .map_err(|error| self.failed(control, error, "activate it"))?; self.check(reported).map_err(Attempt::Failed)?; + if let Some(tracker) = &mut tracker { + tracker.reach(Progress::Ready).map_err(Attempt::Failed)?; + } Ok(session) } - /// Connects to the control endpoint of the VMM of `starting`, as [`session::connect`] does, - /// by the deadline. - fn connect(&self, starting: &Starting<'_>) -> Result> { - let reference = starting.process().reference(); - let endpoint = starting.endpoint(); + /// Connects to the control endpoint `endpoint`, which the VMM `reference` serves, as + /// [`session::connect`] does, by the deadline. + fn connect( + &self, + endpoint: &str, + reference: &ProcessRef, + ) -> std::result::Result, Fault> { session::connect(endpoint, reference.pid, reference.start_time, self.deadline).map_err( |error| { let message = format!("cannot connect to the clone's control endpoint {endpoint}"); match error { - SessionError::TimedOut => Error::timed_out(format!( - "{message} within {:?} of its launch", - self.timeout - )), - SessionError::ProcessExited => { - Error::exited(format!("{message}: its VMM exited")) - } - SessionError::Io(source) => Error::io(message).with_source(source), - error => Error::protocol(message).with_source(error), + SessionError::TimedOut => Fault::new( + Evidence::Timeout, + Error::timed_out(format!( + "{message} within {:?} of {}", + self.timeout, self.since + )), + ), + SessionError::ProcessExited => Fault::new( + Evidence::LostStatus, + Error::exited(format!("{message}: its VMM exited")), + ), + SessionError::Io(source) => Fault::host(Error::io(message).with_source(source)), + error => Fault::new( + Evidence::Protocol, + Error::protocol(message).with_source(error), + ), } }, ) @@ -507,76 +651,85 @@ impl ActivationSession<'_> { /// Reads the activation record in the sandbox, which must be, byte for byte, the encoding of /// the launch's record, and returns it. - fn read_record(&self) -> Result> { + fn read_record(&self) -> std::result::Result, Fault> { let path = self.record; let bytes = open_regular(path) .and_then(|file| read_at_most(file, MAX_RECORD_BYTES as u64, "an activation record")) .map_err(|error| { - Error::io(format!( - "cannot read the clone's activation record {}", - path.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot read the clone's activation record {}", + path.display() + )) + .with_source(error), + ) })?; if bytes != self.encoded { - return Err(Error::io(format!( - "{} no longer holds the activation record that the clone's launch wrote, the only \ - record that its restore sends", - path.display() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::io(format!( + "{} no longer holds the activation record that the clone's launch wrote, the \ + only record that its restore sends", + path.display() + )), + )); } Ok(bytes) } /// Checks `reported`, the activation that the clone's Ready reports, against the record's. - fn check(&self, reported: Option) -> Result<()> { - let expected = &self.expected; - let Some(reported) = reported else { - return Err(Error::protocol( - "the clone's Ready reports no activation, as a VM that boots cold reports none", - )); - }; - if reported == *expected { - return Ok(()); - } - let transaction = |activation: &ReadyActivation| { - activation - .transaction - .map_or_else(|| "none".to_owned(), |transaction| transaction.to_string()) - }; - Err(Error::protocol(format!( - "the clone's Ready reports the transaction {}, not its record's transaction {}", - transaction(&reported), - transaction(expected) - ))) + fn check(&self, reported: Option) -> std::result::Result<(), Fault> { + check_ready(&self.expected, reported).attribute(Evidence::Protocol) } /// Returns how a session failed with `error`, which `control` returned as it tried to `what`. fn failed(&self, control: &C, error: C::Error, what: &str) -> Attempt { if Instant::now() >= self.deadline { - return Attempt::Failed( + return Attempt::Failed(Fault::new( + Evidence::Timeout, Error::timed_out(format!( - "the clone was not ready within {:?} of its launch: cannot {what}", - self.timeout + "the clone was not ready within {:?} of {}: cannot {what}", + self.timeout, self.since )) .with_source(error), - ); + )); } - let lost = control.lost(&error); - let message = if lost { - format!("the clone's control session was lost: cannot {what}") + if control.lost(&error) { + let error = Error::protocol(format!( + "the clone's control session was lost: cannot {what}" + )) + .with_source(error); + Attempt::Lost(Fault::new(Evidence::Reset, error)) } else { - format!("cannot {what}") - }; - let error = Error::protocol(message).with_source(error); - if lost { - Attempt::Lost(error) - } else { - Attempt::Failed(error) + let error = Error::protocol(format!("cannot {what}")).with_source(error); + Attempt::Failed(Fault::new(Evidence::Protocol, error)) } } } +/// Checks `reported`, the activation that a clone's Ready reports, against `expected`, its +/// record's: the transaction ID of the record's restore section. +fn check_ready(expected: &ReadyActivation, reported: Option) -> Result<()> { + let Some(reported) = reported else { + return Err(Error::protocol( + "the clone's Ready reports no activation, as a VM that boots cold reports none", + )); + }; + if reported == *expected { + return Ok(()); + } + let transaction = |activation: &ReadyActivation| { + activation + .transaction + .map_or_else(|| "none".to_owned(), |transaction| transaction.to_string()) + }; + Err(Error::protocol(format!( + "the clone's Ready reports the transaction {}, not its record's transaction {}", + transaction(&reported), + transaction(expected) + ))) +} + /// Waits until `deadline`, `timeout` after the launch, for the restore-ready event of the VMM of /// `starting` at `listener`, as [`Restoring::activate`] describes. fn wait_event( @@ -584,7 +737,7 @@ fn wait_event( starting: &Starting<'_>, deadline: Instant, timeout: Duration, -) -> Result<()> { +) -> std::result::Result<(), Fault> { let process = starting.process(); let pid = process.reference().pid; // OpenVMM writes the event once, and then closes its end, so the wait reads the event's bytes @@ -594,9 +747,12 @@ fn wait_event( while received.len() < length { let left = deadline.saturating_duration_since(Instant::now()); if left.is_zero() { - return Err(Error::timed_out(format!( - "the clone's VMM sent no restore-ready event within {timeout:?} of its launch" - ))); + return Err(Fault::new( + Evidence::Timeout, + Error::timed_out(format!( + "the clone's VMM sent no restore-ready event within {timeout:?} of its launch" + )), + )); } let before = received.len(); let open = listener @@ -604,35 +760,67 @@ fn wait_event( .map_err(|error| { let message = "cannot receive the restore-ready event of the clone's VMM"; if error.kind() == io::ErrorKind::PermissionDenied { - Error::protocol(message).with_source(error) + Fault::new( + Evidence::Protocol, + Error::protocol(message).with_source(error), + ) } else { - Error::io(message).with_source(error) + Fault::host(Error::io(message).with_source(error)) } })?; if !RESTORE_READY_EVENT.starts_with(&received) { - return Err(Error::protocol(format!( - "the clone's VMM sent {:?} at its restore-ready endpoint, which is not OpenVMM's \ - restore-ready event", - String::from_utf8_lossy(&received) - ))); + return Err(Fault::new( + Evidence::Protocol, + Error::protocol(format!( + "the clone's VMM sent {:?} at its restore-ready endpoint, which is not \ + OpenVMM's restore-ready event", + String::from_utf8_lossy(&received) + )), + )); } if received.len() == length { break; } if !open { - return Err(Error::protocol(format!( - "the clone's VMM closed its restore-ready endpoint after {} bytes of the event", - received.len() - ))); + if received.is_empty() { + let wait = deadline.min(Instant::now() + CLOSED_EXIT_WAIT); + let exited = process.wait(wait).map_err(|error| { + Fault::host( + Error::io("cannot tell whether the clone's VMM still runs") + .with_source(error), + ) + })?; + if exited { + return Err(Fault::new( + Evidence::LostStatus, + Error::exited( + "the clone's VMM closed its restore-ready endpoint without the event, \ + and exited, as OpenVMM does once it cannot restore the snapshot", + ), + )); + } + } + return Err(Fault::new( + Evidence::Protocol, + Error::protocol(format!( + "the clone's VMM closed its restore-ready endpoint after {} bytes of the event", + received.len() + )), + )); } if received.len() == before { let exited = process.has_exited().map_err(|error| { - Error::io("cannot tell whether the clone's VMM still runs").with_source(error) + Fault::host( + Error::io("cannot tell whether the clone's VMM still runs").with_source(error), + ) })?; if exited { - return Err(Error::exited( - "the clone's VMM exited before its restore-ready event, as OpenVMM does once \ - it cannot restore the snapshot", + return Err(Fault::new( + Evidence::LostStatus, + Error::exited( + "the clone's VMM exited before its restore-ready event, as OpenVMM does \ + once it cannot restore the snapshot", + ), )); } } @@ -640,32 +828,14 @@ fn wait_event( Ok(()) } -/// Returns the error of the deadline `name`, `timeout` after its start, which this host's clock +/// Returns the fault of the deadline `name`, `timeout` after its start, which this host's clock /// cannot represent. -fn beyond_clock(name: &str, timeout: Duration) -> Error { - Error::unsupported(format!( - "the {name} deadline of a clone's restore, {timeout:?} after its start, is beyond this \ - host's clock" - )) -} - -/// Ends `starting`, a launch whose restore sequence failed with `error`, as -/// [`Starting::abandon`] does, and returns `error`, which says why the launch could not be ended -/// if it could not: its journal then remains for recovery. -fn abandoned(starting: Starting<'_>, error: Error) -> Error { - let reason = match starting.abandon() { - Ok(Recovery::Indeterminate(reason)) => reason, - Ok(_) => return error, - Err(failure) => failure.to_string(), - }; - let code = error.code(); - let (_, message, source) = error.into_parts(); - let error = Error::new( - code, - format!("{message}; the launch could not be ended, and remains for recovery: {reason}"), - ); - match source { - Some(source) => error.with_source(source), - None => error, - } +fn beyond_clock(name: &str, timeout: Duration) -> Fault { + Fault::new( + Evidence::HostFailure, + Error::unsupported(format!( + "the {name} deadline of a clone's restore, {timeout:?} after its start, is beyond \ + this host's clock" + )), + ) } diff --git a/nvx-rs/src/host/template/clone/tests.rs b/nvx-rs/src/host/template/clone/tests.rs index 0aeb338ad..fc9bb98a4 100644 --- a/nvx-rs/src/host/template/clone/tests.rs +++ b/nvx-rs/src/host/template/clone/tests.rs @@ -247,9 +247,42 @@ fn network(access: NetworkAccess) -> SandboxNetwork { SandboxNetwork::new("10.0.0.2/24", NetworkPolicy::new(access)).unwrap() } -/// Returns the kind and the message of the error of `result`. -fn failure(result: Result) -> (ErrorKind, String) { - let error = result.unwrap_err(); +/// Returns the kind and the message of the error of `result`, a clone's plan that failed with +/// `evidence`, as [`refused_at`] checks it. +fn refused( + result: std::result::Result, + evidence: Evidence, +) -> (ErrorKind, String) { + refused_at(result, Stage::Plan, evidence) +} + +/// Returns the kind and the message of the error of `result`, which must have failed at `stage`, +/// the plan or the launch, with `evidence`, before anything of the launch existed: with the +/// transaction of the clone's restore once the plan drew it, and no VM to tear down. +fn refused_at( + result: std::result::Result, + stage: Stage, + evidence: Evidence, +) -> (ErrorKind, String) { + let failure = result.unwrap_err(); + let record = failure.record(); + assert_eq!( + (record.stage, record.evidence, failure.outcome()), + (stage, evidence, evidence.outcome()), + "{failure}" + ); + assert_eq!( + (record.progress, failure.teardown(), record.status), + (Progress::Planned, &Teardown::NoVm, None), + "{failure}" + ); + assert_eq!( + record.transaction.is_some(), + stage != Stage::Plan, + "{failure}" + ); + assert_eq!(record.code, failure.code()); + let error = failure.into_error(); (error.kind(), error.to_string()) } @@ -378,7 +411,7 @@ impl Fixture { acquired: &mut Acquired, request: &CloneRequest, access: Access<'_>, - ) -> Result { + ) -> Result { ClonePlan::new_with( &self.store, acquired, @@ -395,10 +428,18 @@ fn sparse_path(acquired: &Acquired) -> PathBuf { } /// Asserts that the sandbox of `acquired` holds no scratch device's file of a clone in RAM, and -/// that its lease still reserves its launch, as after a plan that failed. +/// no record of a clone's progress or failure, and that its lease still reserves its launch, as +/// after a plan that failed. fn assert_untouched(acquired: &Acquired) { - assert!(!sparse_path(acquired).exists()); - assert_eq!(acquired.locked.reserved(), Some(acquired.lease.launch())); + let locked = &acquired.locked; + for path in [ + sparse_path(acquired), + locked.clone_progress_path(), + locked.clone_failure_path(), + ] { + assert!(!path.exists(), "{}", path.display()); + } + assert_eq!(locked.reserved(), Some(acquired.lease.launch())); } /// Returns a file of `length` bytes on hugetlbfs, whose I/O block size is a huge page of 2 MiB, @@ -602,6 +643,11 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let fixture = Fixture::new(); let mut acquired = fixture.acquire("sandbox"); let sparse = sparse_path(&acquired); + // The image or the scratch image of a request is a file of the sandbox's store: the request's + // settings are invalid. + let store_file = |result: std::result::Result| { + refused(result, Evidence::InvalidSettings) + }; let refusal = |what: &str, path: &Path, owned: &Path| { ( ErrorKind::Unsupported, @@ -620,7 +666,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let binding = DistroBinding::import(&layer(&sparse), PolicyVerdict::ACCEPTED).unwrap(); let staged = seal_of(&sparse); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&sparse, &binding), NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &tmpfs_request(&sparse, &binding), NO_CONTENT)), refusal("image", &sparse, &sparse) ); assert_eq!(seal_of(&sparse), staged); @@ -629,7 +675,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { fs::hard_link(&sparse, &hard).unwrap(); let hard_binding = DistroBinding::import(&layer(&hard), PolicyVerdict::ACCEPTED).unwrap(); assert_eq!( - failure(fixture.plan( + store_file(fixture.plan( &mut acquired, &tmpfs_request(&hard, &hard_binding), NO_CONTENT @@ -643,7 +689,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let aliased_binding = DistroBinding::import(&layer(&aliased), PolicyVerdict::ACCEPTED).unwrap(); assert_eq!( - failure(fixture.plan( + store_file(fixture.plan( &mut acquired, &tmpfs_request(&aliased, &aliased_binding), NO_CONTENT @@ -660,7 +706,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let (scratch, scratch_binding) = fixture.scratch("scratch.ext4", SCRATCH_LEN); let request = scratch_request(&sparse, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal("image", &sparse, &sparse) ); assert_eq!(seal_of(&sparse), staged); @@ -670,7 +716,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let sparse_binding = ScratchBinding::bind(&sparse).unwrap(); let request = scratch_request(&image, &image_binding, &sparse, &sparse_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal("scratch image", &sparse, &sparse) ); assert_eq!(fs::metadata(&sparse).unwrap().len(), SCRATCH_LEN); @@ -685,7 +731,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { stage(&owned, IMAGE_LEN); let binding = DistroBinding::import(&layer(&owned), PolicyVerdict::ACCEPTED).unwrap(); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&owned, &binding), NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &tmpfs_request(&owned, &binding), NO_CONTENT)), refusal("image", &owned, &owned) ); assert_eq!(fs::metadata(&owned).unwrap().len(), IMAGE_LEN); @@ -700,6 +746,11 @@ fn links_and_directories_under_the_names_of_the_sandboxs_store_are_refused() { let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); let sparse = sparse_path(&acquired); + // The sandbox's store holds an entry that it never creates: the host's state is not as the + // store keeps it. + let store_entry = |result: std::result::Result| { + refused(result, Evidence::HostFailure) + }; let refusal = |owned: &Path| { ( ErrorKind::Unavailable, @@ -715,7 +766,7 @@ fn links_and_directories_under_the_names_of_the_sandboxs_store_are_refused() { let owned = acquired.locked.dir().join(name); fs::create_dir(&owned).unwrap(); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_entry(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal(&owned) ); assert!(owned.is_dir()); @@ -729,7 +780,7 @@ fn links_and_directories_under_the_names_of_the_sandboxs_store_are_refused() { DistroBinding::import(&layer(&through), PolicyVerdict::ACCEPTED).unwrap(); for request in [tmpfs_request(&through, &through_binding), request.clone()] { assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_entry(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal(&sparse) ); } @@ -778,7 +829,7 @@ fn a_scratch_image_that_is_a_file_of_a_generation_s_snapshot_is_refused() { let other = other.join("memory.bin"); stage(&other, SCRATCH_LEN); // Through a hard link among the scratch images, another name of the file, and by its own - // path, in the store's directory. + // path, in the store's directory: the request's settings are invalid. let cases = [ (&leased, fixture.path("scratches", "leased.raw")), (&other, fixture.path("scratches", "other.raw")), @@ -811,7 +862,10 @@ fn a_scratch_image_that_is_a_file_of_a_generation_s_snapshot_is_refused() { ) }; assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), (ErrorKind::Unsupported, expected) ); assert_untouched(&acquired); @@ -859,7 +913,10 @@ fn a_scratch_image_in_the_template_store_is_refused() { let request = scratch_request(&image, &image_binding, &scratch, &binding); let location = location_of(&File::open(file).unwrap()).unwrap(); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -922,7 +979,10 @@ fn a_scratch_image_with_another_name_is_refused() { let binding = ScratchBinding::bind(&scratch).unwrap(); let request = scratch_request(&image, &image_binding, &scratch, &binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -945,6 +1005,11 @@ fn names_that_the_file_system_takes_for_those_of_the_stores_files_are_the_stores let (image, image_binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); let dir = acquired.locked.dir().to_owned(); + // The image or the scratch image of a request is a file of the sandbox's store: the request's + // settings are invalid. + let store_file = |result: std::result::Result| { + refused(result, Evidence::InvalidSettings) + }; let refusal = |what: &str, path: &Path, owned: &Path| { ( ErrorKind::Unsupported, @@ -976,7 +1041,7 @@ fn names_that_the_file_system_takes_for_those_of_the_stores_files_are_the_stores let same = same_file(&cased, &owned); let result = fixture.plan(&mut acquired, &tmpfs_request(&cased, &binding), NO_CONTENT); if same { - assert_eq!(failure(result), refusal("image", &cased, &owned)); + assert_eq!(store_file(result), refusal("image", &cased, &owned)); } else { drop(result.unwrap()); } @@ -994,7 +1059,10 @@ fn names_that_the_file_system_takes_for_those_of_the_stores_files_are_the_stores let same = same_file(&cased, &sparse); let result = fixture.plan(&mut acquired, &request, NO_CONTENT); if same { - assert_eq!(failure(result), refusal("scratch image", &cased, &sparse)); + assert_eq!( + store_file(result), + refusal("scratch image", &cased, &sparse) + ); } else { drop(result.unwrap()); assert_eq!( @@ -1023,8 +1091,9 @@ fn templates_that_a_runtime_of_other_contracts_captured_are_refused() { let fixture = Fixture::with(|class, _, _| change(class)); let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)), + refused(plan, Evidence::IncompatibleHost), ( ErrorKind::Unsupported, format!( @@ -1039,6 +1108,166 @@ fn templates_that_a_runtime_of_other_contracts_captured_are_refused() { } } +#[test] +fn images_that_their_binding_does_not_authorize_and_malformed_generations_are_refused() { + let fixture = Fixture::new(); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("sandbox"); + let request = tmpfs_request(&image, &binding); + // A binding whose verdict rejects the image authorizes no clone of it. + let verdict = PolicyVerdict { + result: PolicyResult::Rejected, + ..PolicyVerdict::ACCEPTED + }; + let rejected = DistroBinding::import(&layer(&image), verdict).unwrap(); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &rejected), NO_CONTENT); + refused(plan, Evidence::AuthorizationDenied); + // A binding of another revision of the distro policy than the request's is stale. + let stale = CloneRequest { + policy_revision: DISTRO_POLICY_REVISION + 1, + ..request.clone() + }; + refused( + fixture.plan(&mut acquired, &stale, NO_CONTENT), + Evidence::StaleBinding, + ); + // A failure of the request quarantines nothing. + let failure = fixture.plan(&mut acquired, &stale, NO_CONTENT).unwrap_err(); + assert!(!failure.quarantine(&fixture.store, None).unwrap()); + assert_untouched(&acquired); + // A generation whose key is gone is malformed: a failure of the template, whose generation it + // quarantines through its tombstone, once the sandbox's lock is released, as the lock order + // asks. + let generation = fixture.generation.generation(); + let key = fixture + .store + .payload_dir(generation) + .join(crate::host::template::KEY_NAME); + fs::remove_file(key).unwrap(); + let failure = fixture + .plan(&mut acquired, &request, NO_CONTENT) + .unwrap_err(); + assert_eq!( + (failure.record().evidence, failure.outcome()), + (Evidence::MalformedGeneration, Outcome::Template), + "{failure}" + ); + assert_untouched(&acquired); + drop(acquired); + let other = Fixture::new(); + let error = failure.quarantine(&other.store, None).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Other, "{error}"); + assert!(other.store.tombstones().unwrap().is_empty()); + assert!(failure.quarantine(&fixture.store, None).unwrap()); + assert_eq!(fixture.store.tombstones().unwrap(), [generation]); + // The generation takes no new lease. + let error = fixture + .store + .acquire( + generation, + fixture.provision("next"), + &files(), + &after(&fixture.generation, 1), + None, + ) + .unwrap_err(); + assert_eq!(error.code(), ErrorCode::TemplateTombstoned, "{error}"); +} + +#[test] +fn keys_that_their_read_refuses_for_what_they_are_are_failures_of_the_template() { + let fixture = Fixture::new(); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("sandbox"); + let request = tmpfs_request(&image, &binding); + let generation = fixture.generation.generation(); + let key = fixture + .store + .payload_dir(generation) + .join(crate::host::template::KEY_NAME); + let malformed = |acquired: &mut Acquired, what: &str| { + let failure = fixture.plan(acquired, &request, NO_CONTENT).unwrap_err(); + assert_eq!( + (failure.record().evidence, failure.outcome()), + (Evidence::MalformedGeneration, Outcome::Template), + "{what}: {failure}" + ); + assert_untouched(acquired); + failure + }; + // A link in place of the key, a directory there, and a key that holds more than a key may are + // none that the store writes: the read refuses each for what it is, a failure of the template, + // not of an operation of the host's. Windows denies the open of a directory, as a file's access + // control list may deny one, which is the host's, so only Unix tries the directory. + let moved = key.with_extension("moved"); + fs::rename(&key, &moved).unwrap(); + if symlink_file(&moved, &key) { + malformed(&mut acquired, "a link"); + fs::remove_file(&key).unwrap(); + } + if cfg!(unix) { + fs::create_dir(&key).unwrap(); + malformed(&mut acquired, "a directory"); + fs::remove_dir(&key).unwrap(); + } + fs::rename(&moved, &key).unwrap(); + crate::host::store::files::write_private(&key, &vec![0; TemplateKey::MAX_LEN + 1]).unwrap(); + let failure = malformed(&mut acquired, "an oversized key"); + // So the failure quarantines the generation, once the sandbox's lock is released. + drop(acquired); + assert!(failure.quarantine(&fixture.store, None).unwrap()); + assert_eq!(fixture.store.tombstones().unwrap(), [generation]); +} + +#[test] +fn keys_of_another_encoding_version_are_failures_of_the_host_that_quarantine_nothing() { + // A store that a build of another encoding of keys kept, as one that a rollback of the host's + // build leaves, holds generations whose keys this build cannot read: the host cannot restore + // them, but a build that reads their keys can, so their failures quarantine nothing. + let fixture = Fixture::new(); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("sandbox"); + let generation = fixture.generation.generation(); + let key = fixture + .store + .payload_dir(generation) + .join(crate::host::template::KEY_NAME); + let mut bytes = fs::read(&key).unwrap(); + let version = TemplateKey::VERSION + 1; + bytes[..2].copy_from_slice(&version.to_le_bytes()); + crate::host::store::files::write_private(&key, &bytes).unwrap(); + let failure = fixture + .plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT) + .unwrap_err(); + assert_eq!( + (failure.record().evidence, failure.outcome()), + (Evidence::IncompatibleHost, Outcome::Host), + "{failure}" + ); + assert_eq!(failure.code(), ErrorCode::FormatUnsupported, "{failure}"); + let unread = format!( + "the template key has encoding version {version}; this build reads version {} only", + TemplateKey::VERSION + ); + assert!(failure.message().contains(&unread), "{failure}"); + assert_untouched(&acquired); + drop(acquired); + assert!(!failure.quarantine(&fixture.store, None).unwrap()); + assert!(fixture.store.tombstones().unwrap().is_empty()); + // The generation still takes new leases. + let next = fixture + .store + .acquire( + generation, + fixture.provision("next"), + &files(), + &after(&fixture.generation, 1), + None, + ) + .unwrap(); + assert_eq!(next.lease.generation(), generation); +} + #[test] fn a_clone_on_a_scratch_image_claims_it_and_reads_no_content() { let fixture = Fixture::new(); @@ -1093,6 +1322,9 @@ fn a_scratch_image_serves_one_clone_at_a_time() { assert!(started.elapsed() >= wait); assert_eq!(error.code(), ErrorCode::ScratchClaimed, "{error}"); assert_eq!(error.kind(), ErrorKind::TimedOut, "{error}"); + // The image is not fresh for this clone: the request's. + assert_eq!(error.record().evidence, Evidence::UnusableScratch); + assert_eq!(error.outcome(), Outcome::Request); assert_eq!( error.message(), format!( @@ -1128,7 +1360,10 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { fs::rename(©, &scratch).unwrap(); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), changed(&scratch, "identity") ); // So has a directory in its place. The image stays, under another name, so that the @@ -1138,7 +1373,10 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { fs::create_dir(&scratch).unwrap(); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), changed(&scratch, "identity") ); // A binding of another kind names another file. @@ -1148,7 +1386,10 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { let scratch_binding: ScratchBinding = serde_json::from_value(value).unwrap(); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), changed(&scratch, "kind") ); assert_untouched(&acquired); @@ -1158,16 +1399,16 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { fn images_of_another_length_or_geometry_are_refused() { let fixture = Fixture::new(); let mut acquired = fixture.acquire("sandbox"); - let mut refusal = |length: u64| { + let mut refusal = |length: u64, evidence: Evidence| { let (image, binding) = fixture.image("other.erofs", length); - let failed = - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); + let failed = refused(plan, evidence); fs::remove_file(&image).unwrap(); (shown(&image), failed) }; // A clone's image has the placeholder's length. for length in [IMAGE_LEN - 512, IMAGE_LEN + 512, 2 * IMAGE_LEN] { - let (image, failed) = refusal(length); + let (image, failed) = refusal(length, Evidence::SuppliedLength); assert_eq!( failed, ( @@ -1181,7 +1422,7 @@ fn images_of_another_length_or_geometry_are_refused() { } // OpenVMM attaches no block device of another length. for length in [0, IMAGE_LEN + 1] { - let (image, failed) = refusal(length); + let (image, failed) = refusal(length, Evidence::SuppliedGeometry); assert_eq!( failed, ( @@ -1205,8 +1446,9 @@ fn images_of_another_length_or_geometry_are_refused() { let mut acquired = fixture.acquire("sandbox"); let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let physical = fixture.physical; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)), + refused(plan, Evidence::SuppliedGeometry), ( ErrorKind::Unsupported, format!( @@ -1229,7 +1471,10 @@ fn scratch_images_of_another_length_or_geometry_are_refused() { let (scratch, scratch_binding) = fixture.scratch("other.ext4", length); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedLength + ), ( ErrorKind::Unsupported, format!( @@ -1250,7 +1495,10 @@ fn scratch_images_of_another_length_or_geometry_are_refused() { let (scratch, scratch_binding) = fixture.scratch("unaligned.ext4", SCRATCH_LEN + 1); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedGeometry + ), ( ErrorKind::Unsupported, format!( @@ -1276,8 +1524,10 @@ fn sandboxes_whose_filesystem_presents_another_scratch_geometry_are_refused() { let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); let (physical, other) = (fixture.physical, other_physical(fixture.physical)); + // The sandbox's filesystem is the host's, which cannot restore the template. + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)), + refused(plan, Evidence::IncompatibleHost), ( ErrorKind::Unsupported, format!( @@ -1293,7 +1543,10 @@ fn sandboxes_whose_filesystem_presents_another_scratch_geometry_are_refused() { let (scratch, scratch_binding) = fixture.scratch("scratch.ext4", SCRATCH_LEN); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedGeometry + ), ( ErrorKind::Unsupported, format!( @@ -1354,8 +1607,9 @@ fn files_on_a_filesystem_of_another_block_size_are_refused() { distro: huge_distro, ..NO_CONTENT }; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access)), + refused(plan, Evidence::SuppliedGeometry), presented(format!("the image {}", shown(&image)), "placeholder") ); // The scratch image: its binding and its check reach a file on hugetlbfs at its path. @@ -1371,7 +1625,10 @@ fn files_on_a_filesystem_of_another_block_size_are_refused() { }; let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, access)), + refused( + fixture.plan(&mut acquired, &request, access), + Evidence::SuppliedGeometry + ), presented(format!("the scratch image {}", shown(&scratch)), "scratch") ); // The scratch device of a clone in RAM, which a sandbox on hugetlbfs creates there. @@ -1381,8 +1638,9 @@ fn files_on_a_filesystem_of_another_block_size_are_refused() { create: &create, ..NO_CONTENT }; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access)), + refused(plan, Evidence::IncompatibleHost), presented( format!( "the scratch device of the clone {}, in the sandbox's filesystem,", @@ -1415,7 +1673,10 @@ fn memory_targets_are_whole_blocks_from_the_templates_ram_to_its_capacity() { ..tmpfs_request(&image, &binding) }; assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidTargets + ), ( ErrorKind::Unsupported, format!( @@ -1444,8 +1705,8 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let (scratch, scratch_binding) = fixture.scratch("scratch.ext4", SCRATCH_LEN); let mut acquired = fixture.acquire("sandbox"); - let mut refusal = |request: &CloneRequest| { - let (kind, message) = failure(fixture.plan(&mut acquired, request, NO_CONTENT)); + let mut refusal = |request: &CloneRequest, evidence: Evidence| { + let (kind, message) = refused(fixture.plan(&mut acquired, request, NO_CONTENT), evidence); assert_eq!(kind, ErrorKind::Unsupported, "{message}"); message }; @@ -1461,7 +1722,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { .unwrap(), ..tmpfs.clone() }; - let message = refusal(&profile); + let message = refusal(&profile, Evidence::InvalidSettings); assert!( message.starts_with("a clone restores an adapter's plan, not a plan of the sandbox"), "{message}" @@ -1476,7 +1737,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { ..tmpfs.clone() }; assert_eq!( - refusal(&mistaken), + refusal(&mistaken, Evidence::InvalidFormat), format!( "the image {} is not the erofs layer with identity {UUID} that its distro binding \ names", @@ -1488,17 +1749,19 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { adapter: SandboxPlan::adapter(SandboxDistro::new(path, distro.image()), SandboxUpper::RAM), ..tmpfs.clone() }; - let message = refusal(&at("distro.erofs")); + let message = refusal(&at("distro.erofs"), Evidence::InvalidSettings); assert!( message.starts_with("the image of a clone must be absolute"), "{message}" ); - let message = refusal(&at(&format!("{},x", image.to_str().unwrap()))); + let comma = at(&format!("{},x", image.to_str().unwrap())); + let message = refusal(&comma, Evidence::InvalidSettings); assert!( message.starts_with("disk paths containing commas, semicolons, or unbalanced brackets"), "{message}" ); - let message = refusal(&at(image.with_extension("vhdx").to_str().unwrap())); + let vhdx = at(image.with_extension("vhdx").to_str().unwrap()); + let message = refusal(&vhdx, Evidence::InvalidSettings); assert!( message.starts_with("disk paths ending in .vhdx"), "{message}" @@ -1508,12 +1771,13 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { scratch: Some(scratch_binding.clone()), ..tmpfs.clone() }; - let message = refusal(&on_scratch("scratch.ext4")); + let message = refusal(&on_scratch("scratch.ext4"), Evidence::InvalidSettings); assert!( message.starts_with("the scratch image of a clone must be absolute"), "{message}" ); - let message = refusal(&on_scratch(scratch.with_extension("iso").to_str().unwrap())); + let iso = on_scratch(scratch.with_extension("iso").to_str().unwrap()); + let message = refusal(&iso, Evidence::InvalidSettings); assert!( message.starts_with("disk paths ending in .iso"), "{message}" @@ -1524,7 +1788,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { ..scratch_request(&image, &binding, &scratch, &scratch_binding) }; assert_eq!( - refusal(&unbound), + refusal(&unbound, Evidence::InvalidSettings), format!( "the upper of a clone on the scratch image {} needs the image's scratch binding", shown(&scratch) @@ -1535,7 +1799,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { ..tmpfs.clone() }; assert_eq!( - refusal(&bound), + refusal(&bound, Evidence::InvalidSettings), "the upper of a clone in RAM takes no scratch binding" ); // None of them opened anything: the scratch image is free, and the lease still reserves the @@ -1554,7 +1818,10 @@ fn networks_that_the_template_does_not_have_are_refused() { let generation = fixture.generation.generation(); let attached = with_network(request.clone(), Some(network(NetworkAccess::Allow))); assert_eq!( - failure(fixture.plan(&mut acquired, &attached, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &attached, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -1583,7 +1850,10 @@ fn networks_that_the_template_does_not_have_are_refused() { let request = tmpfs_request(&image, &binding); let generation = fixture.generation.generation(); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -1601,7 +1871,10 @@ fn networks_that_the_template_does_not_have_are_refused() { Some(SandboxNetwork::new(other, NetworkPolicy::new(NetworkAccess::Allow)).unwrap()), ); assert_eq!( - failure(fixture.plan(&mut acquired, &moved, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &moved, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -1625,7 +1898,8 @@ fn clones_are_planned_under_the_lock_that_acquired_their_lease_before_their_laun // A lease of another store. let other = Fixture::new(); let mut elsewhere = other.acquire("sandbox"); - let (kind, message) = failure(fixture.plan(&mut elsewhere, &request, NO_CONTENT)); + let plan = fixture.plan(&mut elsewhere, &request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::HostFailure); assert_eq!(kind, ErrorKind::Other, "{message}"); assert_eq!( message, @@ -1643,7 +1917,8 @@ fn clones_are_planned_under_the_lock_that_acquired_their_lease_before_their_laun drop(locked); let locked = SandboxStore::lock_existing(lease.sandbox(), &files(), None).unwrap(); let mut relocked = Acquired { lease, locked }; - let (kind, message) = failure(fixture.plan(&mut relocked, &request, NO_CONTENT)); + let plan = fixture.plan(&mut relocked, &request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::HostFailure); assert_eq!(kind, ErrorKind::Other, "{message}"); assert!( message.starts_with("a clone is planned under the lock that acquired its lease"), @@ -1659,7 +1934,8 @@ fn clones_are_planned_under_the_lock_that_acquired_their_lease_before_their_laun .begin_launch(acquired.lease.launch(), key.runtime(), &[7; CAPABILITY_LEN]) .unwrap(), ); - let (kind, message) = failure(fixture.plan(&mut acquired, &request, NO_CONTENT)); + let plan = fixture.plan(&mut acquired, &request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::HostFailure); assert_eq!(kind, ErrorKind::Other, "{message}"); assert!(!sparse_path(&acquired).exists()); } @@ -1673,8 +1949,9 @@ fn files_that_their_paths_do_not_name_as_checked_are_refused() { let link = fixture.path("images", "link.erofs"); if symlink_file(&target, &link) { let binding = DistroBinding::import(&layer(&link), PolicyVerdict::ACCEPTED).unwrap(); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&link, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&link, &binding), NO_CONTENT)), + refused(plan, Evidence::SuppliedIdentity), ( ErrorKind::Unavailable, format!( @@ -1691,7 +1968,10 @@ fn files_that_their_paths_do_not_name_as_checked_are_refused() { let scratch_binding = ScratchBinding::bind(&link).unwrap(); let request = scratch_request(&image, &binding, &link, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), ( ErrorKind::Unavailable, format!( @@ -1724,8 +2004,9 @@ fn files_that_their_paths_do_not_name_as_checked_are_refused() { } else { "is writable by other users (mode 666)" }; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&path, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&path, &binding), NO_CONTENT)), + refused(plan, Evidence::SuppliedIdentity), ( ErrorKind::Unavailable, format!("the image {} {problem}", shown(&path)) @@ -1956,8 +2237,9 @@ fn settings_that_no_activation_record_holds_are_refused_before_anything_is_creat "the workload identity's UID or GID is 0 or 4294967295", ), ] { + let plan = fixture.plan(&mut acquired, &with_workload(workload), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &with_workload(workload), NO_CONTENT)), + refused(plan, Evidence::InvalidActivate), ( ErrorKind::Unsupported, format!("no clone activates with the settings of the request: {problem}") @@ -1978,7 +2260,8 @@ fn settings_that_no_activation_record_holds_are_refused_before_anything_is_creat binding: gpt_binding, ..request.clone() }; - let (kind, message) = failure(fixture.plan(&mut acquired, &gpt_request, NO_CONTENT)); + let plan = fixture.plan(&mut acquired, &gpt_request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::InvalidFormat); assert_eq!(kind, ErrorKind::Unsupported, "{message}"); assert!( message.starts_with("the image of a clone must be an EROFS layer"), @@ -1987,3 +2270,53 @@ fn settings_that_no_activation_record_holds_are_refused_before_anything_is_creat assert_untouched(&acquired); drop(fixture.plan(&mut acquired, &request, NO_CONTENT).unwrap()); } + +#[test] +fn only_a_path_that_no_longer_names_its_supplied_file_is_the_request_s() { + // The open's refusal of a link or of a file that is not regular, which the system does not + // report, a file that went, and a path that the current user may no longer open, are the + // path's. + assert!(path_refused(&io::Error::other( + "it is a link, or not a regular file" + ))); + #[cfg(unix)] + let (refusals, failures) = { + use nix::errno::Errno; + + ( + [Errno::ENOENT, Errno::ENOTDIR, Errno::EACCES], + [Errno::EIO, Errno::EMFILE, Errno::ENOMEM], + ) + }; + // ERROR_FILE_NOT_FOUND, ERROR_PATH_NOT_FOUND, and ERROR_ACCESS_DENIED; ERROR_IO_DEVICE, + // ERROR_NOT_ENOUGH_MEMORY, and ERROR_NO_SYSTEM_RESOURCES. + #[cfg(windows)] + let (refusals, failures) = ([2_u32, 3, 5], [1117_u32, 8, 1450]); + for errno in refusals { + let error = io::Error::from_raw_os_error(errno as i32); + assert!(path_refused(&error), "{error}"); + } + for errno in failures { + let error = io::Error::from_raw_os_error(errno as i32); + assert!(!path_refused(&error), "{error}"); + } + // A file that went from its path is the request's. + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("image"); + let fault = check_at_path(&path, (1, 2), "the image").unwrap_err(); + assert_eq!(fault.evidence, Evidence::SuppliedIdentity); + assert_eq!( + fault.error.kind(), + ErrorKind::Unavailable, + "{}", + fault.error + ); + assert!( + fault + .error + .message() + .starts_with("the image cannot be used: "), + "{}", + fault.error + ); +} diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index af9233a2c..f93e85bcf 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -15,16 +15,20 @@ use std::thread; use std::time::{Duration, Instant}; use super::*; -use crate::activation::{ReadyActivation, StoreGeneration}; +use crate::activation::{ReadyActivation, StoreGeneration, TransactionId}; use crate::host::artifacts; use crate::host::platform::Transport; -use crate::host::process::{Process, ProcessRef}; +use crate::host::process::{BootId, Process, ProcessRef}; use crate::host::store::{ - CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, - LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, Recovery, RuntimeIdentity, + CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, + CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, + Recovery, RuntimeIdentity, }; use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; +use crate::host::template::Retirement; use crate::host::template::clone::launch::{Hooks, Point}; +use crate::host::template::clone::reattach::reattach_clone_with; +use crate::host::template::clone::records::{read_failure, write_failure}; use crate::host::template::store::SNAPSHOT_ARTIFACTS; use crate::host::template::test_support::artifact; @@ -37,7 +41,9 @@ const ROLE_SANDBOX: &str = "NVX_CLONE_LAUNCH_TEST_SANDBOX"; /// The environment variable that names the image of the clone of a [`launch_role`]. const ROLE_IMAGE: &str = "NVX_CLONE_LAUNCH_TEST_IMAGE"; /// The environment variable that names where a [`launch_role`] crashes: at the point `prepared` -/// or `begun` of its launch, or once its launch returned, `launched`. +/// or `begun` of its launch, once its launch returned, `launched`, at the steps `activating`, +/// `activated`, `starting`, `started`, and `committed` of its restore sequence, and once its +/// commit left the launch's journal beside the runtime record, `superseded`. const ROLE_CRASH: &str = "NVX_CLONE_LAUNCH_TEST_CRASH"; /// The environment variable that names the working directory of the VMM of a [`launch_role`]. const ROLE_VMM: &str = "NVX_CLONE_LAUNCH_TEST_VMM"; @@ -72,6 +78,9 @@ pub(super) const TIMEOUT: Duration = Duration::from_secs(60); /// Longest life of a [`restore_vmm_role`] that nothing ends, so that a failed test leaves no /// process. const VMM_LIFETIME: Duration = Duration::from_secs(180); +/// How long a [`restore_vmm_role`] whose plan closes the event's endpoint waits once it connected +/// the endpoint, and once it closed it, before it reports its error. +const CLOSE_PAUSE: Duration = Duration::from_millis(300); /// Returns `bytes` in lowercase hexadecimal. pub(super) fn hex(bytes: &[u8]) -> String { @@ -178,12 +187,73 @@ pub(super) fn assert_released(sandbox: &Path) { for name in [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ] { assert!(!exists(&sandbox.join(name)), "{name}"); } } +/// Checks that `failure`, of the clone of `lease` in the sandbox of `locked`, failed at `stage`, +/// once the clone got as far as `progress`, with `evidence`, which attributes it, and the teardown +/// `teardown`, and that the sandbox records it, with the clone's lease and the transaction of its +/// restore. +pub(super) fn assert_recorded( + failure: &CloneFailure, + lease: &Lease, + locked: &LockedStore, + (stage, progress): (Stage, Progress), + evidence: Evidence, + teardown: Teardown, +) { + let record = failure.record(); + assert_eq!( + ( + record.stage, + record.progress, + record.evidence, + &record.teardown + ), + (stage, progress, evidence, &teardown), + "{failure}" + ); + assert_eq!(failure.outcome(), evidence.outcome(), "{failure}"); + assert_eq!( + (record.store, record.lease, record.template, &record.launch), + (lease.store(), lease.id(), lease.template(), lease.launch()) + ); + let activation = fs::read(locked.clone_activation_path()).unwrap(); + let activation = ActivationRecord::decode(&activation).unwrap(); + let transaction = activation.restore().unwrap().transaction; + assert_eq!(record.transaction, Some(transaction)); + assert_eq!(record.code, failure.code()); + assert!(record.message.starts_with(failure.message()), "{failure}"); + let records = CloneRecords::read(locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(record)); + assert_eq!(&records.lease, lease); +} + +/// Returns what a reattach found, as test messages say it. +pub(super) fn describe(reattached: &ReattachedClone) -> String { + match reattached { + ReattachedClone::Live { vmm, .. } => format!("the live VM {:?}", vmm.record()), + ReattachedClone::Ended { progress, report } => { + format!("a clone that ended at its step {progress}, with the report {report:?}") + } + ReattachedClone::Failed(failure) => format!("the failure {:?}", failure.record()), + ReattachedClone::Indeterminate(reason) => format!("an indeterminate VM: {reason}"), + } +} + +/// Returns the failure that a reattach found, and fails on anything else. +pub(super) fn failed(reattached: ReattachedClone) -> CloneFailure { + match reattached { + ReattachedClone::Failed(failure) => failure, + other => panic!("the reattach found {}", describe(&other)), + } +} + /// The seam of the command of a launch that must start no VMM. fn never_spawned(_: VmmCommand<'_>) -> VmmCommand<'_> { panic!("a launch that is refused made the command of a VMM") @@ -298,16 +368,32 @@ impl VmmDir { assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); } + /// Waits until the VMM, which exits on its own, as its [plan](Self::plan) asks, has exited. + pub(super) fn wait_exit(&self) { + let reference = self.reference.as_ref().expect("the test knows the VMM"); + if let Some(process) = Process::open(reference).unwrap() { + assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + } + } + /// Changes how the VMM sends its restore-ready event, before it starts, by the words of /// `plan`: /// /// - `delay:MS` sends it after MS milliseconds; /// - `none` sends nothing, and runs on; /// - `exit` sends nothing, and exits at once; + /// - `close` connects to the event's endpoint, and closes it without a byte of the event, then + /// exits, [`CLOSE_PAUSE`] after each, as OpenVMM does once the launch of its VM worker + /// fails; + /// - `fatal:CODE`, with `exit` or `close`, writes the line with which OpenVMM reports a fatal + /// error of the time ABI code `CODE`, and the error's chain, on its standard error, after a + /// warning that names another code, before it exits; /// - `other` sends a version of the event that OpenVMM does not send; /// - `partial` sends the first bytes of the event, then closes its end of the endpoint; /// - `flush` flushes the event as OpenVMM does, which waits on Windows until the host read - /// it, before the VMM writes [`STARTED`], then notes it in [`FLUSHED`]. + /// it, before the VMM writes [`STARTED`], then notes it in [`FLUSHED`]; + /// - `report:CATEGORY:STATUS` writes OpenVMM's outcome report of a VM whose outcome is of the + /// category `CATEGORY`, with the status `STATUS`, as the VMM exits. pub(super) fn plan(&self, plan: &str) { fs::write(self.path().join(EVENT_PLAN), plan).unwrap(); } @@ -338,9 +424,9 @@ impl Drop for VmmDir { /// the control endpoint beside the snapshot, and on Windows opens the snapshot's directory and /// files sharing nothing but reading, and serves its control endpoint's named pipe; sends the /// restore-ready event as OpenVMM does, or as its [plan](VmmDir::plan) says; writes [`STARTED`]; -/// then runs until a file named [`EXIT_MARKER`] appears, or for [`VMM_LIFETIME`] at most, and -/// removes its control socket as it exits, as OpenVMM does at a normal exit. It never answers on -/// its control endpoint. +/// then runs until a file named [`EXIT_MARKER`] appears, or for [`VMM_LIFETIME`] at most, writes +/// the outcome report that its plan asks for, and removes its control socket as it exits, as +/// OpenVMM does at a normal exit. It never answers on its control endpoint. /// /// Where OpenVMM would refuse its arguments, this VMM writes why in [`STARTED`] after its /// capability and whether it sent its event. Where OpenVMM would abort a restore whose event it @@ -357,11 +443,33 @@ fn restore_vmm_role() { std::io::stdin().read_exact(&mut capability).unwrap(); let plan = fs::read_to_string(directory.join(EVENT_PLAN)).unwrap_or_default(); let words: Vec<&str> = plan.split_whitespace().collect(); - if words.contains(&"exit") { - return; - } let arguments: Vec = serde_json::from_slice(&fs::read(directory.join(ARGUMENTS)).unwrap()).unwrap(); + if words.contains(&"exit") || words.contains(&"close") { + if words.contains(&"close") { + // As OpenVMM does once the launch of its VM worker fails: it connected the event's + // endpoint before it launched the worker, closes the endpoint as the launch fails, + // tears down, and only then reports the error as it exits. + let ready = argument(&arguments, "--restore-ready-path").unwrap(); + let sink = send_ready(Path::new(&ready), b""); + thread::sleep(CLOSE_PAUSE); + drop(sink); + thread::sleep(CLOSE_PAUSE); + } + if let Some(code) = words.iter().find_map(|word| word.strip_prefix("fatal:")) { + // OpenVMM's warnings may name a time ABI code too; its fatal error leads with the + // first code of the error's chain, which follows it. + writeln!( + std::io::stderr(), + "NVX-CPU-PROFILE-FALLBACK: [E_PROFILE_UNSUPPORTED] the test's warning\n\ + fatal error: [{code}] failed to launch vm worker\n\nCaused by:\n \ + 0: [{code}] the test's restore" + ) + .unwrap(); + } + write_report(&words, &arguments); + return; + } let (held, problem) = match hold_snapshot(&arguments) { Ok(held) => (Some(held), String::new()), Err(problem) => (None, format!(" {problem}")), @@ -402,11 +510,31 @@ fn restore_vmm_role() { while !directory.join(EXIT_MARKER).exists() && started.elapsed() < VMM_LIFETIME { thread::sleep(Duration::from_millis(10)); } + write_report(&words, &arguments); if let Some(held) = held { held.release(); } } +/// Writes OpenVMM's outcome report that the word `report:CATEGORY:STATUS` among `words` of a +/// [`restore_vmm_role`]'s plan asks for, if any: a VM whose outcome is of the category +/// `CATEGORY`, with the status `STATUS`, at the `--microvm-report` path among `arguments`, as +/// OpenVMM writes it before it exits. +fn write_report(words: &[&str], arguments: &[String]) { + let Some(report) = words.iter().find_map(|word| word.strip_prefix("report:")) else { + return; + }; + let (category, status) = report.split_once(':').unwrap(); + let status: i64 = status.parse().unwrap(); + let path = argument(arguments, "--microvm-report").unwrap(); + let report = serde_json::json!({ + "schema_version": 1, + "outcome": {"operation": "managed", "category": category, "status_code": status}, + "teardown": {"workers": true}, + }); + fs::write(path, serde_json::to_vec(&report).unwrap()).unwrap(); +} + /// Returns the value of the option `name` among `arguments`, if they name it. fn argument(arguments: &[String], name: &str) -> Option { arguments @@ -994,7 +1122,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let plan = fixture.plan(&mut unbound, &request, NO_CONTENT).unwrap(); fs::remove_file(unbound.locked.binding_path()).unwrap(); assert_eq!( - failure(plan.launch_with(&mut unbound, openvmm, &runtime, hooks)), + refused_at( + plan.launch_with(&mut unbound, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure + ), ( ErrorKind::Other, format!( @@ -1011,7 +1143,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let mut second = fixture.acquire("second"); let plan = fixture.plan(&mut first, &request, NO_CONTENT).unwrap(); assert_eq!( - failure(plan.launch_with(&mut second, openvmm, &runtime, hooks)), + refused_at( + plan.launch_with(&mut second, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure + ), ( ErrorKind::Other, format!( @@ -1031,7 +1167,8 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); assert_ne!(other, runtime); let plan = fixture.plan(&mut first, &request, NO_CONTENT).unwrap(); - let (kind, message) = failure(plan.launch_with(&mut first, openvmm, &other, hooks)); + let launched = plan.launch_with(&mut first, openvmm, &other, hooks); + let (kind, message) = refused_at(launched, Stage::Launch, Evidence::RuntimeMismatch); assert_eq!(kind, ErrorKind::Unsupported, "{message}"); assert!( message.starts_with(&format!( @@ -1054,7 +1191,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { assert!(fixture.plan(&mut first, &request, failing).is_err()); let sparse = first.locked.clone_scratch_path(); assert!(!exists(&sparse)); - let (kind, message) = failure(stale.launch_with(&mut first, openvmm, &runtime, hooks)); + let (kind, message) = refused_at( + stale.launch_with(&mut first, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure, + ); assert_eq!(kind, ErrorKind::Unavailable, "{message}"); assert!( message.starts_with(&format!( @@ -1071,7 +1212,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let mut deep = fixture.acquire(&"d".repeat(100)); let plan = fixture.plan(&mut deep, &request, NO_CONTENT).unwrap(); assert_eq!( - failure(plan.launch_with(&mut deep, openvmm, &runtime, hooks)), + refused_at( + plan.launch_with(&mut deep, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::EndpointBind + ), ( ErrorKind::Unsupported, too_long(&deep.locked.restore_ready_path()) @@ -1095,6 +1240,90 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { vmm.reference = Some(restoring.starting().process().reference().clone()); } +/// Returns OpenVMM's outcome report of a VM whose guest ended it with `status`. +fn guest_exit(status: u8) -> Vec { + serde_json::to_vec(&serde_json::json!({ + "schema_version": 1, + "outcome": {"operation": "managed", "category": "guest-exit", "status_code": status}, + "teardown": {"workers": true}, + })) + .unwrap() +} + +#[test] +fn a_clone_whose_launch_failed_before_it_began_launches_no_more_until_its_release() { + let fixture = Fixture::new(); + complete_snapshot(&fixture); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let request = tmpfs_request(&image, &binding); + let runtime = runtime_of(&fixture); + let openvmm = Path::new(OPENVMM); + // The launch fails once it wrote the clone's files and recorded its progress, before it + // began, which leaves the lock's reservation. + let interrupt = |point: Point| match point { + Point::Prepared => Err(Error::io("the test fails the launch")), + Point::Begun => Ok(()), + }; + let hooks = Hooks { + command: &never_spawned, + interrupt: &interrupt, + ..Hooks::LAUNCH + }; + let mut acquired = fixture.acquire("failed"); + // The report that an earlier VM of the sandbox left, which the acquisition kept, is not the + // clone's: an inspection takes none, and the launch removes it before it records anything. + let report = acquired.locked.outcome_path(); + fs::write(&report, guest_exit(195)).unwrap(); + assert!(crate::host::report::read(&report).is_some()); + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.report), + (Progress::Planned, None) + ); + let plan = fixture.plan(&mut acquired, &request, NO_CONTENT).unwrap(); + let failure = plan + .launch_with(&mut acquired, openvmm, &runtime, hooks) + .unwrap_err(); + assert_recorded( + &failure, + &acquired.lease, + &acquired.locked, + (Stage::Launch, Progress::Launching), + Evidence::HostFailure, + Teardown::NoVm, + ); + assert!(!exists(&report)); + assert_eq!(acquired.locked.reserved(), Some(acquired.lease.launch())); + // Another launch under the lease and the lock that still reserves it is refused before it + // writes anything, so the record of the failure stays as it was. + let path = acquired.locked.clone_failure_path(); + let recorded = fs::read(&path).unwrap(); + let plan = fixture.plan(&mut acquired, &request, NO_CONTENT).unwrap(); + assert_eq!( + refused_at( + plan.launch_with(&mut acquired, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure + ), + ( + ErrorKind::Other, + format!( + "the clone of the lease {} failed, as {} records: a lease's clone fails once, \ + and the release of the lease ends it", + acquired.lease.id(), + path.display() + ) + ) + ); + assert_eq!(fs::read(&path).unwrap(), recorded); + // The release ends the lease with the record. + assert_eq!( + fixture.store.release(&mut acquired.locked).unwrap(), + Some(acquired.lease.clone()) + ); + assert!(!path.exists()); +} + /// Returns the message of the refusal of a clone's launch whose socket at `path` would not fit a /// Unix socket's address. #[cfg(target_os = "linux")] @@ -1122,7 +1351,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() command: &never_spawned, ..Hooks::LAUNCH }; - let refused = |fixture: &Fixture, sandbox: &str| { + let launch_refusal = |fixture: &Fixture, sandbox: &str, evidence: Evidence| { complete_snapshot(fixture); let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire(sandbox); @@ -1130,13 +1359,14 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() .plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT) .unwrap(); let runtime = runtime_of(fixture); - let refusal = failure(plan.launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks)); + let launched = plan.launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks); + let refusal = refused_at(launched, Stage::Launch, evidence); assert_unlaunched(fixture, &acquired); (refusal, fixture.store.clone_socket_path(&acquired.lease)) }; // A store whose path leaves no room for the socket of a clone's VMM in a payload. let fixture = Fixture::in_store(|directory| deep(directory).join("templates"), |_, _, _| {}); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.as_os_str().len() >= 108); assert_eq!((kind, message), (ErrorKind::Unsupported, too_long(&socket))); // A store whose path is short, but long once its links are resolved, at which OpenVMM listens. @@ -1148,7 +1378,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.as_os_str().len() < 108, "{}", socket.display()); let resolved = filesystem::canonicalize(socket.parent().unwrap()) .unwrap() @@ -1168,7 +1398,8 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let bind = Evidence::EndpointBind; + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", bind); assert_eq!( (kind, message), ( @@ -1192,7 +1423,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.to_str().is_none(), "{}", socket.display()); assert_eq!( (kind, message), @@ -1219,7 +1450,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.to_str().is_some(), "{}", socket.display()); let resolved = filesystem::canonicalize(socket.parent().unwrap()) .unwrap() @@ -1248,12 +1479,13 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() .unwrap(); let payload = fixture.store.payload_dir(fixture.generation.generation()); fs::set_permissions(&payload, fs::Permissions::from_mode(0o500)).unwrap(); - let refusal = failure(plan.launch_with( + let launched = plan.launch_with( &mut acquired, Path::new(OPENVMM), &runtime_of(&fixture), hooks, - )); + ); + let refusal = refused_at(launched, Stage::Launch, Evidence::MalformedGeneration); fs::set_permissions(&payload, fs::Permissions::from_mode(0o700)).unwrap(); assert_unlaunched(&fixture, &acquired); assert_eq!( @@ -1291,6 +1523,8 @@ fn launch_role() { let store = Store::open(dir).unwrap(); let generation = StoreGeneration::new(generation.parse().unwrap()); let indexed = store.index().unwrap().get(generation).unwrap().clone(); + let journal = sandbox.join(JOURNAL_NAME); + let kept = sandbox.join(format!("{JOURNAL_NAME}.kept")); let mut acquired = store .acquire(generation, sandbox, &files(), &after(&indexed, 1), None) .unwrap(); @@ -1315,6 +1549,7 @@ fn launch_role() { let hooks = Hooks { command: &command, interrupt: &interrupt, + ..Hooks::LAUNCH }; let restoring = plan .launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks) @@ -1338,7 +1573,16 @@ fn launch_role() { if step == "started" { crash(&step); } + if step == "superseded" { + fs::hard_link(&journal, &kept).unwrap(); + } let committed = started.commit().unwrap(); + if step == "superseded" { + // The commit's removal of the journal failed, as a commit tolerates: the journal, the + // same file, remains beside the runtime record that supersedes it. + fs::rename(&kept, &journal).unwrap(); + crash(&step); + } if step == "committed" { crash(&step); } @@ -1454,15 +1698,23 @@ impl Crashed { /// Leases a fresh fixture's generation to a new sandbox, whose clone a [`launch_role`] /// launches until it crashes at `step`. fn at(step: &str) -> Self { + Self::at_with(step, "") + } + + /// Leases a fresh fixture's generation to a new sandbox, whose clone a [`launch_role`] + /// launches until it crashes at `step`, with a VMM whose [plan](VmmDir::plan) holds `words`. + fn at_with(step: &str, words: &str) -> Self { let fixture = Fixture::new(); complete_snapshot(&fixture); let (image, _) = fixture.image("distro.erofs", IMAGE_LEN); let sandbox = fixture.provision("sandbox"); let mut vmm = VmmDir::new(); + let mut plan = words.to_owned(); if !matches!(step, "prepared" | "begun" | "launched") { // The restore sequence reads the event, whose flush waits for that on Windows. - vmm.plan("flush"); + plan.push_str(" flush"); } + vmm.plan(&plan); vmm.reference = run_launch_role(step, &fixture, &sandbox, &image, vmm.path()); let lease = Lease::of(&SandboxStore::open(&sandbox).unwrap()) .unwrap() @@ -1515,6 +1767,28 @@ impl Crashed { } } + /// Takes the clone of the sandbox of `locked` over, under `runtime`, through `control`, with + /// [`reattach_clone`], until the result is determinate, as a caller retries an indeterminate + /// reattach, and returns it. + fn take_over( + &self, + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + ) -> ReattachedClone { + let deadline = Instant::now() + TIMEOUT; + loop { + let session = Instant::now() + TIMEOUT; + match reattach_clone(locked, runtime, control, session, TIMEOUT).unwrap() { + ReattachedClone::Indeterminate(reason) if Instant::now() < deadline => { + eprintln!("reattach is indeterminate: {reason}"); + thread::sleep(Duration::from_millis(100)); + } + reattached => return reattached, + } + } + } + /// Checks that the store refuses to release the lease of the sandbox of `locked`, whose /// launch has not ended, and that the sandbox keeps it. fn refuses_release(&self, locked: &mut LockedStore) { @@ -1531,6 +1805,7 @@ impl Crashed { Some(self.lease.clone()) ); assert_released(&self.sandbox); + assert!(CloneRecords::read(locked).unwrap().is_none()); assert!(!exists(&self.socket())); let snapshot = self.fixture.store.snapshot_dir(&self.fixture.generation); for name in SNAPSHOT_ARTIFACTS { @@ -1579,6 +1854,58 @@ fn a_launch_whose_caller_crashes_once_it_began_ends_as_reattach_recovers_it() { crashed.releases(&mut locked); } +#[test] +fn a_launch_that_began_shows_the_report_of_a_vmm_that_its_journal_does_not_record() { + // The caller crashed once the launch began, before the journal recorded the VMM that it + // spawned, which left the report: the clone's, since the launch removed any earlier VM's + // report before it recorded anything, though the clone's progress shows no spawn. An + // inspection takes the report once a reattach of the clone ended the launch, recording its + // failure, and once a recovery of the sandbox's launch did, recording none. + let shown = |records: &CloneRecords| { + let report = records + .report + .as_ref() + .expect("the report of the clone's VM"); + (report.category.clone(), report.status_code) + }; + let ended = (Some("guest-exit".to_owned()), Some(3)); + for of_the_clone in [true, false] { + let crashed = Crashed::at("begun"); + let mut locked = crashed.lock(); + fs::write(locked.outcome_path(), guest_exit(3)).unwrap(); + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.journal.is_some()), + (Progress::Launching, true) + ); + assert_eq!(shown(&records), ended); + if of_the_clone { + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.record().teardown, Teardown::Exited); + } else { + match crashed.reattach(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + } + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + ( + records.progress, + records.journal.is_none(), + records.failure.is_some() + ), + (Progress::Launching, true, of_the_clone) + ); + assert_eq!(shown(&records), ended, "{of_the_clone}"); + crashed.releases(&mut locked); + } +} + #[test] fn a_launch_whose_caller_crashes_once_its_vmm_started_ends_the_vmm_as_reattach_recovers_it() { let crashed = Crashed::at("launched"); @@ -1671,3 +1998,704 @@ fn a_restore_whose_caller_crashes_once_it_committed_keeps_its_lease_while_its_vm assert!(!exists(&crashed.socket())); crashed.releases(&mut locked); } + +/// The Ready of a clone that a [`ReattachControl`] answers, from the activation of the record that +/// the clone's Activate carries. +type Answered = fn(ReadyActivation) -> Option; + +/// The adapter's framing of the sessions of a reattach: it attaches a session to each connection +/// to the clone's control endpoint, keeps the capability of each, and the record of each Activate, +/// which it answers as its `answer` says. +struct ReattachControl { + /// The capabilities of the sessions, in order. + capabilities: Vec<[u8; CAPABILITY_LEN]>, + /// The records of the Activates, in order. + records: Vec>, + /// How the clone answers an Activate. + answer: Answered, +} + +impl ReattachControl { + /// Returns a framing whose clone answers each Activate as `answer` says. + fn answering(answer: Answered) -> Self { + Self { + capabilities: Vec::new(), + records: Vec::new(), + answer, + } + } +} + +impl CloneControl for ReattachControl { + type Session = Box; + type Error = io::Error; + + fn attach( + &mut self, + transport: Box, + capability: &[u8; CAPABILITY_LEN], + _deadline: Instant, + ) -> io::Result> { + self.capabilities.push(*capability); + Ok(transport) + } + + fn activate( + &mut self, + _session: &mut Box, + record: &[u8], + _deadline: Instant, + ) -> io::Result> { + self.records.push(record.to_vec()); + let record = ActivationRecord::decode(record).unwrap(); + Ok((self.answer)(ReadyActivation::of(&record))) + } + + fn start(&mut self, _session: &mut Box, _deadline: Instant) -> io::Result<()> { + panic!("a reattach starts no sandbox") + } + + fn lost(&self, _error: &io::Error) -> bool { + false + } +} + +#[test] +fn a_clone_whose_caller_crashed_once_it_committed_is_taken_over_in_a_new_session() { + let crashed = Crashed::at("committed"); + let reference = crashed.vmm.reference.clone().unwrap(); + let started = crashed.vmm.started(); + let (capability, _) = started.split_once(' ').unwrap(); + // A tombstone quarantines the clone's generation, which a retirement deletes once its leases + // have drained: neither takes the clone's lease from it. + let store = &crashed.fixture.store; + let generation = crashed.fixture.generation.generation(); + store.tombstone(generation, None).unwrap(); + assert_eq!( + store.retire(generation, None).unwrap(), + Retirement::Draining(vec![crashed.lease.clone()]) + ); + let mut locked = crashed.lock(); + let recorded = fs::read(locked.clone_activation_path()).unwrap(); + let mut control = ReattachControl::answering(Some); + let (vmm, session) = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, session } => (vmm, session), + other => panic!("the reattach found {}", describe(&other)), + }; + // The VM is the one that the launch committed, whose clone answered the Activate of the + // sandbox's record, byte for byte, in a session that the launch's capability authenticated, + // and runs on, with its lease. + assert_eq!(vmm.process().reference(), &reference); + assert_eq!(control.records, [recorded]); + let attached: Vec = control.capabilities.iter().map(|key| hex(key)).collect(); + assert_eq!(attached, [capability]); + drop(session); + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.failure), + (Progress::Committed, None) + ); + crashed.refuses_release(&mut locked); + assert_eq!( + vmm.terminate(&mut locked, TIMEOUT).unwrap(), + StopOutcome::Forced + ); + crashed.releases(&mut locked); + // Its lease released, the retirement deletes the generation. + assert_eq!(store.retire(generation, None).unwrap(), Retirement::Retired); +} + +#[test] +fn a_journal_that_a_clone_s_commit_left_is_no_failure_of_the_clone() { + // The commit of the clone's launch could not remove the launch's journal, which remains + // beside the runtime record of the same launch, as its caller crashed. + let crashed = Crashed::at("superseded"); + let mut locked = crashed.lock(); + let journal = locked + .journal() + .unwrap() + .expect("the commit left the journal"); + let runtime = locked + .runtime() + .unwrap() + .expect("the commit wrote its record"); + assert_eq!(journal.launch(), runtime.launch()); + // Each reattach takes the clone over, and records no failure: the first removes the journal. + let mut control = ReattachControl::answering(Some); + let mut live = None; + for attempt in 0..2 { + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach {attempt} found {}", describe(&other)), + }; + assert_eq!( + Some(vmm.process().reference()), + crashed.vmm.reference.as_ref() + ); + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.failure, records.journal.is_some()), + (Progress::Committed, None, false), + "{attempt}" + ); + live = Some(vmm); + } + assert_eq!(control.records.len(), 2); + crashed.refuses_release(&mut locked); + let vmm = live.unwrap(); + assert_eq!( + vmm.terminate(&mut locked, TIMEOUT).unwrap(), + StopOutcome::Forced + ); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_ends_a_clone_that_it_may_not_keep_through_its_verified_identity() { + let reattach = (Stage::Reattach, Progress::Committed); + // A clone that runs under another runtime identity, which might not speak this one's + // protocol: the reattach sends its guest nothing. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &other, &mut control)); + assert_eq!(failure.code(), ErrorCode::RuntimeMismatch, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + let mismatch = Evidence::RuntimeMismatch; + assert_recorded(&failure, lease, &locked, reattach, mismatch, terminated); + assert_eq!(failure.outcome(), Outcome::Host); + assert!(control.capabilities.is_empty()); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + // A clone whose Ready reports another activation than its record's. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(|_| Some(ReadyActivation { transaction: None })); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Protocol, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + assert_recorded( + &failure, + lease, + &locked, + reattach, + Evidence::Protocol, + terminated, + ); + assert_eq!(control.records.len(), 1); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_ends_the_launch_of_a_caller_that_crashed_and_records_its_failure() { + let cases = [ + ( + "begun", + (Stage::Launch, Progress::Launching), + Evidence::LostStatus, + Teardown::Exited, + ), + ( + "launched", + (Stage::Restore, Progress::Spawned), + Evidence::ForcedKill, + Teardown::Terminated, + ), + ( + "activated", + (Stage::Start, Progress::Ready), + Evidence::ForcedKill, + Teardown::Terminated, + ), + ( + "started", + (Stage::Commit, Progress::Started), + Evidence::ForcedKill, + Teardown::Terminated, + ), + ]; + for (step, at, evidence, teardown) in cases { + let crashed = Crashed::at(step); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{step}: {failure}"); + assert_recorded(&failure, &crashed.lease, &locked, at, evidence, teardown); + assert!(control.capabilities.is_empty(), "{step}"); + // Once the launch has ended, a reattach finds its failure as it was recorded. + let again = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(again.record(), failure.record(), "{step}"); + assert_eq!(again.message(), failure.record().message, "{step}"); + crashed.releases(&mut locked); + } + // A caller that crashed before its launch began left no VM, and no failure. + let crashed = Crashed::at("prepared"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Launching, + report: None, + } => {} + other => panic!("the reattach found {}", describe(&other)), + } + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_records_the_failure_of_an_interrupted_launch_before_it_ends_the_launch() { + let crashed = Crashed::at("launched"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + // The restarted caller crashes in turn as its reattach would end the interrupted launch. + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Err(Error::other("the test's caller crashes")), + ); + assert!(crash.is_err()); + // The sandbox records the failure, whose teardown is pending, beside the launch's journal, + // which keeps the lease. + let records = CloneRecords::read(&locked).unwrap().unwrap(); + let pending = records + .failure + .expect("the reattach recorded the failure before the recovery"); + assert_eq!( + (pending.stage, pending.progress, pending.evidence), + (Stage::Restore, Progress::Spawned, Evidence::LostStatus) + ); + assert_eq!(pending.teardown, Teardown::Pending); + assert!(records.journal.is_some()); + crashed.refuses_release(&mut locked); + // The next reattach ends the launch, and resolves the teardown of the recorded failure. + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{failure}"); + assert_recorded( + &failure, + &crashed.lease, + &locked, + (Stage::Restore, Progress::Spawned), + Evidence::LostStatus, + Teardown::Terminated, + ); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + +#[test] +fn a_clone_whose_vm_exited_once_it_committed_is_classified_by_its_report() { + let run = (Stage::Run, Progress::Committed); + // A repair's status is evidence of the repair stage, after the clone's Ready too, and a VM + // that ended without a report of the guest's own end lost the clone's status. + let cases = [ + ("report:guest-exit:195", Evidence::RepairStatus, Some(195)), + ("", Evidence::LostStatus, None), + ]; + for (words, evidence, status) in cases { + let crashed = Crashed::at_with("committed", words); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{words}: {failure}"); + let exited = Teardown::Exited; + assert_recorded(&failure, &crashed.lease, &locked, run, evidence, exited); + assert_eq!(failure.record().status, status, "{words}"); + assert!(locked.runtime().unwrap().is_none(), "{words}"); + crashed.releases(&mut locked); + } + // A VM that its guest ended with a status that is no evidence of a failure ended. + let crashed = Crashed::at_with("committed", "report:guest-exit:3"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Committed, + report: Some(report), + } => { + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(3))); + } + other => panic!("the reattach found {}", describe(&other)), + } + // An inspection of the clone takes the report of its VM as the reattach did, and no failure. + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert!(records.failure.is_none()); + let report = records.report.expect("the report of the clone's VM"); + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(3))); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_records_the_failure_of_a_vm_that_exited_before_it_forgets_the_vm() { + let run = (Stage::Run, Progress::Committed); + let crashed = Crashed::at_with("committed", "report:guest-exit:195"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + // The restarted caller crashes once its reattach cleared the runtime record of the VM that + // exited, before it recorded the failure once more. + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + supervisor::reattach_reporting(locked, runtime, exited)?; + Err(Error::other("the test's caller crashes")) + }, + ); + assert!(crash.is_err()); + // The sandbox recorded the failure before the runtime record went, with its teardown pending. + let pending = read_failure(&locked.clone_failure_path()) + .unwrap() + .expect("the reattach recorded the failure before it cleared the runtime record"); + assert_eq!( + (pending.stage, pending.progress, pending.evidence), + (run.0, run.1, Evidence::RepairStatus) + ); + assert_eq!(pending.teardown, Teardown::Pending); + assert_eq!(pending.status, Some(195)); + assert!(locked.runtime().unwrap().is_none()); + // The next reattach finishes it, with the status that it holds, even once no read finds the + // report, as when another process holds it. + fs::write(locked.outcome_path(), b"not a report").unwrap(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded( + &failure, + lease, + &locked, + run, + Evidence::RepairStatus, + exited, + ); + assert_eq!(failure.record().status, Some(195)); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_takes_the_outcome_report_of_a_vm_that_exited_as_it_first_read_it() { + let run = (Stage::Run, Progress::Committed); + let crashed = Crashed::at_with("committed", "report:guest-exit:195"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let outcome = locked.outcome_path(); + let mut control = ReattachControl::answering(Some); + // Once the reattach read the report of the VM that exited, before it cleared the VM's runtime + // record, no read finds the report, as when another process holds it: the reattach takes the + // report as it first read it. + let reattached = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + supervisor::reattach_reporting(locked, runtime, |record| { + exited(record); + fs::write(&outcome, b"not a report").unwrap(); + }) + }, + ) + .unwrap(); + let failure = failed(reattached); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded( + &failure, + lease, + &locked, + run, + Evidence::RepairStatus, + exited, + ); + assert_eq!(failure.record().status, Some(195)); + crashed.releases(&mut locked); +} + +/// Records the failure of the clone of `lease`, in the sandbox of `locked`, at `at`, with +/// `evidence` and `error`, whose teardown is `teardown`, with the transaction `transaction`, as +/// a failure of the clone's restore sequence records it before its teardown, which a crash of its +/// caller interrupted, if the teardown is pending, or once a teardown that could not end the +/// clone's VMM, if it is unresolved. +fn record_earlier( + lease: &Lease, + locked: &LockedStore, + (at, transaction): ((Stage, Progress), Option), + (evidence, teardown): (Evidence, Teardown), + error: &Error, +) { + let record = FailureRecord::new(lease, transaction, at, evidence, None, teardown, error); + write_failure(&locked.clone_failure_path(), &record).unwrap(); +} + +/// Returns the transaction of the restore that the activation record of the clone in the sandbox +/// of `locked` names. +fn transaction_of(locked: &LockedStore) -> Option { + let activation = fs::read(locked.clone_activation_path()).unwrap(); + let activation = ActivationRecord::decode(&activation).unwrap(); + activation.restore().map(|restore| restore.transaction) +} + +#[test] +fn a_reattach_finishes_an_earlier_failure_with_what_the_clone_s_vmm_left_as_it_ended() { + let restore = (Stage::Restore, Progress::Spawned); + let mut control = ReattachControl::answering(Some); + let unresolved = Teardown::Unresolved("the test could not end the clone's VMM".to_owned()); + for earlier in [Teardown::Pending, unresolved] { + // A failure of the clone's restore sequence, whose teardown a crash of its caller + // interrupted, or could not end the clone's VMM, which the guest then ended with the + // status of a failed repair, which prevails. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + let mut locked = crashed.lock(); + let reset = Error::protocol("the test's clone reset its session"); + let at = (restore, transaction_of(&locked)); + let teardown = earlier.clone(); + record_earlier( + &crashed.lease, + &locked, + at, + (Evidence::Reset, teardown), + &reset, + ); + crashed.vmm.end_and_wait(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!( + failure.kind(), + ErrorKind::Protocol, + "{earlier:?}: {failure}" + ); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded( + &failure, + lease, + &locked, + restore, + Evidence::RepairStatus, + exited, + ); + assert_eq!(failure.record().status, Some(195), "{earlier:?}"); + assert_eq!(failure.message(), "the test's clone reset its session"); + crashed.releases(&mut locked); + // One whose VMM exited before its restore-ready event, with OpenVMM's error, which tells + // that this host cannot present the template's CPU, once its record was written. + let crashed = Crashed::at_with("launched", "close fatal:E_CPU_GENERATION"); + let mut locked = crashed.lock(); + let lost = Error::exited("the test's VMM exited before its restore-ready event"); + let at = (restore, transaction_of(&locked)); + let teardown = earlier.clone(); + record_earlier( + &crashed.lease, + &locked, + at, + (Evidence::LostStatus, teardown), + &lost, + ); + crashed.vmm.wait_exit(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{earlier:?}: {failure}"); + let (lease, incompatible) = (&crashed.lease, Evidence::IncompatibleHost); + assert_recorded( + &failure, + lease, + &locked, + restore, + incompatible, + Teardown::Exited, + ); + let message = failure.message(); + assert!( + message.starts_with("the test's VMM exited before its restore-ready event; ") + && message.ends_with("failed to launch vm worker") + && message.matches("[E_CPU_GENERATION]").count() == 1, + "{earlier:?}: {failure}" + ); + // A later reattach finds the failure as it was recorded. + let again = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(again.record(), failure.record(), "{earlier:?}"); + crashed.releases(&mut locked); + } + assert!(control.capabilities.is_empty()); + // The record keeps the transaction of the restore that failed, whatever the sandbox's + // activation record names since. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + let mut locked = crashed.lock(); + let reset = Error::protocol("the test's clone reset its session"); + let transaction = Some(TransactionId::from_bytes([0x5a; 16])); + assert_ne!(transaction, transaction_of(&locked)); + let pending = (Evidence::Reset, Teardown::Pending); + record_earlier( + &crashed.lease, + &locked, + (restore, transaction), + pending, + &reset, + ); + crashed.vmm.end_and_wait(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let record = failure.record(); + assert_eq!( + (record.transaction, record.evidence, &record.teardown), + (transaction, Evidence::RepairStatus, &Teardown::Exited) + ); + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert_eq!(recorded.as_ref(), Some(record)); + crashed.releases(&mut locked); +} + +#[test] +fn a_clone_whose_progress_log_may_have_lost_its_steps_may_have_run_its_workload() { + // A launch whose caller crashed once the clone was ready, or once it sent the start of the + // clone's sandbox, whose VM the guest ended with a status that the clone's root gives before + // that start, and a workload may give after it. + let (current, rebooted) = (BootId::current().unwrap(), BootId::from_bytes([0xee; 16])); + assert_ne!(current, rebooted); + let ready = (Stage::Start, Progress::Ready); + let spawned = (Stage::Restore, Progress::Spawned); + let cases = [ + // Its intact log tells that the clone was only ready: its root's status is evidence. + ("intact", "activated", ready, Evidence::UnusableRoot), + // A log that was lost, or damaged, tells nothing, and one that the launch wrote during + // another boot of the host may have lost the steps of the start with the host's crash: + // the status is no evidence of the root's failure, and the clone lost its status. + ("lost", "started", spawned, Evidence::LostStatus), + ("damaged", "started", spawned, Evidence::LostStatus), + ("rebooted", "started", ready, Evidence::LostStatus), + ]; + for (damage, step, at, evidence) in cases { + let crashed = Crashed::at_with(step, "report:guest-exit:54"); + let mut locked = crashed.lock(); + let log = locked.clone_progress_path(); + let launch = crashed.lease.launch(); + match damage { + "lost" => fs::remove_file(log).unwrap(), + "damaged" => fs::write(log, b"not a progress log\n").unwrap(), + "rebooted" => fs::write( + log, + format!( + "nvx-clone-progress 1 {launch} {rebooted}\nspawned\nevent\nacknowledged\n\ + activating\nready\n" + ), + ) + .unwrap(), + _ => {} + } + crashed.vmm.end_and_wait(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let record = failure.record(); + assert_eq!( + ( + (record.stage, record.progress), + record.evidence, + record.status, + &record.teardown + ), + (at, evidence, Some(54), &Teardown::Exited), + "{damage}: {failure}" + ); + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert_eq!(recorded.as_ref(), Some(record), "{damage}"); + crashed.releases(&mut locked); + } +} + +#[test] +fn a_clone_that_its_holder_stopped_ended_whatever_a_reboot_left_of_its_progress_log() { + let other = BootId::from_bytes([0xee; 16]); + assert_ne!(other, BootId::current().unwrap()); + for rebooted in [false, true] { + let crashed = Crashed::at_with("committed", "report:guest-exit:54"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach found {}", describe(&other)), + }; + // Its holder stops the clone, whose workload ends with a status that the clone's root + // gives before the start of its sandbox, which clears the store's records of its VM. + let stopped = vmm + .stop(&mut locked, Instant::now() + TIMEOUT, None, |_, _| { + crashed.vmm.end(); + Ok::<(), io::Error>(()) + }) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful, "{rebooted}"); + assert!(locked.runtime().unwrap().is_none(), "{rebooted}"); + if rebooted { + // A crash of the host, before the release, lost the last lines of the progress log, + // which the launch wrote during the boot that the crash ended. + let launch = crashed.lease.launch(); + fs::write( + locked.clone_progress_path(), + format!( + "nvx-clone-progress 1 {launch} {other}\nspawned\nevent\nacknowledged\n\ + activating\nready\n" + ), + ) + .unwrap(); + } + // The clone ended, and its workload's status is no failure of its root. + let progress = if rebooted { + Progress::Ready + } else { + Progress::Committed + }; + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: ended, + report: Some(report), + } if ended == progress => { + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(54)), "{rebooted}"); + } + other => panic!("{rebooted}: the reattach found {}", describe(&other)), + } + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert!(recorded.is_none(), "{rebooted}: {recorded:?}"); + crashed.releases(&mut locked); + } +} + +#[test] +fn an_interrupted_launch_that_a_reattach_cannot_end_yet_keeps_its_lease_until_one_can() { + let crashed = Crashed::at("activated"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + // The reattach cannot tell whether the launch's VMM still runs. + let reason = "the test cannot end the launch yet"; + let reattached = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Ok(Reattached::Indeterminate(reason.to_owned())), + ) + .unwrap(); + let failure = failed(reattached); + let ready = (Stage::Start, Progress::Ready); + let unresolved = Teardown::Unresolved(reason.to_owned()); + let lost = Evidence::LostStatus; + assert_recorded(&failure, &crashed.lease, &locked, ready, lost, unresolved); + assert!(failure.message().ends_with(reason), "{failure}"); + // The journal remains, and with it the lease, and the clone's records. + assert!(locked.journal().unwrap().is_some()); + crashed.refuses_release(&mut locked); + // A reattach that can end the launch resolves the teardown, and keeps the failure as it was. + let resolved = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let terminated = Teardown::Terminated; + assert_recorded(&resolved, &crashed.lease, &locked, ready, lost, terminated); + assert_eq!(resolved.message(), failure.record().message); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} diff --git a/nvx-rs/src/host/template/clone/tests/restore.rs b/nvx-rs/src/host/template/clone/tests/restore.rs index e905ee0ad..99f8fe420 100644 --- a/nvx-rs/src/host/template/clone/tests/restore.rs +++ b/nvx-rs/src/host/template/clone/tests/restore.rs @@ -13,14 +13,17 @@ use std::collections::VecDeque; use std::thread; use std::time::{Duration, Instant}; -use super::launch::{OPENVMM, TIMEOUT, VmmDir, complete_snapshot, fake_vmm}; +use super::launch::{ + OPENVMM, TIMEOUT, VmmDir, assert_recorded, complete_snapshot, failed, fake_vmm, runtime_of, +}; use super::*; use crate::activation::{ReadyActivation, RestoreSection, TransactionId}; use crate::host::platform::Transport; -use crate::host::process::Process; +use crate::host::process::{BootId, Process}; use crate::host::store::{CLONE_ACTIVATION_NAME, Recovery}; -use crate::host::supervisor::StopOutcome; +use crate::host::supervisor::{Starting, StopOutcome}; use crate::host::template::clone::launch::Hooks; +use crate::host::template::clone::records::{End, read_failure}; /// How long a VMM whose plan delays its event waits before it sends it. const DELAY: Duration = Duration::from_millis(300); @@ -288,6 +291,17 @@ fn launch<'a>( acquired: &'a mut Acquired, image: &(PathBuf, DistroBinding), vmm: &mut VmmDir, +) -> (Restoring<'a>, Launched) { + launch_ending(fixture, acquired, image, vmm, Hooks::LAUNCH.end) +} + +/// Launches the clone as [`launch`] does, whose failures end the launch with `end`. +fn launch_ending<'a>( + fixture: &Fixture, + acquired: &'a mut Acquired, + image: &(PathBuf, DistroBinding), + vmm: &mut VmmDir, + end: End, ) -> (Restoring<'a>, Launched) { let request = tmpfs_request(&image.0, &image.1); let plan = fixture.plan(acquired, &request, NO_CONTENT).unwrap(); @@ -297,6 +311,7 @@ fn launch<'a>( let command = fake_vmm(&dir, &made); let hooks = Hooks { command: &command, + end, ..Hooks::LAUNCH }; let before = Instant::now(); @@ -345,6 +360,25 @@ fn assert_launch_ended(fixture: &Fixture, acquired: &mut Acquired, vmm: &VmmDir) ); } +/// Checks that `failure`, of the clone of `acquired`, failed at `stage`, once the clone got as far +/// as `progress`, with `evidence`, and the teardown `teardown`, as [`assert_recorded`] does. +fn assert_failed( + failure: &CloneFailure, + acquired: &Acquired, + at: (Stage, Progress), + evidence: Evidence, + teardown: Teardown, +) { + assert_recorded( + failure, + &acquired.lease, + &acquired.locked, + at, + evidence, + teardown, + ); +} + #[test] fn a_clone_is_activated_once_its_restore_is_ready_then_started_and_committed() { let (fixture, image) = fixture(); @@ -417,6 +451,26 @@ fn a_clone_is_activated_once_its_restore_is_ready_then_started_and_committed() { ); let (running, session) = started.commit().unwrap(); assert_eq!(session.number, 1); + // The progress log records each step, the spawn and the acknowledgment with the steps that + // follow them at once, after a first line that names the launch and the current boot of the + // host, and the sandbox's records show the committed clone. + let log = fs::read_to_string(acquired.locked.clone_progress_path()).unwrap(); + assert_eq!( + log, + format!( + "nvx-clone-progress 1 {} {}\nspawned\nevent\nacknowledged\nactivating\nready\n\ + starting\nstarted\ncommitted\n", + acquired.lease.launch(), + BootId::current().unwrap() + ) + ); + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.failure, records.journal), + (Progress::Committed, None, None) + ); + assert_eq!(records.lease, acquired.lease); + assert_eq!(records.runtime, acquired.locked.runtime().unwrap()); // The runtime record names the VM, whose lease lasts as long as it runs. let locked = &mut acquired.locked; let recorded = locked.runtime().unwrap().unwrap(); @@ -471,6 +525,9 @@ fn a_ready_that_does_not_report_the_record_s_activation_ends_the_launch() { ); // One session, which the host ended, and no start. assert_eq!(control.calls.len(), 2, "{name}: {:?}", control.calls); + let ready = (Stage::Activation, Progress::Activating); + let terminated = Teardown::Terminated; + assert_failed(&error, &acquired, ready, Evidence::Protocol, terminated); assert_launch_ended(&fixture, &mut acquired, &vmm); } } @@ -542,6 +599,13 @@ fn a_retry_never_sends_another_record_nor_takes_another_activation() { assert!(error.message().contains("no longer holds"), "{error}"); assert_eq!(control.sessions, 1); assert_eq!(control.records(), [&original.encode()[..]]); + // The host's own record changed: the record of the failure names the launch's transaction. + let record = error.record(); + assert_eq!( + (record.evidence, record.outcome, record.stage), + (Evidence::HostFailure, Outcome::Host, Stage::Activation) + ); + assert_eq!(record.transaction, Some(restore.transaction)); assert_launch_ended(&fixture, &mut acquired, &vmm); // The Ready of the retry reports another transaction. @@ -558,6 +622,14 @@ fn a_retry_never_sends_another_record_nor_takes_another_activation() { .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Protocol, "{error}"); assert_eq!(control.sessions, 2); + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::Protocol, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -576,6 +648,14 @@ fn only_lost_sessions_are_replaced_and_only_so_many_times() { assert_eq!(error.kind(), ErrorKind::Protocol, "{error}"); assert_eq!(error.message(), "cannot activate it"); assert_eq!(control.sessions, 1); + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::Protocol, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); // A session lost again and again ends the sequence once the last of its sessions is lost. @@ -591,6 +671,13 @@ fn only_lost_sessions_are_replaced_and_only_so_many_times() { assert!(error.message().contains("was lost"), "{error}"); control.assert_activates_first(); assert_eq!(control.sessions, MAX_ACTIVATION_SESSIONS); + assert_failed( + &error, + &acquired, + ready, + Evidence::Reset, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -611,6 +698,15 @@ fn a_clone_that_took_the_reset_of_a_lost_session_before_its_ready_ends_the_retri assert_eq!(error.kind(), ErrorKind::Exited, "{error}"); assert_eq!(control.sessions, 1); drop(control); + // The clone's status was lost with its VMM, which ended without a report. + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::LostStatus, + Teardown::Exited, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -631,6 +727,14 @@ fn the_restore_to_ready_deadline_runs_from_the_launch_through_every_step() { assert_eq!(error.kind(), ErrorKind::TimedOut, "{error}"); assert!(launched.before.elapsed() >= SHORT); assert!(control.calls.is_empty()); + let restore = (Stage::Restore, Progress::Spawned); + assert_failed( + &error, + &acquired, + restore, + Evidence::Timeout, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); // A clone that never answers its Activate, after a late event: the attachment and the @@ -650,6 +754,14 @@ fn the_restore_to_ready_deadline_runs_from_the_launch_through_every_step() { launched.assert_deadline(deadline, deadlines.ready); } assert!(launched.before.elapsed() >= deadlines.ready); + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::Timeout, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -681,6 +793,14 @@ fn the_ready_to_start_deadline_runs_from_the_ready_and_a_start_is_never_resent() deadline: start }) ); + let starting = (Stage::Start, Progress::Starting); + assert_failed( + &error, + &acquired, + starting, + Evidence::Timeout, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); // A start that fails, even in a lost session, is not sent again. @@ -698,6 +818,13 @@ fn the_ready_to_start_deadline_runs_from_the_ready_and_a_start_is_never_resent() .filter(|call| matches!(call, Call::Start { .. })) .count(); assert_eq!((control.sessions, starts), (1, 1)); + assert_failed( + &error, + &acquired, + starting, + Evidence::Reset, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -705,14 +832,59 @@ fn the_ready_to_start_deadline_runs_from_the_ready_and_a_start_is_never_resent() fn a_vmm_that_exits_or_sends_anything_but_its_event_ends_the_launch() { let (fixture, image) = fixture(); // A VMM that exits before its event, as OpenVMM does once it cannot restore the snapshot, ends - // the wait at once; another version of the event, and part of it, are refused. + // the wait at once, and loses the clone's status, unless OpenVMM's fatal error tells that the + // host cannot present the template's CPU or time; another version of the event, and part of + // it, are refused, and the VMM that sent them is terminated. let cases = [ - ("exit", ErrorKind::Exited), - ("other flush", ErrorKind::Protocol), - ("partial flush", ErrorKind::Protocol), + ( + "exit", + "exit", + ErrorKind::Exited, + Evidence::LostStatus, + Teardown::Exited, + ), + ( + "incompatible", + "exit fatal:E_TSC_RATE_TOLERANCE", + ErrorKind::Exited, + Evidence::IncompatibleHost, + Teardown::Exited, + ), + ( + "contract", + "exit fatal:E_MANIFEST_TIME", + ErrorKind::Exited, + Evidence::LostStatus, + Teardown::Exited, + ), + // OpenVMM connects the event's endpoint before it launches its VM worker, whose launch + // fails on a host that cannot present the template's CPU or time: it closes the endpoint + // without the event, and reports its error only as it exits, which the wait awaits. + ( + "closed", + "close fatal:E_TSC_RATE_TOLERANCE", + ErrorKind::Exited, + Evidence::IncompatibleHost, + Teardown::Exited, + ), + ( + "other", + "other flush", + ErrorKind::Protocol, + Evidence::Protocol, + Teardown::Terminated, + ), + ( + "partial", + "partial flush", + ErrorKind::Protocol, + Evidence::Protocol, + Teardown::Terminated, + ), ]; - for (plan, kind) in cases { - let mut acquired = fixture.acquire(&plan.replace(' ', "-")); + let restore = (Stage::Restore, Progress::Spawned); + for (name, plan, kind, evidence, teardown) in cases { + let mut acquired = fixture.acquire(name); let mut vmm = vmm_with(plan); let (restoring, launched) = launch(&fixture, &mut acquired, &image, &mut vmm); let mut control = FakeControl::default(); @@ -722,6 +894,17 @@ fn a_vmm_that_exits_or_sends_anything_but_its_event_ends_the_launch() { assert_eq!(error.kind(), kind, "{plan}: {error}"); assert!(launched.before.elapsed() < TIMEOUT, "{plan}"); assert!(control.calls.is_empty(), "{plan}"); + assert_failed(&error, &acquired, restore, evidence, teardown); + // The record keeps the line of OpenVMM's fatal error, and not the warning before it. + let fatal = plan + .split_whitespace() + .find_map(|word| word.strip_prefix("fatal:")); + if let Some(code) = fatal { + let line = + format!("OpenVMM reported: fatal error: [{code}] failed to launch vm worker"); + assert!(error.message().ends_with(&line), "{error}"); + assert!(error.record().message.ends_with(&line), "{error}"); + } assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -742,6 +925,199 @@ fn a_vmm_that_exits_or_sends_anything_but_its_event_ends_the_launch() { let source = std::error::Error::source(&error).unwrap().to_string(); assert!(source.contains("not the VMM"), "{source}"); assert!(control.calls.is_empty()); + assert_failed( + &error, + &acquired, + restore, + Evidence::Protocol, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); assert!(!acquired.locked.dir().join(CLONE_ACTIVATION_NAME).exists()); } + +#[test] +fn a_status_with_which_the_guest_ended_the_clone_s_vm_attributes_its_failure() { + let (fixture, image) = fixture(); + let ready = (Stage::Activation, Progress::Activating); + let starting = (Stage::Start, Progress::Starting); + // Each VMM ends as its clone awaits its Ready, or once the start of its sandbox was sent, with + // OpenVMM's outcome report, which the failure's record reads once the VMM has exited. + let cases = [ + // The guest's activation could not assemble the clone's root, which only the request's + // image and upper make, before any workload ran. + ( + "root", + "guest-exit:54", + ready, + Evidence::UnusableRoot, + Some(54), + ), + // A repair's status is evidence of the repair stage at any point. + ( + "repair", + "guest-exit:195", + ready, + Evidence::RepairStatus, + Some(195), + ), + ( + "late-repair", + "guest-exit:195", + starting, + Evidence::RepairStatus, + Some(195), + ), + // Once the start was sent, a workload may have ended its VM with any status, so the loss + // of the session is all that the host knows. + ( + "late-root", + "guest-exit:54", + starting, + Evidence::Reset, + Some(54), + ), + // A failure of the VMM tells nothing of the guest. + ("vmm", "vmm-failure:1", ready, Evidence::LostStatus, None), + ]; + for (name, report, at, evidence, status) in cases { + let mut acquired = fixture.acquire(name); + let mut vmm = vmm_with(&format!("flush report:{report}")); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let error = if at == starting { + let mut control = FakeControl::default(); + control.starts.push_back(Answer::Fail(Failure::LOST)); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + activated.start(&mut control).unwrap_err() + } else { + let ending = &vmm; + let mut control = FakeControl::activating([Answer::Fail(Failure::LOST)]); + control.on_activate = Some(Box::new(move |_| ending.end_and_wait())); + restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err() + }; + assert_failed(&error, &acquired, at, evidence, Teardown::Exited); + assert_eq!(error.record().status, status, "{name}"); + assert_launch_ended(&fixture, &mut acquired, &vmm); + } +} + +thread_local! { + /// The path of the record of a failure that [`ended_once_recorded`] reads, and the record that + /// it read there. + static RECORDED: RefCell)>> = const { RefCell::new(None) }; +} + +/// Ends a launch as [`Starting::abandon`] does, once it read the record of the failure at the path +/// in [`RECORDED`], as the teardown begins, which a crash of the caller during the teardown would +/// leave. +fn ended_once_recorded(starting: Starting<'_>) -> Result { + RECORDED.with_borrow_mut(|recorded| { + let (path, record) = recorded.as_mut().expect("the test names the record's path"); + *record = read_failure(path).unwrap(); + }); + starting.abandon() +} + +#[test] +fn a_failure_is_recorded_before_its_teardown_and_again_once_the_teardown_ended() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("sandbox"); + RECORDED.set(Some((acquired.locked.clone_failure_path(), None))); + let mut vmm = vmm_with("other flush"); + let (restoring, _) = launch_ending( + &fixture, + &mut acquired, + &image, + &mut vmm, + ended_once_recorded, + ); + let mut control = FakeControl::default(); + let error = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err(); + // As the teardown began, the sandbox recorded the failure, with its teardown pending. + let (_, pending) = RECORDED.take().unwrap(); + let pending = pending.expect("the failure was recorded before its teardown"); + let restore = (Stage::Restore, Progress::Spawned); + assert_eq!( + (pending.stage, pending.progress, pending.evidence), + (restore.0, restore.1, Evidence::Protocol) + ); + assert_eq!(pending.teardown, Teardown::Pending); + assert_eq!(pending.message, error.record().message); + // Once the teardown ended, the record holds its outcome. + assert_failed( + &error, + &acquired, + restore, + Evidence::Protocol, + Teardown::Terminated, + ); + assert_launch_ended(&fixture, &mut acquired, &vmm); +} + +/// Why [`kept`] keeps a launch. +const KEPT: &str = "the test keeps the launch"; + +/// Ends no launch of a clone that failed: keeps its journal and its VMM, as a launch whose VMM's +/// identity or exit is indeterminate keeps them. +fn kept(starting: Starting<'_>) -> Result { + // Nothing of the launch ends: its journal and its VMM outlive it, as a crash leaves them. + std::mem::forget(starting); + Ok(Recovery::Indeterminate(KEPT.to_owned())) +} + +#[test] +fn a_launch_that_cannot_be_ended_keeps_its_records_and_its_lease_until_a_reattach_ends_it() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("sandbox"); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch_ending(&fixture, &mut acquired, &image, &mut vmm, kept); + let mut control = FakeControl::activating([Answer::Fail(Failure::REFUSED)]); + let error = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err(); + let ready = (Stage::Activation, Progress::Activating); + let unresolved = Teardown::Unresolved(KEPT.to_owned()); + assert_failed(&error, &acquired, ready, Evidence::Protocol, unresolved); + let unended = format!("; the launch could not be ended, and remains for recovery: {KEPT}"); + assert!(error.message().ends_with(&unended), "{error}"); + // The journal remains, and with it the lease, which no release ends, and the clone's records. + let reference = vmm.reference.clone().unwrap(); + let locked = &mut acquired.locked; + let refusal = fixture.store.release(locked).unwrap_err(); + assert_eq!(refusal.code(), ErrorCode::SandboxRunning, "{refusal}"); + let records = CloneRecords::read(locked).unwrap().unwrap(); + assert_eq!(records.journal.unwrap().process(), Some(&reference)); + assert!(Process::open(&reference).unwrap().is_some()); + // A reattach ends the launch through its VMM's verified identity: the record keeps the + // failure's stage, evidence, and original outcome, with the teardown that resolved it. + let mut control = FakeControl::default(); + let reattached = reattach_clone( + locked, + &runtime_of(&fixture), + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ) + .unwrap(); + let resolved = failed(reattached); + assert_failed( + &resolved, + &acquired, + ready, + Evidence::Protocol, + Teardown::Terminated, + ); + assert_eq!( + (resolved.code(), resolved.message()), + (error.code(), error.record().message.as_str()) + ); + assert!(control.calls.is_empty()); + assert_launch_ended(&fixture, &mut acquired, &vmm); +} diff --git a/nvx-rs/src/host/template/mod.rs b/nvx-rs/src/host/template/mod.rs index 00af2578c..4e05c8b01 100644 --- a/nvx-rs/src/host/template/mod.rs +++ b/nvx-rs/src/host/template/mod.rs @@ -47,6 +47,17 @@ //! resends the same record in a new session once the adapter lost one. [`Activated::start`] //! then starts the clone's sandbox by a deadline that starts at its Ready, and //! [`Started::commit`] commits the launch. +//! - The plan, the launch, and the restore sequence fail with a [`CloneFailure`], whose +//! [`Evidence`] attributes it to an [`Outcome`], at a [`Stage`], once the clone got as far as +//! its [`Progress`], with the [`Teardown`] that followed: a failure from the launch on ends the +//! clone's VMM through its verified identity, and is recorded in the clone's sandbox, which +//! keeps its lease while a VM of the clone may run. Only a failure of the template quarantines +//! the leased generation, through its tombstone, [`CloneFailure::quarantine`]. +//! - [`reattach_clone`] takes a clone over once the caller of its launch restarted: it checks the +//! clone's activation in a new session, and ends a clone that another runtime identity +//! launched, or that fails its check, or classifies how it ended; [`CloneRecords::read`] reads +//! what the clone's sandbox keeps of it, for the operations that list, inspect, or remove +//! clones. //! //! # Canonical encoding //! @@ -79,9 +90,10 @@ mod test_support; mod wire; pub use self::clone::{ - Activated, CloneControl, ClonePlan, CloneRequest, CloneUpper, MAX_ACTIVATION_SESSIONS, - READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, RestoreDeadlines, - Restoring, Started, + Activated, CloneControl, CloneFailure, ClonePlan, CloneRecords, CloneRequest, CloneUpper, + Evidence, FAILURE_FORMAT, FailureRecord, MAX_ACTIVATION_SESSIONS, Outcome, Progress, + READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, ReattachedClone, + RestoreDeadlines, Restoring, Stage, Started, Teardown, reattach_clone, status_evidence, }; pub use self::eligibility::{Downtime, DowntimeSource, Eligibility, HostTime, MAX_DOWNTIME_NS}; pub use self::key::{CaptureClass, ClassIdentity, Invariants, KeyIdentity, TemplateKey}; diff --git a/nvx-rs/src/host/template/store.rs b/nvx-rs/src/host/template/store.rs index 762f45e8c..a491c9ba0 100644 --- a/nvx-rs/src/host/template/store.rs +++ b/nvx-rs/src/host/template/store.rs @@ -731,8 +731,9 @@ impl Store { } /// Checks that the directory of the generations is private to the current user, and not a - /// link. - fn check_generations_dir(&self) -> Result<()> { + /// link, as every read of a generation does, so that a clone's plan tells a store whose + /// directory fails it, the host's, from a generation that is malformed. + pub(super) fn check_generations_dir(&self) -> Result<()> { let path = self.dir.join(GENERATIONS_NAME); let what = "the generations of the template store"; let problem = platform::check_owner_only_dir(&path).map_err(store_error(format!( diff --git a/nvx-rs/src/host/template/store/leases.rs b/nvx-rs/src/host/template/store/leases.rs index e19d73967..3f73e90b5 100644 --- a/nvx-rs/src/host/template/store/leases.rs +++ b/nvx-rs/src/host/template/store/leases.rs @@ -369,8 +369,11 @@ impl Store { /// launch: the file of the scratch device of a clone whose upper is in RAM, /// [`CLONE_SCRATCH_NAME`](crate::host::store::CLONE_SCRATCH_NAME), which the clone's /// [plan](crate::host::template::ClonePlan) created, the clone's activation record, - /// [`CLONE_ACTIVATION_NAME`](crate::host::store::CLONE_ACTIVATION_NAME), and the socket of - /// its restore-ready event, [`RESTORE_READY_NAME`](crate::host::store::RESTORE_READY_NAME); + /// [`CLONE_ACTIVATION_NAME`](crate::host::store::CLONE_ACTIVATION_NAME), the socket of its + /// restore-ready event, [`RESTORE_READY_NAME`](crate::host::store::RESTORE_READY_NAME), and + /// its records, which last as long as the lease, its progress log, + /// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), and the record of its + /// failure, [`CLONE_FAILURE_NAME`](crate::host::store::CLONE_FAILURE_NAME); /// then, on Linux, the control socket of the clone's VMM, `.sock` in the leased /// generation's payload, after the launch that the lease reserved, whose absence it makes /// durable; then the clone binding, and once their removal is durable, the lease's record in diff --git a/nvx-rs/src/host/template/store/leases/tests.rs b/nvx-rs/src/host/template/store/leases/tests.rs index eeee09695..058f3eb5f 100644 --- a/nvx-rs/src/host/template/store/leases/tests.rs +++ b/nvx-rs/src/host/template/store/leases/tests.rs @@ -18,8 +18,8 @@ use crate::activation::TemplateReference; use crate::host::process::{Process, ProcessRef}; use crate::host::store::files::write_private; use crate::host::store::{ - CLONE_ACTIVATION_NAME, CLONE_SCRATCH_NAME, FORMAT, LOCK_NAME, LaunchState, RESTORE_READY_NAME, - RUNTIME_NAME, Recovery, RuntimeIdentity, + CLONE_ACTIVATION_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, CLONE_SCRATCH_NAME, FORMAT, + LOCK_NAME, LaunchState, RESTORE_READY_NAME, RUNTIME_NAME, Recovery, RuntimeIdentity, }; use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; use crate::host::template::eligibility::MAX_DOWNTIME_NS; @@ -819,6 +819,8 @@ fn a_release_makes_the_removal_of_its_binding_durable_before_it_removes_its_reco let kept = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ]; let synced = RefCell::new(Vec::new()); @@ -1242,6 +1244,8 @@ fn a_release_ends_the_lease_of_the_sandbox_whose_lock_it_holds_whatever_its_path let kept = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ]; for sandbox in [&leased_sandbox, &other] { From 7cd51f0f83059a4633dac7f67146c699ab748e94 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 25 Oct 2026 18:54:06 -0700 Subject: [PATCH 02/12] nvx-rs: let the caller fail a clone once it is ready A step of the caller's own between a clone's Ready and the commit of its launch, such as one that maps a directory of the host into the clone, may fail. Activated::fail and Started::fail fail the clone with the caller's evidence, at the start and the commit stages, through the recorded teardown that the restore sequence's own failures take: the failure is recorded before the teardown and once it ended. Abandoning a launch records nothing, as before. A status of the guest's that is evidence prevails over what the host or the caller saw of the same end, but no longer over evidence of the template, a check of the generation or a diagnosis that attributes a defect to it, which shows the provenance that the status alone leaves to establish. The record holds the status beside it, for a failure's record, the finish of a pending one, and the resolution of an unresolved one alike. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/README.md | 15 +- nvx-rs/src/host/template/clone/failure.rs | 51 ++++++- nvx-rs/src/host/template/clone/reattach.rs | 3 +- nvx-rs/src/host/template/clone/records.rs | 124 +++++++++++++--- nvx-rs/src/host/template/clone/restore.rs | 40 ++++- .../src/host/template/clone/tests/launch.rs | 31 ++++ .../src/host/template/clone/tests/restore.rs | 138 ++++++++++++++++++ 7 files changed, 369 insertions(+), 33 deletions(-) diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 4bf58e062..f4ed5ffdf 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -739,14 +739,16 @@ geometry, which the host's staging takes instead once the caller proves it, bound, resource exhaustion, a host that cannot restore the template, another runtime, or a failure of its own operations; unknown, for a lost status, a timeout, a reset, a forced kill, or a broken protocol; the template, for a -malformed generation, which alone quarantines the generation, through its +malformed generation, or a defect of its capture or repair that a diagnosis +attributes to it, the only outcome that quarantines the generation, through its tombstone, `CloneFailure::quarantine`; and the repair stage, for a status of the clone's repair alone. OpenVMM's outcome report gives the status with which the guest ended the clone's -VM, which `status_evidence` attributes, and which prevails over what the host -saw: a status of the repair at any point, and one of the activation before the -start was sent. +VM, which `status_evidence` attributes, and which prevails over what the host or +the caller saw, but not over evidence of the template, which shows the +provenance that a status alone leaves to establish: a status of the repair at +any point, and one of the activation before the start was sent. A VMM that exits before its restore-ready event, as OpenVMM does once it cannot restore the snapshot, after it closed the event's endpoint, which the sequence @@ -769,6 +771,11 @@ verified, or an unresolved one, which keeps the launch's journal or runtime record, and the lease; a release removes both, and a sandbox that boots cold writes neither. +A step or a check of the caller's own that fails once the clone is ready, or +once its sandbox started, fails the clone with the caller's evidence through +`Activated::fail` or `Started::fail`, which end the launch and record the +failure as the sequence's own failures do. + `reattach_clone` takes a clone over after the caller of its launch restarted: it reattaches by the identity of its VMM's process, checks a committed clone in a new session, in which the clone answers the byte-equal Activate of diff --git a/nvx-rs/src/host/template/clone/failure.rs b/nvx-rs/src/host/template/clone/failure.rs index 1a5bb193e..11c20de46 100644 --- a/nvx-rs/src/host/template/clone/failure.rs +++ b/nvx-rs/src/host/template/clone/failure.rs @@ -51,8 +51,12 @@ use crate::protocol::status; /// before the start of the clone's sandbox was sent, and none after; /// - no other status is evidence. /// -/// A status that is evidence prevails over what the host saw of the same end: the loss of the -/// session, the passing of a deadline, or the VMM's exit only show that the clone ended. +/// A status that is evidence prevails over what the host or the caller saw of the same end: the +/// loss of the session, the passing of a deadline, or the VMM's exit only show that the clone +/// ended, and a step that fails as the guest ends the clone may fail because it ended. It does not +/// prevail over evidence of the template, a check of the generation or a diagnosis that attributes +/// a defect to it, which shows the provenance that the status alone leaves to establish, and the +/// failure's record holds the status beside it. /// /// The host sees the failures of OpenVMM itself through the line with which OpenVMM reports the /// error that ended it, which the launch's log keeps, and which leads with the error's time ABI @@ -439,7 +443,9 @@ pub enum Teardown { /// Whether the clone's VMM still runs cannot be determined, for the reason that this gives: /// its identity or its exit is indeterminate. The launch's journal or the runtime record /// remains, and with it the clone's lease, which a release refuses to end until a reattach - /// resolves the teardown. + /// resolves the teardown. A record that the clone's sandbox keeps holds at most 1024 + /// characters of the reason, as it holds at most 8192 of the + /// [message](FailureRecord::message). Unresolved(String), /// The teardown had not ended when the failure was recorded: the record that a failure writes /// before its teardown, so that a crash of the caller during the teardown leaves it, which the @@ -525,7 +531,9 @@ pub struct FailureRecord { pub teardown: Teardown, /// The code of the failure's error. pub code: ErrorCode, - /// The message of the failure's error, followed by the messages of its sources. + /// The message of the failure's error, followed by the messages of its sources. A record that + /// the clone's sandbox keeps holds at most 8192 of its characters: of a longer message, as a + /// caller's diagnosis may be, its first and its last ones, with an ellipsis between them. pub message: String, } @@ -835,6 +843,17 @@ pub(super) fn reported( (evidence, status) } +/// Returns the evidence of a failure whose step saw `evidence`, once the guest's status, which +/// ended the clone's VM, is `reported`, as [`reported`] gives it, as [the attribution of +/// evidence](Evidence#attribution) describes: the status's evidence prevails, unless `evidence` +/// is of the template, whose provenance the status alone does not show. +pub(super) fn prevailing(evidence: Evidence, reported: Option) -> Evidence { + match reported { + Some(reported) if evidence.outcome() != Outcome::Template => reported, + _ => evidence, + } +} + /// The start of the line with which OpenVMM reports, on its standard error, which the launch's log /// keeps, the error that ended it: the line goes on with the error's first time ABI code in /// brackets, such as `[E_TSC_RATE_TOLERANCE]`, if the error's chain has one. @@ -1159,6 +1178,30 @@ mod tests { ); } + #[test] + fn a_status_prevails_over_what_was_seen_of_the_same_end_but_not_over_the_template_s_evidence() { + let template = [Evidence::MalformedGeneration, Evidence::TemplateDefect]; + for evidence in ALL { + // Without a status that is evidence, what the step saw stands. + assert_eq!(prevailing(evidence, None), evidence, "{evidence}"); + for reported in [ + Evidence::RepairStatus, + Evidence::UnusableRoot, + Evidence::Timeout, + ] { + let prevailed = prevailing(evidence, Some(reported)); + // A check of the generation, or a diagnosis that attributes a defect to it, shows + // the provenance that the status alone leaves to establish. + let expected = if template.contains(&evidence) { + evidence + } else { + reported + }; + assert_eq!(prevailed, expected, "{evidence} {reported}"); + } + } + } + #[test] fn openvmm_s_fatal_error_is_evidence_of_an_incompatible_host_only_for_its_codes() { for code in INCOMPATIBLE_HOST_CODES { diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index a2995a8b6..b288bbc0f 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -97,7 +97,8 @@ pub enum ReattachedClone { /// [`ReattachedClone::Ended`] otherwise. One that cannot be examined is /// [`ReattachedClone::Indeterminate`]. /// -/// The evidence of a status of the guest's prevails, as in the restore sequence. A failure is +/// The evidence of a status of the guest's prevails over any but the template's, as in the restore +/// sequence, and as [the attribution of evidence](Evidence#attribution) describes. A failure is /// recorded in the sandbox, as the restore sequence records it: before the reattach ends a VMM of /// the clone, recovers an interrupted launch, or clears the runtime record of a VM that exited, /// with its teardown [pending](Teardown::Pending), and again once the teardown ended. A clone diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs index b021c8b3d..f5b34d831 100644 --- a/nvx-rs/src/host/template/clone/records.rs +++ b/nvx-rs/src/host/template/clone/records.rs @@ -29,12 +29,15 @@ //! record outlasts a crash of the host, as the clone's binding does. Like the store's records, the //! record, and the directory that a write syncs, are named by their paths: the directories above //! the sandbox's must be ones that only the current user, or an administrator, may change, as the -//! [store](crate::host::store) requires of them. A record of another format -//! fails rather than being read. The failure of a VMM that -//! exited before its restore-ready event reads the end of the launch's log for the line with -//! which OpenVMM reported the error that ended it, which the record's message keeps, and whose -//! time ABI code may attribute the failure, as [the attribution of -//! evidence](Evidence#attribution) describes. +//! [store](crate::host::store) requires of them. The record keeps at most +//! 8192 characters of the failure's message, and 1024 of the reason of an unresolved teardown: of +//! a longer text, as the diagnosis of a caller that fails the clone may be, its first and its last +//! characters, with an ellipsis between them, so that a read, which takes at most 64 KiB of a +//! record, takes every record that a failure writes. A record of another format fails rather +//! than being read. The failure of a VMM that exited before its restore-ready event reads the end +//! of the launch's log for the line with which OpenVMM reported the error that ended it, which the +//! record's message keeps, and whose time ABI code may attribute the failure, as [the attribution +//! of evidence](Evidence#attribution) describes. //! //! A release of the clone's lease removes both, as it removes the clone's other files, so that //! the records of a clone last as long as its lease, which a failure whose teardown is unresolved @@ -79,10 +82,26 @@ const PROGRESS_FORMAT: u32 = 1; /// of its steps, take a few hundred. const MAX_PROGRESS_BYTES: u64 = 4096; -/// The most bytes of the record of a clone's failure that a read takes: the message of an error, -/// which it holds, takes at most a few kilobytes. +/// The most bytes of the record of a clone's failure that a read takes: more than any record that +/// a failure writes takes, whose texts [`bounded`] bounds, as [`MAX_MESSAGE_CHARS`] and +/// [`MAX_REASON_CHARS`] tell. const MAX_FAILURE_BYTES: u64 = 64 * 1024; +/// The most characters of the message of a failure's error that the record of the failure keeps, +/// as [`bounded`] keeps them: the diagnosis of a caller that fails a clone, as +/// [`Activated::fail`](super::Activated::fail) and [`Started::fail`](super::Started::fail) take +/// it, may be longer. +const MAX_MESSAGE_CHARS: usize = 8 * 1024; + +/// The most characters of the reason of an unresolved teardown that the record of a failure keeps, +/// as [`bounded`] keeps them. +const MAX_REASON_CHARS: usize = 1024; + +// JSON takes at most six bytes for a character, as for `\u0001`, and the other fields of a record +// take less than a kilobyte, so that a read takes every record that a failure writes. +const _: () = + assert!(6 * (MAX_MESSAGE_CHARS + MAX_REASON_CHARS) + 1024 <= MAX_FAILURE_BYTES as usize); + /// What messages call the progress log of a clone. const PROGRESS_WHAT: &str = "clone progress log"; @@ -351,8 +370,9 @@ impl Recorder { /// loss of a VMM that exited before its restore-ready event takes the line of OpenVMM's fatal /// error that the VMM's log ends with, as [`diagnosed`](Self::diagnosed) does, and the /// evidence of a status of the guest's, which OpenVMM's outcome report gives once the VMM has - /// exited, as [`observed`](Self::observed) reads it, prevails over the fault's, as - /// [`reported`](Self::reported) gives it. + /// exited, as [`observed`](Self::observed) reads it, and as [`reported`](Self::reported) + /// gives it, prevails over the fault's, unless the fault's is the template's, as + /// [`failure::prevailing`] decides. fn attribute( &self, progress: Progress, @@ -365,7 +385,8 @@ impl Recorder { _ => self.observed(), }; let (reported, status) = self.reported(report, progress); - (reported.unwrap_or(fault.evidence), status, fault.error) + let evidence = failure::prevailing(fault.evidence, reported); + (evidence, status, fault.error) } /// Returns `fault`, of the clone that got as far as `progress`, which is the loss of a VMM @@ -423,8 +444,8 @@ impl Recorder { /// error, for the loss of a VMM that exited before its restore-ready event, as /// [`diagnosed`](Self::diagnosed) adds it, and the guest's status in OpenVMM's outcome /// report, which the record does not hold, whose evidence, as [`reported`](Self::reported) - /// gives it, prevails, as in [`record`](Self::record). Writes the record in place of the - /// earlier one, if they differ, and returns the failure. + /// gives it, prevails as [`failure::prevailing`] decides, as in [`record`](Self::record). + /// Writes the record in place of the earlier one, if they differ, and returns the failure. pub(super) fn resolve(&self, mut earlier: FailureRecord, teardown: Teardown) -> CloneFailure { let error = earlier.error(); if earlier.teardown == teardown { @@ -434,7 +455,7 @@ impl Recorder { let mut evidence = fault.evidence; if earlier.status.is_none() && teardown != Teardown::NoVm { let (reported, status) = self.reported(self.observed(), earlier.progress); - evidence = reported.unwrap_or(evidence); + evidence = failure::prevailing(evidence, reported); earlier.status = status; } if evidence != earlier.evidence { @@ -446,9 +467,11 @@ impl Recorder { self.write(fault.error, earlier) } - /// Writes `record`, the record of the failure of `error`, in place of the sandbox's record, - /// and returns the failure, whose error says so if the record could not be written. + /// Writes `record`, the record of the failure of `error`, in place of the sandbox's record, and + /// returns the failure, with the record as the sandbox keeps it, as [`kept`] returns it, whose + /// error says so if the record could not be written. pub(super) fn write(&self, error: Error, record: FailureRecord) -> CloneFailure { + let record = kept(record); let error = match write_failure(&self.failure, &record) { Ok(()) => error, Err(failed) => amended( @@ -676,7 +699,8 @@ struct Recorded { } impl Recorded { - /// Returns the JSON form of `record`. + /// Returns the JSON form of `record`, as the sandbox keeps it, with texts that [`kept`] + /// bounds. fn new(record: &FailureRecord) -> Self { Self { format: FAILURE_FORMAT, @@ -695,11 +719,11 @@ impl Recorded { outcome: record.outcome, teardown: record.teardown.as_str().to_owned(), unresolved: match &record.teardown { - Teardown::Unresolved(reason) => Some(reason.clone()), + Teardown::Unresolved(reason) => Some(bounded(reason, MAX_REASON_CHARS)), _ => None, }, code: record.code.as_str().to_owned(), - message: record.message.clone(), + message: bounded(&record.message, MAX_MESSAGE_CHARS), } } @@ -760,9 +784,10 @@ impl Recorded { } } -/// Replaces the record of a clone's failure at `path` with `record`, atomically, and durably: once -/// the record is in place, it syncs the record's directory, as [`sync_dir`] does, and fails if the -/// sync does, though the record is in place then. +/// Replaces the record of a clone's failure at `path` with `record`, as the sandbox keeps it, as +/// [`kept`] returns it, atomically, and durably: once the record is in place, it syncs the +/// record's directory, as [`sync_dir`] does, and fails if the sync does, though the record is in +/// place then. pub(super) fn write_failure(path: &Path, record: &FailureRecord) -> Result<()> { write_failure_with(path, record, &sync_dir) } @@ -785,6 +810,35 @@ fn write_failure_with( } } +/// Returns `record` as the sandbox keeps it: with at most [`MAX_MESSAGE_CHARS`] characters of its +/// message, and [`MAX_REASON_CHARS`] of the reason of its teardown, if it is unresolved, as +/// [`bounded`] keeps them. +fn kept(mut record: FailureRecord) -> FailureRecord { + record.message = bounded(&record.message, MAX_MESSAGE_CHARS); + if let Teardown::Unresolved(reason) = &mut record.teardown { + *reason = bounded(reason, MAX_REASON_CHARS); + } + record +} + +/// Returns `text`, or, if it has more than `max` characters, `max` of them: its first ones and its +/// last ones, about as many of each, with an ellipsis between them in place of those that it +/// leaves out, so that the message of an error keeps what it begins with, and the messages of its +/// sources, which end it. +fn bounded(text: &str, max: usize) -> String { + let count = text.chars().count(); + if count <= max { + return text.to_owned(); + } + let first = (max - 1) / 2; + let last = max - 1 - first; + text.chars() + .take(first) + .chain(std::iter::once('…')) + .chain(text.chars().skip(count - last)) + .collect() +} + /// Reads the record of a clone's failure at `path`, or returns `None` if it does not exist. /// /// Fails with [`ErrorCode::FormatUnsupported`] for a record of another format than @@ -1205,6 +1259,32 @@ mod tests { assert_eq!(read_failure(&path).unwrap(), Some(terminated)); } + #[test] + fn a_failure_record_keeps_the_ends_of_texts_too_long_for_a_read_to_take() { + // A text keeps its first and its last characters, which it counts as characters. + assert_eq!(bounded("abcdef", 6), "abcdef"); + assert_eq!(bounded("abcdefg", 6), "ab…efg"); + assert_eq!(bounded("ééééééé", 4), "é…éé"); + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + // JSON takes six bytes for each of these control characters, the most that a character + // takes, so that each text, whole, would take more than a read takes. + let filler = "\u{1}".repeat(64 * 1024); + let mut written = failure(Teardown::Unresolved(format!("why{filler}so"))); + written.message = format!("start{filler}end"); + write_failure(&path, &written).unwrap(); + assert!(fs::metadata(&path).unwrap().len() <= MAX_FAILURE_BYTES); + let read = read_failure(&path).unwrap().unwrap(); + assert_eq!(read, kept(written)); + assert_eq!(read.message.chars().count(), MAX_MESSAGE_CHARS); + assert!(read.message.starts_with("start\u{1}") && read.message.ends_with("\u{1}end")); + let Teardown::Unresolved(reason) = &read.teardown else { + panic!("{:?}", read.teardown); + }; + assert_eq!(reason.chars().count(), MAX_REASON_CHARS); + assert!(reason.starts_with("why\u{1}") && reason.ends_with("\u{1}so")); + } + #[test] fn a_failure_record_of_another_format_or_a_malformed_one_is_refused() { let (_directory, private) = private_dir(); diff --git a/nvx-rs/src/host/template/clone/restore.rs b/nvx-rs/src/host/template/clone/restore.rs index d97858049..93dc92d5a 100644 --- a/nvx-rs/src/host/template/clone/restore.rs +++ b/nvx-rs/src/host/template/clone/restore.rs @@ -2,7 +2,8 @@ //! through the adapter's framing of the clone's control session, a [`CloneControl`]: //! [`Restoring::activate`] activates the clone, [`Activated::start`] starts its sandbox, and //! [`Started::commit`] commits its launch, as the documentation of [`Restoring::activate`] -//! describes. +//! describes, while [`Activated::fail`] and [`Started::fail`] fail the clone for a step of the +//! caller's own between them. use std::fmt; use std::io; @@ -416,6 +417,27 @@ impl<'s, S> Activated<'s, S> { drop(session); starting.abandon() } + + /// Fails the clone, once it is ready, for a step of the caller's own before the start of its + /// sandbox, such as one that maps a directory of the host into the clone, with `evidence`, + /// which attributes the failure as [the attribution of evidence](Evidence#attribution) + /// describes, and `error`, and returns the failure: once the adapter's session has ended, it + /// ends the launch as [`abandon`](Self::abandon) does, and records the failure at + /// [`Stage::Start`], before its teardown and once its teardown ended, as the restore sequence + /// records its own failures, so that a status of the guest's that is evidence, which + /// OpenVMM's outcome report gives once the VMM has exited, prevails over `evidence`, unless + /// `evidence` is of the template, such as a defect of the generation that the caller + /// diagnosed, whose record then holds the status beside it. + pub fn fail(self, evidence: Evidence, error: Error) -> CloneFailure { + let Self { + session, + starting, + tracker, + .. + } = self; + drop(session); + tracker.fail(starting, Stage::Start, Fault::new(evidence, error)) + } } impl fmt::Debug for Activated<'_, S> { @@ -432,7 +454,7 @@ impl fmt::Debug for Activated<'_, S> { /// A clone whose sandbox started, as [`Activated::start`] returns it, whose launch the caller /// [commits](Self::commit), once it checked anything else that it requires of a launch, or -/// [abandons](Self::abandon). +/// [abandons](Self::abandon), or [fails](Self::fail), if a check of its own failed. /// /// It holds the adapter's session, and the launch, as [`Activated`] does. Dropping it abandons the /// launch, as [`abandon`](Self::abandon) does, ignoring failures; a crash of the caller leaves the @@ -515,6 +537,20 @@ impl<'s, S> Started<'s, S> { drop(session); starting.abandon() } + + /// Fails the clone, once its sandbox started, for a check of the caller's own before the + /// commit of its launch, with `evidence` and `error`, as [`Activated::fail`] does, at + /// [`Stage::Commit`]. + pub fn fail(self, evidence: Evidence, error: Error) -> CloneFailure { + let Self { + session, + starting, + tracker, + .. + } = self; + drop(session); + tracker.fail(starting, Stage::Commit, Fault::new(evidence, error)) + } } impl fmt::Debug for Started<'_, S> { diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index f93e85bcf..04fc9d33f 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -2486,6 +2486,37 @@ fn a_reattach_finishes_an_earlier_failure_with_what_the_clone_s_vmm_left_as_it_e assert_eq!(failure.record().status, Some(195), "{earlier:?}"); assert_eq!(failure.message(), "the test's clone reset its session"); crashed.releases(&mut locked); + // One of the template, such as a defect of the generation that the caller diagnosed, + // which the status does not override: the record holds the status beside it. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + let mut locked = crashed.lock(); + let defect = Error::other("the test diagnosed a defect of the generation's repair"); + let at = (restore, transaction_of(&locked)); + let teardown = earlier.clone(); + record_earlier( + &crashed.lease, + &locked, + at, + (Evidence::TemplateDefect, teardown), + &defect, + ); + crashed.vmm.end_and_wait(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (lease, template) = (&crashed.lease, Evidence::TemplateDefect); + assert_recorded( + &failure, + lease, + &locked, + restore, + template, + Teardown::Exited, + ); + assert_eq!( + (failure.outcome(), failure.record().status), + (Outcome::Template, Some(195)), + "{earlier:?}" + ); + crashed.releases(&mut locked); // One whose VMM exited before its restore-ready event, with OpenVMM's error, which tells // that this host cannot present the template's CPU, once its record was written. let crashed = Crashed::at_with("launched", "close fatal:E_CPU_GENERATION"); diff --git a/nvx-rs/src/host/template/clone/tests/restore.rs b/nvx-rs/src/host/template/clone/tests/restore.rs index 99f8fe420..9e9c41dbc 100644 --- a/nvx-rs/src/host/template/clone/tests/restore.rs +++ b/nvx-rs/src/host/template/clone/tests/restore.rs @@ -1121,3 +1121,141 @@ fn a_launch_that_cannot_be_ended_keeps_its_records_and_its_lease_until_a_reattac assert!(control.calls.is_empty()); assert_launch_ended(&fixture, &mut acquired, &vmm); } + +#[test] +fn a_caller_fails_a_ready_or_started_clone_with_its_own_evidence_as_the_sequence_fails_one() { + let (fixture, image) = fixture(); + let ready = (Stage::Start, Progress::Ready); + // A step of the caller's own, such as one that maps a directory of the host into the clone, + // fails once the clone is ready, which the request's mappings attribute to the request. + let mut acquired = fixture.acquire("ready"); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + let mapping = "the test's mapping names no directory of the host"; + let failure = activated.fail(Evidence::InvalidMappings, Error::unsupported(mapping)); + assert_eq!(failure.kind(), ErrorKind::Unsupported, "{failure}"); + assert_eq!(failure.message(), mapping); + let mappings = Evidence::InvalidMappings; + assert_failed(&failure, &acquired, ready, mappings, Teardown::Terminated); + assert_eq!(failure.outcome(), Outcome::Request); + assert_launch_ended(&fixture, &mut acquired, &vmm); + // A check of the caller's own fails once the clone's sandbox started, before the commit. + let mut acquired = fixture.acquire("started"); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + let started = activated.start(&mut control).unwrap(); + let failure = started.fail(Evidence::HostFailure, Error::io("the test's check failed")); + let commit = (Stage::Commit, Progress::Started); + let host = Evidence::HostFailure; + assert_failed(&failure, &acquired, commit, host, Teardown::Terminated); + assert_eq!(failure.outcome(), Outcome::Host); + assert_launch_ended(&fixture, &mut acquired, &vmm); + // The guest ended the clone's VM with the status of a failed repair as the caller's step + // failed: the status prevails, as over the sequence's own failures. + let mut acquired = fixture.acquire("repair"); + let mut vmm = vmm_with("flush report:guest-exit:195"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + let failure = activated.fail(Evidence::InvalidMappings, Error::unsupported(mapping)); + let repair = Evidence::RepairStatus; + assert_failed(&failure, &acquired, ready, repair, Teardown::Exited); + assert_eq!(failure.record().status, Some(195)); + assert_launch_ended(&fixture, &mut acquired, &vmm); + // Evidence of the template, such as a defect of the generation that the caller diagnosed, + // shows the provenance that the status alone leaves to establish: it prevails, and the record + // holds the status beside it. The failure quarantines the generation once the sandbox's lock + // is released, as the lock order asks. + let mut acquired = fixture.acquire("defect"); + let mut vmm = vmm_with("flush report:guest-exit:195"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + let defect = "the test diagnosed a defect of the generation's repair"; + let failure = activated.fail(Evidence::TemplateDefect, Error::other(defect)); + let template = Evidence::TemplateDefect; + assert_failed(&failure, &acquired, ready, template, Teardown::Exited); + assert_eq!( + (failure.outcome(), failure.record().status), + (Outcome::Template, Some(195)) + ); + assert_launch_ended(&fixture, &mut acquired, &vmm); + drop(acquired); + assert!(failure.quarantine(&fixture.store, None).unwrap()); + assert_eq!( + fixture.store.tombstones().unwrap(), + [failure.record().template.generation] + ); +} + +#[test] +fn a_diagnosis_too_long_for_a_record_keeps_its_ends_and_its_evidence_through_a_reattach() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("diagnosis"); + let mut vmm = vmm_with("flush report:guest-exit:195"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + // The caller's diagnosis of a defect of the generation is longer than a record keeps: JSON + // takes six bytes for each of its control characters, the most that a character takes. + let (first, last) = ( + "the test's diagnosis begins", + "and the test's diagnosis ends", + ); + let defect = format!("{first}{}{last}", "\u{1}".repeat(64 * 1024)); + let failure = activated.fail(Evidence::TemplateDefect, Error::other(defect.clone())); + // The failure's error keeps the diagnosis whole, and its record, which the sandbox keeps, + // its ends. + assert_eq!(failure.message(), defect); + let record = failure.record(); + let ready = (Stage::Start, Progress::Ready); + assert_eq!( + ( + (record.stage, record.progress), + record.evidence, + &record.teardown + ), + (ready, Evidence::TemplateDefect, &Teardown::Exited) + ); + assert!(record.message.starts_with(first) && record.message.ends_with(last)); + assert!(record.message.chars().count() < defect.chars().count()); + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(record)); + // A reattach finds the failure as it was recorded, with the template's evidence, which + // quarantines the generation once the sandbox's lock is released. + let mut control = FakeControl::default(); + let reattached = reattach_clone( + &mut acquired.locked, + &runtime_of(&fixture), + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ) + .unwrap(); + let again = failed(reattached); + assert_eq!(again.record(), record); + assert_launch_ended(&fixture, &mut acquired, &vmm); + drop(acquired); + assert!(again.quarantine(&fixture.store, None).unwrap()); + assert_eq!( + fixture.store.tombstones().unwrap(), + [record.template.generation] + ); +} From 36b7399c348d0286d90ebdc445c3b36b87ed69d1 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Tue, 27 Oct 2026 21:05:42 -0700 Subject: [PATCH 03/12] nvx-rs: record the spawn of a clone's VMM that its journal lacks A launch whose VMM was spawned, but whose identity it could not record in its journal, recorded the clone's failure as if no VMM had been spawned: the launch then ends the VMM through its handle, and removes the journal, so nothing else tells it. The failure now records the spawn, and Progress::Spawned says that the journal records the VMM unless the launch could not record it. The documentation of reattach_clone says that its bound applies to a VMM whose launch committed: a launch that a crash interrupted ends as LockedStore::recover ends it, within the wait that the store's recovery allows the VMM's termination. Evidence::Protocol says, as the restore sequence now does, that the bytes that it refuses at the restore-ready endpoint are those that differ from the event. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/template/clone/failure.rs | 9 +++++---- nvx-rs/src/host/template/clone/launch.rs | 9 ++++++++- nvx-rs/src/host/template/clone/reattach.rs | 13 +++++++------ 3 files changed, 20 insertions(+), 11 deletions(-) diff --git a/nvx-rs/src/host/template/clone/failure.rs b/nvx-rs/src/host/template/clone/failure.rs index 11c20de46..ce78efe09 100644 --- a/nvx-rs/src/host/template/clone/failure.rs +++ b/nvx-rs/src/host/template/clone/failure.rs @@ -153,9 +153,9 @@ pub enum Evidence { /// The host's recovery terminated the clone's VMM, whose launch's caller had crashed. ForcedKill, /// The clone's session broke its protocol: another process at one of the clone's endpoints, - /// other bytes than OpenVMM's restore-ready event, a refusal of the adapter's framing, a Ready - /// that does not report the record's activation, or a session that broke the order of the - /// activation, as the guest's + /// bytes that differ from OpenVMM's restore-ready event, a refusal of the adapter's framing, a + /// Ready that does not report the record's activation, or a session that broke the order of + /// the activation, as the guest's /// [`ACTIVATION_PROTOCOL`](crate::protocol::status::ACTIVATION_PROTOCOL) and /// [`ACTIVATION_ORDER`](crate::protocol::status::ACTIVATION_ORDER) tell. Protocol, @@ -363,7 +363,8 @@ pub enum Progress { Planned, /// The launch wrote the clone's files, and may have begun its journal: no VMM was spawned. Launching, - /// The launch's VMM was spawned, and its journal records it. + /// The launch's VMM was spawned, which its journal records, unless the launch could not record + /// it, and then ended it through its handle. Spawned, /// The VMM's restore-ready event was read: the restored VM runs. Event, diff --git a/nvx-rs/src/host/template/clone/launch.rs b/nvx-rs/src/host/template/clone/launch.rs index 860fb2c1d..410c6faeb 100644 --- a/nvx-rs/src/host/template/clone/launch.rs +++ b/nvx-rs/src/host/template/clone/launch.rs @@ -345,7 +345,14 @@ impl ClonePlan { command.inherit = inherit; Ok((hooks.command)(command)) }, - |error, _| tracker.pending(Stage::Launch, host_evidence(error), error), + |error, spawned| { + // A VMM that the launch spawned, and could not record, got the clone that far, + // which only the record of its failure tells once the launch has ended it. + if spawned { + tracker.advance(Progress::Spawned); + } + tracker.pending(Stage::Launch, host_evidence(error), error); + }, ); // The VMM holds the claim, if it started. drop(claim); diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index b288bbc0f..913d88c23 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -61,15 +61,16 @@ pub enum ReattachedClone { /// Reattaches to the clone of the sandbox of `locked`, under the runtime identity `runtime`, /// which the caller would launch the clone's template with now, through the adapter's framing, -/// `control`, which must answer in a new session by `deadline`, and ends a VMM of the clone that -/// it may not keep within `bound`. +/// `control`, which must answer in a new session by `deadline`, and ends a VMM of the clone whose +/// launch committed, and that it may not keep, within `bound`. /// /// It reads the records of the clone that the sandbox keeps, as [`CloneRecords`] describes, then /// reattaches to the sandbox's VM, as [`supervisor::reattach`] does, which ends a launch that a -/// crash of its caller interrupted, through the verified identity of its VMM, removes a journal -/// that the runtime record of its launch supersedes, as the launch's commit may leave it, which is -/// no failure, and finds a VMM whose launch committed by the identity of its process, which says -/// which VM it is: +/// crash of its caller interrupted, through the verified identity of its VMM, as +/// [`LockedStore::recover`] ends it, within the wait that the store's recovery allows the VMM's +/// termination, which `bound` does not shorten, removes a journal that the runtime record of its +/// launch supersedes, as the launch's commit may leave it, which is no failure, and finds a VMM +/// whose launch committed by the identity of its process, which says which VM it is: /// /// - A VMM that runs under `runtime`, whose launch is the one that the clone's lease reserves, is /// the clone's, which the reattach checks in a new session, as the restore sequence of the From b5a706caf782db771a4ac8f814b04660fd7e27b7 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Tue, 27 Oct 2026 22:21:23 -0700 Subject: [PATCH 04/12] nvx-rs: fail a clone whose record of a failure cannot be read A reattach took a record of the clone's failure that it could not read for none, so a clone that had failed came back live if its VM still ran and answered, ended without a failure if its VM had exited, and indeterminate if its VM could not be examined. The record still tells that the clone launched, and failed: the reattach now fails such a clone at its own stage, with the error of the record's read, the host's evidence, once it ended a VMM of the clone that runs through its verified identity, or with the teardown unresolved for one that it cannot examine, which a later reattach resolves. The failure that it records replaces the record that cannot be read. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/template/clone/reattach.rs | 79 ++++++++++++---- .../src/host/template/clone/tests/launch.rs | 91 +++++++++++++++++++ 2 files changed, 153 insertions(+), 17 deletions(-) diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index 913d88c23..a1efe3e8a 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -48,8 +48,9 @@ pub enum ReattachedClone { /// The clone failed, as the record of its failure in its sandbox tells, which the reattach /// found, or wrote: the launch of a caller that crashed, which the reattach ended; a VM that /// exited once its launch committed, with a status of the guest's that is evidence, or - /// without a report of the guest's own end; a check of the clone that failed; or a VM that - /// runs under another runtime identity. The failure's teardown tells whether a VM of the + /// without a report of the guest's own end; a check of the clone that failed; a VM that + /// runs under another runtime identity; or a record of a failure of the clone that cannot be + /// read, which the record that the reattach wrote replaces. The failure's teardown tells whether a VM of the /// clone may still run: until it is resolved, the sandbox keeps the store's records of the VM, /// and with them the clone's lease, for a later reattach. Failed(CloneFailure), @@ -117,11 +118,15 @@ pub enum ReattachedClone { /// of the generation leaves the clone's reattach, as its run, as it is. It signals no process that /// it has not verified. The progress log and the record of a failure only inform it: one that /// cannot be read is none, so that the reattach still ends the clone's VM, and a failure that the -/// reattach records replaces it. A clone that launched, whose progress log, which its launch -/// created before it began, and appends to without a sync, cannot be read, is missing, or was -/// written during another boot of the host, whose crash may have lost its last lines, may have -/// run its workload: a status of the guest's is then attributed as once the start of the clone's -/// sandbox was sent. +/// reattach records replaces it. A record of a failure that cannot be read still tells that the +/// clone launched, and failed, so the reattach returns no VM of the clone, and finds the clone +/// neither ended nor indeterminate: it fails the clone at [`Stage::Reattach`] with the error of +/// the record's read, the host's evidence, once it ended a VMM of the clone that runs, as it ends +/// one that it may not keep, or with its teardown unresolved for one that cannot be examined. A +/// clone that launched, whose progress log, which its launch created before it began, and appends +/// to without a sync, cannot be read, is missing, or was written during another boot of the host, +/// whose crash may have lost its last lines, may have run its workload: a status of the guest's +/// is then attributed as once the start of the clone's sandbox was sent. /// /// Fails with [`ErrorKind::Other`](crate::ErrorKind::Other) if the sandbox holds no clone /// binding, whose VM reattaches as any sandbox's does, with @@ -186,7 +191,13 @@ pub(super) fn reattach_clone_with( let committed = runtime_record .as_ref() .filter(|record| record.launch() == launch); - let earlier = read_failure(&locked.clone_failure_path()).unwrap_or(None); + // A record of a failure that cannot be read is none below, but tells that the clone launched, + // and failed: the reattach fails the clone that it would keep, find ended, or find + // indeterminate, and the failure that it records replaces the record. + let (earlier, unreadable) = match read_failure(&locked.clone_failure_path()) { + Ok(earlier) => (earlier, None), + Err(error) => (None, Some(error)), + }; let logged = read_progress(&locked.clone_progress_path(), launch).unwrap_or(None); // The launch of a clone creates its progress log before it begins, and before it records a // failure, and appends to it without a sync, so the log of a clone that launched tells how @@ -195,8 +206,11 @@ pub(super) fn reattach_clone_with( // the log's last lines, and a workload may then have run in the clone. A log that can be read // tells that the clone launched as well as the store's records do, which a stop of its VM // clears, and a release alone removes it. - let launched = - journal.is_some() || committed.is_some() || earlier.is_some() || logged.is_some(); + let launched = journal.is_some() + || committed.is_some() + || earlier.is_some() + || unreadable.is_some() + || logged.is_some(); let certain = logged.is_some_and(|logged| logged.is_certain(BootId::current().ok())); let uncertain = launched && !certain; let logged = logged.map(|logged| logged.progress); @@ -270,9 +284,12 @@ pub(super) fn reattach_clone_with( }; match reattached { Reattached::Live(vmm) => { - let checked = activation.and_then(|activation| { - check_activation(locked, &lease, &vmm, control, activation, deadline, started) - }); + let checked = match unreadable { + Some(error) => Err(unreadable_failure(error)), + None => activation.and_then(|activation| { + check_activation(locked, &lease, &vmm, control, activation, deadline, started) + }), + }; match checked { Ok(session) => Ok(ReattachedClone::Live { vmm: Box::new(vmm), @@ -305,10 +322,20 @@ pub(super) fn reattach_clone_with( failed(fault, progress.stage(), Teardown::of(Ok(recovery))) } Reattached::Exited { record, .. } => { - match exit_fault(&recorder, progress, record.is_some()) { - Some((fault, _)) => failed(fault, progress.stage(), Teardown::Exited), + let fault = exit_fault(&recorder, progress, record.is_some()); + match (fault, unreadable) { + (Some((fault, _)), _) => failed(fault, progress.stage(), Teardown::Exited), + // A VM of the clone ran only once its launch spawned it, as in `exit_fault`. + (None, Some(error)) => { + let teardown = if progress >= Progress::Spawned { + Teardown::Exited + } else { + Teardown::NoVm + }; + failed(unreadable_failure(error), Stage::Reattach, teardown) + } // A report is of the clone's VM only once its launch began, as in `exit_fault`. - None => Ok(ReattachedClone::Ended { + (None, None) => Ok(ReattachedClone::Ended { progress, report: (progress >= Progress::Spawned) .then(|| recorder.observed().cloned()) @@ -318,7 +345,14 @@ pub(super) fn reattach_clone_with( } Reattached::Indeterminate(reason) => { if locked.journal()?.is_none() { - return Ok(ReattachedClone::Indeterminate(reason)); + return match unreadable { + Some(error) => failed( + unreadable_failure(error), + Stage::Reattach, + Teardown::Unresolved(reason), + ), + None => Ok(ReattachedClone::Indeterminate(reason)), + }; } // The launch that a crash of its caller interrupted cannot be ended yet. let fault = Fault::new(Evidence::LostStatus, interrupted(progress)); @@ -499,3 +533,14 @@ fn mismatched(differences: &[IdentityPart]) -> Error { ), ) } + +/// Returns the fault of a clone whose sandbox holds a record of a failure that cannot be read, as +/// `error` tells: the record still shows that the clone failed, so the reattach keeps none of it, +/// with the host's evidence. +fn unreadable_failure(error: Error) -> Fault { + Fault::host(amended( + error, + "; a clone whose sandbox records a failure failed for good, so the reattach keeps no VM \ + of it", + )) +} diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index 04fc9d33f..931e7bb43 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -19,6 +19,7 @@ use crate::activation::{ReadyActivation, StoreGeneration, TransactionId}; use crate::host::artifacts; use crate::host::platform::Transport; use crate::host::process::{BootId, Process, ProcessRef}; +use crate::host::store::files::write_private; use crate::host::store::{ CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, @@ -1324,6 +1325,34 @@ fn a_clone_whose_launch_failed_before_it_began_launches_no_more_until_its_releas assert!(!path.exists()); } +#[test] +fn a_failure_that_a_reattach_records_before_any_launch_shows_no_report_of_an_earlier_vm() { + // A reattach before any launch of the clone that cannot read the record of a failure, as a + // transient error of the host may keep it from reading one, records a failure of its own, at + // the plan, with no VM: the report that an earlier VM of the sandbox left, which no launch + // removed, stays none of the clone's. + let fixture = Fixture::new(); + let runtime = runtime_of(&fixture); + let mut acquired = fixture.acquire("sandbox"); + let locked = &mut acquired.locked; + fs::write(locked.outcome_path(), guest_exit(195)).unwrap(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let deadline = Instant::now() + TIMEOUT; + let reattached = reattach_clone(locked, &runtime, &mut control, deadline, TIMEOUT).unwrap(); + let failure = failed(reattached); + assert_eq!( + (failure.record().progress, &failure.record().teardown), + (Progress::Planned, &Teardown::NoVm) + ); + let records = CloneRecords::read(locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(failure.record())); + assert_eq!( + (records.progress, records.report), + (Progress::Planned, None) + ); +} + /// Returns the message of the refusal of a clone's launch whose socket at `path` would not fit a /// Unix socket's address. #[cfg(target_os = "linux")] @@ -2188,6 +2217,68 @@ fn a_reattach_ends_a_clone_that_it_may_not_keep_through_its_verified_identity() crashed.releases(&mut locked); } +#[test] +fn a_reattach_fails_a_clone_whose_record_of_a_failure_cannot_be_read() { + let reattach = (Stage::Reattach, Progress::Committed); + let host = Evidence::HostFailure; + // A record that is malformed, and one of a format that this build does not read: the clone + // failed, so the reattach sends its guest nothing, and ends its VM through its verified + // identity, though its Ready would match its activation record. + let unreadable: [(&[u8], ErrorCode); 2] = [ + (b"not a record", ErrorCode::Io), + (br#"{"format": 99}"#, ErrorCode::FormatUnsupported), + ]; + for (bytes, code) in unreadable { + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), bytes).unwrap(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), code, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + } + // A clone whose VM exited with a status that is no evidence of a failure failed all the same. + let crashed = Crashed::at_with("committed", "report:guest-exit:3"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded(&failure, lease, &locked, reattach, host, exited); + assert_eq!(failure.record().status, Some(3)); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // A clone whose VM cannot be examined failed all the same, with its teardown unresolved, which + // a later reattach resolves. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let reason = "the test cannot examine the VM"; + let reattached = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Ok(Reattached::Indeterminate(reason.to_owned())), + ) + .unwrap(); + let failure = failed(reattached); + let (lease, unresolved) = (&crashed.lease, Teardown::Unresolved(reason.to_owned())); + assert_recorded(&failure, lease, &locked, reattach, host, unresolved); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let terminated = Teardown::Terminated; + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + #[test] fn a_reattach_ends_the_launch_of_a_caller_that_crashed_and_records_its_failure() { let cases = [ From 418167db4df60fe9066fc84510dbec382b4d08d4 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Tue, 27 Oct 2026 23:17:32 -0700 Subject: [PATCH 05/12] nvx-rs: take a record of a clone's failure that cannot be read as one A reattach that found a record of the clone's failure that it could not read failed the clone with the error of the read only where it would otherwise have kept the clone, found it ended, or found its VM indeterminate. Elsewhere the error was lost: before it ended a launch that a crash of its caller interrupted, or cleared the runtime record of a VM that exited with a status of the guest's that is evidence, or without a report of the guest's own end, the reattach recorded a new failure in place of the record, at the stage of the clone's progress, as it did for a VM that runs under another runtime identity. The reattach now takes such a record as the record of an earlier failure that cannot tell how the clone failed. If the store holds the journal or the runtime record of the clone's launch, which the reattach may remove, it first replaces the record with its own failure at its own stage, with the error of the record's read and the host's evidence, its teardown pending, which keeps how far the clone got for a reattach after a crash: before the restore-ready event, only the journal holds the spawn. It then ends a VMM of the clone that still runs through its verified identity, and records the failure with the teardown that it found or made, over whose evidence a status of the guest's in OpenVMM's outcome report prevails. Part of #844. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/template/clone/reattach.rs | 123 +++++++-------- .../src/host/template/clone/tests/launch.rs | 142 ++++++++++++++++++ 2 files changed, 205 insertions(+), 60 deletions(-) diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index a1efe3e8a..ee4b22878 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -50,9 +50,9 @@ pub enum ReattachedClone { /// exited once its launch committed, with a status of the guest's that is evidence, or /// without a report of the guest's own end; a check of the clone that failed; a VM that /// runs under another runtime identity; or a record of a failure of the clone that cannot be - /// read, which the record that the reattach wrote replaces. The failure's teardown tells whether a VM of the - /// clone may still run: until it is resolved, the sandbox keeps the store's records of the VM, - /// and with them the clone's lease, for a later reattach. + /// read, which the record that the reattach wrote replaces. The failure's teardown tells + /// whether a VM of the clone may still run: until it is resolved, the sandbox keeps the + /// store's records of the VM, and with them the clone's lease, for a later reattach. Failed(CloneFailure), /// Whether the clone's VMM, whose launch committed, runs cannot be determined, for the reason /// that this gives, and the sandbox records no failure of the clone: the runtime record @@ -111,18 +111,19 @@ pub enum ReattachedClone { /// keeps the stage, the evidence, and the original outcome, unless what the clone's VMM left as /// it ended since tells more, as the guest's status, which the record does not hold, and the /// error that OpenVMM reported, for a VMM that exited before its restore-ready event, may, and -/// returns the failure, with the record's lease, transaction, stage, progress, and code. +/// returns the failure, with the record's lease, transaction, stage, progress, and code. A record +/// of a failure that cannot be read tells as much, though not how the clone failed: if the store +/// holds records of the clone's launch, which the reattach may remove, the reattach first replaces +/// it with the record of its own failure at [`Stage::Reattach`], with the error of the record's +/// read and the host's evidence, its teardown pending, which keeps how far the clone got for a +/// reattach after a crash of the caller. It then ends a VMM of the clone that still runs the same +/// way, and records the failure with the teardown that it found or made, as any failure that it +/// records, over whose evidence a status of the guest's prevails. /// /// The reattach reads no record of the template store: the lease holds the clone's generation, /// and the sandbox's activation record the clone's transaction, so a tombstone, or a retirement, /// of the generation leaves the clone's reattach, as its run, as it is. It signals no process that -/// it has not verified. The progress log and the record of a failure only inform it: one that -/// cannot be read is none, so that the reattach still ends the clone's VM, and a failure that the -/// reattach records replaces it. A record of a failure that cannot be read still tells that the -/// clone launched, and failed, so the reattach returns no VM of the clone, and finds the clone -/// neither ended nor indeterminate: it fails the clone at [`Stage::Reattach`] with the error of -/// the record's read, the host's evidence, once it ended a VMM of the clone that runs, as it ends -/// one that it may not keep, or with its teardown unresolved for one that cannot be examined. A +/// it has not verified. The progress log only informs it: one that cannot be read is none. A /// clone that launched, whose progress log, which its launch created before it began, and appends /// to without a sync, cannot be read, is missing, or was written during another boot of the host, /// whose crash may have lost its last lines, may have run its workload: a status of the guest's @@ -134,7 +135,8 @@ pub enum ReattachedClone { /// [`MAX_TIMEOUT`](crate::host::launch::MAX_TIMEOUT), and for an invalid `runtime`, before it /// writes anything, and as [`Lease::of`] and [`supervisor::reattach`] fail for records that are /// inconsistent, which leave the clone's VM as it was, and the record of the failure of an -/// interrupted launch, if the reattach wrote it, pending. +/// interrupted launch, or the one that replaced a record that could not be read, if the reattach +/// wrote it, pending. /// /// [`CloneRecords`]: super::CloneRecords pub fn reattach_clone( @@ -191,13 +193,11 @@ pub(super) fn reattach_clone_with( let committed = runtime_record .as_ref() .filter(|record| record.launch() == launch); - // A record of a failure that cannot be read is none below, but tells that the clone launched, - // and failed: the reattach fails the clone that it would keep, find ended, or find - // indeterminate, and the failure that it records replaces the record. - let (earlier, unreadable) = match read_failure(&locked.clone_failure_path()) { - Ok(earlier) => (earlier, None), - Err(error) => (None, Some(error)), - }; + // The record of an earlier failure of the clone, or the fault of a record that cannot be read, + // which still tells that the clone launched, and failed for good, though not how. + let earlier = read_failure(&locked.clone_failure_path()) + .map_err(unreadable_failure) + .transpose(); let logged = read_progress(&locked.clone_progress_path(), launch).unwrap_or(None); // The launch of a clone creates its progress log before it begins, and before it records a // failure, and appends to it without a sync, so the log of a clone that launched tells how @@ -206,11 +206,8 @@ pub(super) fn reattach_clone_with( // the log's last lines, and a workload may then have run in the clone. A log that can be read // tells that the clone launched as well as the store's records do, which a stop of its VM // clears, and a release alone removes it. - let launched = journal.is_some() - || committed.is_some() - || earlier.is_some() - || unreadable.is_some() - || logged.is_some(); + let launched = + journal.is_some() || committed.is_some() || earlier.is_some() || logged.is_some(); let certain = logged.is_some_and(|logged| logged.is_certain(BootId::current().ok())); let uncertain = launched && !certain; let logged = logged.map(|logged| logged.progress); @@ -224,6 +221,22 @@ pub(super) fn reattach_clone_with( .and_then(|(_, record)| record.restore().map(|restore| restore.transaction)); let recorder = Recorder::new(lease.clone(), transaction, locked).uncertain(uncertain); let fresh = earlier.is_none(); + // A record that cannot be read is replaced by the reattach's own failure, with its teardown + // pending, before the reattach ends or forgets anything of the clone's launch: the record keeps + // how far the clone got, which the launch's journal and runtime record hold, and the reattach + // removes, for a reattach after a crash of this caller, which finishes it. Without them, the + // reattach removes nothing that the record would keep, and a later reattach finds the clone as + // this one does. + if let Some(Err(fault)) = &earlier + && (journal.is_some() || committed.is_some()) + { + recorder.pending( + Stage::Reattach, + progress, + (fault.evidence, None), + &fault.error, + ); + } // A launch that the journal names, which no runtime record supersedes, and whose failure the // sandbox does not record, was interrupted by a crash of its caller: its failure is recorded // before the recovery ends it, so that a crash of this caller during the recovery leaves the @@ -247,6 +260,12 @@ pub(super) fn reattach_clone_with( } }; let reattached = reattach(locked, runtime, &exiting)?; + let failed = |fault: Fault, stage: Stage, teardown: Teardown| { + let fault = Fault::new(fault.evidence, unended(fault.error, &teardown)); + Ok(ReattachedClone::Failed( + recorder.record(stage, progress, fault, teardown), + )) + }; if let Some(earlier) = earlier { let teardown = match reattached { Reattached::Live(vmm) | Reattached::Mismatched { vmm, .. } => { @@ -260,12 +279,22 @@ pub(super) fn reattach_clone_with( record: Some(_), .. } => Teardown::Exited, // Nothing of the clone remains: a teardown that was not resolved is now. - Reattached::Exited { record: None, .. } => match earlier.teardown { - Teardown::Unresolved(_) | Teardown::Pending => Teardown::Exited, - ref teardown => teardown.clone(), + Reattached::Exited { record: None, .. } => match &earlier { + Ok(earlier) => match earlier.teardown { + Teardown::Unresolved(_) | Teardown::Pending => Teardown::Exited, + ref teardown => teardown.clone(), + }, + // A VM of the clone ran only once its launch spawned it, as in `exit_fault`. + Err(_) if progress >= Progress::Spawned => Teardown::Exited, + Err(_) => Teardown::NoVm, }, Reattached::Indeterminate(reason) => Teardown::Unresolved(reason), }; + let earlier = match earlier { + Ok(earlier) => earlier, + // The record cannot tell how the clone failed: the failure is the reattach's own. + Err(fault) => return failed(fault, Stage::Reattach, teardown), + }; let failure = if earlier.teardown == Teardown::Pending { // The crash of the caller during the teardown left the failure pending, which is // recorded as the failure would have recorded it once its teardown ended. @@ -276,20 +305,11 @@ pub(super) fn reattach_clone_with( }; return Ok(ReattachedClone::Failed(failure)); } - let failed = |fault: Fault, stage: Stage, teardown: Teardown| { - let fault = Fault::new(fault.evidence, unended(fault.error, &teardown)); - Ok(ReattachedClone::Failed( - recorder.record(stage, progress, fault, teardown), - )) - }; match reattached { Reattached::Live(vmm) => { - let checked = match unreadable { - Some(error) => Err(unreadable_failure(error)), - None => activation.and_then(|activation| { - check_activation(locked, &lease, &vmm, control, activation, deadline, started) - }), - }; + let checked = activation.and_then(|activation| { + check_activation(locked, &lease, &vmm, control, activation, deadline, started) + }); match checked { Ok(session) => Ok(ReattachedClone::Live { vmm: Box::new(vmm), @@ -322,20 +342,10 @@ pub(super) fn reattach_clone_with( failed(fault, progress.stage(), Teardown::of(Ok(recovery))) } Reattached::Exited { record, .. } => { - let fault = exit_fault(&recorder, progress, record.is_some()); - match (fault, unreadable) { - (Some((fault, _)), _) => failed(fault, progress.stage(), Teardown::Exited), - // A VM of the clone ran only once its launch spawned it, as in `exit_fault`. - (None, Some(error)) => { - let teardown = if progress >= Progress::Spawned { - Teardown::Exited - } else { - Teardown::NoVm - }; - failed(unreadable_failure(error), Stage::Reattach, teardown) - } + match exit_fault(&recorder, progress, record.is_some()) { + Some((fault, _)) => failed(fault, progress.stage(), Teardown::Exited), // A report is of the clone's VM only once its launch began, as in `exit_fault`. - (None, None) => Ok(ReattachedClone::Ended { + None => Ok(ReattachedClone::Ended { progress, report: (progress >= Progress::Spawned) .then(|| recorder.observed().cloned()) @@ -345,14 +355,7 @@ pub(super) fn reattach_clone_with( } Reattached::Indeterminate(reason) => { if locked.journal()?.is_none() { - return match unreadable { - Some(error) => failed( - unreadable_failure(error), - Stage::Reattach, - Teardown::Unresolved(reason), - ), - None => Ok(ReattachedClone::Indeterminate(reason)), - }; + return Ok(ReattachedClone::Indeterminate(reason)); } // The launch that a crash of its caller interrupted cannot be ended yet. let fault = Fault::new(Evidence::LostStatus, interrupted(progress)); diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index 931e7bb43..ab62b3531 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -2279,6 +2279,148 @@ fn a_reattach_fails_a_clone_whose_record_of_a_failure_cannot_be_read() { crashed.releases(&mut locked); } +#[test] +fn a_record_of_a_failure_that_cannot_be_read_is_a_failure_whatever_the_reattach_finds() { + let reattach = (Stage::Reattach, Progress::Committed); + // A VM that exited once its launch committed, with a status of the guest's that is evidence, + // or without a report of the guest's own end: the clone failed before, so the failure is the + // reattach's, with the error of the record's read, over whose evidence the status's prevails. + let cases = [ + ("report:guest-exit:195", Evidence::RepairStatus, Some(195)), + ("", Evidence::HostFailure, None), + ]; + for (words, evidence, status) in cases { + let crashed = Crashed::at_with("committed", words); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{words}: {failure}"); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded(&failure, lease, &locked, reattach, evidence, exited); + assert_eq!(failure.record().status, status, "{words}"); + assert!(locked.runtime().unwrap().is_none(), "{words}"); + assert!(control.capabilities.is_empty(), "{words}"); + crashed.releases(&mut locked); + } + // A VM that runs under another runtime identity is ended, and the failure is the same. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &other, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + let host = Evidence::HostFailure; + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // A launch that a crash of its caller interrupted: the reattach replaces the record with its + // own failure, pending, before its recovery ends the launch, which a crash of the restarted + // caller leaves, for the next reattach to finish. + let spawned = (Stage::Reattach, Progress::Spawned); + let assert_pending = |locked: &LockedStore| { + let pending = read_failure(&locked.clone_failure_path()) + .unwrap() + .expect("the reattach replaced the record that it could not read"); + assert_eq!( + ( + pending.stage, + pending.progress, + pending.evidence, + pending.code + ), + (spawned.0, spawned.1, host, ErrorCode::Io) + ); + assert_eq!(pending.teardown, Teardown::Pending); + }; + let crashed = Crashed::at("launched"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Err(Error::other("the test's caller crashes")), + ); + assert!(crash.is_err()); + assert_pending(&locked); + assert!(locked.journal().unwrap().is_some()); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + assert_recorded(&failure, lease, &locked, spawned, host, terminated); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // Before the restore-ready event, only the launch's journal, which the recovery removes, holds + // the spawn: once it went, the pending record still tells how far the clone got, so the next + // reattach takes the status with which the guest ended the VM. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + supervisor::reattach_reporting(locked, runtime, exited)?; + Err(Error::other("the test's caller crashes")) + }, + ); + assert!(crash.is_err()); + assert_pending(&locked); + assert!(locked.journal().unwrap().is_none()); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + let repair = Evidence::RepairStatus; + assert_recorded(&failure, lease, &locked, spawned, repair, exited); + assert_eq!(failure.record().status, Some(195)); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // A clone whose launch did not begin: the store holds no record of the launch that the + // reattach would remove, so the record stays as it is, and a reattach after a crash of the + // caller finds the clone as this one does, with no VM, and takes no report of an earlier VM + // of the sandbox for the clone's. + let crashed = Crashed::at("prepared"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let earlier = serde_json::json!({ + "schema_version": 1, + "outcome": {"operation": "managed", "category": "guest-exit", "status_code": 195}, + "teardown": {"workers": true}, + }); + fs::write(locked.outcome_path(), serde_json::to_vec(&earlier).unwrap()).unwrap(); + let mut control = ReattachControl::answering(Some); + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Err(Error::other("the test's caller crashes")), + ); + assert!(crash.is_err()); + let kept = fs::read(locked.clone_failure_path()).unwrap(); + assert_eq!(kept, b"not a record"); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let launching = (Stage::Reattach, Progress::Launching); + let (lease, no_vm) = (&crashed.lease, Teardown::NoVm); + assert_recorded(&failure, lease, &locked, launching, host, no_vm); + assert_eq!(failure.record().status, None); + crashed.releases(&mut locked); +} + #[test] fn a_reattach_ends_the_launch_of_a_caller_that_crashed_and_records_its_failure() { let cases = [ From 504155ad2840e9b32ba8483ed7815d1b9552b018 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Wed, 28 Oct 2026 08:20:55 -0700 Subject: [PATCH 06/12] nvx-rs: record the spawn of a clone's VMM in its progress log as its launch returns A clone's launch noted the spawn of its VMM without writing it in the progress log, which recorded it with the next step, since the launch's journal records the spawn too. A launch that its caller abandoned, or dropped, before the clone's activation removed the journal as it ended, and nothing recorded the spawn any more: a later reattach then took the clone for one whose VMM never spawned, read no outcome report of its VM, and found it ended, even when the guest had ended the VM with a status that is evidence of a failure, such as one of its repair. The launch now appends the spawn to the progress log before it returns. A launch that cannot write it fails at the launch stage, which ends its VMM and records the failure, as the restore sequence does when it cannot record a step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/template/clone/failure.rs | 10 +-- nvx-rs/src/host/template/clone/launch.rs | 15 ++-- nvx-rs/src/host/template/clone/records.rs | 13 ++-- .../src/host/template/clone/tests/launch.rs | 69 ++++++++++++++++++- 4 files changed, 90 insertions(+), 17 deletions(-) diff --git a/nvx-rs/src/host/template/clone/failure.rs b/nvx-rs/src/host/template/clone/failure.rs index ce78efe09..02d6cf15e 100644 --- a/nvx-rs/src/host/template/clone/failure.rs +++ b/nvx-rs/src/host/template/clone/failure.rs @@ -350,11 +350,11 @@ impl fmt::Display for Stage { /// How far a clone got, in the order of its steps: the latest step that it reached. /// -/// The steps from the restore-ready event to the start's confirmation are those of the clone's -/// restore sequence, which its progress log, -/// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), records as each is reached; -/// the store's records tell the others: the launch's journal records the spawn of its VMM, and the -/// runtime record its commit. +/// The steps from the spawn of the clone's VMM to the start's confirmation are those that its +/// launch and restore sequence record in its progress log, +/// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), as each is reached; the +/// store's records tell the spawn too, and the commit: the launch's journal records the spawn of +/// its VMM until the launch ends, and the runtime record its commit. #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] #[serde(rename_all = "kebab-case")] #[non_exhaustive] diff --git a/nvx-rs/src/host/template/clone/launch.rs b/nvx-rs/src/host/template/clone/launch.rs index 410c6faeb..d7d367d42 100644 --- a/nvx-rs/src/host/template/clone/launch.rs +++ b/nvx-rs/src/host/template/clone/launch.rs @@ -113,8 +113,10 @@ impl ClonePlan { /// directory, in place of one that an earlier launch left; on Windows, a named pipe whose /// name the launch draws at random. /// 4. Runs [`supervisor::launch`] as the launch that the lease reserves, which begins it with - /// the store's launch journal, starts the VMM, and records its identity in the journal, - /// which holds the spawn until the progress log records it with the restore-ready event. + /// the store's launch journal, starts the VMM, and records its identity in the journal. + /// 5. Appends the spawn to the progress log before it returns, since a teardown of the launch + /// removes the journal, which records the spawn too. A launch that cannot append it fails + /// at the launch stage, and ends the VMM. /// /// The VMM restores the generation's snapshot in place, in the generation's payload, which it /// only reads, and which the launch neither copies nor changes, so any number of clones @@ -367,9 +369,12 @@ impl ClonePlan { return Err(tracker.record(Stage::Launch, Fault::host(failure.error), teardown)); } }; - // The launch's journal records the spawn, which the progress log records with the next - // step. - tracker.advance(Progress::Spawned); + // The progress log records the spawn before the launch returns: the journal records it too, + // but a teardown of the launch removes the journal, and a reattach reads the outcome report + // of the clone's VM only once its VMM spawned. + if let Err(fault) = tracker.reach(Progress::Spawned) { + return Err(tracker.fail(starting, Stage::Launch, fault)); + } Ok(Restoring { starting, launched, diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs index f5b34d831..cbb71beb1 100644 --- a/nvx-rs/src/host/template/clone/records.rs +++ b/nvx-rs/src/host/template/clone/records.rs @@ -14,10 +14,11 @@ //! caller alone loses no write of. A crash of the host, which ends the clone's VM, may lose the //! log's last lines, so how far the clone got is uncertain, once it launched, by a log that was //! written during another boot, as by one that is damaged or missing: a workload may have run in -//! the clone. A step that another record holds, the spawn, which the launch's journal records, -//! and one that the next step follows at once, the acknowledgment, which the Activate follows, is -//! appended with that next step, in its one write, so that the log takes a write for each step -//! from the restore-ready event on. A reader ignores a last line that a crash cut short. A sandbox +//! the clone. The launch appends the spawn before it returns, although the launch's journal +//! records it too, since a teardown of the launch removes the journal. A step that the next step +//! follows at once, the acknowledgment, which the Activate follows, is appended with that next +//! step, in its one write, so that the log takes a write for the spawn and for each step from the +//! restore-ready event on. A reader ignores a last line that a crash cut short. A sandbox //! that boots cold writes no log. //! //! # The failure record @@ -181,8 +182,8 @@ impl ProgressLog { /// Notes that the clone reached `step`, unless it reached it, or a later step, already, /// without writing it: the log records it with the next step that it writes. It is for a - /// step that another record holds until then, as the launch's journal holds the spawn of its - /// VMM, or that the next step follows at once. + /// step that another record holds, as the record of a launch's failure holds the spawn of a + /// VMM that the launch could not record, or that the next step follows at once. pub(super) fn advance(&mut self, step: Progress) { self.progress = self.progress.max(step); } diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index ab62b3531..8aa02c1e4 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -29,7 +29,7 @@ use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; use crate::host::template::Retirement; use crate::host::template::clone::launch::{Hooks, Point}; use crate::host::template::clone::reattach::reattach_clone_with; -use crate::host::template::clone::records::{read_failure, write_failure}; +use crate::host::template::clone::records::{read_failure, read_progress, write_failure}; use crate::host::template::store::SNAPSHOT_ARTIFACTS; use crate::host::template::test_support::artifact; @@ -2520,6 +2520,73 @@ fn a_reattach_records_the_failure_of_an_interrupted_launch_before_it_ends_the_la crashed.releases(&mut locked); } +/// A launch that its caller abandons before the clone's activation leaves a progress log that +/// records the spawn of the clone's VMM, which the launch's journal, which the teardown removes, +/// records no more, so that a reattach reads the outcome report of the clone's VM, and records the +/// failure that the guest's status gives. +#[test] +fn a_launch_abandoned_before_its_activation_leaves_the_spawn_for_a_reattach() { + let fixture = Fixture::new(); + complete_snapshot(&fixture); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("abandoned"); + let progress = acquired.locked.clone_progress_path(); + let launch = acquired.lease.launch().clone(); + let plan = fixture + .plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT) + .unwrap(); + let runtime = plan.key().runtime().clone(); + let mut vmm = VmmDir::new(); + // The guest's repair ends the clone's VM with its status, which the VMM reports as it exits. + vmm.plan("report:guest-exit:195"); + let dir = vmm.path().to_owned(); + let made = RefCell::new(None); + let command = fake_vmm(&dir, &made); + let hooks = Hooks { + command: &command, + ..Hooks::LAUNCH + }; + let restoring = plan + .launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks) + .unwrap(); + vmm.reference = Some(restoring.starting().process().reference().clone()); + // The log records the spawn once the launch returns. + let logged = read_progress(&progress, &launch).unwrap().unwrap(); + assert_eq!( + (logged.progress, logged.boot), + (Progress::Spawned, Some(BootId::current().unwrap())) + ); + let capability = hex(restoring.capability()); + assert_eq!(vmm.started(), format!("{capability} true")); + vmm.end_and_wait(); + assert_eq!(restoring.abandon().unwrap(), Recovery::Exited); + assert!(acquired.locked.journal().unwrap().is_none()); + let mut control = ReattachControl::answering(Some); + let reattached = reattach_clone( + &mut acquired.locked, + &runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ); + let failure = failed(reattached.unwrap()); + assert_recorded( + &failure, + &acquired.lease, + &acquired.locked, + (Stage::Restore, Progress::Spawned), + Evidence::RepairStatus, + Teardown::Exited, + ); + assert_eq!(failure.record().status, Some(195)); + assert!(control.capabilities.is_empty()); + assert_eq!( + fixture.store.release(&mut acquired.locked).unwrap(), + Some(acquired.lease.clone()) + ); + assert_released(acquired.lease.sandbox()); +} + #[test] fn a_clone_whose_vm_exited_once_it_committed_is_classified_by_its_report() { let run = (Stage::Run, Progress::Committed); From a3981c9b2045a89d4a1b6b5a92ff462dd40582f1 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Wed, 28 Oct 2026 10:05:33 -0700 Subject: [PATCH 07/12] nvx-rs: take a record of another clone's failure as one that cannot be read A reattach accepted any failure record that it could read in the sandbox as the failure of the clone that the sandbox's binding leases, whatever the template store, lease, generation, key, and launch that the record names. A record of another clone's failure, misplaced in the sandbox, was then returned as this clone's, and a failure that it attributed to the template could quarantine a generation that this clone does not lease. The reattach, and the read of a clone's records, now read the failure record only as the failure of the binding's clone: a record that names another store, lease, generation, key, or launch fails to read, as a malformed record does, naming what differs. The reattach takes it as a record that it cannot read: it replaces it with its own failure at the reattach stage, with the host's evidence, so the failure that it returns, and records, is this clone's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/template/clone/reattach.rs | 39 +++-- nvx-rs/src/host/template/clone/records.rs | 150 +++++++++++++++++- .../src/host/template/clone/tests/launch.rs | 41 +++++ 3 files changed, 210 insertions(+), 20 deletions(-) diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index ee4b22878..8da6cf6a9 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -9,7 +9,7 @@ use std::time::{Duration, Instant}; use super::failure::{ Attribute, CloneFailure, Evidence, Fault, Progress, Stage, Teardown, amended, describe, }; -use super::records::{Recorder, progress_of, read_failure, read_progress}; +use super::records::{Recorder, progress_of, read_failure_of, read_progress}; use super::restore::{ActivationSession, CloneControl}; use crate::activation::{ActivationRecord, MAX_RECORD_BYTES, ReadyActivation}; use crate::error::{Error, ErrorCode, Result}; @@ -112,13 +112,17 @@ pub enum ReattachedClone { /// it ended since tells more, as the guest's status, which the record does not hold, and the /// error that OpenVMM reported, for a VMM that exited before its restore-ready event, may, and /// returns the failure, with the record's lease, transaction, stage, progress, and code. A record -/// of a failure that cannot be read tells as much, though not how the clone failed: if the store -/// holds records of the clone's launch, which the reattach may remove, the reattach first replaces -/// it with the record of its own failure at [`Stage::Reattach`], with the error of the record's -/// read and the host's evidence, its teardown pending, which keeps how far the clone got for a +/// of a failure that cannot be read tells as much, though not how the clone failed, and so does +/// a record of another clone's failure, whose template store, lease, leased generation, its key, +/// or launch is not the binding's, and which tells nothing of this clone: if the store holds +/// records of the clone's launch, which the reattach may remove, the reattach first replaces it +/// with the record of its own failure at [`Stage::Reattach`], with the error of the record's read +/// and the host's evidence, its teardown pending, which keeps how far the clone got for a /// reattach after a crash of the caller. It then ends a VMM of the clone that still runs the same /// way, and records the failure with the teardown that it found or made, as any failure that it -/// records, over whose evidence a status of the guest's prevails. +/// records, over whose evidence a status of the guest's prevails. The failure that it returns is +/// then this clone's, so a [quarantine](CloneFailure::quarantine) of it never reaches a +/// generation that the clone does not lease. /// /// The reattach reads no record of the template store: the lease holds the clone's generation, /// and the sandbox's activation record the clone's transaction, so a tombstone, or a retirement, @@ -194,8 +198,9 @@ pub(super) fn reattach_clone_with( .as_ref() .filter(|record| record.launch() == launch); // The record of an earlier failure of the clone, or the fault of a record that cannot be read, - // which still tells that the clone launched, and failed for good, though not how. - let earlier = read_failure(&locked.clone_failure_path()) + // or of one of another clone's failure, which still tells that the clone launched, and failed + // for good, though not how. + let earlier = read_failure_of(&locked.clone_failure_path(), &lease) .map_err(unreadable_failure) .transpose(); let logged = read_progress(&locked.clone_progress_path(), launch).unwrap_or(None); @@ -221,12 +226,12 @@ pub(super) fn reattach_clone_with( .and_then(|(_, record)| record.restore().map(|restore| restore.transaction)); let recorder = Recorder::new(lease.clone(), transaction, locked).uncertain(uncertain); let fresh = earlier.is_none(); - // A record that cannot be read is replaced by the reattach's own failure, with its teardown - // pending, before the reattach ends or forgets anything of the clone's launch: the record keeps - // how far the clone got, which the launch's journal and runtime record hold, and the reattach - // removes, for a reattach after a crash of this caller, which finishes it. Without them, the - // reattach removes nothing that the record would keep, and a later reattach finds the clone as - // this one does. + // A record that cannot be read, or that is another clone's, is replaced by the reattach's own + // failure, with its teardown pending, before the reattach ends or forgets anything of the + // clone's launch: the record keeps how far the clone got, which the launch's journal and + // runtime record hold, and the reattach removes, for a reattach after a crash of this caller, + // which finishes it. Without them, the reattach removes nothing that the record would keep, and + // a later reattach finds the clone as this one does. if let Some(Err(fault)) = &earlier && (journal.is_some() || committed.is_some()) { @@ -537,9 +542,9 @@ fn mismatched(differences: &[IdentityPart]) -> Error { ) } -/// Returns the fault of a clone whose sandbox holds a record of a failure that cannot be read, as -/// `error` tells: the record still shows that the clone failed, so the reattach keeps none of it, -/// with the host's evidence. +/// Returns the fault of a clone whose sandbox holds a record of a failure that cannot be read, or +/// that is another clone's, as `error` tells: the record still shows that the clone failed, so +/// the reattach keeps none of it, with the host's evidence. fn unreadable_failure(error: Error) -> Fault { Fault::host(amended( error, diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs index cbb71beb1..699acc0f9 100644 --- a/nvx-rs/src/host/template/clone/records.rs +++ b/nvx-rs/src/host/template/clone/records.rs @@ -874,6 +874,51 @@ pub(super) fn read_failure(path: &Path) -> Result> { .map_err(|problem| Error::io(format!("{}: {problem}", malformed()))) } +/// Reads the record of the failure of the clone of `lease` at `path`, as [`read_failure`] reads +/// it, or returns `None` if it does not exist. +/// +/// Fails as [`read_failure`] does, and, as for a record that is malformed, with +/// [`ErrorKind::Io`](crate::ErrorKind::Io) for a record of another clone's failure, whose template +/// store, lease, leased generation, its key, or launch is not the lease's: it tells nothing of +/// this clone, and a [quarantine](CloneFailure::quarantine) of its failure would tombstone the +/// record's generation, which may not be the one that this clone leases. +pub(super) fn read_failure_of(path: &Path, lease: &Lease) -> Result> { + let Some(record) = read_failure(path)? else { + return Ok(None); + }; + let leased = lease.template(); + let differences: Vec<&str> = [ + (record.store != lease.store(), "the template store"), + (record.lease != lease.id(), "the lease"), + ( + record.template.generation != leased.generation, + "the leased generation", + ), + ( + record.template.key != leased.key, + "the key of the leased generation", + ), + (record.launch != *lease.launch(), "the launch"), + ] + .into_iter() + .filter_map(|(differs, what)| differs.then_some(what)) + .collect(); + let Some((last, others)) = differences.split_last() else { + return Ok(Some(record)); + }; + let differences = if others.is_empty() { + (*last).to_owned() + } else { + format!("{} and {last}", others.join(", ")) + }; + Err(Error::io(format!( + "the {FAILURE_WHAT} {} belongs to another clone than the one of the lease {}, which the \ + sandbox's clone binding records: they differ in {differences}", + path.display(), + lease.id(), + ))) +} + /// Reads the private regular file at `path`, the clone's `description`, of at most `limit` bytes, /// or returns `None` if it does not exist, or was removed as this opened it, as [`was_removed`] /// tells. @@ -962,8 +1007,8 @@ impl CloneRecords { /// the latest transition that replaced it left it, and the records may come from transitions /// that ran between the reads. A caller that holds the sandbox's lock reads them at one point /// of its lifecycle. Fails as the reads of the clone binding, [`Lease::of`], of the store's - /// records, of the progress log, and of the failure record fail; an outcome report that is - /// missing or malformed is none. + /// records, of the progress log, and of the failure record fail, a failure record of another + /// clone than the binding's included; an outcome report that is missing or malformed is none. /// /// It reads the outcome report only once the clone got past its plan, as the progress log of /// its launch, the launch's journal, the runtime record of its VM, or the record of a failure @@ -978,7 +1023,7 @@ impl CloneRecords { let runtime = sandbox.runtime()?; let logged = read_progress(&sandbox.clone_progress_path(), lease.launch())? .map(|logged| logged.progress); - let failure = read_failure(&sandbox.clone_failure_path())?; + let failure = read_failure_of(&sandbox.clone_failure_path(), &lease)?; // A failure records how far the clone got, which the store's records of its launch no // longer tell once the failure ended the launch. let progress = progress_of(&lease, journal.as_ref(), runtime.as_ref(), logged).max( @@ -1317,6 +1362,105 @@ mod tests { assert_eq!(read_failure(&path).unwrap_err().kind(), ErrorKind::Io); } + #[test] + fn a_failure_record_of_another_clone_than_the_one_of_the_lease_is_refused() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + let written = failure(Teardown::Exited); + let lease = Lease::new( + written.lease, + written.store, + written.template.generation, + written.template.key, + private.clone(), + written.launch.clone(), + ); + assert_eq!(read_failure_of(&path, &lease).unwrap(), None); + write_failure(&path, &written).unwrap(); + assert_eq!( + read_failure_of(&path, &lease).unwrap(), + Some(written.clone()) + ); + // A record of the failure of a clone of another store, of another lease, of another + // generation, or of another key, or of another launch, tells nothing of the lease's clone, + // though `read_failure` reads it. + let generation = StoreGeneration::new(NonZeroU64::new(8).unwrap()); + let launch = LaunchId::try_from("6".repeat(32)).unwrap(); + let others = [ + ( + FailureRecord { + store: StoreId::from_bytes([6; 16]), + ..written.clone() + }, + "the template store", + ), + ( + FailureRecord { + lease: LeaseId::from_bytes([6; 16]), + ..written.clone() + }, + "the lease", + ), + ( + FailureRecord { + template: TemplateReference { + generation, + ..written.template + }, + ..written.clone() + }, + "the leased generation", + ), + ( + FailureRecord { + template: TemplateReference { + key: Identity::from_bytes([6; 32]), + ..written.template + }, + ..written.clone() + }, + "the key of the leased generation", + ), + ( + FailureRecord { + launch: launch.clone(), + ..written.clone() + }, + "the launch", + ), + ( + FailureRecord { + lease: LeaseId::from_bytes([6; 16]), + template: TemplateReference { + generation, + ..written.template + }, + launch, + ..written.clone() + }, + "the lease, the leased generation and the launch", + ), + ]; + for (other, differences) in others { + write_failure(&path, &other).unwrap(); + assert_eq!(read_failure(&path).unwrap(), Some(other)); + let error = read_failure_of(&path, &lease).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + assert!( + error + .to_string() + .ends_with(&format!("they differ in {differences}")), + "{error}" + ); + } + // A record that cannot be read fails as it does for `read_failure`. + write_private(&path, b"{").unwrap(); + assert_eq!( + read_failure_of(&path, &lease).unwrap_err().kind(), + ErrorKind::Io + ); + } + #[test] fn the_line_of_openvmm_s_fatal_error_is_read_from_the_end_of_its_log() { let (_directory, private) = private_dir(); diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index 8aa02c1e4..83f4e2a36 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -2279,6 +2279,47 @@ fn a_reattach_fails_a_clone_whose_record_of_a_failure_cannot_be_read() { crashed.releases(&mut locked); } +#[test] +fn a_reattach_fails_a_clone_whose_sandbox_records_the_failure_of_another_clone() { + // A record of the failure of another lease's clone, of another launch, which leased another + // generation of the same store, whose evidence is the template's: it tells nothing of this + // clone, so the reattach takes it as a record that it cannot read, and the failure that it + // records in its place, and returns, is this clone's, which quarantines nothing. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let lease = &crashed.lease; + let generation = lease.generation().get().checked_add(1).unwrap(); + let other = Lease::new( + crate::host::template::LeaseId::from_bytes([0xa5; 16]), + lease.store(), + StoreGeneration::new(generation), + lease.template().key, + lease.sandbox().to_owned(), + crate::host::store::LaunchId::generate().unwrap(), + ); + let defect = Error::io("the test's generation is malformed"); + let at = ((Stage::Restore, Progress::Spawned), None); + let template = (Evidence::MalformedGeneration, Teardown::Exited); + record_earlier(&other, &locked, at, template, &defect); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + assert!( + failure.message().contains("belongs to another clone"), + "{failure}" + ); + let reattach = (Stage::Reattach, Progress::Committed); + let (host, terminated) = (Evidence::HostFailure, Teardown::Terminated); + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + drop(locked); + let store = &crashed.fixture.store; + assert!(!failure.quarantine(store, None).unwrap()); + assert!(store.tombstones().unwrap().is_empty()); +} + #[test] fn a_record_of_a_failure_that_cannot_be_read_is_a_failure_whatever_the_reattach_finds() { let reattach = (Stage::Reattach, Progress::Committed); From 145ca08724ca7ab54677987e0f7a946e7d321e26 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Wed, 28 Oct 2026 11:24:58 -0700 Subject: [PATCH 08/12] nvx-rs: give a clone's restore-ready socket owner-only access On Linux, a clone's launch listens for the restore-ready event of its VMM at a Unix socket in the sandbox's directory, which the bind creates with the mode that the process's umask leaves. A restrictive umask, such as 0200, could withhold even its owner's permission to write it, and a VMM that is not root cannot connect to a socket without that permission: the launch would then fail as if the VMM had sent nothing. The launch now gives the socket mode 0600 right after the bind, before it spawns the VMM. It opens the socket's path only to locate it, following no link, refuses anything but a socket, and sets the mode through the descriptor's link in /proc, as the store sets the modes of the directories that it creates. The review of #949, which added the restore-ready socket, found this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/platform/linux.rs | 58 ++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/nvx-rs/src/host/platform/linux.rs b/nvx-rs/src/host/platform/linux.rs index 06d11afe2..90f487332 100644 --- a/nvx-rs/src/host/platform/linux.rs +++ b/nvx-rs/src/host/platform/linux.rs @@ -868,7 +868,8 @@ pub(crate) struct ReadyListener { /// Listens at the Unix socket `socket_path`, in place of a socket that an earlier launch left /// there, as the restore-ready endpoint of a clone's launch, and returns the endpoint, as -/// OpenVMM's `--restore-ready-path` takes it, and its listener. +/// OpenVMM's `--restore-ready-path` takes it, and its listener. The socket has mode 0600, +/// whatever the umask, as [`set_ready_socket_mode`] gives it. pub(crate) fn listen_ready(socket_path: &Path) -> io::Result<(String, ReadyListener)> { let endpoint = socket_path .to_str() @@ -879,6 +880,10 @@ pub(crate) fn listen_ready(socket_path: &Path) -> io::Result<(String, ReadyListe _ => {} } let socket = UnixListener::bind(socket_path)?; + // The umask may clear bits of the mode that the bind gives the socket, even its owner's + // permission to write it, without which a VMM that is not root cannot connect: the mode is + // set again before any VMM may connect. + set_ready_socket_mode(socket_path)?; // The host polls the socket before it accepts, so an accept never blocks. socket.set_nonblocking(true)?; let listener = ReadyListener { @@ -965,6 +970,35 @@ impl ReadyListener { } } +/// Gives the restore-ready socket at `socket_path`, which the caller bound, the mode 0600, whatever +/// the umask: its owner, the user of the clone's VMM, may connect to it, which takes the +/// permission to write it, and no one else may. +/// +/// The socket is opened only to locate it, following no link at its name, and refused unless it +/// is a socket. +fn set_ready_socket_mode(socket_path: &Path) -> io::Result<()> { + use std::os::unix::fs::FileTypeExt; + + use nix::fcntl::OFlag; + + let socket = OpenOptions::new() + .read(true) + .custom_flags((OFlag::O_PATH | OFlag::O_NOFOLLOW).bits()) + .open(socket_path)?; + if !socket.metadata()?.file_type().is_socket() { + return Err(io::Error::other(format!( + "{} is not the restore-ready socket that the launch bound", + socket_path.display() + ))); + } + // A descriptor that only locates the socket cannot change its mode itself, but its link in + // /proc names exactly the socket that it opened. + fs::set_permissions( + Path::new("/proc/self/fd").join(socket.as_raw_fd().to_string()), + Permissions::from_mode(0o600), + ) +} + /// Connects to a control endpoint served by the process `expected_pid`. /// /// The connection attempt never blocks: a listener whose backlog is full yields @@ -1111,6 +1145,28 @@ mod tests { assert!(process.wait(Instant::now()).unwrap().unwrap()); } + #[test] + fn a_restore_ready_socket_lets_its_owner_connect_whatever_the_umask() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("ready.sock"); + // A umask that clears even the owner's permission to write what it creates, in a thread + // of the test's own, whose umask no other thread shares. + let (endpoint, _listener) = thread::scope(|scope| { + scope + .spawn(|| { + nix::sched::unshare(nix::sched::CloneFlags::CLONE_FS).unwrap(); + nix::sys::stat::umask(nix::sys::stat::Mode::from_bits_truncate(0o277)); + listen_ready(&path) + }) + .join() + .unwrap() + }) + .unwrap(); + let mode = fs::symlink_metadata(&path).unwrap().permissions().mode(); + assert_eq!(mode & PERMISSION_BITS, 0o600, "{mode:o}"); + UnixStream::connect(&endpoint).unwrap(); + } + #[test] fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() { let pid = std::process::id(); From 096b886d0009d5af66ab7a0264bbb8a443d21244 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Wed, 28 Oct 2026 11:24:58 -0700 Subject: [PATCH 09/12] nvx-rs: check a leased payload as the release of its lease opens it On Linux, an acquisition refused a generation whose payload was not a directory of the current user with mode 0700, but checked the payload by its path, which follows links. The release of the lease opens the payload through the template store's directory instead, following no link at the store's directory, at the directory of the generations, or at the payload. A link put in one of those places once the store was opened thus let the acquisition grant a lease that its release could not end until the link was gone. The acquisition now opens the payload as the release does, through the store's directory, whose identity it reads there, and checks the payload's owner and mode through the payload's handle. Before it writes anything, it refuses a payload that it cannot open that way, one that is gone or whose store's directory holds another store, and one that is not such a directory. The review of #949, which made the release open the payload through a handle, found this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/src/host/platform/linux.rs | 11 +++++ nvx-rs/src/host/template/store/leases.rs | 40 ++++++++++------ .../src/host/template/store/leases/tests.rs | 46 +++++++++++++++---- 3 files changed, 72 insertions(+), 25 deletions(-) diff --git a/nvx-rs/src/host/platform/linux.rs b/nvx-rs/src/host/platform/linux.rs index 90f487332..5c6027769 100644 --- a/nvx-rs/src/host/platform/linux.rs +++ b/nvx-rs/src/host/platform/linux.rs @@ -784,6 +784,17 @@ impl Directory { pub(crate) fn sync(&self) -> io::Result<()> { self.0.sync_all() } + + /// Returns why the directory cannot hold a sandbox store, as [`check_store_dir`] returns it for + /// a path, if it cannot: unless it belongs to the current user with mode 0700. + pub(crate) fn store_dir_problem(&self) -> io::Result> { + let metadata = self.0.metadata()?; + Ok(store_dir_problem( + metadata.uid(), + metadata.mode(), + sys::effective_ids().0, + )) + } } /// Claims a launch's fresh log, which `file` holds open. diff --git a/nvx-rs/src/host/template/store/leases.rs b/nvx-rs/src/host/template/store/leases.rs index 3f73e90b5..820779440 100644 --- a/nvx-rs/src/host/template/store/leases.rs +++ b/nvx-rs/src/host/template/store/leases.rs @@ -165,8 +165,10 @@ impl Store { /// - with [`ErrorCode::TemplateIneligible`] if it is not eligible at `now`; /// - on Linux, with [`ErrorKind::Io`] if the generation's payload is not a directory of the /// current user with mode 0700, which a launch of the clone requires, as OpenVMM does of the - /// directory of a control socket, and which a release needs, to remove the clone's socket - /// from the payload and make its absence durable; + /// directory of a control socket, or if a release could not open it as it opens it, + /// following no link at the store's directory, at the directory of the generations, or at + /// the payload: a release opens the payload to remove the clone's socket from it and make + /// its absence durable; /// - with [`ErrorCode::SandboxNotProvisioned`] if the directory does not exist, or holds no /// configuration record of the caller, as `files` declares it; /// - with [`ErrorCode::SandboxRunning`] if the sandbox records a VM, or a launch, which may @@ -229,11 +231,11 @@ impl Store { )); } // A release removes the socket of the clone's VMM through a handle of the payload, which - // it opens for reading, so that it can make the removal durable: a payload that it could - // not open, or remove an entry from, would keep the lease and its binding past it. - if cfg!(unix) { - self.check_payload(generation)?; - } + // it opens for reading, following no link from the store's directory on, so that it can + // make the removal durable: a payload that it could not open, or remove an entry from, + // would keep the lease and its binding past it. + #[cfg(target_os = "linux")] + self.check_payload(generation)?; let not_provisioned = || { Error::new( ErrorCode::SandboxNotProvisioned, @@ -489,16 +491,24 @@ impl Store { Ok(Some(lease)) } - /// Refuses the payload of the generation `generation` unless it is a directory of the current - /// user with mode 0700, as a launch of a clone requires it, OpenVMM requiring that mode of the - /// directory of a control socket, and as a release of the clone's lease needs it on Linux: the - /// release opens the payload for reading, and removes the clone's socket from it. + /// Refuses the payload of the generation `generation` unless a release of a clone's lease can + /// open it, as [`open_payload`] opens it, following no link at the store's directory, at the + /// directory of the generations, or at the payload, through a directory that holds this + /// store, and it is a directory of the current user with mode 0700. A launch of a clone + /// requires that mode, OpenVMM requiring it of the directory of a control socket, and so does + /// the release: it opens the payload for reading, and removes the clone's socket from it. + #[cfg(target_os = "linux")] fn check_payload(&self, generation: StoreGeneration) -> Result<()> { let payload = self.payload_dir(generation); - let problem = platform::check_store_dir(&payload).map_err(store_error(format!( - "cannot open the payload of the generation {}", - payload.display() - )))?; + let problem = match open_payload(self.dir(), self.id, generation)? { + Some(directory) => directory.store_dir_problem().map_err(store_error(format!( + "cannot examine the payload of the generation {}", + payload.display() + )))?, + None => { + Some("is gone, or the store's directory holds another store, or none".to_owned()) + } + }; match problem { Some(problem) => Err(Error::io(format!( "the payload of the generation {} {problem}", diff --git a/nvx-rs/src/host/template/store/leases/tests.rs b/nvx-rs/src/host/template/store/leases/tests.rs index 058f3eb5f..25066d514 100644 --- a/nvx-rs/src/host/template/store/leases/tests.rs +++ b/nvx-rs/src/host/template/store/leases/tests.rs @@ -285,26 +285,52 @@ fn acquisitions_refuse_what_they_cannot_lease_and_lease_nothing() { assert_eq!(error.code(), ErrorCode::TemplateIneligible, "{error}"); assert!(error.message().contains(condition), "{error}"); } - // On Unix, a payload that is not a directory of the current user with mode 0700, which a + // On Linux, a payload that is not a directory of the current user with mode 0700, which a // launch of the clone requires, and through which a release ends the lease: it opens the - // payload for reading, and removes the clone's socket from it. - #[cfg(unix)] + // payload for reading, and removes the clone's socket from it. A user other than root cannot + // open a payload of mode 0300 for reading at all. + #[cfg(target_os = "linux")] { use std::os::unix::fs::PermissionsExt; let payload = store.payload_dir(generation); + let unopened = format!( + "cannot open the payload of the generation {}", + payload.display() + ); for mode in [0o300, 0o500] { fs::set_permissions(&payload, fs::Permissions::from_mode(mode)).unwrap(); let error = refusal(&store, generation, &sandbox, &now); fs::set_permissions(&payload, fs::Permissions::from_mode(0o700)).unwrap(); assert_eq!(error.kind(), ErrorKind::Io, "{error}"); - assert_eq!( - error.message(), - format!( - "the payload of the generation {} has mode {mode:o}, not the mode 700 that \ - the directory of a control socket must have", - payload.display() - ) + let refused = format!( + "the payload of the generation {} has mode {mode:o}, not the mode 700 that the \ + directory of a control socket must have", + payload.display() + ); + assert!( + error.message() == refused || (mode == 0o300 && error.message() == unopened), + "{error}" + ); + } + // A link in the place of the store's directory, of the directory of the generations, or of + // the payload, even to a directory that holds the same files: a release, which follows + // none of them, could not open the payload. + let root = store.dir().to_owned(); + let generations = payload.parent().unwrap().to_owned(); + let elsewhere = directory.path().join("elsewhere"); + for linked in [&root, &generations, &payload] { + fs::rename(linked, &elsewhere).unwrap(); + std::os::unix::fs::symlink(&elsewhere, linked).unwrap(); + let error = refusal(&store, generation, &sandbox, &now); + fs::remove_file(linked).unwrap(); + fs::rename(&elsewhere, linked).unwrap(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + assert_eq!(error.message(), unopened, "{error}"); + let source = std::error::Error::source(&error).unwrap().to_string(); + assert!( + source.ends_with("is a link, or not a directory"), + "{source}" ); } } From 21a37c11bfce253c91cc84597d2405add059e2a7 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Wed, 28 Oct 2026 14:56:15 -0700 Subject: [PATCH 10/12] nvx-rs: let a holder's stop keep the record of a VMM that had exited A holder that checks whether its VM's VMM has exited, then stops the VM, attributes to the stop a VMM that exits in between: Vmm::stop finds it exited, runs no step, reports StopOutcome::Graceful, and clears the runtime record of its launch. A clone's reattach then finds no record, and ends the clone as one that its holder stopped, rather than with the loss of its status. Vmm::stop_unless_exited stops the VM as Vmm::stop does, unless its own check, before the graceful step, finds the VMM exited. It then returns None, runs no step, signals nothing, and keeps the record, so that reattach_clone finds that the VM exited unheld, records the loss of the clone's status if no report of the guest's own end tells why, and only then clears the record. Once that check found the VMM running, its exit is the stop's, as before. Vmm::stop, Vmm::terminate, and their callers are unchanged, and so are the rules of quarantine and of leases: the loss of a status quarantines nothing, and a lease is released only once its VM's exit is verified. Tests cover both stops through stand-in VMMs, and a clone whose VMM exits once its holder found it running, before the holder's stop begins: the clone fails at its run with the loss of its status. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/README.md | 4 + nvx-rs/src/host/supervisor/mod.rs | 121 ++++++++++++++---- nvx-rs/src/host/supervisor/tests.rs | 106 ++++++++++++++- nvx-rs/src/host/template/clone/reattach.rs | 14 +- .../src/host/template/clone/tests/launch.rs | 71 +++++++++- 5 files changed, 278 insertions(+), 38 deletions(-) diff --git a/nvx-rs/README.md b/nvx-rs/README.md index f4ed5ffdf..63cace754 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -536,6 +536,10 @@ explicit handle list. event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. +`Vmm::stop_unless_exited` stops the VM the same way only if its VMM still runs +as the stop begins, and otherwise stops nothing and keeps the runtime record of +its launch, so that `reattach` finds that the VM exited without its holder. + `Vmm::terminate` takes that forced step alone, within the caller's bound, for a VM whose guest the caller must not ask anything, such as one of another runtime identity. diff --git a/nvx-rs/src/host/supervisor/mod.rs b/nvx-rs/src/host/supervisor/mod.rs index 95a5dbac0..55dcf9f51 100644 --- a/nvx-rs/src/host/supervisor/mod.rs +++ b/nvx-rs/src/host/supervisor/mod.rs @@ -86,6 +86,13 @@ //! [`Vmm::terminate`] takes that forced step alone, with no graceful step: it ends a VM whose //! guest the caller cannot ask to shut down, such as one that another runtime identity started. //! +//! A stop decides by its own check whether the VMM still runs as it begins. [`Vmm::stop`] ends a +//! VM whose VMM had exited by then as one that it stopped, and clears the runtime record of its +//! launch. [`Vmm::stop_unless_exited`] stops nothing then, and keeps the record, so that +//! [`reattach`] finds that the VM exited without its holder, as a caller that tells how the VM +//! ended, such as a clone's, needs: a holder that checked whether the VMM had exited, then stopped +//! the VM, would otherwise attribute to its stop a VMM that exited in between. +//! //! # Reattach //! //! After an orchestrator restarts within the same host boot, [`reattach`] ends an interrupted @@ -542,7 +549,10 @@ impl Vmm { /// The result says whether the VMM exited without the forced step, the forced step ended it, /// or it still runs; see [`StopOutcome`]. Once the VMM has exited, the stop clears the runtime /// record of its launch, as [`LockedStore::clear_runtime`] does, unless the record is another - /// VM's by then. A VMM that still runs keeps its record. + /// VM's by then. A VMM that still runs keeps its record. One that had exited before the stop + /// began, which runs no step, ends as one that the stop ended, as [`StopOutcome::Graceful`], + /// and its record is cleared: [`stop_unless_exited`](Self::stop_unless_exited) tells such a + /// VMM apart, and keeps its record. /// /// `locked` must be the locked store of the VM's sandbox. Fails with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if `forced` is zero or longer @@ -556,19 +566,33 @@ impl Vmm { forced: Option, graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, ) -> Result> { - self.check_store(locked)?; - if let Some(bound) = forced { - check_bound(bound, "the forced step")?; - } - let stopped = stop_process( - &self.process, - self.record.process().pid, - deadline, - forced, - || graceful(self, deadline), - )?; - self.clear_if_exited(locked, stopped.outcome)?; - Ok(stopped) + self.stop_with(locked, deadline, forced, graceful, &self.process) + } + + /// Stops the VM as [`stop`](Self::stop) does, unless its VMM had exited before the stop began, + /// and returns `None` then, with the runtime record of its launch kept. + /// + /// The stop decides by its own check of the VMM, before its graceful step, as + /// [`stop`](Self::stop) does. A VMM that had exited by then ended without its holder, however + /// recently the holder found it running: the stop runs no step, signals nothing, and clears + /// nothing, so that the record tells [`reattach`] that the VM exited unheld, as one that exits + /// while no caller holds it does. A caller that tells how the VM ended then reattaches, as + /// [`reattach_clone`](crate::host::template::reattach_clone) does, which records a clone's + /// failure, with the loss of its status if no report of the guest's own end tells why its VM + /// exited, before it clears the record. [`stop`](Self::stop) ends such a VM as one that it + /// stopped, and clears its record, so a holder that checks whether the VMM has exited, then + /// stops the VM with it, attributes to its stop a VMM that exits in between. + /// + /// A VMM that runs as the stop begins is stopped as [`stop`](Self::stop) stops it, with the + /// same result, and its exit is the stop's from then on. Fails as [`stop`](Self::stop) fails. + pub fn stop_unless_exited( + &self, + locked: &mut LockedStore, + deadline: Instant, + forced: Option, + graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, + ) -> Result>> { + self.stop_unless_exited_with(locked, deadline, forced, graceful, &self.process) } /// Ends the VM by identity-verified forced termination: terminates its VMM through its @@ -597,6 +621,51 @@ impl Vmm { self.terminate_with(locked, bound, &self.process) } + /// Stops the VM as [`stop`](Self::stop) does, through `process`: its VMM, or, in tests, a + /// stand-in for it. + fn stop_with( + &self, + locked: &mut LockedStore, + deadline: Instant, + forced: Option, + graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, + process: &impl Stoppable, + ) -> Result> { + if let Some(stopped) = + self.stop_unless_exited_with(locked, deadline, forced, graceful, process)? + { + return Ok(stopped); + } + self.clear_if_exited(locked, StopOutcome::Graceful)?; + Ok(Stopped { + outcome: StopOutcome::Graceful, + graceful_error: None, + }) + } + + /// Stops the VM as [`stop_unless_exited`](Self::stop_unless_exited) does, through `process`: + /// its VMM, or, in tests, a stand-in for it. + fn stop_unless_exited_with( + &self, + locked: &mut LockedStore, + deadline: Instant, + forced: Option, + graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, + process: &impl Stoppable, + ) -> Result>> { + self.check_store(locked)?; + if let Some(bound) = forced { + check_bound(bound, "the forced step")?; + } + let stopped = stop_process(process, self.record.process().pid, deadline, forced, || { + graceful(self, deadline) + })?; + if let Some(stopped) = &stopped { + self.clear_if_exited(locked, stopped.outcome)?; + } + Ok(stopped) + } + /// Ends the VM as [`terminate`](Self::terminate) does, through `process`: its VMM, or, in /// tests, a stand-in for it. fn terminate_with( @@ -648,11 +717,11 @@ fn check_bound(bound: Duration, what: &str) -> Result<()> { } } -/// How [`Vmm::stop`] or [`Vmm::terminate`] ended. +/// How [`Vmm::stop`], [`Vmm::stop_unless_exited`], or [`Vmm::terminate`] ended. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum StopOutcome { /// The VMM exited without the forced step: by the deadline of the graceful step, or before - /// the stop or the termination began. + /// the stop or the termination began, which [`Vmm::stop_unless_exited`] reports as no stop. Graceful, /// The forced step terminated the VMM through its verified handle, and it exited within the /// step's bound: the graceful step of a stop did not end it, or a termination took the step @@ -664,8 +733,8 @@ pub enum StopOutcome { TimedOut, } -/// What [`Vmm::stop`] did: how the stop ended, and the failure of its graceful step, if it -/// failed. +/// What [`Vmm::stop`] or [`Vmm::stop_unless_exited`] did: how the stop ended, and the failure of +/// its graceful step, if it failed. #[derive(Debug)] pub struct Stopped { /// How the stop ended. @@ -701,23 +770,21 @@ impl Stoppable for Process { } /// Stops the VMM `process`, whose ID is `pid`, with the graceful step `graceful`, the deadline -/// `deadline`, and the forced step that `forced` requests, as [`Vmm::stop`] describes, without -/// clearing its runtime record. +/// `deadline`, and the forced step that `forced` requests, as [`Vmm::stop_unless_exited`] +/// describes, without clearing its runtime record: returns `None`, and takes no step, if the VMM +/// had exited before. fn stop_process( process: &impl Stoppable, pid: u32, deadline: Instant, forced: Option, graceful: impl FnOnce() -> std::result::Result<(), E>, -) -> Result> { +) -> Result>> { if process .has_exited() .map_err(|error| undetermined(pid, error))? { - return Ok(Stopped { - outcome: StopOutcome::Graceful, - graceful_error: None, - }); + return Ok(None); } let graceful_error = graceful().err(); let exited = match graceful_error { @@ -730,10 +797,10 @@ fn stop_process( None => StopOutcome::TimedOut, Some(bound) => force(process, pid, bound)?, }; - Ok(Stopped { + Ok(Some(Stopped { outcome, graceful_error, - }) + })) } /// Terminates the VMM `process`, whose ID is `pid`, unless it has exited, and waits up to `bound` diff --git a/nvx-rs/src/host/supervisor/tests.rs b/nvx-rs/src/host/supervisor/tests.rs index bd8397f8a..d249029a4 100644 --- a/nvx-rs/src/host/supervisor/tests.rs +++ b/nvx-rs/src/host/supervisor/tests.rs @@ -157,12 +157,12 @@ impl Scripted { } /// Stops the VMM with `forced`, and with a graceful step that fails with `graceful_error` if - /// it is given. - fn stop( + /// it is given, unless it had exited, as [`Vmm::stop_unless_exited`] does. + fn stop_unless_exited( &self, forced: Option, graceful_error: Option<&'static str>, - ) -> Result> { + ) -> Result>> { let deadline = Instant::now() + Duration::from_secs(60); stop_process(self, 7, deadline, forced, || { self.note("graceful"); @@ -173,6 +173,17 @@ impl Scripted { }) } + /// Stops the VMM, which runs as the stop begins, as + /// [`stop_unless_exited`](Self::stop_unless_exited) does. + fn stop( + &self, + forced: Option, + graceful_error: Option<&'static str>, + ) -> Result> { + let stopped = self.stop_unless_exited(forced, graceful_error)?; + Ok(stopped.expect("the VMM had exited before the stop began")) + } + /// Returns the operations of the stop. fn operations(&self) -> Vec<&'static str> { self.operations.borrow().clone() @@ -298,8 +309,10 @@ fn a_failed_graceful_step_is_kept_and_not_waited_out() { fn a_vmm_that_exited_already_skips_the_graceful_step() { let vmm = Scripted::default(); vmm.exited.set(true); - let stopped = vmm.stop(Some(Duration::from_secs(10)), None).unwrap(); - assert_eq!(stopped.outcome, StopOutcome::Graceful); + let stopped = vmm + .stop_unless_exited(Some(Duration::from_secs(10)), None) + .unwrap(); + assert!(stopped.is_none(), "{stopped:?}"); assert_eq!(vmm.operations(), ["check"]); } @@ -523,11 +536,19 @@ fn stops_check_their_store_and_forced_bound_before_anything_else() { .stop(&mut other.lock(None).unwrap(), Instant::now(), None, never) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Other); + let error = vmm + .stop_unless_exited(&mut other.lock(None).unwrap(), Instant::now(), None, never) + .unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Other); for bound in [Duration::ZERO, MAX_TIMEOUT + Duration::from_secs(1)] { let error = vmm .stop(&mut locked, Instant::now(), Some(bound), never) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Unsupported, "{bound:?}"); + let error = vmm + .stop_unless_exited(&mut locked, Instant::now(), Some(bound), never) + .unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Unsupported, "{bound:?}"); } assert!(store.runtime().unwrap().is_some()); } @@ -675,6 +696,81 @@ fn a_clone_whose_forced_teardown_fails_keeps_its_lease_until_one_ends_its_vmm() assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); } +#[test] +fn only_a_stop_unless_exited_keeps_the_record_of_a_vmm_that_had_exited() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + let Reattached::Live(vmm) = reattach(&mut locked, &runtime()).unwrap() else { + panic!("the clone's VMM was not reattached"); + }; + // This test process stands for the VMM, which these stops reach only through a stand-in, + // which exited once its holder found it running, before the stops began. + let deadline = Instant::now() + Duration::from_secs(60); + let forced = Some(Duration::from_secs(10)); + let never = |_: &Vmm, _: Instant| -> std::result::Result<(), ()> { + panic!("a stop ran the graceful step of a VMM that had exited") + }; + let exited = Scripted::default(); + exited.exited.set(true); + // The stop takes no step, and keeps the record, which tells a reattach that the VM exited + // unheld, and with it the lease. + let stopped = vmm + .stop_unless_exited_with(&mut locked, deadline, forced, never, &exited) + .unwrap(); + assert!(stopped.is_none(), "{stopped:?}"); + assert_eq!(exited.operations(), ["check"]); + assert_eq!(locked.runtime().unwrap().as_ref(), Some(vmm.record())); + refuses_release(&templates, &mut locked, &lease); + // A stop that ends such a VM as one that it stopped clears the record, so the release ends + // the lease. + let stopped = vmm + .stop_with(&mut locked, deadline, forced, never, &exited) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + assert!(stopped.graceful_error.is_none()); + assert_eq!(exited.operations(), ["check", "check"]); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + +#[test] +fn a_stop_unless_exited_stops_a_running_vmm_as_a_stop_does() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + let Reattached::Live(vmm) = reattach(&mut locked, &runtime()).unwrap() else { + panic!("the clone's VMM was not reattached"); + }; + // This test process stands for the VMM, which these stops reach only through stand-ins. + let deadline = Instant::now() + Duration::from_secs(60); + // One that outlives the stop keeps its record, and the lease. + let survivor = Scripted::default(); + let stopped = vmm + .stop_unless_exited_with( + &mut locked, + deadline, + None, + |_, _| Ok::<_, ()>(()), + &survivor, + ) + .unwrap() + .expect("the stop of a VMM that runs"); + assert_eq!(stopped.outcome, StopOutcome::TimedOut); + assert_eq!(survivor.operations(), ["check", "wait"]); + refuses_release(&templates, &mut locked, &lease); + // One that heeds the graceful step exits through the stop, which clears its record. + let heeding = Scripted::default(); + let shut_down = |_: &Vmm, _: Instant| { + heeding.exited.set(true); + Ok::<_, ()>(()) + }; + let stopped = vmm + .stop_unless_exited_with(&mut locked, deadline, None, shut_down, &heeding) + .unwrap() + .expect("the stop of a VMM that runs"); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + assert_eq!(heeding.operations(), ["check", "wait"]); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + #[test] fn a_stop_keeps_the_record_of_another_vmm_under_the_same_launch_identity() { let (_directory, store) = store(); diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index 8da6cf6a9..15ce6fa31 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -97,7 +97,9 @@ pub enum ReattachedClone { /// evidence, as a status of the clone's repair is at any point, and with /// [`Evidence::LostStatus`] if no report of the guest's own end tells why it exited, and /// [`ReattachedClone::Ended`] otherwise. One that cannot be examined is -/// [`ReattachedClone::Indeterminate`]. +/// [`ReattachedClone::Indeterminate`]. A VM that its holder's stop ended, which cleared its +/// runtime record, lost no status, but one whose VMM had exited before the stop began is one +/// that exited, as above, since [`Vmm::stop_unless_exited`] keeps its record. /// /// The evidence of a status of the guest's prevails over any but the template's, as in the restore /// sequence, and as [the attribution of evidence](Evidence#attribution) describes. A failure is @@ -373,8 +375,9 @@ pub(super) fn reattach_clone_with( /// `progress`, and whose VM exited, as OpenVMM's outcome report, which /// [`Recorder::observed`] reads, tells, with the guest's status, or `None` if the VM ended /// without a failure: a status of the guest's that is evidence, as [`Recorder::reported`] gives -/// it, or, for a VM that exited `unheld`, which the reattach found exited, the loss of the clone's -/// status, if no report of the guest's own end tells why. +/// it, or, for a VM that exited `unheld`, whose runtime record the reattach found, as +/// [`Vmm::stop_unless_exited`] keeps that of a VMM that had exited before the stop began, the loss +/// of the clone's status, if no report of the guest's own end tells why. fn exit_fault( recorder: &Recorder, progress: Progress, @@ -388,8 +391,9 @@ fn exit_fault( let (evidence, status) = recorder.reported(recorder.observed(), progress); let evidence = match evidence { Some(evidence) => evidence, - // Only a VM that exited unheld ends without telling why: its holder's stop ends one - // otherwise. + // Only a VM that exited unheld, whose runtime record the reattach found, ends without + // telling why: its holder's stop ends one otherwise, and clears the record, but keeps that + // of one that had exited before the stop began, as `Vmm::stop_unless_exited` does. None if unheld && status.is_none() => Evidence::LostStatus, None => return None, }; diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index 83f4e2a36..d7eac12f7 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -25,7 +25,7 @@ use crate::host::store::{ CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, Recovery, RuntimeIdentity, }; -use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; +use crate::host::supervisor::{self, Reattached, StopOutcome, Vmm, VmmCommand}; use crate::host::template::Retirement; use crate::host::template::clone::launch::{Hooks, Point}; use crate::host::template::clone::reattach::reattach_clone_with; @@ -3038,6 +3038,75 @@ fn a_clone_that_its_holder_stopped_ended_whatever_a_reboot_left_of_its_progress_ } } +#[test] +fn a_clone_whose_vmm_exited_before_its_holder_s_stop_began_lost_its_status() { + let never = |_: &Vmm, _: Instant| -> io::Result<()> { + panic!("a stop ran the graceful step of a VMM that had exited") + }; + // Its holder found the clone's VMM running, which then exits, without a report of the + // guest's own end, before the holder's stop begins. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach found {}", describe(&other)), + }; + crashed.vmm.end_and_wait(); + let stopped = vmm + .stop_unless_exited(&mut locked, Instant::now() + TIMEOUT, None, never) + .unwrap(); + assert!(stopped.is_none(), "{stopped:?}"); + // The stop kept the runtime record, and with it the lease, so the reattach finds that the VM + // exited unheld: the clone lost its status, which quarantines nothing. + assert_eq!(locked.runtime().unwrap().as_ref(), Some(vmm.record())); + crashed.refuses_release(&mut locked); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{failure}"); + let (run, lost) = ((Stage::Run, Progress::Committed), Evidence::LostStatus); + assert_recorded( + &failure, + &crashed.lease, + &locked, + run, + lost, + Teardown::Exited, + ); + assert_eq!(failure.record().status, None); + let deadline = Some(Instant::now() + TIMEOUT); + assert!( + !failure + .quarantine(&crashed.fixture.store, deadline) + .unwrap() + ); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + + // A stop that ends such a VM as one that it stopped clears the record, and the clone ended. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach found {}", describe(&other)), + }; + crashed.vmm.end_and_wait(); + let stopped = vmm + .stop(&mut locked, Instant::now() + TIMEOUT, None, never) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + assert!(locked.runtime().unwrap().is_none()); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Committed, + report: None, + } => {} + other => panic!("the reattach found {}", describe(&other)), + } + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert!(recorded.is_none(), "{recorded:?}"); + crashed.releases(&mut locked); +} + #[test] fn an_interrupted_launch_that_a_reattach_cannot_end_yet_keeps_its_lease_until_one_can() { let crashed = Crashed::at("activated"); From 1abc2784d1f8566a64dc3b6e827ea6d83abe71f2 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Wed, 28 Oct 2026 16:37:32 -0700 Subject: [PATCH 11/12] nvx-rs: keep a clone's launch until its failure is recorded A clone's failure is recorded with its teardown pending before the teardown, so that a crash of the caller during the teardown leaves the record for a reattach to finish. But a pending record that could not be written, as on a full disk, was ignored: the teardown still removed the launch's journal, or the VM's runtime record, and a crash before the final record then left nothing of the failure. A reattach found a clone that ended, the lease was released as if nothing had failed, and the failure was lost, a diagnosis of the template's included. A clone's lifecycle record, the journal of its launch or the runtime record of its VM, and with it the lease, now outlives every failure that cannot be recorded before its teardown: - A failure of the restore sequence, or of the caller's Activated::fail or Started::fail, ends the clone's VMM through its handle, and keeps the launch's journal, through Starting::end_keeping_journal, as a failed commit does. Its teardown is unresolved until a reattach ends the launch, which records the failure, though not how it happened. - A launch that fails before it returns kills a VMM that it spawned and could not record, as before, and keeps its journal for recovery: the `failing` callback of supervisor::launch_reporting now returns whether the caller recorded the failure. - reattach_clone fails before it ends or forgets anything of the clone's VM: the VM and its records stay as it found them, running or not, for a later reattach. The `exited` callback of supervisor::reattach_reporting now fails the reattach, and keeps the runtime record, if the caller cannot record the VM's exit. - A record that a write put in place, but whose sync of the sandbox's directory failed, may not be durable yet, and a reattach takes it as the record of an earlier failure, which it does not write again. So reattach_clone syncs the sandbox's directory before it ends, or forgets, anything of a launch whose failure the sandbox records, and fails as above if the sync fails. The rules of quarantine and of leases are unchanged: a lease is released only once its VM's exit is verified, and a failure that a caller still holds quarantines as before. Tests make the record of the failure unwritable with a directory in its place: a caller's diagnosis of the template keeps the launch's journal, then a reattach finds the clone failed; reattaches of an interrupted launch, of a VM that exited, of one under another runtime identity, and of one that fails its activation check end and forget nothing, then record the failure once they can; and the supervisor keeps the journal of a launch, and the runtime record of a VM that exited, whose caller cannot record them. Another refuses the sync of an earlier failure's record: the reattach then ends nothing, and one that can sync ends the launch only once it synced. Reviews of this pull request found these. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a424884c-018a-45f5-a7fe-c57548a873f0 --- nvx-rs/README.md | 21 +- nvx-rs/src/host/supervisor/mod.rs | 113 ++++++--- nvx-rs/src/host/supervisor/tests.rs | 59 ++++- nvx-rs/src/host/template/clone/failure.rs | 8 +- nvx-rs/src/host/template/clone/launch.rs | 3 +- nvx-rs/src/host/template/clone/reattach.rs | 99 ++++++-- nvx-rs/src/host/template/clone/records.rs | 44 +++- nvx-rs/src/host/template/clone/restore.rs | 3 +- .../src/host/template/clone/tests/launch.rs | 216 +++++++++++++++++- .../src/host/template/clone/tests/restore.rs | 59 +++++ 10 files changed, 544 insertions(+), 81 deletions(-) diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 63cace754..07895889e 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -773,7 +773,9 @@ evidence and the status, the original outcome, the transaction, the generation, the progress, and the teardown: no VM, an exit or a termination that the host verified, or an unresolved one, which keeps the launch's journal or runtime record, and the lease; a release removes both, and a sandbox that boots cold -writes neither. +writes neither. A failure that cannot be recorded before its teardown keeps the +launch's journal or runtime record, and the lease, so that a reattach still +finds that the clone failed. A step or a check of the caller's own that fails once the clone is ready, or once its sandbox started, fails the clone with the caller's evidence through @@ -788,14 +790,15 @@ that another runtime identity launched, or that fails the check, through its verified identity; it records the failure of a launch that a crash interrupted, but not the journal that a commit may leave beside its runtime record, classifies a VM that exited by its outcome report, records a failure before it -ends a VMM or clears the runtime record of one that exited, finishes a failure -that a crash left pending as its teardown would have recorded it, keeping its -transaction, resolves the teardown of an earlier failure with what the VMM left -as it ended, its outcome report and OpenVMM's fatal error, attributes a status -of the guest's as one that a workload may have given once the clone's progress -log is lost, damaged, or was written during another boot of the host, whose -crash may have lost its last lines, and reads nothing of the template store, so -that a tombstone or a retirement of the generation leaves the clone its lease. +ends a VMM or clears the runtime record of one that exited, and fails, leaving +them as they were, if it cannot, finishes a failure that a crash left pending as +its teardown would have recorded it, keeping its transaction, resolves the +teardown of an earlier failure with what the VMM left as it ended, its outcome +report and OpenVMM's fatal error, attributes a status of the guest's as one that +a workload may have given once the clone's progress log is lost, damaged, or was +written during another boot of the host, whose crash may have lost its last +lines, and reads nothing of the template store, so that a tombstone or a +retirement of the generation leaves the clone its lease. `CloneRecords::read` reads what the sandbox keeps of a clone, for the operations that list, inspect, or remove clones. diff --git a/nvx-rs/src/host/supervisor/mod.rs b/nvx-rs/src/host/supervisor/mod.rs index 55dcf9f51..a364a49f6 100644 --- a/nvx-rs/src/host/supervisor/mod.rs +++ b/nvx-rs/src/host/supervisor/mod.rs @@ -140,7 +140,8 @@ //! [`Starting::commit`] does. A failure ends the launch, as [`Starting::abandon`] does, through //! the VMM's verified identity, and the sandbox records it until the release, with the evidence //! that attributes it, as a [`CloneFailure`](crate::host::template::CloneFailure) describes: a -//! launch that cannot be ended keeps its journal, and the lease. +//! launch that cannot be ended keeps its journal, and the lease, and so does one whose failure +//! cannot be recorded before its end, whose VMM is ended through its handle. //! 4. Once the clone's VM has exited, as a stop, a termination, or a reattach verifies, //! [`Store::release`](crate::host::template::Store::release) ends the lease, under the //! sandbox's lock, taken through the lease's path, @@ -238,8 +239,15 @@ pub fn launch<'s, 'h>( capability: &[u8; CAPABILITY_LEN], command: impl FnOnce(&str) -> Result>, ) -> Result> { - launch_reporting(locked, reserved, runtime, capability, command, |_, _| {}) - .map_err(|failure| failure.error) + launch_reporting( + locked, + reserved, + runtime, + capability, + command, + |_, _| Ok(()), + ) + .map_err(|failure| failure.error) } /// How a launch of [`launch_reporting`] failed. @@ -250,22 +258,25 @@ pub(crate) struct LaunchFailure { pub(crate) error: Error, /// Whether the launch spawned its VMM, which it then ended through the VMM's handle. pub(crate) spawned: bool, - /// Why the launch could not be ended, whose journal then remains for - /// [`LockedStore::recover`], or `None` once it ended. + /// Why the launch was not ended, whose journal then remains for [`LockedStore::recover`]: + /// it could not be, or its caller could not record its failure first. `None` once it ended. pub(crate) unended: Option, } /// Launches a VMM as [`launch`] does, and tells how a launch that fails ended, as a /// [`LaunchFailure`]. A launch that began, and fails, calls `failing` with the step's error, and /// whether it spawned the VMM, before it ends the launch, so that the caller may record the -/// failure, as a clone's launch does, before anything of the launch ends. +/// failure, as a clone's launch does, before anything of the launch ends. If `failing` fails, the +/// launch ends a VMM that it spawned through the VMM's handle, as always, but keeps its journal, +/// as a crash would leave it, for [`LockedStore::recover`]: the journal then still tells that the +/// launch began, though the caller could not record its failure. pub(crate) fn launch_reporting<'s, 'h>( locked: &'s mut LockedStore, reserved: &LaunchId, runtime: &RuntimeIdentity, capability: &[u8; CAPABILITY_LEN], command: impl FnOnce(&str) -> Result>, - failing: impl FnOnce(&Error, bool), + failing: impl FnOnce(&Error, bool) -> Result<()>, ) -> std::result::Result, LaunchFailure> { let store = SandboxStore::clone(locked); // A launch that does not begin leaves nothing to end. @@ -279,15 +290,15 @@ pub(crate) fn launch_reporting<'s, 'h>( let command = match command(launch.endpoint()) { Ok(command) => command, Err(error) => { - failing(&error, false); - return Err(abandon_after(launch, error, false)); + let recorded = failing(&error, false); + return Err(abandon_after(launch, error, false, recorded)); } }; let log = match launch.take_log() { Ok(log) => log, Err(error) => { - failing(&error, false); - return Err(abandon_after(launch, error, false)); + let recorded = failing(&error, false); + return Err(abandon_after(launch, error, false, recorded)); } }; let child = match spawn(&command, capability, log) { @@ -295,18 +306,18 @@ pub(crate) fn launch_reporting<'s, 'h>( Err(error) => { let error = Error::io(format!("cannot launch {}", command.program.display())) .with_source(error); - failing(&error, false); - return Err(abandon_after(launch, error, false)); + let recorded = failing(&error, false); + return Err(abandon_after(launch, error, false, recorded)); } }; let process = match launch.record_process(&child) { Ok(process) => process, Err(error) => { - failing(&error, true); + let recorded = failing(&error, true); // An unrecorded VMM holds the claim of its launch, so only its handle can end it, // before the launch ends. child.kill_and_wait(); - return Err(abandon_after(launch, error, true)); + return Err(abandon_after(launch, error, true, recorded)); } }; Ok(Starting { @@ -319,20 +330,33 @@ pub(crate) fn launch_reporting<'s, 'h>( }) } -/// Ends `launch`, which failed with `error`, after it `spawned` a VMM or not, and returns how it -/// failed, whose error says why the launch could not be ended if it could not: its journal then -/// remains for [`LockedStore::recover`]. -fn abandon_after(launch: Launch<'_>, error: Error, spawned: bool) -> LaunchFailure { - let reason = match launch.abandon() { - Ok(Recovery::Indeterminate(reason)) => reason, - Ok(_) => { - return LaunchFailure { - error, - spawned, - unended: None, - }; +/// Ends `launch`, which failed with `error`, after it `spawned` a VMM or not, once its caller +/// `recorded` the failure, and returns how it failed, whose error says why the launch was not +/// ended if it was not: its journal then remains for [`LockedStore::recover`]. A launch whose +/// caller could not record its failure keeps its journal, as a crash would leave it, and ends no +/// more of it than its caller did. +fn abandon_after( + launch: Launch<'_>, + error: Error, + spawned: bool, + recorded: Result<()>, +) -> LaunchFailure { + let reason = match recorded { + Err(unrecorded) => { + drop(launch); + unrecorded.to_string() } - Err(failure) => failure.to_string(), + Ok(()) => match launch.abandon() { + Ok(Recovery::Indeterminate(reason)) => reason, + Ok(_) => { + return LaunchFailure { + error, + spawned, + unended: None, + }; + } + Err(failure) => failure.to_string(), + }, }; let (kind, message, source) = error.into_parts(); let error = Error::new( @@ -444,7 +468,8 @@ impl<'s> Starting<'s> { /// Commits the launch as [`commit`](Self::commit) does, and calls `failing` with the error of /// a commit that fails, before it ends the VMM, so that the caller may record the failure - /// first, as a clone's launch does. + /// first, as a clone's launch does. The journal remains all the same, whether or not the + /// caller could. pub(crate) fn commit_reporting(mut self, failing: impl FnOnce(&Error)) -> Result { let StartingInner { launch, @@ -480,6 +505,25 @@ impl<'s> Starting<'s> { .expect("a launch is abandoned once") .abandon() } + + /// Ends the launch's VMM through its handle, and keeps the launch's journal, as a crash of the + /// caller would leave it, for [`LockedStore::recover`], which ends the launch once it finds + /// the VMM exited. Returns whether the VMM exited, as [`Launched::kill_and_wait`] does. + /// + /// It ends a launch whose caller could not record its failure before the launch's end, as a + /// clone's launch records it, so that the journal, which keeps the clone's lease, still tells + /// that the launch began. + pub(crate) fn end_keeping_journal(mut self) -> bool { + let StartingInner { + launch, + child, + process: _, + } = self.inner.take().expect("a launch is ended once"); + let exited = child.kill_and_wait(); + // A launch that is dropped keeps its journal. + drop(launch); + exited + } } impl Drop for Starting<'_> { @@ -891,16 +935,17 @@ pub enum Reattached { /// can. Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for an invalid /// `runtime`, and as [`LockedStore::recover`] does for inconsistent records. pub fn reattach(locked: &mut LockedStore, runtime: &RuntimeIdentity) -> Result { - reattach_with(locked, runtime, Process::open, |_| {}) + reattach_with(locked, runtime, Process::open, |_| Ok(())) } /// Reattaches as [`reattach`] does, and calls `exited` with the runtime record of a VMM that it /// found exited, before it clears the record, so that the caller may record what ended the VM -/// while the sandbox still names it, as a clone's reattach does. +/// while the sandbox still names it, as a clone's reattach does. Fails, and keeps the record, if +/// `exited` fails. pub(crate) fn reattach_reporting( locked: &mut LockedStore, runtime: &RuntimeIdentity, - exited: impl FnOnce(&RuntimeRecord), + exited: impl FnOnce(&RuntimeRecord) -> Result<()>, ) -> Result { reattach_with(locked, runtime, Process::open, exited) } @@ -911,7 +956,7 @@ fn reattach_with( locked: &mut LockedStore, runtime: &RuntimeIdentity, open: impl FnOnce(&ProcessRef) -> io::Result>, - exited: impl FnOnce(&RuntimeRecord), + exited: impl FnOnce(&RuntimeRecord) -> Result<()>, ) -> Result { runtime.check()?; let recovery = locked.recover()?; @@ -930,7 +975,7 @@ fn reattach_with( let process = match open(record.process()) { Ok(Some(process)) => process, Ok(None) => { - exited(&record); + exited(&record)?; locked.clear_runtime_of(Some(&record))?; return Ok(Reattached::Exited { record: Some(record), diff --git a/nvx-rs/src/host/supervisor/tests.rs b/nvx-rs/src/host/supervisor/tests.rs index d249029a4..fc105f296 100644 --- a/nvx-rs/src/host/supervisor/tests.rs +++ b/nvx-rs/src/host/supervisor/tests.rs @@ -474,7 +474,7 @@ fn a_vmm_that_cannot_be_examined_is_indeterminate_and_keeps_its_record() { Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) }; let Reattached::Indeterminate(reason) = - reattach_with(&mut locked, &runtime(), hidden, |_| {}).unwrap() + reattach_with(&mut locked, &runtime(), hidden, |_| Ok(())).unwrap() else { panic!("a VMM that cannot be examined was classified"); }; @@ -577,6 +577,61 @@ fn a_launch_whose_command_fails_ends_before_anything_runs() { assert_eq!(locked.recover().unwrap(), Recovery::Clean); } +#[test] +fn a_launch_whose_caller_cannot_record_its_failure_keeps_its_journal_for_recovery() { + let (_directory, store) = store(); + let mut locked = store.lock(None).unwrap(); + let failure = launch_reporting( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |_| Err(Error::unsupported("no command")), + |error, spawned| { + assert_eq!((error.message(), spawned), ("no command", false)); + Err(Error::io("the test cannot record the failure")) + }, + ) + .unwrap_err(); + assert!(!failure.spawned); + let reason = "the test cannot record the failure"; + assert_eq!(failure.unended.as_deref(), Some(reason)); + assert_eq!(failure.error.kind(), ErrorKind::Unsupported); + assert_eq!( + failure.error.message(), + format!("no command; the launch could not be ended, and remains for recovery: {reason}") + ); + // The journal remains, as a crash leaves it, until a recovery ends the launch, of which + // nothing runs. + assert!(store.journal().unwrap().is_some()); + assert_eq!(locked.recover().unwrap(), Recovery::Exited); + assert!(store.journal().unwrap().is_none()); +} + +#[test] +fn a_reattach_whose_caller_cannot_record_an_exit_keeps_the_record_until_one_can() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + // The recorded VMM exited, but the caller cannot record what ended it. + let exited = |_: &ProcessRef| -> io::Result> { Ok(None) }; + let error = reattach_with(&mut locked, &runtime(), exited, |_| { + Err(Error::io("the test cannot record the exit")) + }) + .unwrap_err(); + assert_eq!(error.message(), "the test cannot record the exit"); + // The record, and with it the lease, remain for a later reattach. + assert!(locked.runtime().unwrap().is_some()); + refuses_release(&templates, &mut locked, &lease); + let Reattached::Exited { record, recovery } = + reattach_with(&mut locked, &runtime(), exited, |_| Ok(())).unwrap() + else { + panic!("a VMM that exited was not reported as exited"); + }; + assert_eq!(record.unwrap().launch(), lease.launch()); + assert_eq!(recovery, Recovery::Clean); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + #[test] fn a_launch_whose_vmm_cannot_start_ends() { let (directory, store) = store(); @@ -646,7 +701,7 @@ fn a_clone_whose_vmm_cannot_be_examined_keeps_its_lease() { Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) }; let Reattached::Indeterminate(reason) = - reattach_with(&mut locked, &runtime(), hidden, |_| {}).unwrap() + reattach_with(&mut locked, &runtime(), hidden, |_| Ok(())).unwrap() else { panic!("a VMM that cannot be examined was classified"); }; diff --git a/nvx-rs/src/host/template/clone/failure.rs b/nvx-rs/src/host/template/clone/failure.rs index 02d6cf15e..3b3af078e 100644 --- a/nvx-rs/src/host/template/clone/failure.rs +++ b/nvx-rs/src/host/template/clone/failure.rs @@ -496,8 +496,12 @@ impl Teardown { /// caller during the teardown leaves the record, which a reattach finishes, and once its teardown /// is known, so that the record holds the stage, the evidence and the status, the original /// outcome, the transaction of the clone's restore, the leased generation, the clone's progress, -/// and the teardown's outcome. A failure of the plan, before the launch, leaves no VM and no -/// record: the caller holds the failure, and releases the lease. A launch keeps its lease, and +/// and the teardown's outcome. A failure that cannot be recorded before its teardown removes +/// nothing that tells that the clone launched: the journal of its launch, or the runtime record of +/// its VM, remains, and with it the lease, so that a reattach still finds that the clone failed, +/// though not how, if a crash of the caller comes before the record. A failure of the plan, before +/// the launch, leaves no VM and no record: the caller holds the failure, and releases the lease. A +/// launch keeps its lease, and /// with it the leased generation, until its VM has exited, so a failure that comes after the /// clone's Ready, such as one of the repair's deferred work, is classified against that /// generation all the same, and so is the status with which the guest ended a VM whose exit a diff --git a/nvx-rs/src/host/template/clone/launch.rs b/nvx-rs/src/host/template/clone/launch.rs index d7d367d42..af4cbe510 100644 --- a/nvx-rs/src/host/template/clone/launch.rs +++ b/nvx-rs/src/host/template/clone/launch.rs @@ -353,7 +353,8 @@ impl ClonePlan { if spawned { tracker.advance(Progress::Spawned); } - tracker.pending(Stage::Launch, host_evidence(error), error); + // A failure that cannot be recorded keeps the launch's journal. + tracker.pending(Stage::Launch, host_evidence(error), error) }, ); // The VMM holds the claim, if it started. diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index 15ce6fa31..d6886d722 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -16,6 +16,7 @@ use crate::error::{Error, ErrorCode, Result}; use crate::host::launch::timeout_problem; use crate::host::process::BootId; use crate::host::report::Report; +use crate::host::store::files::sync_dir; use crate::host::store::regular::{open_regular, read_at_most}; use crate::host::store::{IdentityPart, LockedStore, Recovery, RuntimeIdentity, RuntimeRecord}; use crate::host::supervisor::{self, Reattached, StopOutcome, Vmm}; @@ -105,7 +106,9 @@ pub enum ReattachedClone { /// sequence, and as [the attribution of evidence](Evidence#attribution) describes. A failure is /// recorded in the sandbox, as the restore sequence records it: before the reattach ends a VMM of /// the clone, recovers an interrupted launch, or clears the runtime record of a VM that exited, -/// with its teardown [pending](Teardown::Pending), and again once the teardown ended. A clone +/// with its teardown [pending](Teardown::Pending), and again once the teardown ended. A reattach +/// that cannot record it then fails, before it ends or forgets anything of the clone's VM, as does +/// one that cannot make the record of an earlier failure durable. A clone /// whose sandbox records a failure already failed for good: the reattach ends a VMM of the clone /// that still runs, through its verified identity, then finishes a failure whose teardown a crash /// of its caller left pending, as the failure would have recorded it once its teardown ended, or @@ -142,7 +145,10 @@ pub enum ReattachedClone { /// writes anything, and as [`Lease::of`] and [`supervisor::reattach`] fail for records that are /// inconsistent, which leave the clone's VM as it was, and the record of the failure of an /// interrupted launch, or the one that replaced a record that could not be read, if the reattach -/// wrote it, pending. +/// wrote it, pending. Fails with [`ErrorKind::Io`](crate::ErrorKind::Io) if it cannot record the +/// clone's failure, or make the record of an earlier one durable, before it would end, or forget, +/// anything of the clone's VM, which it then leaves as it found it, running or not, with the +/// journal of its launch or its runtime record, and the lease, for a later reattach. /// /// [`CloneRecords`]: super::CloneRecords pub fn reattach_clone( @@ -164,7 +170,8 @@ pub fn reattach_clone( /// Reattaches to a clone as [`reattach_clone`] does, reattaching to its VM with `reattach`, which /// is [`supervisor::reattach_reporting`] outside tests, and which calls its last argument with -/// the runtime record of a VMM that exited before it clears the record. +/// the runtime record of a VMM that exited before it clears the record, and fails, keeping the +/// record, if that fails. pub(super) fn reattach_clone_with( locked: &mut LockedStore, runtime: &RuntimeIdentity, @@ -174,7 +181,27 @@ pub(super) fn reattach_clone_with( reattach: impl FnOnce( &mut LockedStore, &RuntimeIdentity, - &dyn Fn(&RuntimeRecord), + &dyn Fn(&RuntimeRecord) -> Result<()>, + ) -> Result, +) -> Result> { + reattach_clone_syncing( + locked, runtime, control, deadline, bound, &sync_dir, reattach, + ) +} + +/// Reattaches to a clone as [`reattach_clone_with`] does, syncing the sandbox's directory, to make +/// the record of an earlier failure durable, with `sync`, which is [`sync_dir`] outside tests. +pub(super) fn reattach_clone_syncing( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + deadline: Instant, + bound: Duration, + sync: &dyn Fn(&Path) -> Result<()>, + reattach: impl FnOnce( + &mut LockedStore, + &RuntimeIdentity, + &dyn Fn(&RuntimeRecord) -> Result<()>, ) -> Result, ) -> Result> { let started = Instant::now(); @@ -228,6 +255,20 @@ pub(super) fn reattach_clone_with( .and_then(|(_, record)| record.restore().map(|restore| restore.transaction)); let recorder = Recorder::new(lease.clone(), transaction, locked).uncertain(uncertain); let fresh = earlier.is_none(); + // A record of an earlier failure may be in place without being durable, as a write whose sync + // of the sandbox's directory failed leaves it: the reattach makes it durable before it ends, + // or forgets, anything that tells that the clone launched, as it records a failure of its own. + if let Some(Ok(_)) = &earlier + && (journal.is_some() || committed.is_some()) + { + sync(locked.dir()).map_err(|error| { + unrecorded(Error::io(format!( + "the record of the clone's failure could not be made durable before its \ + teardown: {}", + describe(&error) + ))) + })?; + } // A record that cannot be read, or that is another clone's, is replaced by the reattach's own // failure, with its teardown pending, before the reattach ends or forgets anything of the // clone's launch: the record keeps how far the clone got, which the launch's journal and @@ -237,12 +278,14 @@ pub(super) fn reattach_clone_with( if let Some(Err(fault)) = &earlier && (journal.is_some() || committed.is_some()) { - recorder.pending( - Stage::Reattach, - progress, - (fault.evidence, None), - &fault.error, - ); + recorder + .pending( + Stage::Reattach, + progress, + (fault.evidence, None), + &fault.error, + ) + .map_err(unrecorded)?; } // A launch that the journal names, which no runtime record supersedes, and whose failure the // sandbox does not record, was interrupted by a crash of its caller: its failure is recorded @@ -250,12 +293,14 @@ pub(super) fn reattach_clone_with( // record. A journal that the runtime record of its launch supersedes, which the launch's // commit may leave, is no failure: the recovery removes it. if fresh && journal.is_some() && runtime_record.is_none() { - recorder.pending( - progress.stage(), - progress, - (Evidence::LostStatus, None), - &interrupted(progress), - ); + recorder + .pending( + progress.stage(), + progress, + (Evidence::LostStatus, None), + &interrupted(progress), + ) + .map_err(unrecorded)?; } // So is the failure of a VM whose launch committed, and which exited, before the reattach // clears its runtime record, as the outcome report that the recorder reads then tells, which @@ -263,8 +308,11 @@ pub(super) fn reattach_clone_with( let exiting = |_: &RuntimeRecord| { if fresh && let Some((fault, status)) = exit_fault(&recorder, progress, true) { let evidence = (fault.evidence, status); - recorder.pending(progress.stage(), progress, evidence, &fault.error); + return recorder + .pending(progress.stage(), progress, evidence, &fault.error) + .map_err(unrecorded); } + Ok(()) }; let reattached = reattach(locked, runtime, &exiting)?; let failed = |fault: Fault, stage: Stage, teardown: Teardown| { @@ -324,7 +372,9 @@ pub(super) fn reattach_clone_with( }), Err(fault) => { let evidence = (fault.evidence, None); - recorder.pending(Stage::Reattach, progress, evidence, &fault.error); + recorder + .pending(Stage::Reattach, progress, evidence, &fault.error) + .map_err(unrecorded)?; let teardown = stopped(vmm.terminate(locked, bound), bound); failed(fault, Stage::Reattach, teardown) } @@ -333,7 +383,9 @@ pub(super) fn reattach_clone_with( Reattached::Mismatched { vmm, differences } => { let fault = Fault::new(Evidence::RuntimeMismatch, mismatched(&differences)); let evidence = (fault.evidence, None); - recorder.pending(Stage::Reattach, progress, evidence, &fault.error); + recorder + .pending(Stage::Reattach, progress, evidence, &fault.error) + .map_err(unrecorded)?; let teardown = stopped(vmm.terminate(locked, bound), bound); failed(fault, Stage::Reattach, teardown) } @@ -511,6 +563,15 @@ fn unended(error: Error, teardown: &Teardown) -> Error { } } +/// Returns the error of a reattach that could not record the clone's failure, as `error` tells, +/// before it would end, or forget, anything of the clone's VM, which it leaves as it found it. +fn unrecorded(error: Error) -> Error { + amended( + error, + "; the reattach left the clone's VM and its records as it found them, for a later reattach", + ) +} + /// Returns the error of the launch of a clone that got as far as `progress` before a crash of its /// caller interrupted it. fn interrupted(progress: Progress) -> Error { diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs index 699acc0f9..8deacfbdd 100644 --- a/nvx-rs/src/host/template/clone/records.rs +++ b/nvx-rs/src/host/template/clone/records.rs @@ -418,15 +418,20 @@ impl Recorder { /// teardown that follows it, whose record is [`Teardown::Pending`], so that a crash of the /// caller during the teardown leaves the record, which a reattach finishes, as /// [`reattach_clone`](super::reattach_clone) describes. The record that the failure writes - /// once its teardown ended replaces it. A record that cannot be written now is written then, - /// if it can be. + /// once its teardown ended replaces it. + /// + /// Fails if the record cannot be written, durably, as [`write_failure`] writes it: the caller + /// then removes nothing that tells that the clone launched, the journal of its launch or the + /// runtime record of its VM, which keep its lease, so that a reattach still finds that the + /// clone failed, though not how, if a crash of the caller comes before the failure is + /// recorded. pub(super) fn pending( &self, stage: Stage, progress: Progress, (evidence, status): (Evidence, Option), error: &Error, - ) { + ) -> Result<()> { let record = FailureRecord::new( &self.lease, self.transaction, @@ -436,7 +441,12 @@ impl Recorder { Teardown::Pending, error, ); - let _ = write_failure(&self.failure, &record); + write_failure(&self.failure, &record).map_err(|failed| { + Error::io(format!( + "the failure could not be recorded before its teardown: {}", + describe(&failed) + )) + }) } /// Resolves `earlier`, the record of a failure that its teardown left unresolved, with @@ -521,19 +531,35 @@ impl Tracker { } /// Records the failure of the clone at `stage` with `evidence` and `error` before the teardown - /// that follows it, once the clone got as far as its progress, as [`Recorder::pending`] does. - pub(super) fn pending(&self, stage: Stage, evidence: Evidence, error: &Error) { + /// that follows it, once the clone got as far as its progress, as [`Recorder::pending`] does, + /// and fails as it fails. + pub(super) fn pending(&self, stage: Stage, evidence: Evidence, error: &Error) -> Result<()> { self.recorder - .pending(stage, self.log.progress(), (evidence, None), error); + .pending(stage, self.log.progress(), (evidence, None), error) } /// Ends `starting`, the launch of the clone, which failed at `stage` with `fault`, as `end` /// does, and records the failure before the teardown, as [`pending`](Self::pending) does, and /// with the teardown's outcome, as [`record`](Self::record) does. The error of a launch that /// could not be ended says so, and that its journal remains for recovery. + /// + /// A failure that cannot be recorded before the teardown ends the clone's VMM through its + /// handle, and keeps the launch's journal, as [`Starting::end_keeping_journal`] does, and with + /// it the lease, until a reattach ends the launch: the teardown is unresolved, so that a crash + /// of the caller before the failure is recorded leaves a launch that a reattach records as + /// failed, rather than a clone that ended. pub(super) fn fail(&self, starting: Starting<'_>, stage: Stage, fault: Fault) -> CloneFailure { - self.pending(stage, fault.evidence, &fault.error); - let teardown = Teardown::of((self.end)(starting)); + let teardown = match self.pending(stage, fault.evidence, &fault.error) { + Ok(()) => Teardown::of((self.end)(starting)), + Err(unrecorded) => { + let ended = if starting.end_keeping_journal() { + "the clone's VMM ended through its handle" + } else { + "the clone's VMM did not exit once it was ended through its handle" + }; + Teardown::Unresolved(format!("{ended}, but {unrecorded}")) + } + }; let fault = match &teardown { Teardown::Unresolved(reason) => Fault::new( fault.evidence, diff --git a/nvx-rs/src/host/template/clone/restore.rs b/nvx-rs/src/host/template/clone/restore.rs index 93dc92d5a..5387c403f 100644 --- a/nvx-rs/src/host/template/clone/restore.rs +++ b/nvx-rs/src/host/template/clone/restore.rs @@ -510,7 +510,8 @@ impl<'s, S> Started<'s, S> { .. } = self; let committed = starting.commit_reporting(|error| { - tracker.pending(Stage::Commit, host_evidence(error), error); + // The launch's journal remains, whether or not the failure can be recorded. + let _ = tracker.pending(Stage::Commit, host_evidence(error), error); }); match committed { Ok(vmm) => { diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index d7eac12f7..950ae87f7 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -19,16 +19,16 @@ use crate::activation::{ReadyActivation, StoreGeneration, TransactionId}; use crate::host::artifacts; use crate::host::platform::Transport; use crate::host::process::{BootId, Process, ProcessRef}; -use crate::host::store::files::write_private; +use crate::host::store::files::{sync_dir, write_private}; use crate::host::store::{ CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, - Recovery, RuntimeIdentity, + Recovery, RuntimeIdentity, RuntimeRecord, }; use crate::host::supervisor::{self, Reattached, StopOutcome, Vmm, VmmCommand}; use crate::host::template::Retirement; use crate::host::template::clone::launch::{Hooks, Point}; -use crate::host::template::clone::reattach::reattach_clone_with; +use crate::host::template::clone::reattach::{reattach_clone_syncing, reattach_clone_with}; use crate::host::template::clone::records::{read_failure, read_progress, write_failure}; use crate::host::template::store::SNAPSHOT_ARTIFACTS; use crate::host::template::test_support::artifact; @@ -2743,8 +2743,9 @@ fn a_reattach_takes_the_outcome_report_of_a_vm_that_exited_as_it_first_read_it() TIMEOUT, |locked, runtime, exited| { supervisor::reattach_reporting(locked, runtime, |record| { - exited(record); + exited(record)?; fs::write(&outcome, b"not a report").unwrap(); + Ok(()) }) }, ) @@ -3140,3 +3141,210 @@ fn an_interrupted_launch_that_a_reattach_cannot_end_yet_keeps_its_lease_until_on assert!(control.capabilities.is_empty()); crashed.releases(&mut locked); } + +/// Returns the error of `reattached`, which must have failed. +fn reattach_error(reattached: crate::Result>) -> Error { + match reattached { + Err(error) => error, + Ok(other) => panic!("the reattach found {}", describe(&other)), + } +} + +/// Takes the clone of the sandbox of `locked` over, under `runtime`, through `control`, with +/// [`reattach_clone`], while a directory takes the place of the record of the clone's failure, as +/// on a storage that refuses to write it, once the reattach read the sandbox's records, and +/// returns the error of the reattach. +fn unrecorded_take_over( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut ReattachControl, +) -> Error { + let path = locked.clone_failure_path(); + let reattached = reattach_clone_with( + locked, + runtime, + control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + fs::create_dir(&path).unwrap(); + supervisor::reattach_reporting(locked, runtime, exited) + }, + ); + fs::remove_dir(&path).unwrap(); + reattach_error(reattached) +} + +#[test] +fn a_reattach_that_cannot_record_a_failure_ends_and_forgets_nothing_of_the_clone() { + let unrecorded = |error: &Error| { + let left = "; the reattach left the clone's VM and its records as it found them, for a \ + later reattach"; + let message = error.message(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + let start = "the failure could not be recorded before its teardown: "; + assert!( + message.starts_with(start) && message.ends_with(left), + "{error}" + ); + }; + let mut control = ReattachControl::answering(Some); + // A record of a failure that cannot be read, which the reattach would replace with its own + // before it ends the interrupted launch that the journal names. + let crashed = Crashed::at("activated"); + let mut locked = crashed.lock(); + let path = locked.clone_failure_path(); + fs::create_dir(&path).unwrap(); + let deadline = Instant::now() + TIMEOUT; + let runtime = &crashed.runtime; + let reattached = reattach_clone(&mut locked, runtime, &mut control, deadline, TIMEOUT); + unrecorded(&reattach_error(reattached)); + let journal = locked + .journal() + .unwrap() + .expect("the reattach kept the journal"); + assert!(Process::open(journal.process().unwrap()).unwrap().is_some()); + crashed.refuses_release(&mut locked); + // Once the record can be written, the reattach ends the launch, and records its failure. + fs::remove_dir(&path).unwrap(); + let failure = failed(crashed.take_over(&mut locked, runtime, &mut control)); + let (ready, killed) = ((Stage::Start, Progress::Ready), Evidence::ForcedKill); + let terminated = Teardown::Terminated; + assert_recorded(&failure, &crashed.lease, &locked, ready, killed, terminated); + crashed.releases(&mut locked); + // A VM whose launch committed, and that exited without a report of the guest's own end: the + // reattach would record that the clone lost its status before it clears the VM's record. + let crashed = Crashed::at("committed"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let record = locked.runtime().unwrap(); + assert!(record.is_some()); + unrecorded(&unrecorded_take_over( + &mut locked, + &crashed.runtime, + &mut control, + )); + assert_eq!(locked.runtime().unwrap(), record); + crashed.refuses_release(&mut locked); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (run, lost) = ((Stage::Run, Progress::Committed), Evidence::LostStatus); + assert_recorded( + &failure, + &crashed.lease, + &locked, + run, + lost, + Teardown::Exited, + ); + crashed.releases(&mut locked); + // A VM that runs under another runtime identity, and one whose Ready reports another + // activation than its record's: the reattach would end either once it recorded why. + let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); + let mut refusing = ReattachControl::answering(|_| Some(ReadyActivation { transaction: None })); + let reattach = (Stage::Reattach, Progress::Committed); + let cases = [ + (Some(&other), Evidence::RuntimeMismatch), + (None, Evidence::Protocol), + ]; + for (runtime, evidence) in cases { + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let (runtime, control) = match runtime { + Some(other) => (other, &mut control), + None => (&crashed.runtime, &mut refusing), + }; + unrecorded(&unrecorded_take_over(&mut locked, runtime, control)); + let record = locked + .runtime() + .unwrap() + .expect("the reattach kept the record"); + assert!( + Process::open(record.process()).unwrap().is_some(), + "{evidence:?}" + ); + crashed.refuses_release(&mut locked); + let failure = failed(crashed.take_over(&mut locked, runtime, control)); + let terminated = Teardown::Terminated; + assert_recorded( + &failure, + &crashed.lease, + &locked, + reattach, + evidence, + terminated, + ); + crashed.releases(&mut locked); + } +} + +#[test] +fn a_reattach_makes_the_record_of_an_earlier_failure_durable_before_it_ends_the_launch() { + // A failure of the clone's launch, whose record is in place, but whose teardown a crash of + // its caller interrupted, maybe once the sync of the record's directory failed. + let crashed = Crashed::at("activated"); + let mut locked = crashed.lock(); + let ready = (Stage::Start, Progress::Ready); + let error = Error::other("the test's launch failed"); + let at = (ready, transaction_of(&locked)); + let earlier = (Evidence::Protocol, Teardown::Pending); + record_earlier(&crashed.lease, &locked, at, earlier, &error); + let mut control = ReattachControl::answering(Some); + let reattach = |locked: &mut LockedStore, + runtime: &RuntimeIdentity, + exited: &dyn Fn(&RuntimeRecord) -> Result<()>| { + supervisor::reattach_reporting(locked, runtime, exited) + }; + // A reattach that cannot make the record durable ends nothing. + let refused = |_: &Path| -> Result<()> { Err(Error::io("the test's sync failed")) }; + let deadline = Instant::now() + TIMEOUT; + let runtime = &crashed.runtime; + let reattached = reattach_clone_syncing( + &mut locked, + runtime, + &mut control, + deadline, + TIMEOUT, + &refused, + reattach, + ); + let error = reattach_error(reattached); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + let refusal = "could not be made durable before its teardown: the test's sync failed"; + assert!(error.message().contains(refusal), "{error}"); + let journal = locked + .journal() + .unwrap() + .expect("the reattach kept the journal"); + assert!(Process::open(journal.process().unwrap()).unwrap().is_some()); + crashed.refuses_release(&mut locked); + // One that can syncs the sandbox's directory before it ends the launch. + let synced = RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced.borrow_mut().push(dir.to_owned()); + sync_dir(dir) + }; + let dir = locked.dir().to_owned(); + let reattached = reattach_clone_syncing( + &mut locked, + runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + &sync, + |locked, runtime, exited| { + assert_eq!(*synced.borrow(), std::slice::from_ref(&dir)); + reattach(locked, runtime, exited) + }, + ); + let failure = failed(reattached.unwrap()); + let (protocol, terminated) = (Evidence::Protocol, Teardown::Terminated); + assert_recorded( + &failure, + &crashed.lease, + &locked, + ready, + protocol, + terminated, + ); + crashed.releases(&mut locked); +} diff --git a/nvx-rs/src/host/template/clone/tests/restore.rs b/nvx-rs/src/host/template/clone/tests/restore.rs index 9e9c41dbc..753b735d8 100644 --- a/nvx-rs/src/host/template/clone/tests/restore.rs +++ b/nvx-rs/src/host/template/clone/tests/restore.rs @@ -1202,6 +1202,65 @@ fn a_caller_fails_a_ready_or_started_clone_with_its_own_evidence_as_the_sequence ); } +#[test] +fn a_failure_that_cannot_be_recorded_before_its_teardown_keeps_its_launch_for_a_reattach() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("unrecorded"); + let path = acquired.locked.clone_failure_path(); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + // The caller diagnoses a defect of the template once the clone is ready, but the record of + // its failure cannot be written, as on a full disk: a directory takes its place. + fs::create_dir(&path).unwrap(); + let defect = "the test diagnosed a defect of the generation's repair"; + let failure = activated.fail(Evidence::TemplateDefect, Error::other(defect)); + let ready = (Stage::Start, Progress::Ready); + let record = failure.record(); + assert_eq!( + ((record.stage, record.progress), record.evidence), + (ready, Evidence::TemplateDefect) + ); + // The caller holds the failure, the template's, whose teardown ended the clone's VMM through + // its handle, and kept the launch's journal, and with it the lease. + assert_eq!(failure.outcome(), Outcome::Template); + let Teardown::Unresolved(reason) = &record.teardown else { + panic!("the teardown of {failure} was resolved"); + }; + let unrecorded = "the failure could not be recorded before its teardown: "; + let ended = format!("the clone's VMM ended through its handle, but {unrecorded}"); + assert!(reason.starts_with(&ended), "{reason}"); + assert!(failure.message().starts_with(defect), "{failure}"); + let lost = "; the failure could not be recorded: "; + assert!(failure.message().contains(lost), "{failure}"); + assert_vmm_ended(&vmm); + let journal = acquired.locked.journal().unwrap(); + let journal = journal.expect("the launch kept its journal"); + assert_eq!(journal.process(), vmm.reference.as_ref()); + let refusal = fixture.store.release(&mut acquired.locked).unwrap_err(); + assert_eq!(refusal.code(), ErrorCode::SandboxRunning, "{refusal}"); + // Once the record can be written, a reattach, as after a crash of the caller, finds that the + // clone failed, though not how, rather than a clone that ended. + fs::remove_dir(&path).unwrap(); + let mut control = FakeControl::default(); + let reattached = reattach_clone( + &mut acquired.locked, + &runtime_of(&fixture), + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ) + .unwrap(); + let resolved = failed(reattached); + let exited = Teardown::Exited; + assert_failed(&resolved, &acquired, ready, Evidence::LostStatus, exited); + assert!(control.calls.is_empty()); + assert_launch_ended(&fixture, &mut acquired, &vmm); +} + #[test] fn a_diagnosis_too_long_for_a_record_keeps_its_ends_and_its_evidence_through_a_reattach() { let (fixture, image) = fixture(); From a06737dd23380905247ae66b945b4a1b7ee97ed7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 28 Oct 2026 21:30:17 -0700 Subject: [PATCH 12/12] nvx-rs: take the report of a clone's VM whose spawn no record kept A reattach read OpenVMM's outcome report of a clone's VM only once the clone's progress showed the spawn of its VMM, which the launch's journal records, and its progress log once the launch returns. A launch whose caller crashed after the spawn, before either recorded it, is one that a recovery of the sandbox, as supervisor::reattach runs it, ends once the VMM has exited, removing the journal: nothing recorded the spawn any more, and a later reattach_clone found the clone ended at its step launching, even when the guest had ended the VM with a status that is evidence of a failure, such as one of its repair, and recorded no failure. The launch removes the report of an earlier VM of the sandbox before it creates the clone's progress log, and CloneRecords::read takes a report as the clone's VM's from then on. The reattach now does too: exit_fault attributes the report's status once the clone got past its plan, ReattachedClone::Ended returns the report from that point, and a reattach that cannot read the record of a failure takes the teardown of a VM that left a report as an exit, whose status then prevails, as it does once a record keeps the spawn. Tests crash a launch once it began, write the report of the VMM that its journal does not record, and let a recovery of the sandbox end the launch: a reattach then fails the clone with the evidence and the status of the guest's repair, or finds it ended with the report of its VM, and one that cannot read the record of a failure takes the repair's status too. The case of a launch that never began no longer writes a report, which no VM of the clone could have left. The review of this pull request found this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- nvx-rs/README.md | 21 ++--- nvx-rs/src/host/template/clone/launch.rs | 4 +- nvx-rs/src/host/template/clone/reattach.rs | 41 ++++++--- .../src/host/template/clone/tests/launch.rs | 84 +++++++++++++++++-- 4 files changed, 119 insertions(+), 31 deletions(-) diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 07895889e..0214c8868 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -789,16 +789,17 @@ new session, in which the clone answers the byte-equal Activate of that another runtime identity launched, or that fails the check, through its verified identity; it records the failure of a launch that a crash interrupted, but not the journal that a commit may leave beside its runtime record, -classifies a VM that exited by its outcome report, records a failure before it -ends a VMM or clears the runtime record of one that exited, and fails, leaving -them as they were, if it cannot, finishes a failure that a crash left pending as -its teardown would have recorded it, keeping its transaction, resolves the -teardown of an earlier failure with what the VMM left as it ended, its outcome -report and OpenVMM's fatal error, attributes a status of the guest's as one that -a workload may have given once the clone's progress log is lost, damaged, or was -written during another boot of the host, whose crash may have lost its last -lines, and reads nothing of the template store, so that a tombstone or a -retirement of the generation leaves the clone its lease. +classifies a VM that exited by its outcome report, which is the clone's VM's +once its launch began, whether or not a crash left a record of its spawn, +records a failure before it ends a VMM or clears the runtime record of one that +exited, and fails, leaving them as they were, if it cannot, finishes a failure +that a crash left pending as its teardown would have recorded it, keeping its +transaction, resolves the teardown of an earlier failure with what the VMM left +as it ended, its outcome report and OpenVMM's fatal error, attributes a status +of the guest's as one that a workload may have given once the clone's progress +log is lost, damaged, or was written during another boot of the host, whose +crash may have lost its last lines, and reads nothing of the template store, so +that a tombstone or a retirement of the generation leaves the clone its lease. `CloneRecords::read` reads what the sandbox keeps of a clone, for the operations that list, inspect, or remove clones. diff --git a/nvx-rs/src/host/template/clone/launch.rs b/nvx-rs/src/host/template/clone/launch.rs index af4cbe510..77e12d5b3 100644 --- a/nvx-rs/src/host/template/clone/launch.rs +++ b/nvx-rs/src/host/template/clone/launch.rs @@ -371,8 +371,8 @@ impl ClonePlan { } }; // The progress log records the spawn before the launch returns: the journal records it too, - // but a teardown of the launch removes the journal, and a reattach reads the outcome report - // of the clone's VM only once its VMM spawned. + // but a teardown of the launch removes the journal, and a reattach then takes how far the + // clone got, and so the stage of its failure, from the log. if let Err(fault) = tracker.reach(Progress::Spawned) { return Err(tracker.fail(starting, Stage::Launch, fault)); } diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs index d6886d722..c11afa28b 100644 --- a/nvx-rs/src/host/template/clone/reattach.rs +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -47,12 +47,13 @@ pub enum ReattachedClone { report: Option, }, /// The clone failed, as the record of its failure in its sandbox tells, which the reattach - /// found, or wrote: the launch of a caller that crashed, which the reattach ended; a VM that - /// exited once its launch committed, with a status of the guest's that is evidence, or - /// without a report of the guest's own end; a check of the clone that failed; a VM that - /// runs under another runtime identity; or a record of a failure of the clone that cannot be - /// read, which the record that the reattach wrote replaces. The failure's teardown tells - /// whether a VM of the clone may still run: until it is resolved, the sandbox keeps the + /// found, or wrote: the launch of a caller that crashed, which the reattach ended, or which a + /// recovery of the sandbox ended before, whose VM ended with a status of the guest's that is + /// evidence; a VM that exited once its launch committed, with a status of the guest's that is + /// evidence, or without a report of the guest's own end; a check of the clone that failed; a + /// VM that runs under another runtime identity; or a record of a failure of the clone that + /// cannot be read, which the record that the reattach wrote replaces. The failure's teardown + /// tells whether a VM of the clone may still run: until it is resolved, the sandbox keeps the /// store's records of the VM, and with them the clone's lease, for a later reattach. Failed(CloneFailure), /// Whether the clone's VMM, whose launch committed, runs cannot be determined, for the reason @@ -93,6 +94,14 @@ pub enum ReattachedClone { /// a VMM that exited before its restore-ready event, attributes as the restore sequence does. One /// that it cannot end yet fails the same way with [`Evidence::LostStatus`], whose teardown is /// unresolved: its journal remains, and the lease with it, for a later reattach. +/// - A launch that a recovery of the sandbox ended before the reattach, as [`supervisor::reattach`] +/// ends one, and of which the store holds no record, fails at the stage of its progress, with +/// [`ErrorCode::Exited`], if OpenVMM's outcome report gives a status of the guest's that is +/// evidence, and is [`ReattachedClone::Ended`] otherwise. Once the launch began, which removes +/// the report of an earlier VM of the sandbox before it creates the clone's progress log, a +/// report that the sandbox holds is the clone's VM's, even if no record kept the spawn of its +/// VMM, as a crash of the launch's caller before the journal or the progress log recorded it +/// leaves it. /// - A VMM whose launch committed, and that exited, fails at [`Stage::Run`] with /// [`ErrorCode::Exited`] if OpenVMM's outcome report gives a status of the guest's that is /// evidence, as a status of the clone's repair is at any point, and with @@ -339,8 +348,15 @@ pub(super) fn reattach_clone_syncing( Teardown::Unresolved(_) | Teardown::Pending => Teardown::Exited, ref teardown => teardown.clone(), }, - // A VM of the clone ran only once its launch spawned it, as in `exit_fault`. - Err(_) if progress >= Progress::Spawned => Teardown::Exited, + // A VM of the clone ran once its launch spawned it, and so did one whose report the + // sandbox holds once the launch began, as in `exit_fault`, though no record kept + // its spawn. + Err(_) + if progress >= Progress::Spawned + || (progress > Progress::Planned && recorder.observed().is_some()) => + { + Teardown::Exited + } Err(_) => Teardown::NoVm, }, Reattached::Indeterminate(reason) => Teardown::Unresolved(reason), @@ -406,7 +422,7 @@ pub(super) fn reattach_clone_syncing( // A report is of the clone's VM only once its launch began, as in `exit_fault`. None => Ok(ReattachedClone::Ended { progress, - report: (progress >= Progress::Spawned) + report: (progress > Progress::Planned) .then(|| recorder.observed().cloned()) .flatten(), }), @@ -436,8 +452,11 @@ fn exit_fault( unheld: bool, ) -> Option<(Fault, Option)> { // A report is of the clone's VM only once its launch began, which removes the report of an - // earlier VM of the sandbox. - if progress < Progress::Spawned { + // earlier VM of the sandbox before it creates the clone's progress log, as + // `CloneRecords::read` takes it: a VMM that the launch spawned wrote it, though no record may + // keep the spawn, if the launch's caller crashed before the journal or the progress log + // recorded it, and a recovery of the sandbox then removed the journal. + if progress <= Progress::Planned { return None; } let (evidence, status) = recorder.reported(recorder.observed(), progress); diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index 950ae87f7..1cde8a7d1 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -1935,6 +1935,59 @@ fn a_launch_that_began_shows_the_report_of_a_vmm_that_its_journal_does_not_recor } } +#[test] +fn a_reattach_takes_the_report_of_a_vmm_whose_launch_a_recovery_of_the_sandbox_ended() { + // The caller crashed once the launch began, before the journal recorded the VMM that it + // spawned, which left the report, and a recovery of the sandbox's launch then removed the + // journal: nothing records the spawn, but the report is the clone's VM's, as an inspection + // takes it, so a reattach of the clone takes it too. + let recovered = |status: u8| { + let crashed = Crashed::at("begun"); + let mut locked = crashed.lock(); + fs::write(locked.outcome_path(), guest_exit(status)).unwrap(); + match crashed.reattach(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + assert!(locked.journal().unwrap().is_none()); + (crashed, locked) + }; + let mut control = ReattachControl::answering(Some); + // The status of the guest's repair is evidence of the clone's failure at any point. + let (crashed, mut locked) = recovered(195); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{failure}"); + assert_recorded( + &failure, + &crashed.lease, + &locked, + (Stage::Launch, Progress::Launching), + Evidence::RepairStatus, + Teardown::Exited, + ); + assert_eq!(failure.record().status, Some(195)); + crashed.releases(&mut locked); + // Another status is no evidence of a failure: the clone ended, with the report of its VM. + let (crashed, mut locked) = recovered(3); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Launching, + report: Some(report), + } => { + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(3))); + } + other => panic!("the reattach found {}", describe(&other)), + } + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert!(records.failure.is_none()); + crashed.releases(&mut locked); + assert!(control.capabilities.is_empty()); +} + #[test] fn a_launch_whose_caller_crashes_once_its_vmm_started_ends_the_vmm_as_reattach_recovers_it() { let crashed = Crashed::at("launched"); @@ -2430,17 +2483,10 @@ fn a_record_of_a_failure_that_cannot_be_read_is_a_failure_whatever_the_reattach_ crashed.releases(&mut locked); // A clone whose launch did not begin: the store holds no record of the launch that the // reattach would remove, so the record stays as it is, and a reattach after a crash of the - // caller finds the clone as this one does, with no VM, and takes no report of an earlier VM - // of the sandbox for the clone's. + // caller finds the clone as this one does, with no VM. let crashed = Crashed::at("prepared"); let mut locked = crashed.lock(); write_private(&locked.clone_failure_path(), b"not a record").unwrap(); - let earlier = serde_json::json!({ - "schema_version": 1, - "outcome": {"operation": "managed", "category": "guest-exit", "status_code": 195}, - "teardown": {"workers": true}, - }); - fs::write(locked.outcome_path(), serde_json::to_vec(&earlier).unwrap()).unwrap(); let mut control = ReattachControl::answering(Some); let crash = reattach_clone_with( &mut locked, @@ -2460,6 +2506,28 @@ fn a_record_of_a_failure_that_cannot_be_read_is_a_failure_whatever_the_reattach_ assert_recorded(&failure, lease, &locked, launching, host, no_vm); assert_eq!(failure.record().status, None); crashed.releases(&mut locked); + // A clone whose launch a recovery of the sandbox ended, once its caller crashed before the + // journal recorded the VMM that it spawned, which left the report: nothing records the spawn, + // but the report is the clone's VM's, since the launch removed any earlier VM's report before + // it recorded anything, and the status of the guest's repair prevails. + let crashed = Crashed::at("begun"); + let mut locked = crashed.lock(); + fs::write(locked.outcome_path(), guest_exit(195)).unwrap(); + match crashed.reattach(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + let repair = Evidence::RepairStatus; + assert_recorded(&failure, lease, &locked, launching, repair, exited); + assert_eq!(failure.record().status, Some(195)); + crashed.releases(&mut locked); } #[test]