diff --git a/nvx-rs/README.md b/nvx-rs/README.md index df06835a6..0214c8868 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -536,6 +536,10 @@ explicit handle list. event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. +`Vmm::stop_unless_exited` stops the VM the same way only if its VMM still runs +as the stop begins, and otherwise stops nothing and keeps the runtime record of +its launch, so that `reattach` finds that the VM exited without its holder. + `Vmm::terminate` takes that forced step alone, within the caller's bound, for a VM whose guest the caller must not ask anything, such as one of another runtime identity. @@ -678,11 +682,12 @@ store's names is refused. `ClonePlan::launch` then launches the clone as the launch that its lease reserves, once it has checked that the sandbox's clone binding records the -lease, and returns a `Restoring`: it writes the clone's activation record in its -canonical encoding, `clone-activation.bin`, from which a caller resends it -unchanged; it listens for the VMM's restore-ready event, at the socket -`restore.sock` on Linux, and at a named pipe of a random name on Windows; and it -runs `supervisor::launch` with OpenVMM's restore of the clone's devices, +lease, and that the sandbox records no failure of the clone, which fails once +under its lease, and returns a `Restoring`: it writes the clone's activation +record in its canonical encoding, `clone-activation.bin`, from which a caller +resends it unchanged; it listens for the VMM's restore-ready event, at the +socket `restore.sock` on Linux, and at a named pipe of a random name on Windows; +and it runs `supervisor::launch` with OpenVMM's restore of the clone's devices, `--restore-snapshot` with the generation's snapshot in place, which no clone copies or changes, so that clones of one generation restore it at once, `--restore-memory` with the memory target, and `--restore-ready-path`, under the @@ -728,6 +733,77 @@ before the commit leaves the launch's journal, whose recovery ends the VMM, while the lease lasts until the release, and a crash after it leaves a VM that reattaches, and keeps its lease while it runs. +A failure of a clone is a `CloneFailure`, whose `Evidence` attributes it to an +`Outcome`, at a `Stage`, once the clone got as far as its `Progress`: the +request, for a denied authorization, a stale binding, invalid targets, format, +settings, Activate, or mappings, or an image, a scratch, or a root that the +template cannot serve, and for a supplied file of the wrong identity, length, or +geometry, which the host's staging takes instead once the caller proves it, +`CloneFailure::with_staging_proven`; the host, for an endpoint that cannot be +bound, resource exhaustion, a host that cannot restore the template, another +runtime, or a failure of its own operations; unknown, for a lost status, a +timeout, a reset, a forced kill, or a broken protocol; the template, for a +malformed generation, or a defect of its capture or repair that a diagnosis +attributes to it, the only outcome that quarantines the generation, through its +tombstone, `CloneFailure::quarantine`; and the repair stage, for a status of the +clone's repair alone. + +OpenVMM's outcome report gives the status with which the guest ended the clone's +VM, which `status_evidence` attributes, and which prevails over what the host or +the caller saw, but not over evidence of the template, which shows the +provenance that a status alone leaves to establish: a status of the repair at +any point, and one of the activation before the start was sent. + +A VMM that exits before its restore-ready event, as OpenVMM does once it cannot +restore the snapshot, after it closed the event's endpoint, which the sequence +then gives ten seconds to exit, leaves the line with which OpenVMM reported its +fatal error in its log, which the failure's message keeps, and whose time ABI +code, if it says that the host cannot present the template's CPU profile, TSC, +or LAPIC timer, attributes the failure to the host. + +From the launch on, a failure ends the clone's VMM through its verified +identity, and the sandbox keeps the clone's progress log, `clone-progress.log`, +which records each step from the spawn to the commit, after a first line that +names the launch and the boot of the host, a few lines that it appends without a +sync, and the record of its failure, `clone-failure.json`, JSON in format +`FAILURE_FORMAT`, which it writes before the teardown, with the teardown +pending, so that a crash of the caller during the teardown leaves it for a +reattach to finish, and again once the teardown ended, with the stage, the +evidence and the status, the original outcome, the transaction, the generation, +the progress, and the teardown: no VM, an exit or a termination that the host +verified, or an unresolved one, which keeps the launch's journal or runtime +record, and the lease; a release removes both, and a sandbox that boots cold +writes neither. A failure that cannot be recorded before its teardown keeps the +launch's journal or runtime record, and the lease, so that a reattach still +finds that the clone failed. + +A step or a check of the caller's own that fails once the clone is ready, or +once its sandbox started, fails the clone with the caller's evidence through +`Activated::fail` or `Started::fail`, which end the launch and record the +failure as the sequence's own failures do. + +`reattach_clone` takes a clone over after the caller of its launch restarted: it +reattaches by the identity of its VMM's process, checks a committed clone in a +new session, in which the clone answers the byte-equal Activate of +`clone-activation.bin` with the Ready that it answered once, and ends a clone +that another runtime identity launched, or that fails the check, through its +verified identity; it records the failure of a launch that a crash interrupted, +but not the journal that a commit may leave beside its runtime record, +classifies a VM that exited by its outcome report, which is the clone's VM's +once its launch began, whether or not a crash left a record of its spawn, +records a failure before it ends a VMM or clears the runtime record of one that +exited, and fails, leaving them as they were, if it cannot, finishes a failure +that a crash left pending as its teardown would have recorded it, keeping its +transaction, resolves the teardown of an earlier failure with what the VMM left +as it ended, its outcome report and OpenVMM's fatal error, attributes a status +of the guest's as one that a workload may have given once the clone's progress +log is lost, damaged, or was written during another boot of the host, whose +crash may have lost its last lines, and reads nothing of the template store, so +that a tombstone or a retirement of the generation leaves the clone its lease. + +`CloneRecords::read` reads what the sandbox keeps of a clone, for the operations +that list, inspect, or remove clones. + `Store::tombstone` quarantines a generation, which takes no new lease while its leases keep working, and `Store::retire` deletes it once its leases have drained, crash-safely, removing the records of leases that no clone binding diff --git a/nvx-rs/src/host/platform/linux.rs b/nvx-rs/src/host/platform/linux.rs index eae6c4477..5c6027769 100644 --- a/nvx-rs/src/host/platform/linux.rs +++ b/nvx-rs/src/host/platform/linux.rs @@ -784,6 +784,17 @@ impl Directory { pub(crate) fn sync(&self) -> io::Result<()> { self.0.sync_all() } + + /// Returns why the directory cannot hold a sandbox store, as [`check_store_dir`] returns it for + /// a path, if it cannot: unless it belongs to the current user with mode 0700. + pub(crate) fn store_dir_problem(&self) -> io::Result> { + let metadata = self.0.metadata()?; + Ok(store_dir_problem( + metadata.uid(), + metadata.mode(), + sys::effective_ids().0, + )) + } } /// Claims a launch's fresh log, which `file` holds open. @@ -868,7 +879,8 @@ pub(crate) struct ReadyListener { /// Listens at the Unix socket `socket_path`, in place of a socket that an earlier launch left /// there, as the restore-ready endpoint of a clone's launch, and returns the endpoint, as -/// OpenVMM's `--restore-ready-path` takes it, and its listener. +/// OpenVMM's `--restore-ready-path` takes it, and its listener. The socket has mode 0600, +/// whatever the umask, as [`set_ready_socket_mode`] gives it. pub(crate) fn listen_ready(socket_path: &Path) -> io::Result<(String, ReadyListener)> { let endpoint = socket_path .to_str() @@ -879,6 +891,10 @@ pub(crate) fn listen_ready(socket_path: &Path) -> io::Result<(String, ReadyListe _ => {} } let socket = UnixListener::bind(socket_path)?; + // The umask may clear bits of the mode that the bind gives the socket, even its owner's + // permission to write it, without which a VMM that is not root cannot connect: the mode is + // set again before any VMM may connect. + set_ready_socket_mode(socket_path)?; // The host polls the socket before it accepts, so an accept never blocks. socket.set_nonblocking(true)?; let listener = ReadyListener { @@ -965,6 +981,35 @@ impl ReadyListener { } } +/// Gives the restore-ready socket at `socket_path`, which the caller bound, the mode 0600, whatever +/// the umask: its owner, the user of the clone's VMM, may connect to it, which takes the +/// permission to write it, and no one else may. +/// +/// The socket is opened only to locate it, following no link at its name, and refused unless it +/// is a socket. +fn set_ready_socket_mode(socket_path: &Path) -> io::Result<()> { + use std::os::unix::fs::FileTypeExt; + + use nix::fcntl::OFlag; + + let socket = OpenOptions::new() + .read(true) + .custom_flags((OFlag::O_PATH | OFlag::O_NOFOLLOW).bits()) + .open(socket_path)?; + if !socket.metadata()?.file_type().is_socket() { + return Err(io::Error::other(format!( + "{} is not the restore-ready socket that the launch bound", + socket_path.display() + ))); + } + // A descriptor that only locates the socket cannot change its mode itself, but its link in + // /proc names exactly the socket that it opened. + fs::set_permissions( + Path::new("/proc/self/fd").join(socket.as_raw_fd().to_string()), + Permissions::from_mode(0o600), + ) +} + /// Connects to a control endpoint served by the process `expected_pid`. /// /// The connection attempt never blocks: a listener whose backlog is full yields @@ -1110,4 +1155,65 @@ mod tests { assert!(elapsed < Duration::from_secs(1), "{elapsed:?}"); assert!(process.wait(Instant::now()).unwrap().unwrap()); } + + #[test] + fn a_restore_ready_socket_lets_its_owner_connect_whatever_the_umask() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("ready.sock"); + // A umask that clears even the owner's permission to write what it creates, in a thread + // of the test's own, whose umask no other thread shares. + let (endpoint, _listener) = thread::scope(|scope| { + scope + .spawn(|| { + nix::sched::unshare(nix::sched::CloneFlags::CLONE_FS).unwrap(); + nix::sys::stat::umask(nix::sys::stat::Mode::from_bits_truncate(0o277)); + listen_ready(&path) + }) + .join() + .unwrap() + }) + .unwrap(); + let mode = fs::symlink_metadata(&path).unwrap().permissions().mode(); + assert_eq!(mode & PERMISSION_BITS, 0o600, "{mode:o}"); + UnixStream::connect(&endpoint).unwrap(); + } + + #[test] + fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() { + let pid = std::process::id(); + let receive_all = |listener: &mut ReadyListener| { + let mut received = Vec::new(); + let deadline = Instant::now() + Duration::from_secs(10); + while listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + { + assert!(Instant::now() < deadline, "the endpoint did not close"); + } + received + }; + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("ready.sock"); + // A client that sent nothing, as a VMM that fails as it starts, and one that sent its + // event, each closed before the listener accepted their connection: what they sent + // remains to read, and then the end of the connection. + for sent in [&b""[..], b"the event\n"] { + let (endpoint, mut listener) = listen_ready(&path).unwrap(); + let mut client = UnixStream::connect(&endpoint).unwrap(); + std::io::Write::write_all(&mut client, sent).unwrap(); + drop(client); + assert_eq!(receive_all(&mut listener), sent); + } + // A client that closes once the listener accepted its connection. + let (endpoint, mut listener) = listen_ready(&path).unwrap(); + let client = UnixStream::connect(&endpoint).unwrap(); + let mut received = Vec::new(); + assert!( + listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + ); + drop(client); + assert_eq!(receive_all(&mut listener), b""); + } } diff --git a/nvx-rs/src/host/platform/windows.rs b/nvx-rs/src/host/platform/windows.rs index f249d9343..56c051bd2 100644 --- a/nvx-rs/src/host/platform/windows.rs +++ b/nvx-rs/src/host/platform/windows.rs @@ -2137,7 +2137,7 @@ pub(crate) fn interrupted() -> bool { #[cfg(test)] mod tests { - use std::io::Read; + use std::io::{Read, Write}; use super::*; @@ -2165,6 +2165,43 @@ mod tests { end_process(false, 7, || panic!("terminated"), || Ok(true)).unwrap(); } + #[test] + fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() { + let pid = std::process::id(); + let receive_all = |listener: &mut ReadyListener| { + let mut received = Vec::new(); + let deadline = Instant::now() + Duration::from_secs(10); + while listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + { + assert!(Instant::now() < deadline, "the endpoint did not close"); + } + received + }; + // A client that sent nothing, as a VMM that fails as it starts, and one that sent its + // event, each closed before the listener accepted their connection: what they sent + // remains to read, and then the end of the pipe. + for sent in [&b""[..], b"the event\n"] { + let (endpoint, mut listener) = listen_ready(Path::new("unused")).unwrap(); + let mut client = fs::OpenOptions::new().write(true).open(&endpoint).unwrap(); + client.write_all(sent).unwrap(); + drop(client); + assert_eq!(receive_all(&mut listener), sent); + } + // A client that closes once the listener accepted its connection. + let (endpoint, mut listener) = listen_ready(Path::new("unused")).unwrap(); + let client = fs::OpenOptions::new().write(true).open(&endpoint).unwrap(); + let mut received = Vec::new(); + assert!( + listener + .receive(pid, &mut received, 64, Duration::from_millis(50)) + .unwrap() + ); + drop(client); + assert_eq!(receive_all(&mut listener), b""); + } + #[test] fn windows_refuses_to_terminate_a_process_that_exited_with_access_denied() { let mut child = std::process::Command::new("ping") diff --git a/nvx-rs/src/host/sandbox/distro.rs b/nvx-rs/src/host/sandbox/distro.rs index 0774bddc1..a3dbe7663 100644 --- a/nvx-rs/src/host/sandbox/distro.rs +++ b/nvx-rs/src/host/sandbox/distro.rs @@ -337,6 +337,12 @@ impl DistroBinding { self.policy } + /// Returns the binding's version, which a [check](Self::check) requires to be + /// [`BINDING_VERSION`]. + pub(crate) fn version(&self) -> u32 { + self.version + } + /// Returns the binding of `layer`, with the seals of its image and manifest and `policy`. fn new( layer: &SandboxLayer, diff --git a/nvx-rs/src/host/sandbox/scratch.rs b/nvx-rs/src/host/sandbox/scratch.rs index ccea75636..d69cfe4e5 100644 --- a/nvx-rs/src/host/sandbox/scratch.rs +++ b/nvx-rs/src/host/sandbox/scratch.rs @@ -257,6 +257,12 @@ impl ScratchBinding { self.length } + /// Returns the binding's version, which a [check](Self::check) requires to be + /// [`SCRATCH_BINDING_VERSION`]. + pub(crate) fn version(&self) -> u32 { + self.version + } + /// Returns the binding of the image that `seal` seals. fn of(seal: &FileSeal) -> Self { Self { diff --git a/nvx-rs/src/host/store/lifecycle/tests.rs b/nvx-rs/src/host/store/lifecycle/tests.rs index ee248e2f6..9b09876a8 100644 --- a/nvx-rs/src/host/store/lifecycle/tests.rs +++ b/nvx-rs/src/host/store/lifecycle/tests.rs @@ -11,8 +11,8 @@ use std::time::Duration; use super::*; use crate::host::store::{ - CAPABILITY_NAME, CLONE_ACTIVATION_NAME, CLONE_SCRATCH_NAME, CONSOLE_NAME, LOG_NAME, - OUTCOME_NAME, RESTORE_READY_NAME, SOCKET_NAME, + CAPABILITY_NAME, CLONE_ACTIVATION_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, + CLONE_SCRATCH_NAME, CONSOLE_NAME, LOG_NAME, OUTCOME_NAME, RESTORE_READY_NAME, SOCKET_NAME, }; use crate::host::test_support::released; @@ -326,6 +326,8 @@ fn a_removal_removes_the_store_the_callers_files_and_the_directory() { CLONE_BINDING_NAME, CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, BOOT, ".runtime.json.12345.tmp", @@ -417,6 +419,8 @@ fn write_clone_files(dir: &Path) { CLONE_BINDING_NAME, CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ] { fs::write(dir.join(name), b"{}").unwrap(); @@ -536,6 +540,8 @@ fn linked_entries(more: &[&str]) -> Vec { let mut names = vec![ CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, + CLONE_FAILURE_NAME, + CLONE_PROGRESS_NAME, CLONE_SCRATCH_NAME, CONFIG, LOCK_NAME, diff --git a/nvx-rs/src/host/store/mod.rs b/nvx-rs/src/host/store/mod.rs index d6a66e3c8..45833869c 100644 --- a/nvx-rs/src/host/store/mod.rs +++ b/nvx-rs/src/host/store/mod.rs @@ -19,6 +19,8 @@ //! clone-scratch.raw the file of the scratch device of a clone whose upper is in RAM, a //! sparse file that the clone's plan creates //! clone-activation.bin the activation record of a clone, which its launch writes +//! clone-progress.log the progress log of a clone's launch and restore sequence +//! clone-failure.json the record of a clone's failure //! restore.sock the endpoint at which the launch of a clone awaits the restore-ready //! event of its VMM, on Linux //! ... the caller's configuration record and other files, which it declares @@ -316,19 +318,34 @@ pub const CLONE_ACTIVATION_NAME: &str = "clone-activation.bin"; /// random. [`template::Store::release`](crate::host::template::Store::release) removes it once the /// clone's VM has exited, as a [removal](self#removal) of the sandbox does. pub const RESTORE_READY_NAME: &str = "restore.sock"; +/// Name of the progress log of a clone's launch, which the launch, +/// [`template::ClonePlan::launch`](crate::host::template::ClonePlan::launch), creates, and to which +/// it and the clone's restore sequence append a line for each step that the clone reaches, without +/// syncing it, so that a reattach tells how far the clone got once the caller of its launch +/// crashed, and which [`template::Store::release`](crate::host::template::Store::release) removes, +/// as a [removal](self#removal) of the sandbox does. No other launch writes it. +pub const CLONE_PROGRESS_NAME: &str = "clone-progress.log"; +/// Name of the record of a clone's failure, which a failure of the clone from its launch on +/// writes before its teardown, and again once its teardown is known, as +/// [`template::CloneFailure`](crate::host::template::CloneFailure) describes, and which +/// [`template::Store::release`](crate::host::template::Store::release) removes, as a +/// [removal](self#removal) of the sandbox does. +pub const CLONE_FAILURE_NAME: &str = "clone-failure.json"; /// The files that the clone of a sandbox keeps of its launch, which a release removes once the /// clone's VM has exited, as a [removal](self#removal) of the sandbox does, before the clone /// binding. -const CLONE_FILE_NAMES: [&str; 3] = [ +const CLONE_FILE_NAMES: [&str; 5] = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, RESTORE_READY_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, ]; /// Every file that the store keeps in a sandbox's directory. Replacing a record leaves a /// temporary file named `.NAME.NUMBER.tmp` beside it only if the replacement was interrupted, and /// a VMM that is killed while it writes its outcome report leaves the file in which it stages it; /// see [the report staging file](self#the-report-staging-file). -pub const FILE_NAMES: [&str; 12] = [ +pub const FILE_NAMES: [&str; 14] = [ LOCK_NAME, JOURNAL_NAME, RUNTIME_NAME, @@ -341,6 +358,8 @@ pub const FILE_NAMES: [&str; 12] = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, RESTORE_READY_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, ]; /// Interval at which a lock that another holder has is tried again: the sandbox's lock, by a @@ -550,6 +569,16 @@ impl SandboxStore { self.path(RESTORE_READY_NAME) } + /// Returns the path of the progress log of a clone's launch, [`CLONE_PROGRESS_NAME`]. + pub fn clone_progress_path(&self) -> PathBuf { + self.path(CLONE_PROGRESS_NAME) + } + + /// Returns the path of the record of a clone's failure, [`CLONE_FAILURE_NAME`]. + pub fn clone_failure_path(&self) -> PathBuf { + self.path(CLONE_FAILURE_NAME) + } + /// Reads the runtime record, which names the sandbox's VM from the commit of its launch until /// its runtime is cleared. pub fn runtime(&self) -> Result> { @@ -807,8 +836,9 @@ impl LockedStore { /// Removes what the clone of the sandbox kept of its launch in the sandbox's directory, whose /// VM has exited, through `directory`, the handle of the directory that /// [`open_dir`](Self::open_dir) returned: the file of its scratch device, - /// [`CLONE_SCRATCH_NAME`], its activation record, [`CLONE_ACTIVATION_NAME`], and the socket of - /// its restore-ready event, [`RESTORE_READY_NAME`]. A release, as a [removal](self#removal) + /// [`CLONE_SCRATCH_NAME`], its activation record, [`CLONE_ACTIVATION_NAME`], the socket of its + /// restore-ready event, [`RESTORE_READY_NAME`], its progress log, [`CLONE_PROGRESS_NAME`], and + /// the record of its failure, [`CLONE_FAILURE_NAME`]. A release, as a [removal](self#removal) /// does, removes them before the clone binding, whose removal, once durable, makes theirs /// durable too, since all are entries of the sandbox's directory. A file whose deletion is /// pending, as a Windows file system without POSIX deletion leaves a removed file that another diff --git a/nvx-rs/src/host/store/regular.rs b/nvx-rs/src/host/store/regular.rs index e85f2845b..da503f91b 100644 --- a/nvx-rs/src/host/store/regular.rs +++ b/nvx-rs/src/host/store/regular.rs @@ -2,6 +2,7 @@ //! binding: regular files that are not links, which they read within a bound, and the identity of //! a file that a template module holds open. +use std::fmt; use std::fs::{File, OpenOptions}; use std::io::{self, Read}; use std::path::Path; @@ -9,9 +10,29 @@ use std::path::Path; use crate::host::seal::FileSeal; use crate::host::{filesystem, platform}; +/// Why a read of this module refused a file for what the file is, as [`is_refused`] tells. +#[derive(Debug)] +struct Refused(String); + +impl fmt::Display for Refused { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&self.0) + } +} + +impl std::error::Error for Refused {} + +/// Returns whether `error` is one with which a read of this module refused a file for what the +/// file is, rather than an operation of the host's that failed: a link, or a file that is not a +/// regular file, which [`open_regular`] and [`open_regular_in`] refuse, and a file that holds +/// more than its bound, which [`read_at_most`] refuses. +pub(crate) fn is_refused(error: &io::Error) -> bool { + error.get_ref().is_some_and(|inner| inner.is::()) +} + /// Returns the error of a file that is a link, or not a regular file. fn not_regular() -> io::Error { - io::Error::other("it is a link, or not a regular file") + io::Error::other(Refused("it is a link, or not a regular file".to_owned())) } /// Returns whether `error`, with which an open or a removal of the file at `path` failed, means @@ -99,9 +120,9 @@ pub(crate) fn read_at_most(file: File, limit: u64, what: &str) -> io::Result limit { - return Err(io::Error::other(format!( + return Err(io::Error::other(Refused(format!( "it holds more than the {limit} bytes that {what} may hold" - ))); + )))); } Ok(bytes) } diff --git a/nvx-rs/src/host/store/tests.rs b/nvx-rs/src/host/store/tests.rs index ef11598a0..12ddc19ea 100644 --- a/nvx-rs/src/host/store/tests.rs +++ b/nvx-rs/src/host/store/tests.rs @@ -410,6 +410,8 @@ fn the_files_that_a_clone_kept_of_its_launch_go_and_nothing_else() { for path in [ store.clone_scratch_path(), store.clone_activation_path(), + store.clone_progress_path(), + store.clone_failure_path(), store.restore_ready_path(), store.log_path(), store.binding_path(), diff --git a/nvx-rs/src/host/supervisor/mod.rs b/nvx-rs/src/host/supervisor/mod.rs index 5d1b81005..a364a49f6 100644 --- a/nvx-rs/src/host/supervisor/mod.rs +++ b/nvx-rs/src/host/supervisor/mod.rs @@ -86,6 +86,13 @@ //! [`Vmm::terminate`] takes that forced step alone, with no graceful step: it ends a VM whose //! guest the caller cannot ask to shut down, such as one that another runtime identity started. //! +//! A stop decides by its own check whether the VMM still runs as it begins. [`Vmm::stop`] ends a +//! VM whose VMM had exited by then as one that it stopped, and clears the runtime record of its +//! launch. [`Vmm::stop_unless_exited`] stops nothing then, and keeps the record, so that +//! [`reattach`] finds that the VM exited without its holder, as a caller that tells how the VM +//! ended, such as a clone's, needs: a holder that checked whether the VMM had exited, then stopped +//! the VM, would otherwise attribute to its stop a VMM that exited in between. +//! //! # Reattach //! //! After an orchestrator restarts within the same host boot, [`reattach`] ends an interrupted @@ -130,7 +137,11 @@ //! session, and checks its Ready, by a deadline that runs from the launch; //! [`Activated::start`](crate::host::template::Activated::start) starts its sandbox; and //! [`Started::commit`](crate::host::template::Started::commit) commits the launch, as -//! [`Starting::commit`] does. A failure ends the launch, as [`Starting::abandon`] does. +//! [`Starting::commit`] does. A failure ends the launch, as [`Starting::abandon`] does, through +//! the VMM's verified identity, and the sandbox records it until the release, with the evidence +//! that attributes it, as a [`CloneFailure`](crate::host::template::CloneFailure) describes: a +//! launch that cannot be ended keeps its journal, and the lease, and so does one whose failure +//! cannot be recorded before its end, whose VMM is ended through its handle. //! 4. Once the clone's VM has exited, as a stop, a termination, or a reattach verifies, //! [`Store::release`](crate::host::template::Store::release) ends the lease, under the //! sandbox's lock, taken through the lease's path, @@ -174,6 +185,12 @@ //! - During a release, the binding goes first, durably, and leaves the record of the lease, which //! a retirement removes. //! +//! The caller of a clone's launch that restarts takes the clone over with +//! [`reattach_clone`](crate::host::template::reattach_clone), which reattaches as [`reattach`] +//! does, records the failure of a launch that it ended, checks the activation of a clone whose +//! launch committed in a new session, and ends a clone that another runtime identity launched, or +//! that fails the check, through its verified identity. +//! //! The evidence of the VM's exit decides each outcome. A record of an earlier boot, or of a process //! ID that another process now has, names a VM that has exited: reattach clears it, and the lease //! may be released. A VM that cannot be examined is indeterminate, and keeps its records, and so @@ -222,31 +239,85 @@ pub fn launch<'s, 'h>( capability: &[u8; CAPABILITY_LEN], command: impl FnOnce(&str) -> Result>, ) -> Result> { + launch_reporting( + locked, + reserved, + runtime, + capability, + command, + |_, _| Ok(()), + ) + .map_err(|failure| failure.error) +} + +/// How a launch of [`launch_reporting`] failed. +#[derive(Debug)] +pub(crate) struct LaunchFailure { + /// The error of the step that failed, which says why the launch could not be ended if it + /// could not. + pub(crate) error: Error, + /// Whether the launch spawned its VMM, which it then ended through the VMM's handle. + pub(crate) spawned: bool, + /// Why the launch was not ended, whose journal then remains for [`LockedStore::recover`]: + /// it could not be, or its caller could not record its failure first. `None` once it ended. + pub(crate) unended: Option, +} + +/// Launches a VMM as [`launch`] does, and tells how a launch that fails ended, as a +/// [`LaunchFailure`]. A launch that began, and fails, calls `failing` with the step's error, and +/// whether it spawned the VMM, before it ends the launch, so that the caller may record the +/// failure, as a clone's launch does, before anything of the launch ends. If `failing` fails, the +/// launch ends a VMM that it spawned through the VMM's handle, as always, but keeps its journal, +/// as a crash would leave it, for [`LockedStore::recover`]: the journal then still tells that the +/// launch began, though the caller could not record its failure. +pub(crate) fn launch_reporting<'s, 'h>( + locked: &'s mut LockedStore, + reserved: &LaunchId, + runtime: &RuntimeIdentity, + capability: &[u8; CAPABILITY_LEN], + command: impl FnOnce(&str) -> Result>, + failing: impl FnOnce(&Error, bool) -> Result<()>, +) -> std::result::Result, LaunchFailure> { let store = SandboxStore::clone(locked); - let mut launch = locked.begin_launch(reserved, runtime, capability)?; + // A launch that does not begin leaves nothing to end. + let mut launch = locked + .begin_launch(reserved, runtime, capability) + .map_err(|error| LaunchFailure { + error, + spawned: false, + unended: None, + })?; let command = match command(launch.endpoint()) { Ok(command) => command, - Err(error) => return Err(abandon_after(launch, error)), + Err(error) => { + let recorded = failing(&error, false); + return Err(abandon_after(launch, error, false, recorded)); + } }; let log = match launch.take_log() { Ok(log) => log, - Err(error) => return Err(abandon_after(launch, error)), + Err(error) => { + let recorded = failing(&error, false); + return Err(abandon_after(launch, error, false, recorded)); + } }; let child = match spawn(&command, capability, log) { Ok(child) => child, Err(error) => { let error = Error::io(format!("cannot launch {}", command.program.display())) .with_source(error); - return Err(abandon_after(launch, error)); + let recorded = failing(&error, false); + return Err(abandon_after(launch, error, false, recorded)); } }; let process = match launch.record_process(&child) { Ok(process) => process, Err(error) => { + let recorded = failing(&error, true); // An unrecorded VMM holds the claim of its launch, so only its handle can end it, // before the launch ends. child.kill_and_wait(); - return Err(abandon_after(launch, error)); + return Err(abandon_after(launch, error, true, recorded)); } }; Ok(Starting { @@ -259,22 +330,46 @@ pub fn launch<'s, 'h>( }) } -/// Ends `launch`, which failed with `error`, and returns `error`, which says why the launch could -/// not be ended if it could not: its journal then remains for [`LockedStore::recover`]. -fn abandon_after(launch: Launch<'_>, error: Error) -> Error { - let reason = match launch.abandon() { - Ok(Recovery::Indeterminate(reason)) => reason, - Ok(_) => return error, - Err(failure) => failure.to_string(), +/// Ends `launch`, which failed with `error`, after it `spawned` a VMM or not, once its caller +/// `recorded` the failure, and returns how it failed, whose error says why the launch was not +/// ended if it was not: its journal then remains for [`LockedStore::recover`]. A launch whose +/// caller could not record its failure keeps its journal, as a crash would leave it, and ends no +/// more of it than its caller did. +fn abandon_after( + launch: Launch<'_>, + error: Error, + spawned: bool, + recorded: Result<()>, +) -> LaunchFailure { + let reason = match recorded { + Err(unrecorded) => { + drop(launch); + unrecorded.to_string() + } + Ok(()) => match launch.abandon() { + Ok(Recovery::Indeterminate(reason)) => reason, + Ok(_) => { + return LaunchFailure { + error, + spawned, + unended: None, + }; + } + Err(failure) => failure.to_string(), + }, }; let (kind, message, source) = error.into_parts(); let error = Error::new( kind, format!("{message}; the launch could not be ended, and remains for recovery: {reason}"), ); - match source { - Some(source) => error.with_source(source), - None => error, + LaunchFailure { + error: match source { + Some(source) => error.with_source(source), + None => error, + }, + spawned, + unended: Some(reason), } } @@ -367,7 +462,15 @@ impl<'s> Starting<'s> { /// /// Fails if the runtime record cannot be written. The VMM is then ended through its handle, /// and the journal that remains shows [`LockedStore::recover`] that it exited. - pub fn commit(mut self) -> Result { + pub fn commit(self) -> Result { + self.commit_reporting(|_| {}) + } + + /// Commits the launch as [`commit`](Self::commit) does, and calls `failing` with the error of + /// a commit that fails, before it ends the VMM, so that the caller may record the failure + /// first, as a clone's launch does. The journal remains all the same, whether or not the + /// caller could. + pub(crate) fn commit_reporting(mut self, failing: impl FnOnce(&Error)) -> Result { let StartingInner { launch, child, @@ -383,6 +486,7 @@ impl<'s> Starting<'s> { }) } Err(error) => { + failing(&error); // No runtime record names the VM, so its VMM must not run on. child.kill_and_wait(); Err(error) @@ -401,6 +505,25 @@ impl<'s> Starting<'s> { .expect("a launch is abandoned once") .abandon() } + + /// Ends the launch's VMM through its handle, and keeps the launch's journal, as a crash of the + /// caller would leave it, for [`LockedStore::recover`], which ends the launch once it finds + /// the VMM exited. Returns whether the VMM exited, as [`Launched::kill_and_wait`] does. + /// + /// It ends a launch whose caller could not record its failure before the launch's end, as a + /// clone's launch records it, so that the journal, which keeps the clone's lease, still tells + /// that the launch began. + pub(crate) fn end_keeping_journal(mut self) -> bool { + let StartingInner { + launch, + child, + process: _, + } = self.inner.take().expect("a launch is ended once"); + let exited = child.kill_and_wait(); + // A launch that is dropped keeps its journal. + drop(launch); + exited + } } impl Drop for Starting<'_> { @@ -470,7 +593,10 @@ impl Vmm { /// The result says whether the VMM exited without the forced step, the forced step ended it, /// or it still runs; see [`StopOutcome`]. Once the VMM has exited, the stop clears the runtime /// record of its launch, as [`LockedStore::clear_runtime`] does, unless the record is another - /// VM's by then. A VMM that still runs keeps its record. + /// VM's by then. A VMM that still runs keeps its record. One that had exited before the stop + /// began, which runs no step, ends as one that the stop ended, as [`StopOutcome::Graceful`], + /// and its record is cleared: [`stop_unless_exited`](Self::stop_unless_exited) tells such a + /// VMM apart, and keeps its record. /// /// `locked` must be the locked store of the VM's sandbox. Fails with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if `forced` is zero or longer @@ -484,19 +610,33 @@ impl Vmm { forced: Option, graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, ) -> Result> { - self.check_store(locked)?; - if let Some(bound) = forced { - check_bound(bound, "the forced step")?; - } - let stopped = stop_process( - &self.process, - self.record.process().pid, - deadline, - forced, - || graceful(self, deadline), - )?; - self.clear_if_exited(locked, stopped.outcome)?; - Ok(stopped) + self.stop_with(locked, deadline, forced, graceful, &self.process) + } + + /// Stops the VM as [`stop`](Self::stop) does, unless its VMM had exited before the stop began, + /// and returns `None` then, with the runtime record of its launch kept. + /// + /// The stop decides by its own check of the VMM, before its graceful step, as + /// [`stop`](Self::stop) does. A VMM that had exited by then ended without its holder, however + /// recently the holder found it running: the stop runs no step, signals nothing, and clears + /// nothing, so that the record tells [`reattach`] that the VM exited unheld, as one that exits + /// while no caller holds it does. A caller that tells how the VM ended then reattaches, as + /// [`reattach_clone`](crate::host::template::reattach_clone) does, which records a clone's + /// failure, with the loss of its status if no report of the guest's own end tells why its VM + /// exited, before it clears the record. [`stop`](Self::stop) ends such a VM as one that it + /// stopped, and clears its record, so a holder that checks whether the VMM has exited, then + /// stops the VM with it, attributes to its stop a VMM that exits in between. + /// + /// A VMM that runs as the stop begins is stopped as [`stop`](Self::stop) stops it, with the + /// same result, and its exit is the stop's from then on. Fails as [`stop`](Self::stop) fails. + pub fn stop_unless_exited( + &self, + locked: &mut LockedStore, + deadline: Instant, + forced: Option, + graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, + ) -> Result>> { + self.stop_unless_exited_with(locked, deadline, forced, graceful, &self.process) } /// Ends the VM by identity-verified forced termination: terminates its VMM through its @@ -525,6 +665,51 @@ impl Vmm { self.terminate_with(locked, bound, &self.process) } + /// Stops the VM as [`stop`](Self::stop) does, through `process`: its VMM, or, in tests, a + /// stand-in for it. + fn stop_with( + &self, + locked: &mut LockedStore, + deadline: Instant, + forced: Option, + graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, + process: &impl Stoppable, + ) -> Result> { + if let Some(stopped) = + self.stop_unless_exited_with(locked, deadline, forced, graceful, process)? + { + return Ok(stopped); + } + self.clear_if_exited(locked, StopOutcome::Graceful)?; + Ok(Stopped { + outcome: StopOutcome::Graceful, + graceful_error: None, + }) + } + + /// Stops the VM as [`stop_unless_exited`](Self::stop_unless_exited) does, through `process`: + /// its VMM, or, in tests, a stand-in for it. + fn stop_unless_exited_with( + &self, + locked: &mut LockedStore, + deadline: Instant, + forced: Option, + graceful: impl FnOnce(&Self, Instant) -> std::result::Result<(), E>, + process: &impl Stoppable, + ) -> Result>> { + self.check_store(locked)?; + if let Some(bound) = forced { + check_bound(bound, "the forced step")?; + } + let stopped = stop_process(process, self.record.process().pid, deadline, forced, || { + graceful(self, deadline) + })?; + if let Some(stopped) = &stopped { + self.clear_if_exited(locked, stopped.outcome)?; + } + Ok(stopped) + } + /// Ends the VM as [`terminate`](Self::terminate) does, through `process`: its VMM, or, in /// tests, a stand-in for it. fn terminate_with( @@ -576,11 +761,11 @@ fn check_bound(bound: Duration, what: &str) -> Result<()> { } } -/// How [`Vmm::stop`] or [`Vmm::terminate`] ended. +/// How [`Vmm::stop`], [`Vmm::stop_unless_exited`], or [`Vmm::terminate`] ended. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum StopOutcome { /// The VMM exited without the forced step: by the deadline of the graceful step, or before - /// the stop or the termination began. + /// the stop or the termination began, which [`Vmm::stop_unless_exited`] reports as no stop. Graceful, /// The forced step terminated the VMM through its verified handle, and it exited within the /// step's bound: the graceful step of a stop did not end it, or a termination took the step @@ -592,8 +777,8 @@ pub enum StopOutcome { TimedOut, } -/// What [`Vmm::stop`] did: how the stop ended, and the failure of its graceful step, if it -/// failed. +/// What [`Vmm::stop`] or [`Vmm::stop_unless_exited`] did: how the stop ended, and the failure of +/// its graceful step, if it failed. #[derive(Debug)] pub struct Stopped { /// How the stop ended. @@ -629,23 +814,21 @@ impl Stoppable for Process { } /// Stops the VMM `process`, whose ID is `pid`, with the graceful step `graceful`, the deadline -/// `deadline`, and the forced step that `forced` requests, as [`Vmm::stop`] describes, without -/// clearing its runtime record. +/// `deadline`, and the forced step that `forced` requests, as [`Vmm::stop_unless_exited`] +/// describes, without clearing its runtime record: returns `None`, and takes no step, if the VMM +/// had exited before. fn stop_process( process: &impl Stoppable, pid: u32, deadline: Instant, forced: Option, graceful: impl FnOnce() -> std::result::Result<(), E>, -) -> Result> { +) -> Result>> { if process .has_exited() .map_err(|error| undetermined(pid, error))? { - return Ok(Stopped { - outcome: StopOutcome::Graceful, - graceful_error: None, - }); + return Ok(None); } let graceful_error = graceful().err(); let exited = match graceful_error { @@ -658,10 +841,10 @@ fn stop_process( None => StopOutcome::TimedOut, Some(bound) => force(process, pid, bound)?, }; - Ok(Stopped { + Ok(Some(Stopped { outcome, graceful_error, - }) + })) } /// Terminates the VMM `process`, whose ID is `pid`, unless it has exited, and waits up to `bound` @@ -752,15 +935,28 @@ pub enum Reattached { /// can. Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for an invalid /// `runtime`, and as [`LockedStore::recover`] does for inconsistent records. pub fn reattach(locked: &mut LockedStore, runtime: &RuntimeIdentity) -> Result { - reattach_with(locked, runtime, Process::open) + reattach_with(locked, runtime, Process::open, |_| Ok(())) +} + +/// Reattaches as [`reattach`] does, and calls `exited` with the runtime record of a VMM that it +/// found exited, before it clears the record, so that the caller may record what ended the VM +/// while the sandbox still names it, as a clone's reattach does. Fails, and keeps the record, if +/// `exited` fails. +pub(crate) fn reattach_reporting( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + exited: impl FnOnce(&RuntimeRecord) -> Result<()>, +) -> Result { + reattach_with(locked, runtime, Process::open, exited) } -/// Reattaches as [`reattach`] does, holding the recorded VMM with `open`, which is +/// Reattaches as [`reattach_reporting`] does, holding the recorded VMM with `open`, which is /// [`Process::open`] outside tests. fn reattach_with( locked: &mut LockedStore, runtime: &RuntimeIdentity, open: impl FnOnce(&ProcessRef) -> io::Result>, + exited: impl FnOnce(&RuntimeRecord) -> Result<()>, ) -> Result { runtime.check()?; let recovery = locked.recover()?; @@ -779,6 +975,7 @@ fn reattach_with( let process = match open(record.process()) { Ok(Some(process)) => process, Ok(None) => { + exited(&record)?; locked.clear_runtime_of(Some(&record))?; return Ok(Reattached::Exited { record: Some(record), diff --git a/nvx-rs/src/host/supervisor/tests.rs b/nvx-rs/src/host/supervisor/tests.rs index 4a71a08a1..fc105f296 100644 --- a/nvx-rs/src/host/supervisor/tests.rs +++ b/nvx-rs/src/host/supervisor/tests.rs @@ -157,12 +157,12 @@ impl Scripted { } /// Stops the VMM with `forced`, and with a graceful step that fails with `graceful_error` if - /// it is given. - fn stop( + /// it is given, unless it had exited, as [`Vmm::stop_unless_exited`] does. + fn stop_unless_exited( &self, forced: Option, graceful_error: Option<&'static str>, - ) -> Result> { + ) -> Result>> { let deadline = Instant::now() + Duration::from_secs(60); stop_process(self, 7, deadline, forced, || { self.note("graceful"); @@ -173,6 +173,17 @@ impl Scripted { }) } + /// Stops the VMM, which runs as the stop begins, as + /// [`stop_unless_exited`](Self::stop_unless_exited) does. + fn stop( + &self, + forced: Option, + graceful_error: Option<&'static str>, + ) -> Result> { + let stopped = self.stop_unless_exited(forced, graceful_error)?; + Ok(stopped.expect("the VMM had exited before the stop began")) + } + /// Returns the operations of the stop. fn operations(&self) -> Vec<&'static str> { self.operations.borrow().clone() @@ -298,8 +309,10 @@ fn a_failed_graceful_step_is_kept_and_not_waited_out() { fn a_vmm_that_exited_already_skips_the_graceful_step() { let vmm = Scripted::default(); vmm.exited.set(true); - let stopped = vmm.stop(Some(Duration::from_secs(10)), None).unwrap(); - assert_eq!(stopped.outcome, StopOutcome::Graceful); + let stopped = vmm + .stop_unless_exited(Some(Duration::from_secs(10)), None) + .unwrap(); + assert!(stopped.is_none(), "{stopped:?}"); assert_eq!(vmm.operations(), ["check"]); } @@ -460,7 +473,8 @@ fn a_vmm_that_cannot_be_examined_is_indeterminate_and_keeps_its_record() { let hidden = |_: &ProcessRef| -> io::Result> { Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) }; - let Reattached::Indeterminate(reason) = reattach_with(&mut locked, &runtime(), hidden).unwrap() + let Reattached::Indeterminate(reason) = + reattach_with(&mut locked, &runtime(), hidden, |_| Ok(())).unwrap() else { panic!("a VMM that cannot be examined was classified"); }; @@ -522,11 +536,19 @@ fn stops_check_their_store_and_forced_bound_before_anything_else() { .stop(&mut other.lock(None).unwrap(), Instant::now(), None, never) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Other); + let error = vmm + .stop_unless_exited(&mut other.lock(None).unwrap(), Instant::now(), None, never) + .unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Other); for bound in [Duration::ZERO, MAX_TIMEOUT + Duration::from_secs(1)] { let error = vmm .stop(&mut locked, Instant::now(), Some(bound), never) .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Unsupported, "{bound:?}"); + let error = vmm + .stop_unless_exited(&mut locked, Instant::now(), Some(bound), never) + .unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Unsupported, "{bound:?}"); } assert!(store.runtime().unwrap().is_some()); } @@ -555,6 +577,61 @@ fn a_launch_whose_command_fails_ends_before_anything_runs() { assert_eq!(locked.recover().unwrap(), Recovery::Clean); } +#[test] +fn a_launch_whose_caller_cannot_record_its_failure_keeps_its_journal_for_recovery() { + let (_directory, store) = store(); + let mut locked = store.lock(None).unwrap(); + let failure = launch_reporting( + &mut locked, + &LaunchId::generate().unwrap(), + &runtime(), + &CAPABILITY, + |_| Err(Error::unsupported("no command")), + |error, spawned| { + assert_eq!((error.message(), spawned), ("no command", false)); + Err(Error::io("the test cannot record the failure")) + }, + ) + .unwrap_err(); + assert!(!failure.spawned); + let reason = "the test cannot record the failure"; + assert_eq!(failure.unended.as_deref(), Some(reason)); + assert_eq!(failure.error.kind(), ErrorKind::Unsupported); + assert_eq!( + failure.error.message(), + format!("no command; the launch could not be ended, and remains for recovery: {reason}") + ); + // The journal remains, as a crash leaves it, until a recovery ends the launch, of which + // nothing runs. + assert!(store.journal().unwrap().is_some()); + assert_eq!(locked.recover().unwrap(), Recovery::Exited); + assert!(store.journal().unwrap().is_none()); +} + +#[test] +fn a_reattach_whose_caller_cannot_record_an_exit_keeps_the_record_until_one_can() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + // The recorded VMM exited, but the caller cannot record what ended it. + let exited = |_: &ProcessRef| -> io::Result> { Ok(None) }; + let error = reattach_with(&mut locked, &runtime(), exited, |_| { + Err(Error::io("the test cannot record the exit")) + }) + .unwrap_err(); + assert_eq!(error.message(), "the test cannot record the exit"); + // The record, and with it the lease, remain for a later reattach. + assert!(locked.runtime().unwrap().is_some()); + refuses_release(&templates, &mut locked, &lease); + let Reattached::Exited { record, recovery } = + reattach_with(&mut locked, &runtime(), exited, |_| Ok(())).unwrap() + else { + panic!("a VMM that exited was not reported as exited"); + }; + assert_eq!(record.unwrap().launch(), lease.launch()); + assert_eq!(recovery, Recovery::Clean); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + #[test] fn a_launch_whose_vmm_cannot_start_ends() { let (directory, store) = store(); @@ -623,7 +700,8 @@ fn a_clone_whose_vmm_cannot_be_examined_keeps_its_lease() { let hidden = |_: &ProcessRef| -> io::Result> { Err(io::Error::new(io::ErrorKind::PermissionDenied, "hidden")) }; - let Reattached::Indeterminate(reason) = reattach_with(&mut locked, &runtime(), hidden).unwrap() + let Reattached::Indeterminate(reason) = + reattach_with(&mut locked, &runtime(), hidden, |_| Ok(())).unwrap() else { panic!("a VMM that cannot be examined was classified"); }; @@ -673,6 +751,81 @@ fn a_clone_whose_forced_teardown_fails_keeps_its_lease_until_one_ends_its_vmm() assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); } +#[test] +fn only_a_stop_unless_exited_keeps_the_record_of_a_vmm_that_had_exited() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + let Reattached::Live(vmm) = reattach(&mut locked, &runtime()).unwrap() else { + panic!("the clone's VMM was not reattached"); + }; + // This test process stands for the VMM, which these stops reach only through a stand-in, + // which exited once its holder found it running, before the stops began. + let deadline = Instant::now() + Duration::from_secs(60); + let forced = Some(Duration::from_secs(10)); + let never = |_: &Vmm, _: Instant| -> std::result::Result<(), ()> { + panic!("a stop ran the graceful step of a VMM that had exited") + }; + let exited = Scripted::default(); + exited.exited.set(true); + // The stop takes no step, and keeps the record, which tells a reattach that the VM exited + // unheld, and with it the lease. + let stopped = vmm + .stop_unless_exited_with(&mut locked, deadline, forced, never, &exited) + .unwrap(); + assert!(stopped.is_none(), "{stopped:?}"); + assert_eq!(exited.operations(), ["check"]); + assert_eq!(locked.runtime().unwrap().as_ref(), Some(vmm.record())); + refuses_release(&templates, &mut locked, &lease); + // A stop that ends such a VM as one that it stopped clears the record, so the release ends + // the lease. + let stopped = vmm + .stop_with(&mut locked, deadline, forced, never, &exited) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + assert!(stopped.graceful_error.is_none()); + assert_eq!(exited.operations(), ["check", "check"]); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + +#[test] +fn a_stop_unless_exited_stops_a_running_vmm_as_a_stop_does() { + let (_directory, templates, lease, mut locked) = committed_clone(&this_process()); + let Reattached::Live(vmm) = reattach(&mut locked, &runtime()).unwrap() else { + panic!("the clone's VMM was not reattached"); + }; + // This test process stands for the VMM, which these stops reach only through stand-ins. + let deadline = Instant::now() + Duration::from_secs(60); + // One that outlives the stop keeps its record, and the lease. + let survivor = Scripted::default(); + let stopped = vmm + .stop_unless_exited_with( + &mut locked, + deadline, + None, + |_, _| Ok::<_, ()>(()), + &survivor, + ) + .unwrap() + .expect("the stop of a VMM that runs"); + assert_eq!(stopped.outcome, StopOutcome::TimedOut); + assert_eq!(survivor.operations(), ["check", "wait"]); + refuses_release(&templates, &mut locked, &lease); + // One that heeds the graceful step exits through the stop, which clears its record. + let heeding = Scripted::default(); + let shut_down = |_: &Vmm, _: Instant| { + heeding.exited.set(true); + Ok::<_, ()>(()) + }; + let stopped = vmm + .stop_unless_exited_with(&mut locked, deadline, None, shut_down, &heeding) + .unwrap() + .expect("the stop of a VMM that runs"); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + assert_eq!(heeding.operations(), ["check", "wait"]); + assert!(locked.runtime().unwrap().is_none()); + assert_eq!(templates.release(&mut locked).unwrap(), Some(lease)); +} + #[test] fn a_stop_keeps_the_record_of_another_vmm_under_the_same_launch_identity() { let (_directory, store) = store(); diff --git a/nvx-rs/src/host/template/clone.rs b/nvx-rs/src/host/template/clone.rs index a99d49872..fa0cfe003 100644 --- a/nvx-rs/src/host/template/clone.rs +++ b/nvx-rs/src/host/template/clone.rs @@ -7,11 +7,19 @@ #[cfg(all(test, any(target_os = "linux", windows)))] mod tests; +mod failure; mod launch; +mod reattach; mod record; +mod records; mod restore; +pub use self::failure::{ + CloneFailure, Evidence, FailureRecord, Outcome, Progress, Stage, Teardown, status_evidence, +}; pub use self::launch::Restoring; +pub use self::reattach::{ReattachedClone, reattach_clone}; +pub use self::records::{CloneRecords, FAILURE_FORMAT}; pub use self::restore::{ Activated, CloneControl, MAX_ACTIVATION_SESSIONS, READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, RestoreDeadlines, Started, @@ -23,18 +31,21 @@ use std::io; use std::path::{Path, PathBuf}; use std::time::Instant; +use self::failure::{Attribute, Fault, host_evidence}; use super::key::TemplateKey; use super::manifest::BlockGeometry; use super::plan::MEMORY_BLOCK_SIZE; use super::store::{Acquired, Generation, Lease, Store}; use crate::activation::{ActivationRecord, Journal, OverlayMode, WorkloadView}; use crate::contract; -use crate::error::{Error, ErrorKind, Result}; +use crate::error::{Error, ErrorCode, ErrorKind, Result}; use crate::host::filesystem; use crate::host::instance::generate_capability; use crate::host::paths; use crate::host::platform; -use crate::host::sandbox::distro::{self, CheckedDistro, DistroBinding, DistroFormat}; +use crate::host::sandbox::distro::{ + self, CheckedDistro, DistroBinding, DistroFormat, PolicyResult, +}; use crate::host::sandbox::plan::bound_image; use crate::host::sandbox::scratch::{self, CheckedScratch, ScratchBinding}; use crate::host::sandbox::{ @@ -43,7 +54,7 @@ use crate::host::sandbox::{ }; use crate::host::store::LockedStore; use crate::host::store::files::{create_new, remove_if_present}; -use crate::host::store::regular::{identity_of, open_regular, was_removed}; +use crate::host::store::regular::{identity_of, is_refused, open_regular, was_removed}; use crate::protocol::CAPABILITY_LEN; /// Bytes in a mebibyte, the unit in which messages give the size of memory blocks. @@ -286,59 +297,94 @@ impl ClonePlan { /// snapshot, [`Store::snapshot_dir`], then checks the clone's files, as the [type](Self) /// describes. /// - /// Fails, and holds nothing, keeping no file that it created, checking in this order: + /// Fails with a [`CloneFailure`] at [`Stage::Plan`], and holds nothing, keeping no file that + /// it created. The failure's record holds the evidence of the check that failed, as below, no + /// transaction, and no VM, [`Teardown::NoVm`], and the plan records it nowhere: the caller + /// holds it, and releases the lease. The evidence of a failure of the host's own operations is + /// [`Evidence::ResourceExhaustion`] if an I/O error among the error's sources says that the + /// host ran out of storage, memory, or descriptors, and [`Evidence::HostFailure`] otherwise. + /// The plan checks in this order: /// /// - with [`ErrorKind::Unsupported`] for a request that no clone takes: a plan of the sandbox - /// profile, with workload settings and its shares; a plan whose image is not the format and - /// identity that the binding names, as for a binding whose identity is not a UUID in its - /// canonical text; an image that is a GPT disk, of which this build restores no clone, since - /// a clone's guest assembles its root only from an EROFS layer; an image or a scratch image - /// whose path is relative, which OpenVMM would resolve against the directory in which it - /// runs, holds a comma or a semicolon, which OpenVMM's block device options would split, or - /// whose name ends in `.vhd`, `.vhdx`, `.vmgs`, or `.iso`, which OpenVMM would not attach as - /// a raw image; and an upper on a scratch image without the image's binding, or one in RAM - /// with a binding; + /// profile, with workload settings and its shares, [`Evidence::InvalidSettings`]; a plan + /// whose image is not the format and identity that the binding names, as for a binding + /// whose identity is not a UUID in its canonical text, and an image that is a GPT disk, of + /// which this build restores no clone, since a clone's guest assembles its root only from an + /// EROFS layer, [`Evidence::InvalidFormat`]; an image or a scratch image whose path is + /// relative, which OpenVMM would resolve against the directory in which it runs, holds a + /// comma or a semicolon, which OpenVMM's block device options would split, or whose name + /// ends in `.vhd`, `.vhdx`, `.vmgs`, or `.iso`, which OpenVMM would not attach as a raw + /// image, and an upper on a scratch image without the image's binding, or one in RAM with a + /// binding, [`Evidence::InvalidSettings`]; /// - with [`ErrorKind::Other`] for a lease of another store, and a lock that is not the one - /// that acquired the lease, before the launch that it reserves began; - /// - with [`ErrorKind::NotFound`] if the store no longer indexes the leased generation, with - /// [`ErrorKind::Io`] if it indexes it with another key than the lease names, and as - /// [`Store::index`], [`Store::read_key`], and [`Store::snapshot_dir`] fail; + /// that acquired the lease, before the launch that it reserves began, + /// [`Evidence::HostFailure`]; + /// - with [`ErrorKind::NotFound`] if the store no longer indexes the leased generation, and as + /// [`Store::index`] fails, the host's evidence; with [`ErrorKind::Io`] if it indexes it with + /// another key than the lease names, [`Evidence::MalformedGeneration`]; and as + /// [`Store::read_key`] and [`Store::snapshot_dir`] fail, once the store's directory of + /// generations passed their check, whose failure is the host's: the host's evidence for an + /// I/O error, [`Evidence::IncompatibleHost`] for a key of another encoding version than this + /// build reads, with [`ErrorCode::FormatUnsupported`], which a build that reads it may + /// serve, and [`Evidence::MalformedGeneration`] for a generation whose payload, key, or + /// snapshot is missing, is not private to the current user, or is malformed, as a key that + /// [`Store::read_key`] refuses for what it is: a link, a file that is not a regular file, or + /// one that holds more than a key may; /// - with [`ErrorKind::Unsupported`] for a request that the template cannot serve: a template /// whose key names other contracts than this build speaks, the boot and control contracts - /// of the runtime that captured it included; a plan with a network for a template without a - /// NIC, without one for a template with a NIC, or with another guest network than the - /// template's NIC, whose address OpenVMM restores from the snapshot; and a memory target - /// that is not whole blocks of [`MEMORY_BLOCK_SIZE`] and of the template's memory blocks, - /// from the template's RAM to its RAM capacity; + /// of the runtime that captured it included, [`Evidence::IncompatibleHost`]; a plan with a + /// network for a template without a NIC, without one for a template with a NIC, or with + /// another guest network than the template's NIC, whose address OpenVMM restores from the + /// snapshot, [`Evidence::InvalidSettings`]; and a memory target that is not whole blocks of + /// [`MEMORY_BLOCK_SIZE`] and of the template's memory blocks, from the template's RAM to its + /// RAM capacity, [`Evidence::InvalidTargets`]; /// - with [`ErrorKind::Io`] for a template whose command line names the DNS server of its /// guest's resolver in another form than OpenVMM writes: more than once, without a NIC, or - /// other than as one IP address in its canonical spelling; + /// other than as one IP address in its canonical spelling, [`Evidence::MalformedGeneration`]; /// - with [`ErrorKind::Unsupported`] for settings that no activation record holds, as /// [`Configuration::check`] finds them, such as a hostname that is empty, longer than /// [`MAX_HOSTNAME_BYTES`](crate::activation::MAX_HOSTNAME_BYTES), or holds a NUL, and a - /// workload identity whose UID or GID is 0, and with [`ErrorKind::Other`] if the - /// transaction ID of the clone's restore cannot be drawn; + /// workload identity whose UID or GID is 0, [`Evidence::InvalidActivate`], and with + /// [`ErrorKind::Other`] if the transaction ID of the clone's restore cannot be drawn, the + /// host's evidence; /// - with [`ErrorKind::Unavailable`] for a link or a directory under a name that the sandbox's - /// store owns, and with [`ErrorKind::Io`] if the store's files cannot be listed or examined; - /// - as [`DistroBinding::check`] and [`CheckedDistro::geometry`] fail, with - /// [`ErrorKind::Unsupported`] for an image of another geometry than the template's - /// placeholder, its length included, with [`ErrorKind::Unavailable`] for an image that is - /// not, at its path, the regular file that the check found, of the current user or an - /// administrator, and that no one else may change, and with [`ErrorKind::Unsupported`] for - /// an image that is a file of the sandbox's store; + /// store owns, and with [`ErrorKind::Io`] if the store's files cannot be listed or examined, + /// the host's evidence; + /// - as [`DistroBinding::check`] and [`CheckedDistro::geometry`] fail: with + /// [`ErrorKind::Unavailable`] for a binding that holds a verdict of another revision of the + /// distro policy than the request accepts, [`Evidence::StaleBinding`], and with + /// [`ErrorKind::Unsupported`] for one whose verdict rejects the image, + /// [`Evidence::AuthorizationDenied`]; with [`ErrorKind::Unavailable`] for an image that + /// changed, was replaced, or went since its binding was made, or whose manifest did, and for + /// an image that is not, at its path, the regular file that the check found, of the current + /// user or an administrator, and that no one else may change, + /// [`Evidence::SuppliedIdentity`], and with [`ErrorKind::Io`] if the image cannot be opened + /// at its path otherwise, or identified or examined once open, the host's evidence; with + /// [`ErrorKind::Unsupported`] for an image of another length than the template's + /// placeholder, [`Evidence::SuppliedLength`], or of other block sizes, or of a geometry that + /// OpenVMM refuses, [`Evidence::SuppliedGeometry`]; and with + /// [`ErrorKind::Unsupported`] for an image that is a file of the sandbox's store, + /// [`Evidence::InvalidSettings`]; /// - for an upper in RAM, with [`ErrorKind::Unsupported`] for a sandbox whose path holds a - /// comma or a semicolon, or whose filesystem presents another geometry than the template's - /// scratch, and with [`ErrorKind::Io`] if the sparse file cannot be created, sized, or - /// checked; + /// comma or a semicolon, [`Evidence::HostFailure`], or whose filesystem presents another + /// geometry than the template's scratch, [`Evidence::IncompatibleHost`], and with + /// [`ErrorKind::Io`] if the sparse file cannot be created, sized, or checked, the host's + /// evidence; /// - for an upper on a scratch image, with [`ErrorKind::Io`] if the image's location cannot be - /// compared with the template store's directory, or its names cannot be counted, as - /// [`ScratchBinding::check`] and [`CheckedScratch::geometry`] fail, with - /// [`ErrorKind::Unsupported`] for an image of another geometry than the template's scratch, - /// its length included, with [`ErrorKind::Unavailable`] for an image that is not, at its - /// path, the regular file that the check found, of the current user or an administrator, - /// and that no one else may change, and with [`ErrorKind::Unsupported`] for an image that - /// is a file of the sandbox's store, or that is in the template store's directory, as the - /// files of its snapshots are, or has another name. + /// compared with the template store's directory, or its names cannot be counted, the host's + /// evidence, and as [`ScratchBinding::check`] and [`CheckedScratch::geometry`] fail: with + /// [`ErrorKind::TimedOut`] and [`ErrorCode::ScratchClaimed`] for an image that another + /// launch still claims, [`Evidence::UnusableScratch`]; with [`ErrorKind::Unavailable`] for an + /// image that changed, was replaced, or went since its binding was made, or that is not, at + /// its path, the regular file that the check found, of the current user or an administrator, + /// and that no one else may change, [`Evidence::SuppliedIdentity`], and with + /// [`ErrorKind::Io`] if the image cannot be opened at its path otherwise, or identified or + /// examined once open, the host's evidence; with [`ErrorKind::Unsupported`] for an image of + /// another length than the template's scratch, [`Evidence::SuppliedLength`], or of other + /// block sizes, [`Evidence::SuppliedGeometry`]; and with [`ErrorKind::Unsupported`] for an + /// image that is a file of the sandbox's store, or that is in the template store's + /// directory, as the files of its snapshots are, or has another name, or whose path is not + /// Unicode, [`Evidence::InvalidSettings`]. /// /// [`Configuration::check`]: crate::activation::Configuration::check pub fn new( @@ -346,7 +392,7 @@ impl ClonePlan { acquired: &mut Acquired, request: &CloneRequest, deadline: Instant, - ) -> Result { + ) -> Result { Self::new_with(store, acquired, request, deadline, Access::CHECKED) } @@ -357,43 +403,78 @@ impl ClonePlan { request: &CloneRequest, deadline: Instant, access: Access<'_>, - ) -> Result { + ) -> Result { + let lease = acquired.lease.clone(); + Self::plan(store, acquired, request, deadline, access).map_err(|fault| { + let record = FailureRecord::new( + &lease, + None, + (Stage::Plan, Progress::Planned), + fault.evidence, + None, + Teardown::NoVm, + &fault.error, + ); + CloneFailure::new(fault.error, record) + }) + } + + /// Plans a clone as [`new`](Self::new) does, reaching its files through `access`, and + /// returns the fault of the check that failed. + fn plan( + store: &Store, + acquired: &mut Acquired, + request: &CloneRequest, + deadline: Instant, + access: Access<'_>, + ) -> Result { check_request(request)?; let Acquired { lease, locked } = acquired; - check_lease(store, lease, locked)?; + check_lease(store, lease, locked).attribute(Evidence::HostFailure)?; let generation = store - .index()? + .index() + .on_host()? .get(lease.generation()) .cloned() .ok_or_else(|| { - Error::new( - ErrorKind::NotFound, - format!( - "the template store {} holds no generation {}, which the lease {} names", - store.dir().display(), - lease.generation(), - lease.id() + Fault::new( + Evidence::HostFailure, + Error::new( + ErrorKind::NotFound, + format!( + "the template store {} holds no generation {}, which the lease {} \ + names", + store.dir().display(), + lease.generation(), + lease.id() + ), ), ) })?; if generation.key() != lease.key() { - return Err(Error::io(format!( - "the generation {} of the template store {} has the key {}, not the key {} that \ - the lease {} names", - generation.generation(), - store.dir().display(), - generation.key(), - lease.key(), - lease.id() - ))); + return Err(Fault::new( + Evidence::MalformedGeneration, + Error::io(format!( + "the generation {} of the template store {} has the key {}, not the key {} \ + that the lease {} names", + generation.generation(), + store.dir().display(), + generation.key(), + lease.key(), + lease.id() + )), + )); } - let key = store.read_key(&generation)?; - let snapshot = store.snapshot_dir(&generation)?; + // A store whose directory of generations is not private fails every generation: the + // host's, not the generation's. + store.check_generations_dir().on_host()?; + let key = store.read_key(&generation).map_err(generation_fault)?; + let snapshot = store.snapshot_dir(&generation).map_err(generation_fault)?; check_template(request, &key, &generation)?; let activation = record::build(request, lease, &key)?; - let owned = StoreFiles::of(locked)?; + let owned = StoreFiles::of(locked).on_host()?; let image = check_image(request, &key, &owned, access)?; - let capability = generate_capability()?; + let capability = generate_capability().on_host()?; let upper = match (request.adapter.upper().scratch_path(), &request.scratch) { (Some(path), Some(binding)) => CloneUpper::Scratch(check_scratch( path, @@ -530,41 +611,80 @@ fn devices_of(request: &CloneRequest, upper: &CloneUpper) -> SandboxPlan { /// Refuses a request that no clone takes, as [`ClonePlan::new`] describes, without reading or /// opening anything. -fn check_request(request: &CloneRequest) -> Result<()> { +fn check_request(request: &CloneRequest) -> Result<(), Fault> { let adapter = &request.adapter; if adapter.settings().is_some() { - return Err(Error::unsupported( - "a clone restores an adapter's plan, not a plan of the sandbox profile, whose \ - workload settings no template holds", + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported( + "a clone restores an adapter's plan, not a plan of the sandbox profile, whose \ + workload settings no template holds", + ), )); } let distro = adapter.distro(); let binding = &request.binding; - if bound_image(binding)? != distro.image() { - return Err(Error::unsupported(format!( - "the image {} is not the {} layer with identity {} that its distro binding names", - distro.path().display(), - binding.format(), - binding.identity() - ))); + if bound_image(binding).attribute(Evidence::InvalidFormat)? != distro.image() { + return Err(Fault::new( + Evidence::InvalidFormat, + Error::unsupported(format!( + "the image {} is not the {} layer with identity {} that its distro binding names", + distro.path().display(), + binding.format(), + binding.identity() + )), + )); } if matches!(binding.format(), DistroFormat::Gpt) { - return Err(record::gpt_refused()); + return Err(Fault::new(Evidence::InvalidFormat, record::gpt_refused())); } - check_disk_path(distro.path(), "image")?; + check_disk_path(distro.path(), "image").attribute(Evidence::InvalidSettings)?; match (adapter.upper().scratch_path(), &request.scratch) { - (Some(path), Some(_)) => check_disk_path(path, "scratch image"), + (Some(path), Some(_)) => { + check_disk_path(path, "scratch image").attribute(Evidence::InvalidSettings) + } (None, None) => Ok(()), - (Some(path), None) => Err(Error::unsupported(format!( - "the upper of a clone on the scratch image {} needs the image's scratch binding", - path.display() - ))), - (None, Some(_)) => Err(Error::unsupported( - "the upper of a clone in RAM takes no scratch binding", + (Some(path), None) => Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the upper of a clone on the scratch image {} needs the image's scratch binding", + path.display() + )), + )), + (None, Some(_)) => Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported("the upper of a clone in RAM takes no scratch binding"), )), } } +/// Returns the fault of `error`, with which the template store's read of the key or of the +/// snapshot of the leased generation failed, once the store's directory of generations passed its +/// check: the host's, which cannot restore the template, for a key of another encoding version +/// than this build reads, as [`TemplateKey::from_bytes`] refuses it with +/// [`ErrorCode::FormatUnsupported`], since a build that reads it may serve the generation; the +/// generation's, which is malformed, for a payload, a key, or a snapshot that is missing, that is +/// not private to the current user, or that is not one that the store writes, such as a key that +/// the read refuses for what it is, as [`is_refused`] tells: a link, a file that is not a regular +/// file, or one that holds more than a key may; and the host's for any other I/O error among the +/// error's sources. +fn generation_fault(error: Error) -> Fault { + if error.code() == ErrorCode::FormatUnsupported { + return Fault::new(Evidence::IncompatibleHost, error); + } + let mut source = std::error::Error::source(&error); + while let Some(cause) = source { + if let Some(io) = cause.downcast_ref::() + && io.kind() != io::ErrorKind::NotFound + && !is_refused(io) + { + return Fault::new(host_evidence(&error), error); + } + source = cause.source(); + } + Fault::new(Evidence::MalformedGeneration, error) +} + /// Refuses the path of the file of a clone's block device, which messages call `what`, that /// OpenVMM would not open as the plan checks it: a relative one, which it would resolve against the /// directory in which it runs, one with a comma or a semicolon, which its block device options @@ -612,7 +732,7 @@ fn check_template( request: &CloneRequest, key: &TemplateKey, generation: &Generation, -) -> Result<()> { +) -> Result<(), Fault> { let name = generation.generation(); // The key's contracts are those through which its clones are restored; the runtime's are // those through which this build boots and drives the guest that the template holds. @@ -622,28 +742,37 @@ fn check_template( || runtime.control_contract != contract::CONTROL_CONTRACT_REVISION || runtime.control_session != contract::CONTROL_SESSION_PROTOCOL_VERSION { - return Err(Error::unsupported(format!( - "the template's generation {name} speaks other contracts than this build, which \ - restores none of its clones" - ))); + return Err(Fault::new( + Evidence::IncompatibleHost, + Error::unsupported(format!( + "the template's generation {name} speaks other contracts than this build, which \ + restores none of its clones" + )), + )); } // A network that OpenVMM attaches no device for is no NIC, as a capture class's. let attached = match request.adapter.network() { None => false, - Some(network) => !network.openvmm_arguments()?.is_empty(), + Some(network) => !network + .openvmm_arguments() + .attribute(Evidence::InvalidSettings)? + .is_empty(), }; if attached != key.has_network() { - return Err(Error::unsupported(if attached { - format!( - "the adapter's plan of a clone attaches a network, but the template's generation \ - {name} has no NIC" - ) - } else { - format!( - "the template's generation {name} has a NIC, which its clones restore with the \ - network of the adapter's plan, but the plan has none" - ) - })); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(if attached { + format!( + "the adapter's plan of a clone attaches a network, but the template's \ + generation {name} has no NIC" + ) + } else { + format!( + "the template's generation {name} has a NIC, which its clones restore with \ + the network of the adapter's plan, but the plan has none" + ) + }), + )); } // OpenVMM restores the guest's address on the NIC from the snapshot, and a clone's launch // drops the plan's, so the plan must name the template's. @@ -651,11 +780,14 @@ fn check_template( && let Some(template) = key.guest_network() && network.guest_network() != template { - return Err(Error::unsupported(format!( - "the adapter's plan of a clone gives its guest the network {}, but the template's \ - generation {name} restores its NIC on {template}, from its snapshot", - network.guest_network() - ))); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the adapter's plan of a clone gives its guest the network {}, but the template's \ + generation {name} restores its NIC on {template}, from its snapshot", + network.guest_network() + )), + )); } let memory = request.memory; let block = key.memory_block_size(); @@ -664,15 +796,18 @@ fn check_template( || !memory.is_multiple_of(MEMORY_BLOCK_SIZE) || !memory.is_multiple_of(block) { - return Err(Error::unsupported(format!( - "a clone of the template's generation {name} cannot restore with {memory} bytes of \ - RAM: its memory target must be whole blocks of {} MiB, and of the template's memory \ - blocks of {block} bytes, from the template's {} bytes of RAM to its capacity of {} \ - bytes", - MEMORY_BLOCK_SIZE / MIB, - key.memory(), - key.memory_capacity() - ))); + return Err(Fault::new( + Evidence::InvalidTargets, + Error::unsupported(format!( + "a clone of the template's generation {name} cannot restore with {memory} bytes \ + of RAM: its memory target must be whole blocks of {} MiB, and of the template's \ + memory blocks of {block} bytes, from the template's {} bytes of RAM to its \ + capacity of {} bytes", + MEMORY_BLOCK_SIZE / MIB, + key.memory(), + key.memory_capacity() + )), + )); } Ok(()) } @@ -686,29 +821,54 @@ fn check_image( key: &TemplateKey, owned: &StoreFiles, access: Access<'_>, -) -> Result { +) -> Result { let path = request.adapter.distro().path(); let shown = path.display(); - let refused = |problem: String| Error::unavailable(format!("the image {shown} {problem}")); - let text = path - .to_str() - .ok_or_else(|| refused("has a path that is not valid Unicode".to_owned()))?; + let text = path.to_str().ok_or_else(|| { + Fault::new( + Evidence::InvalidSettings, + Error::unavailable(format!( + "the image {shown} has a path that is not valid Unicode" + )), + ) + })?; let binding = &request.binding; let layer = match binding.format() { DistroFormat::Erofs => SandboxLayer::new(LayerRole::Distro, text, binding.identity()), DistroFormat::Gpt => SandboxLayer::gpt(LayerRole::Distro, text, binding.identity()), }; - let checked = binding.check_with(&layer, request.policy_revision, access.distro)?; + let policy = binding.policy(); + let checked = binding + .check_with(&layer, request.policy_revision, access.distro) + .map_err(|error| { + // The check refuses a stale binding, then one whose verdict rejects the image, before + // it opens anything. + let evidence = if binding.version() != distro::BINDING_VERSION + || policy.revision != request.policy_revision + { + Evidence::StaleBinding + } else if policy.result != PolicyResult::Accepted { + Evidence::AuthorizationDenied + } else { + binding_evidence(&error) + }; + Fault::new(evidence, error) + })?; + let geometry = checked.geometry().map_err(supplied_geometry_fault)?; check_geometry( - checked.geometry()?, + geometry, key.distro(), &format!("the image {shown}"), "placeholder", + Mismatch::SUPPLIED, )?; - let identity = identity_of(checked.file()) - .map_err(|error| refused(format!("cannot be identified: {error}")))?; - check_at_path(path, identity, refused)?; - owned.check(identity, "image", path)?; + let identity = identity_of(checked.file()).map_err(|error| { + Fault::host(Error::io(format!("the image {shown} cannot be identified")).with_source(error)) + })?; + check_at_path(path, identity, &format!("the image {shown}"))?; + owned + .check(identity, "image", path) + .attribute(Evidence::InvalidSettings)?; Ok(checked) } @@ -725,21 +885,38 @@ fn check_scratch( (owned, store): (&StoreFiles, &Store), deadline: Instant, access: Access<'_>, -) -> Result { +) -> Result { let shown = path.display(); - let checked = binding.check_with(path, deadline, access.scratch)?; + let checked = binding + .check_with(path, deadline, access.scratch) + .map_err(|error| { + let evidence = if binding.version() != scratch::SCRATCH_BINDING_VERSION { + Evidence::StaleBinding + } else if error.code() == ErrorCode::ScratchClaimed { + // Another launch holds the image, which is not fresh for this clone. + Evidence::UnusableScratch + } else { + binding_evidence(&error) + }; + Fault::new(evidence, error) + })?; + let geometry = checked.geometry().map_err(supplied_geometry_fault)?; check_geometry( - checked.geometry()?, + geometry, key.scratch(), &format!("the scratch image {shown}"), "scratch", + Mismatch::SUPPLIED, )?; - let refused = - |problem: String| Error::unavailable(format!("the scratch image {shown} {problem}")); - let identity = identity_of(checked.file()) - .map_err(|error| refused(format!("cannot be identified: {error}")))?; - check_at_path(path, identity, refused)?; - owned.check(identity, "scratch image", path)?; + let identity = identity_of(checked.file()).map_err(|error| { + Fault::host( + Error::io(format!("the scratch image {shown} cannot be identified")).with_source(error), + ) + })?; + check_at_path(path, identity, &format!("the scratch image {shown}"))?; + owned + .check(identity, "scratch image", path) + .attribute(Evidence::InvalidSettings)?; check_outside_store(&checked, store, path)?; check_single_name(&checked, path)?; Ok(checked) @@ -754,26 +931,33 @@ fn check_scratch( /// file into the store's directory, so one outside it stays outside it. The location resolves the /// aliases that the file system makes, links and junctions on the way and directories held open, /// but not a mount or a network share that exposes the store's directories elsewhere, which -/// [`ClonePlan`] leaves to the system's administration. -fn check_outside_store(checked: &CheckedScratch, store: &Store, path: &Path) -> Result<()> { +/// [`ClonePlan`] leaves to the system's administration. The request supplied the image, so its +/// refusal is invalid settings, while the failure to locate it is the host's. +fn check_outside_store(checked: &CheckedScratch, store: &Store, path: &Path) -> Result<(), Fault> { let failed = |error: io::Error| { - Error::io(format!( - "cannot compare the location of the scratch image {} with the template store in {}", - path.display(), - store.dir().display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot compare the location of the scratch image {} with the template store in \ + {}", + path.display(), + store.dir().display() + )) + .with_source(error), + ) }; let location = location_of(checked.file()).map_err(failed)?; if is_within(&location, store.dir()).map_err(failed)? { - return Err(Error::unsupported(format!( - "the scratch image {} is {}, in the directory of the template store {}, where its \ - captures write the snapshots that every clone of a generation restores in place: a \ - clone's scratch image must be outside it", - path.display(), - location.display(), - store.dir().display() - ))); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the scratch image {} is {}, in the directory of the template store {}, where its \ + captures write the snapshots that every clone of a generation restores in \ + place: a clone's scratch image must be outside it", + path.display(), + location.display(), + store.dir().display() + )), + )); } Ok(()) } @@ -825,26 +1009,54 @@ fn location_of(file: &File) -> io::Result { /// Refuses the clone's scratch image at `path`, which `checked` holds open, unless it has a single /// name: another name, a hard link, could make it a file of a snapshot that the clone's guest then /// writes, of one of the store's generations or captures, or of another store, which the image's -/// location does not show. -fn check_single_name(checked: &CheckedScratch, path: &Path) -> Result<()> { +/// location does not show. The request supplied the image, so its refusal is invalid settings, +/// while the failure to count its names is the host's. +fn check_single_name(checked: &CheckedScratch, path: &Path) -> Result<(), Fault> { let links = platform::link_count(checked.file()).map_err(|error| { - Error::io(format!( - "cannot count the names of the scratch image {}", - path.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot count the names of the scratch image {}", + path.display() + )) + .with_source(error), + ) })?; if links > 1 { - return Err(Error::unsupported(format!( - "the scratch image {} has {links} names, but a clone's scratch image must have a \ - single name: another, a hard link, could make it a file of a snapshot, which every \ - clone of its generation restores in place", - path.display() - ))); + return Err(Fault::new( + Evidence::InvalidSettings, + Error::unsupported(format!( + "the scratch image {} has {links} names, but a clone's scratch image must have a \ + single name: another, a hard link, could make it a file of a snapshot, which \ + every clone of its generation restores in place", + path.display() + )), + )); } Ok(()) } +/// Returns the evidence of `error`, with which the check of the binding of a file that the request +/// supplied failed, past the binding's own version and verdict: a file that changed, was replaced, +/// or went since its binding was made, a supplied file of the wrong identity; a path that the +/// check refuses, invalid settings; and any other failure, the host's. +fn binding_evidence(error: &Error) -> Evidence { + match error.kind() { + ErrorKind::Unavailable => Evidence::SuppliedIdentity, + ErrorKind::Unsupported => Evidence::InvalidSettings, + _ => host_evidence(error), + } +} + +/// Returns the fault of `error`, with which the geometry of a file that the request supplied +/// cannot be read: a geometry that OpenVMM refuses, a supplied file of the wrong geometry, and any +/// other failure, the host's. +fn supplied_geometry_fault(error: Error) -> Fault { + match error.kind() { + ErrorKind::Unsupported => Fault::new(Evidence::SuppliedGeometry, error), + _ => Fault::host(error), + } +} + /// The regular files that the store of a sandbox owns, with their identities, which no image or /// scratch image of a clone may be. struct StoreFiles(Vec<(PathBuf, (u64, u128))>); @@ -928,34 +1140,53 @@ impl StoreFiles { /// the capture of a template checks its own, as [`ClonePlan`] describes. Returns its path, once /// the file is closed. It removes the file if any step fails, its creation included, which can /// fail once it made the file, as a file that cannot be made sparse does. -fn create_scratch(locked: &LockedStore, key: &TemplateKey, access: Access<'_>) -> Result { +fn create_scratch( + locked: &LockedStore, + key: &TemplateKey, + access: Access<'_>, +) -> Result { let path = locked.clone_scratch_path(); let shown = path.display(); - reject_disk_path(&path, &shown.to_string())?; + reject_disk_path(&path, &shown.to_string()).attribute(Evidence::HostFailure)?; let failed = |error: io::Error| { - Error::io(format!("cannot create the {SPARSE_WHAT} {shown}")).with_source(error) + Fault::host( + Error::io(format!("cannot create the {SPARSE_WHAT} {shown}")).with_source(error), + ) }; let created = (|| { let file = (access.create)(&path).map_err(&failed)?; let scratch = key.scratch(); file.set_len(scratch.length).map_err(&failed)?; + let geometry = + block_geometry(&file, &path, SPARSE_WHAT).map_err(|error| match error.kind() { + // The sandbox's filesystem presents a geometry that OpenVMM refuses. + ErrorKind::Unsupported => Fault::new(Evidence::IncompatibleHost, error), + _ => Fault::host(error), + })?; check_geometry( - block_geometry(&file, &path, SPARSE_WHAT)?, + geometry, scratch, &format!("the {SPARSE_WHAT} {shown}, in the sandbox's filesystem,"), "scratch", + Mismatch::HOST, )?; let identity = identity_of(&file).map_err(&failed)?; drop(file); - check_at_path(&path, identity, |problem| { - Error::io(format!("the {SPARSE_WHAT} {shown} {problem}")) + // The host made the file, so what its path shows of it is the host's. + check_at_path(&path, identity, &format!("the {SPARSE_WHAT} {shown}")).map_err(|fault| { + let (_, message, source) = fault.error.into_parts(); + let error = Error::io(message); + Fault::host(match source { + Some(source) => error.with_source(source), + None => error, + }) }) })(); match created { Ok(()) => Ok(path), - Err(error) => { + Err(fault) => { let _ = remove_if_present(&path); - Err(error) + Err(fault) } } } @@ -973,54 +1204,119 @@ fn create_sparse(path: &Path) -> io::Result { Ok(file) } +/// The evidence of a file of a clone's block device whose geometry is not the one that the +/// template's device presented. +#[derive(Clone, Copy)] +struct Mismatch { + /// The evidence of another length. + length: Evidence, + /// The evidence of other block sizes. + blocks: Evidence, +} + +impl Mismatch { + /// A file that the request supplied. + const SUPPLIED: Self = Self { + length: Evidence::SuppliedLength, + blocks: Evidence::SuppliedGeometry, + }; + + /// The sparse file of an upper in RAM, which the plan sizes as the template's scratch, and + /// whose block sizes the sandbox's filesystem presents: the host's. + const HOST: Self = Self { + length: Evidence::IncompatibleHost, + blocks: Evidence::IncompatibleHost, + }; +} + /// Refuses `found`, the geometry of the file of a clone's block device, which messages call -/// `what`, unless it is `expected`, the geometry that the template's `device` presented. +/// `what`, unless it is `expected`, the geometry that the template's `device` presented, with the +/// evidence that `mismatch` gives. fn check_geometry( found: BlockGeometry, expected: BlockGeometry, what: &str, device: &str, -) -> Result<()> { + mismatch: Mismatch, +) -> Result<(), Fault> { if found.length != expected.length { - return Err(Error::unsupported(format!( - "{what} is {} bytes long, not the {} bytes of the template's {device}", - found.length, expected.length - ))); + return Err(Fault::new( + mismatch.length, + Error::unsupported(format!( + "{what} is {} bytes long, not the {} bytes of the template's {device}", + found.length, expected.length + )), + )); } if found != expected { - return Err(Error::unsupported(format!( - "{what} presents logical blocks of {} bytes and physical blocks of {} bytes, but the \ - template's {device} presented {} and {} bytes", - found.logical_block_size, - found.physical_block_size, - expected.logical_block_size, - expected.physical_block_size - ))); + return Err(Fault::new( + mismatch.blocks, + Error::unsupported(format!( + "{what} presents logical blocks of {} bytes and physical blocks of {} bytes, but \ + the template's {device} presented {} and {} bytes", + found.logical_block_size, + found.physical_block_size, + expected.logical_block_size, + expected.physical_block_size + )), + )); } Ok(()) } -/// Checks the file at `path` as OpenVMM will open it, by its path: a regular file, not a link, -/// nor on Windows any other reparse point, whose identity is `expected`, the identity of the file -/// that a check found, and that no one but the current user, or an administrator, may change. -/// Fails with the error that `refused` makes of the problem that it found. -fn check_at_path( - path: &Path, - expected: (u64, u128), - refused: impl Fn(String) -> Error, -) -> Result<()> { - let file = open_regular(path).map_err(|error| refused(format!("cannot be used: {error}")))?; - let identity = - identity_of(&file).map_err(|error| refused(format!("cannot be identified: {error}")))?; +/// Checks the file at `path`, which messages call `what`, as OpenVMM will open it, by its path: a +/// regular file, not a link, nor on Windows any other reparse point, whose identity is +/// `expected`, the identity of the file that a check found, and that no one but the current user, +/// or an administrator, may change. +/// +/// Fails with [`ErrorKind::Unavailable`] and [`Evidence::SuppliedIdentity`] for a path that no +/// longer names that file: one whose file went, a component of which is no longer a directory, +/// that the current user may no longer open, or that names a link, a file that is not regular, or +/// another file, and for a file that others may change; and with [`ErrorKind::Io`] and the host's +/// evidence if the file cannot be opened at its path otherwise, as when the host runs out of +/// descriptors, or if the file that it opened cannot be identified or examined. +fn check_at_path(path: &Path, expected: (u64, u128), what: &str) -> Result<(), Fault> { + let refused = |problem: String| { + Fault::new( + Evidence::SuppliedIdentity, + Error::unavailable(format!("{what} {problem}")), + ) + }; + let failed = |action: &str, error: io::Error| { + Fault::host(Error::io(format!("{what} cannot be {action}")).with_source(error)) + }; + let file = open_regular(path).map_err(|error| { + if path_refused(&error) { + refused(format!("cannot be used: {error}")) + } else { + failed("opened", error) + } + })?; + let identity = identity_of(&file).map_err(|error| failed("identified", error))?; if identity != expected { return Err(refused( "changed while it was checked: its path names another file".to_owned(), )); } - if let Some(problem) = platform::foreign_writer_problem(&file) - .map_err(|error| refused(format!("cannot be checked: {error}")))? + if let Some(problem) = + platform::foreign_writer_problem(&file).map_err(|error| failed("checked", error))? { return Err(refused(problem)); } Ok(()) } + +/// Returns whether `error`, with which [`open_regular`] failed to open a file that a request +/// supplied at its path, says that the path no longer names the file that a check found: the +/// file went, a component of the path is no longer a directory, the current user may no longer +/// open it, or the open refused a link or a file that is not regular, a failure that the system +/// does not report. Any other failure is the host's. +fn path_refused(error: &io::Error) -> bool { + error.raw_os_error().is_none() + || matches!( + error.kind(), + io::ErrorKind::NotFound + | io::ErrorKind::NotADirectory + | io::ErrorKind::PermissionDenied + ) +} diff --git a/nvx-rs/src/host/template/clone/failure.rs b/nvx-rs/src/host/template/clone/failure.rs new file mode 100644 index 000000000..3b3af078e --- /dev/null +++ b/nvx-rs/src/host/template/clone/failure.rs @@ -0,0 +1,1279 @@ +//! The failures of clones, [`CloneFailure`]: the [`Evidence`] of a failure of a clone's plan, +//! launch, restore sequence, or reattach, the [`Outcome`] to which the evidence attributes it, as +//! [the attribution of evidence](Evidence#attribution) describes, the [`Stage`] at which the clone +//! failed, the clone's [`Progress`] by then, and the [`Teardown`] that followed, which the +//! failure's [`FailureRecord`] keeps, and the clone's sandbox records, as [the records of +//! failures](FailureRecord#records) describe. + +use std::fmt; +use std::io; +use std::time::Instant; + +use serde::{Deserialize, Serialize}; + +use crate::activation::{TemplateReference, TransactionId}; +use crate::error::{Error, ErrorCode, ErrorKind, Result}; +use crate::host::report::Report; +use crate::host::store::{LaunchId, Recovery}; +use crate::host::template::{Lease, LeaseId, Store, StoreId}; +use crate::protocol::status; + +/// What the host saw of a failure of a clone, which attributes the failure to an [`Outcome`]. +/// +/// # Attribution +/// +/// The outcome of a failure follows from its evidence alone, as [`outcome`](Self::outcome) gives +/// it: +/// +/// | Evidence | Outcome | Quarantine | +/// | --- | --- | --- | +/// | An authorization that the image's binding denies; a stale image binding; invalid targets, format, settings, Activate, or mappings; an image that the template can't serve; a scratch that isn't ext4 or isn't fresh; a root that does not assemble from them | [`Request`](Outcome::Request) | No | +/// | A supplied file with the wrong identity, length, or geometry | [`Request`](Outcome::Request), or the host's [`Staging`](Outcome::Staging) once the caller proves it | No | +/// | An endpoint or forward that cannot be bound; resource exhaustion; a destination that cannot restore the template, such as an incompatible CPU or TSC; another runtime; another failure of the host's own operations | [`Host`](Outcome::Host) | No | +/// | A lost status; a timeout; a reset; a forced kill; a broken protocol | [`Unknown`](Outcome::Unknown): the host, the protocol, or neither | No | +/// | A malformed or inconsistent generation; a diagnosed capture or repair defect attributable to it | [`Template`](Outcome::Template) | Yes | +/// | A repair status alone, 193, 194, and 195 included | [`Repair`](Outcome::Repair): the repair stage, whose provenance is still to establish | No | +/// +/// Only a failure of the template quarantines the template's generation, through the generation's +/// tombstone, as [`CloneFailure::quarantine`] does: no other evidence shows that the generation, +/// rather than the request, the host, or the session, failed the clone, and a quarantine changes +/// nothing of the generation's payload or snapshot, which its other clones keep restoring. +/// +/// The host sees the guest of a clone through its control session, and through the status with +/// which its VM ends, which OpenVMM's outcome report gives: a clone's restore discards the +/// guest's console, so no diagnostic tells where a status came from. [`status_evidence`] +/// attributes a status alone, as [`nvx::protocol::status`](crate::protocol::status) classes it: +/// +/// - a status of the time ABI or of a clone's repair is evidence of the repair stage, at any point, +/// since the repair's deferred work may end a clone that is ready, and its provenance is still to +/// establish, since a workload may end its VM with any status; +/// - a status of a clone's activation is evidence of its class while no workload can have run, +/// before the start of the clone's sandbox was sent, and none after; +/// - no other status is evidence. +/// +/// A status that is evidence prevails over what the host or the caller saw of the same end: the +/// loss of the session, the passing of a deadline, or the VMM's exit only show that the clone +/// ended, and a step that fails as the guest ends the clone may fail because it ended. It does not +/// prevail over evidence of the template, a check of the generation or a diagnosis that attributes +/// a defect to it, which shows the provenance that the status alone leaves to establish, and the +/// failure's record holds the status beside it. +/// +/// The host sees the failures of OpenVMM itself through the line with which OpenVMM reports the +/// error that ended it, which the launch's log keeps, and which leads with the error's time ABI +/// code, a spelling that OpenVMM keeps stable for its callers. A VMM that exits before its +/// restore-ready event with a code that says that the host cannot present the template's CPU +/// profile, TSC, or LAPIC timer failed on an [incompatible host](Self::IncompatibleHost). Any +/// other code, such as one of the snapshot's time contract, keeps what the host saw, since it does +/// not tell a defect of the generation from one of this host's OpenVMM, and the failure's message +/// keeps the line. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Evidence { + /// The verdict of the distro policy that the binding of the clone's image holds rejects the + /// image: the request may not run it. + AuthorizationDenied, + /// The binding of the clone's image is stale: it holds a verdict of another revision of the + /// distro policy than the caller accepts, or it is of another version than this build checks, + /// or the image's manifest changed after it was made. + StaleBinding, + /// The request's targets are invalid: a memory target that is not whole blocks of the + /// template's, from its RAM to its capacity, or resources that the repaired clone does not + /// have, as the guest's + /// [`ACTIVATION_RESOURCES`](crate::protocol::status::ACTIVATION_RESOURCES) tells. + InvalidTargets, + /// The request's image is of a format that no clone restores, a GPT disk, or is not of the + /// format and identity that its binding names. + InvalidFormat, + /// The request's settings are invalid: a plan that no clone takes, a path that OpenVMM would + /// not open as checked, a network that the template's NIC does not match, an image or a + /// scratch image that is a file of the sandbox's store, or settings that the guest cannot + /// apply, as its [`ACTIVATION_WORKLOAD`](crate::protocol::status::ACTIVATION_WORKLOAD) and + /// [`ACTIVATION_NETWORK`](crate::protocol::status::ACTIVATION_NETWORK) tell. + InvalidSettings, + /// The request's Activate is invalid: settings that no activation record holds, or a record + /// that the guest refused, as its + /// [`ACTIVATION_RECORD`](crate::protocol::status::ACTIVATION_RECORD) tells. + InvalidActivate, + /// The request's mappings are invalid: the guest cannot mount the host export that they map, + /// as its [`ACTIVATION_EXPORT`](crate::protocol::status::ACTIVATION_EXPORT) tells. + InvalidMappings, + /// The request's image is one that the template cannot serve, as a diagnostic tells. No + /// clone's restore keeps the guest's diagnostics yet, so the guest's status alone, + /// [`UnusableRoot`](Self::UnusableRoot), tells of such an image. + UnservableImage, + /// The request's scratch image is not ext4, or not fresh: another launch holds it, or, as a + /// diagnostic tells, the guest found a filesystem on it. No clone's restore keeps the guest's + /// diagnostics yet, so the guest's status alone, [`UnusableRoot`](Self::UnusableRoot), tells + /// of a filesystem on it. + UnusableScratch, + /// The clone's root does not assemble from the request's image over its upper, as the guest's + /// [`ACTIVATION_ROOT`](crate::protocol::status::ACTIVATION_ROOT) tells: an image that the + /// template cannot serve, or a scratch that is not ext4 or not fresh, which the status alone + /// does not tell apart. + UnusableRoot, + /// A file that the request supplied is not the one that its binding names: it was replaced, + /// changed, or removed after its binding was made, or its path names another file than the + /// check found, or one that others may change. + SuppliedIdentity, + /// A file that the request supplied has another length than the template's device. + SuppliedLength, + /// A file that the request supplied presents other block sizes than the template's device, or + /// a geometry that OpenVMM refuses. + SuppliedGeometry, + /// An endpoint of the clone's launch cannot be bound: the endpoint of the VMM's restore-ready + /// event, or a control socket whose path a Unix socket's address, or OpenVMM's option + /// `listen=`, cannot hold. + EndpointBind, + /// A forward of the clone's network cannot be bound. No clone's network forwards a port yet. + ForwardBind, + /// The host ran out of a resource, such as its storage, its memory, or its descriptors. + ResourceExhaustion, + /// The host cannot restore the template: this build speaks other contracts than the template, + /// or reads no key of the template's encoding version, or the sandbox's filesystem presents + /// another geometry than the template's scratch, or OpenVMM refused to restore the snapshot + /// with a time ABI code that says that the host cannot present the template's CPU profile, + /// TSC, or LAPIC timer, as [the attribution](Self#attribution) describes. + IncompatibleHost, + /// The clone runs, or would launch, under another runtime identity than the template's or the + /// caller's, which the clone's guest might not speak. + RuntimeMismatch, + /// Another failure of the host's own operations: an I/O error of its files, a store whose + /// records are inconsistent, a clock that cannot hold a deadline, or a caller that drives the + /// clone through another lease or lock than the clone's, or launches it again once it failed. + HostFailure, + /// The clone's status was lost: its VMM ended without an outcome report that tells why, as one + /// that is killed does, or the caller of its launch crashed. + LostStatus, + /// A deadline of the clone's restore passed. + Timeout, + /// The adapter lost the clone's session: its reset, the end of its stream, or a failure of its + /// transport. + Reset, + /// The host's recovery terminated the clone's VMM, whose launch's caller had crashed. + ForcedKill, + /// The clone's session broke its protocol: another process at one of the clone's endpoints, + /// bytes that differ from OpenVMM's restore-ready event, a refusal of the adapter's framing, a + /// Ready that does not report the record's activation, or a session that broke the order of + /// the activation, as the guest's + /// [`ACTIVATION_PROTOCOL`](crate::protocol::status::ACTIVATION_PROTOCOL) and + /// [`ACTIVATION_ORDER`](crate::protocol::status::ACTIVATION_ORDER) tell. + Protocol, + /// The leased generation is malformed or inconsistent: the store indexes it with another key + /// than the lease names, its key or snapshot is malformed, its key names its guest's DNS + /// server in a form that OpenVMM does not write, or its payload is not a directory of the + /// current user with mode 0700. + MalformedGeneration, + /// A defect of the generation's capture or repair, which a diagnostic attributes to the + /// generation. No clone's restore keeps the guest's diagnostics yet, so a status of the repair + /// alone is [`RepairStatus`](Self::RepairStatus), and no time ABI code of OpenVMM's tells a + /// defect of the generation from one of the host's OpenVMM. + TemplateDefect, + /// A status of the time ABI or of a clone's repair, alone, which + /// [`FailureRecord::status`] holds. + RepairStatus, +} + +impl Evidence { + /// Returns the outcome to which the evidence attributes a failure, as [the + /// attribution](Self#attribution) describes. + pub const fn outcome(self) -> Outcome { + match self { + Self::AuthorizationDenied + | Self::StaleBinding + | Self::InvalidTargets + | Self::InvalidFormat + | Self::InvalidSettings + | Self::InvalidActivate + | Self::InvalidMappings + | Self::UnservableImage + | Self::UnusableScratch + | Self::UnusableRoot + | Self::SuppliedIdentity + | Self::SuppliedLength + | Self::SuppliedGeometry => Outcome::Request, + Self::EndpointBind + | Self::ForwardBind + | Self::ResourceExhaustion + | Self::IncompatibleHost + | Self::RuntimeMismatch + | Self::HostFailure => Outcome::Host, + Self::LostStatus | Self::Timeout | Self::Reset | Self::ForcedKill | Self::Protocol => { + Outcome::Unknown + } + Self::MalformedGeneration | Self::TemplateDefect => Outcome::Template, + Self::RepairStatus => Outcome::Repair, + } + } + + /// Returns whether the evidence is of a file that the request supplied, which the host's + /// staging may have made wrong, as [`CloneFailure::with_staging_proven`] attributes it once + /// the caller proves it. + pub const fn is_supplied_file(self) -> bool { + matches!( + self, + Self::SuppliedIdentity | Self::SuppliedLength | Self::SuppliedGeometry + ) + } + + /// Returns the evidence's name, in lowercase words joined by hyphens, as its record spells it. + pub const fn as_str(self) -> &'static str { + match self { + Self::AuthorizationDenied => "authorization-denied", + Self::StaleBinding => "stale-binding", + Self::InvalidTargets => "invalid-targets", + Self::InvalidFormat => "invalid-format", + Self::InvalidSettings => "invalid-settings", + Self::InvalidActivate => "invalid-activate", + Self::InvalidMappings => "invalid-mappings", + Self::UnservableImage => "unservable-image", + Self::UnusableScratch => "unusable-scratch", + Self::UnusableRoot => "unusable-root", + Self::SuppliedIdentity => "supplied-identity", + Self::SuppliedLength => "supplied-length", + Self::SuppliedGeometry => "supplied-geometry", + Self::EndpointBind => "endpoint-bind", + Self::ForwardBind => "forward-bind", + Self::ResourceExhaustion => "resource-exhaustion", + Self::IncompatibleHost => "incompatible-host", + Self::RuntimeMismatch => "runtime-mismatch", + Self::HostFailure => "host-failure", + Self::LostStatus => "lost-status", + Self::Timeout => "timeout", + Self::Reset => "reset", + Self::ForcedKill => "forced-kill", + Self::Protocol => "protocol", + Self::MalformedGeneration => "malformed-generation", + Self::TemplateDefect => "template-defect", + Self::RepairStatus => "repair-status", + } + } +} + +impl fmt::Display for Evidence { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// What a failure of a clone is attributed to, by its [`Evidence`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Outcome { + /// The request: its image, upper, targets, or settings. + Request, + /// The host's staging of a file that the request supplied, which the caller proved. + Staging, + /// The host. + Host, + /// The host, the protocol, or neither: the evidence shows no provenance. + Unknown, + /// The leased generation of the template, which [`CloneFailure::quarantine`] quarantines. + Template, + /// The repair stage of the clone, whose provenance is still to establish. + Repair, +} + +impl Outcome { + /// Returns whether a failure of this outcome quarantines the leased generation, through its + /// tombstone: only a failure of the [template](Self::Template) does. + pub const fn quarantines(self) -> bool { + matches!(self, Self::Template) + } + + /// Returns the outcome's name, in lowercase, as its record spells it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Request => "request", + Self::Staging => "staging", + Self::Host => "host", + Self::Unknown => "unknown", + Self::Template => "template", + Self::Repair => "repair", + } + } +} + +impl fmt::Display for Outcome { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// The stage of a clone at which it failed. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Stage { + /// The clone's plan, [`ClonePlan::new`](super::ClonePlan::new), before anything of its launch. + Plan, + /// The clone's launch, [`ClonePlan::launch`](super::ClonePlan::launch), until its VMM runs. + Launch, + /// OpenVMM's restore, from the spawn of the VMM until its restore-ready event. + Restore, + /// The clone's repair and activation, from the restore-ready event until the check of the + /// clone's Ready. + Activation, + /// The start of the clone's sandbox, from the check of its Ready until the start's + /// confirmation. + Start, + /// The commit of the clone's launch. + Commit, + /// The run of the clone's VM, from the commit of its launch until the VM exits. + Run, + /// A reattach's check of the clone, [`reattach_clone`](super::reattach_clone). + Reattach, +} + +impl Stage { + /// Returns the stage's name, in lowercase, as its record spells it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Plan => "plan", + Self::Launch => "launch", + Self::Restore => "restore", + Self::Activation => "activation", + Self::Start => "start", + Self::Commit => "commit", + Self::Run => "run", + Self::Reattach => "reattach", + } + } +} + +impl fmt::Display for Stage { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// How far a clone got, in the order of its steps: the latest step that it reached. +/// +/// The steps from the spawn of the clone's VMM to the start's confirmation are those that its +/// launch and restore sequence record in its progress log, +/// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), as each is reached; the +/// store's records tell the spawn too, and the commit: the launch's journal records the spawn of +/// its VMM until the launch ends, and the runtime record its commit. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum Progress { + /// The clone's plan: nothing of its launch exists. + Planned, + /// The launch wrote the clone's files, and may have begun its journal: no VMM was spawned. + Launching, + /// The launch's VMM was spawned, which its journal records, unless the launch could not record + /// it, and then ended it through its handle. + Spawned, + /// The VMM's restore-ready event was read: the restored VM runs. + Event, + /// A session of the adapter's framing attached: the guest's repair acknowledged the restore, + /// before which OpenVMM reads none of the host's input. + Acknowledged, + /// The Activate of the clone's record was sent. + Activating, + /// The clone's Ready was checked: the clone is activated, with the record's transaction. + Ready, + /// The start of the clone's sandbox was sent. + Starting, + /// The start was confirmed: the clone's sandbox started. + Started, + /// The launch committed: the runtime record names the clone's VM. + Committed, +} + +impl Progress { + /// Returns the stage of a clone that fails once it got this far: the stage of the step that + /// follows. + pub const fn stage(self) -> Stage { + match self { + Self::Planned => Stage::Plan, + Self::Launching => Stage::Launch, + Self::Spawned => Stage::Restore, + Self::Event | Self::Acknowledged | Self::Activating => Stage::Activation, + Self::Ready | Self::Starting => Stage::Start, + Self::Started => Stage::Commit, + Self::Committed => Stage::Run, + } + } + + /// Returns the step's name, in lowercase, as its records spell it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Planned => "planned", + Self::Launching => "launching", + Self::Spawned => "spawned", + Self::Event => "event", + Self::Acknowledged => "acknowledged", + Self::Activating => "activating", + Self::Ready => "ready", + Self::Starting => "starting", + Self::Started => "started", + Self::Committed => "committed", + } + } + + /// Returns whether a workload may have run in the clone once it got this far: once the start + /// of its sandbox was sent. + pub const fn may_have_run(self) -> bool { + matches!(self, Self::Starting | Self::Started | Self::Committed) + } +} + +impl fmt::Display for Progress { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// How the teardown that follows a failure of a clone ended. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Teardown { + /// The clone failed before its VMM was spawned: no VM runs. + NoVm, + /// No VMM of the clone runs: the VMM that the launch spawned has exited, as its verified + /// handle proved, or a launch's claim on its log, which no process holds, proved for the + /// launch, whether it spawned a VMM or not. The store's records of the VMM are cleared. + Exited, + /// The teardown terminated the clone's VMM through its verified identity, or through its + /// handle, a VMM that the launch could not record, and confirmed its exit, and the store's + /// records of it are cleared. + Terminated, + /// Whether the clone's VMM still runs cannot be determined, for the reason that this gives: + /// its identity or its exit is indeterminate. The launch's journal or the runtime record + /// remains, and with it the clone's lease, which a release refuses to end until a reattach + /// resolves the teardown. A record that the clone's sandbox keeps holds at most 1024 + /// characters of the reason, as it holds at most 8192 of the + /// [message](FailureRecord::message). + Unresolved(String), + /// The teardown had not ended when the failure was recorded: the record that a failure writes + /// before its teardown, so that a crash of the caller during the teardown leaves it, which the + /// record that the failure writes once its teardown ended replaces. A reattach that finds it + /// finishes the failure, as [`reattach_clone`](super::reattach_clone) describes, so no + /// [`CloneFailure`] that a step of a clone returns has it. + Pending, +} + +impl Teardown { + /// Returns whether the teardown is resolved: whether no VM of the clone runs. + pub fn is_resolved(&self) -> bool { + !matches!(self, Self::Unresolved(_) | Self::Pending) + } + + /// Returns the teardown of a launch that [`Starting::abandon`](crate::host::supervisor::Starting::abandon), + /// or a recovery, ended as `ended` tells. + pub(super) fn of(ended: Result) -> Self { + match ended { + Ok(Recovery::Clean | Recovery::Exited) => Self::Exited, + Ok(Recovery::Terminated(_)) => Self::Terminated, + Ok(Recovery::Indeterminate(reason)) => Self::Unresolved(reason), + Err(error) => Self::Unresolved(error.to_string()), + } + } + + /// Returns the teardown's name, in lowercase words joined by hyphens, as its record spells it. + pub const fn as_str(&self) -> &'static str { + match self { + Self::NoVm => "no-vm", + Self::Exited => "exited", + Self::Terminated => "terminated", + Self::Unresolved(_) => "unresolved", + Self::Pending => "pending", + } + } +} + +/// The record of a failure of a clone: what the failure was about, where and how it failed, and +/// how its teardown ended, as [`CloneFailure::record`] returns it. +/// +/// # Records +/// +/// A failure from the clone's launch on is recorded in the clone's sandbox, as +/// [`CLONE_FAILURE_NAME`](crate::host::store::CLONE_FAILURE_NAME): before the teardown that +/// follows it, whose record is [pending](Teardown::Pending) until then, so that a crash of the +/// caller during the teardown leaves the record, which a reattach finishes, and once its teardown +/// is known, so that the record holds the stage, the evidence and the status, the original +/// outcome, the transaction of the clone's restore, the leased generation, the clone's progress, +/// and the teardown's outcome. A failure that cannot be recorded before its teardown removes +/// nothing that tells that the clone launched: the journal of its launch, or the runtime record of +/// its VM, remains, and with it the lease, so that a reattach still finds that the clone failed, +/// though not how, if a crash of the caller comes before the record. A failure of the plan, before +/// the launch, leaves no VM and no record: the caller holds the failure, and releases the lease. A +/// launch keeps its lease, and +/// with it the leased generation, until its VM has exited, so a failure that comes after the +/// clone's Ready, such as one of the repair's deferred work, is classified against that +/// generation all the same, and so is the status with which the guest ended a VM whose exit a +/// reattach confirms only later. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct FailureRecord { + /// The identity of the template store that leased the clone's generation. + pub store: StoreId, + /// The identity of the clone's lease. + pub lease: LeaseId, + /// The leased generation, and the identity of its key. + pub template: TemplateReference, + /// The launch that the lease reserves. + pub launch: LaunchId, + /// The transaction ID of the clone's restore, once its plan drew it. + pub transaction: Option, + /// The stage at which the clone failed. + pub stage: Stage, + /// How far the clone got: its acknowledgment and the progress of its activation included. + pub progress: Progress, + /// The evidence of the failure. + pub evidence: Evidence, + /// The status with which the guest ended the clone's VM, as OpenVMM's outcome report gives + /// it, if the report has one. + pub status: Option, + /// The original outcome of the failure, to which its evidence attributes it, whatever the + /// teardown that followed it, and the status with which the guest ended the VM, once the VMM + /// exited. + pub outcome: Outcome, + /// How the teardown that followed the failure ended. + pub teardown: Teardown, + /// The code of the failure's error. + pub code: ErrorCode, + /// The message of the failure's error, followed by the messages of its sources. A record that + /// the clone's sandbox keeps holds at most 8192 of its characters: of a longer message, as a + /// caller's diagnosis may be, its first and its last ones, with an ellipsis between them. + pub message: String, +} + +impl FailureRecord { + /// Returns the record of a failure of the clone that holds `lease`, with `transaction`, at + /// the stage and the progress `at`, with `evidence`, the guest's `status`, and `teardown`, + /// whose error is `error`. + pub(super) fn new( + lease: &Lease, + transaction: Option, + (stage, progress): (Stage, Progress), + evidence: Evidence, + status: Option, + teardown: Teardown, + error: &Error, + ) -> Self { + Self { + store: lease.store(), + lease: lease.id(), + template: lease.template(), + launch: lease.launch().clone(), + transaction, + stage, + progress, + evidence, + status, + outcome: evidence.outcome(), + teardown, + code: error.code(), + message: describe(error), + } + } + + /// Returns the error that the record describes, with its code and its message. + pub fn error(&self) -> Error { + Error::new(self.code, self.message.clone()) + } +} + +/// A failure of a clone: the error of the step that failed, with the [`FailureRecord`] that +/// attributes it, as the clone's plan, launch, restore sequence, and reattach return it. +/// +/// It displays as its error, whose [`kind`](Self::kind) and [`code`](Self::code) it has, and +/// converts into it, without the record. A failure from the clone's launch on is recorded in the +/// clone's sandbox as well, as [the records of failures](FailureRecord#records) describe. +#[derive(Debug)] +pub struct CloneFailure { + /// The error of the step that failed. + error: Error, + /// The failure's record. + record: Box, +} + +impl CloneFailure { + /// Returns the failure of `error`, which `record` attributes. + pub(super) fn new(error: Error, record: FailureRecord) -> Self { + Self { + error, + record: Box::new(record), + } + } + + /// Returns the error of the step that failed. + pub fn error(&self) -> &Error { + &self.error + } + + /// Returns the error of the step that failed, without the record. + pub fn into_error(self) -> Error { + self.error + } + + /// Returns the error's kind. + pub fn kind(&self) -> ErrorKind { + self.error.kind() + } + + /// Returns the error's code. + pub fn code(&self) -> ErrorCode { + self.error.code() + } + + /// Returns the error's message. + pub fn message(&self) -> &str { + self.error.message() + } + + /// Returns the failure's record. + pub fn record(&self) -> &FailureRecord { + &self.record + } + + /// Returns the outcome to which the failure is attributed. + pub fn outcome(&self) -> Outcome { + self.record.outcome + } + + /// Returns how the teardown that followed the failure ended. + pub fn teardown(&self) -> &Teardown { + &self.record.teardown + } + + /// Returns the failure attributed to the host's staging, once the caller proved that its + /// staging, rather than the request, made a file that the request supplied wrong: a failure + /// whose evidence is a [supplied file](Evidence::is_supplied_file) with the wrong identity, + /// length, or geometry, as the plan finds it, which records nothing. Any other failure is + /// returned unchanged. + pub fn with_staging_proven(mut self) -> Self { + if self.record.evidence.is_supplied_file() { + self.record.outcome = Outcome::Staging; + } + self + } + + /// Quarantines the leased generation through its tombstone, as + /// [`Store::tombstone`] does, waiting for the store's lock until `deadline`, or for as long as + /// it takes without one, if the failure is the [template's](Outcome::Template), and returns + /// whether it did. A failure of any other outcome quarantines nothing. + /// + /// The tombstone changes nothing of the generation's payload or snapshot: the generation takes + /// no new lease, while the clones that lease it keep running, and a retirement deletes it once + /// they have released it. The store's lock comes before a sandbox's in the + /// [lock order](crate::host::supervisor#the-lock-order), so the caller quarantines once it + /// released the lock of the clone's sandbox, such as the one that an + /// [`Acquired`](crate::host::template::Acquired) holds. + /// + /// Fails with [`ErrorKind::Other`] for a store of another identity than the one that leased + /// the generation, and as [`Store::tombstone`] fails. + pub fn quarantine(&self, store: &Store, deadline: Option) -> Result { + if !self.record.outcome.quarantines() { + return Ok(false); + } + if store.id() != self.record.store { + return Err(Error::other(format!( + "the generation {} that a failure quarantines is one of the template store {}, \ + not of this one, {}", + self.record.template.generation, + self.record.store, + store.id() + ))); + } + store.tombstone(self.record.template.generation, deadline)?; + Ok(true) + } +} + +impl fmt::Display for CloneFailure { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(&self.error, formatter) + } +} + +impl std::error::Error for CloneFailure { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + std::error::Error::source(&self.error) + } +} + +impl From for Error { + fn from(failure: CloneFailure) -> Self { + failure.error + } +} + +/// The error of a step of a clone, with the evidence that attributes it, before the stage at which +/// the clone failed, its progress, and the teardown that follows are known. +#[derive(Debug)] +pub(super) struct Fault { + /// The evidence of the failure. + pub(super) evidence: Evidence, + /// The step's error. + pub(super) error: Error, +} + +impl Fault { + /// Returns the fault of `error`, which `evidence` attributes. + pub(super) fn new(evidence: Evidence, error: Error) -> Self { + Self { evidence, error } + } + + /// Returns the fault of `error`, a failure of the host's own operations, as + /// [`host_evidence`] attributes it. + pub(super) fn host(error: Error) -> Self { + Self::new(host_evidence(&error), error) + } +} + +/// Attributes the error of a step of a clone. +pub(super) trait Attribute { + /// Returns the result with its error attributed by `evidence`. + fn attribute(self, evidence: Evidence) -> std::result::Result; + + /// Returns the result with its error attributed as a failure of the host's own operations, as + /// [`host_evidence`] attributes it. + fn on_host(self) -> std::result::Result; +} + +impl Attribute for Result { + fn attribute(self, evidence: Evidence) -> std::result::Result { + self.map_err(|error| Fault::new(evidence, error)) + } + + fn on_host(self) -> std::result::Result { + self.map_err(Fault::host) + } +} + +/// Returns the evidence of `error`, a failure of the host's own operations: resource exhaustion +/// if an I/O error among its sources says that the host ran out of storage, memory, or +/// descriptors, and a host failure otherwise. +pub(super) fn host_evidence(error: &Error) -> Evidence { + let mut source = std::error::Error::source(error); + while let Some(cause) = source { + if let Some(io) = cause.downcast_ref::() + && exhausted(io) + { + return Evidence::ResourceExhaustion; + } + source = cause.source(); + } + Evidence::HostFailure +} + +/// Returns whether `error` says that the host ran out of a resource. +fn exhausted(error: &io::Error) -> bool { + // Too many open files, in the system or in the process. + #[cfg(unix)] + if [nix::errno::Errno::ENFILE, nix::errno::Errno::EMFILE] + .into_iter() + .any(|errno| error.raw_os_error() == Some(errno as i32)) + { + return true; + } + matches!( + error.kind(), + io::ErrorKind::StorageFull + | io::ErrorKind::QuotaExceeded + | io::ErrorKind::FileTooLarge + | io::ErrorKind::OutOfMemory + ) +} + +/// Returns the message of `error`, followed by the messages of its sources, as the record of a +/// failure keeps it. +pub(super) fn describe(error: &Error) -> String { + let mut text = error.to_string(); + let mut source = std::error::Error::source(error); + while let Some(cause) = source { + text.push_str(": "); + text.push_str(&cause.to_string()); + source = cause.source(); + } + text +} + +/// Returns `error` with `addition` after its message, with its code and its source. +pub(super) fn amended(error: Error, addition: &str) -> Error { + let code = error.code(); + let (_, message, source) = error.into_parts(); + let error = Error::new(code, format!("{message}{addition}")); + match source { + Some(source) => error.with_source(source), + None => error, + } +} + +/// Returns the evidence of `status`, with which the guest ended a clone's VM once the clone got +/// as far as `progress`, if the status alone is evidence, as [the attribution of +/// evidence](Evidence#attribution) describes: a status of the time ABI or of a clone's repair is +/// evidence of the repair stage at any point, a status of a clone's activation is evidence of its +/// class before the start of the clone's sandbox was sent, and no other status is evidence. +pub fn status_evidence(status: u8, progress: Progress) -> Option { + if status::TIME_STATUSES.contains(&status) || status::REPAIR_STATUSES.contains(&status) { + return Some(Evidence::RepairStatus); + } + if progress.may_have_run() { + return None; + } + Some(match status { + status::ACTIVATION_DEADLINE => Evidence::Timeout, + status::ACTIVATION_PROTOCOL | status::ACTIVATION_ORDER => Evidence::Protocol, + status::ACTIVATION_RECORD => Evidence::InvalidActivate, + status::ACTIVATION_RESOURCES => Evidence::InvalidTargets, + status::ACTIVATION_ROOT => Evidence::UnusableRoot, + status::ACTIVATION_WORKLOAD | status::ACTIVATION_NETWORK => Evidence::InvalidSettings, + status::ACTIVATION_EXPORT => Evidence::InvalidMappings, + _ => return None, + }) +} + +/// Returns the guest's status in OpenVMM's outcome report `report` of a clone's VM, and the +/// evidence that the status alone is once the clone got as far as `progress`, as +/// [`status_evidence`] gives it: the status of a report whose category is `guest-exit` or +/// `success`, the guest's own end, and none for any other, such as a failure of the VMM. +pub(super) fn reported( + report: Option<&Report>, + progress: Progress, +) -> (Option, Option) { + let Some(report) = report else { + return (None, None); + }; + let status = match (report.category.as_deref(), report.status_code) { + (Some("guest-exit" | "success"), Some(code)) => u8::try_from(code).ok(), + _ => None, + }; + let evidence = status.and_then(|status| status_evidence(status, progress)); + (evidence, status) +} + +/// Returns the evidence of a failure whose step saw `evidence`, once the guest's status, which +/// ended the clone's VM, is `reported`, as [`reported`] gives it, as [the attribution of +/// evidence](Evidence#attribution) describes: the status's evidence prevails, unless `evidence` +/// is of the template, whose provenance the status alone does not show. +pub(super) fn prevailing(evidence: Evidence, reported: Option) -> Evidence { + match reported { + Some(reported) if evidence.outcome() != Outcome::Template => reported, + _ => evidence, + } +} + +/// The start of the line with which OpenVMM reports, on its standard error, which the launch's log +/// keeps, the error that ended it: the line goes on with the error's first time ABI code in +/// brackets, such as `[E_TSC_RATE_TOLERANCE]`, if the error's chain has one. +pub(super) const FATAL_PREFIX: &str = "fatal error: "; + +/// The time ABI codes with which OpenVMM refuses to restore a snapshot on a host that cannot +/// present the template's CPU or time: a CPU that the template's CPU profile does not serve, a +/// hypervisor that cannot present the profile, as OpenVMM groups the failures of a profile, and a +/// TSC or a LAPIC timer that the snapshot's time contract does not accept. +const INCOMPATIBLE_HOST_CODES: [&str; 13] = [ + "E_PROFILE_HOST_UNKNOWN", + "E_CPU_GENERATION", + "E_PROFILE_UNSUPPORTED", + "E_PROFILE_TIME_BITS", + "E_CPU_SURFACE", + "E_CPU_UNLISTED", + "E_IDENTITY_ROUTING", + "E_TSC_SYNC_UNSUPPORTED", + "E_TSC_SCALING_ACTIVE", + "E_TSC_RATE_UNAVAILABLE", + "E_TSC_RATE_TOLERANCE", + "E_LAPIC_RATE_UNAVAILABLE", + "E_LAPIC_RATE_MISMATCH", +]; + +/// Returns the evidence of `line`, the line with which OpenVMM reported the error that ended a +/// clone's VMM before its restore-ready event, as [the attribution of +/// evidence](Evidence#attribution) describes: [`Evidence::IncompatibleHost`] for an error whose +/// time ABI code says that the host cannot present the template's CPU or time, and none for any +/// other. +pub(super) fn fatal_evidence(line: &str) -> Option { + let code = line.strip_prefix(FATAL_PREFIX)?.strip_prefix('[')?; + let code = &code[..code.find(']')?]; + INCOMPATIBLE_HOST_CODES + .contains(&code) + .then_some(Evidence::IncompatibleHost) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Every evidence, in the order of the module's table. + const ALL: [Evidence; 27] = [ + Evidence::AuthorizationDenied, + Evidence::StaleBinding, + Evidence::InvalidTargets, + Evidence::InvalidFormat, + Evidence::InvalidSettings, + Evidence::InvalidActivate, + Evidence::InvalidMappings, + Evidence::UnservableImage, + Evidence::UnusableScratch, + Evidence::UnusableRoot, + Evidence::SuppliedIdentity, + Evidence::SuppliedLength, + Evidence::SuppliedGeometry, + Evidence::EndpointBind, + Evidence::ForwardBind, + Evidence::ResourceExhaustion, + Evidence::IncompatibleHost, + Evidence::RuntimeMismatch, + Evidence::HostFailure, + Evidence::LostStatus, + Evidence::Timeout, + Evidence::Reset, + Evidence::ForcedKill, + Evidence::Protocol, + Evidence::MalformedGeneration, + Evidence::TemplateDefect, + Evidence::RepairStatus, + ]; + + #[test] + fn each_row_of_the_table_attributes_its_evidence() { + let rows: [(&[Evidence], Outcome); 6] = [ + ( + &[ + Evidence::AuthorizationDenied, + Evidence::StaleBinding, + Evidence::InvalidTargets, + Evidence::InvalidFormat, + Evidence::InvalidSettings, + Evidence::InvalidActivate, + Evidence::InvalidMappings, + Evidence::UnservableImage, + Evidence::UnusableScratch, + Evidence::UnusableRoot, + ], + Outcome::Request, + ), + ( + &[ + Evidence::SuppliedIdentity, + Evidence::SuppliedLength, + Evidence::SuppliedGeometry, + ], + Outcome::Request, + ), + ( + &[ + Evidence::EndpointBind, + Evidence::ForwardBind, + Evidence::ResourceExhaustion, + Evidence::IncompatibleHost, + Evidence::RuntimeMismatch, + Evidence::HostFailure, + ], + Outcome::Host, + ), + ( + &[ + Evidence::LostStatus, + Evidence::Timeout, + Evidence::Reset, + Evidence::ForcedKill, + Evidence::Protocol, + ], + Outcome::Unknown, + ), + ( + &[Evidence::MalformedGeneration, Evidence::TemplateDefect], + Outcome::Template, + ), + (&[Evidence::RepairStatus], Outcome::Repair), + ]; + let mut seen = Vec::new(); + for (evidence, outcome) in rows { + for evidence in evidence { + assert_eq!(evidence.outcome(), outcome, "{evidence}"); + assert_eq!( + evidence.outcome().quarantines(), + outcome == Outcome::Template, + "{evidence}" + ); + assert_eq!( + evidence.is_supplied_file(), + evidence.as_str().starts_with("supplied-"), + "{evidence}" + ); + seen.push(*evidence); + } + } + // The table attributes every evidence once. + assert_eq!(seen, ALL); + // Only the template's outcome quarantines. + for outcome in [ + Outcome::Request, + Outcome::Staging, + Outcome::Host, + Outcome::Unknown, + Outcome::Repair, + ] { + assert!(!outcome.quarantines(), "{outcome}"); + } + assert!(Outcome::Template.quarantines()); + } + + #[test] + fn names_are_those_of_the_records() { + for evidence in ALL { + let json = serde_json::to_string(&evidence).unwrap(); + assert_eq!(json, format!("\"{evidence}\"")); + assert_eq!(serde_json::from_str::(&json).unwrap(), evidence); + } + let progress = [ + Progress::Planned, + Progress::Launching, + Progress::Spawned, + Progress::Event, + Progress::Acknowledged, + Progress::Activating, + Progress::Ready, + Progress::Starting, + Progress::Started, + Progress::Committed, + ]; + // The steps are in order, and spelled as their records spell them. + assert!(progress.windows(2).all(|pair| pair[0] < pair[1])); + for step in progress { + assert_eq!(serde_json::to_string(&step).unwrap(), format!("\"{step}\"")); + } + let stages: Vec = progress.iter().map(|step| step.stage()).collect(); + assert_eq!( + stages, + [ + Stage::Plan, + Stage::Launch, + Stage::Restore, + Stage::Activation, + Stage::Activation, + Stage::Activation, + Stage::Start, + Stage::Start, + Stage::Commit, + Stage::Run, + ] + ); + for stage in [ + Stage::Plan, + Stage::Launch, + Stage::Restore, + Stage::Activation, + Stage::Start, + Stage::Commit, + Stage::Run, + Stage::Reattach, + ] { + assert_eq!( + serde_json::to_string(&stage).unwrap(), + format!("\"{stage}\"") + ); + } + for outcome in [ + Outcome::Request, + Outcome::Staging, + Outcome::Host, + Outcome::Unknown, + Outcome::Template, + Outcome::Repair, + ] { + assert_eq!( + serde_json::to_string(&outcome).unwrap(), + format!("\"{outcome}\"") + ); + } + } + + #[test] + fn a_repair_status_alone_is_evidence_of_the_repair_stage_at_any_point() { + for status in status::TIME_STATUSES.chain(status::REPAIR_STATUSES) { + for progress in [Progress::Spawned, Progress::Ready, Progress::Committed] { + assert_eq!( + status_evidence(status, progress), + Some(Evidence::RepairStatus), + "{status} {progress}" + ); + } + } + } + + #[test] + fn an_activation_status_is_evidence_of_its_class_until_a_workload_may_have_run() { + let classes = [ + (status::ACTIVATION_DEADLINE, Evidence::Timeout), + (status::ACTIVATION_PROTOCOL, Evidence::Protocol), + (status::ACTIVATION_RECORD, Evidence::InvalidActivate), + (status::ACTIVATION_RESOURCES, Evidence::InvalidTargets), + (status::ACTIVATION_ROOT, Evidence::UnusableRoot), + (status::ACTIVATION_WORKLOAD, Evidence::InvalidSettings), + (status::ACTIVATION_EXPORT, Evidence::InvalidMappings), + (status::ACTIVATION_NETWORK, Evidence::InvalidSettings), + (status::ACTIVATION_ORDER, Evidence::Protocol), + ]; + assert_eq!( + classes.map(|(status, _)| status).to_vec(), + status::ACTIVATION_STATUSES.collect::>() + ); + for (status, evidence) in classes { + for progress in [Progress::Spawned, Progress::Activating, Progress::Ready] { + assert_eq!( + status_evidence(status, progress), + Some(evidence), + "{status} {progress}" + ); + } + // Once the start was sent, a workload may end the VM with any status. + for progress in [Progress::Starting, Progress::Started, Progress::Committed] { + assert_eq!( + status_evidence(status, progress), + None, + "{status} {progress}" + ); + } + } + // No other status is evidence: a success, the generic failure, or a workload's. + for status in [0, 1, 2, 3, 39, 46, 49, 59, 137, 192, 196, 255] { + assert_eq!( + status_evidence(status, Progress::Activating), + None, + "{status}" + ); + } + } + + #[test] + fn a_report_gives_the_guest_s_status_and_its_evidence() { + let report = |category: &str, status_code: i64| Report { + category: Some(category.to_owned()), + status_code: Some(status_code), + teardown_complete: Some(true), + }; + assert_eq!(reported(None, Progress::Event), (None, None)); + assert_eq!( + reported(Some(&report("guest-exit", 195)), Progress::Event), + (Some(Evidence::RepairStatus), Some(195)) + ); + assert_eq!( + reported(Some(&report("guest-exit", 54)), Progress::Activating), + (Some(Evidence::UnusableRoot), Some(54)) + ); + assert_eq!( + reported(Some(&report("guest-exit", 54)), Progress::Committed), + (None, Some(54)) + ); + assert_eq!( + reported(Some(&report("success", 0)), Progress::Committed), + (None, Some(0)) + ); + // A failure of the VMM, or of its teardown, has no status of the guest's. + assert_eq!( + reported(Some(&report("vmm-failure", 1)), Progress::Spawned), + (None, None) + ); + assert_eq!( + reported(Some(&report("teardown-failure", 1)), Progress::Spawned), + (None, None) + ); + assert_eq!( + reported(Some(&report("guest-exit", 256)), Progress::Spawned), + (None, None) + ); + } + + #[test] + fn a_status_prevails_over_what_was_seen_of_the_same_end_but_not_over_the_template_s_evidence() { + let template = [Evidence::MalformedGeneration, Evidence::TemplateDefect]; + for evidence in ALL { + // Without a status that is evidence, what the step saw stands. + assert_eq!(prevailing(evidence, None), evidence, "{evidence}"); + for reported in [ + Evidence::RepairStatus, + Evidence::UnusableRoot, + Evidence::Timeout, + ] { + let prevailed = prevailing(evidence, Some(reported)); + // A check of the generation, or a diagnosis that attributes a defect to it, shows + // the provenance that the status alone leaves to establish. + let expected = if template.contains(&evidence) { + evidence + } else { + reported + }; + assert_eq!(prevailed, expected, "{evidence} {reported}"); + } + } + } + + #[test] + fn openvmm_s_fatal_error_is_evidence_of_an_incompatible_host_only_for_its_codes() { + for code in INCOMPATIBLE_HOST_CODES { + // OpenVMM leads its fatal error with the first time ABI code of the error's chain. + let line = format!("fatal error: [{code}] failed to launch vm worker"); + assert_eq!( + fatal_evidence(&line), + Some(Evidence::IncompatibleHost), + "{line}" + ); + } + // A code of the snapshot's time contract, or of this OpenVMM, does not tell the + // generation's defect from the host's, and an error without a code tells nothing. + for line in [ + "fatal error: [E_MANIFEST_TIME] failed to launch vm worker", + "fatal error: [E_PROFILE_DIGEST] failed to launch vm worker", + "fatal error: [E_SNAPSHOT_VERSION] failed to restore", + "fatal error: failed to open the snapshot", + "fatal error: [E_TSC_RATE_TOLERANCE failed to launch vm worker", + // A line that is not the fatal error, such as a warning, names no code of it. + "NVX-CPU-PROFILE-FALLBACK: [E_PROFILE_UNSUPPORTED] the host profile serves", + "warning: fatal error: [E_CPU_GENERATION] quoted", + ] { + assert_eq!(fatal_evidence(line), None, "{line}"); + } + } + + #[test] + fn host_failures_that_ran_out_of_a_resource_are_resource_exhaustion() { + let failed = |kind: io::ErrorKind| { + host_evidence(&Error::io("cannot write").with_source(io::Error::from(kind))) + }; + assert_eq!( + failed(io::ErrorKind::StorageFull), + Evidence::ResourceExhaustion + ); + assert_eq!( + failed(io::ErrorKind::OutOfMemory), + Evidence::ResourceExhaustion + ); + assert_eq!( + failed(io::ErrorKind::PermissionDenied), + Evidence::HostFailure + ); + assert_eq!( + host_evidence(&Error::io("no source")), + Evidence::HostFailure + ); + // A cause deeper in the chain counts too. + let nested = Error::io("outer").with_source( + Error::io("inner").with_source(io::Error::from(io::ErrorKind::StorageFull)), + ); + assert_eq!(host_evidence(&nested), Evidence::ResourceExhaustion); + } + + #[test] + fn teardowns_follow_the_end_of_the_launch() { + assert_eq!(Teardown::of(Ok(Recovery::Exited)), Teardown::Exited); + assert_eq!(Teardown::of(Ok(Recovery::Clean)), Teardown::Exited); + assert_eq!( + Teardown::of(Ok(Recovery::Indeterminate("held".to_owned()))), + Teardown::Unresolved("held".to_owned()) + ); + let failed = Teardown::of(Err(Error::io("cannot read"))); + assert!(!failed.is_resolved()); + assert!(!Teardown::Pending.is_resolved()); + assert!(Teardown::NoVm.is_resolved()); + assert!(Teardown::Terminated.is_resolved()); + } +} diff --git a/nvx-rs/src/host/template/clone/launch.rs b/nvx-rs/src/host/template/clone/launch.rs index 2354f622b..77e12d5b3 100644 --- a/nvx-rs/src/host/template/clone/launch.rs +++ b/nvx-rs/src/host/template/clone/launch.rs @@ -4,10 +4,17 @@ use std::ffi::{OsStr, OsString}; use std::fmt; +use std::fs; +use std::io; use std::path::{Path, PathBuf}; use std::time::Instant; -use super::{ClonePlan, CloneUpper, MIB, SPARSE_WHAT, check_geometry}; +use super::failure::{ + Attribute, CloneFailure, Evidence, FailureRecord, Fault, Progress, Stage, Teardown, + host_evidence, +}; +use super::records::{End, ProgressLog, Recorder, Tracker, abandon}; +use super::{ClonePlan, CloneUpper, MIB, Mismatch, SPARSE_WHAT, check_geometry}; use crate::activation::ActivationRecord; use crate::error::{Error, Result}; use crate::host::artifacts; @@ -42,6 +49,8 @@ pub(super) struct Hooks<'a> { pub(super) command: &'a dyn for<'h> Fn(VmmCommand<'h>) -> VmmCommand<'h>, /// Fails the launch at a point, as a test asks. pub(super) interrupt: &'a dyn Fn(Point) -> Result<()>, + /// Ends the launch of a clone that failed after its VMM was spawned. + pub(super) end: End, } impl Hooks<'static> { @@ -49,6 +58,7 @@ impl Hooks<'static> { pub(super) const LAUNCH: Self = Self { command: &unchanged, interrupt: &uninterrupted, + end: abandon, }; } @@ -73,26 +83,40 @@ impl ClonePlan { /// launch does. /// /// The launch first checks, without writing anything, that `acquired` holds the plan's lease, - /// under the lock that acquired it, before the launch that it reserves began, and that the + /// under the lock that acquired it, before the launch that it reserves began, that the /// sandbox's clone binding records the lease: the acquisition made the binding durable, and no - /// VMM of the clone starts without it. For an upper in RAM, it checks that the file of the - /// scratch device that the plan created is still there: a later plan under the same - /// acquisition replaces it, and removes it if that plan fails. It then: + /// VMM of the clone starts without it, and that the sandbox records no failure of the clone: + /// a lease's clone fails once, and the record of its failure lasts until the release, even + /// for a launch that failed before it began, which leaves the lock's reservation. For an upper + /// in RAM, it checks that the file of the scratch device that the plan created is still + /// there: a later plan under the same acquisition replaces it, and removes it if that plan + /// fails. It then: /// - /// 1. Writes the clone's activation record, in its canonical encoding, as the file + /// 1. Creates the clone's progress log, + /// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME) in the sandbox's + /// directory, in place of one that an earlier launch left, which it removes through a + /// handle of the directory that holds the sandbox's lock, once it removed the same way the + /// outcome report that an earlier VM of the sandbox left, so that a report beside the + /// clone's records is the clone's VM's. The launch and the restore sequence append to the + /// log each step that the clone reaches, without syncing it, as + /// [`CloneRecords`](super::CloneRecords) describes. + /// 2. Writes the clone's activation record, in its canonical encoding, as the file /// [`CLONE_ACTIVATION_NAME`](crate::host::store::CLONE_ACTIVATION_NAME) in the sandbox's /// directory, in place of one that an earlier launch left, before the VMM starts: the /// record, with the transaction ID of its restore section, is the one that the clone /// activates with, which the caller, or a caller that takes the sandbox over after a /// crash, reads there again and resends unchanged. - /// 2. Listens at the endpoint of the VMM's restore-ready event, which OpenVMM connects to once + /// 3. Listens at the endpoint of the VMM's restore-ready event, which OpenVMM connects to once /// it has prepared the restore, before the restored VM runs, and which must exist before /// the VMM starts: on Linux, the socket /// [`RESTORE_READY_NAME`](crate::host::store::RESTORE_READY_NAME) in the sandbox's /// directory, in place of one that an earlier launch left; on Windows, a named pipe whose /// name the launch draws at random. - /// 3. Runs [`supervisor::launch`] as the launch that the lease reserves, which begins it with - /// the store's launch journal, starts the VMM, and records its identity. + /// 4. Runs [`supervisor::launch`] as the launch that the lease reserves, which begins it with + /// the store's launch journal, starts the VMM, and records its identity in the journal. + /// 5. Appends the spawn to the progress log before it returns, since a teardown of the launch + /// removes the journal, which records the spawn too. A launch that cannot append it fails + /// at the launch stage, and ends the VMM. /// /// The VMM restores the generation's snapshot in place, in the generation's payload, which it /// only reads, and which the launch neither copies nor changes, so any number of clones @@ -132,51 +156,72 @@ impl ClonePlan { /// VMM, if it started, left beside the snapshot. /// /// Like every launch of the sandbox's store, the launch names the sandbox's files by their - /// paths, under its lock: the activation record that it writes, the restore-ready socket at - /// which it listens on Linux, and the journal, capability, and log with which it begins; and - /// the VMM, which runs in the sandbox's directory, opens the files that its arguments name - /// there. So the directories above the sandbox's must be ones that only the current user, or - /// an administrator, may change, as the [store](crate::host::store) requires of them, and as - /// [Block devices by path](crate::host::template::Store#block-devices-by-path) requires of - /// those above the clone's block devices. + /// paths, under its lock: the progress log that it creates, the activation record that it + /// writes, the restore-ready socket at which it listens on Linux, and the journal, capability, + /// and log with which it begins; and the VMM, which runs in the sandbox's directory, opens the + /// files that its arguments name there. So the directories above the sandbox's must be ones + /// that only the current user, or an administrator, may change, as the + /// [store](crate::host::store) requires of them, and as [Block devices by + /// path](crate::host::template::Store#block-devices-by-path) requires of those above the + /// clone's block devices. The launch removes the outcome report of an earlier VM of the + /// sandbox, and the progress log of an earlier launch, through a handle of the directory that + /// holds the sandbox's lock, though, as a release removes the clone's files, and creates its + /// own log at its path, where no file may exist: no change of the path makes the launch remove + /// another sandbox's report or log, and another sandbox's log fails the launch instead, before + /// it writes anything else. /// - /// Fails, and starts no VMM, checking in this order: + /// Fails with a [`CloneFailure`] at [`Stage::Launch`], whose evidence is the host's for a + /// failure of its own operations, as [`ClonePlan::new`] describes it, checking in this order: /// - /// - with [`ErrorKind::Other`](crate::ErrorKind::Other) if `acquired` holds another lease - /// than the plan's, under another lock than the one that acquired it, or after the launch - /// that it reserves began, and if the sandbox's clone binding does not record the plan's - /// lease, and as [`Lease::of`] fails, before anything is written; - /// - with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a runtime identity - /// other than the template's; - /// - for an upper in RAM, with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) if - /// the file of its scratch device, which the plan created, cannot be opened, as when a later - /// plan of the clone replaced it and failed, and with + /// - before anything is written, and so recording nothing, as a plan does: with + /// [`ErrorKind::Other`](crate::ErrorKind::Other) if `acquired` holds another lease than the + /// plan's, under another lock than the one that acquired it, or after the launch that it + /// reserves began, if the sandbox's clone binding does not record the plan's lease, and if + /// the sandbox records a failure of the clone, [`Evidence::HostFailure`], and as + /// [`Lease::of`] fails, or the record of a failure cannot be examined, the host's evidence; + /// with + /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a runtime identity other + /// than the template's, [`Evidence::RuntimeMismatch`]; for an upper in RAM, with + /// [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) if the file of its scratch + /// device, which the plan created, cannot be opened, as when a later plan of the clone + /// replaced it and failed, the host's evidence, and with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if it no longer presents the - /// template's scratch geometry; - /// - on Linux, with [`ErrorKind::Other`](crate::ErrorKind::Other) if the control socket that - /// the lease reserves is not beside the plan's snapshot, as for a plan and a lease that - /// name the template store by different directories; with + /// template's scratch geometry, [`Evidence::IncompatibleHost`]; on Linux, with + /// [`ErrorKind::Other`](crate::ErrorKind::Other) if the control socket that the lease + /// reserves is not beside the plan's snapshot, as for a plan and a lease that name the + /// template store by different directories, [`Evidence::HostFailure`], with /// [`ErrorKind::Io`](crate::ErrorKind::Io) if the generation's payload cannot be resolved - /// or opened, or is not a directory of the current user with mode 0700, as OpenVMM requires - /// of the directory of a control socket; and with + /// or opened, the host's evidence, or is not a directory of the current user with mode + /// 0700, as OpenVMM requires of the directory of a control socket, + /// [`Evidence::MalformedGeneration`], and with /// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if the path of the control /// socket, with or without the payload's links resolved, or of the restore-ready socket, is /// not shorter than 108 bytes, as a Unix socket's address requires, or is not valid UTF-8, /// as OpenVMM and the host take it, or if the path of the control socket holds a comma or an - /// equals sign, which OpenVMM's option `listen=` cannot take; - /// - with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) for an `openvmm` whose - /// path cannot be resolved; - /// - with [`ErrorKind::Io`](crate::ErrorKind::Io) if the activation record cannot be written, - /// or the endpoint of the restore-ready event cannot be opened; - /// - and as [`supervisor::launch`] fails, which ends the launch. + /// equals sign, which OpenVMM's option `listen=` cannot take, + /// [`Evidence::EndpointBind`]; with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) + /// for an `openvmm` whose path cannot be resolved, the host's evidence; with + /// [`ErrorKind::Other`](crate::ErrorKind::Other) if the sandbox's path no longer names the + /// directory that holds its lock, and if the progress log cannot be created, as when the + /// path names a directory that holds one already, the host's evidence; + /// - once the progress log exists, recording the failure in the sandbox, as the + /// [failures](CloneFailure) of a clone's launch are: with + /// [`ErrorKind::Io`](crate::ErrorKind::Io) if the activation record cannot be written, the + /// host's evidence, or the endpoint of the restore-ready event cannot be opened, + /// [`Evidence::EndpointBind`] unless the host ran out of a resource; and as + /// [`supervisor::launch`] fails, which ends the launch, the host's evidence, with the + /// teardown [`Teardown::NoVm`] before the spawn of the VMM, [`Teardown::Terminated`] once + /// the launch ended the VMM that it spawned through its handle, and + /// [`Teardown::Unresolved`] for a launch that could not be ended, whose journal remains. /// - /// A launch that fails leaves the clone's files that it wrote for the release. + /// A launch that fails leaves the clone's files that it wrote, and the record of its failure, + /// for the release. pub fn launch<'s>( self, acquired: &'s mut Acquired, openvmm: &Path, runtime: &RuntimeIdentity, - ) -> Result> { + ) -> Result, CloneFailure> { self.launch_with(acquired, openvmm, runtime, Hooks::LAUNCH) } @@ -187,12 +232,29 @@ impl ClonePlan { openvmm: &Path, runtime: &RuntimeIdentity, hooks: Hooks<'_>, - ) -> Result> { + ) -> Result, CloneFailure> { // The restore-to-Ready deadline of the clone's restore sequence starts here. let launched = Instant::now(); let Acquired { lease, locked } = acquired; - self.check_launch(lease, locked, runtime)?; - let openvmm = artifacts::absolute(openvmm)?; + let planned = self.lease.clone(); + let transaction = self.activation.restore().map(|restore| restore.transaction); + // Until the launch writes a file, the sandbox may not be the clone's, so its failure is + // recorded nowhere. + let unrecorded = |fault: Fault| { + let record = FailureRecord::new( + &planned, + transaction, + (Stage::Launch, Progress::Planned), + fault.evidence, + None, + Teardown::NoVm, + &fault.error, + ); + CloneFailure::new(fault.error, record) + }; + self.check_launch(lease, locked, runtime) + .map_err(unrecorded)?; + let openvmm = artifacts::absolute(openvmm).on_host().map_err(unrecorded)?; let Self { lease: _, generation: _, @@ -205,18 +267,40 @@ impl ClonePlan { capability, devices, } = self; - let devices = device_arguments(&devices)?; + let devices = device_arguments(&devices) + .attribute(Evidence::InvalidSettings) + .map_err(unrecorded)?; + let log = locked + .open_dir() + .and_then(|directory| ProgressLog::create(&directory, locked.dir(), lease.launch())) + .on_host() + .map_err(unrecorded)?; + let recorder = Recorder::new(lease.clone(), transaction, locked); + let mut tracker = Tracker::new(recorder, log, hooks.end); + let unspawned = + |tracker: &Tracker, fault: Fault| tracker.record(Stage::Launch, fault, Teardown::NoVm); let record = locked.clone_activation_path(); - write_private(&record, &activation.encode())?; + write_private(&record, &activation.encode()) + .on_host() + .map_err(|fault| unspawned(&tracker, fault))?; let (ready, listener) = platform::listen_ready(&locked.restore_ready_path()).map_err(|error| { - Error::io(format!( - "cannot listen for the restore-ready event of the clone in {}", - locked.dir().display() - )) - .with_source(error) + let fault = Fault::host( + Error::io(format!( + "cannot listen for the restore-ready event of the clone in {}", + locked.dir().display() + )) + .with_source(error), + ); + let fault = match fault.evidence { + Evidence::HostFailure => Fault::new(Evidence::EndpointBind, fault.error), + _ => fault, + }; + unspawned(&tracker, fault) })?; - (hooks.interrupt)(Point::Prepared)?; + (hooks.interrupt)(Point::Prepared) + .on_host() + .map_err(|fault| unspawned(&tracker, fault))?; // On Windows, this closes the claim's handle, and keeps the image's guard. let claim = match upper { CloneUpper::Scratch(checked) => Some(checked.into_file()), @@ -228,8 +312,15 @@ impl ClonePlan { let claim = match claim { Some(mut file) => { crate::host::sys::move_above_standard_streams(&mut file).map_err(|error| { - Error::io("cannot pass the claim of the clone's scratch image to its VMM") - .with_source(error) + unspawned( + &tracker, + Fault::host( + Error::io( + "cannot pass the claim of the clone's scratch image to its VMM", + ) + .with_source(error), + ), + ) })?; Some(file) } @@ -245,17 +336,48 @@ impl ClonePlan { }; let dir = locked.dir().to_owned(); let inherit: Vec> = claim.iter().map(Inherited::new).collect(); - let starting = - supervisor::launch(locked, lease.launch(), runtime, &capability, |endpoint| { + let starting = supervisor::launch_reporting( + locked, + lease.launch(), + runtime, + &capability, + |endpoint| { (hooks.interrupt)(Point::Begun)?; let mut command = VmmCommand::new(&openvmm, restore.arguments(endpoint), &dir); command.inherit = inherit; Ok((hooks.command)(command)) - }); + }, + |error, spawned| { + // A VMM that the launch spawned, and could not record, got the clone that far, + // which only the record of its failure tells once the launch has ended it. + if spawned { + tracker.advance(Progress::Spawned); + } + // A failure that cannot be recorded keeps the launch's journal. + tracker.pending(Stage::Launch, host_evidence(error), error) + }, + ); // The VMM holds the claim, if it started. drop(claim); + let starting = match starting { + Ok(starting) => starting, + Err(failure) => { + let teardown = match (failure.unended, failure.spawned) { + (Some(reason), _) => Teardown::Unresolved(reason), + (None, true) => Teardown::Terminated, + (None, false) => Teardown::NoVm, + }; + return Err(tracker.record(Stage::Launch, Fault::host(failure.error), teardown)); + } + }; + // The progress log records the spawn before the launch returns: the journal records it too, + // but a teardown of the launch removes the journal, and a reattach then takes how far the + // clone got, and so the stage of its failure, from the log. + if let Err(fault) = tracker.reach(Progress::Spawned) { + return Err(tracker.fail(starting, Stage::Launch, fault)); + } Ok(Restoring { - starting: starting?, + starting, launched, ready, listener, @@ -263,6 +385,7 @@ impl ClonePlan { activation, record, capability, + tracker, }) } @@ -275,37 +398,72 @@ impl ClonePlan { lease: &Lease, locked: &LockedStore, runtime: &RuntimeIdentity, - ) -> Result<()> { + ) -> Result<(), Fault> { let planned = &self.lease; if lease != planned || locked.dir() != planned.sandbox() || locked.reserved() != Some(planned.launch()) { - return Err(Error::other(format!( - "a clone is launched under the lock that acquired the lease of its plan, {}, \ - before the launch that the lease reserves begins, but the lock of the sandbox in \ - {} holds the lease {}, and reserves no launch {} of it", - planned.id(), - locked.dir().display(), - lease.id(), - planned.launch() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "a clone is launched under the lock that acquired the lease of its plan, {}, \ + before the launch that the lease reserves begins, but the lock of the \ + sandbox in {} holds the lease {}, and reserves no launch {} of it", + planned.id(), + locked.dir().display(), + lease.id(), + planned.launch() + )), + )); } - if Lease::of(locked)?.as_ref() != Some(planned) { - return Err(Error::other(format!( - "the sandbox in {} records no clone binding of the lease {}, which its \ - acquisition makes durable before any VMM of the clone starts", - locked.dir().display(), - planned.id() - ))); + if Lease::of(locked).on_host()?.as_ref() != Some(planned) { + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the sandbox in {} records no clone binding of the lease {}, which its \ + acquisition makes durable before any VMM of the clone starts", + locked.dir().display(), + planned.id() + )), + )); + } + // A launch that failed before it began leaves the lock's reservation, but the record of + // its failure, which no launch removes, ends the clone's launches until the release. + let failure = locked.clone_failure_path(); + match fs::symlink_metadata(&failure) { + Ok(_) => { + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the clone of the lease {} failed, as {} records: a lease's clone fails \ + once, and the release of the lease ends it", + planned.id(), + failure.display() + )), + )); + } + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + Err(error) => { + return Err(Fault::host( + Error::io(format!( + "cannot examine the record of a failure of the clone {}", + failure.display() + )) + .with_source(error), + )); + } } if runtime != self.key.runtime() { - return Err(Error::unsupported(format!( - "the template's generation {} restores under the runtime that captured it, {:?}, \ - not under {runtime:?}", - self.generation.generation(), - self.key.runtime() - ))); + return Err(Fault::new( + Evidence::RuntimeMismatch, + Error::unsupported(format!( + "the template's generation {} restores under the runtime that captured it, \ + {:?}, not under {runtime:?}", + self.generation.generation(), + self.key.runtime() + )), + )); } if let CloneUpper::Tmpfs { path } = &self.upper { check_sparse(path, self.key.scratch())?; @@ -330,20 +488,25 @@ impl ClonePlan { /// socket. OpenVMM takes the control socket's path in its option `listen=`, among /// options that commas separate, and whose values follow an equals sign, so the path must hold /// neither. - fn check_sockets(&self, socket: &Path, ready: &Path) -> Result<()> { + fn check_sockets(&self, socket: &Path, ready: &Path) -> Result<(), Fault> { let (Some(payload), Some(name)) = (socket.parent(), socket.file_name()) else { - return Err(Error::other(format!( - "the control socket {} of the clone's VMM names no directory", - socket.display() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the control socket {} of the clone's VMM names no directory", + socket.display() + )), + )); }; let resolve = |path: &Path| { filesystem::canonicalize(path).map_err(|error| { - Error::io(format!( - "cannot resolve the payload of the generation {}", - path.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot resolve the payload of the generation {}", + path.display() + )) + .with_source(error), + ) }) }; let canonical = resolve(payload)?; @@ -353,51 +516,68 @@ impl ClonePlan { None => false, }; if !beside { - return Err(Error::other(format!( - "the control socket {} of the clone's VMM, which its lease reserves, is not beside \ - the snapshot {} of its plan, where OpenVMM requires it", - socket.display(), - self.snapshot.display() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::other(format!( + "the control socket {} of the clone's VMM, which its lease reserves, is not \ + beside the snapshot {} of its plan, where OpenVMM requires it", + socket.display(), + self.snapshot.display() + )), + )); } let problem = platform::check_store_dir(payload).map_err(|error| { - Error::io(format!( - "cannot open the payload of the generation {}", - payload.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot open the payload of the generation {}", + payload.display() + )) + .with_source(error), + ) })?; if let Some(problem) = problem { - return Err(Error::io(format!( - "the payload of the generation {} {problem}", - payload.display() - ))); + return Err(Fault::new( + Evidence::MalformedGeneration, + Error::io(format!( + "the payload of the generation {} {problem}", + payload.display() + )), + )); } let resolved = canonical.join(name); for path in [socket, &resolved, ready] { if path.as_os_str().len() >= SOCKET_PATH_LIMIT { - return Err(Error::unsupported(format!( - "the path of the socket {} of a clone's launch must be shorter than \ - {SOCKET_PATH_LIMIT} bytes, as a Unix socket's address requires", - path.display() - ))); + return Err(Fault::new( + Evidence::EndpointBind, + Error::unsupported(format!( + "the path of the socket {} of a clone's launch must be shorter than \ + {SOCKET_PATH_LIMIT} bytes, as a Unix socket's address requires", + path.display() + )), + )); } } for path in [socket, &resolved, ready] { if path.to_str().is_none() { - return Err(Error::unsupported(format!( - "the path of the socket {} of a clone's launch must be valid UTF-8, as \ - OpenVMM takes it", - path.display() - ))); + return Err(Fault::new( + Evidence::EndpointBind, + Error::unsupported(format!( + "the path of the socket {} of a clone's launch must be valid UTF-8, as \ + OpenVMM takes it", + path.display() + )), + )); } } if socket.to_string_lossy().contains([',', '=']) { - return Err(Error::unsupported(format!( - "the path of the control socket {} of a clone's VMM must hold no ',' or '=', \ - which end the path in OpenVMM's option `listen=`", - socket.display() - ))); + return Err(Fault::new( + Evidence::EndpointBind, + Error::unsupported(format!( + "the path of the control socket {} of a clone's VMM must hold no ',' or '=', \ + which end the path in OpenVMM's option `listen=`", + socket.display() + )), + )); } Ok(()) } @@ -406,21 +586,29 @@ impl ClonePlan { /// Refuses the file of the `scratch` block device of a clone whose upper is in RAM, at `path`, /// which the clone's plan created, unless it is still a regular file that presents the template's /// scratch geometry, `scratch`: OpenVMM opens it by its path, and a later plan under the same -/// acquisition replaces it, or removes it if that plan fails. -fn check_sparse(path: &Path, scratch: BlockGeometry) -> Result<()> { +/// acquisition replaces it, or removes it if that plan fails. The file is the host's, as is the +/// evidence of its refusal. +fn check_sparse(path: &Path, scratch: BlockGeometry) -> Result<(), Fault> { let shown = path.display(); let file = open_regular(path).map_err(|error| { - Error::unavailable(format!( - "the {SPARSE_WHAT} {shown}, which the clone's plan created, cannot be opened, as when \ - a later plan of the clone replaced it and failed" - )) - .with_source(error) + Fault::host( + Error::unavailable(format!( + "the {SPARSE_WHAT} {shown}, which the clone's plan created, cannot be opened, as \ + when a later plan of the clone replaced it and failed" + )) + .with_source(error), + ) })?; check_geometry( - block_geometry(&file, path, SPARSE_WHAT)?, + block_geometry(&file, path, SPARSE_WHAT).map_err(|error| match error.kind() { + // The sandbox's filesystem presents a geometry that OpenVMM refuses. + crate::ErrorKind::Unsupported => Fault::new(Evidence::IncompatibleHost, error), + _ => Fault::host(error), + })?, scratch, &format!("the {SPARSE_WHAT} {shown}"), "scratch", + Mismatch::HOST, ) } @@ -521,6 +709,8 @@ pub struct Restoring<'s> { pub(super) record: PathBuf, /// The capability of the launch. pub(super) capability: [u8; CAPABILITY_LEN], + /// How far the clone got, and how its failures are recorded. + pub(super) tracker: Tracker, } impl<'s> Restoring<'s> { diff --git a/nvx-rs/src/host/template/clone/reattach.rs b/nvx-rs/src/host/template/clone/reattach.rs new file mode 100644 index 000000000..c11afa28b --- /dev/null +++ b/nvx-rs/src/host/template/clone/reattach.rs @@ -0,0 +1,638 @@ +//! The reattach of a clone, [`reattach_clone`], once the caller of its launch restarted: it finds +//! the clone's VM by the identity of its VMM's process, takes it over if this host may manage it, +//! and checks the clone's activation in a new session, or tells how the clone ended, from the +//! records of its sandbox. + +use std::path::Path; +use std::time::{Duration, Instant}; + +use super::failure::{ + Attribute, CloneFailure, Evidence, Fault, Progress, Stage, Teardown, amended, describe, +}; +use super::records::{Recorder, progress_of, read_failure_of, read_progress}; +use super::restore::{ActivationSession, CloneControl}; +use crate::activation::{ActivationRecord, MAX_RECORD_BYTES, ReadyActivation}; +use crate::error::{Error, ErrorCode, Result}; +use crate::host::launch::timeout_problem; +use crate::host::process::BootId; +use crate::host::report::Report; +use crate::host::store::files::sync_dir; +use crate::host::store::regular::{open_regular, read_at_most}; +use crate::host::store::{IdentityPart, LockedStore, Recovery, RuntimeIdentity, RuntimeRecord}; +use crate::host::supervisor::{self, Reattached, StopOutcome, Vmm}; +use crate::host::template::Lease; + +/// What [`reattach_clone`] found of a clone. +#[derive(Debug)] +#[non_exhaustive] +pub enum ReattachedClone { + /// The clone's VMM runs under the caller's runtime identity, and the clone answered the + /// Activate of its record, byte for byte, in a new session of the adapter's framing, with the + /// Ready of the record's activation: the caller holds the VM, and the session, in which it + /// drives the clone. + Live { + /// The clone's VM. + vmm: Box, + /// The session in which the clone answered. + session: S, + }, + /// No VM of the clone runs, and the sandbox records no failure of it: its VM exited once its + /// launch committed, with an outcome report of the guest's own end whose status is no evidence + /// of a failure, or its holder stopped it, or its launch ended, or never began, without a + /// failure, as its progress tells. The caller releases the clone's lease. + Ended { + /// How far the clone got. + progress: Progress, + /// OpenVMM's outcome report of the clone's VM, if its VMM wrote one. + report: Option, + }, + /// The clone failed, as the record of its failure in its sandbox tells, which the reattach + /// found, or wrote: the launch of a caller that crashed, which the reattach ended, or which a + /// recovery of the sandbox ended before, whose VM ended with a status of the guest's that is + /// evidence; a VM that exited once its launch committed, with a status of the guest's that is + /// evidence, or without a report of the guest's own end; a check of the clone that failed; a + /// VM that runs under another runtime identity; or a record of a failure of the clone that + /// cannot be read, which the record that the reattach wrote replaces. The failure's teardown + /// tells whether a VM of the clone may still run: until it is resolved, the sandbox keeps the + /// store's records of the VM, and with them the clone's lease, for a later reattach. + Failed(CloneFailure), + /// Whether the clone's VMM, whose launch committed, runs cannot be determined, for the reason + /// that this gives, and the sandbox records no failure of the clone: the runtime record + /// remains, and with it the clone's lease, for a later reattach, which may succeed. + Indeterminate(String), +} + +/// Reattaches to the clone of the sandbox of `locked`, under the runtime identity `runtime`, +/// which the caller would launch the clone's template with now, through the adapter's framing, +/// `control`, which must answer in a new session by `deadline`, and ends a VMM of the clone whose +/// launch committed, and that it may not keep, within `bound`. +/// +/// It reads the records of the clone that the sandbox keeps, as [`CloneRecords`] describes, then +/// reattaches to the sandbox's VM, as [`supervisor::reattach`] does, which ends a launch that a +/// crash of its caller interrupted, through the verified identity of its VMM, as +/// [`LockedStore::recover`] ends it, within the wait that the store's recovery allows the VMM's +/// termination, which `bound` does not shorten, removes a journal that the runtime record of its +/// launch supersedes, as the launch's commit may leave it, which is no failure, and finds a VMM +/// whose launch committed by the identity of its process, which says which VM it is: +/// +/// - A VMM that runs under `runtime`, whose launch is the one that the clone's lease reserves, is +/// the clone's, which the reattach checks in a new session, as the restore sequence of the +/// clone's launch does, [`Restoring::activate`](super::Restoring::activate), sending the +/// sandbox's activation record, byte for byte, which must still be the canonical encoding of a +/// record of the restore of the lease's generation: the clone answers with the Ready that it +/// answered once, and the reattach checks the transaction ID that it reports, which +/// [`ReattachedClone::Live`] returns with the session. A check that fails, by the evidence that +/// the restore sequence gives it, ends the VMM through its verified identity, as +/// [`Vmm::terminate`] does within `bound`, and the clone fails at [`Stage::Reattach`]. +/// - A VMM that runs under another runtime identity, which might not speak the caller's +/// protocol, is ended the same way, without a word to its guest, and the clone fails at +/// [`Stage::Reattach`] with [`Evidence::RuntimeMismatch`], and +/// [`ErrorCode::RuntimeMismatch`]. +/// - A launch that the reattach ended fails at the stage of its progress, with +/// [`ErrorCode::Exited`], and with [`Evidence::ForcedKill`] for a VMM that the reattach +/// terminated, and [`Evidence::LostStatus`] otherwise, which the error that OpenVMM reported, for +/// a VMM that exited before its restore-ready event, attributes as the restore sequence does. One +/// that it cannot end yet fails the same way with [`Evidence::LostStatus`], whose teardown is +/// unresolved: its journal remains, and the lease with it, for a later reattach. +/// - A launch that a recovery of the sandbox ended before the reattach, as [`supervisor::reattach`] +/// ends one, and of which the store holds no record, fails at the stage of its progress, with +/// [`ErrorCode::Exited`], if OpenVMM's outcome report gives a status of the guest's that is +/// evidence, and is [`ReattachedClone::Ended`] otherwise. Once the launch began, which removes +/// the report of an earlier VM of the sandbox before it creates the clone's progress log, a +/// report that the sandbox holds is the clone's VM's, even if no record kept the spawn of its +/// VMM, as a crash of the launch's caller before the journal or the progress log recorded it +/// leaves it. +/// - A VMM whose launch committed, and that exited, fails at [`Stage::Run`] with +/// [`ErrorCode::Exited`] if OpenVMM's outcome report gives a status of the guest's that is +/// evidence, as a status of the clone's repair is at any point, and with +/// [`Evidence::LostStatus`] if no report of the guest's own end tells why it exited, and +/// [`ReattachedClone::Ended`] otherwise. One that cannot be examined is +/// [`ReattachedClone::Indeterminate`]. A VM that its holder's stop ended, which cleared its +/// runtime record, lost no status, but one whose VMM had exited before the stop began is one +/// that exited, as above, since [`Vmm::stop_unless_exited`] keeps its record. +/// +/// The evidence of a status of the guest's prevails over any but the template's, as in the restore +/// sequence, and as [the attribution of evidence](Evidence#attribution) describes. A failure is +/// recorded in the sandbox, as the restore sequence records it: before the reattach ends a VMM of +/// the clone, recovers an interrupted launch, or clears the runtime record of a VM that exited, +/// with its teardown [pending](Teardown::Pending), and again once the teardown ended. A reattach +/// that cannot record it then fails, before it ends or forgets anything of the clone's VM, as does +/// one that cannot make the record of an earlier failure durable. A clone +/// whose sandbox records a failure already failed for good: the reattach ends a VMM of the clone +/// that still runs, through its verified identity, then finishes a failure whose teardown a crash +/// of its caller left pending, as the failure would have recorded it once its teardown ended, or +/// records the teardown of one whose teardown was unresolved in place of the record's, which +/// keeps the stage, the evidence, and the original outcome, unless what the clone's VMM left as +/// it ended since tells more, as the guest's status, which the record does not hold, and the +/// error that OpenVMM reported, for a VMM that exited before its restore-ready event, may, and +/// returns the failure, with the record's lease, transaction, stage, progress, and code. A record +/// of a failure that cannot be read tells as much, though not how the clone failed, and so does +/// a record of another clone's failure, whose template store, lease, leased generation, its key, +/// or launch is not the binding's, and which tells nothing of this clone: if the store holds +/// records of the clone's launch, which the reattach may remove, the reattach first replaces it +/// with the record of its own failure at [`Stage::Reattach`], with the error of the record's read +/// and the host's evidence, its teardown pending, which keeps how far the clone got for a +/// reattach after a crash of the caller. It then ends a VMM of the clone that still runs the same +/// way, and records the failure with the teardown that it found or made, as any failure that it +/// records, over whose evidence a status of the guest's prevails. The failure that it returns is +/// then this clone's, so a [quarantine](CloneFailure::quarantine) of it never reaches a +/// generation that the clone does not lease. +/// +/// The reattach reads no record of the template store: the lease holds the clone's generation, +/// and the sandbox's activation record the clone's transaction, so a tombstone, or a retirement, +/// of the generation leaves the clone's reattach, as its run, as it is. It signals no process that +/// it has not verified. The progress log only informs it: one that cannot be read is none. A +/// clone that launched, whose progress log, which its launch created before it began, and appends +/// to without a sync, cannot be read, is missing, or was written during another boot of the host, +/// whose crash may have lost its last lines, may have run its workload: a status of the guest's +/// is then attributed as once the start of the clone's sandbox was sent. +/// +/// Fails with [`ErrorKind::Other`](crate::ErrorKind::Other) if the sandbox holds no clone +/// binding, whose VM reattaches as any sandbox's does, with +/// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) if `bound` is zero or longer than +/// [`MAX_TIMEOUT`](crate::host::launch::MAX_TIMEOUT), and for an invalid `runtime`, before it +/// writes anything, and as [`Lease::of`] and [`supervisor::reattach`] fail for records that are +/// inconsistent, which leave the clone's VM as it was, and the record of the failure of an +/// interrupted launch, or the one that replaced a record that could not be read, if the reattach +/// wrote it, pending. Fails with [`ErrorKind::Io`](crate::ErrorKind::Io) if it cannot record the +/// clone's failure, or make the record of an earlier one durable, before it would end, or forget, +/// anything of the clone's VM, which it then leaves as it found it, running or not, with the +/// journal of its launch or its runtime record, and the lease, for a later reattach. +/// +/// [`CloneRecords`]: super::CloneRecords +pub fn reattach_clone( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + deadline: Instant, + bound: Duration, +) -> Result> { + reattach_clone_with( + locked, + runtime, + control, + deadline, + bound, + |locked, runtime, exited| supervisor::reattach_reporting(locked, runtime, exited), + ) +} + +/// Reattaches to a clone as [`reattach_clone`] does, reattaching to its VM with `reattach`, which +/// is [`supervisor::reattach_reporting`] outside tests, and which calls its last argument with +/// the runtime record of a VMM that exited before it clears the record, and fails, keeping the +/// record, if that fails. +pub(super) fn reattach_clone_with( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + deadline: Instant, + bound: Duration, + reattach: impl FnOnce( + &mut LockedStore, + &RuntimeIdentity, + &dyn Fn(&RuntimeRecord) -> Result<()>, + ) -> Result, +) -> Result> { + reattach_clone_syncing( + locked, runtime, control, deadline, bound, &sync_dir, reattach, + ) +} + +/// Reattaches to a clone as [`reattach_clone_with`] does, syncing the sandbox's directory, to make +/// the record of an earlier failure durable, with `sync`, which is [`sync_dir`] outside tests. +pub(super) fn reattach_clone_syncing( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + deadline: Instant, + bound: Duration, + sync: &dyn Fn(&Path) -> Result<()>, + reattach: impl FnOnce( + &mut LockedStore, + &RuntimeIdentity, + &dyn Fn(&RuntimeRecord) -> Result<()>, + ) -> Result, +) -> Result> { + let started = Instant::now(); + if let Some(problem) = timeout_problem(bound, true) { + return Err(Error::unsupported(format!( + "the bound of a clone's termination {problem}" + ))); + } + // The reattach fails for an invalid runtime identity before it writes anything. + runtime.check()?; + let Some(lease) = Lease::of(locked)? else { + return Err(Error::other(format!( + "the sandbox in {} holds no clone binding, and so no clone to reattach to", + locked.dir().display() + ))); + }; + let launch = lease.launch(); + let journal = locked + .journal()? + .filter(|journal| journal.launch() == launch); + let runtime_record = locked.runtime()?; + let committed = runtime_record + .as_ref() + .filter(|record| record.launch() == launch); + // The record of an earlier failure of the clone, or the fault of a record that cannot be read, + // or of one of another clone's failure, which still tells that the clone launched, and failed + // for good, though not how. + let earlier = read_failure_of(&locked.clone_failure_path(), &lease) + .map_err(unreadable_failure) + .transpose(); + let logged = read_progress(&locked.clone_progress_path(), launch).unwrap_or(None); + // The launch of a clone creates its progress log before it begins, and before it records a + // failure, and appends to it without a sync, so the log of a clone that launched tells how + // far the clone got for sure only if it can be read, and was written during the current boot + // of the host, whose crash would have ended the clone's VM: a crash of the host may have lost + // the log's last lines, and a workload may then have run in the clone. A log that can be read + // tells that the clone launched as well as the store's records do, which a stop of its VM + // clears, and a release alone removes it. + let launched = + journal.is_some() || committed.is_some() || earlier.is_some() || logged.is_some(); + let certain = logged.is_some_and(|logged| logged.is_certain(BootId::current().ok())); + let uncertain = launched && !certain; + let logged = logged.map(|logged| logged.progress); + let progress = progress_of(&lease, journal.as_ref(), committed, logged); + let activation = read_activation(&locked.clone_activation_path(), &lease); + // A failure that the reattach records has the transaction that the clone's activation record + // names, while the record of an earlier failure keeps its own. + let transaction = activation + .as_ref() + .ok() + .and_then(|(_, record)| record.restore().map(|restore| restore.transaction)); + let recorder = Recorder::new(lease.clone(), transaction, locked).uncertain(uncertain); + let fresh = earlier.is_none(); + // A record of an earlier failure may be in place without being durable, as a write whose sync + // of the sandbox's directory failed leaves it: the reattach makes it durable before it ends, + // or forgets, anything that tells that the clone launched, as it records a failure of its own. + if let Some(Ok(_)) = &earlier + && (journal.is_some() || committed.is_some()) + { + sync(locked.dir()).map_err(|error| { + unrecorded(Error::io(format!( + "the record of the clone's failure could not be made durable before its \ + teardown: {}", + describe(&error) + ))) + })?; + } + // A record that cannot be read, or that is another clone's, is replaced by the reattach's own + // failure, with its teardown pending, before the reattach ends or forgets anything of the + // clone's launch: the record keeps how far the clone got, which the launch's journal and + // runtime record hold, and the reattach removes, for a reattach after a crash of this caller, + // which finishes it. Without them, the reattach removes nothing that the record would keep, and + // a later reattach finds the clone as this one does. + if let Some(Err(fault)) = &earlier + && (journal.is_some() || committed.is_some()) + { + recorder + .pending( + Stage::Reattach, + progress, + (fault.evidence, None), + &fault.error, + ) + .map_err(unrecorded)?; + } + // A launch that the journal names, which no runtime record supersedes, and whose failure the + // sandbox does not record, was interrupted by a crash of its caller: its failure is recorded + // before the recovery ends it, so that a crash of this caller during the recovery leaves the + // record. A journal that the runtime record of its launch supersedes, which the launch's + // commit may leave, is no failure: the recovery removes it. + if fresh && journal.is_some() && runtime_record.is_none() { + recorder + .pending( + progress.stage(), + progress, + (Evidence::LostStatus, None), + &interrupted(progress), + ) + .map_err(unrecorded)?; + } + // So is the failure of a VM whose launch committed, and which exited, before the reattach + // clears its runtime record, as the outcome report that the recorder reads then tells, which + // the rest of the reattach takes as it found it. + let exiting = |_: &RuntimeRecord| { + if fresh && let Some((fault, status)) = exit_fault(&recorder, progress, true) { + let evidence = (fault.evidence, status); + return recorder + .pending(progress.stage(), progress, evidence, &fault.error) + .map_err(unrecorded); + } + Ok(()) + }; + let reattached = reattach(locked, runtime, &exiting)?; + let failed = |fault: Fault, stage: Stage, teardown: Teardown| { + let fault = Fault::new(fault.evidence, unended(fault.error, &teardown)); + Ok(ReattachedClone::Failed( + recorder.record(stage, progress, fault, teardown), + )) + }; + if let Some(earlier) = earlier { + let teardown = match reattached { + Reattached::Live(vmm) | Reattached::Mismatched { vmm, .. } => { + stopped(vmm.terminate(locked, bound), bound) + } + Reattached::Exited { + recovery: recovery @ (Recovery::Exited | Recovery::Terminated(_)), + .. + } => Teardown::of(Ok(recovery)), + Reattached::Exited { + record: Some(_), .. + } => Teardown::Exited, + // Nothing of the clone remains: a teardown that was not resolved is now. + Reattached::Exited { record: None, .. } => match &earlier { + Ok(earlier) => match earlier.teardown { + Teardown::Unresolved(_) | Teardown::Pending => Teardown::Exited, + ref teardown => teardown.clone(), + }, + // A VM of the clone ran once its launch spawned it, and so did one whose report the + // sandbox holds once the launch began, as in `exit_fault`, though no record kept + // its spawn. + Err(_) + if progress >= Progress::Spawned + || (progress > Progress::Planned && recorder.observed().is_some()) => + { + Teardown::Exited + } + Err(_) => Teardown::NoVm, + }, + Reattached::Indeterminate(reason) => Teardown::Unresolved(reason), + }; + let earlier = match earlier { + Ok(earlier) => earlier, + // The record cannot tell how the clone failed: the failure is the reattach's own. + Err(fault) => return failed(fault, Stage::Reattach, teardown), + }; + let failure = if earlier.teardown == Teardown::Pending { + // The crash of the caller during the teardown left the failure pending, which is + // recorded as the failure would have recorded it once its teardown ended. + let error = unended(earlier.error(), &teardown); + recorder.finish(earlier, error, teardown) + } else { + recorder.resolve(earlier, teardown) + }; + return Ok(ReattachedClone::Failed(failure)); + } + match reattached { + Reattached::Live(vmm) => { + let checked = activation.and_then(|activation| { + check_activation(locked, &lease, &vmm, control, activation, deadline, started) + }); + match checked { + Ok(session) => Ok(ReattachedClone::Live { + vmm: Box::new(vmm), + session, + }), + Err(fault) => { + let evidence = (fault.evidence, None); + recorder + .pending(Stage::Reattach, progress, evidence, &fault.error) + .map_err(unrecorded)?; + let teardown = stopped(vmm.terminate(locked, bound), bound); + failed(fault, Stage::Reattach, teardown) + } + } + } + Reattached::Mismatched { vmm, differences } => { + let fault = Fault::new(Evidence::RuntimeMismatch, mismatched(&differences)); + let evidence = (fault.evidence, None); + recorder + .pending(Stage::Reattach, progress, evidence, &fault.error) + .map_err(unrecorded)?; + let teardown = stopped(vmm.terminate(locked, bound), bound); + failed(fault, Stage::Reattach, teardown) + } + Reattached::Exited { + recovery: recovery @ (Recovery::Exited | Recovery::Terminated(_)), + .. + } => { + let evidence = match recovery { + Recovery::Terminated(_) => Evidence::ForcedKill, + _ => Evidence::LostStatus, + }; + let fault = Fault::new(evidence, interrupted(progress)); + failed(fault, progress.stage(), Teardown::of(Ok(recovery))) + } + Reattached::Exited { record, .. } => { + match exit_fault(&recorder, progress, record.is_some()) { + Some((fault, _)) => failed(fault, progress.stage(), Teardown::Exited), + // A report is of the clone's VM only once its launch began, as in `exit_fault`. + None => Ok(ReattachedClone::Ended { + progress, + report: (progress > Progress::Planned) + .then(|| recorder.observed().cloned()) + .flatten(), + }), + } + } + Reattached::Indeterminate(reason) => { + if locked.journal()?.is_none() { + return Ok(ReattachedClone::Indeterminate(reason)); + } + // The launch that a crash of its caller interrupted cannot be ended yet. + let fault = Fault::new(Evidence::LostStatus, interrupted(progress)); + failed(fault, progress.stage(), Teardown::Unresolved(reason)) + } + } +} + +/// Returns the fault of the clone whose failures `recorder` records, which got as far as +/// `progress`, and whose VM exited, as OpenVMM's outcome report, which +/// [`Recorder::observed`] reads, tells, with the guest's status, or `None` if the VM ended +/// without a failure: a status of the guest's that is evidence, as [`Recorder::reported`] gives +/// it, or, for a VM that exited `unheld`, whose runtime record the reattach found, as +/// [`Vmm::stop_unless_exited`] keeps that of a VMM that had exited before the stop began, the loss +/// of the clone's status, if no report of the guest's own end tells why. +fn exit_fault( + recorder: &Recorder, + progress: Progress, + unheld: bool, +) -> Option<(Fault, Option)> { + // A report is of the clone's VM only once its launch began, which removes the report of an + // earlier VM of the sandbox before it creates the clone's progress log, as + // `CloneRecords::read` takes it: a VMM that the launch spawned wrote it, though no record may + // keep the spawn, if the launch's caller crashed before the journal or the progress log + // recorded it, and a recovery of the sandbox then removed the journal. + if progress <= Progress::Planned { + return None; + } + let (evidence, status) = recorder.reported(recorder.observed(), progress); + let evidence = match evidence { + Some(evidence) => evidence, + // Only a VM that exited unheld, whose runtime record the reattach found, ends without + // telling why: its holder's stop ends one otherwise, and clears the record, but keeps that + // of one that had exited before the stop began, as `Vmm::stop_unless_exited` does. + None if unheld && status.is_none() => Evidence::LostStatus, + None => return None, + }; + Some((Fault::new(evidence, exited(progress, status)), status)) +} + +/// Reads the clone's activation record at `path`, which must be the canonical encoding of a +/// record of the restore of the generation that `lease` holds, as the clone's launch wrote it, +/// and returns its bytes and the record, or the fault, the host's, of one that is not. +fn read_activation( + path: &Path, + lease: &Lease, +) -> std::result::Result<(Vec, ActivationRecord), Fault> { + let bytes = open_regular(path) + .and_then(|file| read_at_most(file, MAX_RECORD_BYTES as u64, "an activation record")) + .map_err(|error| { + Fault::host( + Error::io(format!( + "cannot read the clone's activation record {}", + path.display() + )) + .with_source(error), + ) + })?; + let malformed = |problem: &str| { + Error::io(format!( + "the clone's activation record {} {problem}", + path.display() + )) + }; + let record = ActivationRecord::decode(&bytes) + .map_err(|error| malformed("is malformed").with_source(error)) + .attribute(Evidence::HostFailure)?; + if record.encode() != bytes { + return Err(Fault::new( + Evidence::HostFailure, + malformed("is not in its canonical encoding, in which the clone's launch wrote it"), + )); + } + if !record + .restore() + .is_some_and(|restore| restore.template == lease.template()) + { + return Err(Fault::new( + Evidence::HostFailure, + malformed(&format!( + "does not restore the generation {} that the clone's lease holds", + lease.generation() + )), + )); + } + Ok((bytes, record)) +} + +/// Checks the clone of `lease`, whose VM `vmm` runs in the sandbox of `locked`, in a new session +/// of `control`, with the sandbox's capability and its activation record, `activation`, by +/// `deadline`, as the reattach that `started` describes, and returns the session in which the +/// clone answered. +fn check_activation( + locked: &LockedStore, + lease: &Lease, + vmm: &Vmm, + control: &mut C, + (encoded, record): (Vec, ActivationRecord), + deadline: Instant, + started: Instant, +) -> std::result::Result { + let launch = vmm.record().launch(); + if launch != lease.launch() { + return Err(Fault::new( + Evidence::HostFailure, + Error::io(format!( + "the VM of the sandbox in {} is of the launch {launch}, not of the launch {} that \ + the clone's lease reserves", + locked.dir().display(), + lease.launch() + )), + )); + } + let capability = locked.read_capability().on_host()?; + let path = locked.clone_activation_path(); + let session = ActivationSession { + capability: &capability, + record: &path, + expected: ReadyActivation::of(&record), + encoded, + deadline, + timeout: deadline.saturating_duration_since(started), + since: "its reattach", + }; + session.activate(control, vmm.endpoint(), vmm.process().reference(), None) +} + +/// Returns the teardown of a termination of a clone's VMM within `bound` that ended as `outcome` +/// tells, as [`Vmm::terminate`] returns it. +fn stopped(outcome: Result, bound: Duration) -> Teardown { + match outcome { + Ok(StopOutcome::Forced) => Teardown::Terminated, + Ok(StopOutcome::Graceful) => Teardown::Exited, + Ok(StopOutcome::TimedOut) => Teardown::Unresolved(format!( + "the clone's VMM did not exit within {bound:?} of its termination" + )), + Err(error) => Teardown::Unresolved(describe(&error)), + } +} + +/// Returns `error`, which says so if `teardown` is unresolved. +fn unended(error: Error, teardown: &Teardown) -> Error { + match teardown { + Teardown::Unresolved(reason) => amended( + error, + &format!("; the clone's VMM could not be ended, and remains for a reattach: {reason}"), + ), + _ => error, + } +} + +/// Returns the error of a reattach that could not record the clone's failure, as `error` tells, +/// before it would end, or forget, anything of the clone's VM, which it leaves as it found it. +fn unrecorded(error: Error) -> Error { + amended( + error, + "; the reattach left the clone's VM and its records as it found them, for a later reattach", + ) +} + +/// Returns the error of the launch of a clone that got as far as `progress` before a crash of its +/// caller interrupted it. +fn interrupted(progress: Progress) -> Error { + Error::exited(format!( + "the launch of the clone was interrupted at its step {progress}, as a crash of its \ + caller interrupts it" + )) +} + +/// Returns the error of a clone's VM that exited once the clone got as far as `progress`, with +/// the guest's `status`, if OpenVMM's outcome report gives one. +fn exited(progress: Progress, status: Option) -> Error { + let ending = match status { + Some(status) => format!("with the guest's status {status}"), + None => "without an outcome report of the guest's own end".to_owned(), + }; + Error::exited(format!( + "the clone's VM exited at its step {progress}, {ending}" + )) +} + +/// Returns the error of a clone's VM that runs under another runtime identity, which differs in +/// `differences`. +fn mismatched(differences: &[IdentityPart]) -> Error { + let parts: Vec<&str> = differences.iter().map(|part| part.name()).collect(); + Error::new( + ErrorCode::RuntimeMismatch, + format!( + "the clone's VM runs under another runtime identity, which differs in {}, so it is \ + not driven, and is ended through its verified identity", + parts.join(", ") + ), + ) +} + +/// Returns the fault of a clone whose sandbox holds a record of a failure that cannot be read, or +/// that is another clone's, as `error` tells: the record still shows that the clone failed, so +/// the reattach keeps none of it, with the host's evidence. +fn unreadable_failure(error: Error) -> Fault { + Fault::host(amended( + error, + "; a clone whose sandbox records a failure failed for good, so the reattach keeps no VM \ + of it", + )) +} diff --git a/nvx-rs/src/host/template/clone/record.rs b/nvx-rs/src/host/template/clone/record.rs index f644a6fc2..9f04ab631 100644 --- a/nvx-rs/src/host/template/clone/record.rs +++ b/nvx-rs/src/host/template/clone/record.rs @@ -12,6 +12,7 @@ use serde::Serialize; use sha2::{Digest, Sha256}; use super::CloneRequest; +use super::failure::{Attribute, Evidence, Fault}; use crate::activation::{ ActivationRecord, Configuration, ExpectedResources, Identity, Image, Network, Publication, RestoreSection, TransactionId, Upper, @@ -53,33 +54,34 @@ const SCRATCH_DOMAIN: &[u8] = b"nvx scratch binding\0"; /// online, and the request's memory target. /// /// Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a request whose image -/// is not an EROFS layer, and for a configuration that no record holds, as -/// [`Configuration::check`] finds it, such as a hostname that is empty or longer than +/// is not an EROFS layer, [`Evidence::InvalidFormat`], and for a configuration that no record +/// holds, as [`Configuration::check`] finds it, such as a hostname that is empty or longer than /// [`MAX_HOSTNAME_BYTES`](crate::activation::MAX_HOSTNAME_BYTES), or a workload identity whose -/// UID or GID is 0; with [`ErrorKind::Io`](crate::ErrorKind::Io) for a key whose command line -/// names its DNS server in a form that OpenVMM does not write, as [`TemplateKey::dns_server`] -/// finds it; and with [`ErrorKind::Other`](crate::ErrorKind::Other) if the transaction ID cannot -/// be drawn. +/// UID or GID is 0, [`Evidence::InvalidActivate`]; with [`ErrorKind::Io`](crate::ErrorKind::Io) +/// for a key whose command line names its DNS server in a form that OpenVMM does not write, as +/// [`TemplateKey::dns_server`] finds it, [`Evidence::MalformedGeneration`]; and with +/// [`ErrorKind::Other`](crate::ErrorKind::Other) if the transaction ID cannot be drawn, or a +/// binding's identity computed, the host's evidence. pub(super) fn build( request: &CloneRequest, lease: &Lease, key: &TemplateKey, -) -> Result { +) -> std::result::Result { let image = match request.binding.format() { DistroFormat::Erofs => Image::Erofs { - binding: distro_identity(&request.binding)?, + binding: distro_identity(&request.binding).on_host()?, uuid: request.adapter.distro().image().identity, }, - DistroFormat::Gpt => return Err(gpt_refused()), + DistroFormat::Gpt => return Err(Fault::new(Evidence::InvalidFormat, gpt_refused())), }; let upper = match &request.scratch { None => Upper::Tmpfs, Some(binding) => Upper::Scratch { - binding: scratch_identity(binding)?, + binding: scratch_identity(binding).on_host()?, journal: request.journal, }, }; - let dns = key.dns_server()?; + let dns = key.dns_server().attribute(Evidence::MalformedGeneration)?; let network = key.network_policy_digest().map(|policy| Network { policy: Identity::from_bytes(policy), dns, @@ -103,13 +105,16 @@ pub(super) fn build( }, }; let restore = RestoreSection { - transaction: transaction()?, + transaction: transaction().on_host()?, template: lease.template(), }; ActivationRecord::new(Some(restore), configuration).map_err(|error| { - Error::unsupported(format!( - "no clone activates with the settings of the request: {error}" - )) + Fault::new( + Evidence::InvalidActivate, + Error::unsupported(format!( + "no clone activates with the settings of the request: {error}" + )), + ) }) } diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs new file mode 100644 index 000000000..8deacfbdd --- /dev/null +++ b/nvx-rs/src/host/template/clone/records.rs @@ -0,0 +1,1519 @@ +//! The records that a clone keeps in its sandbox about its launch, beside its activation record: +//! its progress log, [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), and the +//! record of its failure, [`CLONE_FAILURE_NAME`](crate::host::store::CLONE_FAILURE_NAME), which +//! [`CloneRecords::read`] reads with the clone's other records. +//! +//! # The progress log +//! +//! The launch of a clone creates its progress log, in place of one that an earlier launch left, +//! with a first line that names its format, the launch, and the boot of the host during which the +//! launch writes it: `nvx-clone-progress 1 `. The launch and its restore sequence +//! then append a line for each step of the clone's [`Progress`] as the clone reaches it, from the +//! spawn of its VMM to the commit of its launch, in order, and sync none of them: the log tells a +//! reattach how far the clone got once the caller of its launch crashed, which a crash of the +//! caller alone loses no write of. A crash of the host, which ends the clone's VM, may lose the +//! log's last lines, so how far the clone got is uncertain, once it launched, by a log that was +//! written during another boot, as by one that is damaged or missing: a workload may have run in +//! the clone. The launch appends the spawn before it returns, although the launch's journal +//! records it too, since a teardown of the launch removes the journal. A step that the next step +//! follows at once, the acknowledgment, which the Activate follows, is appended with that next +//! step, in its one write, so that the log takes a write for the spawn and for each step from the +//! restore-ready event on. A reader ignores a last line that a crash cut short. A sandbox +//! that boots cold writes no log. +//! +//! # The failure record +//! +//! The record of a clone's failure is JSON in format [`FAILURE_FORMAT`], which the failure writes +//! before its teardown, with the teardown [pending](Teardown::Pending), and again once its +//! teardown is known, replacing a record atomically, as the sandbox's store replaces its records, +//! and durably: each write syncs the sandbox's directory once the record is in place, so that the +//! record outlasts a crash of the host, as the clone's binding does. Like the store's records, the +//! record, and the directory that a write syncs, are named by their paths: the directories above +//! the sandbox's must be ones that only the current user, or an administrator, may change, as the +//! [store](crate::host::store) requires of them. The record keeps at most +//! 8192 characters of the failure's message, and 1024 of the reason of an unresolved teardown: of +//! a longer text, as the diagnosis of a caller that fails the clone may be, its first and its last +//! characters, with an ellipsis between them, so that a read, which takes at most 64 KiB of a +//! record, takes every record that a failure writes. A record of another format fails rather +//! than being read. The failure of a VMM that exited before its restore-ready event reads the end +//! of the launch's log for the line with which OpenVMM reported the error that ended it, which the +//! record's message keeps, and whose time ABI code may attribute the failure, as [the attribution +//! of evidence](Evidence#attribution) describes. +//! +//! A release of the clone's lease removes both, as it removes the clone's other files, so that +//! the records of a clone last as long as its lease, which a failure whose teardown is unresolved +//! keeps. + +use std::cell::OnceCell; +use std::fs::File; +use std::io::{self, Read, Seek, SeekFrom, Write}; +use std::num::NonZeroU64; +use std::path::{Path, PathBuf}; + +use serde::{Deserialize, Serialize}; + +use super::failure::{ + self, CloneFailure, Evidence, FailureRecord, Fault, Outcome, Progress, Stage, Teardown, + amended, describe, +}; +use crate::activation::{Identity, StoreGeneration, TemplateReference, TransactionId}; +use crate::error::{Error, ErrorCode, Result}; +use crate::host::platform::{self, parse_hex_id}; +use crate::host::process::BootId; +use crate::host::report::{self, Report}; +use crate::host::store::files::{create_new, sync_dir, write_private}; +use crate::host::store::regular::{open_regular, read_at_most, was_removed}; +use crate::host::store::{ + CLONE_PROGRESS_NAME, LaunchId, LaunchJournal, OUTCOME_NAME, Recovery, RuntimeRecord, + SandboxStore, +}; +use crate::host::supervisor::Starting; +use crate::host::template::{KeyIdentity, Lease}; + +/// The format of the record of a clone's failure. +pub const FAILURE_FORMAT: u32 = 1; + +/// The first word of the first line of a clone's progress log. +const PROGRESS_MAGIC: &str = "nvx-clone-progress"; + +/// The format of a clone's progress log, after [`PROGRESS_MAGIC`] in its first line. +const PROGRESS_FORMAT: u32 = 1; + +/// The most bytes of a clone's progress log that a read takes: its first line, and a line for each +/// of its steps, take a few hundred. +const MAX_PROGRESS_BYTES: u64 = 4096; + +/// The most bytes of the record of a clone's failure that a read takes: more than any record that +/// a failure writes takes, whose texts [`bounded`] bounds, as [`MAX_MESSAGE_CHARS`] and +/// [`MAX_REASON_CHARS`] tell. +const MAX_FAILURE_BYTES: u64 = 64 * 1024; + +/// The most characters of the message of a failure's error that the record of the failure keeps, +/// as [`bounded`] keeps them: the diagnosis of a caller that fails a clone, as +/// [`Activated::fail`](super::Activated::fail) and [`Started::fail`](super::Started::fail) take +/// it, may be longer. +const MAX_MESSAGE_CHARS: usize = 8 * 1024; + +/// The most characters of the reason of an unresolved teardown that the record of a failure keeps, +/// as [`bounded`] keeps them. +const MAX_REASON_CHARS: usize = 1024; + +// JSON takes at most six bytes for a character, as for `\u0001`, and the other fields of a record +// take less than a kilobyte, so that a read takes every record that a failure writes. +const _: () = + assert!(6 * (MAX_MESSAGE_CHARS + MAX_REASON_CHARS) + 1024 <= MAX_FAILURE_BYTES as usize); + +/// What messages call the progress log of a clone. +const PROGRESS_WHAT: &str = "clone progress log"; + +/// What messages call the record of a clone's failure. +const FAILURE_WHAT: &str = "clone failure record"; + +/// The most bytes at the end of a VMM's log in which a failure looks for the line of OpenVMM's +/// fatal error, which OpenVMM writes as it exits, followed by the error's chain alone. +const FATAL_TAIL_BYTES: u64 = 64 * 1024; + +/// The most characters of the line of OpenVMM's fatal error that the record of a failure keeps. +const MAX_FATAL_CHARS: usize = 1024; + +/// The progress log of a clone's launch, open for writing, with the latest step that the clone +/// reached, and the latest that the log records, as the [module](self#the-progress-log) +/// describes. +#[derive(Debug)] +pub(super) struct ProgressLog { + /// The log, open for writing at its end. + file: File, + /// The log's path. + path: Box, + /// The latest step that the clone reached. + progress: Progress, + /// The latest step that the log records. + logged: Progress, +} + +impl ProgressLog { + /// Creates the progress log of the launch `launch` in the sandbox's directory `dir`, in place + /// of one that an earlier launch left, with its first line, which names the current boot of + /// the host, and returns it, at [`Progress::Launching`]. It first removes the outcome report + /// that an earlier VM of the sandbox left, which no acquisition, clearing of a VM's runtime + /// record, or release removes, so that a report beside the clone's records is the clone's VM's, + /// as [`CloneRecords::read`] takes it. + /// + /// `directory` holds the sandbox's directory open, as the sandbox's lock opened and checked it + /// through `dir`. The earlier report and log go through it, as a release removes the clone's + /// files, so that the removals reach the directory that holds the lock, whatever `dir` names + /// meanwhile. The log is then created at its path, where no file may exist, so that a path that + /// names another sandbox's directory by then fails the creation if that sandbox holds a log, + /// rather than replace it. + pub(super) fn create( + directory: &platform::Directory, + dir: &Path, + launch: &LaunchId, + ) -> Result { + let path = dir.join(CLONE_PROGRESS_NAME); + let failed = |error: io::Error| { + Error::io(format!( + "cannot create the {PROGRESS_WHAT} {}", + path.display() + )) + .with_source(error) + }; + let boot = BootId::current().map_err(failed)?; + for name in [OUTCOME_NAME, CLONE_PROGRESS_NAME] { + directory.remove_file(name).map_err(|error| { + Error::io(format!("cannot remove {}", dir.join(name).display())).with_source(error) + })?; + } + let mut file = create_new(&path).map_err(failed)?; + file.write_all(format!("{PROGRESS_MAGIC} {PROGRESS_FORMAT} {launch} {boot}\n").as_bytes()) + .map_err(failed)?; + Ok(Self { + file, + path: path.into(), + progress: Progress::Launching, + logged: Progress::Launching, + }) + } + + /// Returns the latest step that the clone reached. + pub(super) fn progress(&self) -> Progress { + self.progress + } + + /// Notes that the clone reached `step`, unless it reached it, or a later step, already, + /// without writing it: the log records it with the next step that it writes. It is for a + /// step that another record holds, as the record of a launch's failure holds the spawn of a + /// VMM that the launch could not record, or that the next step follows at once. + pub(super) fn advance(&mut self, step: Progress) { + self.progress = self.progress.max(step); + } + + /// Records that the clone reached `step`, unless it reached it, or a later step, already: + /// appends the line of each step that the clone reached since the latest that the log + /// records, through `step`, in order, with one write, which nothing syncs. + pub(super) fn reach(&mut self, step: Progress) -> Result<()> { + self.advance(step); + if self.progress <= self.logged { + return Ok(()); + } + let lines: String = LOGGED + .iter() + .filter(|&&logged| self.logged < logged && logged <= self.progress) + .map(|logged| format!("{logged}\n")) + .collect(); + self.file.write_all(lines.as_bytes()).map_err(|error| { + Error::io(format!( + "cannot record that the clone reached its step {} in the {PROGRESS_WHAT} {}", + self.progress, + self.path.display() + )) + .with_source(error) + })?; + self.logged = self.progress; + Ok(()) + } +} + +/// The steps that a progress log records after its first line, in order. +const LOGGED: [Progress; 8] = [ + Progress::Spawned, + Progress::Event, + Progress::Acknowledged, + Progress::Activating, + Progress::Ready, + Progress::Starting, + Progress::Started, + Progress::Committed, +]; + +/// How the launch of a clone that failed is ended: [`Starting::abandon`], outside tests, which +/// replace it to keep a launch that cannot be ended. +pub(super) type End = for<'a> fn(Starting<'a>) -> Result; + +/// Ends `starting` as [`Starting::abandon`] does: how the launch of a clone that failed is ended +/// outside tests. +pub(super) fn abandon(starting: Starting<'_>) -> Result { + starting.abandon() +} + +/// What a clone's failures are recorded with: the clone's lease and the transaction of its +/// restore, the paths, in its sandbox, of the record of its failure, of OpenVMM's outcome report +/// of its VM, and of its VMM's log, whether a workload may have run in the clone whatever its +/// records say, as for a clone whose progress log was damaged, or written during another boot, +/// and the report, once it was read. +#[derive(Debug)] +pub(super) struct Recorder { + /// The clone's lease. + lease: Lease, + /// The transaction ID of the clone's restore. + transaction: Option, + /// The path of the record of the clone's failure. + failure: PathBuf, + /// The path of OpenVMM's outcome report of the clone's VM. + report: PathBuf, + /// The path of the log of the clone's VMM. + log: PathBuf, + /// Whether how far the clone got is uncertain beyond the progress that its records give, so + /// that a workload may have run in it. + uncertain: bool, + /// OpenVMM's outcome report of the clone's VM, as [`observed`](Self::observed) first read it. + observed: OnceCell>, +} + +impl Recorder { + /// Returns the recorder of the failures of the clone of `lease`, with the transaction of its + /// restore, `transaction`, in the sandbox of `sandbox`. + pub(super) fn new( + lease: Lease, + transaction: Option, + sandbox: &SandboxStore, + ) -> Self { + Self { + lease, + transaction, + failure: sandbox.clone_failure_path(), + report: sandbox.outcome_path(), + log: sandbox.log_path(), + uncertain: false, + observed: OnceCell::new(), + } + } + + /// Returns the recorder, for a clone whose progress is uncertain beyond what its records + /// give, if `uncertain`: a status of the guest's is then attributed as once a workload may + /// have run, as [`reported`](Self::reported) does. + pub(super) fn uncertain(self, uncertain: bool) -> Self { + Self { uncertain, ..self } + } + + /// Returns OpenVMM's outcome report of the clone's VM, or `None` if it is missing or + /// malformed, as the recorder first read it, the first time that it needed it: once the VM + /// has ended, whose report is final then, so that a read that fails later, as a read that + /// another process's open of the report refuses may, takes nothing back of what the first + /// one showed. A recorder serves one step, a launch that fails or a reattach, which reads the + /// report only once the clone's VM ended, or its teardown did. + pub(super) fn observed(&self) -> Option<&Report> { + self.observed + .get_or_init(|| report::read(&self.report)) + .as_ref() + } + + /// Returns the guest's status in OpenVMM's outcome report `report` of the clone's VM, and the + /// evidence that the status alone is once the clone got as far as `progress`, as + /// [`failure::reported`] gives them, or as once a workload may have run, if the clone's + /// progress is [uncertain](Self::uncertain). + pub(super) fn reported( + &self, + report: Option<&Report>, + progress: Progress, + ) -> (Option, Option) { + let progress = if self.uncertain { + progress.max(Progress::Starting) + } else { + progress + }; + failure::reported(report, progress) + } + + /// Records the failure of the clone at `stage`, once it got as far as `progress`, with + /// `fault`, whose teardown ended as `teardown` tells, as [the records of + /// failures](FailureRecord#records) describe, attributed as [`attribute`](Self::attribute) + /// does, and returns it. A failure that cannot be recorded says so. + pub(super) fn record( + &self, + stage: Stage, + progress: Progress, + fault: Fault, + teardown: Teardown, + ) -> CloneFailure { + let (evidence, status, error) = self.attribute(progress, fault, &teardown); + let record = FailureRecord::new( + &self.lease, + self.transaction, + (stage, progress), + evidence, + status, + teardown, + &error, + ); + self.write(error, record) + } + + /// Finishes `earlier`, the record of a failure whose teardown a crash of the caller left + /// [pending](Teardown::Pending), once its teardown ended as `teardown` tells, with the + /// failure's `error`, as the failure would have recorded it then: attributed from its + /// evidence, as [`attribute`](Self::attribute) does, keeping its lease, generation, launch, + /// transaction, stage, and progress, and the guest's status that it holds, if the outcome + /// report gives none now. Writes the record in place of the earlier one, and returns the + /// failure. + pub(super) fn finish( + &self, + earlier: FailureRecord, + error: Error, + teardown: Teardown, + ) -> CloneFailure { + let fault = Fault::new(earlier.evidence, error); + let (evidence, status, error) = self.attribute(earlier.progress, fault, &teardown); + let record = FailureRecord { + evidence, + status: status.or(earlier.status), + outcome: evidence.outcome(), + teardown, + code: error.code(), + message: describe(&error), + ..earlier + }; + self.write(error, record) + } + + /// Returns the evidence, the guest's status, and the error of the failure of the clone with + /// `fault`, once it got as far as `progress`, whose teardown ended as `teardown` tells: the + /// loss of a VMM that exited before its restore-ready event takes the line of OpenVMM's fatal + /// error that the VMM's log ends with, as [`diagnosed`](Self::diagnosed) does, and the + /// evidence of a status of the guest's, which OpenVMM's outcome report gives once the VMM has + /// exited, as [`observed`](Self::observed) reads it, and as [`reported`](Self::reported) + /// gives it, prevails over the fault's, unless the fault's is the template's, as + /// [`failure::prevailing`] decides. + fn attribute( + &self, + progress: Progress, + fault: Fault, + teardown: &Teardown, + ) -> (Evidence, Option, Error) { + let fault = self.diagnosed(progress, fault); + let report = match teardown { + Teardown::NoVm => None, + _ => self.observed(), + }; + let (reported, status) = self.reported(report, progress); + let evidence = failure::prevailing(fault.evidence, reported); + (evidence, status, fault.error) + } + + /// Returns `fault`, of the clone that got as far as `progress`, which is the loss of a VMM + /// that exited before its restore-ready event, with the line with which OpenVMM reported the + /// error that ended the VMM after its message, if the VMM's log ends with one, as + /// [`read_fatal`] reads it, and the message does not hold it yet, as the message of a failure + /// that a reattach resolves may, and with the line's evidence, if it is evidence, as [the + /// attribution of evidence](Evidence#attribution) describes. Returns any other fault as it is. + fn diagnosed(&self, progress: Progress, fault: Fault) -> Fault { + if fault.evidence != Evidence::LostStatus || progress >= Progress::Event { + return fault; + } + let Some(line) = read_fatal(&self.log) else { + return fault; + }; + let evidence = failure::fatal_evidence(&line).unwrap_or(fault.evidence); + let reported = format!("; OpenVMM reported: {line}"); + let error = if describe(&fault.error).contains(&reported) { + fault.error + } else { + amended(fault.error, &reported) + }; + Fault::new(evidence, error) + } + + /// Records the failure of the clone at `stage`, once it got as far as `progress`, with + /// `evidence`, the guest's `status`, if the failure knows it already, and `error`, before the + /// teardown that follows it, whose record is [`Teardown::Pending`], so that a crash of the + /// caller during the teardown leaves the record, which a reattach finishes, as + /// [`reattach_clone`](super::reattach_clone) describes. The record that the failure writes + /// once its teardown ended replaces it. + /// + /// Fails if the record cannot be written, durably, as [`write_failure`] writes it: the caller + /// then removes nothing that tells that the clone launched, the journal of its launch or the + /// runtime record of its VM, which keep its lease, so that a reattach still finds that the + /// clone failed, though not how, if a crash of the caller comes before the failure is + /// recorded. + pub(super) fn pending( + &self, + stage: Stage, + progress: Progress, + (evidence, status): (Evidence, Option), + error: &Error, + ) -> Result<()> { + let record = FailureRecord::new( + &self.lease, + self.transaction, + (stage, progress), + evidence, + status, + Teardown::Pending, + error, + ); + write_failure(&self.failure, &record).map_err(|failed| { + Error::io(format!( + "the failure could not be recorded before its teardown: {}", + describe(&failed) + )) + }) + } + + /// Resolves `earlier`, the record of a failure that its teardown left unresolved, with + /// `teardown`, which ended the clone's VMM, keeping its stage, evidence, outcome, code, and + /// message, unless what the VMM left as it ended tells more: the line of OpenVMM's fatal + /// error, for the loss of a VMM that exited before its restore-ready event, as + /// [`diagnosed`](Self::diagnosed) adds it, and the guest's status in OpenVMM's outcome + /// report, which the record does not hold, whose evidence, as [`reported`](Self::reported) + /// gives it, prevails as [`failure::prevailing`] decides, as in [`record`](Self::record). + /// Writes the record in place of the earlier one, if they differ, and returns the failure. + pub(super) fn resolve(&self, mut earlier: FailureRecord, teardown: Teardown) -> CloneFailure { + let error = earlier.error(); + if earlier.teardown == teardown { + return CloneFailure::new(error, earlier); + } + let fault = self.diagnosed(earlier.progress, Fault::new(earlier.evidence, error)); + let mut evidence = fault.evidence; + if earlier.status.is_none() && teardown != Teardown::NoVm { + let (reported, status) = self.reported(self.observed(), earlier.progress); + evidence = failure::prevailing(evidence, reported); + earlier.status = status; + } + if evidence != earlier.evidence { + earlier.evidence = evidence; + earlier.outcome = evidence.outcome(); + } + earlier.message = describe(&fault.error); + earlier.teardown = teardown; + self.write(fault.error, earlier) + } + + /// Writes `record`, the record of the failure of `error`, in place of the sandbox's record, and + /// returns the failure, with the record as the sandbox keeps it, as [`kept`] returns it, whose + /// error says so if the record could not be written. + pub(super) fn write(&self, error: Error, record: FailureRecord) -> CloneFailure { + let record = kept(record); + let error = match write_failure(&self.failure, &record) { + Ok(()) => error, + Err(failed) => amended( + error, + &format!("; the failure could not be recorded: {}", describe(&failed)), + ), + }; + CloneFailure::new(error, record) + } +} + +/// What the launch of a clone keeps to record how far the clone got and how it failed: the +/// [`Recorder`] of its failures, its progress log, and how a failure ends the launch. +#[derive(Debug)] +pub(super) struct Tracker { + /// The recorder of the clone's failures. + recorder: Recorder, + /// The clone's progress log. + log: ProgressLog, + /// How a failure ends the launch. + end: End, +} + +impl Tracker { + /// Returns the tracker of the launch of a clone whose failures `recorder` records, with its + /// progress `log`, and the end of a launch that fails, `end`. + pub(super) fn new(recorder: Recorder, log: ProgressLog, end: End) -> Self { + Self { recorder, log, end } + } + + /// Notes that the clone reached `step`, as [`ProgressLog::advance`] does, without writing it. + pub(super) fn advance(&mut self, step: Progress) { + self.log.advance(step); + } + + /// Records that the clone reached `step` in its progress log, as [`ProgressLog::reach`] does, + /// and returns the fault of a failure to record it, the host's. + pub(super) fn reach(&mut self, step: Progress) -> std::result::Result<(), Fault> { + self.log.reach(step).map_err(Fault::host) + } + + /// Records that the clone reached `step` in its progress log if it can: a step that another + /// record holds, such as the commit, which the runtime record holds while the VM runs. + pub(super) fn note(&mut self, step: Progress) { + let _ = self.log.reach(step); + } + + /// Records the failure of the clone at `stage` with `evidence` and `error` before the teardown + /// that follows it, once the clone got as far as its progress, as [`Recorder::pending`] does, + /// and fails as it fails. + pub(super) fn pending(&self, stage: Stage, evidence: Evidence, error: &Error) -> Result<()> { + self.recorder + .pending(stage, self.log.progress(), (evidence, None), error) + } + + /// Ends `starting`, the launch of the clone, which failed at `stage` with `fault`, as `end` + /// does, and records the failure before the teardown, as [`pending`](Self::pending) does, and + /// with the teardown's outcome, as [`record`](Self::record) does. The error of a launch that + /// could not be ended says so, and that its journal remains for recovery. + /// + /// A failure that cannot be recorded before the teardown ends the clone's VMM through its + /// handle, and keeps the launch's journal, as [`Starting::end_keeping_journal`] does, and with + /// it the lease, until a reattach ends the launch: the teardown is unresolved, so that a crash + /// of the caller before the failure is recorded leaves a launch that a reattach records as + /// failed, rather than a clone that ended. + pub(super) fn fail(&self, starting: Starting<'_>, stage: Stage, fault: Fault) -> CloneFailure { + let teardown = match self.pending(stage, fault.evidence, &fault.error) { + Ok(()) => Teardown::of((self.end)(starting)), + Err(unrecorded) => { + let ended = if starting.end_keeping_journal() { + "the clone's VMM ended through its handle" + } else { + "the clone's VMM did not exit once it was ended through its handle" + }; + Teardown::Unresolved(format!("{ended}, but {unrecorded}")) + } + }; + let fault = match &teardown { + Teardown::Unresolved(reason) => Fault::new( + fault.evidence, + amended( + fault.error, + &format!("; the launch could not be ended, and remains for recovery: {reason}"), + ), + ), + _ => fault, + }; + self.record(stage, fault, teardown) + } + + /// Records the failure of the clone at `stage` with `fault`, whose teardown ended as + /// `teardown` tells, once the clone got as far as its progress, as [`Recorder::record`] + /// does, and returns it. + pub(super) fn record(&self, stage: Stage, fault: Fault, teardown: Teardown) -> CloneFailure { + self.recorder + .record(stage, self.log.progress(), fault, teardown) + } +} + +/// What a clone's progress log records. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct Logged { + /// The latest step that the log records. + pub(super) progress: Progress, + /// The boot of the host during which the launch wrote the log, as its first line names it, + /// unless a crash cut that line short. + pub(super) boot: Option, +} + +impl Logged { + /// Returns whether the log tells for sure how far the clone got: whether the launch wrote it + /// during the boot `current`, the current boot of the host, if it is known, so that no crash + /// of the host lost a line of it since, as [the module](self#the-progress-log) describes. + pub(super) fn is_certain(&self, current: Option) -> bool { + self.boot.is_some() && self.boot == current + } +} + +/// Reads the progress log at `path` of the launch `launch`, and returns what it records, or +/// `None` if it does not exist. +/// +/// Fails with [`ErrorKind::Io`](crate::ErrorKind::Io) for a log that cannot be read, is not a +/// private regular file, names another launch or format, or no boot, or records other lines than +/// its steps, in order, besides a last line that a crash cut short. +pub(super) fn read_progress(path: &Path, launch: &LaunchId) -> Result> { + let Some(bytes) = read_private(path, PROGRESS_WHAT, MAX_PROGRESS_BYTES)? else { + return Ok(None); + }; + let malformed = + |problem: String| Error::io(format!("the {PROGRESS_WHAT} {} {problem}", path.display())); + let text = String::from_utf8(bytes).map_err(|_| malformed("is not UTF-8".to_owned()))?; + // A line without its newline is one that a crash cut short. + let mut lines = text + .split_inclusive('\n') + .filter_map(|line| line.strip_suffix('\n')); + let header = format!("{PROGRESS_MAGIC} {PROGRESS_FORMAT} {launch} "); + let boot = match lines.next() { + Some(line) => match line + .strip_prefix(&header) + .and_then(|boot| boot.parse::().ok()) + { + Some(boot) => boot, + None => { + return Err(malformed(format!( + "begins with {line:?}, not with the first line of the launch {launch}, \ + {header:?} followed by the identity of a boot" + ))); + } + }, + // A log that a crash cut short before its first line ended records the launch alone. + None => { + return Ok(Some(Logged { + progress: Progress::Launching, + boot: None, + })); + } + }; + let mut progress = Progress::Launching; + for line in lines { + match LOGGED.iter().find(|step| step.as_str() == line) { + Some(&step) if step > progress => progress = step, + _ => { + return Err(malformed(format!( + "records {line:?} after the step {progress}, which is not a later step" + ))); + } + } + } + Ok(Some(Logged { + progress, + boot: Some(boot), + })) +} + +/// Returns how far the clone of `lease` got, by the latest step that its progress log records, +/// `logged`, and by the store's records of its launch: its `journal`, which records the spawn of +/// its VMM, and its `runtime` record, which records the commit of its launch. +pub(super) fn progress_of( + lease: &Lease, + journal: Option<&LaunchJournal>, + runtime: Option<&RuntimeRecord>, + logged: Option, +) -> Progress { + let launch = lease.launch(); + let recorded = if runtime.is_some_and(|record| record.launch() == launch) { + Some(Progress::Committed) + } else { + journal + .filter(|journal| journal.launch() == launch) + .map(|journal| match journal.process() { + Some(_) => Progress::Spawned, + None => Progress::Launching, + }) + }; + [logged, recorded] + .into_iter() + .flatten() + .max() + .unwrap_or(Progress::Planned) +} + +/// The record of a clone's failure, as JSON. +#[derive(Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct Recorded { + /// [`FAILURE_FORMAT`]. + format: u32, + /// The identity of the template store that leased the clone's generation. + store: String, + /// The identity of the clone's lease. + lease: String, + /// The leased generation. + generation: u64, + /// The identity of the leased generation's key. + key: String, + /// The launch that the lease reserves. + launch: String, + /// The transaction ID of the clone's restore, in hexadecimal, once the plan drew it. + #[serde(default, skip_serializing_if = "Option::is_none")] + transaction: Option, + /// The stage at which the clone failed. + stage: Stage, + /// How far the clone got. + progress: Progress, + /// The evidence of the failure. + evidence: Evidence, + /// The guest's status, if OpenVMM's outcome report has one. + #[serde(default, skip_serializing_if = "Option::is_none")] + status: Option, + /// The original outcome of the failure. + outcome: Outcome, + /// How the teardown ended, as [`Teardown::as_str`] spells it. + teardown: String, + /// Why the teardown is unresolved, if it is. + #[serde(default, skip_serializing_if = "Option::is_none")] + unresolved: Option, + /// The code of the failure's error, as [`ErrorCode::as_str`] spells it. + code: String, + /// The message of the failure's error. + message: String, +} + +impl Recorded { + /// Returns the JSON form of `record`, as the sandbox keeps it, with texts that [`kept`] + /// bounds. + fn new(record: &FailureRecord) -> Self { + Self { + format: FAILURE_FORMAT, + store: record.store.to_string(), + lease: record.lease.to_string(), + generation: record.template.generation.get().get(), + key: KeyIdentity::from(record.template.key).to_string(), + launch: record.launch.to_string(), + transaction: record + .transaction + .map(|transaction| transaction.to_string()), + stage: record.stage, + progress: record.progress, + evidence: record.evidence, + status: record.status, + outcome: record.outcome, + teardown: record.teardown.as_str().to_owned(), + unresolved: match &record.teardown { + Teardown::Unresolved(reason) => Some(bounded(reason, MAX_REASON_CHARS)), + _ => None, + }, + code: record.code.as_str().to_owned(), + message: bounded(&record.message, MAX_MESSAGE_CHARS), + } + } + + /// Checks the record, and returns it, or says why it is malformed. + fn into_record(self) -> std::result::Result { + let field = + |what: &str, error: Error| format!("its {what} is invalid: {}", error.message()); + let teardown = match (self.teardown.as_str(), self.unresolved) { + ("no-vm", None) => Teardown::NoVm, + ("exited", None) => Teardown::Exited, + ("terminated", None) => Teardown::Terminated, + ("pending", None) => Teardown::Pending, + ("unresolved", Some(reason)) => Teardown::Unresolved(reason), + (teardown, _) => { + return Err(format!( + "its teardown, {teardown:?}, is not one that a failure records, or names a \ + reason only when it is unresolved" + )); + } + }; + let transaction = match self.transaction { + None => None, + Some(digits) => Some(TransactionId::from_bytes( + parse_hex_id(&digits).ok_or_else(|| { + format!("its transaction, {digits:?}, is not 32 hexadecimal digits") + })?, + )), + }; + let key: KeyIdentity = self.key.parse().map_err(|error| field("key", error))?; + // A later build may record a code that this one does not know, which is a failure all + // the same. + let code = ErrorCode::ALL + .iter() + .copied() + .find(|code| code.as_str() == self.code) + .unwrap_or(ErrorCode::Other); + Ok(FailureRecord { + store: self.store.parse().map_err(|error| field("store", error))?, + lease: self.lease.parse().map_err(|error| field("lease", error))?, + template: TemplateReference { + generation: NonZeroU64::new(self.generation) + .map(StoreGeneration::new) + .ok_or("it names store generation 0, which no generation has")?, + key: Identity::from(key), + }, + launch: LaunchId::try_from(self.launch) + .map_err(|problem| format!("its launch is invalid: {problem}"))?, + transaction, + stage: self.stage, + progress: self.progress, + evidence: self.evidence, + status: self.status, + outcome: self.outcome, + teardown, + code, + message: self.message, + }) + } +} + +/// Replaces the record of a clone's failure at `path` with `record`, as the sandbox keeps it, as +/// [`kept`] returns it, atomically, and durably: once the record is in place, it syncs the +/// record's directory, as [`sync_dir`] does, and fails if the sync does, though the record is in +/// place then. +pub(super) fn write_failure(path: &Path, record: &FailureRecord) -> Result<()> { + write_failure_with(path, record, &sync_dir) +} + +/// Replaces the record of a clone's failure as [`write_failure`] does, syncing its directory with +/// `sync`, which tests observe, and fail. +fn write_failure_with( + path: &Path, + record: &FailureRecord, + sync: &dyn Fn(&Path) -> Result<()>, +) -> Result<()> { + let mut text = serde_json::to_vec_pretty(&Recorded::new(record)).map_err(|error| { + Error::other(format!("cannot encode the {FAILURE_WHAT}")).with_source(error) + })?; + text.push(b'\n'); + write_private(path, &text)?; + match path.parent() { + Some(dir) if !dir.as_os_str().is_empty() => sync(dir), + _ => sync(Path::new(".")), + } +} + +/// Returns `record` as the sandbox keeps it: with at most [`MAX_MESSAGE_CHARS`] characters of its +/// message, and [`MAX_REASON_CHARS`] of the reason of its teardown, if it is unresolved, as +/// [`bounded`] keeps them. +fn kept(mut record: FailureRecord) -> FailureRecord { + record.message = bounded(&record.message, MAX_MESSAGE_CHARS); + if let Teardown::Unresolved(reason) = &mut record.teardown { + *reason = bounded(reason, MAX_REASON_CHARS); + } + record +} + +/// Returns `text`, or, if it has more than `max` characters, `max` of them: its first ones and its +/// last ones, about as many of each, with an ellipsis between them in place of those that it +/// leaves out, so that the message of an error keeps what it begins with, and the messages of its +/// sources, which end it. +fn bounded(text: &str, max: usize) -> String { + let count = text.chars().count(); + if count <= max { + return text.to_owned(); + } + let first = (max - 1) / 2; + let last = max - 1 - first; + text.chars() + .take(first) + .chain(std::iter::once('…')) + .chain(text.chars().skip(count - last)) + .collect() +} + +/// Reads the record of a clone's failure at `path`, or returns `None` if it does not exist. +/// +/// Fails with [`ErrorCode::FormatUnsupported`] for a record of another format than +/// [`FAILURE_FORMAT`], and with [`ErrorKind::Io`](crate::ErrorKind::Io) for one that cannot be +/// read, is not a private regular file, or is malformed. +pub(super) fn read_failure(path: &Path) -> Result> { + let Some(bytes) = read_private(path, FAILURE_WHAT, MAX_FAILURE_BYTES)? else { + return Ok(None); + }; + let malformed = || format!("the {FAILURE_WHAT} {} is malformed", path.display()); + let value: serde_json::Value = serde_json::from_slice(&bytes) + .map_err(|error| Error::io(malformed()).with_source(error))?; + match value.get("format") { + Some(format) if format.as_u64() == Some(FAILURE_FORMAT.into()) => {} + Some(format) => { + return Err(Error::new( + ErrorCode::FormatUnsupported, + format!( + "the {FAILURE_WHAT} {} has the unsupported format {format}; this build reads \ + format {FAILURE_FORMAT} only", + path.display() + ), + )); + } + None => return Err(Error::io(format!("{}: it has no format", malformed()))), + } + let recorded: Recorded = + serde_json::from_value(value).map_err(|error| Error::io(malformed()).with_source(error))?; + recorded + .into_record() + .map(Some) + .map_err(|problem| Error::io(format!("{}: {problem}", malformed()))) +} + +/// Reads the record of the failure of the clone of `lease` at `path`, as [`read_failure`] reads +/// it, or returns `None` if it does not exist. +/// +/// Fails as [`read_failure`] does, and, as for a record that is malformed, with +/// [`ErrorKind::Io`](crate::ErrorKind::Io) for a record of another clone's failure, whose template +/// store, lease, leased generation, its key, or launch is not the lease's: it tells nothing of +/// this clone, and a [quarantine](CloneFailure::quarantine) of its failure would tombstone the +/// record's generation, which may not be the one that this clone leases. +pub(super) fn read_failure_of(path: &Path, lease: &Lease) -> Result> { + let Some(record) = read_failure(path)? else { + return Ok(None); + }; + let leased = lease.template(); + let differences: Vec<&str> = [ + (record.store != lease.store(), "the template store"), + (record.lease != lease.id(), "the lease"), + ( + record.template.generation != leased.generation, + "the leased generation", + ), + ( + record.template.key != leased.key, + "the key of the leased generation", + ), + (record.launch != *lease.launch(), "the launch"), + ] + .into_iter() + .filter_map(|(differs, what)| differs.then_some(what)) + .collect(); + let Some((last, others)) = differences.split_last() else { + return Ok(Some(record)); + }; + let differences = if others.is_empty() { + (*last).to_owned() + } else { + format!("{} and {last}", others.join(", ")) + }; + Err(Error::io(format!( + "the {FAILURE_WHAT} {} belongs to another clone than the one of the lease {}, which the \ + sandbox's clone binding records: they differ in {differences}", + path.display(), + lease.id(), + ))) +} + +/// Reads the private regular file at `path`, the clone's `description`, of at most `limit` bytes, +/// or returns `None` if it does not exist, or was removed as this opened it, as [`was_removed`] +/// tells. +fn read_private(path: &Path, description: &str, limit: u64) -> Result>> { + let failed = |error: io::Error| { + Error::io(format!("cannot read the {description} {}", path.display())).with_source(error) + }; + let file = match open_regular(path) { + Ok(file) => file, + Err(error) if error.kind() == io::ErrorKind::NotFound || was_removed(path, &error) => { + return Ok(None); + } + Err(error) => return Err(failed(error)), + }; + if let Some(problem) = platform::private_file_problem(&file).map_err(failed)? { + return Err(Error::io(format!( + "the {description} {} {problem}", + path.display() + ))); + } + read_at_most(file, limit, &format!("the {description}")) + .map(Some) + .map_err(failed) +} + +/// Returns the last line with which OpenVMM reported the error that ended it, among the last +/// [`FATAL_TAIL_BYTES`] of the VMM's log at `path`, cut to [`MAX_FATAL_CHARS`], or none if they +/// hold none, or the log cannot be read: the line that leads OpenVMM's fatal error, then the +/// error's time ABI code, if it has one, as [the attribution of evidence](Evidence#attribution) +/// describes. +fn read_fatal(path: &Path) -> Option { + let mut log = open_regular(path).ok()?; + let length = log.metadata().ok()?.len(); + log.seek(SeekFrom::Start(length.saturating_sub(FATAL_TAIL_BYTES))) + .ok()?; + let mut tail = Vec::new(); + log.take(FATAL_TAIL_BYTES).read_to_end(&mut tail).ok()?; + let text = String::from_utf8_lossy(&tail); + let line = text + .lines() + .rev() + .find(|line| line.starts_with(failure::FATAL_PREFIX))?; + Some(line.chars().take(MAX_FATAL_CHARS).collect()) +} + +/// What the sandbox of a clone keeps of it, as [`CloneRecords::read`] reads it: the lease that its +/// clone binding records, the store's records of its launch and of its VM, its progress, the +/// record of its failure, and OpenVMM's outcome report of its VM. +/// +/// The records tell the state of a clone, as an operation that lists, inspects, or removes +/// clones reads it: +/// +/// - a [`journal`](Self::journal) without a [`runtime`](Self::runtime) record: a launch that runs, +/// under the lock of the sandbox, or one that was interrupted, which +/// [`reattach_clone`](super::reattach_clone) ends; +/// - a runtime record: a VM that may run, which a reattach holds again, or a stop ends; +/// - a [`failure`](Self::failure): a clone that failed, whose teardown is unresolved while the +/// journal or the runtime record remains, and [pending](Teardown::Pending) if a crash of the +/// caller interrupted its teardown, which a reattach finishes; +/// - neither a journal nor a runtime record: no VM of the clone runs, and a release ends the +/// lease, which removes the clone's records, before a removal of the sandbox. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct CloneRecords { + /// The lease that the sandbox's clone binding records. + pub lease: Lease, + /// The journal of the clone's launch, while it runs, or once it was interrupted. + pub journal: Option, + /// The runtime record of the clone's VM, from the commit of its launch until a stop, a + /// termination, or a reattach verifies that the VM exited. + pub runtime: Option, + /// How far the clone got, as its progress log and the store's records of its launch tell. + pub progress: Progress, + /// The record of the clone's failure, if it failed from its launch on. + pub failure: Option, + /// OpenVMM's outcome report of the clone's VM, once its VMM wrote one, and none before the + /// clone got past its plan, as [`read`](Self::read) tells. + pub report: Option, +} + +impl CloneRecords { + /// Reads the records of the clone of the sandbox of `sandbox`, or returns `None` if the + /// sandbox holds no clone binding. + /// + /// It takes no lock, as an inspection of the sandbox does not: each record is read whole, as + /// the latest transition that replaced it left it, and the records may come from transitions + /// that ran between the reads. A caller that holds the sandbox's lock reads them at one point + /// of its lifecycle. Fails as the reads of the clone binding, [`Lease::of`], of the store's + /// records, of the progress log, and of the failure record fail, a failure record of another + /// clone than the binding's included; an outcome report that is missing or malformed is none. + /// + /// It reads the outcome report only once the clone got past its plan, as the progress log of + /// its launch, the launch's journal, the runtime record of its VM, or the record of a failure + /// past the plan shows. Before the launch writes the first of those, its progress log, it + /// removes the report that an earlier VM of the sandbox left, which no acquisition, clearing + /// of a VM's runtime record, or release removes, and which is not the clone's. + pub fn read(sandbox: &SandboxStore) -> Result> { + let Some(lease) = Lease::of(sandbox)? else { + return Ok(None); + }; + let journal = sandbox.journal()?; + let runtime = sandbox.runtime()?; + let logged = read_progress(&sandbox.clone_progress_path(), lease.launch())? + .map(|logged| logged.progress); + let failure = read_failure_of(&sandbox.clone_failure_path(), &lease)?; + // A failure records how far the clone got, which the store's records of its launch no + // longer tell once the failure ended the launch. + let progress = progress_of(&lease, journal.as_ref(), runtime.as_ref(), logged).max( + failure + .as_ref() + .map_or(Progress::Planned, |failure| failure.progress), + ); + // The launch removed the report of an earlier VM of the sandbox before it wrote the first + // of the clone's records, which a progress past the plan shows. + let report = (progress > Progress::Planned) + .then(|| report::read(&sandbox.outcome_path())) + .flatten(); + Ok(Some(Self { + lease, + journal, + runtime, + progress, + failure, + report, + })) + } +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + use crate::error::ErrorKind; + use crate::host::store::CLONE_FAILURE_NAME; + use crate::host::template::{LeaseId, StoreId}; + + /// Returns a directory private to the current user, as a sandbox's is, whose files are + /// private too, in a temporary directory, which removes it as it drops. + fn private_dir() -> (tempfile::TempDir, PathBuf) { + let directory = tempfile::tempdir().unwrap(); + let private = directory.path().join("sandbox"); + assert_eq!(platform::create_owner_only_dir(&private).unwrap(), None); + (directory, private) + } + + /// Returns the record of a failure of a clone whose teardown ended as `teardown`. + fn failure(teardown: Teardown) -> FailureRecord { + FailureRecord { + store: StoreId::from_bytes([1; 16]), + lease: LeaseId::from_bytes([2; 16]), + template: TemplateReference { + generation: StoreGeneration::new(NonZeroU64::new(7).unwrap()), + key: Identity::from_bytes([3; 32]), + }, + launch: LaunchId::try_from("4".repeat(32)).unwrap(), + transaction: Some(TransactionId::from_bytes([5; 16])), + stage: Stage::Activation, + progress: Progress::Activating, + evidence: Evidence::UnusableRoot, + status: Some(54), + outcome: Outcome::Request, + teardown, + code: ErrorCode::Exited, + message: "the clone's VMM exited".to_owned(), + } + } + + #[test] + fn a_progress_log_records_each_step_once_with_the_steps_that_it_deferred() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_PROGRESS_NAME); + let launch = LaunchId::generate().unwrap(); + let boot = BootId::current().unwrap(); + let logged = |progress| { + Some(Logged { + progress, + boot: Some(boot), + }) + }; + // The log of an earlier launch goes, and so does the report of an earlier VM. + write_private(&path, b"left").unwrap(); + let report = private.join(OUTCOME_NAME); + write_private(&report, b"left").unwrap(); + let directory = platform::Directory::open(&private).unwrap(); + let mut log = ProgressLog::create(&directory, &private, &launch).unwrap(); + assert!(!report.exists()); + assert_eq!( + read_progress(&path, &launch).unwrap(), + logged(Progress::Launching) + ); + // A step that another record holds waits for the next step that the log writes. + log.advance(Progress::Spawned); + assert_eq!(log.progress(), Progress::Spawned); + assert_eq!( + read_progress(&path, &launch).unwrap(), + logged(Progress::Launching) + ); + log.reach(Progress::Event).unwrap(); + log.advance(Progress::Acknowledged); + log.reach(Progress::Activating).unwrap(); + // A step that the clone reached already is not recorded again. + log.reach(Progress::Event).unwrap(); + log.advance(Progress::Spawned); + assert_eq!(log.progress(), Progress::Activating); + assert_eq!( + fs::read_to_string(&path).unwrap(), + format!( + "{PROGRESS_MAGIC} 1 {launch} {boot}\nspawned\nevent\nacknowledged\nactivating\n" + ) + ); + assert_eq!( + read_progress(&path, &launch).unwrap(), + logged(Progress::Activating) + ); + } + + #[test] + fn a_progress_log_replaces_an_earlier_one_only_in_the_directory_that_it_holds_open() { + let (temporary, private) = private_dir(); + let launch = LaunchId::generate().unwrap(); + let directory = platform::Directory::open(&private).unwrap(); + write_private(&private.join(CLONE_PROGRESS_NAME), b"left").unwrap(); + write_private(&private.join(OUTCOME_NAME), b"left").unwrap(); + // The sandbox's path names another sandbox's directory by the time that the log is + // created: the earlier log and report of the directory that the handle holds go, and the + // other sandbox's stay, whose log fails the creation. + let moved = temporary.path().join("moved"); + fs::rename(&private, &moved).unwrap(); + let other = temporary.path().join("other"); + assert_eq!(platform::create_owner_only_dir(&other).unwrap(), None); + for name in [CLONE_PROGRESS_NAME, OUTCOME_NAME] { + write_private(&other.join(name), b"the other's").unwrap(); + } + fs::rename(&other, &private).unwrap(); + let error = ProgressLog::create(&directory, &private, &launch).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + for name in [CLONE_PROGRESS_NAME, OUTCOME_NAME] { + assert!(!moved.join(name).exists(), "{name}"); + assert_eq!( + fs::read(private.join(name)).unwrap(), + b"the other's", + "{name}" + ); + } + } + + #[test] + fn a_progress_log_that_a_crash_cut_short_is_read_and_one_of_another_launch_is_refused() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_PROGRESS_NAME); + let launch = LaunchId::generate().unwrap(); + assert_eq!(read_progress(&path, &launch).unwrap(), None); + let (boot, other) = (BootId::current().unwrap(), BootId::from_bytes([0xee; 16])); + let header = format!("{PROGRESS_MAGIC} 1 {launch} {boot}\n"); + // A last line that a crash cut short is not a step, nor is the first line, without which + // the log names no boot. A log of another boot is read, with its boot, which a crash of + // the host may have cut short. + for (text, progress, written) in [ + (format!("{header}event\nackno"), Progress::Event, Some(boot)), + (header[..10].to_owned(), Progress::Launching, None), + ( + header[..header.len() - 1].to_owned(), + Progress::Launching, + None, + ), + (String::new(), Progress::Launching, None), + ( + format!("{header}committed\n"), + Progress::Committed, + Some(boot), + ), + ( + format!("{PROGRESS_MAGIC} 1 {launch} {other}\nevent\n"), + Progress::Event, + Some(other), + ), + ] { + write_private(&path, text.as_bytes()).unwrap(); + let read = read_progress(&path, &launch).unwrap().unwrap(); + assert_eq!((read.progress, read.boot), (progress, written), "{text:?}"); + // Only a log of the current boot tells for sure how far the clone got. + assert_eq!( + read.is_certain(Some(boot)), + written == Some(boot), + "{text:?}" + ); + assert!(!read.is_certain(None), "{text:?}"); + } + // Another launch's log, another format, a first line without a boot, and steps out of + // their order, or that no log records, are refused. + let other = LaunchId::generate().unwrap(); + for text in [ + format!("{PROGRESS_MAGIC} 1 {other} {boot}\nevent\n"), + format!("{PROGRESS_MAGIC} 2 {launch} {boot}\n"), + format!("{PROGRESS_MAGIC} 1 {launch}\n"), + format!("{PROGRESS_MAGIC} 1 {launch} the-boot\n"), + format!("{header}ready\nevent\n"), + format!("{header}event\nevent\n"), + format!("{header}planned\n"), + format!("{header}launching\n"), + format!("{header}event\nbogus\n"), + ] { + write_private(&path, text.as_bytes()).unwrap(); + let error = read_progress(&path, &launch).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{text:?}: {error}"); + } + } + + #[test] + fn a_failure_record_is_read_as_it_was_written() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + assert_eq!(read_failure(&path).unwrap(), None); + let unresolved = Teardown::Unresolved("its VMM did not exit".to_owned()); + for teardown in [ + Teardown::NoVm, + Teardown::Exited, + Teardown::Terminated, + unresolved, + ] { + let written = failure(teardown); + write_failure(&path, &written).unwrap(); + assert_eq!(read_failure(&path).unwrap(), Some(written)); + } + // Its JSON spells each field as the record's types do. + let value: serde_json::Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + assert_eq!( + value, + serde_json::json!({ + "format": 1, + "store": "01".repeat(16), + "lease": "02".repeat(16), + "generation": 7, + "key": "03".repeat(32), + "launch": "4".repeat(32), + "transaction": "05".repeat(16), + "stage": "activation", + "progress": "activating", + "evidence": "unusable-root", + "status": 54, + "outcome": "request", + "teardown": "unresolved", + "unresolved": "its VMM did not exit", + "code": "exited", + "message": "the clone's VMM exited", + }) + ); + // A code that a later build records, which this one does not know, is still a failure. + let mut later = value.clone(); + later["code"] = "a-later-code".into(); + write_private(&path, &serde_json::to_vec(&later).unwrap()).unwrap(); + assert_eq!(read_failure(&path).unwrap().unwrap().code, ErrorCode::Other); + } + + #[test] + fn a_failure_record_is_in_place_when_its_directory_is_synced() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + // Each write syncs the record's directory once the record is in place, so that the sync + // makes the record's rename durable. + let synced = std::cell::RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced + .borrow_mut() + .push((dir.to_owned(), read_failure(&path).unwrap())); + sync_dir(dir) + }; + let (pending, exited) = (failure(Teardown::Pending), failure(Teardown::Exited)); + write_failure_with(&path, &pending, &sync).unwrap(); + write_failure_with(&path, &exited, &sync).unwrap(); + assert_eq!( + synced.into_inner(), + [ + (private.clone(), Some(pending)), + (private.clone(), Some(exited)) + ] + ); + // A sync that fails fails the write, whose record is in place, though maybe not durably. + let refused = |_: &Path| -> Result<()> { Err(Error::io("the test's sync failed")) }; + let terminated = failure(Teardown::Terminated); + let error = write_failure_with(&path, &terminated, &refused).unwrap_err(); + assert_eq!(error.message(), "the test's sync failed"); + assert_eq!(read_failure(&path).unwrap(), Some(terminated)); + } + + #[test] + fn a_failure_record_keeps_the_ends_of_texts_too_long_for_a_read_to_take() { + // A text keeps its first and its last characters, which it counts as characters. + assert_eq!(bounded("abcdef", 6), "abcdef"); + assert_eq!(bounded("abcdefg", 6), "ab…efg"); + assert_eq!(bounded("ééééééé", 4), "é…éé"); + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + // JSON takes six bytes for each of these control characters, the most that a character + // takes, so that each text, whole, would take more than a read takes. + let filler = "\u{1}".repeat(64 * 1024); + let mut written = failure(Teardown::Unresolved(format!("why{filler}so"))); + written.message = format!("start{filler}end"); + write_failure(&path, &written).unwrap(); + assert!(fs::metadata(&path).unwrap().len() <= MAX_FAILURE_BYTES); + let read = read_failure(&path).unwrap().unwrap(); + assert_eq!(read, kept(written)); + assert_eq!(read.message.chars().count(), MAX_MESSAGE_CHARS); + assert!(read.message.starts_with("start\u{1}") && read.message.ends_with("\u{1}end")); + let Teardown::Unresolved(reason) = &read.teardown else { + panic!("{:?}", read.teardown); + }; + assert_eq!(reason.chars().count(), MAX_REASON_CHARS); + assert!(reason.starts_with("why\u{1}") && reason.ends_with("\u{1}so")); + } + + #[test] + fn a_failure_record_of_another_format_or_a_malformed_one_is_refused() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + write_failure(&path, &failure(Teardown::Exited)).unwrap(); + let value: serde_json::Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + let changed = |change: &dyn Fn(&mut serde_json::Value)| { + let mut value = value.clone(); + change(&mut value); + write_private(&path, &serde_json::to_vec(&value).unwrap()).unwrap(); + read_failure(&path).unwrap_err() + }; + let error = changed(&|value| value["format"] = 2.into()); + assert_eq!(error.code(), ErrorCode::FormatUnsupported, "{error}"); + type Change = fn(&mut serde_json::Value); + let malformed: [Change; 6] = [ + |value| value["unresolved"] = "a reason of a resolved teardown".into(), + |value| value["teardown"] = "unresolved".into(), + |value| value["teardown"] = "vanished".into(), + |value| value["generation"] = 0.into(), + |value| value["transaction"] = "05".into(), + |value| value["extra"] = true.into(), + ]; + for change in malformed { + let error = changed(&change); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + } + write_private(&path, b"{").unwrap(); + assert_eq!(read_failure(&path).unwrap_err().kind(), ErrorKind::Io); + } + + #[test] + fn a_failure_record_of_another_clone_than_the_one_of_the_lease_is_refused() { + let (_directory, private) = private_dir(); + let path = private.join(CLONE_FAILURE_NAME); + let written = failure(Teardown::Exited); + let lease = Lease::new( + written.lease, + written.store, + written.template.generation, + written.template.key, + private.clone(), + written.launch.clone(), + ); + assert_eq!(read_failure_of(&path, &lease).unwrap(), None); + write_failure(&path, &written).unwrap(); + assert_eq!( + read_failure_of(&path, &lease).unwrap(), + Some(written.clone()) + ); + // A record of the failure of a clone of another store, of another lease, of another + // generation, or of another key, or of another launch, tells nothing of the lease's clone, + // though `read_failure` reads it. + let generation = StoreGeneration::new(NonZeroU64::new(8).unwrap()); + let launch = LaunchId::try_from("6".repeat(32)).unwrap(); + let others = [ + ( + FailureRecord { + store: StoreId::from_bytes([6; 16]), + ..written.clone() + }, + "the template store", + ), + ( + FailureRecord { + lease: LeaseId::from_bytes([6; 16]), + ..written.clone() + }, + "the lease", + ), + ( + FailureRecord { + template: TemplateReference { + generation, + ..written.template + }, + ..written.clone() + }, + "the leased generation", + ), + ( + FailureRecord { + template: TemplateReference { + key: Identity::from_bytes([6; 32]), + ..written.template + }, + ..written.clone() + }, + "the key of the leased generation", + ), + ( + FailureRecord { + launch: launch.clone(), + ..written.clone() + }, + "the launch", + ), + ( + FailureRecord { + lease: LeaseId::from_bytes([6; 16]), + template: TemplateReference { + generation, + ..written.template + }, + launch, + ..written.clone() + }, + "the lease, the leased generation and the launch", + ), + ]; + for (other, differences) in others { + write_failure(&path, &other).unwrap(); + assert_eq!(read_failure(&path).unwrap(), Some(other)); + let error = read_failure_of(&path, &lease).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + assert!( + error + .to_string() + .ends_with(&format!("they differ in {differences}")), + "{error}" + ); + } + // A record that cannot be read fails as it does for `read_failure`. + write_private(&path, b"{").unwrap(); + assert_eq!( + read_failure_of(&path, &lease).unwrap_err().kind(), + ErrorKind::Io + ); + } + + #[test] + fn the_line_of_openvmm_s_fatal_error_is_read_from_the_end_of_its_log() { + let (_directory, private) = private_dir(); + let path = private.join("openvmm.log"); + assert_eq!(read_fatal(&path), None); + // OpenVMM's warnings may name a time ABI code too; the line that leads its fatal error, + // which the error's chain follows, is the one that tells why it exited. + let fatal = "fatal error: [E_TSC_RATE_TOLERANCE] failed to launch vm worker"; + let log = format!( + "NVX-CPU-PROFILE-FALLBACK: [E_PROFILE_UNSUPPORTED] a warning\r\n{fatal}\r\n\r\n\ + Caused by:\r\n 0: [E_TSC_RATE_TOLERANCE] the destination's TSC rate\r\n" + ); + fs::write(&path, &log).unwrap(); + assert_eq!(read_fatal(&path).as_deref(), Some(fatal)); + // The line is looked for at the end of a long log, and kept to its first characters. + let long = format!("fatal error: {}", "x".repeat(2 * MAX_FATAL_CHARS)); + let filler = "a line of the VMM's log\n".repeat(2 * FATAL_TAIL_BYTES as usize / 24); + fs::write( + &path, + format!("fatal error: an earlier one\n{filler}{long}\n"), + ) + .unwrap(); + let read = read_fatal(&path).unwrap(); + assert_eq!(read.chars().count(), MAX_FATAL_CHARS); + assert!(long.starts_with(&read)); + // A log without a fatal error, or one that only names it, holds none. + fs::write(&path, "the VMM exited\nwarning: fatal error: quoted\n").unwrap(); + assert_eq!(read_fatal(&path), None); + } +} diff --git a/nvx-rs/src/host/template/clone/restore.rs b/nvx-rs/src/host/template/clone/restore.rs index f2ef00613..5387c403f 100644 --- a/nvx-rs/src/host/template/clone/restore.rs +++ b/nvx-rs/src/host/template/clone/restore.rs @@ -2,7 +2,8 @@ //! through the adapter's framing of the clone's control session, a [`CloneControl`]: //! [`Restoring::activate`] activates the clone, [`Activated::start`] starts its sandbox, and //! [`Started::commit`] commits its launch, as the documentation of [`Restoring::activate`] -//! describes. +//! describes, while [`Activated::fail`] and [`Started::fail`] fail the clone for a step of the +//! caller's own between them. use std::fmt; use std::io; @@ -10,10 +11,13 @@ use std::path::Path; use std::thread; use std::time::{Duration, Instant}; +use super::failure::{Attribute, CloneFailure, Evidence, Fault, Progress, Stage, host_evidence}; use super::launch::Restoring; +use super::records::Tracker; use crate::activation::{ActivationRecord, MAX_RECORD_BYTES, ReadyActivation}; use crate::error::{Error, Result}; use crate::host::platform::{ReadyListener, Transport}; +use crate::host::process::ProcessRef; use crate::host::sandbox::distro::CheckedDistro; use crate::host::session::{self, SessionError}; use crate::host::store::Recovery; @@ -48,6 +52,11 @@ pub const MAX_ACTIVATION_SESSIONS: usize = 3; /// How often the wait for the restore-ready event checks whether the VMM still runs. const EXIT_CHECK_INTERVAL: Duration = Duration::from_millis(50); +/// How long the wait for the restore-ready event waits for the VMM to exit once it closed the +/// event's endpoint without a byte of the event, as OpenVMM does as it fails to launch its VM +/// worker, which it does after it connected the endpoint, before it reports the error and exits. +const CLOSED_EXIT_WAIT: Duration = Duration::from_secs(10); + /// How long the sequence waits before a session that replaces a lost one, so that the VMM's /// control console notices the end of the lost session, which frees the console's one attachment. const SESSION_RETRY_INTERVAL: Duration = Duration::from_millis(50); @@ -149,6 +158,12 @@ impl<'s> Restoring<'s> { /// identities in it: the transaction ID of the record's restore section, as /// [`ReadyActivation::of`] returns it. /// + /// The sequence records each step that the clone reaches in its progress log, as + /// [`CloneRecords`](super::CloneRecords) describes: the event; its acknowledgment, which the + /// attachment of a session shows, since OpenVMM reads none of the host's input, its + /// authentication included, until the guest's repair acknowledged a restore that gates the + /// input; the Activate; and the Ready. + /// /// Returns the [`Activated`] clone, whose Ready-to-Start deadline, `deadlines.start`, starts /// at the check of its Ready, and whose sandbox [`Activated::start`] starts, after which /// [`Started::commit`] commits the launch, as the cold boot of a sandbox commits its launch @@ -172,30 +187,46 @@ impl<'s> Restoring<'s> { /// # Failures /// /// A step that fails ends the launch, as [`abandon`](Self::abandon) does, which terminates the - /// VMM, and ends the launch once nothing of it runs, and fails with the step's error: + /// VMM through its verified identity, and ends the launch once nothing of it runs, and fails + /// with a [`CloneFailure`], which the sandbox records, at [`Stage::Restore`] until the + /// restore-ready event and at [`Stage::Activation`] after it, with the step's error and the + /// evidence below, unless OpenVMM's outcome report of a VMM that exited gives a status of the + /// guest's that is evidence, as [`status_evidence`](super::status_evidence) tells, which + /// prevails: /// /// - [`ErrorKind::TimedOut`](crate::ErrorKind::TimedOut) once the restore-to-Ready deadline - /// passed; + /// passed, [`Evidence::Timeout`]; /// - [`ErrorKind::Exited`](crate::ErrorKind::Exited) once the VMM exited before its event, or - /// before a connection; + /// before a connection, and once it closed the event's endpoint without a byte of the event, + /// and then exited, as OpenVMM does as it fails to launch its VM worker, within ten seconds + /// of the close, [`Evidence::LostStatus`], and, for a VMM that exited before its + /// event, the error that OpenVMM reported, as its log ends with it, after the error's + /// message, and [`Evidence::IncompatibleHost`] for an error whose time ABI code says that + /// the host cannot present the template's CPU profile, TSC, or LAPIC timer, as [the + /// attribution of evidence](Evidence#attribution) describes; /// - [`ErrorKind::Protocol`](crate::ErrorKind::Protocol) for a connection to the restore-ready - /// endpoint of another process than the VMM, bytes there that differ from the event, a - /// control endpoint that another process serves, a failure of the adapter's framing, a - /// session that it lost in the last of the sessions, and a Ready that reports another - /// activation than the record's, or none; - /// - [`ErrorKind::Io`](crate::ErrorKind::Io) if the endpoints fail otherwise, and if the - /// sandbox's activation record cannot be read, or is no longer the launch's; + /// endpoint of another process than the VMM, bytes there that differ from the event, an + /// endpoint that the VMM closed after a part of the event, or without a byte of it and then + /// ran on for ten seconds, a control endpoint that another process serves, a failure of the + /// adapter's framing, and a Ready that reports another activation than the record's, or + /// none, [`Evidence::Protocol`], and for a session that the adapter lost in the last of the + /// sessions, [`Evidence::Reset`]; + /// - [`ErrorKind::Io`](crate::ErrorKind::Io) if the endpoints fail otherwise, if the sandbox's + /// activation record cannot be read, or is no longer the launch's, and if the progress log + /// cannot be appended to, the host's evidence, as [`ClonePlan::new`](super::ClonePlan::new) + /// describes it; /// - [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a deadline beyond the - /// host's clock. + /// host's clock, [`Evidence::HostFailure`]. /// - /// The error says so if the launch could not be ended, whose journal then remains for + /// The failure's teardown says whether the launch could be ended, and the error says so if it + /// could not: its journal then remains for /// [`LockedStore::recover`](crate::host::store::LockedStore::recover), and the lease with it, /// as [Crashes](crate::host::supervisor#crashes) describes. pub fn activate( self, control: &mut C, deadlines: RestoreDeadlines, - ) -> Result> { + ) -> Result, CloneFailure> { let Self { starting, launched, @@ -205,16 +236,20 @@ impl<'s> Restoring<'s> { activation, record, capability, + mut tracker, } = self; let Some(deadline) = launched.checked_add(deadlines.ready) else { - let error = beyond_clock("restore-to-Ready", deadlines.ready); - return Err(abandoned(starting, error)); + let fault = beyond_clock("restore-to-Ready", deadlines.ready); + return Err(tracker.fail(starting, Stage::Restore, fault)); }; - if let Err(error) = wait_event(&mut listener, &starting, deadline, deadlines.ready) { - return Err(abandoned(starting, error)); + if let Err(fault) = wait_event(&mut listener, &starting, deadline, deadlines.ready) { + return Err(tracker.fail(starting, Stage::Restore, fault)); } // OpenVMM sends one event, and the endpoint takes nothing more. drop(listener); + if let Err(fault) = tracker.reach(Progress::Event) { + return Err(tracker.fail(starting, Stage::Activation, fault)); + } let activation_session = ActivationSession { capability: &capability, record: &record, @@ -222,25 +257,20 @@ impl<'s> Restoring<'s> { expected: ReadyActivation::of(&activation), deadline, timeout: deadlines.ready, + since: "its launch", }; - let mut sessions = 0; - let session = loop { - sessions += 1; - match activation_session.run(control, &starting) { - Ok(session) => break session, - Err(Attempt::Lost(_)) if sessions < MAX_ACTIVATION_SESSIONS => { - let left = deadline.saturating_duration_since(Instant::now()); - thread::sleep(SESSION_RETRY_INTERVAL.min(left)); - } - Err(Attempt::Lost(error) | Attempt::Failed(error)) => { - return Err(abandoned(starting, error)); - } - } + let endpoint = starting.endpoint().to_owned(); + let reference = starting.process().reference().clone(); + let activated = + activation_session.activate(control, &endpoint, &reference, Some(&mut tracker)); + let session = match activated { + Ok(session) => session, + Err(fault) => return Err(tracker.fail(starting, Stage::Activation, fault)), }; let Some(start) = Instant::now().checked_add(deadlines.start) else { drop(session); - let error = beyond_clock("Ready-to-Start", deadlines.start); - return Err(abandoned(starting, error)); + let fault = beyond_clock("Ready-to-Start", deadlines.start); + return Err(tracker.fail(starting, Stage::Start, fault)); }; Ok(Activated { session, @@ -248,6 +278,7 @@ impl<'s> Restoring<'s> { deadline: start, image, activation, + tracker, }) } } @@ -273,6 +304,8 @@ pub struct Activated<'s, S> { image: CheckedDistro, /// The clone's activation record. activation: ActivationRecord, + /// How far the clone got, and how its failures are recorded. + tracker: Tracker, } impl<'s, S> Activated<'s, S> { @@ -303,41 +336,76 @@ impl<'s, S> Activated<'s, S> { /// Starts the clone's sandbox through the adapter's framing, `control`, in the session in /// which the clone answered, by the Ready-to-Start deadline, and returns the [`Started`] - /// clone, whose launch the caller commits. + /// clone, whose launch the caller commits. The progress log records that the start was sent, + /// then that it was confirmed. /// /// The start is not sent again: a guest that started its sandbox refuses another start, so a /// session that fails once the start was sent leaves the start unknown. A failure ends the - /// launch, as [`abandon`](Self::abandon) does, and fails with - /// [`ErrorKind::TimedOut`](crate::ErrorKind::TimedOut) once the deadline passed, and with - /// [`ErrorKind::Protocol`](crate::ErrorKind::Protocol) before, and says so if the launch - /// could not be ended, as [`Restoring::activate`] does. - pub fn start>(self, control: &mut C) -> Result> { + /// launch, as [`abandon`](Self::abandon) does, and fails with a [`CloneFailure`] at + /// [`Stage::Start`], as [`Restoring::activate`] does: with + /// [`ErrorKind::TimedOut`](crate::ErrorKind::TimedOut) once the deadline passed, + /// [`Evidence::Timeout`], and with [`ErrorKind::Protocol`](crate::ErrorKind::Protocol) + /// before, [`Evidence::Reset`] for a session that the adapter lost and + /// [`Evidence::Protocol`] otherwise, unless OpenVMM's outcome report of a VMM that exited + /// gives evidence that prevails; and with the host's evidence if the progress log cannot be + /// appended to. + pub fn start>( + self, + control: &mut C, + ) -> Result, CloneFailure> { let Self { mut session, starting, deadline, image, activation, + mut tracker, } = self; - match control.start(&mut session, deadline) { - Ok(()) => Ok(Started { - session, - starting, - image, - activation, - }), + if let Err(fault) = tracker.reach(Progress::Starting) { + drop(session); + return Err(tracker.fail(starting, Stage::Start, fault)); + } + let started = control.start(&mut session, deadline); + let fault = match started { + Ok(()) => match tracker.reach(Progress::Started) { + Ok(()) => { + return Ok(Started { + session, + starting, + image, + activation, + tracker, + }); + } + Err(fault) => fault, + }, Err(source) => { - drop(session); - let error = if Instant::now() >= deadline { - Error::timed_out( - "the clone's sandbox did not start by its Ready-to-Start deadline", + let (evidence, error) = if Instant::now() >= deadline { + ( + Evidence::Timeout, + Error::timed_out( + "the clone's sandbox did not start by its Ready-to-Start deadline", + ), + ) + } else if control.lost(&source) { + ( + Evidence::Reset, + Error::protocol( + "the clone's control session was lost: cannot start the clone's \ + sandbox", + ), ) } else { - Error::protocol("cannot start the clone's sandbox") + ( + Evidence::Protocol, + Error::protocol("cannot start the clone's sandbox"), + ) }; - Err(abandoned(starting, error.with_source(source))) + Fault::new(evidence, error.with_source(source)) } - } + }; + drop(session); + Err(tracker.fail(starting, Stage::Start, fault)) } /// Ends the launch, as [`Starting::abandon`] does, which terminates its VMM and ends the @@ -349,6 +417,27 @@ impl<'s, S> Activated<'s, S> { drop(session); starting.abandon() } + + /// Fails the clone, once it is ready, for a step of the caller's own before the start of its + /// sandbox, such as one that maps a directory of the host into the clone, with `evidence`, + /// which attributes the failure as [the attribution of evidence](Evidence#attribution) + /// describes, and `error`, and returns the failure: once the adapter's session has ended, it + /// ends the launch as [`abandon`](Self::abandon) does, and records the failure at + /// [`Stage::Start`], before its teardown and once its teardown ended, as the restore sequence + /// records its own failures, so that a status of the guest's that is evidence, which + /// OpenVMM's outcome report gives once the VMM has exited, prevails over `evidence`, unless + /// `evidence` is of the template, such as a defect of the generation that the caller + /// diagnosed, whose record then holds the status beside it. + pub fn fail(self, evidence: Evidence, error: Error) -> CloneFailure { + let Self { + session, + starting, + tracker, + .. + } = self; + drop(session); + tracker.fail(starting, Stage::Start, Fault::new(evidence, error)) + } } impl fmt::Debug for Activated<'_, S> { @@ -365,7 +454,7 @@ impl fmt::Debug for Activated<'_, S> { /// A clone whose sandbox started, as [`Activated::start`] returns it, whose launch the caller /// [commits](Self::commit), once it checked anything else that it requires of a launch, or -/// [abandons](Self::abandon). +/// [abandons](Self::abandon), or [fails](Self::fail), if a check of its own failed. /// /// It holds the adapter's session, and the launch, as [`Activated`] does. Dropping it abandons the /// launch, as [`abandon`](Self::abandon) does, ignoring failures; a crash of the caller leaves the @@ -380,6 +469,8 @@ pub struct Started<'s, S> { image: CheckedDistro, /// The clone's activation record. activation: ActivationRecord, + /// How far the clone got, and how its failures are recorded. + tracker: Tracker, } impl<'s, S> Started<'s, S> { @@ -404,15 +495,39 @@ impl<'s, S> Started<'s, S> { } /// Commits the launch, as [`Starting::commit`] does, which writes its runtime record, and - /// returns the VM, with the adapter's session. The clone's image closes. + /// returns the VM, with the adapter's session. The clone's image closes, and the progress log + /// records the commit, if it can: the runtime record holds it while the VM runs. /// - /// Fails as [`Starting::commit`] fails, which ends the VMM. - pub fn commit(self) -> Result<(Vmm, S)> { + /// Fails as [`Starting::commit`] fails, which ends the VMM through its handle and keeps the + /// launch's journal, with a [`CloneFailure`] at [`Stage::Commit`], whose evidence is the + /// host's, and whose teardown is [`Teardown::Unresolved`](super::Teardown::Unresolved) until + /// a reattach ends the launch that the journal names, and releases the lease's hold. + pub fn commit(self) -> Result<(Vmm, S), CloneFailure> { let Self { - session, starting, .. + session, + starting, + mut tracker, + .. } = self; - let vmm = starting.commit()?; - Ok((vmm, session)) + let committed = starting.commit_reporting(|error| { + // The launch's journal remains, whether or not the failure can be recorded. + let _ = tracker.pending(Stage::Commit, host_evidence(error), error); + }); + match committed { + Ok(vmm) => { + tracker.note(Progress::Committed); + Ok((vmm, session)) + } + Err(error) => { + drop(session); + let teardown = super::Teardown::Unresolved( + "the clone's VMM ended through its handle, and the launch's journal remains \ + until a reattach ends the launch" + .to_owned(), + ); + Err(tracker.record(Stage::Commit, Fault::host(error), teardown)) + } + } } /// Ends the launch, as [`Activated::abandon`] does. @@ -423,6 +538,20 @@ impl<'s, S> Started<'s, S> { drop(session); starting.abandon() } + + /// Fails the clone, once its sandbox started, for a check of the caller's own before the + /// commit of its launch, with `evidence` and `error`, as [`Activated::fail`] does, at + /// [`Stage::Commit`]. + pub fn fail(self, evidence: Evidence, error: Error) -> CloneFailure { + let Self { + session, + starting, + tracker, + .. + } = self; + drop(session); + tracker.fail(starting, Stage::Commit, Fault::new(evidence, error)) + } } impl fmt::Debug for Started<'_, S> { @@ -438,68 +567,120 @@ impl fmt::Debug for Started<'_, S> { /// What every session of a clone's activation takes: the launch's capability, the sandbox's /// activation record, and the activation that its Ready must report, by the restore-to-Ready -/// deadline. -struct ActivationSession<'a> { +/// deadline, or by the deadline of a reattach. +pub(super) struct ActivationSession<'a> { /// The capability of the launch. - capability: &'a [u8; CAPABILITY_LEN], + pub(super) capability: &'a [u8; CAPABILITY_LEN], /// The path of the activation record in the sandbox. - record: &'a Path, + pub(super) record: &'a Path, /// The canonical encoding of the launch's activation record. - encoded: Vec, + pub(super) encoded: Vec, /// The activation that the clone's Ready must report. - expected: ReadyActivation, - /// The restore-to-Ready deadline. - deadline: Instant, - /// The restore-to-Ready deadline's duration from the launch. - timeout: Duration, + pub(super) expected: ReadyActivation, + /// The deadline by which the clone must be ready. + pub(super) deadline: Instant, + /// The deadline's duration from its start. + pub(super) timeout: Duration, + /// What starts the deadline, as messages say: `its launch`, or `its reattach`. + pub(super) since: &'static str, } /// How a session of a clone's activation failed. enum Attempt { /// The adapter lost the session, after which another session may find the clone ready. - Lost(Error), + Lost(Fault), /// Any other failure, which is final. - Failed(Error), + Failed(Fault), } impl ActivationSession<'_> { - /// Activates the clone of `starting` in a new session of `control`, as - /// [`Restoring::activate`] describes, and returns the session once the clone is ready. + /// Activates the clone whose VMM, `reference`, serves its control endpoint at `endpoint`, in + /// sessions of `control`, as [`Restoring::activate`] describes: in a new session after a + /// session that the adapter lost, in [`MAX_ACTIVATION_SESSIONS`] at most, and returns the + /// session in which the clone answered with its Ready. A launch records the clone's progress + /// in its `tracker`. + pub(super) fn activate( + &self, + control: &mut C, + endpoint: &str, + reference: &ProcessRef, + mut tracker: Option<&mut Tracker>, + ) -> std::result::Result { + let mut sessions = 0; + loop { + sessions += 1; + match self.run(control, endpoint, reference, tracker.as_deref_mut()) { + Ok(session) => return Ok(session), + Err(Attempt::Lost(_)) if sessions < MAX_ACTIVATION_SESSIONS => { + let left = self.deadline.saturating_duration_since(Instant::now()); + thread::sleep(SESSION_RETRY_INTERVAL.min(left)); + } + Err(Attempt::Lost(fault) | Attempt::Failed(fault)) => return Err(fault), + } + } + } + + /// Activates the clone in a new session of `control`, as [`activate`](Self::activate) + /// describes, records its progress in `tracker`, if it has one, and returns the session once + /// the clone is ready. fn run( &self, control: &mut C, - starting: &Starting<'_>, + endpoint: &str, + reference: &ProcessRef, + mut tracker: Option<&mut Tracker>, ) -> std::result::Result { let record = self.read_record().map_err(Attempt::Failed)?; - let transport = self.connect(starting).map_err(Attempt::Failed)?; + let transport = self.connect(endpoint, reference).map_err(Attempt::Failed)?; let attached = control.attach(transport, self.capability, self.deadline); let mut session = attached.map_err(|error| self.failed(control, error, "attach its control session"))?; + if let Some(tracker) = &mut tracker { + // OpenVMM authenticated the session, which it does only once the guest acknowledged + // a restore that gates the host's input. The Activate follows at once, with which + // the progress log records the acknowledgment. + tracker.advance(Progress::Acknowledged); + tracker + .reach(Progress::Activating) + .map_err(Attempt::Failed)?; + } let reported = control .activate(&mut session, &record, self.deadline) .map_err(|error| self.failed(control, error, "activate it"))?; self.check(reported).map_err(Attempt::Failed)?; + if let Some(tracker) = &mut tracker { + tracker.reach(Progress::Ready).map_err(Attempt::Failed)?; + } Ok(session) } - /// Connects to the control endpoint of the VMM of `starting`, as [`session::connect`] does, - /// by the deadline. - fn connect(&self, starting: &Starting<'_>) -> Result> { - let reference = starting.process().reference(); - let endpoint = starting.endpoint(); + /// Connects to the control endpoint `endpoint`, which the VMM `reference` serves, as + /// [`session::connect`] does, by the deadline. + fn connect( + &self, + endpoint: &str, + reference: &ProcessRef, + ) -> std::result::Result, Fault> { session::connect(endpoint, reference.pid, reference.start_time, self.deadline).map_err( |error| { let message = format!("cannot connect to the clone's control endpoint {endpoint}"); match error { - SessionError::TimedOut => Error::timed_out(format!( - "{message} within {:?} of its launch", - self.timeout - )), - SessionError::ProcessExited => { - Error::exited(format!("{message}: its VMM exited")) - } - SessionError::Io(source) => Error::io(message).with_source(source), - error => Error::protocol(message).with_source(error), + SessionError::TimedOut => Fault::new( + Evidence::Timeout, + Error::timed_out(format!( + "{message} within {:?} of {}", + self.timeout, self.since + )), + ), + SessionError::ProcessExited => Fault::new( + Evidence::LostStatus, + Error::exited(format!("{message}: its VMM exited")), + ), + SessionError::Io(source) => Fault::host(Error::io(message).with_source(source)), + error => Fault::new( + Evidence::Protocol, + Error::protocol(message).with_source(error), + ), } }, ) @@ -507,76 +688,85 @@ impl ActivationSession<'_> { /// Reads the activation record in the sandbox, which must be, byte for byte, the encoding of /// the launch's record, and returns it. - fn read_record(&self) -> Result> { + fn read_record(&self) -> std::result::Result, Fault> { let path = self.record; let bytes = open_regular(path) .and_then(|file| read_at_most(file, MAX_RECORD_BYTES as u64, "an activation record")) .map_err(|error| { - Error::io(format!( - "cannot read the clone's activation record {}", - path.display() - )) - .with_source(error) + Fault::host( + Error::io(format!( + "cannot read the clone's activation record {}", + path.display() + )) + .with_source(error), + ) })?; if bytes != self.encoded { - return Err(Error::io(format!( - "{} no longer holds the activation record that the clone's launch wrote, the only \ - record that its restore sends", - path.display() - ))); + return Err(Fault::new( + Evidence::HostFailure, + Error::io(format!( + "{} no longer holds the activation record that the clone's launch wrote, the \ + only record that its restore sends", + path.display() + )), + )); } Ok(bytes) } /// Checks `reported`, the activation that the clone's Ready reports, against the record's. - fn check(&self, reported: Option) -> Result<()> { - let expected = &self.expected; - let Some(reported) = reported else { - return Err(Error::protocol( - "the clone's Ready reports no activation, as a VM that boots cold reports none", - )); - }; - if reported == *expected { - return Ok(()); - } - let transaction = |activation: &ReadyActivation| { - activation - .transaction - .map_or_else(|| "none".to_owned(), |transaction| transaction.to_string()) - }; - Err(Error::protocol(format!( - "the clone's Ready reports the transaction {}, not its record's transaction {}", - transaction(&reported), - transaction(expected) - ))) + fn check(&self, reported: Option) -> std::result::Result<(), Fault> { + check_ready(&self.expected, reported).attribute(Evidence::Protocol) } /// Returns how a session failed with `error`, which `control` returned as it tried to `what`. fn failed(&self, control: &C, error: C::Error, what: &str) -> Attempt { if Instant::now() >= self.deadline { - return Attempt::Failed( + return Attempt::Failed(Fault::new( + Evidence::Timeout, Error::timed_out(format!( - "the clone was not ready within {:?} of its launch: cannot {what}", - self.timeout + "the clone was not ready within {:?} of {}: cannot {what}", + self.timeout, self.since )) .with_source(error), - ); + )); } - let lost = control.lost(&error); - let message = if lost { - format!("the clone's control session was lost: cannot {what}") + if control.lost(&error) { + let error = Error::protocol(format!( + "the clone's control session was lost: cannot {what}" + )) + .with_source(error); + Attempt::Lost(Fault::new(Evidence::Reset, error)) } else { - format!("cannot {what}") - }; - let error = Error::protocol(message).with_source(error); - if lost { - Attempt::Lost(error) - } else { - Attempt::Failed(error) + let error = Error::protocol(format!("cannot {what}")).with_source(error); + Attempt::Failed(Fault::new(Evidence::Protocol, error)) } } } +/// Checks `reported`, the activation that a clone's Ready reports, against `expected`, its +/// record's: the transaction ID of the record's restore section. +fn check_ready(expected: &ReadyActivation, reported: Option) -> Result<()> { + let Some(reported) = reported else { + return Err(Error::protocol( + "the clone's Ready reports no activation, as a VM that boots cold reports none", + )); + }; + if reported == *expected { + return Ok(()); + } + let transaction = |activation: &ReadyActivation| { + activation + .transaction + .map_or_else(|| "none".to_owned(), |transaction| transaction.to_string()) + }; + Err(Error::protocol(format!( + "the clone's Ready reports the transaction {}, not its record's transaction {}", + transaction(&reported), + transaction(expected) + ))) +} + /// Waits until `deadline`, `timeout` after the launch, for the restore-ready event of the VMM of /// `starting` at `listener`, as [`Restoring::activate`] describes. fn wait_event( @@ -584,7 +774,7 @@ fn wait_event( starting: &Starting<'_>, deadline: Instant, timeout: Duration, -) -> Result<()> { +) -> std::result::Result<(), Fault> { let process = starting.process(); let pid = process.reference().pid; // OpenVMM writes the event once, and then closes its end, so the wait reads the event's bytes @@ -594,9 +784,12 @@ fn wait_event( while received.len() < length { let left = deadline.saturating_duration_since(Instant::now()); if left.is_zero() { - return Err(Error::timed_out(format!( - "the clone's VMM sent no restore-ready event within {timeout:?} of its launch" - ))); + return Err(Fault::new( + Evidence::Timeout, + Error::timed_out(format!( + "the clone's VMM sent no restore-ready event within {timeout:?} of its launch" + )), + )); } let before = received.len(); let open = listener @@ -604,35 +797,67 @@ fn wait_event( .map_err(|error| { let message = "cannot receive the restore-ready event of the clone's VMM"; if error.kind() == io::ErrorKind::PermissionDenied { - Error::protocol(message).with_source(error) + Fault::new( + Evidence::Protocol, + Error::protocol(message).with_source(error), + ) } else { - Error::io(message).with_source(error) + Fault::host(Error::io(message).with_source(error)) } })?; if !RESTORE_READY_EVENT.starts_with(&received) { - return Err(Error::protocol(format!( - "the clone's VMM sent {:?} at its restore-ready endpoint, which is not OpenVMM's \ - restore-ready event", - String::from_utf8_lossy(&received) - ))); + return Err(Fault::new( + Evidence::Protocol, + Error::protocol(format!( + "the clone's VMM sent {:?} at its restore-ready endpoint, which is not \ + OpenVMM's restore-ready event", + String::from_utf8_lossy(&received) + )), + )); } if received.len() == length { break; } if !open { - return Err(Error::protocol(format!( - "the clone's VMM closed its restore-ready endpoint after {} bytes of the event", - received.len() - ))); + if received.is_empty() { + let wait = deadline.min(Instant::now() + CLOSED_EXIT_WAIT); + let exited = process.wait(wait).map_err(|error| { + Fault::host( + Error::io("cannot tell whether the clone's VMM still runs") + .with_source(error), + ) + })?; + if exited { + return Err(Fault::new( + Evidence::LostStatus, + Error::exited( + "the clone's VMM closed its restore-ready endpoint without the event, \ + and exited, as OpenVMM does once it cannot restore the snapshot", + ), + )); + } + } + return Err(Fault::new( + Evidence::Protocol, + Error::protocol(format!( + "the clone's VMM closed its restore-ready endpoint after {} bytes of the event", + received.len() + )), + )); } if received.len() == before { let exited = process.has_exited().map_err(|error| { - Error::io("cannot tell whether the clone's VMM still runs").with_source(error) + Fault::host( + Error::io("cannot tell whether the clone's VMM still runs").with_source(error), + ) })?; if exited { - return Err(Error::exited( - "the clone's VMM exited before its restore-ready event, as OpenVMM does once \ - it cannot restore the snapshot", + return Err(Fault::new( + Evidence::LostStatus, + Error::exited( + "the clone's VMM exited before its restore-ready event, as OpenVMM does \ + once it cannot restore the snapshot", + ), )); } } @@ -640,32 +865,14 @@ fn wait_event( Ok(()) } -/// Returns the error of the deadline `name`, `timeout` after its start, which this host's clock +/// Returns the fault of the deadline `name`, `timeout` after its start, which this host's clock /// cannot represent. -fn beyond_clock(name: &str, timeout: Duration) -> Error { - Error::unsupported(format!( - "the {name} deadline of a clone's restore, {timeout:?} after its start, is beyond this \ - host's clock" - )) -} - -/// Ends `starting`, a launch whose restore sequence failed with `error`, as -/// [`Starting::abandon`] does, and returns `error`, which says why the launch could not be ended -/// if it could not: its journal then remains for recovery. -fn abandoned(starting: Starting<'_>, error: Error) -> Error { - let reason = match starting.abandon() { - Ok(Recovery::Indeterminate(reason)) => reason, - Ok(_) => return error, - Err(failure) => failure.to_string(), - }; - let code = error.code(); - let (_, message, source) = error.into_parts(); - let error = Error::new( - code, - format!("{message}; the launch could not be ended, and remains for recovery: {reason}"), - ); - match source { - Some(source) => error.with_source(source), - None => error, - } +fn beyond_clock(name: &str, timeout: Duration) -> Fault { + Fault::new( + Evidence::HostFailure, + Error::unsupported(format!( + "the {name} deadline of a clone's restore, {timeout:?} after its start, is beyond \ + this host's clock" + )), + ) } diff --git a/nvx-rs/src/host/template/clone/tests.rs b/nvx-rs/src/host/template/clone/tests.rs index 0aeb338ad..fc9bb98a4 100644 --- a/nvx-rs/src/host/template/clone/tests.rs +++ b/nvx-rs/src/host/template/clone/tests.rs @@ -247,9 +247,42 @@ fn network(access: NetworkAccess) -> SandboxNetwork { SandboxNetwork::new("10.0.0.2/24", NetworkPolicy::new(access)).unwrap() } -/// Returns the kind and the message of the error of `result`. -fn failure(result: Result) -> (ErrorKind, String) { - let error = result.unwrap_err(); +/// Returns the kind and the message of the error of `result`, a clone's plan that failed with +/// `evidence`, as [`refused_at`] checks it. +fn refused( + result: std::result::Result, + evidence: Evidence, +) -> (ErrorKind, String) { + refused_at(result, Stage::Plan, evidence) +} + +/// Returns the kind and the message of the error of `result`, which must have failed at `stage`, +/// the plan or the launch, with `evidence`, before anything of the launch existed: with the +/// transaction of the clone's restore once the plan drew it, and no VM to tear down. +fn refused_at( + result: std::result::Result, + stage: Stage, + evidence: Evidence, +) -> (ErrorKind, String) { + let failure = result.unwrap_err(); + let record = failure.record(); + assert_eq!( + (record.stage, record.evidence, failure.outcome()), + (stage, evidence, evidence.outcome()), + "{failure}" + ); + assert_eq!( + (record.progress, failure.teardown(), record.status), + (Progress::Planned, &Teardown::NoVm, None), + "{failure}" + ); + assert_eq!( + record.transaction.is_some(), + stage != Stage::Plan, + "{failure}" + ); + assert_eq!(record.code, failure.code()); + let error = failure.into_error(); (error.kind(), error.to_string()) } @@ -378,7 +411,7 @@ impl Fixture { acquired: &mut Acquired, request: &CloneRequest, access: Access<'_>, - ) -> Result { + ) -> Result { ClonePlan::new_with( &self.store, acquired, @@ -395,10 +428,18 @@ fn sparse_path(acquired: &Acquired) -> PathBuf { } /// Asserts that the sandbox of `acquired` holds no scratch device's file of a clone in RAM, and -/// that its lease still reserves its launch, as after a plan that failed. +/// no record of a clone's progress or failure, and that its lease still reserves its launch, as +/// after a plan that failed. fn assert_untouched(acquired: &Acquired) { - assert!(!sparse_path(acquired).exists()); - assert_eq!(acquired.locked.reserved(), Some(acquired.lease.launch())); + let locked = &acquired.locked; + for path in [ + sparse_path(acquired), + locked.clone_progress_path(), + locked.clone_failure_path(), + ] { + assert!(!path.exists(), "{}", path.display()); + } + assert_eq!(locked.reserved(), Some(acquired.lease.launch())); } /// Returns a file of `length` bytes on hugetlbfs, whose I/O block size is a huge page of 2 MiB, @@ -602,6 +643,11 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let fixture = Fixture::new(); let mut acquired = fixture.acquire("sandbox"); let sparse = sparse_path(&acquired); + // The image or the scratch image of a request is a file of the sandbox's store: the request's + // settings are invalid. + let store_file = |result: std::result::Result| { + refused(result, Evidence::InvalidSettings) + }; let refusal = |what: &str, path: &Path, owned: &Path| { ( ErrorKind::Unsupported, @@ -620,7 +666,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let binding = DistroBinding::import(&layer(&sparse), PolicyVerdict::ACCEPTED).unwrap(); let staged = seal_of(&sparse); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&sparse, &binding), NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &tmpfs_request(&sparse, &binding), NO_CONTENT)), refusal("image", &sparse, &sparse) ); assert_eq!(seal_of(&sparse), staged); @@ -629,7 +675,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { fs::hard_link(&sparse, &hard).unwrap(); let hard_binding = DistroBinding::import(&layer(&hard), PolicyVerdict::ACCEPTED).unwrap(); assert_eq!( - failure(fixture.plan( + store_file(fixture.plan( &mut acquired, &tmpfs_request(&hard, &hard_binding), NO_CONTENT @@ -643,7 +689,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let aliased_binding = DistroBinding::import(&layer(&aliased), PolicyVerdict::ACCEPTED).unwrap(); assert_eq!( - failure(fixture.plan( + store_file(fixture.plan( &mut acquired, &tmpfs_request(&aliased, &aliased_binding), NO_CONTENT @@ -660,7 +706,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let (scratch, scratch_binding) = fixture.scratch("scratch.ext4", SCRATCH_LEN); let request = scratch_request(&sparse, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal("image", &sparse, &sparse) ); assert_eq!(seal_of(&sparse), staged); @@ -670,7 +716,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { let sparse_binding = ScratchBinding::bind(&sparse).unwrap(); let request = scratch_request(&image, &image_binding, &sparse, &sparse_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal("scratch image", &sparse, &sparse) ); assert_eq!(fs::metadata(&sparse).unwrap().len(), SCRATCH_LEN); @@ -685,7 +731,7 @@ fn files_of_the_sandboxs_store_are_refused_as_images() { stage(&owned, IMAGE_LEN); let binding = DistroBinding::import(&layer(&owned), PolicyVerdict::ACCEPTED).unwrap(); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&owned, &binding), NO_CONTENT)), + store_file(fixture.plan(&mut acquired, &tmpfs_request(&owned, &binding), NO_CONTENT)), refusal("image", &owned, &owned) ); assert_eq!(fs::metadata(&owned).unwrap().len(), IMAGE_LEN); @@ -700,6 +746,11 @@ fn links_and_directories_under_the_names_of_the_sandboxs_store_are_refused() { let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); let sparse = sparse_path(&acquired); + // The sandbox's store holds an entry that it never creates: the host's state is not as the + // store keeps it. + let store_entry = |result: std::result::Result| { + refused(result, Evidence::HostFailure) + }; let refusal = |owned: &Path| { ( ErrorKind::Unavailable, @@ -715,7 +766,7 @@ fn links_and_directories_under_the_names_of_the_sandboxs_store_are_refused() { let owned = acquired.locked.dir().join(name); fs::create_dir(&owned).unwrap(); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_entry(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal(&owned) ); assert!(owned.is_dir()); @@ -729,7 +780,7 @@ fn links_and_directories_under_the_names_of_the_sandboxs_store_are_refused() { DistroBinding::import(&layer(&through), PolicyVerdict::ACCEPTED).unwrap(); for request in [tmpfs_request(&through, &through_binding), request.clone()] { assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + store_entry(fixture.plan(&mut acquired, &request, NO_CONTENT)), refusal(&sparse) ); } @@ -778,7 +829,7 @@ fn a_scratch_image_that_is_a_file_of_a_generation_s_snapshot_is_refused() { let other = other.join("memory.bin"); stage(&other, SCRATCH_LEN); // Through a hard link among the scratch images, another name of the file, and by its own - // path, in the store's directory. + // path, in the store's directory: the request's settings are invalid. let cases = [ (&leased, fixture.path("scratches", "leased.raw")), (&other, fixture.path("scratches", "other.raw")), @@ -811,7 +862,10 @@ fn a_scratch_image_that_is_a_file_of_a_generation_s_snapshot_is_refused() { ) }; assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), (ErrorKind::Unsupported, expected) ); assert_untouched(&acquired); @@ -859,7 +913,10 @@ fn a_scratch_image_in_the_template_store_is_refused() { let request = scratch_request(&image, &image_binding, &scratch, &binding); let location = location_of(&File::open(file).unwrap()).unwrap(); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -922,7 +979,10 @@ fn a_scratch_image_with_another_name_is_refused() { let binding = ScratchBinding::bind(&scratch).unwrap(); let request = scratch_request(&image, &image_binding, &scratch, &binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -945,6 +1005,11 @@ fn names_that_the_file_system_takes_for_those_of_the_stores_files_are_the_stores let (image, image_binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); let dir = acquired.locked.dir().to_owned(); + // The image or the scratch image of a request is a file of the sandbox's store: the request's + // settings are invalid. + let store_file = |result: std::result::Result| { + refused(result, Evidence::InvalidSettings) + }; let refusal = |what: &str, path: &Path, owned: &Path| { ( ErrorKind::Unsupported, @@ -976,7 +1041,7 @@ fn names_that_the_file_system_takes_for_those_of_the_stores_files_are_the_stores let same = same_file(&cased, &owned); let result = fixture.plan(&mut acquired, &tmpfs_request(&cased, &binding), NO_CONTENT); if same { - assert_eq!(failure(result), refusal("image", &cased, &owned)); + assert_eq!(store_file(result), refusal("image", &cased, &owned)); } else { drop(result.unwrap()); } @@ -994,7 +1059,10 @@ fn names_that_the_file_system_takes_for_those_of_the_stores_files_are_the_stores let same = same_file(&cased, &sparse); let result = fixture.plan(&mut acquired, &request, NO_CONTENT); if same { - assert_eq!(failure(result), refusal("scratch image", &cased, &sparse)); + assert_eq!( + store_file(result), + refusal("scratch image", &cased, &sparse) + ); } else { drop(result.unwrap()); assert_eq!( @@ -1023,8 +1091,9 @@ fn templates_that_a_runtime_of_other_contracts_captured_are_refused() { let fixture = Fixture::with(|class, _, _| change(class)); let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)), + refused(plan, Evidence::IncompatibleHost), ( ErrorKind::Unsupported, format!( @@ -1039,6 +1108,166 @@ fn templates_that_a_runtime_of_other_contracts_captured_are_refused() { } } +#[test] +fn images_that_their_binding_does_not_authorize_and_malformed_generations_are_refused() { + let fixture = Fixture::new(); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("sandbox"); + let request = tmpfs_request(&image, &binding); + // A binding whose verdict rejects the image authorizes no clone of it. + let verdict = PolicyVerdict { + result: PolicyResult::Rejected, + ..PolicyVerdict::ACCEPTED + }; + let rejected = DistroBinding::import(&layer(&image), verdict).unwrap(); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &rejected), NO_CONTENT); + refused(plan, Evidence::AuthorizationDenied); + // A binding of another revision of the distro policy than the request's is stale. + let stale = CloneRequest { + policy_revision: DISTRO_POLICY_REVISION + 1, + ..request.clone() + }; + refused( + fixture.plan(&mut acquired, &stale, NO_CONTENT), + Evidence::StaleBinding, + ); + // A failure of the request quarantines nothing. + let failure = fixture.plan(&mut acquired, &stale, NO_CONTENT).unwrap_err(); + assert!(!failure.quarantine(&fixture.store, None).unwrap()); + assert_untouched(&acquired); + // A generation whose key is gone is malformed: a failure of the template, whose generation it + // quarantines through its tombstone, once the sandbox's lock is released, as the lock order + // asks. + let generation = fixture.generation.generation(); + let key = fixture + .store + .payload_dir(generation) + .join(crate::host::template::KEY_NAME); + fs::remove_file(key).unwrap(); + let failure = fixture + .plan(&mut acquired, &request, NO_CONTENT) + .unwrap_err(); + assert_eq!( + (failure.record().evidence, failure.outcome()), + (Evidence::MalformedGeneration, Outcome::Template), + "{failure}" + ); + assert_untouched(&acquired); + drop(acquired); + let other = Fixture::new(); + let error = failure.quarantine(&other.store, None).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Other, "{error}"); + assert!(other.store.tombstones().unwrap().is_empty()); + assert!(failure.quarantine(&fixture.store, None).unwrap()); + assert_eq!(fixture.store.tombstones().unwrap(), [generation]); + // The generation takes no new lease. + let error = fixture + .store + .acquire( + generation, + fixture.provision("next"), + &files(), + &after(&fixture.generation, 1), + None, + ) + .unwrap_err(); + assert_eq!(error.code(), ErrorCode::TemplateTombstoned, "{error}"); +} + +#[test] +fn keys_that_their_read_refuses_for_what_they_are_are_failures_of_the_template() { + let fixture = Fixture::new(); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("sandbox"); + let request = tmpfs_request(&image, &binding); + let generation = fixture.generation.generation(); + let key = fixture + .store + .payload_dir(generation) + .join(crate::host::template::KEY_NAME); + let malformed = |acquired: &mut Acquired, what: &str| { + let failure = fixture.plan(acquired, &request, NO_CONTENT).unwrap_err(); + assert_eq!( + (failure.record().evidence, failure.outcome()), + (Evidence::MalformedGeneration, Outcome::Template), + "{what}: {failure}" + ); + assert_untouched(acquired); + failure + }; + // A link in place of the key, a directory there, and a key that holds more than a key may are + // none that the store writes: the read refuses each for what it is, a failure of the template, + // not of an operation of the host's. Windows denies the open of a directory, as a file's access + // control list may deny one, which is the host's, so only Unix tries the directory. + let moved = key.with_extension("moved"); + fs::rename(&key, &moved).unwrap(); + if symlink_file(&moved, &key) { + malformed(&mut acquired, "a link"); + fs::remove_file(&key).unwrap(); + } + if cfg!(unix) { + fs::create_dir(&key).unwrap(); + malformed(&mut acquired, "a directory"); + fs::remove_dir(&key).unwrap(); + } + fs::rename(&moved, &key).unwrap(); + crate::host::store::files::write_private(&key, &vec![0; TemplateKey::MAX_LEN + 1]).unwrap(); + let failure = malformed(&mut acquired, "an oversized key"); + // So the failure quarantines the generation, once the sandbox's lock is released. + drop(acquired); + assert!(failure.quarantine(&fixture.store, None).unwrap()); + assert_eq!(fixture.store.tombstones().unwrap(), [generation]); +} + +#[test] +fn keys_of_another_encoding_version_are_failures_of_the_host_that_quarantine_nothing() { + // A store that a build of another encoding of keys kept, as one that a rollback of the host's + // build leaves, holds generations whose keys this build cannot read: the host cannot restore + // them, but a build that reads their keys can, so their failures quarantine nothing. + let fixture = Fixture::new(); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("sandbox"); + let generation = fixture.generation.generation(); + let key = fixture + .store + .payload_dir(generation) + .join(crate::host::template::KEY_NAME); + let mut bytes = fs::read(&key).unwrap(); + let version = TemplateKey::VERSION + 1; + bytes[..2].copy_from_slice(&version.to_le_bytes()); + crate::host::store::files::write_private(&key, &bytes).unwrap(); + let failure = fixture + .plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT) + .unwrap_err(); + assert_eq!( + (failure.record().evidence, failure.outcome()), + (Evidence::IncompatibleHost, Outcome::Host), + "{failure}" + ); + assert_eq!(failure.code(), ErrorCode::FormatUnsupported, "{failure}"); + let unread = format!( + "the template key has encoding version {version}; this build reads version {} only", + TemplateKey::VERSION + ); + assert!(failure.message().contains(&unread), "{failure}"); + assert_untouched(&acquired); + drop(acquired); + assert!(!failure.quarantine(&fixture.store, None).unwrap()); + assert!(fixture.store.tombstones().unwrap().is_empty()); + // The generation still takes new leases. + let next = fixture + .store + .acquire( + generation, + fixture.provision("next"), + &files(), + &after(&fixture.generation, 1), + None, + ) + .unwrap(); + assert_eq!(next.lease.generation(), generation); +} + #[test] fn a_clone_on_a_scratch_image_claims_it_and_reads_no_content() { let fixture = Fixture::new(); @@ -1093,6 +1322,9 @@ fn a_scratch_image_serves_one_clone_at_a_time() { assert!(started.elapsed() >= wait); assert_eq!(error.code(), ErrorCode::ScratchClaimed, "{error}"); assert_eq!(error.kind(), ErrorKind::TimedOut, "{error}"); + // The image is not fresh for this clone: the request's. + assert_eq!(error.record().evidence, Evidence::UnusableScratch); + assert_eq!(error.outcome(), Outcome::Request); assert_eq!( error.message(), format!( @@ -1128,7 +1360,10 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { fs::rename(©, &scratch).unwrap(); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), changed(&scratch, "identity") ); // So has a directory in its place. The image stays, under another name, so that the @@ -1138,7 +1373,10 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { fs::create_dir(&scratch).unwrap(); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), changed(&scratch, "identity") ); // A binding of another kind names another file. @@ -1148,7 +1386,10 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { let scratch_binding: ScratchBinding = serde_json::from_value(value).unwrap(); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), changed(&scratch, "kind") ); assert_untouched(&acquired); @@ -1158,16 +1399,16 @@ fn scratch_images_whose_identity_or_kind_changed_are_refused() { fn images_of_another_length_or_geometry_are_refused() { let fixture = Fixture::new(); let mut acquired = fixture.acquire("sandbox"); - let mut refusal = |length: u64| { + let mut refusal = |length: u64, evidence: Evidence| { let (image, binding) = fixture.image("other.erofs", length); - let failed = - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); + let failed = refused(plan, evidence); fs::remove_file(&image).unwrap(); (shown(&image), failed) }; // A clone's image has the placeholder's length. for length in [IMAGE_LEN - 512, IMAGE_LEN + 512, 2 * IMAGE_LEN] { - let (image, failed) = refusal(length); + let (image, failed) = refusal(length, Evidence::SuppliedLength); assert_eq!( failed, ( @@ -1181,7 +1422,7 @@ fn images_of_another_length_or_geometry_are_refused() { } // OpenVMM attaches no block device of another length. for length in [0, IMAGE_LEN + 1] { - let (image, failed) = refusal(length); + let (image, failed) = refusal(length, Evidence::SuppliedGeometry); assert_eq!( failed, ( @@ -1205,8 +1446,9 @@ fn images_of_another_length_or_geometry_are_refused() { let mut acquired = fixture.acquire("sandbox"); let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let physical = fixture.physical; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)), + refused(plan, Evidence::SuppliedGeometry), ( ErrorKind::Unsupported, format!( @@ -1229,7 +1471,10 @@ fn scratch_images_of_another_length_or_geometry_are_refused() { let (scratch, scratch_binding) = fixture.scratch("other.ext4", length); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedLength + ), ( ErrorKind::Unsupported, format!( @@ -1250,7 +1495,10 @@ fn scratch_images_of_another_length_or_geometry_are_refused() { let (scratch, scratch_binding) = fixture.scratch("unaligned.ext4", SCRATCH_LEN + 1); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedGeometry + ), ( ErrorKind::Unsupported, format!( @@ -1276,8 +1524,10 @@ fn sandboxes_whose_filesystem_presents_another_scratch_geometry_are_refused() { let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire("sandbox"); let (physical, other) = (fixture.physical, other_physical(fixture.physical)); + // The sandbox's filesystem is the host's, which cannot restore the template. + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT)), + refused(plan, Evidence::IncompatibleHost), ( ErrorKind::Unsupported, format!( @@ -1293,7 +1543,10 @@ fn sandboxes_whose_filesystem_presents_another_scratch_geometry_are_refused() { let (scratch, scratch_binding) = fixture.scratch("scratch.ext4", SCRATCH_LEN); let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedGeometry + ), ( ErrorKind::Unsupported, format!( @@ -1354,8 +1607,9 @@ fn files_on_a_filesystem_of_another_block_size_are_refused() { distro: huge_distro, ..NO_CONTENT }; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access)), + refused(plan, Evidence::SuppliedGeometry), presented(format!("the image {}", shown(&image)), "placeholder") ); // The scratch image: its binding and its check reach a file on hugetlbfs at its path. @@ -1371,7 +1625,10 @@ fn files_on_a_filesystem_of_another_block_size_are_refused() { }; let request = scratch_request(&image, &binding, &scratch, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, access)), + refused( + fixture.plan(&mut acquired, &request, access), + Evidence::SuppliedGeometry + ), presented(format!("the scratch image {}", shown(&scratch)), "scratch") ); // The scratch device of a clone in RAM, which a sandbox on hugetlbfs creates there. @@ -1381,8 +1638,9 @@ fn files_on_a_filesystem_of_another_block_size_are_refused() { create: &create, ..NO_CONTENT }; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&image, &binding), access)), + refused(plan, Evidence::IncompatibleHost), presented( format!( "the scratch device of the clone {}, in the sandbox's filesystem,", @@ -1415,7 +1673,10 @@ fn memory_targets_are_whole_blocks_from_the_templates_ram_to_its_capacity() { ..tmpfs_request(&image, &binding) }; assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidTargets + ), ( ErrorKind::Unsupported, format!( @@ -1444,8 +1705,8 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let (scratch, scratch_binding) = fixture.scratch("scratch.ext4", SCRATCH_LEN); let mut acquired = fixture.acquire("sandbox"); - let mut refusal = |request: &CloneRequest| { - let (kind, message) = failure(fixture.plan(&mut acquired, request, NO_CONTENT)); + let mut refusal = |request: &CloneRequest, evidence: Evidence| { + let (kind, message) = refused(fixture.plan(&mut acquired, request, NO_CONTENT), evidence); assert_eq!(kind, ErrorKind::Unsupported, "{message}"); message }; @@ -1461,7 +1722,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { .unwrap(), ..tmpfs.clone() }; - let message = refusal(&profile); + let message = refusal(&profile, Evidence::InvalidSettings); assert!( message.starts_with("a clone restores an adapter's plan, not a plan of the sandbox"), "{message}" @@ -1476,7 +1737,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { ..tmpfs.clone() }; assert_eq!( - refusal(&mistaken), + refusal(&mistaken, Evidence::InvalidFormat), format!( "the image {} is not the erofs layer with identity {UUID} that its distro binding \ names", @@ -1488,17 +1749,19 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { adapter: SandboxPlan::adapter(SandboxDistro::new(path, distro.image()), SandboxUpper::RAM), ..tmpfs.clone() }; - let message = refusal(&at("distro.erofs")); + let message = refusal(&at("distro.erofs"), Evidence::InvalidSettings); assert!( message.starts_with("the image of a clone must be absolute"), "{message}" ); - let message = refusal(&at(&format!("{},x", image.to_str().unwrap()))); + let comma = at(&format!("{},x", image.to_str().unwrap())); + let message = refusal(&comma, Evidence::InvalidSettings); assert!( message.starts_with("disk paths containing commas, semicolons, or unbalanced brackets"), "{message}" ); - let message = refusal(&at(image.with_extension("vhdx").to_str().unwrap())); + let vhdx = at(image.with_extension("vhdx").to_str().unwrap()); + let message = refusal(&vhdx, Evidence::InvalidSettings); assert!( message.starts_with("disk paths ending in .vhdx"), "{message}" @@ -1508,12 +1771,13 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { scratch: Some(scratch_binding.clone()), ..tmpfs.clone() }; - let message = refusal(&on_scratch("scratch.ext4")); + let message = refusal(&on_scratch("scratch.ext4"), Evidence::InvalidSettings); assert!( message.starts_with("the scratch image of a clone must be absolute"), "{message}" ); - let message = refusal(&on_scratch(scratch.with_extension("iso").to_str().unwrap())); + let iso = on_scratch(scratch.with_extension("iso").to_str().unwrap()); + let message = refusal(&iso, Evidence::InvalidSettings); assert!( message.starts_with("disk paths ending in .iso"), "{message}" @@ -1524,7 +1788,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { ..scratch_request(&image, &binding, &scratch, &scratch_binding) }; assert_eq!( - refusal(&unbound), + refusal(&unbound, Evidence::InvalidSettings), format!( "the upper of a clone on the scratch image {} needs the image's scratch binding", shown(&scratch) @@ -1535,7 +1799,7 @@ fn requests_that_no_clone_takes_are_refused_before_anything_is_opened() { ..tmpfs.clone() }; assert_eq!( - refusal(&bound), + refusal(&bound, Evidence::InvalidSettings), "the upper of a clone in RAM takes no scratch binding" ); // None of them opened anything: the scratch image is free, and the lease still reserves the @@ -1554,7 +1818,10 @@ fn networks_that_the_template_does_not_have_are_refused() { let generation = fixture.generation.generation(); let attached = with_network(request.clone(), Some(network(NetworkAccess::Allow))); assert_eq!( - failure(fixture.plan(&mut acquired, &attached, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &attached, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -1583,7 +1850,10 @@ fn networks_that_the_template_does_not_have_are_refused() { let request = tmpfs_request(&image, &binding); let generation = fixture.generation.generation(); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -1601,7 +1871,10 @@ fn networks_that_the_template_does_not_have_are_refused() { Some(SandboxNetwork::new(other, NetworkPolicy::new(NetworkAccess::Allow)).unwrap()), ); assert_eq!( - failure(fixture.plan(&mut acquired, &moved, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &moved, NO_CONTENT), + Evidence::InvalidSettings + ), ( ErrorKind::Unsupported, format!( @@ -1625,7 +1898,8 @@ fn clones_are_planned_under_the_lock_that_acquired_their_lease_before_their_laun // A lease of another store. let other = Fixture::new(); let mut elsewhere = other.acquire("sandbox"); - let (kind, message) = failure(fixture.plan(&mut elsewhere, &request, NO_CONTENT)); + let plan = fixture.plan(&mut elsewhere, &request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::HostFailure); assert_eq!(kind, ErrorKind::Other, "{message}"); assert_eq!( message, @@ -1643,7 +1917,8 @@ fn clones_are_planned_under_the_lock_that_acquired_their_lease_before_their_laun drop(locked); let locked = SandboxStore::lock_existing(lease.sandbox(), &files(), None).unwrap(); let mut relocked = Acquired { lease, locked }; - let (kind, message) = failure(fixture.plan(&mut relocked, &request, NO_CONTENT)); + let plan = fixture.plan(&mut relocked, &request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::HostFailure); assert_eq!(kind, ErrorKind::Other, "{message}"); assert!( message.starts_with("a clone is planned under the lock that acquired its lease"), @@ -1659,7 +1934,8 @@ fn clones_are_planned_under_the_lock_that_acquired_their_lease_before_their_laun .begin_launch(acquired.lease.launch(), key.runtime(), &[7; CAPABILITY_LEN]) .unwrap(), ); - let (kind, message) = failure(fixture.plan(&mut acquired, &request, NO_CONTENT)); + let plan = fixture.plan(&mut acquired, &request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::HostFailure); assert_eq!(kind, ErrorKind::Other, "{message}"); assert!(!sparse_path(&acquired).exists()); } @@ -1673,8 +1949,9 @@ fn files_that_their_paths_do_not_name_as_checked_are_refused() { let link = fixture.path("images", "link.erofs"); if symlink_file(&target, &link) { let binding = DistroBinding::import(&layer(&link), PolicyVerdict::ACCEPTED).unwrap(); + let plan = fixture.plan(&mut acquired, &tmpfs_request(&link, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&link, &binding), NO_CONTENT)), + refused(plan, Evidence::SuppliedIdentity), ( ErrorKind::Unavailable, format!( @@ -1691,7 +1968,10 @@ fn files_that_their_paths_do_not_name_as_checked_are_refused() { let scratch_binding = ScratchBinding::bind(&link).unwrap(); let request = scratch_request(&image, &binding, &link, &scratch_binding); assert_eq!( - failure(fixture.plan(&mut acquired, &request, NO_CONTENT)), + refused( + fixture.plan(&mut acquired, &request, NO_CONTENT), + Evidence::SuppliedIdentity + ), ( ErrorKind::Unavailable, format!( @@ -1724,8 +2004,9 @@ fn files_that_their_paths_do_not_name_as_checked_are_refused() { } else { "is writable by other users (mode 666)" }; + let plan = fixture.plan(&mut acquired, &tmpfs_request(&path, &binding), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &tmpfs_request(&path, &binding), NO_CONTENT)), + refused(plan, Evidence::SuppliedIdentity), ( ErrorKind::Unavailable, format!("the image {} {problem}", shown(&path)) @@ -1956,8 +2237,9 @@ fn settings_that_no_activation_record_holds_are_refused_before_anything_is_creat "the workload identity's UID or GID is 0 or 4294967295", ), ] { + let plan = fixture.plan(&mut acquired, &with_workload(workload), NO_CONTENT); assert_eq!( - failure(fixture.plan(&mut acquired, &with_workload(workload), NO_CONTENT)), + refused(plan, Evidence::InvalidActivate), ( ErrorKind::Unsupported, format!("no clone activates with the settings of the request: {problem}") @@ -1978,7 +2260,8 @@ fn settings_that_no_activation_record_holds_are_refused_before_anything_is_creat binding: gpt_binding, ..request.clone() }; - let (kind, message) = failure(fixture.plan(&mut acquired, &gpt_request, NO_CONTENT)); + let plan = fixture.plan(&mut acquired, &gpt_request, NO_CONTENT); + let (kind, message) = refused(plan, Evidence::InvalidFormat); assert_eq!(kind, ErrorKind::Unsupported, "{message}"); assert!( message.starts_with("the image of a clone must be an EROFS layer"), @@ -1987,3 +2270,53 @@ fn settings_that_no_activation_record_holds_are_refused_before_anything_is_creat assert_untouched(&acquired); drop(fixture.plan(&mut acquired, &request, NO_CONTENT).unwrap()); } + +#[test] +fn only_a_path_that_no_longer_names_its_supplied_file_is_the_request_s() { + // The open's refusal of a link or of a file that is not regular, which the system does not + // report, a file that went, and a path that the current user may no longer open, are the + // path's. + assert!(path_refused(&io::Error::other( + "it is a link, or not a regular file" + ))); + #[cfg(unix)] + let (refusals, failures) = { + use nix::errno::Errno; + + ( + [Errno::ENOENT, Errno::ENOTDIR, Errno::EACCES], + [Errno::EIO, Errno::EMFILE, Errno::ENOMEM], + ) + }; + // ERROR_FILE_NOT_FOUND, ERROR_PATH_NOT_FOUND, and ERROR_ACCESS_DENIED; ERROR_IO_DEVICE, + // ERROR_NOT_ENOUGH_MEMORY, and ERROR_NO_SYSTEM_RESOURCES. + #[cfg(windows)] + let (refusals, failures) = ([2_u32, 3, 5], [1117_u32, 8, 1450]); + for errno in refusals { + let error = io::Error::from_raw_os_error(errno as i32); + assert!(path_refused(&error), "{error}"); + } + for errno in failures { + let error = io::Error::from_raw_os_error(errno as i32); + assert!(!path_refused(&error), "{error}"); + } + // A file that went from its path is the request's. + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("image"); + let fault = check_at_path(&path, (1, 2), "the image").unwrap_err(); + assert_eq!(fault.evidence, Evidence::SuppliedIdentity); + assert_eq!( + fault.error.kind(), + ErrorKind::Unavailable, + "{}", + fault.error + ); + assert!( + fault + .error + .message() + .starts_with("the image cannot be used: "), + "{}", + fault.error + ); +} diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index af9233a2c..1cde8a7d1 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -15,16 +15,21 @@ use std::thread; use std::time::{Duration, Instant}; use super::*; -use crate::activation::{ReadyActivation, StoreGeneration}; +use crate::activation::{ReadyActivation, StoreGeneration, TransactionId}; use crate::host::artifacts; use crate::host::platform::Transport; -use crate::host::process::{Process, ProcessRef}; +use crate::host::process::{BootId, Process, ProcessRef}; +use crate::host::store::files::{sync_dir, write_private}; use crate::host::store::{ - CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, - LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, Recovery, RuntimeIdentity, + CLONE_ACTIVATION_NAME, CLONE_BINDING_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, + CLONE_SCRATCH_NAME, JOURNAL_NAME, LaunchState, LockedStore, OUTCOME_NAME, RESTORE_READY_NAME, + Recovery, RuntimeIdentity, RuntimeRecord, }; -use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; +use crate::host::supervisor::{self, Reattached, StopOutcome, Vmm, VmmCommand}; +use crate::host::template::Retirement; use crate::host::template::clone::launch::{Hooks, Point}; +use crate::host::template::clone::reattach::{reattach_clone_syncing, reattach_clone_with}; +use crate::host::template::clone::records::{read_failure, read_progress, write_failure}; use crate::host::template::store::SNAPSHOT_ARTIFACTS; use crate::host::template::test_support::artifact; @@ -37,7 +42,9 @@ const ROLE_SANDBOX: &str = "NVX_CLONE_LAUNCH_TEST_SANDBOX"; /// The environment variable that names the image of the clone of a [`launch_role`]. const ROLE_IMAGE: &str = "NVX_CLONE_LAUNCH_TEST_IMAGE"; /// The environment variable that names where a [`launch_role`] crashes: at the point `prepared` -/// or `begun` of its launch, or once its launch returned, `launched`. +/// or `begun` of its launch, once its launch returned, `launched`, at the steps `activating`, +/// `activated`, `starting`, `started`, and `committed` of its restore sequence, and once its +/// commit left the launch's journal beside the runtime record, `superseded`. const ROLE_CRASH: &str = "NVX_CLONE_LAUNCH_TEST_CRASH"; /// The environment variable that names the working directory of the VMM of a [`launch_role`]. const ROLE_VMM: &str = "NVX_CLONE_LAUNCH_TEST_VMM"; @@ -72,6 +79,9 @@ pub(super) const TIMEOUT: Duration = Duration::from_secs(60); /// Longest life of a [`restore_vmm_role`] that nothing ends, so that a failed test leaves no /// process. const VMM_LIFETIME: Duration = Duration::from_secs(180); +/// How long a [`restore_vmm_role`] whose plan closes the event's endpoint waits once it connected +/// the endpoint, and once it closed it, before it reports its error. +const CLOSE_PAUSE: Duration = Duration::from_millis(300); /// Returns `bytes` in lowercase hexadecimal. pub(super) fn hex(bytes: &[u8]) -> String { @@ -178,12 +188,73 @@ pub(super) fn assert_released(sandbox: &Path) { for name in [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ] { assert!(!exists(&sandbox.join(name)), "{name}"); } } +/// Checks that `failure`, of the clone of `lease` in the sandbox of `locked`, failed at `stage`, +/// once the clone got as far as `progress`, with `evidence`, which attributes it, and the teardown +/// `teardown`, and that the sandbox records it, with the clone's lease and the transaction of its +/// restore. +pub(super) fn assert_recorded( + failure: &CloneFailure, + lease: &Lease, + locked: &LockedStore, + (stage, progress): (Stage, Progress), + evidence: Evidence, + teardown: Teardown, +) { + let record = failure.record(); + assert_eq!( + ( + record.stage, + record.progress, + record.evidence, + &record.teardown + ), + (stage, progress, evidence, &teardown), + "{failure}" + ); + assert_eq!(failure.outcome(), evidence.outcome(), "{failure}"); + assert_eq!( + (record.store, record.lease, record.template, &record.launch), + (lease.store(), lease.id(), lease.template(), lease.launch()) + ); + let activation = fs::read(locked.clone_activation_path()).unwrap(); + let activation = ActivationRecord::decode(&activation).unwrap(); + let transaction = activation.restore().unwrap().transaction; + assert_eq!(record.transaction, Some(transaction)); + assert_eq!(record.code, failure.code()); + assert!(record.message.starts_with(failure.message()), "{failure}"); + let records = CloneRecords::read(locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(record)); + assert_eq!(&records.lease, lease); +} + +/// Returns what a reattach found, as test messages say it. +pub(super) fn describe(reattached: &ReattachedClone) -> String { + match reattached { + ReattachedClone::Live { vmm, .. } => format!("the live VM {:?}", vmm.record()), + ReattachedClone::Ended { progress, report } => { + format!("a clone that ended at its step {progress}, with the report {report:?}") + } + ReattachedClone::Failed(failure) => format!("the failure {:?}", failure.record()), + ReattachedClone::Indeterminate(reason) => format!("an indeterminate VM: {reason}"), + } +} + +/// Returns the failure that a reattach found, and fails on anything else. +pub(super) fn failed(reattached: ReattachedClone) -> CloneFailure { + match reattached { + ReattachedClone::Failed(failure) => failure, + other => panic!("the reattach found {}", describe(&other)), + } +} + /// The seam of the command of a launch that must start no VMM. fn never_spawned(_: VmmCommand<'_>) -> VmmCommand<'_> { panic!("a launch that is refused made the command of a VMM") @@ -298,16 +369,32 @@ impl VmmDir { assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); } + /// Waits until the VMM, which exits on its own, as its [plan](Self::plan) asks, has exited. + pub(super) fn wait_exit(&self) { + let reference = self.reference.as_ref().expect("the test knows the VMM"); + if let Some(process) = Process::open(reference).unwrap() { + assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + } + } + /// Changes how the VMM sends its restore-ready event, before it starts, by the words of /// `plan`: /// /// - `delay:MS` sends it after MS milliseconds; /// - `none` sends nothing, and runs on; /// - `exit` sends nothing, and exits at once; + /// - `close` connects to the event's endpoint, and closes it without a byte of the event, then + /// exits, [`CLOSE_PAUSE`] after each, as OpenVMM does once the launch of its VM worker + /// fails; + /// - `fatal:CODE`, with `exit` or `close`, writes the line with which OpenVMM reports a fatal + /// error of the time ABI code `CODE`, and the error's chain, on its standard error, after a + /// warning that names another code, before it exits; /// - `other` sends a version of the event that OpenVMM does not send; /// - `partial` sends the first bytes of the event, then closes its end of the endpoint; /// - `flush` flushes the event as OpenVMM does, which waits on Windows until the host read - /// it, before the VMM writes [`STARTED`], then notes it in [`FLUSHED`]. + /// it, before the VMM writes [`STARTED`], then notes it in [`FLUSHED`]; + /// - `report:CATEGORY:STATUS` writes OpenVMM's outcome report of a VM whose outcome is of the + /// category `CATEGORY`, with the status `STATUS`, as the VMM exits. pub(super) fn plan(&self, plan: &str) { fs::write(self.path().join(EVENT_PLAN), plan).unwrap(); } @@ -338,9 +425,9 @@ impl Drop for VmmDir { /// the control endpoint beside the snapshot, and on Windows opens the snapshot's directory and /// files sharing nothing but reading, and serves its control endpoint's named pipe; sends the /// restore-ready event as OpenVMM does, or as its [plan](VmmDir::plan) says; writes [`STARTED`]; -/// then runs until a file named [`EXIT_MARKER`] appears, or for [`VMM_LIFETIME`] at most, and -/// removes its control socket as it exits, as OpenVMM does at a normal exit. It never answers on -/// its control endpoint. +/// then runs until a file named [`EXIT_MARKER`] appears, or for [`VMM_LIFETIME`] at most, writes +/// the outcome report that its plan asks for, and removes its control socket as it exits, as +/// OpenVMM does at a normal exit. It never answers on its control endpoint. /// /// Where OpenVMM would refuse its arguments, this VMM writes why in [`STARTED`] after its /// capability and whether it sent its event. Where OpenVMM would abort a restore whose event it @@ -357,11 +444,33 @@ fn restore_vmm_role() { std::io::stdin().read_exact(&mut capability).unwrap(); let plan = fs::read_to_string(directory.join(EVENT_PLAN)).unwrap_or_default(); let words: Vec<&str> = plan.split_whitespace().collect(); - if words.contains(&"exit") { - return; - } let arguments: Vec = serde_json::from_slice(&fs::read(directory.join(ARGUMENTS)).unwrap()).unwrap(); + if words.contains(&"exit") || words.contains(&"close") { + if words.contains(&"close") { + // As OpenVMM does once the launch of its VM worker fails: it connected the event's + // endpoint before it launched the worker, closes the endpoint as the launch fails, + // tears down, and only then reports the error as it exits. + let ready = argument(&arguments, "--restore-ready-path").unwrap(); + let sink = send_ready(Path::new(&ready), b""); + thread::sleep(CLOSE_PAUSE); + drop(sink); + thread::sleep(CLOSE_PAUSE); + } + if let Some(code) = words.iter().find_map(|word| word.strip_prefix("fatal:")) { + // OpenVMM's warnings may name a time ABI code too; its fatal error leads with the + // first code of the error's chain, which follows it. + writeln!( + std::io::stderr(), + "NVX-CPU-PROFILE-FALLBACK: [E_PROFILE_UNSUPPORTED] the test's warning\n\ + fatal error: [{code}] failed to launch vm worker\n\nCaused by:\n \ + 0: [{code}] the test's restore" + ) + .unwrap(); + } + write_report(&words, &arguments); + return; + } let (held, problem) = match hold_snapshot(&arguments) { Ok(held) => (Some(held), String::new()), Err(problem) => (None, format!(" {problem}")), @@ -402,11 +511,31 @@ fn restore_vmm_role() { while !directory.join(EXIT_MARKER).exists() && started.elapsed() < VMM_LIFETIME { thread::sleep(Duration::from_millis(10)); } + write_report(&words, &arguments); if let Some(held) = held { held.release(); } } +/// Writes OpenVMM's outcome report that the word `report:CATEGORY:STATUS` among `words` of a +/// [`restore_vmm_role`]'s plan asks for, if any: a VM whose outcome is of the category +/// `CATEGORY`, with the status `STATUS`, at the `--microvm-report` path among `arguments`, as +/// OpenVMM writes it before it exits. +fn write_report(words: &[&str], arguments: &[String]) { + let Some(report) = words.iter().find_map(|word| word.strip_prefix("report:")) else { + return; + }; + let (category, status) = report.split_once(':').unwrap(); + let status: i64 = status.parse().unwrap(); + let path = argument(arguments, "--microvm-report").unwrap(); + let report = serde_json::json!({ + "schema_version": 1, + "outcome": {"operation": "managed", "category": category, "status_code": status}, + "teardown": {"workers": true}, + }); + fs::write(path, serde_json::to_vec(&report).unwrap()).unwrap(); +} + /// Returns the value of the option `name` among `arguments`, if they name it. fn argument(arguments: &[String], name: &str) -> Option { arguments @@ -994,7 +1123,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let plan = fixture.plan(&mut unbound, &request, NO_CONTENT).unwrap(); fs::remove_file(unbound.locked.binding_path()).unwrap(); assert_eq!( - failure(plan.launch_with(&mut unbound, openvmm, &runtime, hooks)), + refused_at( + plan.launch_with(&mut unbound, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure + ), ( ErrorKind::Other, format!( @@ -1011,7 +1144,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let mut second = fixture.acquire("second"); let plan = fixture.plan(&mut first, &request, NO_CONTENT).unwrap(); assert_eq!( - failure(plan.launch_with(&mut second, openvmm, &runtime, hooks)), + refused_at( + plan.launch_with(&mut second, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure + ), ( ErrorKind::Other, format!( @@ -1031,7 +1168,8 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); assert_ne!(other, runtime); let plan = fixture.plan(&mut first, &request, NO_CONTENT).unwrap(); - let (kind, message) = failure(plan.launch_with(&mut first, openvmm, &other, hooks)); + let launched = plan.launch_with(&mut first, openvmm, &other, hooks); + let (kind, message) = refused_at(launched, Stage::Launch, Evidence::RuntimeMismatch); assert_eq!(kind, ErrorKind::Unsupported, "{message}"); assert!( message.starts_with(&format!( @@ -1054,7 +1192,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { assert!(fixture.plan(&mut first, &request, failing).is_err()); let sparse = first.locked.clone_scratch_path(); assert!(!exists(&sparse)); - let (kind, message) = failure(stale.launch_with(&mut first, openvmm, &runtime, hooks)); + let (kind, message) = refused_at( + stale.launch_with(&mut first, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure, + ); assert_eq!(kind, ErrorKind::Unavailable, "{message}"); assert!( message.starts_with(&format!( @@ -1071,7 +1213,11 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { let mut deep = fixture.acquire(&"d".repeat(100)); let plan = fixture.plan(&mut deep, &request, NO_CONTENT).unwrap(); assert_eq!( - failure(plan.launch_with(&mut deep, openvmm, &runtime, hooks)), + refused_at( + plan.launch_with(&mut deep, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::EndpointBind + ), ( ErrorKind::Unsupported, too_long(&deep.locked.restore_ready_path()) @@ -1095,6 +1241,118 @@ fn a_clone_launches_only_as_the_launch_that_its_durable_binding_reserves() { vmm.reference = Some(restoring.starting().process().reference().clone()); } +/// Returns OpenVMM's outcome report of a VM whose guest ended it with `status`. +fn guest_exit(status: u8) -> Vec { + serde_json::to_vec(&serde_json::json!({ + "schema_version": 1, + "outcome": {"operation": "managed", "category": "guest-exit", "status_code": status}, + "teardown": {"workers": true}, + })) + .unwrap() +} + +#[test] +fn a_clone_whose_launch_failed_before_it_began_launches_no_more_until_its_release() { + let fixture = Fixture::new(); + complete_snapshot(&fixture); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let request = tmpfs_request(&image, &binding); + let runtime = runtime_of(&fixture); + let openvmm = Path::new(OPENVMM); + // The launch fails once it wrote the clone's files and recorded its progress, before it + // began, which leaves the lock's reservation. + let interrupt = |point: Point| match point { + Point::Prepared => Err(Error::io("the test fails the launch")), + Point::Begun => Ok(()), + }; + let hooks = Hooks { + command: &never_spawned, + interrupt: &interrupt, + ..Hooks::LAUNCH + }; + let mut acquired = fixture.acquire("failed"); + // The report that an earlier VM of the sandbox left, which the acquisition kept, is not the + // clone's: an inspection takes none, and the launch removes it before it records anything. + let report = acquired.locked.outcome_path(); + fs::write(&report, guest_exit(195)).unwrap(); + assert!(crate::host::report::read(&report).is_some()); + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.report), + (Progress::Planned, None) + ); + let plan = fixture.plan(&mut acquired, &request, NO_CONTENT).unwrap(); + let failure = plan + .launch_with(&mut acquired, openvmm, &runtime, hooks) + .unwrap_err(); + assert_recorded( + &failure, + &acquired.lease, + &acquired.locked, + (Stage::Launch, Progress::Launching), + Evidence::HostFailure, + Teardown::NoVm, + ); + assert!(!exists(&report)); + assert_eq!(acquired.locked.reserved(), Some(acquired.lease.launch())); + // Another launch under the lease and the lock that still reserves it is refused before it + // writes anything, so the record of the failure stays as it was. + let path = acquired.locked.clone_failure_path(); + let recorded = fs::read(&path).unwrap(); + let plan = fixture.plan(&mut acquired, &request, NO_CONTENT).unwrap(); + assert_eq!( + refused_at( + plan.launch_with(&mut acquired, openvmm, &runtime, hooks), + Stage::Launch, + Evidence::HostFailure + ), + ( + ErrorKind::Other, + format!( + "the clone of the lease {} failed, as {} records: a lease's clone fails once, \ + and the release of the lease ends it", + acquired.lease.id(), + path.display() + ) + ) + ); + assert_eq!(fs::read(&path).unwrap(), recorded); + // The release ends the lease with the record. + assert_eq!( + fixture.store.release(&mut acquired.locked).unwrap(), + Some(acquired.lease.clone()) + ); + assert!(!path.exists()); +} + +#[test] +fn a_failure_that_a_reattach_records_before_any_launch_shows_no_report_of_an_earlier_vm() { + // A reattach before any launch of the clone that cannot read the record of a failure, as a + // transient error of the host may keep it from reading one, records a failure of its own, at + // the plan, with no VM: the report that an earlier VM of the sandbox left, which no launch + // removed, stays none of the clone's. + let fixture = Fixture::new(); + let runtime = runtime_of(&fixture); + let mut acquired = fixture.acquire("sandbox"); + let locked = &mut acquired.locked; + fs::write(locked.outcome_path(), guest_exit(195)).unwrap(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let deadline = Instant::now() + TIMEOUT; + let reattached = reattach_clone(locked, &runtime, &mut control, deadline, TIMEOUT).unwrap(); + let failure = failed(reattached); + assert_eq!( + (failure.record().progress, &failure.record().teardown), + (Progress::Planned, &Teardown::NoVm) + ); + let records = CloneRecords::read(locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(failure.record())); + assert_eq!( + (records.progress, records.report), + (Progress::Planned, None) + ); +} + /// Returns the message of the refusal of a clone's launch whose socket at `path` would not fit a /// Unix socket's address. #[cfg(target_os = "linux")] @@ -1122,7 +1380,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() command: &never_spawned, ..Hooks::LAUNCH }; - let refused = |fixture: &Fixture, sandbox: &str| { + let launch_refusal = |fixture: &Fixture, sandbox: &str, evidence: Evidence| { complete_snapshot(fixture); let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); let mut acquired = fixture.acquire(sandbox); @@ -1130,13 +1388,14 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() .plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT) .unwrap(); let runtime = runtime_of(fixture); - let refusal = failure(plan.launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks)); + let launched = plan.launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks); + let refusal = refused_at(launched, Stage::Launch, evidence); assert_unlaunched(fixture, &acquired); (refusal, fixture.store.clone_socket_path(&acquired.lease)) }; // A store whose path leaves no room for the socket of a clone's VMM in a payload. let fixture = Fixture::in_store(|directory| deep(directory).join("templates"), |_, _, _| {}); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.as_os_str().len() >= 108); assert_eq!((kind, message), (ErrorKind::Unsupported, too_long(&socket))); // A store whose path is short, but long once its links are resolved, at which OpenVMM listens. @@ -1148,7 +1407,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.as_os_str().len() < 108, "{}", socket.display()); let resolved = filesystem::canonicalize(socket.parent().unwrap()) .unwrap() @@ -1168,7 +1427,8 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let bind = Evidence::EndpointBind; + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", bind); assert_eq!( (kind, message), ( @@ -1192,7 +1452,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.to_str().is_none(), "{}", socket.display()); assert_eq!( (kind, message), @@ -1219,7 +1479,7 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() }, |_, _, _| {}, ); - let ((kind, message), socket) = refused(&fixture, "sandbox"); + let ((kind, message), socket) = launch_refusal(&fixture, "sandbox", Evidence::EndpointBind); assert!(socket.to_str().is_some(), "{}", socket.display()); let resolved = filesystem::canonicalize(socket.parent().unwrap()) .unwrap() @@ -1248,12 +1508,13 @@ fn a_clone_whose_control_socket_openvmm_or_the_host_cannot_use_starts_nothing() .unwrap(); let payload = fixture.store.payload_dir(fixture.generation.generation()); fs::set_permissions(&payload, fs::Permissions::from_mode(0o500)).unwrap(); - let refusal = failure(plan.launch_with( + let launched = plan.launch_with( &mut acquired, Path::new(OPENVMM), &runtime_of(&fixture), hooks, - )); + ); + let refusal = refused_at(launched, Stage::Launch, Evidence::MalformedGeneration); fs::set_permissions(&payload, fs::Permissions::from_mode(0o700)).unwrap(); assert_unlaunched(&fixture, &acquired); assert_eq!( @@ -1291,6 +1552,8 @@ fn launch_role() { let store = Store::open(dir).unwrap(); let generation = StoreGeneration::new(generation.parse().unwrap()); let indexed = store.index().unwrap().get(generation).unwrap().clone(); + let journal = sandbox.join(JOURNAL_NAME); + let kept = sandbox.join(format!("{JOURNAL_NAME}.kept")); let mut acquired = store .acquire(generation, sandbox, &files(), &after(&indexed, 1), None) .unwrap(); @@ -1315,6 +1578,7 @@ fn launch_role() { let hooks = Hooks { command: &command, interrupt: &interrupt, + ..Hooks::LAUNCH }; let restoring = plan .launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks) @@ -1338,7 +1602,16 @@ fn launch_role() { if step == "started" { crash(&step); } + if step == "superseded" { + fs::hard_link(&journal, &kept).unwrap(); + } let committed = started.commit().unwrap(); + if step == "superseded" { + // The commit's removal of the journal failed, as a commit tolerates: the journal, the + // same file, remains beside the runtime record that supersedes it. + fs::rename(&kept, &journal).unwrap(); + crash(&step); + } if step == "committed" { crash(&step); } @@ -1454,15 +1727,23 @@ impl Crashed { /// Leases a fresh fixture's generation to a new sandbox, whose clone a [`launch_role`] /// launches until it crashes at `step`. fn at(step: &str) -> Self { + Self::at_with(step, "") + } + + /// Leases a fresh fixture's generation to a new sandbox, whose clone a [`launch_role`] + /// launches until it crashes at `step`, with a VMM whose [plan](VmmDir::plan) holds `words`. + fn at_with(step: &str, words: &str) -> Self { let fixture = Fixture::new(); complete_snapshot(&fixture); let (image, _) = fixture.image("distro.erofs", IMAGE_LEN); let sandbox = fixture.provision("sandbox"); let mut vmm = VmmDir::new(); + let mut plan = words.to_owned(); if !matches!(step, "prepared" | "begun" | "launched") { // The restore sequence reads the event, whose flush waits for that on Windows. - vmm.plan("flush"); + plan.push_str(" flush"); } + vmm.plan(&plan); vmm.reference = run_launch_role(step, &fixture, &sandbox, &image, vmm.path()); let lease = Lease::of(&SandboxStore::open(&sandbox).unwrap()) .unwrap() @@ -1515,6 +1796,28 @@ impl Crashed { } } + /// Takes the clone of the sandbox of `locked` over, under `runtime`, through `control`, with + /// [`reattach_clone`], until the result is determinate, as a caller retries an indeterminate + /// reattach, and returns it. + fn take_over( + &self, + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut C, + ) -> ReattachedClone { + let deadline = Instant::now() + TIMEOUT; + loop { + let session = Instant::now() + TIMEOUT; + match reattach_clone(locked, runtime, control, session, TIMEOUT).unwrap() { + ReattachedClone::Indeterminate(reason) if Instant::now() < deadline => { + eprintln!("reattach is indeterminate: {reason}"); + thread::sleep(Duration::from_millis(100)); + } + reattached => return reattached, + } + } + } + /// Checks that the store refuses to release the lease of the sandbox of `locked`, whose /// launch has not ended, and that the sandbox keeps it. fn refuses_release(&self, locked: &mut LockedStore) { @@ -1531,6 +1834,7 @@ impl Crashed { Some(self.lease.clone()) ); assert_released(&self.sandbox); + assert!(CloneRecords::read(locked).unwrap().is_none()); assert!(!exists(&self.socket())); let snapshot = self.fixture.store.snapshot_dir(&self.fixture.generation); for name in SNAPSHOT_ARTIFACTS { @@ -1579,6 +1883,111 @@ fn a_launch_whose_caller_crashes_once_it_began_ends_as_reattach_recovers_it() { crashed.releases(&mut locked); } +#[test] +fn a_launch_that_began_shows_the_report_of_a_vmm_that_its_journal_does_not_record() { + // The caller crashed once the launch began, before the journal recorded the VMM that it + // spawned, which left the report: the clone's, since the launch removed any earlier VM's + // report before it recorded anything, though the clone's progress shows no spawn. An + // inspection takes the report once a reattach of the clone ended the launch, recording its + // failure, and once a recovery of the sandbox's launch did, recording none. + let shown = |records: &CloneRecords| { + let report = records + .report + .as_ref() + .expect("the report of the clone's VM"); + (report.category.clone(), report.status_code) + }; + let ended = (Some("guest-exit".to_owned()), Some(3)); + for of_the_clone in [true, false] { + let crashed = Crashed::at("begun"); + let mut locked = crashed.lock(); + fs::write(locked.outcome_path(), guest_exit(3)).unwrap(); + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.journal.is_some()), + (Progress::Launching, true) + ); + assert_eq!(shown(&records), ended); + if of_the_clone { + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.record().teardown, Teardown::Exited); + } else { + match crashed.reattach(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + } + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + ( + records.progress, + records.journal.is_none(), + records.failure.is_some() + ), + (Progress::Launching, true, of_the_clone) + ); + assert_eq!(shown(&records), ended, "{of_the_clone}"); + crashed.releases(&mut locked); + } +} + +#[test] +fn a_reattach_takes_the_report_of_a_vmm_whose_launch_a_recovery_of_the_sandbox_ended() { + // The caller crashed once the launch began, before the journal recorded the VMM that it + // spawned, which left the report, and a recovery of the sandbox's launch then removed the + // journal: nothing records the spawn, but the report is the clone's VM's, as an inspection + // takes it, so a reattach of the clone takes it too. + let recovered = |status: u8| { + let crashed = Crashed::at("begun"); + let mut locked = crashed.lock(); + fs::write(locked.outcome_path(), guest_exit(status)).unwrap(); + match crashed.reattach(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + assert!(locked.journal().unwrap().is_none()); + (crashed, locked) + }; + let mut control = ReattachControl::answering(Some); + // The status of the guest's repair is evidence of the clone's failure at any point. + let (crashed, mut locked) = recovered(195); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{failure}"); + assert_recorded( + &failure, + &crashed.lease, + &locked, + (Stage::Launch, Progress::Launching), + Evidence::RepairStatus, + Teardown::Exited, + ); + assert_eq!(failure.record().status, Some(195)); + crashed.releases(&mut locked); + // Another status is no evidence of a failure: the clone ended, with the report of its VM. + let (crashed, mut locked) = recovered(3); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Launching, + report: Some(report), + } => { + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(3))); + } + other => panic!("the reattach found {}", describe(&other)), + } + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert!(records.failure.is_none()); + crashed.releases(&mut locked); + assert!(control.capabilities.is_empty()); +} + #[test] fn a_launch_whose_caller_crashes_once_its_vmm_started_ends_the_vmm_as_reattach_recovers_it() { let crashed = Crashed::at("launched"); @@ -1671,3 +2080,1339 @@ fn a_restore_whose_caller_crashes_once_it_committed_keeps_its_lease_while_its_vm assert!(!exists(&crashed.socket())); crashed.releases(&mut locked); } + +/// The Ready of a clone that a [`ReattachControl`] answers, from the activation of the record that +/// the clone's Activate carries. +type Answered = fn(ReadyActivation) -> Option; + +/// The adapter's framing of the sessions of a reattach: it attaches a session to each connection +/// to the clone's control endpoint, keeps the capability of each, and the record of each Activate, +/// which it answers as its `answer` says. +struct ReattachControl { + /// The capabilities of the sessions, in order. + capabilities: Vec<[u8; CAPABILITY_LEN]>, + /// The records of the Activates, in order. + records: Vec>, + /// How the clone answers an Activate. + answer: Answered, +} + +impl ReattachControl { + /// Returns a framing whose clone answers each Activate as `answer` says. + fn answering(answer: Answered) -> Self { + Self { + capabilities: Vec::new(), + records: Vec::new(), + answer, + } + } +} + +impl CloneControl for ReattachControl { + type Session = Box; + type Error = io::Error; + + fn attach( + &mut self, + transport: Box, + capability: &[u8; CAPABILITY_LEN], + _deadline: Instant, + ) -> io::Result> { + self.capabilities.push(*capability); + Ok(transport) + } + + fn activate( + &mut self, + _session: &mut Box, + record: &[u8], + _deadline: Instant, + ) -> io::Result> { + self.records.push(record.to_vec()); + let record = ActivationRecord::decode(record).unwrap(); + Ok((self.answer)(ReadyActivation::of(&record))) + } + + fn start(&mut self, _session: &mut Box, _deadline: Instant) -> io::Result<()> { + panic!("a reattach starts no sandbox") + } + + fn lost(&self, _error: &io::Error) -> bool { + false + } +} + +#[test] +fn a_clone_whose_caller_crashed_once_it_committed_is_taken_over_in_a_new_session() { + let crashed = Crashed::at("committed"); + let reference = crashed.vmm.reference.clone().unwrap(); + let started = crashed.vmm.started(); + let (capability, _) = started.split_once(' ').unwrap(); + // A tombstone quarantines the clone's generation, which a retirement deletes once its leases + // have drained: neither takes the clone's lease from it. + let store = &crashed.fixture.store; + let generation = crashed.fixture.generation.generation(); + store.tombstone(generation, None).unwrap(); + assert_eq!( + store.retire(generation, None).unwrap(), + Retirement::Draining(vec![crashed.lease.clone()]) + ); + let mut locked = crashed.lock(); + let recorded = fs::read(locked.clone_activation_path()).unwrap(); + let mut control = ReattachControl::answering(Some); + let (vmm, session) = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, session } => (vmm, session), + other => panic!("the reattach found {}", describe(&other)), + }; + // The VM is the one that the launch committed, whose clone answered the Activate of the + // sandbox's record, byte for byte, in a session that the launch's capability authenticated, + // and runs on, with its lease. + assert_eq!(vmm.process().reference(), &reference); + assert_eq!(control.records, [recorded]); + let attached: Vec = control.capabilities.iter().map(|key| hex(key)).collect(); + assert_eq!(attached, [capability]); + drop(session); + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.failure), + (Progress::Committed, None) + ); + crashed.refuses_release(&mut locked); + assert_eq!( + vmm.terminate(&mut locked, TIMEOUT).unwrap(), + StopOutcome::Forced + ); + crashed.releases(&mut locked); + // Its lease released, the retirement deletes the generation. + assert_eq!(store.retire(generation, None).unwrap(), Retirement::Retired); +} + +#[test] +fn a_journal_that_a_clone_s_commit_left_is_no_failure_of_the_clone() { + // The commit of the clone's launch could not remove the launch's journal, which remains + // beside the runtime record of the same launch, as its caller crashed. + let crashed = Crashed::at("superseded"); + let mut locked = crashed.lock(); + let journal = locked + .journal() + .unwrap() + .expect("the commit left the journal"); + let runtime = locked + .runtime() + .unwrap() + .expect("the commit wrote its record"); + assert_eq!(journal.launch(), runtime.launch()); + // Each reattach takes the clone over, and records no failure: the first removes the journal. + let mut control = ReattachControl::answering(Some); + let mut live = None; + for attempt in 0..2 { + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach {attempt} found {}", describe(&other)), + }; + assert_eq!( + Some(vmm.process().reference()), + crashed.vmm.reference.as_ref() + ); + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.failure, records.journal.is_some()), + (Progress::Committed, None, false), + "{attempt}" + ); + live = Some(vmm); + } + assert_eq!(control.records.len(), 2); + crashed.refuses_release(&mut locked); + let vmm = live.unwrap(); + assert_eq!( + vmm.terminate(&mut locked, TIMEOUT).unwrap(), + StopOutcome::Forced + ); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_ends_a_clone_that_it_may_not_keep_through_its_verified_identity() { + let reattach = (Stage::Reattach, Progress::Committed); + // A clone that runs under another runtime identity, which might not speak this one's + // protocol: the reattach sends its guest nothing. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &other, &mut control)); + assert_eq!(failure.code(), ErrorCode::RuntimeMismatch, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + let mismatch = Evidence::RuntimeMismatch; + assert_recorded(&failure, lease, &locked, reattach, mismatch, terminated); + assert_eq!(failure.outcome(), Outcome::Host); + assert!(control.capabilities.is_empty()); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + // A clone whose Ready reports another activation than its record's. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(|_| Some(ReadyActivation { transaction: None })); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Protocol, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + assert_recorded( + &failure, + lease, + &locked, + reattach, + Evidence::Protocol, + terminated, + ); + assert_eq!(control.records.len(), 1); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_fails_a_clone_whose_record_of_a_failure_cannot_be_read() { + let reattach = (Stage::Reattach, Progress::Committed); + let host = Evidence::HostFailure; + // A record that is malformed, and one of a format that this build does not read: the clone + // failed, so the reattach sends its guest nothing, and ends its VM through its verified + // identity, though its Ready would match its activation record. + let unreadable: [(&[u8], ErrorCode); 2] = [ + (b"not a record", ErrorCode::Io), + (br#"{"format": 99}"#, ErrorCode::FormatUnsupported), + ]; + for (bytes, code) in unreadable { + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), bytes).unwrap(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), code, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + } + // A clone whose VM exited with a status that is no evidence of a failure failed all the same. + let crashed = Crashed::at_with("committed", "report:guest-exit:3"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded(&failure, lease, &locked, reattach, host, exited); + assert_eq!(failure.record().status, Some(3)); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // A clone whose VM cannot be examined failed all the same, with its teardown unresolved, which + // a later reattach resolves. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let reason = "the test cannot examine the VM"; + let reattached = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Ok(Reattached::Indeterminate(reason.to_owned())), + ) + .unwrap(); + let failure = failed(reattached); + let (lease, unresolved) = (&crashed.lease, Teardown::Unresolved(reason.to_owned())); + assert_recorded(&failure, lease, &locked, reattach, host, unresolved); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let terminated = Teardown::Terminated; + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_fails_a_clone_whose_sandbox_records_the_failure_of_another_clone() { + // A record of the failure of another lease's clone, of another launch, which leased another + // generation of the same store, whose evidence is the template's: it tells nothing of this + // clone, so the reattach takes it as a record that it cannot read, and the failure that it + // records in its place, and returns, is this clone's, which quarantines nothing. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let lease = &crashed.lease; + let generation = lease.generation().get().checked_add(1).unwrap(); + let other = Lease::new( + crate::host::template::LeaseId::from_bytes([0xa5; 16]), + lease.store(), + StoreGeneration::new(generation), + lease.template().key, + lease.sandbox().to_owned(), + crate::host::store::LaunchId::generate().unwrap(), + ); + let defect = Error::io("the test's generation is malformed"); + let at = ((Stage::Restore, Progress::Spawned), None); + let template = (Evidence::MalformedGeneration, Teardown::Exited); + record_earlier(&other, &locked, at, template, &defect); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + assert!( + failure.message().contains("belongs to another clone"), + "{failure}" + ); + let reattach = (Stage::Reattach, Progress::Committed); + let (host, terminated) = (Evidence::HostFailure, Teardown::Terminated); + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + drop(locked); + let store = &crashed.fixture.store; + assert!(!failure.quarantine(store, None).unwrap()); + assert!(store.tombstones().unwrap().is_empty()); +} + +#[test] +fn a_record_of_a_failure_that_cannot_be_read_is_a_failure_whatever_the_reattach_finds() { + let reattach = (Stage::Reattach, Progress::Committed); + // A VM that exited once its launch committed, with a status of the guest's that is evidence, + // or without a report of the guest's own end: the clone failed before, so the failure is the + // reattach's, with the error of the record's read, over whose evidence the status's prevails. + let cases = [ + ("report:guest-exit:195", Evidence::RepairStatus, Some(195)), + ("", Evidence::HostFailure, None), + ]; + for (words, evidence, status) in cases { + let crashed = Crashed::at_with("committed", words); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{words}: {failure}"); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded(&failure, lease, &locked, reattach, evidence, exited); + assert_eq!(failure.record().status, status, "{words}"); + assert!(locked.runtime().unwrap().is_none(), "{words}"); + assert!(control.capabilities.is_empty(), "{words}"); + crashed.releases(&mut locked); + } + // A VM that runs under another runtime identity is ended, and the failure is the same. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &other, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + let host = Evidence::HostFailure; + assert_recorded(&failure, lease, &locked, reattach, host, terminated); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // A launch that a crash of its caller interrupted: the reattach replaces the record with its + // own failure, pending, before its recovery ends the launch, which a crash of the restarted + // caller leaves, for the next reattach to finish. + let spawned = (Stage::Reattach, Progress::Spawned); + let assert_pending = |locked: &LockedStore| { + let pending = read_failure(&locked.clone_failure_path()) + .unwrap() + .expect("the reattach replaced the record that it could not read"); + assert_eq!( + ( + pending.stage, + pending.progress, + pending.evidence, + pending.code + ), + (spawned.0, spawned.1, host, ErrorCode::Io) + ); + assert_eq!(pending.teardown, Teardown::Pending); + }; + let crashed = Crashed::at("launched"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Err(Error::other("the test's caller crashes")), + ); + assert!(crash.is_err()); + assert_pending(&locked); + assert!(locked.journal().unwrap().is_some()); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, terminated) = (&crashed.lease, Teardown::Terminated); + assert_recorded(&failure, lease, &locked, spawned, host, terminated); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // Before the restore-ready event, only the launch's journal, which the recovery removes, holds + // the spawn: once it went, the pending record still tells how far the clone got, so the next + // reattach takes the status with which the guest ended the VM. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + supervisor::reattach_reporting(locked, runtime, exited)?; + Err(Error::other("the test's caller crashes")) + }, + ); + assert!(crash.is_err()); + assert_pending(&locked); + assert!(locked.journal().unwrap().is_none()); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + let repair = Evidence::RepairStatus; + assert_recorded(&failure, lease, &locked, spawned, repair, exited); + assert_eq!(failure.record().status, Some(195)); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); + // A clone whose launch did not begin: the store holds no record of the launch that the + // reattach would remove, so the record stays as it is, and a reattach after a crash of the + // caller finds the clone as this one does, with no VM. + let crashed = Crashed::at("prepared"); + let mut locked = crashed.lock(); + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let mut control = ReattachControl::answering(Some); + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Err(Error::other("the test's caller crashes")), + ); + assert!(crash.is_err()); + let kept = fs::read(locked.clone_failure_path()).unwrap(); + assert_eq!(kept, b"not a record"); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let launching = (Stage::Reattach, Progress::Launching); + let (lease, no_vm) = (&crashed.lease, Teardown::NoVm); + assert_recorded(&failure, lease, &locked, launching, host, no_vm); + assert_eq!(failure.record().status, None); + crashed.releases(&mut locked); + // A clone whose launch a recovery of the sandbox ended, once its caller crashed before the + // journal recorded the VMM that it spawned, which left the report: nothing records the spawn, + // but the report is the clone's VM's, since the launch removed any earlier VM's report before + // it recorded anything, and the status of the guest's repair prevails. + let crashed = Crashed::at("begun"); + let mut locked = crashed.lock(); + fs::write(locked.outcome_path(), guest_exit(195)).unwrap(); + match crashed.reattach(&mut locked) { + Reattached::Exited { + record: None, + recovery: Recovery::Exited, + } => {} + other => panic!("unexpected reattach: {other:?}"), + } + write_private(&locked.clone_failure_path(), b"not a record").unwrap(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.code(), ErrorCode::Io, "{failure}"); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + let repair = Evidence::RepairStatus; + assert_recorded(&failure, lease, &locked, launching, repair, exited); + assert_eq!(failure.record().status, Some(195)); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_ends_the_launch_of_a_caller_that_crashed_and_records_its_failure() { + let cases = [ + ( + "begun", + (Stage::Launch, Progress::Launching), + Evidence::LostStatus, + Teardown::Exited, + ), + ( + "launched", + (Stage::Restore, Progress::Spawned), + Evidence::ForcedKill, + Teardown::Terminated, + ), + ( + "activated", + (Stage::Start, Progress::Ready), + Evidence::ForcedKill, + Teardown::Terminated, + ), + ( + "started", + (Stage::Commit, Progress::Started), + Evidence::ForcedKill, + Teardown::Terminated, + ), + ]; + for (step, at, evidence, teardown) in cases { + let crashed = Crashed::at(step); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{step}: {failure}"); + assert_recorded(&failure, &crashed.lease, &locked, at, evidence, teardown); + assert!(control.capabilities.is_empty(), "{step}"); + // Once the launch has ended, a reattach finds its failure as it was recorded. + let again = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(again.record(), failure.record(), "{step}"); + assert_eq!(again.message(), failure.record().message, "{step}"); + crashed.releases(&mut locked); + } + // A caller that crashed before its launch began left no VM, and no failure. + let crashed = Crashed::at("prepared"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Launching, + report: None, + } => {} + other => panic!("the reattach found {}", describe(&other)), + } + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_records_the_failure_of_an_interrupted_launch_before_it_ends_the_launch() { + let crashed = Crashed::at("launched"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + // The restarted caller crashes in turn as its reattach would end the interrupted launch. + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Err(Error::other("the test's caller crashes")), + ); + assert!(crash.is_err()); + // The sandbox records the failure, whose teardown is pending, beside the launch's journal, + // which keeps the lease. + let records = CloneRecords::read(&locked).unwrap().unwrap(); + let pending = records + .failure + .expect("the reattach recorded the failure before the recovery"); + assert_eq!( + (pending.stage, pending.progress, pending.evidence), + (Stage::Restore, Progress::Spawned, Evidence::LostStatus) + ); + assert_eq!(pending.teardown, Teardown::Pending); + assert!(records.journal.is_some()); + crashed.refuses_release(&mut locked); + // The next reattach ends the launch, and resolves the teardown of the recorded failure. + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{failure}"); + assert_recorded( + &failure, + &crashed.lease, + &locked, + (Stage::Restore, Progress::Spawned), + Evidence::LostStatus, + Teardown::Terminated, + ); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + +/// A launch that its caller abandons before the clone's activation leaves a progress log that +/// records the spawn of the clone's VMM, which the launch's journal, which the teardown removes, +/// records no more, so that a reattach reads the outcome report of the clone's VM, and records the +/// failure that the guest's status gives. +#[test] +fn a_launch_abandoned_before_its_activation_leaves_the_spawn_for_a_reattach() { + let fixture = Fixture::new(); + complete_snapshot(&fixture); + let (image, binding) = fixture.image("distro.erofs", IMAGE_LEN); + let mut acquired = fixture.acquire("abandoned"); + let progress = acquired.locked.clone_progress_path(); + let launch = acquired.lease.launch().clone(); + let plan = fixture + .plan(&mut acquired, &tmpfs_request(&image, &binding), NO_CONTENT) + .unwrap(); + let runtime = plan.key().runtime().clone(); + let mut vmm = VmmDir::new(); + // The guest's repair ends the clone's VM with its status, which the VMM reports as it exits. + vmm.plan("report:guest-exit:195"); + let dir = vmm.path().to_owned(); + let made = RefCell::new(None); + let command = fake_vmm(&dir, &made); + let hooks = Hooks { + command: &command, + ..Hooks::LAUNCH + }; + let restoring = plan + .launch_with(&mut acquired, Path::new(OPENVMM), &runtime, hooks) + .unwrap(); + vmm.reference = Some(restoring.starting().process().reference().clone()); + // The log records the spawn once the launch returns. + let logged = read_progress(&progress, &launch).unwrap().unwrap(); + assert_eq!( + (logged.progress, logged.boot), + (Progress::Spawned, Some(BootId::current().unwrap())) + ); + let capability = hex(restoring.capability()); + assert_eq!(vmm.started(), format!("{capability} true")); + vmm.end_and_wait(); + assert_eq!(restoring.abandon().unwrap(), Recovery::Exited); + assert!(acquired.locked.journal().unwrap().is_none()); + let mut control = ReattachControl::answering(Some); + let reattached = reattach_clone( + &mut acquired.locked, + &runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ); + let failure = failed(reattached.unwrap()); + assert_recorded( + &failure, + &acquired.lease, + &acquired.locked, + (Stage::Restore, Progress::Spawned), + Evidence::RepairStatus, + Teardown::Exited, + ); + assert_eq!(failure.record().status, Some(195)); + assert!(control.capabilities.is_empty()); + assert_eq!( + fixture.store.release(&mut acquired.locked).unwrap(), + Some(acquired.lease.clone()) + ); + assert_released(acquired.lease.sandbox()); +} + +#[test] +fn a_clone_whose_vm_exited_once_it_committed_is_classified_by_its_report() { + let run = (Stage::Run, Progress::Committed); + // A repair's status is evidence of the repair stage, after the clone's Ready too, and a VM + // that ended without a report of the guest's own end lost the clone's status. + let cases = [ + ("report:guest-exit:195", Evidence::RepairStatus, Some(195)), + ("", Evidence::LostStatus, None), + ]; + for (words, evidence, status) in cases { + let crashed = Crashed::at_with("committed", words); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{words}: {failure}"); + let exited = Teardown::Exited; + assert_recorded(&failure, &crashed.lease, &locked, run, evidence, exited); + assert_eq!(failure.record().status, status, "{words}"); + assert!(locked.runtime().unwrap().is_none(), "{words}"); + crashed.releases(&mut locked); + } + // A VM that its guest ended with a status that is no evidence of a failure ended. + let crashed = Crashed::at_with("committed", "report:guest-exit:3"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Committed, + report: Some(report), + } => { + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(3))); + } + other => panic!("the reattach found {}", describe(&other)), + } + // An inspection of the clone takes the report of its VM as the reattach did, and no failure. + let records = CloneRecords::read(&locked).unwrap().unwrap(); + assert!(records.failure.is_none()); + let report = records.report.expect("the report of the clone's VM"); + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(3))); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_records_the_failure_of_a_vm_that_exited_before_it_forgets_the_vm() { + let run = (Stage::Run, Progress::Committed); + let crashed = Crashed::at_with("committed", "report:guest-exit:195"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + // The restarted caller crashes once its reattach cleared the runtime record of the VM that + // exited, before it recorded the failure once more. + let crash = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + supervisor::reattach_reporting(locked, runtime, exited)?; + Err(Error::other("the test's caller crashes")) + }, + ); + assert!(crash.is_err()); + // The sandbox recorded the failure before the runtime record went, with its teardown pending. + let pending = read_failure(&locked.clone_failure_path()) + .unwrap() + .expect("the reattach recorded the failure before it cleared the runtime record"); + assert_eq!( + (pending.stage, pending.progress, pending.evidence), + (run.0, run.1, Evidence::RepairStatus) + ); + assert_eq!(pending.teardown, Teardown::Pending); + assert_eq!(pending.status, Some(195)); + assert!(locked.runtime().unwrap().is_none()); + // The next reattach finishes it, with the status that it holds, even once no read finds the + // report, as when another process holds it. + fs::write(locked.outcome_path(), b"not a report").unwrap(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded( + &failure, + lease, + &locked, + run, + Evidence::RepairStatus, + exited, + ); + assert_eq!(failure.record().status, Some(195)); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + +#[test] +fn a_reattach_takes_the_outcome_report_of_a_vm_that_exited_as_it_first_read_it() { + let run = (Stage::Run, Progress::Committed); + let crashed = Crashed::at_with("committed", "report:guest-exit:195"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let outcome = locked.outcome_path(); + let mut control = ReattachControl::answering(Some); + // Once the reattach read the report of the VM that exited, before it cleared the VM's runtime + // record, no read finds the report, as when another process holds it: the reattach takes the + // report as it first read it. + let reattached = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + supervisor::reattach_reporting(locked, runtime, |record| { + exited(record)?; + fs::write(&outcome, b"not a report").unwrap(); + Ok(()) + }) + }, + ) + .unwrap(); + let failure = failed(reattached); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded( + &failure, + lease, + &locked, + run, + Evidence::RepairStatus, + exited, + ); + assert_eq!(failure.record().status, Some(195)); + crashed.releases(&mut locked); +} + +/// Records the failure of the clone of `lease`, in the sandbox of `locked`, at `at`, with +/// `evidence` and `error`, whose teardown is `teardown`, with the transaction `transaction`, as +/// a failure of the clone's restore sequence records it before its teardown, which a crash of its +/// caller interrupted, if the teardown is pending, or once a teardown that could not end the +/// clone's VMM, if it is unresolved. +fn record_earlier( + lease: &Lease, + locked: &LockedStore, + (at, transaction): ((Stage, Progress), Option), + (evidence, teardown): (Evidence, Teardown), + error: &Error, +) { + let record = FailureRecord::new(lease, transaction, at, evidence, None, teardown, error); + write_failure(&locked.clone_failure_path(), &record).unwrap(); +} + +/// Returns the transaction of the restore that the activation record of the clone in the sandbox +/// of `locked` names. +fn transaction_of(locked: &LockedStore) -> Option { + let activation = fs::read(locked.clone_activation_path()).unwrap(); + let activation = ActivationRecord::decode(&activation).unwrap(); + activation.restore().map(|restore| restore.transaction) +} + +#[test] +fn a_reattach_finishes_an_earlier_failure_with_what_the_clone_s_vmm_left_as_it_ended() { + let restore = (Stage::Restore, Progress::Spawned); + let mut control = ReattachControl::answering(Some); + let unresolved = Teardown::Unresolved("the test could not end the clone's VMM".to_owned()); + for earlier in [Teardown::Pending, unresolved] { + // A failure of the clone's restore sequence, whose teardown a crash of its caller + // interrupted, or could not end the clone's VMM, which the guest then ended with the + // status of a failed repair, which prevails. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + let mut locked = crashed.lock(); + let reset = Error::protocol("the test's clone reset its session"); + let at = (restore, transaction_of(&locked)); + let teardown = earlier.clone(); + record_earlier( + &crashed.lease, + &locked, + at, + (Evidence::Reset, teardown), + &reset, + ); + crashed.vmm.end_and_wait(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!( + failure.kind(), + ErrorKind::Protocol, + "{earlier:?}: {failure}" + ); + let (lease, exited) = (&crashed.lease, Teardown::Exited); + assert_recorded( + &failure, + lease, + &locked, + restore, + Evidence::RepairStatus, + exited, + ); + assert_eq!(failure.record().status, Some(195), "{earlier:?}"); + assert_eq!(failure.message(), "the test's clone reset its session"); + crashed.releases(&mut locked); + // One of the template, such as a defect of the generation that the caller diagnosed, + // which the status does not override: the record holds the status beside it. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + let mut locked = crashed.lock(); + let defect = Error::other("the test diagnosed a defect of the generation's repair"); + let at = (restore, transaction_of(&locked)); + let teardown = earlier.clone(); + record_earlier( + &crashed.lease, + &locked, + at, + (Evidence::TemplateDefect, teardown), + &defect, + ); + crashed.vmm.end_and_wait(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (lease, template) = (&crashed.lease, Evidence::TemplateDefect); + assert_recorded( + &failure, + lease, + &locked, + restore, + template, + Teardown::Exited, + ); + assert_eq!( + (failure.outcome(), failure.record().status), + (Outcome::Template, Some(195)), + "{earlier:?}" + ); + crashed.releases(&mut locked); + // One whose VMM exited before its restore-ready event, with OpenVMM's error, which tells + // that this host cannot present the template's CPU, once its record was written. + let crashed = Crashed::at_with("launched", "close fatal:E_CPU_GENERATION"); + let mut locked = crashed.lock(); + let lost = Error::exited("the test's VMM exited before its restore-ready event"); + let at = (restore, transaction_of(&locked)); + let teardown = earlier.clone(); + record_earlier( + &crashed.lease, + &locked, + at, + (Evidence::LostStatus, teardown), + &lost, + ); + crashed.vmm.wait_exit(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{earlier:?}: {failure}"); + let (lease, incompatible) = (&crashed.lease, Evidence::IncompatibleHost); + assert_recorded( + &failure, + lease, + &locked, + restore, + incompatible, + Teardown::Exited, + ); + let message = failure.message(); + assert!( + message.starts_with("the test's VMM exited before its restore-ready event; ") + && message.ends_with("failed to launch vm worker") + && message.matches("[E_CPU_GENERATION]").count() == 1, + "{earlier:?}: {failure}" + ); + // A later reattach finds the failure as it was recorded. + let again = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(again.record(), failure.record(), "{earlier:?}"); + crashed.releases(&mut locked); + } + assert!(control.capabilities.is_empty()); + // The record keeps the transaction of the restore that failed, whatever the sandbox's + // activation record names since. + let crashed = Crashed::at_with("launched", "report:guest-exit:195"); + let mut locked = crashed.lock(); + let reset = Error::protocol("the test's clone reset its session"); + let transaction = Some(TransactionId::from_bytes([0x5a; 16])); + assert_ne!(transaction, transaction_of(&locked)); + let pending = (Evidence::Reset, Teardown::Pending); + record_earlier( + &crashed.lease, + &locked, + (restore, transaction), + pending, + &reset, + ); + crashed.vmm.end_and_wait(); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let record = failure.record(); + assert_eq!( + (record.transaction, record.evidence, &record.teardown), + (transaction, Evidence::RepairStatus, &Teardown::Exited) + ); + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert_eq!(recorded.as_ref(), Some(record)); + crashed.releases(&mut locked); +} + +#[test] +fn a_clone_whose_progress_log_may_have_lost_its_steps_may_have_run_its_workload() { + // A launch whose caller crashed once the clone was ready, or once it sent the start of the + // clone's sandbox, whose VM the guest ended with a status that the clone's root gives before + // that start, and a workload may give after it. + let (current, rebooted) = (BootId::current().unwrap(), BootId::from_bytes([0xee; 16])); + assert_ne!(current, rebooted); + let ready = (Stage::Start, Progress::Ready); + let spawned = (Stage::Restore, Progress::Spawned); + let cases = [ + // Its intact log tells that the clone was only ready: its root's status is evidence. + ("intact", "activated", ready, Evidence::UnusableRoot), + // A log that was lost, or damaged, tells nothing, and one that the launch wrote during + // another boot of the host may have lost the steps of the start with the host's crash: + // the status is no evidence of the root's failure, and the clone lost its status. + ("lost", "started", spawned, Evidence::LostStatus), + ("damaged", "started", spawned, Evidence::LostStatus), + ("rebooted", "started", ready, Evidence::LostStatus), + ]; + for (damage, step, at, evidence) in cases { + let crashed = Crashed::at_with(step, "report:guest-exit:54"); + let mut locked = crashed.lock(); + let log = locked.clone_progress_path(); + let launch = crashed.lease.launch(); + match damage { + "lost" => fs::remove_file(log).unwrap(), + "damaged" => fs::write(log, b"not a progress log\n").unwrap(), + "rebooted" => fs::write( + log, + format!( + "nvx-clone-progress 1 {launch} {rebooted}\nspawned\nevent\nacknowledged\n\ + activating\nready\n" + ), + ) + .unwrap(), + _ => {} + } + crashed.vmm.end_and_wait(); + let mut control = ReattachControl::answering(Some); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let record = failure.record(); + assert_eq!( + ( + (record.stage, record.progress), + record.evidence, + record.status, + &record.teardown + ), + (at, evidence, Some(54), &Teardown::Exited), + "{damage}: {failure}" + ); + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert_eq!(recorded.as_ref(), Some(record), "{damage}"); + crashed.releases(&mut locked); + } +} + +#[test] +fn a_clone_that_its_holder_stopped_ended_whatever_a_reboot_left_of_its_progress_log() { + let other = BootId::from_bytes([0xee; 16]); + assert_ne!(other, BootId::current().unwrap()); + for rebooted in [false, true] { + let crashed = Crashed::at_with("committed", "report:guest-exit:54"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach found {}", describe(&other)), + }; + // Its holder stops the clone, whose workload ends with a status that the clone's root + // gives before the start of its sandbox, which clears the store's records of its VM. + let stopped = vmm + .stop(&mut locked, Instant::now() + TIMEOUT, None, |_, _| { + crashed.vmm.end(); + Ok::<(), io::Error>(()) + }) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful, "{rebooted}"); + assert!(locked.runtime().unwrap().is_none(), "{rebooted}"); + if rebooted { + // A crash of the host, before the release, lost the last lines of the progress log, + // which the launch wrote during the boot that the crash ended. + let launch = crashed.lease.launch(); + fs::write( + locked.clone_progress_path(), + format!( + "nvx-clone-progress 1 {launch} {other}\nspawned\nevent\nacknowledged\n\ + activating\nready\n" + ), + ) + .unwrap(); + } + // The clone ended, and its workload's status is no failure of its root. + let progress = if rebooted { + Progress::Ready + } else { + Progress::Committed + }; + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: ended, + report: Some(report), + } if ended == progress => { + let outcome = (report.category.as_deref(), report.status_code); + assert_eq!(outcome, (Some("guest-exit"), Some(54)), "{rebooted}"); + } + other => panic!("{rebooted}: the reattach found {}", describe(&other)), + } + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert!(recorded.is_none(), "{rebooted}: {recorded:?}"); + crashed.releases(&mut locked); + } +} + +#[test] +fn a_clone_whose_vmm_exited_before_its_holder_s_stop_began_lost_its_status() { + let never = |_: &Vmm, _: Instant| -> io::Result<()> { + panic!("a stop ran the graceful step of a VMM that had exited") + }; + // Its holder found the clone's VMM running, which then exits, without a report of the + // guest's own end, before the holder's stop begins. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach found {}", describe(&other)), + }; + crashed.vmm.end_and_wait(); + let stopped = vmm + .stop_unless_exited(&mut locked, Instant::now() + TIMEOUT, None, never) + .unwrap(); + assert!(stopped.is_none(), "{stopped:?}"); + // The stop kept the runtime record, and with it the lease, so the reattach finds that the VM + // exited unheld: the clone lost its status, which quarantines nothing. + assert_eq!(locked.runtime().unwrap().as_ref(), Some(vmm.record())); + crashed.refuses_release(&mut locked); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + assert_eq!(failure.kind(), ErrorKind::Exited, "{failure}"); + let (run, lost) = ((Stage::Run, Progress::Committed), Evidence::LostStatus); + assert_recorded( + &failure, + &crashed.lease, + &locked, + run, + lost, + Teardown::Exited, + ); + assert_eq!(failure.record().status, None); + let deadline = Some(Instant::now() + TIMEOUT); + assert!( + !failure + .quarantine(&crashed.fixture.store, deadline) + .unwrap() + ); + assert!(locked.runtime().unwrap().is_none()); + crashed.releases(&mut locked); + + // A stop that ends such a VM as one that it stopped clears the record, and the clone ended. + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let vmm = match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Live { vmm, .. } => vmm, + other => panic!("the reattach found {}", describe(&other)), + }; + crashed.vmm.end_and_wait(); + let stopped = vmm + .stop(&mut locked, Instant::now() + TIMEOUT, None, never) + .unwrap(); + assert_eq!(stopped.outcome, StopOutcome::Graceful); + assert!(locked.runtime().unwrap().is_none()); + match crashed.take_over(&mut locked, &crashed.runtime, &mut control) { + ReattachedClone::Ended { + progress: Progress::Committed, + report: None, + } => {} + other => panic!("the reattach found {}", describe(&other)), + } + let recorded = read_failure(&locked.clone_failure_path()).unwrap(); + assert!(recorded.is_none(), "{recorded:?}"); + crashed.releases(&mut locked); +} + +#[test] +fn an_interrupted_launch_that_a_reattach_cannot_end_yet_keeps_its_lease_until_one_can() { + let crashed = Crashed::at("activated"); + let mut locked = crashed.lock(); + let mut control = ReattachControl::answering(Some); + // The reattach cannot tell whether the launch's VMM still runs. + let reason = "the test cannot end the launch yet"; + let reattached = reattach_clone_with( + &mut locked, + &crashed.runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + |_, _, _| Ok(Reattached::Indeterminate(reason.to_owned())), + ) + .unwrap(); + let failure = failed(reattached); + let ready = (Stage::Start, Progress::Ready); + let unresolved = Teardown::Unresolved(reason.to_owned()); + let lost = Evidence::LostStatus; + assert_recorded(&failure, &crashed.lease, &locked, ready, lost, unresolved); + assert!(failure.message().ends_with(reason), "{failure}"); + // The journal remains, and with it the lease, and the clone's records. + assert!(locked.journal().unwrap().is_some()); + crashed.refuses_release(&mut locked); + // A reattach that can end the launch resolves the teardown, and keeps the failure as it was. + let resolved = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let terminated = Teardown::Terminated; + assert_recorded(&resolved, &crashed.lease, &locked, ready, lost, terminated); + assert_eq!(resolved.message(), failure.record().message); + assert!(control.capabilities.is_empty()); + crashed.releases(&mut locked); +} + +/// Returns the error of `reattached`, which must have failed. +fn reattach_error(reattached: crate::Result>) -> Error { + match reattached { + Err(error) => error, + Ok(other) => panic!("the reattach found {}", describe(&other)), + } +} + +/// Takes the clone of the sandbox of `locked` over, under `runtime`, through `control`, with +/// [`reattach_clone`], while a directory takes the place of the record of the clone's failure, as +/// on a storage that refuses to write it, once the reattach read the sandbox's records, and +/// returns the error of the reattach. +fn unrecorded_take_over( + locked: &mut LockedStore, + runtime: &RuntimeIdentity, + control: &mut ReattachControl, +) -> Error { + let path = locked.clone_failure_path(); + let reattached = reattach_clone_with( + locked, + runtime, + control, + Instant::now() + TIMEOUT, + TIMEOUT, + |locked, runtime, exited| { + fs::create_dir(&path).unwrap(); + supervisor::reattach_reporting(locked, runtime, exited) + }, + ); + fs::remove_dir(&path).unwrap(); + reattach_error(reattached) +} + +#[test] +fn a_reattach_that_cannot_record_a_failure_ends_and_forgets_nothing_of_the_clone() { + let unrecorded = |error: &Error| { + let left = "; the reattach left the clone's VM and its records as it found them, for a \ + later reattach"; + let message = error.message(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + let start = "the failure could not be recorded before its teardown: "; + assert!( + message.starts_with(start) && message.ends_with(left), + "{error}" + ); + }; + let mut control = ReattachControl::answering(Some); + // A record of a failure that cannot be read, which the reattach would replace with its own + // before it ends the interrupted launch that the journal names. + let crashed = Crashed::at("activated"); + let mut locked = crashed.lock(); + let path = locked.clone_failure_path(); + fs::create_dir(&path).unwrap(); + let deadline = Instant::now() + TIMEOUT; + let runtime = &crashed.runtime; + let reattached = reattach_clone(&mut locked, runtime, &mut control, deadline, TIMEOUT); + unrecorded(&reattach_error(reattached)); + let journal = locked + .journal() + .unwrap() + .expect("the reattach kept the journal"); + assert!(Process::open(journal.process().unwrap()).unwrap().is_some()); + crashed.refuses_release(&mut locked); + // Once the record can be written, the reattach ends the launch, and records its failure. + fs::remove_dir(&path).unwrap(); + let failure = failed(crashed.take_over(&mut locked, runtime, &mut control)); + let (ready, killed) = ((Stage::Start, Progress::Ready), Evidence::ForcedKill); + let terminated = Teardown::Terminated; + assert_recorded(&failure, &crashed.lease, &locked, ready, killed, terminated); + crashed.releases(&mut locked); + // A VM whose launch committed, and that exited without a report of the guest's own end: the + // reattach would record that the clone lost its status before it clears the VM's record. + let crashed = Crashed::at("committed"); + crashed.vmm.end_and_wait(); + let mut locked = crashed.lock(); + let record = locked.runtime().unwrap(); + assert!(record.is_some()); + unrecorded(&unrecorded_take_over( + &mut locked, + &crashed.runtime, + &mut control, + )); + assert_eq!(locked.runtime().unwrap(), record); + crashed.refuses_release(&mut locked); + let failure = failed(crashed.take_over(&mut locked, &crashed.runtime, &mut control)); + let (run, lost) = ((Stage::Run, Progress::Committed), Evidence::LostStatus); + assert_recorded( + &failure, + &crashed.lease, + &locked, + run, + lost, + Teardown::Exited, + ); + crashed.releases(&mut locked); + // A VM that runs under another runtime identity, and one whose Ready reports another + // activation than its record's: the reattach would end either once it recorded why. + let other = RuntimeIdentity::new(artifact('d'), artifact('e'), artifact('f')); + let mut refusing = ReattachControl::answering(|_| Some(ReadyActivation { transaction: None })); + let reattach = (Stage::Reattach, Progress::Committed); + let cases = [ + (Some(&other), Evidence::RuntimeMismatch), + (None, Evidence::Protocol), + ]; + for (runtime, evidence) in cases { + let crashed = Crashed::at("committed"); + let mut locked = crashed.lock(); + let (runtime, control) = match runtime { + Some(other) => (other, &mut control), + None => (&crashed.runtime, &mut refusing), + }; + unrecorded(&unrecorded_take_over(&mut locked, runtime, control)); + let record = locked + .runtime() + .unwrap() + .expect("the reattach kept the record"); + assert!( + Process::open(record.process()).unwrap().is_some(), + "{evidence:?}" + ); + crashed.refuses_release(&mut locked); + let failure = failed(crashed.take_over(&mut locked, runtime, control)); + let terminated = Teardown::Terminated; + assert_recorded( + &failure, + &crashed.lease, + &locked, + reattach, + evidence, + terminated, + ); + crashed.releases(&mut locked); + } +} + +#[test] +fn a_reattach_makes_the_record_of_an_earlier_failure_durable_before_it_ends_the_launch() { + // A failure of the clone's launch, whose record is in place, but whose teardown a crash of + // its caller interrupted, maybe once the sync of the record's directory failed. + let crashed = Crashed::at("activated"); + let mut locked = crashed.lock(); + let ready = (Stage::Start, Progress::Ready); + let error = Error::other("the test's launch failed"); + let at = (ready, transaction_of(&locked)); + let earlier = (Evidence::Protocol, Teardown::Pending); + record_earlier(&crashed.lease, &locked, at, earlier, &error); + let mut control = ReattachControl::answering(Some); + let reattach = |locked: &mut LockedStore, + runtime: &RuntimeIdentity, + exited: &dyn Fn(&RuntimeRecord) -> Result<()>| { + supervisor::reattach_reporting(locked, runtime, exited) + }; + // A reattach that cannot make the record durable ends nothing. + let refused = |_: &Path| -> Result<()> { Err(Error::io("the test's sync failed")) }; + let deadline = Instant::now() + TIMEOUT; + let runtime = &crashed.runtime; + let reattached = reattach_clone_syncing( + &mut locked, + runtime, + &mut control, + deadline, + TIMEOUT, + &refused, + reattach, + ); + let error = reattach_error(reattached); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + let refusal = "could not be made durable before its teardown: the test's sync failed"; + assert!(error.message().contains(refusal), "{error}"); + let journal = locked + .journal() + .unwrap() + .expect("the reattach kept the journal"); + assert!(Process::open(journal.process().unwrap()).unwrap().is_some()); + crashed.refuses_release(&mut locked); + // One that can syncs the sandbox's directory before it ends the launch. + let synced = RefCell::new(Vec::new()); + let sync = |dir: &Path| -> Result<()> { + synced.borrow_mut().push(dir.to_owned()); + sync_dir(dir) + }; + let dir = locked.dir().to_owned(); + let reattached = reattach_clone_syncing( + &mut locked, + runtime, + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + &sync, + |locked, runtime, exited| { + assert_eq!(*synced.borrow(), std::slice::from_ref(&dir)); + reattach(locked, runtime, exited) + }, + ); + let failure = failed(reattached.unwrap()); + let (protocol, terminated) = (Evidence::Protocol, Teardown::Terminated); + assert_recorded( + &failure, + &crashed.lease, + &locked, + ready, + protocol, + terminated, + ); + crashed.releases(&mut locked); +} diff --git a/nvx-rs/src/host/template/clone/tests/restore.rs b/nvx-rs/src/host/template/clone/tests/restore.rs index e905ee0ad..753b735d8 100644 --- a/nvx-rs/src/host/template/clone/tests/restore.rs +++ b/nvx-rs/src/host/template/clone/tests/restore.rs @@ -13,14 +13,17 @@ use std::collections::VecDeque; use std::thread; use std::time::{Duration, Instant}; -use super::launch::{OPENVMM, TIMEOUT, VmmDir, complete_snapshot, fake_vmm}; +use super::launch::{ + OPENVMM, TIMEOUT, VmmDir, assert_recorded, complete_snapshot, failed, fake_vmm, runtime_of, +}; use super::*; use crate::activation::{ReadyActivation, RestoreSection, TransactionId}; use crate::host::platform::Transport; -use crate::host::process::Process; +use crate::host::process::{BootId, Process}; use crate::host::store::{CLONE_ACTIVATION_NAME, Recovery}; -use crate::host::supervisor::StopOutcome; +use crate::host::supervisor::{Starting, StopOutcome}; use crate::host::template::clone::launch::Hooks; +use crate::host::template::clone::records::{End, read_failure}; /// How long a VMM whose plan delays its event waits before it sends it. const DELAY: Duration = Duration::from_millis(300); @@ -288,6 +291,17 @@ fn launch<'a>( acquired: &'a mut Acquired, image: &(PathBuf, DistroBinding), vmm: &mut VmmDir, +) -> (Restoring<'a>, Launched) { + launch_ending(fixture, acquired, image, vmm, Hooks::LAUNCH.end) +} + +/// Launches the clone as [`launch`] does, whose failures end the launch with `end`. +fn launch_ending<'a>( + fixture: &Fixture, + acquired: &'a mut Acquired, + image: &(PathBuf, DistroBinding), + vmm: &mut VmmDir, + end: End, ) -> (Restoring<'a>, Launched) { let request = tmpfs_request(&image.0, &image.1); let plan = fixture.plan(acquired, &request, NO_CONTENT).unwrap(); @@ -297,6 +311,7 @@ fn launch<'a>( let command = fake_vmm(&dir, &made); let hooks = Hooks { command: &command, + end, ..Hooks::LAUNCH }; let before = Instant::now(); @@ -345,6 +360,25 @@ fn assert_launch_ended(fixture: &Fixture, acquired: &mut Acquired, vmm: &VmmDir) ); } +/// Checks that `failure`, of the clone of `acquired`, failed at `stage`, once the clone got as far +/// as `progress`, with `evidence`, and the teardown `teardown`, as [`assert_recorded`] does. +fn assert_failed( + failure: &CloneFailure, + acquired: &Acquired, + at: (Stage, Progress), + evidence: Evidence, + teardown: Teardown, +) { + assert_recorded( + failure, + &acquired.lease, + &acquired.locked, + at, + evidence, + teardown, + ); +} + #[test] fn a_clone_is_activated_once_its_restore_is_ready_then_started_and_committed() { let (fixture, image) = fixture(); @@ -417,6 +451,26 @@ fn a_clone_is_activated_once_its_restore_is_ready_then_started_and_committed() { ); let (running, session) = started.commit().unwrap(); assert_eq!(session.number, 1); + // The progress log records each step, the spawn and the acknowledgment with the steps that + // follow them at once, after a first line that names the launch and the current boot of the + // host, and the sandbox's records show the committed clone. + let log = fs::read_to_string(acquired.locked.clone_progress_path()).unwrap(); + assert_eq!( + log, + format!( + "nvx-clone-progress 1 {} {}\nspawned\nevent\nacknowledged\nactivating\nready\n\ + starting\nstarted\ncommitted\n", + acquired.lease.launch(), + BootId::current().unwrap() + ) + ); + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!( + (records.progress, records.failure, records.journal), + (Progress::Committed, None, None) + ); + assert_eq!(records.lease, acquired.lease); + assert_eq!(records.runtime, acquired.locked.runtime().unwrap()); // The runtime record names the VM, whose lease lasts as long as it runs. let locked = &mut acquired.locked; let recorded = locked.runtime().unwrap().unwrap(); @@ -471,6 +525,9 @@ fn a_ready_that_does_not_report_the_record_s_activation_ends_the_launch() { ); // One session, which the host ended, and no start. assert_eq!(control.calls.len(), 2, "{name}: {:?}", control.calls); + let ready = (Stage::Activation, Progress::Activating); + let terminated = Teardown::Terminated; + assert_failed(&error, &acquired, ready, Evidence::Protocol, terminated); assert_launch_ended(&fixture, &mut acquired, &vmm); } } @@ -542,6 +599,13 @@ fn a_retry_never_sends_another_record_nor_takes_another_activation() { assert!(error.message().contains("no longer holds"), "{error}"); assert_eq!(control.sessions, 1); assert_eq!(control.records(), [&original.encode()[..]]); + // The host's own record changed: the record of the failure names the launch's transaction. + let record = error.record(); + assert_eq!( + (record.evidence, record.outcome, record.stage), + (Evidence::HostFailure, Outcome::Host, Stage::Activation) + ); + assert_eq!(record.transaction, Some(restore.transaction)); assert_launch_ended(&fixture, &mut acquired, &vmm); // The Ready of the retry reports another transaction. @@ -558,6 +622,14 @@ fn a_retry_never_sends_another_record_nor_takes_another_activation() { .unwrap_err(); assert_eq!(error.kind(), ErrorKind::Protocol, "{error}"); assert_eq!(control.sessions, 2); + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::Protocol, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -576,6 +648,14 @@ fn only_lost_sessions_are_replaced_and_only_so_many_times() { assert_eq!(error.kind(), ErrorKind::Protocol, "{error}"); assert_eq!(error.message(), "cannot activate it"); assert_eq!(control.sessions, 1); + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::Protocol, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); // A session lost again and again ends the sequence once the last of its sessions is lost. @@ -591,6 +671,13 @@ fn only_lost_sessions_are_replaced_and_only_so_many_times() { assert!(error.message().contains("was lost"), "{error}"); control.assert_activates_first(); assert_eq!(control.sessions, MAX_ACTIVATION_SESSIONS); + assert_failed( + &error, + &acquired, + ready, + Evidence::Reset, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -611,6 +698,15 @@ fn a_clone_that_took_the_reset_of_a_lost_session_before_its_ready_ends_the_retri assert_eq!(error.kind(), ErrorKind::Exited, "{error}"); assert_eq!(control.sessions, 1); drop(control); + // The clone's status was lost with its VMM, which ended without a report. + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::LostStatus, + Teardown::Exited, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -631,6 +727,14 @@ fn the_restore_to_ready_deadline_runs_from_the_launch_through_every_step() { assert_eq!(error.kind(), ErrorKind::TimedOut, "{error}"); assert!(launched.before.elapsed() >= SHORT); assert!(control.calls.is_empty()); + let restore = (Stage::Restore, Progress::Spawned); + assert_failed( + &error, + &acquired, + restore, + Evidence::Timeout, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); // A clone that never answers its Activate, after a late event: the attachment and the @@ -650,6 +754,14 @@ fn the_restore_to_ready_deadline_runs_from_the_launch_through_every_step() { launched.assert_deadline(deadline, deadlines.ready); } assert!(launched.before.elapsed() >= deadlines.ready); + let ready = (Stage::Activation, Progress::Activating); + assert_failed( + &error, + &acquired, + ready, + Evidence::Timeout, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -681,6 +793,14 @@ fn the_ready_to_start_deadline_runs_from_the_ready_and_a_start_is_never_resent() deadline: start }) ); + let starting = (Stage::Start, Progress::Starting); + assert_failed( + &error, + &acquired, + starting, + Evidence::Timeout, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); // A start that fails, even in a lost session, is not sent again. @@ -698,6 +818,13 @@ fn the_ready_to_start_deadline_runs_from_the_ready_and_a_start_is_never_resent() .filter(|call| matches!(call, Call::Start { .. })) .count(); assert_eq!((control.sessions, starts), (1, 1)); + assert_failed( + &error, + &acquired, + starting, + Evidence::Reset, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -705,14 +832,59 @@ fn the_ready_to_start_deadline_runs_from_the_ready_and_a_start_is_never_resent() fn a_vmm_that_exits_or_sends_anything_but_its_event_ends_the_launch() { let (fixture, image) = fixture(); // A VMM that exits before its event, as OpenVMM does once it cannot restore the snapshot, ends - // the wait at once; another version of the event, and part of it, are refused. + // the wait at once, and loses the clone's status, unless OpenVMM's fatal error tells that the + // host cannot present the template's CPU or time; another version of the event, and part of + // it, are refused, and the VMM that sent them is terminated. let cases = [ - ("exit", ErrorKind::Exited), - ("other flush", ErrorKind::Protocol), - ("partial flush", ErrorKind::Protocol), + ( + "exit", + "exit", + ErrorKind::Exited, + Evidence::LostStatus, + Teardown::Exited, + ), + ( + "incompatible", + "exit fatal:E_TSC_RATE_TOLERANCE", + ErrorKind::Exited, + Evidence::IncompatibleHost, + Teardown::Exited, + ), + ( + "contract", + "exit fatal:E_MANIFEST_TIME", + ErrorKind::Exited, + Evidence::LostStatus, + Teardown::Exited, + ), + // OpenVMM connects the event's endpoint before it launches its VM worker, whose launch + // fails on a host that cannot present the template's CPU or time: it closes the endpoint + // without the event, and reports its error only as it exits, which the wait awaits. + ( + "closed", + "close fatal:E_TSC_RATE_TOLERANCE", + ErrorKind::Exited, + Evidence::IncompatibleHost, + Teardown::Exited, + ), + ( + "other", + "other flush", + ErrorKind::Protocol, + Evidence::Protocol, + Teardown::Terminated, + ), + ( + "partial", + "partial flush", + ErrorKind::Protocol, + Evidence::Protocol, + Teardown::Terminated, + ), ]; - for (plan, kind) in cases { - let mut acquired = fixture.acquire(&plan.replace(' ', "-")); + let restore = (Stage::Restore, Progress::Spawned); + for (name, plan, kind, evidence, teardown) in cases { + let mut acquired = fixture.acquire(name); let mut vmm = vmm_with(plan); let (restoring, launched) = launch(&fixture, &mut acquired, &image, &mut vmm); let mut control = FakeControl::default(); @@ -722,6 +894,17 @@ fn a_vmm_that_exits_or_sends_anything_but_its_event_ends_the_launch() { assert_eq!(error.kind(), kind, "{plan}: {error}"); assert!(launched.before.elapsed() < TIMEOUT, "{plan}"); assert!(control.calls.is_empty(), "{plan}"); + assert_failed(&error, &acquired, restore, evidence, teardown); + // The record keeps the line of OpenVMM's fatal error, and not the warning before it. + let fatal = plan + .split_whitespace() + .find_map(|word| word.strip_prefix("fatal:")); + if let Some(code) = fatal { + let line = + format!("OpenVMM reported: fatal error: [{code}] failed to launch vm worker"); + assert!(error.message().ends_with(&line), "{error}"); + assert!(error.record().message.ends_with(&line), "{error}"); + } assert_launch_ended(&fixture, &mut acquired, &vmm); } @@ -742,6 +925,396 @@ fn a_vmm_that_exits_or_sends_anything_but_its_event_ends_the_launch() { let source = std::error::Error::source(&error).unwrap().to_string(); assert!(source.contains("not the VMM"), "{source}"); assert!(control.calls.is_empty()); + assert_failed( + &error, + &acquired, + restore, + Evidence::Protocol, + Teardown::Terminated, + ); assert_launch_ended(&fixture, &mut acquired, &vmm); assert!(!acquired.locked.dir().join(CLONE_ACTIVATION_NAME).exists()); } + +#[test] +fn a_status_with_which_the_guest_ended_the_clone_s_vm_attributes_its_failure() { + let (fixture, image) = fixture(); + let ready = (Stage::Activation, Progress::Activating); + let starting = (Stage::Start, Progress::Starting); + // Each VMM ends as its clone awaits its Ready, or once the start of its sandbox was sent, with + // OpenVMM's outcome report, which the failure's record reads once the VMM has exited. + let cases = [ + // The guest's activation could not assemble the clone's root, which only the request's + // image and upper make, before any workload ran. + ( + "root", + "guest-exit:54", + ready, + Evidence::UnusableRoot, + Some(54), + ), + // A repair's status is evidence of the repair stage at any point. + ( + "repair", + "guest-exit:195", + ready, + Evidence::RepairStatus, + Some(195), + ), + ( + "late-repair", + "guest-exit:195", + starting, + Evidence::RepairStatus, + Some(195), + ), + // Once the start was sent, a workload may have ended its VM with any status, so the loss + // of the session is all that the host knows. + ( + "late-root", + "guest-exit:54", + starting, + Evidence::Reset, + Some(54), + ), + // A failure of the VMM tells nothing of the guest. + ("vmm", "vmm-failure:1", ready, Evidence::LostStatus, None), + ]; + for (name, report, at, evidence, status) in cases { + let mut acquired = fixture.acquire(name); + let mut vmm = vmm_with(&format!("flush report:{report}")); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let error = if at == starting { + let mut control = FakeControl::default(); + control.starts.push_back(Answer::Fail(Failure::LOST)); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + activated.start(&mut control).unwrap_err() + } else { + let ending = &vmm; + let mut control = FakeControl::activating([Answer::Fail(Failure::LOST)]); + control.on_activate = Some(Box::new(move |_| ending.end_and_wait())); + restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err() + }; + assert_failed(&error, &acquired, at, evidence, Teardown::Exited); + assert_eq!(error.record().status, status, "{name}"); + assert_launch_ended(&fixture, &mut acquired, &vmm); + } +} + +thread_local! { + /// The path of the record of a failure that [`ended_once_recorded`] reads, and the record that + /// it read there. + static RECORDED: RefCell)>> = const { RefCell::new(None) }; +} + +/// Ends a launch as [`Starting::abandon`] does, once it read the record of the failure at the path +/// in [`RECORDED`], as the teardown begins, which a crash of the caller during the teardown would +/// leave. +fn ended_once_recorded(starting: Starting<'_>) -> Result { + RECORDED.with_borrow_mut(|recorded| { + let (path, record) = recorded.as_mut().expect("the test names the record's path"); + *record = read_failure(path).unwrap(); + }); + starting.abandon() +} + +#[test] +fn a_failure_is_recorded_before_its_teardown_and_again_once_the_teardown_ended() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("sandbox"); + RECORDED.set(Some((acquired.locked.clone_failure_path(), None))); + let mut vmm = vmm_with("other flush"); + let (restoring, _) = launch_ending( + &fixture, + &mut acquired, + &image, + &mut vmm, + ended_once_recorded, + ); + let mut control = FakeControl::default(); + let error = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err(); + // As the teardown began, the sandbox recorded the failure, with its teardown pending. + let (_, pending) = RECORDED.take().unwrap(); + let pending = pending.expect("the failure was recorded before its teardown"); + let restore = (Stage::Restore, Progress::Spawned); + assert_eq!( + (pending.stage, pending.progress, pending.evidence), + (restore.0, restore.1, Evidence::Protocol) + ); + assert_eq!(pending.teardown, Teardown::Pending); + assert_eq!(pending.message, error.record().message); + // Once the teardown ended, the record holds its outcome. + assert_failed( + &error, + &acquired, + restore, + Evidence::Protocol, + Teardown::Terminated, + ); + assert_launch_ended(&fixture, &mut acquired, &vmm); +} + +/// Why [`kept`] keeps a launch. +const KEPT: &str = "the test keeps the launch"; + +/// Ends no launch of a clone that failed: keeps its journal and its VMM, as a launch whose VMM's +/// identity or exit is indeterminate keeps them. +fn kept(starting: Starting<'_>) -> Result { + // Nothing of the launch ends: its journal and its VMM outlive it, as a crash leaves them. + std::mem::forget(starting); + Ok(Recovery::Indeterminate(KEPT.to_owned())) +} + +#[test] +fn a_launch_that_cannot_be_ended_keeps_its_records_and_its_lease_until_a_reattach_ends_it() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("sandbox"); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch_ending(&fixture, &mut acquired, &image, &mut vmm, kept); + let mut control = FakeControl::activating([Answer::Fail(Failure::REFUSED)]); + let error = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap_err(); + let ready = (Stage::Activation, Progress::Activating); + let unresolved = Teardown::Unresolved(KEPT.to_owned()); + assert_failed(&error, &acquired, ready, Evidence::Protocol, unresolved); + let unended = format!("; the launch could not be ended, and remains for recovery: {KEPT}"); + assert!(error.message().ends_with(&unended), "{error}"); + // The journal remains, and with it the lease, which no release ends, and the clone's records. + let reference = vmm.reference.clone().unwrap(); + let locked = &mut acquired.locked; + let refusal = fixture.store.release(locked).unwrap_err(); + assert_eq!(refusal.code(), ErrorCode::SandboxRunning, "{refusal}"); + let records = CloneRecords::read(locked).unwrap().unwrap(); + assert_eq!(records.journal.unwrap().process(), Some(&reference)); + assert!(Process::open(&reference).unwrap().is_some()); + // A reattach ends the launch through its VMM's verified identity: the record keeps the + // failure's stage, evidence, and original outcome, with the teardown that resolved it. + let mut control = FakeControl::default(); + let reattached = reattach_clone( + locked, + &runtime_of(&fixture), + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ) + .unwrap(); + let resolved = failed(reattached); + assert_failed( + &resolved, + &acquired, + ready, + Evidence::Protocol, + Teardown::Terminated, + ); + assert_eq!( + (resolved.code(), resolved.message()), + (error.code(), error.record().message.as_str()) + ); + assert!(control.calls.is_empty()); + assert_launch_ended(&fixture, &mut acquired, &vmm); +} + +#[test] +fn a_caller_fails_a_ready_or_started_clone_with_its_own_evidence_as_the_sequence_fails_one() { + let (fixture, image) = fixture(); + let ready = (Stage::Start, Progress::Ready); + // A step of the caller's own, such as one that maps a directory of the host into the clone, + // fails once the clone is ready, which the request's mappings attribute to the request. + let mut acquired = fixture.acquire("ready"); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + let mapping = "the test's mapping names no directory of the host"; + let failure = activated.fail(Evidence::InvalidMappings, Error::unsupported(mapping)); + assert_eq!(failure.kind(), ErrorKind::Unsupported, "{failure}"); + assert_eq!(failure.message(), mapping); + let mappings = Evidence::InvalidMappings; + assert_failed(&failure, &acquired, ready, mappings, Teardown::Terminated); + assert_eq!(failure.outcome(), Outcome::Request); + assert_launch_ended(&fixture, &mut acquired, &vmm); + // A check of the caller's own fails once the clone's sandbox started, before the commit. + let mut acquired = fixture.acquire("started"); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + let started = activated.start(&mut control).unwrap(); + let failure = started.fail(Evidence::HostFailure, Error::io("the test's check failed")); + let commit = (Stage::Commit, Progress::Started); + let host = Evidence::HostFailure; + assert_failed(&failure, &acquired, commit, host, Teardown::Terminated); + assert_eq!(failure.outcome(), Outcome::Host); + assert_launch_ended(&fixture, &mut acquired, &vmm); + // The guest ended the clone's VM with the status of a failed repair as the caller's step + // failed: the status prevails, as over the sequence's own failures. + let mut acquired = fixture.acquire("repair"); + let mut vmm = vmm_with("flush report:guest-exit:195"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + let failure = activated.fail(Evidence::InvalidMappings, Error::unsupported(mapping)); + let repair = Evidence::RepairStatus; + assert_failed(&failure, &acquired, ready, repair, Teardown::Exited); + assert_eq!(failure.record().status, Some(195)); + assert_launch_ended(&fixture, &mut acquired, &vmm); + // Evidence of the template, such as a defect of the generation that the caller diagnosed, + // shows the provenance that the status alone leaves to establish: it prevails, and the record + // holds the status beside it. The failure quarantines the generation once the sandbox's lock + // is released, as the lock order asks. + let mut acquired = fixture.acquire("defect"); + let mut vmm = vmm_with("flush report:guest-exit:195"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + let defect = "the test diagnosed a defect of the generation's repair"; + let failure = activated.fail(Evidence::TemplateDefect, Error::other(defect)); + let template = Evidence::TemplateDefect; + assert_failed(&failure, &acquired, ready, template, Teardown::Exited); + assert_eq!( + (failure.outcome(), failure.record().status), + (Outcome::Template, Some(195)) + ); + assert_launch_ended(&fixture, &mut acquired, &vmm); + drop(acquired); + assert!(failure.quarantine(&fixture.store, None).unwrap()); + assert_eq!( + fixture.store.tombstones().unwrap(), + [failure.record().template.generation] + ); +} + +#[test] +fn a_failure_that_cannot_be_recorded_before_its_teardown_keeps_its_launch_for_a_reattach() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("unrecorded"); + let path = acquired.locked.clone_failure_path(); + let mut vmm = vmm_with("flush"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + // The caller diagnoses a defect of the template once the clone is ready, but the record of + // its failure cannot be written, as on a full disk: a directory takes its place. + fs::create_dir(&path).unwrap(); + let defect = "the test diagnosed a defect of the generation's repair"; + let failure = activated.fail(Evidence::TemplateDefect, Error::other(defect)); + let ready = (Stage::Start, Progress::Ready); + let record = failure.record(); + assert_eq!( + ((record.stage, record.progress), record.evidence), + (ready, Evidence::TemplateDefect) + ); + // The caller holds the failure, the template's, whose teardown ended the clone's VMM through + // its handle, and kept the launch's journal, and with it the lease. + assert_eq!(failure.outcome(), Outcome::Template); + let Teardown::Unresolved(reason) = &record.teardown else { + panic!("the teardown of {failure} was resolved"); + }; + let unrecorded = "the failure could not be recorded before its teardown: "; + let ended = format!("the clone's VMM ended through its handle, but {unrecorded}"); + assert!(reason.starts_with(&ended), "{reason}"); + assert!(failure.message().starts_with(defect), "{failure}"); + let lost = "; the failure could not be recorded: "; + assert!(failure.message().contains(lost), "{failure}"); + assert_vmm_ended(&vmm); + let journal = acquired.locked.journal().unwrap(); + let journal = journal.expect("the launch kept its journal"); + assert_eq!(journal.process(), vmm.reference.as_ref()); + let refusal = fixture.store.release(&mut acquired.locked).unwrap_err(); + assert_eq!(refusal.code(), ErrorCode::SandboxRunning, "{refusal}"); + // Once the record can be written, a reattach, as after a crash of the caller, finds that the + // clone failed, though not how, rather than a clone that ended. + fs::remove_dir(&path).unwrap(); + let mut control = FakeControl::default(); + let reattached = reattach_clone( + &mut acquired.locked, + &runtime_of(&fixture), + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ) + .unwrap(); + let resolved = failed(reattached); + let exited = Teardown::Exited; + assert_failed(&resolved, &acquired, ready, Evidence::LostStatus, exited); + assert!(control.calls.is_empty()); + assert_launch_ended(&fixture, &mut acquired, &vmm); +} + +#[test] +fn a_diagnosis_too_long_for_a_record_keeps_its_ends_and_its_evidence_through_a_reattach() { + let (fixture, image) = fixture(); + let mut acquired = fixture.acquire("diagnosis"); + let mut vmm = vmm_with("flush report:guest-exit:195"); + let (restoring, _) = launch(&fixture, &mut acquired, &image, &mut vmm); + let mut control = FakeControl::default(); + let activated = restoring + .activate(&mut control, RestoreDeadlines::default()) + .unwrap(); + vmm.end_and_wait(); + // The caller's diagnosis of a defect of the generation is longer than a record keeps: JSON + // takes six bytes for each of its control characters, the most that a character takes. + let (first, last) = ( + "the test's diagnosis begins", + "and the test's diagnosis ends", + ); + let defect = format!("{first}{}{last}", "\u{1}".repeat(64 * 1024)); + let failure = activated.fail(Evidence::TemplateDefect, Error::other(defect.clone())); + // The failure's error keeps the diagnosis whole, and its record, which the sandbox keeps, + // its ends. + assert_eq!(failure.message(), defect); + let record = failure.record(); + let ready = (Stage::Start, Progress::Ready); + assert_eq!( + ( + (record.stage, record.progress), + record.evidence, + &record.teardown + ), + (ready, Evidence::TemplateDefect, &Teardown::Exited) + ); + assert!(record.message.starts_with(first) && record.message.ends_with(last)); + assert!(record.message.chars().count() < defect.chars().count()); + let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); + assert_eq!(records.failure.as_ref(), Some(record)); + // A reattach finds the failure as it was recorded, with the template's evidence, which + // quarantines the generation once the sandbox's lock is released. + let mut control = FakeControl::default(); + let reattached = reattach_clone( + &mut acquired.locked, + &runtime_of(&fixture), + &mut control, + Instant::now() + TIMEOUT, + TIMEOUT, + ) + .unwrap(); + let again = failed(reattached); + assert_eq!(again.record(), record); + assert_launch_ended(&fixture, &mut acquired, &vmm); + drop(acquired); + assert!(again.quarantine(&fixture.store, None).unwrap()); + assert_eq!( + fixture.store.tombstones().unwrap(), + [record.template.generation] + ); +} diff --git a/nvx-rs/src/host/template/mod.rs b/nvx-rs/src/host/template/mod.rs index 00af2578c..4e05c8b01 100644 --- a/nvx-rs/src/host/template/mod.rs +++ b/nvx-rs/src/host/template/mod.rs @@ -47,6 +47,17 @@ //! resends the same record in a new session once the adapter lost one. [`Activated::start`] //! then starts the clone's sandbox by a deadline that starts at its Ready, and //! [`Started::commit`] commits the launch. +//! - The plan, the launch, and the restore sequence fail with a [`CloneFailure`], whose +//! [`Evidence`] attributes it to an [`Outcome`], at a [`Stage`], once the clone got as far as +//! its [`Progress`], with the [`Teardown`] that followed: a failure from the launch on ends the +//! clone's VMM through its verified identity, and is recorded in the clone's sandbox, which +//! keeps its lease while a VM of the clone may run. Only a failure of the template quarantines +//! the leased generation, through its tombstone, [`CloneFailure::quarantine`]. +//! - [`reattach_clone`] takes a clone over once the caller of its launch restarted: it checks the +//! clone's activation in a new session, and ends a clone that another runtime identity +//! launched, or that fails its check, or classifies how it ended; [`CloneRecords::read`] reads +//! what the clone's sandbox keeps of it, for the operations that list, inspect, or remove +//! clones. //! //! # Canonical encoding //! @@ -79,9 +90,10 @@ mod test_support; mod wire; pub use self::clone::{ - Activated, CloneControl, ClonePlan, CloneRequest, CloneUpper, MAX_ACTIVATION_SESSIONS, - READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, RestoreDeadlines, - Restoring, Started, + Activated, CloneControl, CloneFailure, ClonePlan, CloneRecords, CloneRequest, CloneUpper, + Evidence, FAILURE_FORMAT, FailureRecord, MAX_ACTIVATION_SESSIONS, Outcome, Progress, + READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, ReattachedClone, + RestoreDeadlines, Restoring, Stage, Started, Teardown, reattach_clone, status_evidence, }; pub use self::eligibility::{Downtime, DowntimeSource, Eligibility, HostTime, MAX_DOWNTIME_NS}; pub use self::key::{CaptureClass, ClassIdentity, Invariants, KeyIdentity, TemplateKey}; diff --git a/nvx-rs/src/host/template/store.rs b/nvx-rs/src/host/template/store.rs index 762f45e8c..a491c9ba0 100644 --- a/nvx-rs/src/host/template/store.rs +++ b/nvx-rs/src/host/template/store.rs @@ -731,8 +731,9 @@ impl Store { } /// Checks that the directory of the generations is private to the current user, and not a - /// link. - fn check_generations_dir(&self) -> Result<()> { + /// link, as every read of a generation does, so that a clone's plan tells a store whose + /// directory fails it, the host's, from a generation that is malformed. + pub(super) fn check_generations_dir(&self) -> Result<()> { let path = self.dir.join(GENERATIONS_NAME); let what = "the generations of the template store"; let problem = platform::check_owner_only_dir(&path).map_err(store_error(format!( diff --git a/nvx-rs/src/host/template/store/leases.rs b/nvx-rs/src/host/template/store/leases.rs index e19d73967..820779440 100644 --- a/nvx-rs/src/host/template/store/leases.rs +++ b/nvx-rs/src/host/template/store/leases.rs @@ -165,8 +165,10 @@ impl Store { /// - with [`ErrorCode::TemplateIneligible`] if it is not eligible at `now`; /// - on Linux, with [`ErrorKind::Io`] if the generation's payload is not a directory of the /// current user with mode 0700, which a launch of the clone requires, as OpenVMM does of the - /// directory of a control socket, and which a release needs, to remove the clone's socket - /// from the payload and make its absence durable; + /// directory of a control socket, or if a release could not open it as it opens it, + /// following no link at the store's directory, at the directory of the generations, or at + /// the payload: a release opens the payload to remove the clone's socket from it and make + /// its absence durable; /// - with [`ErrorCode::SandboxNotProvisioned`] if the directory does not exist, or holds no /// configuration record of the caller, as `files` declares it; /// - with [`ErrorCode::SandboxRunning`] if the sandbox records a VM, or a launch, which may @@ -229,11 +231,11 @@ impl Store { )); } // A release removes the socket of the clone's VMM through a handle of the payload, which - // it opens for reading, so that it can make the removal durable: a payload that it could - // not open, or remove an entry from, would keep the lease and its binding past it. - if cfg!(unix) { - self.check_payload(generation)?; - } + // it opens for reading, following no link from the store's directory on, so that it can + // make the removal durable: a payload that it could not open, or remove an entry from, + // would keep the lease and its binding past it. + #[cfg(target_os = "linux")] + self.check_payload(generation)?; let not_provisioned = || { Error::new( ErrorCode::SandboxNotProvisioned, @@ -369,8 +371,11 @@ impl Store { /// launch: the file of the scratch device of a clone whose upper is in RAM, /// [`CLONE_SCRATCH_NAME`](crate::host::store::CLONE_SCRATCH_NAME), which the clone's /// [plan](crate::host::template::ClonePlan) created, the clone's activation record, - /// [`CLONE_ACTIVATION_NAME`](crate::host::store::CLONE_ACTIVATION_NAME), and the socket of - /// its restore-ready event, [`RESTORE_READY_NAME`](crate::host::store::RESTORE_READY_NAME); + /// [`CLONE_ACTIVATION_NAME`](crate::host::store::CLONE_ACTIVATION_NAME), the socket of its + /// restore-ready event, [`RESTORE_READY_NAME`](crate::host::store::RESTORE_READY_NAME), and + /// its records, which last as long as the lease, its progress log, + /// [`CLONE_PROGRESS_NAME`](crate::host::store::CLONE_PROGRESS_NAME), and the record of its + /// failure, [`CLONE_FAILURE_NAME`](crate::host::store::CLONE_FAILURE_NAME); /// then, on Linux, the control socket of the clone's VMM, `.sock` in the leased /// generation's payload, after the launch that the lease reserved, whose absence it makes /// durable; then the clone binding, and once their removal is durable, the lease's record in @@ -486,16 +491,24 @@ impl Store { Ok(Some(lease)) } - /// Refuses the payload of the generation `generation` unless it is a directory of the current - /// user with mode 0700, as a launch of a clone requires it, OpenVMM requiring that mode of the - /// directory of a control socket, and as a release of the clone's lease needs it on Linux: the - /// release opens the payload for reading, and removes the clone's socket from it. + /// Refuses the payload of the generation `generation` unless a release of a clone's lease can + /// open it, as [`open_payload`] opens it, following no link at the store's directory, at the + /// directory of the generations, or at the payload, through a directory that holds this + /// store, and it is a directory of the current user with mode 0700. A launch of a clone + /// requires that mode, OpenVMM requiring it of the directory of a control socket, and so does + /// the release: it opens the payload for reading, and removes the clone's socket from it. + #[cfg(target_os = "linux")] fn check_payload(&self, generation: StoreGeneration) -> Result<()> { let payload = self.payload_dir(generation); - let problem = platform::check_store_dir(&payload).map_err(store_error(format!( - "cannot open the payload of the generation {}", - payload.display() - )))?; + let problem = match open_payload(self.dir(), self.id, generation)? { + Some(directory) => directory.store_dir_problem().map_err(store_error(format!( + "cannot examine the payload of the generation {}", + payload.display() + )))?, + None => { + Some("is gone, or the store's directory holds another store, or none".to_owned()) + } + }; match problem { Some(problem) => Err(Error::io(format!( "the payload of the generation {} {problem}", diff --git a/nvx-rs/src/host/template/store/leases/tests.rs b/nvx-rs/src/host/template/store/leases/tests.rs index eeee09695..25066d514 100644 --- a/nvx-rs/src/host/template/store/leases/tests.rs +++ b/nvx-rs/src/host/template/store/leases/tests.rs @@ -18,8 +18,8 @@ use crate::activation::TemplateReference; use crate::host::process::{Process, ProcessRef}; use crate::host::store::files::write_private; use crate::host::store::{ - CLONE_ACTIVATION_NAME, CLONE_SCRATCH_NAME, FORMAT, LOCK_NAME, LaunchState, RESTORE_READY_NAME, - RUNTIME_NAME, Recovery, RuntimeIdentity, + CLONE_ACTIVATION_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME, CLONE_SCRATCH_NAME, FORMAT, + LOCK_NAME, LaunchState, RESTORE_READY_NAME, RUNTIME_NAME, Recovery, RuntimeIdentity, }; use crate::host::supervisor::{self, Reattached, StopOutcome, VmmCommand}; use crate::host::template::eligibility::MAX_DOWNTIME_NS; @@ -285,26 +285,52 @@ fn acquisitions_refuse_what_they_cannot_lease_and_lease_nothing() { assert_eq!(error.code(), ErrorCode::TemplateIneligible, "{error}"); assert!(error.message().contains(condition), "{error}"); } - // On Unix, a payload that is not a directory of the current user with mode 0700, which a + // On Linux, a payload that is not a directory of the current user with mode 0700, which a // launch of the clone requires, and through which a release ends the lease: it opens the - // payload for reading, and removes the clone's socket from it. - #[cfg(unix)] + // payload for reading, and removes the clone's socket from it. A user other than root cannot + // open a payload of mode 0300 for reading at all. + #[cfg(target_os = "linux")] { use std::os::unix::fs::PermissionsExt; let payload = store.payload_dir(generation); + let unopened = format!( + "cannot open the payload of the generation {}", + payload.display() + ); for mode in [0o300, 0o500] { fs::set_permissions(&payload, fs::Permissions::from_mode(mode)).unwrap(); let error = refusal(&store, generation, &sandbox, &now); fs::set_permissions(&payload, fs::Permissions::from_mode(0o700)).unwrap(); assert_eq!(error.kind(), ErrorKind::Io, "{error}"); - assert_eq!( - error.message(), - format!( - "the payload of the generation {} has mode {mode:o}, not the mode 700 that \ - the directory of a control socket must have", - payload.display() - ) + let refused = format!( + "the payload of the generation {} has mode {mode:o}, not the mode 700 that the \ + directory of a control socket must have", + payload.display() + ); + assert!( + error.message() == refused || (mode == 0o300 && error.message() == unopened), + "{error}" + ); + } + // A link in the place of the store's directory, of the directory of the generations, or of + // the payload, even to a directory that holds the same files: a release, which follows + // none of them, could not open the payload. + let root = store.dir().to_owned(); + let generations = payload.parent().unwrap().to_owned(); + let elsewhere = directory.path().join("elsewhere"); + for linked in [&root, &generations, &payload] { + fs::rename(linked, &elsewhere).unwrap(); + std::os::unix::fs::symlink(&elsewhere, linked).unwrap(); + let error = refusal(&store, generation, &sandbox, &now); + fs::remove_file(linked).unwrap(); + fs::rename(&elsewhere, linked).unwrap(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + assert_eq!(error.message(), unopened, "{error}"); + let source = std::error::Error::source(&error).unwrap().to_string(); + assert!( + source.ends_with("is a link, or not a directory"), + "{source}" ); } } @@ -819,6 +845,8 @@ fn a_release_makes_the_removal_of_its_binding_durable_before_it_removes_its_reco let kept = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ]; let synced = RefCell::new(Vec::new()); @@ -1242,6 +1270,8 @@ fn a_release_ends_the_lease_of_the_sandbox_whose_lock_it_holds_whatever_its_path let kept = [ CLONE_SCRATCH_NAME, CLONE_ACTIVATION_NAME, + CLONE_PROGRESS_NAME, + CLONE_FAILURE_NAME, RESTORE_READY_NAME, ]; for sandbox in [&leased_sandbox, &other] {