Skip to content

execution: select restore consumers while preserving checkpoint provenance #767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zibi-pplx wants to merge 1 commit into from
+39 −3
Open

execution: select restore consumers while preserving checkpoint provenance #767

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion crates/libs/sbx/mgr/server/src/service/create.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use super::activation_support::{

Check warning on line 1 in crates/libs/sbx/mgr/server/src/service/create.rs

View workflow job for this annotation

GitHub Actions / rust_static / Rust clippy 

[archcheck] reported by reviewdog 🐶
crates/libs/sbx/mgr/server/src/service/create.rs grew from 1195 to 1248 code lines. Once this rule is required, split the file before adding to it.

Raw Output:
message:"crates/libs/sbx/mgr/server/src/service/create.rs grew from 1195 to 1248 code lines. Once this rule is required, split the file before adding to it."  location:{path:"crates/libs/sbx/mgr/server/src/service/create.rs"  range:{start:{line:1}}}  severity:WARNING  source:{name:"archcheck"}  code:{value:"file_size.listed_no_growth"}
ActivationFailureStage, IdentityFreeDirectActivationAuthority, complete_restore_and_mark_final,
fresh_runtime_activation_identity, runtime_activation_wire_identity_can_open_claim,
space_api_route_identity_for_runtime_activation, space_api_route_identity_from_create_request,
Expand Down Expand Up @@ -714,7 +714,13 @@
.map_err(|error| Status::failed_precondition(error.to_string()))?;
let cpu_budget = restore_cpu_budget(vcpu, snapshot_execution.cpu_budget())?;
let metadata_json = activation_metadata_json(sandbox, cpu_budget)?;
let requested_execution = snapshot_execution.into_execution();
let producer_execution = snapshot_execution.into_execution();
let requested_execution = self
.select_compatible_execution(
&producer_execution,
pplx_space_node_mgr_client::proto::ExecutionEnvironmentSelection::Restore,
)
.await?;
let snapshot_memory_mib = ensure_snapshot_memory_allowed(
snapshot_memory_file_mib(&snapshot_dir)?,
node_memory_policy.guest_memory_max_mib,
Expand Down Expand Up @@ -808,6 +814,15 @@
},
)?;

// Commit the consumer before Prepare can rewrite disk metadata. The
// checkpoint manifest remains producer evidence; a cancelled attempt
// keeps a valid exact consumer for subsequent admission and recovery.
{
let db = self.db_conn()?;
crate::db::upsert_sandbox_execution_metadata(&db, id, &requested_execution)
.map_err(|error| Status::internal(format!("persist restore consumer: {error}")))?;
}

// Allocate network slot
let execution_fence = read_node_execution_fence(&self.db, id)?.ok_or_else(|| {
Status::failed_precondition("activation has no durable node execution identity")
Expand Down
25 changes: 23 additions & 2 deletions docs/execution-environments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,8 +563,8 @@ store and stages its kernel into the execution cache. Cache loss triggers
reconstruction. Missing or corrupt non-default artifacts cannot replace or
withdraw E0. Each compiled binding records its qualified lifecycle support; the
1.17.0 / 6.18.44 and 1.17.0 / 6.1.155 bindings support cold execution and
same-environment memory snapshots. Restoring across environment identities is
not an upgrade path: each snapshot retains its original VMM and kernel binding.
same-environment memory snapshots. Cross-environment restore requires a
qualified compatible loader; each snapshot retains its original producer binding.
The 6.1.155 kernel keeps its existing artifact identity when paired with 1.17.0.
Non-default snapshot capture carries an environment-bound sandbox identity on
`CreateSnapshot`, with the legacy sandbox ID omitted. Old node-managers reject
Expand Down Expand Up @@ -761,3 +761,24 @@ evidence. An unrelated publication does not renew it. Policy is not proof of
installation or usability, and reporting does not implement placement gating.
Deploy the additive migration and cluster readers before node publishers. Inspect
cohort drift and collection freshness before enabling workloads.

### Selecting a restore consumer

The snapshot manifest retains original runtime, kernel and state-format identity.
After claiming activation ownership, sandbox-manager saves the selected loader in
the sandbox row before preparing files or allocating a slot. Prepare and Launch
receive that exact consumer. A failed attempt can leave it recorded; cleanup
retains the checkpoint and a later restore attempt resolves again from producer
evidence. A subsequent snapshot describes the runtime that actually ran.

Pause/resume keeps the resident VM. This policy change does not kill running VMs
or add isolated resume refusal: incident-driven stopping, resource reclamation and
resume revocation must ship together. Stop and delete remain available. Operators
must stop affected resident VMs when revoking execution during an incident.

A future kernel or incompatible-state upgrade requires explicit memory-loss
consent, activation fencing and a preserved matching disk/checkpoint pair. Keep
the old pair until the new cold boot is verified; a failed cold boot must not
mutate the only rollback disk. Retention is not permission to restore a revoked
runtime. Retire the checkpoint through a durable lifecycle operation before later
object garbage collection, never by deleting S3 vmstate and editing lifecycle rows.
Loading