feat(prewarm): single-use retained-pod lifecycle (task 34.01.03)

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -476,6 +476,26 @@ kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\
 kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
 ```
 
+For the experimental native prewarm RPCs, check the explicit Kubernetes
+`prewarm` approval section first. New preparation/binding stay disabled without
+it; ordinary cold creation is unaffected. The supported native profile requires
+shared namespace, combined topology, explicit UID/GID 1000, digest-pinned
+agent/recipe/supervisor images and no provider SPIFFE mount. The recipe requires
+image-volume support. A changed native driver profile cannot be adopted into an
+existing receipt. Restore its original profile rather than recreating resources.
+Indeterminate CREATE tickets remain quarantined until the original resource is
+discovered; do not delete permanent fence records or report absence from a timeout.
+See `docs/reference/gateway-config.mdx` for additional platform-only RBAC.
+
+The staging sidecar profile is not native-prewarm compatible: its operator
+bootstrap, dual-supervisor release and `fsGroup`-affected Secret permissions need
+a reviewed private-grant handoff. Never remove `fsGroup`, weaken root-only grants
+or switch staging topology as a debugging workaround. Native v1's legacy agent
+launch also is not current assistant acceptance. Reserved assistant schema
+version 2 returns `UNIMPLEMENTED` before tenant or resource mutation. Do not
+substitute a legacy recipe to bypass it. Keep all customer pools disabled
+until the actual artifact/profile/storage and joined customer journey pass.
+
 If `topology = "sidecar"` is rendered under `[openshell.drivers.kubernetes]`,
 sandbox pods should have an `openshell-network-init` init container running
 `--mode=network-init`, an `agent` container running

architecture/compute-runtimes.md

@@ -19,6 +19,101 @@ Each runtime receives a sandbox spec from the gateway and is responsible for:
 - Reporting lifecycle and platform events back to the gateway.
 - Cleaning up runtime-owned resources.
 
+### Native prewarm status
+
+`PreparePrewarmSlot`, `ObservePrewarmSlot`, `BindPrewarmSlot` and
+`RetirePrewarmSlot` provide the opt-in native prewarm v1 transport on `openshell.v1.OpenShell`.
+They use `PrewarmSlotRequest.request_json` and
+`PrewarmSlotResponse.evidence_json`. The request is marked secret in protobuf
+metadata. Requests are bounded to 1 MiB of UTF-8, and the evidence limit is
+also 1 MiB. All four methods require a user/service platform-admin principal;
+observation requires `sandbox:read`, and mutations require `sandbox:write` when
+scope enforcement is enabled. Sandbox callbacks cannot invoke them.
+
+Only the in-tree Kubernetes backend implements these methods. New preparation
+and binding require an explicit approved immutable-artifact configuration.
+Disabled/other backends return `UNIMPLEMENTED`; unsupported placement, topology,
+artifacts and initial power states never fall back to cold creation.
+
+The warming boundary is **held init**, not an already-running supervisor. Prepare
+creates a real suspended AgentSandbox CR, one direct Pod and a blank exclusive
+RWO PVC. Its receipt contains their actual Kubernetes UIDs. There is no ordinary
+tenant Sandbox row, organization or credential in the idle runtime. The inert
+init container proves the claim is blank and waits for an identity-bound grant.
+Ordinary SA/mTLS mounts remain unavailable to that init container.
+
+A permanent generic database reservation grants one attempt to create the
+permanent fencing ConfigMap. ConfigMap resource-version CAS, not a process-local
+lock, linearizes allocation versus retirement. Every CREATE has a durable,
+single-attempt ticket. Retries discover the original resource and validate its
+UID/ownership; they never replay an issued CREATE. An ambiguous transport outcome,
+including a crash after claiming a ticket but before sending the request, is
+**quarantined** until that original creation becomes discoverable. There is no
+timeout that converts uncertainty into absence. Definitive API rejection permits
+retirement, but not recreation. Permanent reservations and ConfigMaps are never
+garbage-collected by these methods.
+
+Bind commits one allocation, stamps Pod/PVC organization ownership, and uses the
+ordinary workspace, provider-profile, credential-driver, policy and Sandbox
+validation/persistence paths with the reserved workload UUID. Credentials do not
+cross into privileged supervisor environment variables. One writer initializes
+the retained PVC before the main container's persistence subPath is resolved;
+see [Sandbox](sandbox.md). Only a matching successful init termination receipt
+permits adoption by the original Sandbox CR and publication of the startup grant.
+Observe can resume this handoff without replaying credential writes or requiring
+the original request secrets. Exact-allocation retries retain the original IDs.
+Before allocation CAS, all three resource UIDs must still match and none may be
+terminating. Reservations also freeze a hash of the native driver configuration.
+Preparation, binding and a pending adoption handoff reject profile drift rather
+than reinterpreting retained resources with a different supervisor, bootstrap,
+identity or storage configuration. Approval settings are checked separately;
+disabling new preparation does not invalidate retirement.
+
+Unused-slot retirement fences future creation first, settles outstanding
+creation tickets, removes the controller and Pod, observes compute absence, and
+only then removes storage. Allocated slots cannot return to inventory; ordinary
+tenant lifecycle owns them after handoff. A missing or changed retained resource
+is not a replacement warm hit. Partial or indeterminate provisioning produces an
+explicit error, never success-shaped absence.
+
+This implementation and native resource-fixture checks are not approved-artifact
+or customer readiness evidence. Bound results require a real supervisor session,
+loaded policy, current providers, and the agent's authenticated `/health/ready`
+probe before advertising a runtime endpoint. Do not enable a customer pool until
+the maintained agent/recipe/supervisor artifacts and actual create/open journey
+have been verified.
+
+Native v1 currently binds the gateway's legacy agent launch contract, not the
+current assistant lifecycle. The historical gateway adapter supplied this draft
+transport contract; it is not evidence of a current gateway consumer, allocator
+or product integration. Schema version 2 is reserved for the incompatible
+no-recipe assistant contract and returns `UNIMPLEMENTED` before any identity,
+resource or tenant mutation. Supporting it requires the ordinary assistant
+command, data-directory initialization and readiness mapping, not a fabricated
+legacy recipe. A current-assistant consumer cannot enable allocation until that
+native mapping and joined customer behavior are verified.
+
+The replacement gateway contract is the separate draft
+`globster-monday/globster-workload-gateway#250` at
+`bec528d52f978204f4acedd78a811823dae8e1a8`, not a published provider capability.
+Its `src/domain/prewarm-slots.ts` and `openapi/workload-gateway.v1.json` own the
+recipe-free assistant v2 contract; `docs/prewarm-native-contract.md` records the
+native handoff requirements. Preparation includes explicit immutable release
+`serviceVersion` and uses the `globster/prewarm-preparation/v2\0` hash domain.
+The native regression checks that settled golden vector and rejects its v2
+identity. It does not implement assistant initialization or approve an artifact.
+
+The supported native profile is shared-namespace, combined topology with
+explicit UID/GID 1000 and no provider SPIFFE mount. It is not compatible with the
+checked-in staging sidecar profile. That profile retains an operator-owned
+process bootstrap, separate network supervisor and pod `fsGroup`; its Secret
+projection permissions do not satisfy the gate's strict root-only mode-`0400`
+contract. Supporting it requires a reviewed private-grant transport and ordered
+dual-supervisor handoff that preserves the bootstrap, identity drop and original
+Pod/PVC. Do not remove `fsGroup`, relax grant permissions or switch staging to
+combined topology as a workaround. No compatible immutable lifecycle artifact
+or actual staging storage acceptance is established by this implementation.
+
 Drivers report **backend state only**. A driver snapshot with `Ready=True` means
 the underlying compute resource (container, pod, VM) is healthy and running —
 nothing more. Drivers must not gate on supervisor session state or hold
@@ -155,6 +250,11 @@ compute to zero, and VM retains its launch request and writable overlay beside
 a stop marker. Delete remains a separate operation that removes these
 resources.
 
+Kubernetes ignores conditions whose `observedGeneration` predates the current
+Sandbox generation. A retained `Suspended` condition must not override readiness
+after adoption or restart; older controllers without generation fields retain
+their existing condition behavior.
+
 On graceful gateway shutdown, persisted running intent for Docker, Podman, and
 VM is stopped through the shared `StopSandbox` RPC before any gateway-managed
 driver process exits. The gateway does not persist `Stopped` for this

architecture/sandbox.md

@@ -35,6 +35,95 @@ only when the set is already empty; any other outcome fails the spawn.
 6. It opens a supervisor session back to the gateway for connect, exec, file
    sync, config polling, and log push.
 
+### Trusted binding startup gate (experimental)
+
+On Linux, the internal `await-binding` command can hold supervisor startup
+behind a read-only projected, private root-owned binding grant. The grant
+must match the prepared slot, generation, pod UID and sandbox ID and carry
+a nonempty allocation ID. No supervisor, tenant credential acquisition or
+workload child starts while the grant is absent. Invalid grants fail closed;
+waiting is bounded. The accepted grant supplies the exact main-process argv
+and serialized workload environment before the gate replaces itself with
+the normal supervisor. Workload environment is applied to the child, not
+the privileged supervisor's environment.
+
+The gate can initialize the retained pod's workspace before supervisor exec.
+With `--require-workspace-initialization`, a grant without
+`workspace_initialization` is rejected. The plan contains `recipe_id`,
+`recipe_digest` and optional `initial_configuration_json`. The latter carries
+the gateway's exact canonical UTF-8 bytes, not a reserialized JSON value.
+The gate checks the complete configuration shape and its 64 KiB limit without
+logging its contents. Grants and initialization plans deliberately have no
+`Debug` implementation.
+
+For this path, the resource owner must supply:
+
+- An approved immutable recipe projected read-only at `/recipe/template`.
+  The initializer verifies the read-only filesystem; it does not authenticate
+  the image or infer its identity from files.
+- The exclusive writable claim at `/workspace`, writable by UID/GID 1000.
+  No other pod or sidecar may write this claim during initialization.
+- A separate pod-private writable state volume at
+  `/run/openshell-binding-state`, owned by root and not group- or world-writable.
+  The gate creates a mode-`0700` `private` child directory. The state volume
+  must not be the customer claim or the read-only grant projection.
+
+The root gate holds a nonblocking exclusive writer lock in the private directory.
+It starts its own fixed-function `initialize-bound-workspace` subprocess with
+UID/GID 1000, an empty environment and a bounded stdin plan. No shell, recipe
+script or tenant executable runs. The initializer accepts regular files and
+directories, rejects symlinks, special files and reserved state, and copies into
+a private staging directory. It creates the `workspace` and
+`.globster-openshell-persist` directories, writes the private initial configuration,
+and publishes `<workload UUID>/` using a Linux no-replace rename after syncing
+files and directories. Copying is bounded to 100,000 entries, 1 GiB and 64 levels.
+Recipes outside those limits are unsupported and fail closed.
+
+The `.globster-recipe.json` marker matches the cold materializer's field names,
+including the SHA-256 digest of the exact initial-configuration bytes. A retry
+accepts only the same recipe and configuration marker, preserves existing tenant
+data, and can discard an abandoned private staging directory. It never adopts an
+unmarked workspace or overwrites a different marker. The pod-private lock spans
+initialization and supervisor exec. After the child succeeds, the gate atomically
+publishes `private/initialized.json`, containing the exact binding identity
+and recipe marker but no configuration or credentials. A receipt for another
+allocation is rejected. Initialization failure or deadline expiry prevents
+supervisor startup. The receipt proves initialization only, not a live supervisor
+session or customer readiness.
+
+The gate is not an allocator or authorization API. Its resource owner must
+commit single-use ownership with resource-side fencing and establish policy,
+credentials and ordinary sandbox ownership before publishing the grant.
+The pod-private writer lock does not fence competing gateway replicas or prevent
+another Kubernetes pod from mounting an RWO claim. RWO alone is not a one-writer
+guarantee.
+
+The opt-in native Kubernetes lifecycle uses this gate in a **held init container**
+and again as the main container's startup gate. Preparation verifies a blank
+claim (allowing only an empty filesystem-created `lost+found`) and publishes a
+root-private prepared receipt in the Pod's emptyDir. It does not create tenant
+directories or start the privileged supervisor. The init container has only the
+binary, read-only recipe, blank PVC, private state and optional private init grant;
+it does not mount the ordinary service-account token or mTLS credentials.
+
+At bind, the same init container makes the blank mount writable by UID/GID 1000,
+runs the fixed unprivileged writer, and exits with a sanitized identity- and
+grant-digest-bound termination receipt. The main gate accepts the same private
+initialized receipt and does not write the PVC a second time. Native allocation
+is independently fenced by permanent DB/Kubernetes records; the pod-local lock
+is only the sequential writer guard. See [Compute Runtimes](compute-runtimes.md)
+for creation quarantine, adoption and retirement.
+
+The ordinary Kubernetes launch mounts
+`<workload UUID>/.globster-openshell-persist` at `/sandbox/persist` using a PVC
+`subPath`. Kubelet resolves it before starting the main container, but **after**
+the held init completes. Only the init container mounts the root claim during
+initialization; it does not request that not-yet-existing subPath. This preserves
+the exact ordinary launch paths without changing Pod/PVC identities or adding
+another writer Pod. A restarted/changed physical Pod cannot consume the original
+UID-bound grant. The init gate's wait is bounded to 24 hours; expired or failed
+idle capacity must be retired, never restarted/recycled as a fresh slot.
+
 ## Isolation Layers
 
 OpenShell uses overlapping controls rather than a single sandbox primitive:

crates/openshell-cli/tests/ensure_providers_integration.rs

@@ -81,6 +81,34 @@ impl TestOpenShell {
 
 #[tonic::async_trait]
 impl OpenShell for TestOpenShell {
+    async fn prepare_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn observe_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn bind_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn retire_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
     async fn report_main_process_exit(
         &self,
         _request: tonic::Request<openshell_core::proto::ReportMainProcessExitRequest>,

crates/openshell-cli/tests/mtls_integration.rs

@@ -34,6 +34,34 @@ struct TestOpenShell;
 
 #[tonic::async_trait]
 impl OpenShell for TestOpenShell {
+    async fn prepare_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn observe_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn bind_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn retire_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
     async fn report_main_process_exit(
         &self,
         _request: tonic::Request<openshell_core::proto::ReportMainProcessExitRequest>,

crates/openshell-cli/tests/provider_commands_integration.rs

@@ -99,6 +99,34 @@ struct TestOpenShell {
 
 #[tonic::async_trait]
 impl OpenShell for TestOpenShell {
+    async fn prepare_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn observe_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn bind_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
+    async fn retire_prewarm_slot(
+        &self,
+        _request: tonic::Request<openshell_core::proto::PrewarmSlotRequest>,
+    ) -> Result<Response<openshell_core::proto::PrewarmSlotResponse>, Status> {
+        Err(Status::unimplemented("unused"))
+    }
+
     async fn report_main_process_exit(
         &self,
         _request: tonic::Request<openshell_core::proto::ReportMainProcessExitRequest>,