Skip to content

feat(prewarm): single-use retained-pod lifecycle (task 34.01.03) - #32

Draft
avivka wants to merge 2 commits into
mainfrom
aviv/feature/34.01.03-single-use-prewarm-binding
Draft

avivka wants to merge 2 commits into
mainfrom
aviv/feature/34.01.03-single-use-prewarm-binding

Conversation

@avivka

@avivka avivka commented Oct 12, 2026 •

Copy link
Copy Markdown

Task-ID: 34.01.03

Summary

Publish the complete existing native single-use prewarm implementation, not only the earlier startup gate. This remains a draft prerequisite, not task completion, an approved artifact, assistant-v2 support, a staging rollout or customer-readiness acceptance. All pools remain disabled.

Related Issue

Workspace task 34.01.03, directly assigned by the owner. Issues are disabled in this maintained fork. Historical source-built evidence remains in open Workspace NVIDIA#1808; it is not current published-artifact or joined customer acceptance.

Changes

  • Add platform-authorized, secret-marked, bounded prepare/observe/bind/retire RPCs and generated Go bindings. Other/disabled backends fail closed without ordinary creation.
  • Persist permanent reservation/fencing records and one-attempt CREATE tickets. Resource-version CAS commits one allocation; uncertain CREATE outcomes quarantine rather than replay or expire into false absence.
  • Retain exact Sandbox/Pod/exclusive-PVC receipts. Check all three UIDs and termination before allocation and tenant handoff; retire compute before storage and never recycle a bound slot.
  • Implement held-init, retained-pod single-writer recipe initialization, normal workspace/provider/credential/policy persistence, controller adoption and identity-bound private launch grants. Ordinary lifecycle/access is blocked while initial binding is pending.
  • Freeze the native driver profile and require explicit UID/GID 1000. Changed supervisor/bootstrap/identity/storage cannot reinterpret an existing receipt. Fix stale Kubernetes observed-generation handling and preserve disabled standalone-driver defaults.
  • Reserve and explicitly reject assistant schema version 2 before identity/resource/tenant processing. Verify the exact recipe-free preparation/hash fixture settled in gateway draft docs: Updated tutorial formatting NVIDIA/OpenShell#250 at bec528d52f978204f4acedd78a811823dae8e1a8; do not manufacture a legacy recipe.
  • Update compute/sandbox architecture, gateway configuration, compute-driver support and cluster-debugging documentation with operational fencing, disabled examples and exact compatibility limits.

Testing

Current local checks on October 25, 2026:

  • cargo test -p openshell-server --features bundled-z3 --lib prewarm: 14 passed, 1 explicitly ignored live resource fixture. Covers one-shot tickets, retained UID/termination checks, profile drift, unsupported profiles/v2, settled gateway golden hash, tenant context/fingerprints, authorization/bounds and pending ordinary lifecycle.
  • cargo test -p openshell-server --features bundled-z3 --lib auth::descriptor_authz: 9 passed (one test overlaps the prewarm filter).
  • cargo test -p openshell-driver-kubernetes --lib: 225 passed.
  • cargo clippy -p openshell-server -p openshell-driver-kubernetes --lib --tests -- -D warnings: passed after fixing the standalone-driver constructor and a new fixture's semicolon.
  • cargo check -p openshell-sdk -p openshell-cli --tests: passed, including changed RPC mocks.
  • go test ./... in sdk/go: passed. Canonical Go protobuf regeneration with the repository's pinned generators matches; buf lint passed.
  • cargo fmt --all -- --check, git diff --check and the dependency-free SPDX check (878 files) passed.

Native tests use the repository-supported bundled solver, the existing macOS 15.4 SDK and checksum-verified session-local CMake 3.31.6. The initial default-SDK link and missing-system-Z3 link failed; an exact Z3 wheel restore also failed at its CDN. These were recovered for the successful checks above, not counted as passes. No validation toolchain/cache was added to source.

The canonical bounded mise ... pre-commit retry failed, not passed: downloading pinned ty==0.0.14 failed with macOS socket error 57. The independent SPDX check passed; full pre-commit remains unverified. The earlier session's protobuf download failure is historical.

No current Linux gate/initializer execution, live Kubernetes fixture, staging proof or joined customer journey was run. Docker's local daemon is unavailable and there is no authorized cloud budget. Historical October 20 source-built combined/offline-model/empty-recipe/local-path proof retained Sandbox/Pod/PVC UIDs and reached readiness in 10.6 seconds; it does not qualify a published artifact or current assistant/customer acceptance.

Compatibility and next owner action

Native v1 still implements the legacy recipe/agent-harness command and initialization contract. The current assistant gateway draft uses incompatible recipe-free v2; its adapter rejects native-v1 evidence before binding tenant intent/credentials. Generic native allocation/receipt/retirement fencing remains reusable, but there is no working current-assistant native integration yet.

The actual checked-in staging profile is sidecar, with an operator bootstrap wrapper, separate network/process supervisors, shared process namespace and fsGroup=1000. Its Secret projection does not satisfy the current strict root-owned mode-0400 grant. Supporting it requires reviewed private-grant transport, bootstrap/identity-drop preservation, ordered dual-supervisor release and assistant initialization/launch/readiness mapping. Do not remove fsGroup, relax grants, silently switch staging topology or treat a supplied hash/accepted flag as approval.

Next owner: review this scoped native draft and settle/authorize that sidecar/private-grant plus assistant-v2 mapping; then obtain supported Linux CI and explicitly authorized exact immutable-artifact/storage/retained-resource and joined customer acceptance. No allocator restoration, release/tag/image publication, live mutation, promotion, auto-merge or enablement was performed.

Checklist

  • Complete local native implementation preserved and published with scoped fixes
  • Targeted native/driver/auth/client/protobuf checks and operational documentation
  • Conventional commit, DCO signoff and Copilot provenance
  • Full canonical pre-commit and supported Linux execution
  • Native assistant-v2 and actual staging sidecar compatibility
  • Approved immutable artifacts, actual storage-mode and joined customer-readiness acceptance

Keep draft. Do not merge or mark the task done on this evidence.

Foundation for task 34.01.03; resource-side prepare/bind/retire remains unimplemented and warm allocation remains disabled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3ed4c100-7879-4cbd-a298-d6b0f6aadeba
Signed-off-by: Your Name <avivka8@gmail.com>
@avivka

avivka commented Oct 18, 2026

Copy link
Copy Markdown
Author

Gateway task 34.01.02 now has a composed native adapter in globster-monday/globster-workload-gateway#238 (7125a74), with 83 focused local checks and green gateway CI. Its docs/prewarm-native-contract.md records the proposed v1 prepare/observe/bind/retire RPC and evidence handoff. This is a consumer contract, not a native implementation or live warm-start proof.

The owner approved finishing the gateway outcome and then continuing 34.01.03 on October 18. Actual native lifecycle/fencing, retained-resource initialization and immutable artifact publication remain incomplete. The October 14 upstream mutation-scope comment on NVIDIA#3508 must be considered against the maintained fork version; the upstream pool is still unmerged.

Current live verification is blocked: the prior Azure sandbox authorization expired October 13; this host has no project VPN profile, staging Kubernetes contexts or visible Globster staging subscriptions. The full local platform check also stops on missing mise. No release, staging promotion, Azure recreation or warm enablement has been triggered. Keep this PR draft; neither HTTP health nor fixture success proves normal customer create/open on retained compute/PVC.

@avivka

avivka commented Oct 19, 2026

Copy link
Copy Markdown
Author

Owner decisions confirmed on October 19: use held-init prewarm (retain the pod/PVC, initialize tenant state with one writer inside that pod, then start the supervisor after the ordinary persistence subPath exists) and durably quarantine indeterminate one-shot CREATE outcomes rather than retrying CREATE or claiming retirement. Stable native sandbox identity and actual resource receipts remain required.

The existing isolated dev VM has been resumed without additional Azure resources or extending its October 20 15:04:55 UTC expiry. The exact AgentSandbox 0.5.3 controller is Ready, the native gateway baseline builds, and 12 focused gate/initializer tests passed. None of this proves actual prepare/bind, a retained-resource warm hit, or normal customer create/open. Native lifecycle implementation and verification are continuing; keep this PR draft and allocation disabled. No release or staging promotion has been authorized.

Publish native prepare/observe/bind/retire with durable one-shot CREATE tickets, resource-side allocation fencing, exact retained identities and held-init single-writer initialization. Preserve ordinary tenant lifecycle and fail closed on changed native profiles or unsupported assistant-v2 and staging sidecar contracts. Keep the capability disabled and document acceptance gaps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 78c54bc7-92e0-4d2a-9d48-3a49da0cf391
Signed-off-by: Aviv Kabesa <avivka8@gmail.com>
@avivka avivka changed the title feat(sandbox): trusted binding startup gate (task 34.01.03) feat(prewarm): single-use retained-pod lifecycle (task 34.01.03) Oct 25, 2026
@avivka

avivka commented Oct 25, 2026

Copy link
Copy Markdown
Author

@NaveCohenMonday @DorHason @orsharon7 Full native lifecycle is pushed at 7399147: prepare/observe/bind/retire, durable one-shot CREATE/CAS fences, exact retained identities, held-init initialization and profile freezing. Current native14/auth9/driver225 tests, strict Clippy, client test-target compilation and Go/protobuf checks passed; full pre-commit remains blocked by pinned dependency download. Updated docs/body distinguish historical proof. Still draft: assistant-v2 and actual staging sidecar private-grant/bootstrap compatibility are unsupported; no immutable artifact, live/customer acceptance, release or pool enablement is claimed.

Comment on lines +190 to +191
"OpenShell held-init ready slot={} generation={} pod={} sandbox={}",
args.slot_id, args.generation, args.pod_uid, args.sandbox_id
Comment on lines +310 to +311
"NATIVE_RESOURCE_BIND_PROOF workload={} receipt={} initialized=true retained=true customerReady=false",
input["workloadId"], receipt
Comment on lines +348 to +349
"NATIVE_RESOURCE_RETIRE_PROOF workload={} receipt={} computeAbsent=true storageAbsent=true",
unused["workloadId"], unused_receipt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants