feat(agent-runtime): durable sandbox allocation, teardown-gated deletion, and Tensorlake integration

[gabrik]

Summary

This PR implements durable sandbox allocation, teardown-gated deletion, and Tensorlake integration for the Temper agent runtime.

Durable sandbox allocation before bootstrap

Split provisioning into two governed stages:

Provision → provision_sandbox → SandboxAllocated (Provisioning)
SandboxAllocated → bootstrap_sandbox → SandboxReady (Thinking)

The provider sandbox ID, URL, and actual provider are persisted immediately after creation succeeds, before TemperFS bootstrap or private-repository clone can fail. A failed bootstrap now retains the provider resource identity needed for later teardown.

Teardown-gated deletion lifecycle (ADR-0167)

Added DELETE /v1/agent-runs/{id} with a governed state-machine lifecycle:

Completed | Failed | Cancelled
  → RequestDeletion → Deleting
  → delete_sandbox (sandbox_destroyer WASM)
  → DeletionTeardownSucceeded → Deleted

DeletionFailed → RetryDeletion → Deleting

Key properties:

• Deletion is strict: missing provider credentials or a missing sandbox ID for a remote provider produces DeletionFailed, not Deleted
• Provider 404 responses are treated as successful idempotent teardown
• Cancellation remains best-effort with an empty callback (no invalid second Cancel transition from terminal Cancelled)
• DELETE on an already-Deleted run Cedar-authorizes the caller without creating an outgoing transition from the terminal state
• Concurrent DELETE race is resolved by rereading authoritative actor state only for known stale-dispatch errors, never masking Cedar denials

Tensorlake integration

• Clone private repos using x-access-token + GIT_ASKPASS (command-scoped env injection, token never in URL or argv)
• Public repos clone without a token
• Readiness polling with bounded delay between attempts
• Provider 404 treated as already-absent during teardown
• E2B backward compatibility: historical runs with provider=local but e2b.app URLs are still destroyed correctly

Dead provider abstraction removed

Removed the unused SandboxProvider trait, RunSpec, SandboxHandle, ProviderType, and related types from crates/temper-sandbox/src/provider.rs. These had no tracked consumers; runtime behavior is implemented directly in the WASM modules.

Live E2E validation

A full Tensorlake agent run succeeded against the private fixture repository gabrik/agent-runtime-fixture:

{
  "status": "Completed",
  "error": null,
  "result": "## Summary\n\nI've successfully fixed the failing test! ...\n### The Fix\nChanged the `divide()` function to correctly perform division:\n- **Before**: `return a * b` (multiplication)\n- **After**: `return a / b` (division)\n### Test Results\n- **Before**: 1 failed, 3 passed\n- **After**: 4 passed ✅"
}

The agent:

1. Provisioned a Tensorlake sandbox
2. Cloned the private fixture repo using the stored GitHub PAT
3. Identified and fixed the divide bug (a * b → a / b)
4. Ran pytest (4 passed)
5. Committed the change
6. Completed successfully

How to test end-to-end

Prerequisites

• Rust with wasm32-unknown-unknown target
• Anthropic API key
• Tensorlake API key
• GitHub fine-grained PAT with Contents:Read on gabrik/agent-runtime-fixture

Steps

# 1. Build WASM modules
cd os-apps/temper-agent/wasm && ./build.sh && cd -

# 2. Start the server
export ANTHROPIC_API_KEY="sk-ant-..."
export TEMPER_API_KEY="temper-local-dev-0001"
export TENSORLAKE_API_KEY="tlk_..."
export GITHUB_TOKEN="github_pat_..."

cargo run -p temper-cli -- serve --port 3000 --app temper-agent --no-observe

# 3. Store secrets (in a separate terminal)
curl -X PUT \
  "http://localhost:3000/api/tenants/default/secrets/tensorlake_api_key" \
  -H "authorization: Bearer $TEMPER_API_KEY" \
  -H "content-type: application/json" \
  -d "{\"value\":\"$TENSORLAKE_API_KEY\"}"

curl -X PUT \
  "http://localhost:3000/api/tenants/default/secrets/github_token" \
  -H "authorization: Bearer $TEMPER_API_KEY" \
  -H "content-type: application/json" \
  -d "{\"value\":\"$GITHUB_TOKEN\"}"

# 4. Create an agent run
RUN_ID=$(curl -sf -X POST http://localhost:3000/v1/agent-runs \
  -H "content-type: application/json" \
  -H "x-tenant-id: default" \
  -H "authorization: Bearer $TEMPER_API_KEY" \
  -d '{
    "prompt": "Fix the failing test in tests/test_calculator.py. The divide function has a bug — it returns a*b instead of a/b. Fix it, run the tests, show the git diff.",
    "sandbox_provider": "tensorlake",
    "repo_url": "https://github.com/gabrik/agent-runtime-fixture",
    "repo_ref": "main",
    "tools": ["read", "write", "edit", "bash"],
    "max_turns": "15"
  }' | python3 -c "import sys,json; print(json.load(sys.stdin)['run_id'])")

# 5. Poll until Completed
curl http://localhost:3000/v1/agent-runs/$RUN_ID \
  -H "x-tenant-id: default" \
  -H "authorization: Bearer $TEMPER_API_KEY"

# 6. Delete the run and its sandbox
curl -X DELETE http://localhost:3000/v1/agent-runs/$RUN_ID \
  -H "x-tenant-id: default" \
  -H "authorization: Bearer $TEMPER_API_KEY"

# 7. Confirm it's gone
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/v1/agent-runs/$RUN_ID \
  -H "x-tenant-id: default" \
  -H "authorization: Bearer $TEMPER_API_KEY"
# Should return 404

See os-apps/temper-agent/AGENT_RUNTIME_POC_GUIDE.md for the full guide including local sandbox setup, steering, cancellation, and troubleshooting.

Validation

Check	Result
IOA L0–L3 cascade	✅ all passed
Migration differential	✅ 3 passed
Destroyer unit tests	✅ 3 passed
Provisioner unit tests	✅ 2 passed
Handler tests	✅ 4 passed
cargo check -p temper-sandbox	✅ passed
cargo check -p temper-server	✅ passed
All WASM modules built	✅ 13 modules
git diff --check	✅ clean
Live Tensorlake E2E	✅ Completed, 4 tests passed

Files changed

• docs/adrs/0167-agent-run-deletion-lifecycle.md — new ADR
• os-apps/temper-agent/specs/temper_agent.ioa.toml — allocation/bootstrap split, deletion lifecycle states/actions
• os-apps/temper-agent/policies/agent.cedar — Cedar permits for new actions
• os-apps/temper-agent/wasm/sandbox_provisioner/src/lib.rs — allocation/bootstrap split, Tensorlake clone, readiness polling
• os-apps/temper-agent/wasm/sandbox_destroyer/src/lib.rs — strict deletion, fail-closed, 404 success, empty cancel callback
• os-apps/temper-agent/wasm/tool_runner/src/lib.rs — Tensorlake API key injection
• crates/temper-server/src/agent_runtime/handlers.rs — DELETE route, race resolution, Cedar authorization
• crates/temper-server/src/agent_runtime/models.rs — DeleteRunResponse model
• crates/temper-spec/tests/migration_differential.rs — allowed config keys for new triggers
• crates/temper-sandbox/src/provider.rs — removed (dead abstraction)
• crates/temper-sandbox/src/lib.rs — removed provider module
• crates/temper-sandbox/Cargo.toml — removed async-trait dependency
• os-apps/temper-agent/AGENT_RUNTIME_POC_GUIDE.md — updated with bearer auth and Tensorlake instructions

Greptile Summary

This PR adds durable two-stage sandbox provisioning, teardown-gated run deletion, and Tensorlake-backed sandbox execution.

• Persists remote sandbox identity before bootstrap so failed runs remain teardown-capable.
• Adds Cedar-governed deletion and retry transitions with idempotent provider teardown.
• Adds Tensorlake provisioning, repository cloning, tool execution, and workspace restoration support.
• Refactors the agent-runtime HTTP handlers into endpoint-specific modules and uses deterministic run IDs.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
os-apps/temper-agent/specs/temper_agent.ioa.toml	Splits allocation from bootstrap and adds governed deletion, teardown-success, failure, and retry transitions.
os-apps/temper-agent/wasm/sandbox_provisioner/src/lib.rs	Adds durable provider allocation and Tensorlake bootstrap, readiness, and repository-cloning behavior.
os-apps/temper-agent/wasm/sandbox_destroyer/src/lib.rs	Implements strict provider teardown with idempotent not-found handling and lifecycle callbacks.
crates/temper-server/src/agent_runtime/handlers/delete.rs	Adds authenticated, Cedar-governed asynchronous deletion with stale-dispatch race handling.
crates/temper-server/src/agent_runtime/handlers/create.rs	Uses deterministic run IDs and dispatches the application Configure and Provision stages.
os-apps/temper-agent/wasm/tool_runner/src/lib.rs	Routes sandbox tool operations through the new Tensorlake process and filesystem APIs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Created -->|Provision| Provisioning
  Provisioning -->|provision_sandbox| Allocated[SandboxAllocated]
  Allocated -->|bootstrap_sandbox| Thinking[SandboxReady / Thinking]
  Thinking --> Completed
  Thinking --> Failed
  Thinking --> Cancelled
  Completed -->|RequestDeletion| Deleting
  Failed -->|RequestDeletion| Deleting
  Cancelled -->|RequestDeletion| Deleting
  Deleting -->|DeletionTeardownSucceeded| Deleted
  Deleting -->|teardown failure| DeletionFailed
  DeletionFailed -->|RetryDeletion| Deleting

Loading

_{Reviews (3): Last reviewed commit: "chore: restore tracked codex spec file" | Re-trigger Greptile}

Context used (4)

• Context used - CLAUDE.md (source)
• Knowledge Base — Temper Agent os-app
• Knowledge Base — temper-server
• Knowledge Base — Temper WASM Runtime

[rita-aga]

CI Failure: Readability Ratchet

The Integrity & DST Patterns job is failing with two ratchet violations:

FAIL PROD_FILES_GT500: baseline=84 current=86
FAIL ALLOW_DEAD_CODE_COUNT: baseline=16 current=19

Fix #1 — Split handlers.rs (PROD_FILES_GT500)

crates/temper-server/src/agent_runtime/handlers.rs exceeds 500 lines. Split it by logical concern into sub-modules:

crates/temper-server/src/agent_runtime/handlers/
├── mod.rs       (re-exports, router registration)
├── allocation.rs
├── teardown.rs
└── tensorlake.rs

Each file must be under 500 lines. Verify with:

bash scripts/readability-ratchet.sh check .ci/readability-baseline.env

Fix #2 — Remove #[allow(dead_code)] (ALLOW_DEAD_CODE_COUNT)

There are 4 new #[allow(dead_code)] annotations added (diff lines 1965, 1993, 2001, 2018). Baseline is 16, current is 19 (3 net new). For each one, either:

• Use the field/function (remove the annotation), OR
• Delete the dead code entirely if it's POC scaffolding not ready to ship

Do not snapshot the baseline to hide the problem — fix the code.

Run the ratchet check after both fixes to confirm both show OK before pushing.

[rita-aga]

CI: Compile & Lint — Clippy failures (5 errors)

All errors are in crates/temper-server/src/agent_runtime/handlers.rs. Straightforward fixes:

---

1. result_large_err — line 46

fn ... -> Result<(TenantId, AuthenticatedRequestContext), Response>

The Err variant (axum::http::Response<Body>) is ≥128 bytes. Fix: box it.

-> Result<(TenantId, AuthenticatedRequestContext), Box<Response>>

(Update all Err(...)) callsites in the function to Err(Box::new(...)).)

---

2. needless_borrow — lines 124 and 147

// before
return error_response(StatusCode::INTERNAL_SERVER_ERROR, &e);
// after
return error_response(StatusCode::INTERNAL_SERVER_ERROR, e);

(Two occurrences — lines 124 and 147.)

---

3. collapsible_if — lines 126 and 149

// before (line 126)
if let Ok(resp) = &configure_result {
    if !resp.success {
        let msg = resp.error.as_deref().unwrap_or("configure failed");
        return error_response(StatusCode::BAD_REQUEST, msg);
    }
}
// after
if let Ok(resp) = &configure_result
    && !resp.success {
    let msg = resp.error.as_deref().unwrap_or("configure failed");
    return error_response(StatusCode::BAD_REQUEST, msg);
}

Same pattern at line 149 (provision_result / "provision failed").

---

Run cargo clippy --workspace --all-targets -- -D warnings locally to verify clean before pushing.