Add DatabricksDurableRuntime for Lakebase-backed execution

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Improvements
 - databricks-openai: `DatabricksOpenAI` and `AsyncDatabricksOpenAI` clients now follow HTTP redirects by default, configurable via the new `follow_redirects` parameter (#445)
+- databricks-ai-bridge: Add a transport-neutral `DatabricksDurableRuntime` with Lakebase request/response persistence and stale-attempt recovery
 
 ### Bug Fixes
 - databricks-ai-bridge: Genie now returns the full answer text aggregated from all text attachments (#432)

README.md

@@ -24,6 +24,20 @@ For frameworks without dedicated integration packages:
 pip install databricks-ai-bridge
 ```
 
+## Durable Runtime
+
+[`DatabricksDurableRuntime`](./src/databricks_ai_bridge/durable_runtime/README.md)
+adds Lakebase-backed request/response persistence, heartbeat detection, and
+stale-attempt recovery around a caller-owned async handler. The handler remains
+responsible for agent sessions and checkpoints.
+
+See the [OpenAI Agents SDK App](./examples/openai-sdk-agent/README.md) for a
+complete FastAPI, background polling, SDK session, and Databricks Apps example.
+
+```sh
+pip install 'databricks-ai-bridge[memory]'
+```
+
 ### Install from source
 
 With https:

examples/openai-sdk-agent/OBSERVATIONS.md

@@ -0,0 +1,155 @@
+# Live Test Observations
+
+Tests ran on 2026-08-19 against the example in this PR, using an isolated App
+name and schemas so they did not affect the earlier experiment.
+
+- App: `open-ai-sdk-runtime`
+- URL: `https://open-ai-sdk-runtime-1653573648247579.staging.aws.databricksapps.com`
+- Lakebase branch: `projects/shivam-openai-agent-on-apps/branches/agent-app`
+- Database: `databricks_postgres`
+- Runtime schema: `openai_sdk_runtime_durability`
+- SDK session schema: `openai_sdk_runtime_sessions`
+
+The PR packages were built as local wheels and included in the test deployment
+because `DatabricksDurableRuntime` was not released yet. All remaining packages
+installed through the workspace package repository, and the App finished in
+`ACTIVE / RUNNING`; no package-proxy failure occurred.
+
+## Common request and database query
+
+Each test supplied a stable ID in `custom_inputs.session_id`:
+
+```bash
+curl -sS --max-time 900 \
+  -o response.json \
+  -w 'http_code=%{http_code}\ntime_total=%{time_total}\n' \
+  -X POST "$APP_URL/responses" \
+  -H "Authorization: Bearer $APP_TOKEN" \
+  -H 'Content-Type: application/json' \
+  --data @request.json
+```
+
+The following query joined runtime state with SDK history:
+
+```sql
+SELECT e.status, e.attempt, e.heartbeat_at,
+       e.request, e.response, count(m.id) AS sdk_messages
+FROM openai_sdk_runtime_durability.executions e
+LEFT JOIN openai_sdk_runtime_sessions.agent_messages m
+  ON m.session_id = e.execution_id
+WHERE e.execution_id = :execution_id
+GROUP BY e.execution_id;
+```
+
+## Test 1: blocking happy path
+
+Execution: `runtime-happy-20260819T221342Z`
+
+```text
+http_code=200
+time_total=74.347104
+curl_exit=0
+```
+
+Lakebase after completion:
+
+```text
+status=COMPLETED  attempt=1  sdk_messages=30
+request=persisted  response=persisted
+```
+
+The runtime row contained the normalized Responses request and final response.
+The SDK table independently contained the model and tool history.
+
+## Test 2: cache and conflict
+
+Posting the exact Test 1 request and ID again returned in 0.258 seconds. The
+original and cached response files had the same SHA-256:
+
+```text
+726b28b5a378e0b6e176a9894deeceb07f28ed8c7865ba21b81fd3324a98a23e
+```
+
+Changing only the input while retaining the ID returned:
+
+```text
+http_code=409
+execution 'runtime-happy-20260819T221342Z' was already accepted with a different request
+```
+
+This verifies exact-request idempotency rather than ID-only response reuse.
+
+## Test 3: blocking client disconnect
+
+Execution: `runtime-disconnect-20260819T221458Z`
+
+The client process was terminated after Lakebase showed `ACTIVE`, attempt `1`,
+and one SDK message:
+
+```text
+curl_exit=143
+```
+
+The runtime task continued without the HTTP client. Its final state was:
+
+```text
+status=COMPLETED  attempt=1  sdk_messages=36
+request=persisted  response=persisted
+```
+
+`GET /responses/runtime-disconnect-20260819T221458Z` then returned the completed
+response with HTTP `200`. Unlike the earlier custom supervisor experiment, a
+disconnected client can retrieve the persisted result by its stable ID.
+
+## Test 4: background stop/start recovery
+
+Execution: `runtime-crash-20260819T222154Z`
+
+The background request returned immediately:
+
+```text
+http_code=202
+time_total=0.223404
+status=in_progress
+```
+
+The App was stopped after the first attempt had persisted SDK history:
+
+```text
+before stop: status=ACTIVE  attempt=1  sdk_messages=5  response=NULL
+after stop:  status=ACTIVE  attempt=1  sdk_messages=7  response=NULL
+```
+
+While compute was stopped, retrieval returned HTTP `503`. The runtime row and
+SDK history remained in Lakebase. After `databricks apps start`, the scanner
+claimed the stale row:
+
+```text
+after restart: status=ACTIVE     attempt=2  sdk_messages=12  response=NULL
+final:         status=COMPLETED  attempt=2  sdk_messages=31  response=persisted
+```
+
+SDK message `112` was the fixed recovery note. It contained neither the PR URL
+nor the old temporary workspace. The following recovered tool calls nevertheless
+used `/tmp/openai-sdk-agent-0oxbysef/repo` and checked out PR 459. Those values
+were present only in messages `105` through `111`, proving that attempt `2`
+reopened and used the persisted SDK session history. The old pod-local directory
+was gone, so the agent recreated it.
+
+After completion:
+
+- `GET /responses/runtime-crash-20260819T222154Z` returned HTTP `200`.
+- Reposting the exact background request returned the cached response in 0.221 seconds.
+- Retrieved and cached responses had the same SHA-256:
+  `4929b07e0931a5a487efac00f4d13910b4ccf2d15e3edb470f670775eb34f5bc`.
+
+## Result
+
+The live tests verify the intended separation:
+
+- `DatabricksDurableRuntime` persists request, response, status, attempt, and heartbeat.
+- `AsyncDatabricksSession` persists replayable agent and tool history.
+- Recovery replays the persisted request to the executor, while this executor
+  intentionally resumes the agent with only the SDK session and recovery note.
+- A stable execution ID lets a client poll after a disconnect or App restart.
+- Pod-local files and in-flight tool processes are not durable.

examples/openai-sdk-agent/README.md

@@ -0,0 +1,114 @@
+# Durable OpenAI Agents SDK App
+
+This example is the PR-review agent from the custom-runtime experiment, with
+its application-owned durability package replaced by `DatabricksDurableRuntime`.
+The agent loop in `review_agent.py` and the OpenAI Agents SDK session in
+`sessions.py` remain application concerns.
+
+See [Live Test Observations](./OBSERVATIONS.md) for blocking, cache/conflict,
+client-disconnect, and real App stop/start recovery results.
+
+## Responsibilities
+
+```text
+client
+  -> FastAPI adapter (app.py)
+       -> DatabricksDurableRuntime
+            -> Lakebase: openai_sdk_agent_durability.executions
+            -> executor (execute_durable_review)
+                 -> OpenAI Agents SDK + tools
+                 -> Lakebase: openai_sdk_agent_sessions.agent_messages
+```
+
+`DatabricksDurableRuntime` owns request/response persistence, exact-request
+idempotency, heartbeats, stale-attempt claims, and process-start recovery. The
+executor owns the SDK session and recovery behavior. On attempt 1 it starts the
+review from the request. On attempt 2 or later it reopens the same SDK session
+and supplies only the fixed recovery note; it does not reconstruct an agent
+prompt from the durability request.
+
+This example intentionally allows one durable request per SDK session, so
+`custom_inputs.session_id` is also the runtime `execution_id`. A multi-turn
+application should use a separate execution ID for each invocation and keep its
+conversation or session ID in the persisted request.
+
+## HTTP contract
+
+- `POST /responses` and `POST /invocations` run in blocking mode by default.
+- Set `background: true` to receive `202` with an ID and poll
+  `GET /responses/{execution_id}`.
+- Repeating the same normalized request and ID returns the cached response.
+- Reusing an ID with a different request returns `409 Conflict`.
+- `background` and `stream` are transport fields and are not persisted.
+  Streaming is rejected because this example does not implement it.
+
+Clients should supply a stable `custom_inputs.session_id`. A generated ID can
+be returned to a connected client, but a blocking client that loses its
+connection before receiving that ID cannot later identify the execution.
+
+Example background request:
+
+```bash
+SESSION_ID="review-$(date -u +%Y%m%dT%H%M%SZ)"
+
+curl -X POST "$APP_URL/responses" \
+  -H "Authorization: Bearer $APP_TOKEN" \
+  -H 'Content-Type: application/json' \
+  -d "{
+    \"background\": true,
+    \"input\": [{\"role\": \"user\", \"content\": \"Execute the complete PR CUJ.\"}],
+    \"custom_inputs\": {
+      \"session_id\": \"$SESSION_ID\",
+      \"pr_url\": \"https://github.com/databricks/databricks-ai-bridge/pull/459\",
+      \"minimum_minutes\": 0
+    }
+  }"
+
+curl "$APP_URL/responses/$SESSION_ID" \
+  -H "Authorization: Bearer $APP_TOKEN"
+```
+
+## Lakebase state
+
+Both stores use the App's `postgres` resource but separate schemas:
+
+| Owner | Schema and table | Persisted state |
+| --- | --- | --- |
+| Runtime | `openai_sdk_agent_durability.executions` | execution ID, status, attempt, heartbeat, normalized request, final response |
+| OpenAI Agents SDK | `openai_sdk_agent_sessions.agent_messages` | replayable user, assistant, tool-call, and tool-output items |
+
+The runtime provides at-least-once recovery. Pod-local files and in-flight tool
+processes do not survive a crash, and tools must tolerate retries.
+
+## Deploy
+
+Install from the repository checkout while developing this unreleased runtime:
+
+```bash
+uv venv
+uv pip install -e '../..[memory]' -e '../../integrations/openai[memory]'
+uv pip install 'openai-agents>=0.19.4,<0.20' 'mcp>=1.29.0,<2' \
+  'mlflow>=3.10.1' 'fastapi>=0.129.0' 'uvicorn>=0.41.0'
+```
+
+For Databricks Apps, configure one Lakebase branch/database and one secret, then
+deploy with an explicitly selected profile:
+
+```bash
+databricks bundle deploy -t dev --profile <PROFILE> \
+  --var="lakebase_branch=projects/<project>/branches/<branch>" \
+  --var="lakebase_database=projects/<project>/branches/<branch>/databases/<database>" \
+  --var="openai_secret_scope=<scope>" \
+  --var="openai_secret_key=<key>"
+
+databricks bundle run open_ai_sdk_agent -t dev --profile <PROFILE> \
+  --var="lakebase_branch=projects/<project>/branches/<branch>" \
+  --var="lakebase_database=projects/<project>/branches/<branch>/databases/<database>" \
+  --var="openai_secret_scope=<scope>" \
+  --var="openai_secret_key=<key>"
+```
+
+After the runtime is released, the App build installs `requirements.txt`
+directly. When deploying this PR before release, replace the
+`databricks-ai-bridge` requirement with an installable wheel or Git ref that
+contains `DatabricksDurableRuntime`.