Add DatabricksDurableRuntime for Lakebase-backed execution

[shivam5]

What did you change, and why?

Change: Add a standalone, transport-neutral DatabricksDurableRuntime and LakebaseDurabilityStore. The runtime persists one idempotent JSON request per execution_id, heartbeats the active attempt, atomically claims queued or stale work, passes the exact persisted request to a caller-owned executor, and stores the terminal JSON response in the same Lakebase row.

The public contract supports background submit, blocking invoke, status/result get, and wait. Reusing an ID with the same request returns the cached response; reusing it with a different request raises a conflict. DurableExecutionContext exposes the attempt number so an agent harness can decide how to use its own session/checkpointer history during recovery.

This PR also adds a complete OpenAI Agents SDK Databricks App example. It keeps the existing PR-review agent and AsyncDatabricksSession, replaces the application-owned durability package with DatabricksDurableRuntime, and shows the thin FastAPI adapter for blocking invocation, background submission, status/result retrieval, exact-request replay, and conflict handling. The example README separates runtime, HTTP adapter, agent, and SDK-session responsibilities; live observations record the Lakebase state across each scenario.

Why: The OpenAI SDK agent experiment showed that heartbeat/restart and request/response recovery are reusable runtime concerns, while SDK session history must remain harness-managed. This separates those layers and makes the required HTTP/idempotency wiring explicit without coupling durability to MLflow, FastAPI, or an agent SDK.

LongRunningAgentServer is unchanged in this PR. Making it extend DatabricksDurableRuntime is intentionally deferred.

How do you know it works?

Automated testing: Added 20 focused tests covering request conflicts, cached responses, background and blocking execution, cross-process polling, stale recovery with the exact request, attempt metadata, failure state, timeouts that leave execution running, store schema/JSON persistence, and subclass wiring. Repository-wide Ruff format/check passes. Repository-wide ty check has only three pre-existing warnings outside this change, and the example passes a dedicated ty check against MLflow 3.10.1 and the current OpenAI integration.

Live testing: Built this PR as local wheels and deployed the example to Databricks Apps with Lakebase. Verified:

• blocking invocation completed with HTTP 200 and persisted request, response, and 30 SDK messages;
• an exact retry returned the byte-identical cached response in 0.258 seconds, while a changed request returned HTTP 409;
• killing a blocking client left execution running, and the completed response was retrievable later;
• a background request returned HTTP 202, survived a real App stop/start, was reclaimed as attempt 2 from a stale heartbeat, reopened the existing SDK history, completed, and returned a persisted response;
• post-recovery retrieval and exact retry returned byte-identical responses; and
• package installation completed through the workspace package repository with no proxy failure.

The commands, IDs, timing, message counts, request/response state, and recovery-history evidence are documented in examples/openai-sdk-agent/OBSERVATIONS.md. Built the wheel and verified all durable_runtime modules and its README are packaged.

[elainemwang]

can we expose a durable_runtime.run_foreground method for streaming?

[shivam5]

Questions to answer:

1. Do we want to expose this as a server in sdk, or as a library so that any server can use this. Trades off flexibility with ease of use.
2. Do we want a general JSON request/response or ResponsesAgentRequest/Response
3. How is streaming handled? Do we need to store stream events to provide durability for streaming?

[shivam5]

Questions to answer:

1. Do we want to expose this as a server in sdk, or as a library so that any server can use this. Trades off flexibility with ease of use.
2. Do we want a general JSON request/response or ResponsesAgentRequest/Response
3. How is streaming handled? Do we need to store stream events to provide durability for streaming?

#467 and #468 to help answer 1 and 2

[shivam5]

For resumption, should we expose a handler/hook such @onResume or should @invoke be responsible for starting/resuming as well?

[shivam5]

We want to go with server approach, and handle durable streaming, and want to generalize longRunningAgentServer - moved PR to #467