Scale the agentic runtime to distributed workers and messaging

[jeremymanning]

Origin

Deferred from the MVP proposed as a fresh realization of #485.

Goal

Scale the single-machine durable runtime into a multi-process and eventually distributed worker system without changing its event-sourced execution semantics.

Scope

• Replace local worker leases/notifications with database- and queue-backed primitives.
• Define worker identity, heartbeats, lease expiry, orphan recovery, backpressure, and fair scheduling.
• Support direct typed addressing in addition to parent/child routing, with authorization and hop/fan-out limits.
• Add cancellation and interruption at deterministic checkpoints.
• Detect deadlock/livelock and expose operator diagnostics.
• Load-test hundreds of concurrent logical nodes and multiple simultaneous runs.

Prerequisite

The MVP must first establish replay, idempotency, message-boundary, and state-transition semantics on one machine. Distribution must preserve those contracts rather than define new ones accidentally.

Acceptance criteria

• Multi-worker execution produces the same projections as the single-process reference runner.
• Worker death, duplicate delivery, delayed messages, and lease theft are covered by fault-injection tests.
• Backpressure and budget limits remain enforceable under concurrency.
• A run can be cancelled and resumed without duplicate completed effects.
• Throughput/latency curves and operational limits are documented.