Sign workflow run mail with the run principal's key

apps/hub/src/server.ts

@@ -306,7 +306,6 @@ export async function createHubServer({
     materializeMailTriggeredRunGrants: createMailTriggeredRunGrantsMaterializer(
       {
         db,
-        principalKeyStore,
         runTokenStore,
         grantStoreFor: createGrantStore,
         // The registry createHubApp uses, so a conditioned operator
@@ -393,7 +392,6 @@ export async function createHubServer({
   createHubSessionOrchestrator({
     events: sidecarRouter.events,
     router: sidecarRouter,
-    db,
     eventCollectors,
   });
 

apps/sidecar/src/index.ts

@@ -547,9 +547,9 @@ const orchestrator = createSidecarOrchestrator({
   // Report the sidecar's cached rotatable senders on every (re)connect so the
   // hub re-resolves each current key and re-pushes it, catching a rotation that
   // landed while the sidecar was disconnected. The cache owns the "only user
-  // senders rotate" filter (a run sender's key is the immutable
-  // workflow_run.public_key); the link stays source-opaque and reports whatever
-  // this returns.
+  // senders rotate" filter (a run sender's key reaches the cache with that
+  // run's mail, on its grants barrier); the link stays source-opaque and
+  // reports whatever this returns.
   getCachedSenderAddresses: () => senderKeyCache.rotatableAddresses(),
   // When the Hub's `welcome` routes a held incarnation again, re-drive any
   // workflow-run pack the disconnect cancelled. The Hub has routed the

bin/backfill-principal-keys.ts

@@ -4,9 +4,9 @@
 // per-principal-key feature.
 //
 // Run ONCE as the last step of the deploy that introduces per-principal keys,
-// AFTER the key-minting hub is live. It is safe against a live hub: nothing
-// reads principal keys yet, the hub only mints for newly-created principals,
-// and the active-key unique index blocks a double key. Idempotent -- a
+// AFTER the key-minting hub is live. It is safe against a live hub: the hub
+// only mints for newly-created principals, and the active-key unique index
+// blocks a double key. It keys only user principals. Idempotent -- a
 // principal that already holds an active key is skipped -- so it is safe to
 // re-run or resume after a partial failure. On a database where every principal
 // was created after the feature landed (e.g. local dev), it does nothing.

docs/API.md

@@ -585,7 +585,7 @@ Body: SendMessage
 202: unknown -- Trigger accepted for delivery
 400: ErrorResponse -- Attachment validation error. Each variant carries a structured code (too_many_attachments, oversize_attachment, disallowed_mime_type, invalid_attachment_name, malformed_base64, oversize_total) with the offending index and limits. A malformed request body that fails SendMessage validation returns the generic error shape instead.
 404: ErrorResponse -- Run not found
-409: ErrorResponse -- Run address is not routable, its allocation is no longer active, the run is stopping or terminal, or (`deployer_unavailable`) the definition has invoker-sourced grant requirements and the deployer is not recorded or no longer active
+409: ErrorResponse -- Run address is not routable, its allocation is no longer active, the run is stopping or terminal, (`deployer_unavailable`) the definition has invoker-sourced grant requirements and the deployer is not recorded or no longer active, or (`workflow_run_principal_inactive`) the first trigger finds the run principal suspended or deactivated
 413: ErrorResponse -- Request body exceeds the maximum allowed size
 503: ErrorResponse -- Run trigger substrate unavailable
 
@@ -712,7 +712,7 @@ Body: SendMessage
 202: unknown -- Trigger accepted for delivery
 400: ErrorResponse -- Attachment validation error. Each variant carries a structured code (too_many_attachments, oversize_attachment, disallowed_mime_type, invalid_attachment_name, malformed_base64, oversize_total) with the offending index and limits. A malformed request body that fails SendMessage validation returns the generic error shape instead.
 404: ErrorResponse -- Workflow deployment not found
-409: ErrorResponse -- Deployment address is not routable, its allocation is no longer active, its top-level run is stopping or terminal, or (`deployer_unavailable`) the definition has invoker-sourced grant requirements and the deployer is not recorded or no longer active
+409: ErrorResponse -- Deployment address is not routable, its allocation is no longer active, its top-level run is stopping or terminal, (`deployer_unavailable`) the definition has invoker-sourced grant requirements and the deployer is not recorded or no longer active, or (`workflow_run_principal_inactive`) the first trigger finds the run principal suspended or deactivated
 413: ErrorResponse -- Request body exceeds the maximum allowed size
 503: ErrorResponse -- Durable workflow dispatch unavailable
 

docs/IMPLEMENTATION.md

@@ -369,7 +369,7 @@ The reconciler owns acknowledged undeploys for releasing and replacing deploymen
 
 A non-retryable destroy rejection moves a `releasing` or `replacing` allocation to terminal `destroy_failed`. The Hub records the failure, keeps a cleanup-only binding, and stops reconciliation. It fails remaining live runs and unsettled dispatches unless deployment removal is confirmed and cancellation is pending; then lifecycle reconciliation settles cancellation and dispatches after recovering accepted history. The allocation retains its sidecar id and external reference for operator recovery, but cannot dispatch work or create a replacement. Its copy occupies an active slot until removal or retained storage is confirmed. Reconnect preserves inventory access without an untracked hello undeploy or a new cleanup attempt. Cleanup must preserve other work on a shared sidecar. The deployment list reports `destroy_failed` explicitly.
 
-Every replacement increments `generation`. The generation is persisted before the provisioner is called and appears in the ensure/destroy request and the sidecar's authenticated identity. Provisioners must make ensure/destroy idempotent and reject a generation older than one they have already observed. The Hub separately records `ensure_accepted_generation`; registration may occur while capacity is converging, but readiness and routing require the exact generation to be accepted. A reconnecting sidecar takes a deployment's route back only once the deployment's first deploy completed: its anchor's key is committed and no initialization is in flight. Releasing a deployment on a sidecar that hosts other work does not destroy the sidecar, so the Hub undeploys a copy whose deploy is still uncertain rather than routing to it. A copy whose run has ended takes no route either. The lifecycle service stops and transfers it under the [retained-capacity policy](SIDECAR_PLACEMENT.md#active-and-retained-capacity). A deferred failure whose run is still live remains ineligible for reclaim and is undeployed.
+Every replacement increments `generation`. The generation is persisted before the provisioner is called and appears in the ensure/destroy request and the sidecar's authenticated identity. Provisioners must make ensure/destroy idempotent and reject a generation older than one they have already observed. The Hub separately records `ensure_accepted_generation`; registration may occur while capacity is converging, but readiness and routing require the exact generation to be accepted. A reconnecting sidecar takes a deployment's route back only once the deployment's first deploy completed: its run principal holds the deployment's key and no initialization is in flight. Releasing a deployment on a sidecar that hosts other work does not destroy the sidecar, so the Hub undeploys a copy whose deploy is still uncertain rather than routing to it. A copy whose run has ended takes no route either. The lifecycle service stops and transfers it under the [retained-capacity policy](SIDECAR_PLACEMENT.md#active-and-retained-capacity). A deferred failure whose run is still live remains ineligible for reclaim and is undeployed.
 
 The Hub never persists raw sidecar bearer tokens; it stores only their hashes. A sidecar connection grants cleanup while it hosts a `releasing`, `replacing`, or `destroy_failed` allocation, including a permanent provider rejection. A disconnect timeout resumes cleanup on reconnect; an exhausted retry budget remains terminal. A cleanup binding permits registration, reporting existing retained-copy proofs, and cleanup. It grants no readiness, routing, reclaim, or permission to request new retention. A sidecar with neither current work nor cleanup obligations stays connected, idle, after the Hub asks it to undeploy what its hello reports, so its provisioner can place new work on it. If the Hub restarts during a probe, startup marks its temporary capacity for release; background reconciliation performs and retries destruction. If it restarts after binding an allocation generation but before recording ensure acceptance, it treats that outcome as uncertain, fences the generation, destroys it idempotently, and mints a replacement identity. On Hub startup, interrupted probes are durably marked for cleanup before active allocation fences are rebuilt; no provider call is awaited during this initialization.
 
@@ -427,13 +427,13 @@ The allocation reconciler uses database leases and retry timestamps rather than
 
 Before sending an allocated deploy, the Hub commits an `initialization_lease_id` on the allocation under the current accepted generation and unexpired reconciliation lease. Restoration and preparation happen before this reservation. The router rechecks cancellation and the exact socket after the reservation commits. A crash between reservation and send can conservatively require cleanup even if no frame reached the worker. A proven unsent failure of a confirmed reservation clears its own initialization marker, even after cancellation or loss of its reconciliation lease, provided the exact marker, allocation status, anchor identity, and accepted generation still match. If reservation confirmation is lost, the caller cannot prove its reservation and leaves the marker for conservative cleanup. Cancellation, disconnects, retries, and lease changes otherwise preserve the marker; a stale clear cannot remove a newer attempt or affect another generation.
 
-An allocated supervisor's deploy acknowledgement does not publish its public key immediately. The Hub publishes the key and credential references and clears the exact initialization marker in one allocation-locked transaction, rechecking lease expiry before committing the mutations. A later reconciliation with an outstanding marker fences and releases the generation before checking connectivity or restoring history. That cleanup requires the exact observed marker to remain present when the generation advances. A concurrent proven-unsent rollback instead requeues ordinary initialization checks; absence of the marker alone does not publish readiness. Readiness and dispatch require the marker to be absent, while bootstrap Git uploads remain permitted. Release consumes the marker atomically with the generation advance. A publication error whose transaction actually committed must not release the completed worker: cleanup revalidates the marker and key after acquiring the allocation lock, and completed initialization re-enters the normal callback to finish dispatch requeue and readiness. An active allocated supervisor without a completed initialization key remains uncertain and must be fenced and released, never redeployed in place.
+An allocated supervisor's deploy acknowledgement does not publish its public key immediately. The Hub records the key as the active key of the deployment's run principal, stores the credential references, and clears the exact initialization marker in one allocation-locked transaction, rechecking lease expiry before committing the mutations. A later reconciliation with an outstanding marker fences and releases the generation before checking connectivity or restoring history. That cleanup requires the exact observed marker to remain present when the generation advances. A concurrent proven-unsent rollback instead requeues ordinary initialization checks; absence of the marker alone does not publish readiness. Readiness and dispatch require the marker to be absent, while bootstrap Git uploads remain permitted. Release consumes the marker atomically with the generation advance. A publication error whose transaction actually committed must not release the completed worker: cleanup revalidates the marker and key after acquiring the allocation lock, and completed initialization re-enters the normal callback to finish dispatch requeue and readiness. An active allocated supervisor without a completed initialization key remains uncertain and must be fenced and released, never redeployed in place.
 
 Mail sent before initialization publishes its signing key is deferred under the allocation generation and reconciliation attempt. Cancellation leaves that mail pending because the key transaction may already have committed. A confirmed unsent rollback fails queued mail. A new reconciliation resolves the previous attempt's marker and key before waiting for a connection. A completed initialization releases its queued mail and a rolled-back one fails it; an outstanding marker leaves mail pending until fencing confirms failure, because the prior caller may still complete a proven-unsent rollback. A lost rollback response is resolved on a later claim, which finds neither a marker nor a key. A durable generation advance also clears an unfinished sender attempt, including when the cleanup transaction's response was lost. Late callbacks can settle only their own attempt. Deferred delivery carries the confirmed signing key so a newer attempt cannot capture the queued message or change its verification key. The Hub completes asynchronous delivery preparation before sending the sender key and mail consecutively on the same socket. The existing deferred-mail TTL remains the fallback if recovery cannot resolve the outcome.
 
 Hub-originated triggers have a separate durable replay path in `workflow_run_dispatch`:
 
-- Mail stores the immutable MIME bytes and the stable run's canonical step grants in the same transaction that reserves the run principal/grants. The exact generation receives `run.grants` then `mail.inbound` on one FIFO WebSocket. The hub loads the run's committed grants each time it sends or redelivers the trigger, and the `run.grants` frame carries them. So a decision that commits after the enqueue rides the trigger. A run whose principal is gone gets no grants. When the lookup is not wired or its load faults, the frame carries the grants the trigger last carried: the stored snapshot on a first send, or the last loaded grants on a redelivery. Later trigger occurrences reuse that snapshot as their fallback; a grants-only reservation with no `RunStarted` event is still eligible for its first delivery.
+- Mail stores the immutable MIME bytes and the stable run's canonical step grants in the same transaction that commits the run's grants and links its run principal. The exact generation receives `run.grants` then `mail.inbound` on one FIFO WebSocket. The hub loads the run's committed grants each time it sends or redelivers the trigger, and the `run.grants` frame carries them. So a decision that commits after the enqueue rides the trigger. A run with no linked run principal gets no grants. When the lookup is not wired or its load faults, the frame carries the grants the trigger last carried: the stored snapshot on a first send, or the last loaded grants on a redelivery. Later trigger occurrences reuse that snapshot as their fallback; a grants-only reservation with no `RunStarted` event is still eligible for its first delivery.
 - A sidecar `mail.inbound.ack` proves only that the local inbox accepted the message. The Hub retains it until an accepted workflow-run Git pack shows either the matching `RunStarted` event or a claim-check entry under `addresses/*/consumed/`. An explicit supervisor rejection fails the dispatch instead of settling it.
 - Signals store a validated `signal.deliver` frame and are redelivered with the same `signalId`. Approval decisions for provisioned deployments enqueue that frame in the same transaction that resolves the approval. There is no sidecar signal ack; the Hub settles the row only after an accepted pack contains the corresponding `SignalReceived` event.
 - When a generation becomes ready, pending and acknowledged dispatches are requeued. Abandoned attempts remain stopped. Mail claim checks and signal ids make replay idempotent across a worker loss at any point in delivery.

docs/INBOUND_MAIL_POLICY.md

@@ -18,8 +18,9 @@ not receiving mail an author expected is the symptom this document exists for.
 The signature gate is the second of two gates. Before it, the hub admits or
 refuses the sender itself against the workflow's `mailAccept` rules. A
 workflow that declares no `mailAccept` admits a sender only through an
-operator grant on the run principal, so it refuses the first mail that fires
-it. See _The sender gate runs first_ below.
+operator grant on the run principal, and such a grant cannot admit the first
+mail that fires the run, so it refuses that mail. See _The sender gate runs
+first_ below.
 
 ## Declaring a policy
 
@@ -105,7 +106,8 @@ sender if its principal is gone, which deletes the file. A hub-side resolve that
 itself faults changes nothing, so the entry stays unreadable until a later
 reconnect. A run sender is repaired by the next grants barrier that co-delivers
 its key instead, because the hub never re-resolves a run sender on reconnect: a
-run's key is immutable.
+run's key changes only when its sidecar is replaced, and the run's next mail
+carries the replacement's key.
 
 ## Admission weighs every finding, not one
 
@@ -173,9 +175,11 @@ Each relation admits a sender that stands in that relation to the deployment:
 
 The default is deny. A workflow that declares no `mailAccept`, or declares no
 rule that matches the sender, refuses the mail unless an operator grant on the
-run principal admits the sender. The first mail that fires a run has no run
-principal yet, so no operator grant can admit it. An unknown sender is always
-refused. The operator approves the rules at deploy time, as the `mail.accept:`
+run principal admits the sender. The run principal exists from deploy, but an
+operator grant on it cannot admit the first mail that fires the run: that mail
+needs a declared rule. An operator `mail.accept` deny on the run principal
+does refuse the first mail. An unknown sender is always refused. The
+operator approves the rules at deploy time, as the `mail.accept:`
 strings of the capability walk. `defineWorkflow` rejects a grant requirement in
 the `mail.accept` namespace, so `mailAccept` is the only way to author a rule.
 
@@ -261,11 +265,11 @@ spec of its deployment:
   `mail.accept:tenant:<id>` grant for its own tenant. The backfill adds no
   invoker row and does not know the parties it mailed, so an existing run
   admits only senders in its own tenant.
-- A deployment that has not fired has no run principal, so it gets only the
-  legacy rule and no tenant grant. After its first fire, a mail-trigger
-  deployment admits its invoker and the parties it mails. A sending deployment
-  admits only the parties it mails, so mail from the HTTP caller that started
-  the run is refused. Neither admits other senders in its own tenant, so each
+- A deployment that had not fired had no run principal when the migration
+  ran, so it got only the legacy rule and no tenant grant. After its first
+  fire, a mail-trigger deployment admits its invoker and the parties it
+  mails. A sending deployment admits only the parties it mails, so mail from
+  the HTTP caller that started the run is refused. Neither admits other senders in its own tenant, so each
   admits fewer senders than a deployment that fired before the migration.
 - A known gap remains. A live run that mailed a party outside its tenant before
   the migration has no `correspondent` grant for that party, so the reply from