Repository navigation
Chat delivers room mail through Interchange's run-trigger route (CL-7490) - #680
Merged
TheGreatAxios merged 64 commits intoSep 6, 2026
Conversation
The native workflow deploy route now requires sourceOfferingIds/ defaultSourceOfferingId resolved against the tenant's own catalog, rather than a raw provider/baseURL/apiKey source chain. Update workflowDeployBody accordingly, and extract the zero-cost noop catalog chain smoke-webhook.test.ts already built (an anthropic- plugin provider pointed at the hub's noop-inference endpoint) into a shared seedNoopCatalogOffering harness helper. Every deploying e2e suite now seeds or resolves a real catalog offering before deploying: walking-skeleton keeps a never-real placeholder host (assertNeverRealProvider still guards it), chat resolves the offering seedCatalog already planted rather than seeding a second chain, and heartbeat/workbench-digest/smoke-webhook seed the noop chain via the shared helper. cl-6324-launch-proof's section-mode deploy now reuses the offering id the proof already resolved when it narrowed the bench catalog to its pinned model.
seedTenant currently deploys every default workflow with a bare provider/baseURL/apiKey source, but the hub's deploy route now requires sourceOfferingIds/defaultSourceOfferingId resolved from the tenant's catalog. Update the seeding, onboarding, and hub config test suites to expect the new offering-id based deploy body and provider/model-only ModelSource shape ahead of the implementation change.
ensureDeployment posted a bare {provider, baseURL, apiKey, model}
source, but the hub's deploy route requires sourceOfferingIds and a
defaultSourceOfferingId resolved from the tenant's catalog, so seeding
a connect flow's "assistant" workflow 400'd. seedTenant now resolves
the tenant's real catalog offerings (sorted by priority, lowest wins
the default) for a normal workflow, and plants a dedicated noop
catalog offering for a modelSource-pinned one (heartbeat), pointed at
the hub's own noop-inference endpoint. ModelSource shrinks to a bare
provider/model pair — the baseURL/apiKey it used to carry only mattered
for the old deploy body, and callers that still need a key for seeding
a catalog (system-seed's root tenant, config's env-key auto-plant) keep
it on their own local type instead.
Another workflowDeployBody caller CI's typecheck job caught: this stack seeds one catalog offering per real inference target already, so track each one's id and pass it through to the deploy body instead of the raw provider/baseURL/apiKey shape.
…' into cl-7471-generation-zero
The allocation service's initial sidecar_allocation and workflow_probe rows start at generation 0 (@intx/db enforces generation >= 0, not > 0), so the first ensure() for a freshly allocated sidecar must be accepted at generation 0. These tests cover that acceptance, later generations, non-integer/negative rejection, and that stale-generation fencing still rejects an older generation once a newer one has been observed.
@intx/db's sidecar_allocation and workflow_probe tables both check generation >= 0 and the allocation service records the first row for an allocation at generation 0, but validateEnsureRequest() and validateDestroyRequest() in packages/sandbox-sidecar rejected anything that wasn't strictly positive, and the on-disk fencing schema in state-store.ts required generation > 0. The first ensure() for any newly allocated sidecar was rejected with 'generation must be a positive integer', failing every deploy. Accept generation >= 0 in both validators and the state-store schema; stale-generation fencing in observeEnsure()/observeDestroy() already compares against the stored generation rather than assuming positivity, so it is unaffected.
…t-against-interchange-main-before-the' into cl-7461-local
… signature Covers the new GET .../catalog/resolved-offerings route (denied grant, and both directly-owned and ancestor-inherited offerings sorted by priority) and updates seeding/onboarding test fixtures to the route's response shape ahead of the implementation change.
…owned ones resolveRealSourceOfferingIds listed only offerings a tenant owns directly, missing anything inherited from an ancestor tenant — the hub's own workflow deployer already resolves that wider, ancestor-inheriting view via listVisibleOfferings. Adds GET /api/tenants/:tenantId/catalog/resolved-offerings (mounted from apps/hub, implemented in @corbits/inference-catalog so the deployer and this route share one source of truth) and points seed at it instead. Also drops the two long-dead unused parameters from onboarding's modelSourceFor, and exports ensureNoopCatalogOffering from @corbits/seeding for the e2e harness to reuse.
…' into cl-7472-sidecar-ws
At the interchange pin this branch stacks on (692c3106), the hub authenticates a sidecar's register/reconnect frame only against a sidecar row that resolves to a live sidecar_allocation (or workflow_probe) row. Upstream dropped the older "shared" credential scope this repo's e2e harness used to provision a single, standalone sidecar identity for, so a provisionSidecar-only row now connects and is rejected with "invalid token" in a loop rather than ever dialing in. This test pins that negative so a future re-vendor that silently restores (or relies on) the old scope surfaces here first.
Every e2e suite dial-in relied on a single, harness-started sidecar process registered against a bare `provisionSidecar` identity row. At pin 692c3106, the hub only accepts a register/reconnect frame from a sidecar whose identity resolves to a live sidecar_allocation (or workflow_probe) row; a standalone identity with neither now fails the handshake and gets rejected with "invalid token" every ~3s, forever (chat mint 500s, deploys never dial in — CI run 34042907014). The hub's own process-provisioner already spawns a dedicated, correctly-allocated sidecar process per deployment on demand — the same path production already takes for a real workflow run or a minted agent chat. Every affected suite (walking skeleton, chat, heartbeat, workbench-digest, smoke-chat, smoke-webhook, greeting-delivery, channel-reconnect, local-rip, and the isolation suite) now lets that happen instead of dialing in its own sidecar: the existing 502/500 retry-until-ready loops already cover the wait, and diagnostics that used to read the one sidecar process's output now read the hub's — the process-provisioner backend spawns each sidecar with its stdio inherited from the hub, so the hub's own captured output already carries every spawned sidecar's log lines. `provisionSidecar`/`startSidecar` stay in the harness (documented as non-authenticating at this pin) for the handful of standalone proof scripts not yet migrated off a directly-controlled sidecar process.
The e2e job's hub/sidecar output was never captured, so the CI run that surfaced the sidecar disconnect loop (34042907014) had no sidecar-side log to read past the transcript. Point E2E_LOG_DIR at a runner-temp directory and upload it on every run, pass or fail.
…t-against-interchange-main-before-the' into cl-7474-cross-tenant-launch # Conflicts: # scripts/e2e/harness.ts # scripts/e2e/heartbeat.test.ts # scripts/e2e/smoke-webhook.test.ts # scripts/e2e/walking-skeleton.test.ts # scripts/e2e/workbench-digest.test.ts
readBindingByAddress and resolveLiveByStableId took an address or
stable id and resolved whichever workbench_launch row matched it,
regardless of which tenant that row belonged to. Every caller that
already knows its own tenant (sendMail's requireLive, and
refreshAgentInstanceFromDefinition) now passes it through and gets
undefined instead of another tenant's launch if the two ever
disagree. The one legitimate tenant-agnostic caller — the event
stream handler discovering which tenant an inbound address even
belongs to before it has one to check against — keeps working by
simply omitting the new optional parameter.
Also fixes offeringDigest: it joined catalog offering ids with a
NUL byte, which is not valid UTF-8 to Postgres the moment a tenant's
catalog carries more than one visible offering ("invalid byte
sequence for encoding \"UTF8\": 0x00"), permanently failing the
workbench_launch insert for every such launch. This is what was
failing 4 of scripts/e2e/chat.test.ts's cases under per-allocation
sidecars, where seedCatalog now produces more than one offering.
…t-against-interchange-main-before-the' into cl-7476-run-turns # Conflicts: # scripts/e2e/harness.test.ts # scripts/e2e/smoke-webhook.test.ts # scripts/e2e/walking-skeleton.test.ts
…o cl-7476-run-turns
…t-against-interchange-main-before-the' into cl-7479-rebase # Conflicts: # packages/agent-directory/src/one-shot-prompt.ts # packages/chat/src/platform-adapter.ts # packages/chat/src/send-run-mail.ts # packages/webhook-triggers/src/launch.ts # packages/webhook-triggers/test/launch.test.ts # packages/workflows/src/deliver-when-routable.ts # packages/workflows/src/index.ts # packages/workflows/src/launch/agent-session.ts # packages/workflows/test/agent-session.test.ts
Replaces the post-send recordAgentSessionForRun (CL-7477/CL-7479) with one idempotent ensureRunSession, called lazily at the hub seams that actually see a run become mail-routable: the mailbox persist wrapper and the event-collector dispatch path, plus chat's pre-send mail dispatch. Pre-creating the run's principal to dodge the timing isn't an option -- Interchange's grant materialization treats a conflict on that insert as grants already committed and throws.
…t-against-interchange-main-before-the' into cl-7486-wake-binding # Conflicts: # packages/agent-directory/src/one-shot-prompt.ts # packages/chat/src/platform-adapter.ts # packages/chat/src/send-run-mail.ts # packages/webhook-triggers/src/launch.ts # packages/webhook-triggers/test/launch.test.ts # packages/workflows/src/index.ts # packages/workflows/src/launch/agent-session.ts # packages/workflows/test/agent-session.test.ts
…uplicate key Merging cl-7107's repin/re-vendor audit into this branch left packages/workflows/src/index.ts re-exporting the pre-CL-7481 agent-session API (recordAgentSessionForRun) that no longer exists, its matching test file testing that same stale API, and a duplicated eventCollectors key in apps/hub/src/index.ts's webhook-trigger launch wiring. Restores the current exports (recordAgentSessionAtProvision, ensureRunSession) and their test, and drops the duplicate object key.
sendRunMailWithReclaimRetry retried every attempt against the address it started with. Proves the reproduction of the production failure: a participant already one relaunch generation in that dies again mid-retry must have its next retry chase the run that replaced it, not keep hammering the address that just died.
sendRunMailWithReclaimRetry retried every delivery attempt, and woke, against the address it captured at the top of the loop. A wake in between retries can relaunch the run (its dead-run branch repoints the room's stable participant at a fresh address without telling this loop), and a participant already one relaunch generation in that dies again mid-retry leaves this loop holding an address that is no longer either the stable participant id or the current run id — invisible to readBindingByAddressAnyTenant. The next wake for it then threw "No workbench_launch binding" instead of finding the run that replaced it, which is what chat e2e's mention-fanout and invite-drives-a-turn cases were hitting. Re-resolves the live address (and its session) off the stable participant id after each wake, so the retry loop always chases whichever run is actually current. Also hardens publishCorbitsToolsRegistry's create-then-list-on-conflict ensure: two overlapping seed runs racing the same tenant's package registry asset can still throw a raw asset_tenant_kind_name violation if the substrate's own 409 mapping is bypassed; the ensure now also recognizes that violation by name and resolves the winner's row instead of throwing.
…486-wake-binding # Conflicts: # packages/webhook-triggers/test/launch.test.ts
…thReclaimRetry Drives the CL-7486 reclaim-retry test off routing-table transitions instead of fixed backoff timing, and updates the CL-7214 coalescing test's fake sidecar router to model a redeploy actually registering.
…ving up on first-turn delivery sendRunMailWithReclaimRetry used to exhaust a ~7.75s backoff and report the invited agent unreachable while its sidecar was still booting and registering with the hub (CL-7476) -- fast enough to make the fan-out relaunch a perfectly healthy run. It now shares deliverWhenRoutable with the webhook trigger's ingress delivery: wake once on the first unreachable failure (still chasing a mid-wait relaunch by re-resolving the live address inside the retry's send closure), then wait on the hub's routing table up to the same boot-and-register budget (DEFAULT_ROUTABLE_DEADLINE_MS, now exported from @corbits/workflows) before retrying exactly once more.
…s missing Runs deployed straight through Interchange's own deployments route never pass through a Workbench launcher, so recordAgentSessionAtProvision never runs for them and no agent_session row exists for the run. By the time persist/dispatch call ensureRunSession the run is anchored with a principal, so the seam has everything needed to create the row itself instead of treating the missing row as a bug.
isBeyondWake's completed-run test assumed status alone marks a run terminal; a deployment settled "completed" before its first trigger is still booting, not dead, so the predicate now needs a run that took a turn to treat "completed" as terminal. Update platform-adapter's lifecycle/ensureAwake/sendMail coverage to match: a live run that is merely not yet routable is waited on rather than redeployed, so those tests now exercise a genuinely terminal run instead of an offline one.
A provisioned anchor stays "deployed" until its first trigger, so a run mid-boot is indistinguishable from an offline one by routability alone. wakeByAddress's fallback branch treated the two the same and redeployed either — minting a fresh run and address out from under the sidecar's own in-flight registration, which supersedes its allocation and restarts the boot. Repeated on every wake attempt, this produced an endless reconnect loop with no run ever finishing its boot. wakeByAddress now only relaunches a run whose status is genuinely terminal; a live run that is not yet routable is left alone; the caller's own routable wait (deliverWhenRoutable, via sendRunMailWithReclaimRetry) is what actually resolves it once the sidecar finishes registering. isBeyondWake also no longer treats every "completed" status as terminal outright: a deployment settled complete before its first trigger ever landed is still indistinguishable from one mid-boot, so it is only terminal once the store can tell it took a turn (a principal was ever anchored to it).
…t-against-interchange-main-before-the' into cl-7491-chat-trigger # Conflicts: # packages/chat/test/platform-adapter.test.ts
Retargets platform-adapter's fan-out assertions from a raw sessionService.sendUserMessage call to the new RunTriggerClient seam: the fake now records anchorRunId-keyed trigger calls and mints a messageId the same shape the real route does, so the reclaim-retry and threading assertions exercise the seam production code will actually use.
…490) A provisioned run is deployed to its sidecar lazily by its first native trigger (vendor/intx/hub-api's createWorkflowRunTrigger: anchor -> grants/principal -> durable dispatch's agent.deploy -> deliver mail). Chat sent via sessionService.sendUserMessage, which delivers straight over the sidecar's ordinary channel and never reaches that durable-dispatch branch, so an invited run never triggered natively never got its first agent.deploy and stayed permanently unreachable. createWorkflowRunTrigger itself is not exported from @intx/hub-api (it backs the route, not a published seam), so chat now calls the route itself: a new RunTriggerClient (run-trigger-client.ts) posts to POST /api/tenants/:tenantId/workflows/runs/:runId/mail in-process against the hub's own composed Hono app. The route authenticates through @intx/hub-api's ordinary session contract, which chat cannot present a browser cookie for (most sends have no inbound HTTP request in scope at all), so the client signs a short-lived internal token naming the principal to authenticate as (run-trigger-internal-auth.ts); apps/hub's own getSession implementation -- a seam @intx/hub-api documents as pluggable -- verifies that token before falling through to the real cookie-session path. Neither side touches vendor/intx. send-run-mail.ts now delivers through the trigger client and uses its returned messageId for session_mail correlation, still assembling a local signed MIME copy (with @intx/mime/@intx/crypto directly) purely for this workbench's own history and attachment blob reads -- the trigger route returns no raw bytes, and nothing here delivers that local copy anywhere a second time.
Prettier formatting on index.ts/run-trigger-client.ts, and the report-error check's one real finding: readJson's parse-failure catch now carries a report-error-ignore note explaining that a malformed response body is still caught (loudly) by the caller's own arktype parse of the response.
TheGreatAxios
changed the base branch from
cl-7490-wake-booting
to
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the
September 6, 2026 23:32
…t-against-interchange-main-before-the' into cl-7491-chat-trigger # Conflicts: # packages/chat/test/platform-adapter.test.ts
TheGreatAxios
merged commit Sep 6, 2026
f10b776
into
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the
22 of 24 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves CL-7490.
Cause
A provisioned run is deployed to its sidecar lazily by its first native
trigger (
vendor/intx/hub-api/src/workflow-run-trigger.ts'screateWorkflowRunTrigger: anchor -> grants/principal -> durabledispatch's
agent.deploy-> deliver mail). Chat sent viasessionService.sendUserMessage, which delivers over the sidecar'sordinary channel directly and never reaches that durable-dispatch
branch, so an invited run that was never triggered natively never got
its first
agent.deployand stayed permanently unreachable.Fix
createWorkflowRunTriggeris not exported from@intx/hub-api(it isthe route's own private dependency, not a published seam), so chat now
calls the mounted route itself as a client:
packages/chat/src/run-trigger-client.tsposts toPOST /api/tenants/:tenantId/workflows/runs/:runId/mailin-processagainst the hub's own composed Hono app (
app.request(...)).@intx/hub-api's ordinary sessioncontract, which chat has no browser cookie to present for (most sends
run with no inbound HTTP request in scope at all — mention fan-out, a
relaunch resend, the mailbox-fanout persist seam).
@intx/hub-api'sGetSessionis a documented pluggable seam ("so a third-partyidentity provider can be plugged in"), and
apps/hubis the one thatsupplies it — so the client signs a short-lived internal token naming
the principal to authenticate as
(
packages/chat/src/run-trigger-internal-auth.ts), andapps/hub'sown
getSessionimplementation verifies it before falling through tothe real cookie-session path. Neither side touches
vendor/intx.packages/chat/src/send-run-mail.tsnow delivers through the triggerclient and correlates on its returned
messageId, still assembling alocal signed MIME copy for this workbench's own
session_mailhistory and attachment blob reads (the trigger route returns no raw
bytes) — never delivered anywhere a second time.
sessionService.sendUserMessagepath is gone for chat, nofallback beside it.
Stacks on #679.
Open gap: durable dispatch does not deploy a probe-adopted allocation
bun test scripts/e2e/chat.test.tsstill fails locally and in CI'se2e-suite(chat) job on the invited-run assertions. Root cause isupstream of this PR's change, inside vendored
@intx/hub-sessions, not in Workbench code:vendor/intx/hub-api/src/workflow-run-trigger.ts(~lines 401-486)correctly: it locks the anchor's allocation and enqueues a
dispatch:<anchorRunId>:<messageId>row viaworkflowDispatchService,returning 202.
vendor/intx/hub-sessions/src/workflow-allocation-service.ts'sdeployReadyAllocation(~line 653) has actually deployed the workflowonto the sidecar (
sendWorkflowRunDispatchToAllocationneeds a routedagent address).
deployReadyAllocationruns fromonReadyinsidesidecar-allocation/reconciler.ts'swaitUntilReady, fired once theallocation's sidecar registers as
kind: "allocated".prepareProvisionedDeployment(same file, ~line 452) callscreateDeploymentwithadoptProbe: truewhenever the deploymentprovisioner it selects equals the probe provisioner
(~lines 617-628) — which is always true here, since
apps/hub/src/index.tsintentionally gives
workflowAllocationServiceone shared provisionerlist for both
deploymentPluginsandprobePlugins("Workbench hasone provisioner list; probes and frozen deployments share it until a
separate probe backend is configured"). The adopt path
(~lines 391-443) writes the
workflowRun/launch-spec/allocation rows,then calls
allocationRouter.disconnectAllocationon the probe's ownconnection to force the same sidecar to reconnect under the
allocatedidentity.e2e-logs-chatartifact, that reconnect isexactly what happens (
sc_67738d3f97572262c8c09c872e9e71c1/sal_7be88209bdd221c1b6fe4812c19cd05aregisters at 23:28:40.331 asthe probe, disconnects at 23:28:42.514, and registers again at
23:28:45.523 with the same allocation and generation) — but nothing
follows: no
workflow-host·definition-loaderline, noDeployed agentline, no warning or error, for the rest of the run.e2e-logs-smoke-webhook's webhook run never hits this branch:its probe and deployment allocations get different ids and different
sidecars (
adoptProbe: false, a genuinely fresh allocation), andDeployed agentappears within ~3.7s of that fresh allocation's firstregistration.
the chat e2e's three invited-run assertions time out waiting for
agent progress that never arrives; the local hub log shows the same
probe-adopt-then-silent-reconnect pattern with no
Deployed agentline, no thrown error, and no scheduled retry logged anywhere in
sidecar-allocation/reconciler.tsorworkflow-dispatch-service.ts.This is a genuine defect in the vendored
adoptProbepath (it neverreaches
session-service.ts'sdeployPreparedCodeSourcedWorkflow/emits no deploy frame after the reconnect, and fails silently instead of
retrying or erroring), not something this PR's chat-side trigger change
can work around — it is upstream of the trigger call entirely, in how
the allocation gets a workflow deployed onto it in the first place.
Leaving
check:structural's vendoring rule intact: no vendor edit here.The workaround of giving
probePluginsanddeploymentPluginsgenuinelydifferent provisioner lists so
adoptProbenever triggers would dodgeit, but that is a separate, deliberate infra decision (today's comment
in
apps/hub/src/index.tssays as much: "until a separate probe backendis configured") and needs its own ticket rather than a silent side
effect of this PR.
Test plan
bunx tsc --noEmit -p packages/chat -p apps/hub— clean.bun test packages/chat/src packages/chat/test apps/hub/src— allgreen (1007 pass, 0 fail, 81 DB-gated skips without
DATABASE_URL).bun test scripts/e2e/chat.test.tsagainst a local Postgres — 7 pass,3 fail on the invited-run assertions, root-caused above.