Repository navigation
Chat: first-turn delivery waits for the provisioned sidecar to become routable (CL-7487) - #676
Merged
TheGreatAxios merged 55 commits intoSep 6, 2026
Conversation
The native workflow deploy route now requires sourceOfferingIds/ defaultSourceOfferingId resolved against the tenant's own catalog, rather than a raw provider/baseURL/apiKey source chain. Update workflowDeployBody accordingly, and extract the zero-cost noop catalog chain smoke-webhook.test.ts already built (an anthropic- plugin provider pointed at the hub's noop-inference endpoint) into a shared seedNoopCatalogOffering harness helper. Every deploying e2e suite now seeds or resolves a real catalog offering before deploying: walking-skeleton keeps a never-real placeholder host (assertNeverRealProvider still guards it), chat resolves the offering seedCatalog already planted rather than seeding a second chain, and heartbeat/workbench-digest/smoke-webhook seed the noop chain via the shared helper. cl-6324-launch-proof's section-mode deploy now reuses the offering id the proof already resolved when it narrowed the bench catalog to its pinned model.
seedTenant currently deploys every default workflow with a bare provider/baseURL/apiKey source, but the hub's deploy route now requires sourceOfferingIds/defaultSourceOfferingId resolved from the tenant's catalog. Update the seeding, onboarding, and hub config test suites to expect the new offering-id based deploy body and provider/model-only ModelSource shape ahead of the implementation change.
ensureDeployment posted a bare {provider, baseURL, apiKey, model}
source, but the hub's deploy route requires sourceOfferingIds and a
defaultSourceOfferingId resolved from the tenant's catalog, so seeding
a connect flow's "assistant" workflow 400'd. seedTenant now resolves
the tenant's real catalog offerings (sorted by priority, lowest wins
the default) for a normal workflow, and plants a dedicated noop
catalog offering for a modelSource-pinned one (heartbeat), pointed at
the hub's own noop-inference endpoint. ModelSource shrinks to a bare
provider/model pair — the baseURL/apiKey it used to carry only mattered
for the old deploy body, and callers that still need a key for seeding
a catalog (system-seed's root tenant, config's env-key auto-plant) keep
it on their own local type instead.
Another workflowDeployBody caller CI's typecheck job caught: this stack seeds one catalog offering per real inference target already, so track each one's id and pass it through to the deploy body instead of the raw provider/baseURL/apiKey shape.
…' into cl-7471-generation-zero
The allocation service's initial sidecar_allocation and workflow_probe rows start at generation 0 (@intx/db enforces generation >= 0, not > 0), so the first ensure() for a freshly allocated sidecar must be accepted at generation 0. These tests cover that acceptance, later generations, non-integer/negative rejection, and that stale-generation fencing still rejects an older generation once a newer one has been observed.
@intx/db's sidecar_allocation and workflow_probe tables both check generation >= 0 and the allocation service records the first row for an allocation at generation 0, but validateEnsureRequest() and validateDestroyRequest() in packages/sandbox-sidecar rejected anything that wasn't strictly positive, and the on-disk fencing schema in state-store.ts required generation > 0. The first ensure() for any newly allocated sidecar was rejected with 'generation must be a positive integer', failing every deploy. Accept generation >= 0 in both validators and the state-store schema; stale-generation fencing in observeEnsure()/observeDestroy() already compares against the stored generation rather than assuming positivity, so it is unaffected.
…t-against-interchange-main-before-the' into cl-7461-local
… signature Covers the new GET .../catalog/resolved-offerings route (denied grant, and both directly-owned and ancestor-inherited offerings sorted by priority) and updates seeding/onboarding test fixtures to the route's response shape ahead of the implementation change.
…owned ones resolveRealSourceOfferingIds listed only offerings a tenant owns directly, missing anything inherited from an ancestor tenant — the hub's own workflow deployer already resolves that wider, ancestor-inheriting view via listVisibleOfferings. Adds GET /api/tenants/:tenantId/catalog/resolved-offerings (mounted from apps/hub, implemented in @corbits/inference-catalog so the deployer and this route share one source of truth) and points seed at it instead. Also drops the two long-dead unused parameters from onboarding's modelSourceFor, and exports ensureNoopCatalogOffering from @corbits/seeding for the e2e harness to reuse.
…' into cl-7472-sidecar-ws
At the interchange pin this branch stacks on (692c3106), the hub authenticates a sidecar's register/reconnect frame only against a sidecar row that resolves to a live sidecar_allocation (or workflow_probe) row. Upstream dropped the older "shared" credential scope this repo's e2e harness used to provision a single, standalone sidecar identity for, so a provisionSidecar-only row now connects and is rejected with "invalid token" in a loop rather than ever dialing in. This test pins that negative so a future re-vendor that silently restores (or relies on) the old scope surfaces here first.
Every e2e suite dial-in relied on a single, harness-started sidecar process registered against a bare `provisionSidecar` identity row. At pin 692c3106, the hub only accepts a register/reconnect frame from a sidecar whose identity resolves to a live sidecar_allocation (or workflow_probe) row; a standalone identity with neither now fails the handshake and gets rejected with "invalid token" every ~3s, forever (chat mint 500s, deploys never dial in — CI run 34042907014). The hub's own process-provisioner already spawns a dedicated, correctly-allocated sidecar process per deployment on demand — the same path production already takes for a real workflow run or a minted agent chat. Every affected suite (walking skeleton, chat, heartbeat, workbench-digest, smoke-chat, smoke-webhook, greeting-delivery, channel-reconnect, local-rip, and the isolation suite) now lets that happen instead of dialing in its own sidecar: the existing 502/500 retry-until-ready loops already cover the wait, and diagnostics that used to read the one sidecar process's output now read the hub's — the process-provisioner backend spawns each sidecar with its stdio inherited from the hub, so the hub's own captured output already carries every spawned sidecar's log lines. `provisionSidecar`/`startSidecar` stay in the harness (documented as non-authenticating at this pin) for the handful of standalone proof scripts not yet migrated off a directly-controlled sidecar process.
The e2e job's hub/sidecar output was never captured, so the CI run that surfaced the sidecar disconnect loop (34042907014) had no sidecar-side log to read past the transcript. Point E2E_LOG_DIR at a runner-temp directory and upload it on every run, pass or fail.
…t-against-interchange-main-before-the' into cl-7474-cross-tenant-launch # Conflicts: # scripts/e2e/harness.ts # scripts/e2e/heartbeat.test.ts # scripts/e2e/smoke-webhook.test.ts # scripts/e2e/walking-skeleton.test.ts # scripts/e2e/workbench-digest.test.ts
readBindingByAddress and resolveLiveByStableId took an address or
stable id and resolved whichever workbench_launch row matched it,
regardless of which tenant that row belonged to. Every caller that
already knows its own tenant (sendMail's requireLive, and
refreshAgentInstanceFromDefinition) now passes it through and gets
undefined instead of another tenant's launch if the two ever
disagree. The one legitimate tenant-agnostic caller — the event
stream handler discovering which tenant an inbound address even
belongs to before it has one to check against — keeps working by
simply omitting the new optional parameter.
Also fixes offeringDigest: it joined catalog offering ids with a
NUL byte, which is not valid UTF-8 to Postgres the moment a tenant's
catalog carries more than one visible offering ("invalid byte
sequence for encoding \"UTF8\": 0x00"), permanently failing the
workbench_launch insert for every such launch. This is what was
failing 4 of scripts/e2e/chat.test.ts's cases under per-allocation
sidecars, where seedCatalog now produces more than one offering.
…t-against-interchange-main-before-the' into cl-7476-run-turns # Conflicts: # scripts/e2e/harness.test.ts # scripts/e2e/smoke-webhook.test.ts # scripts/e2e/walking-skeleton.test.ts
…o cl-7476-run-turns
…nd 'origin/cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the' into cl-7477-agent-session
resolveRunSessionId (Interchange's mail-routing lookup) reads agent_session keyed by a run's principal, but nothing writes that table for a native launch since folded-runs was deleted -- a run's first turn silently fails to persist mail. Proves recordAgentSessionForRun writes a row resolveRunSessionId reads back, and updates the webhook launch fake DB to model workflow_run/agent_session the way the fix now touches them.
…st (CL-7477) Before #632 deleted folded-runs, mintFoldedRun was the only writer of agent_session, the table resolveRunSessionId (Interchange's mail-routing lookup, vendor/intx/hub-sessions/src/hub-session-lookups.ts) reads by a run's principal to find its session. Every current native launch path mints its own sessionId and hands it to prepareProvisionedDeployment, but nothing ever wrote it into agent_session -- a run's first turn logged "Endpoint ... has no session for address" and silently dropped its mail. workflow_run.principal_id is still null the instant prepareProvisionedDeployment returns: a provisioned anchor is born "deployed" with no principal, and only the run's first trigger reconciles one onto it (Interchange's anchorWithPrincipal), through the exact same trigger path every sendUserMessage drives. So the write has to land after that first send, not right after provisioning. @corbits/workflows now exports recordAgentSessionForRun, a shared helper that reads a run's tenant/definition/principal fresh off workflow_run and inserts the session row (idempotent, since every send after the first is a no-op). Every call site that sends a message wires it in right after that send: webhook-triggers' launch, agent-directory's one-shot prompt, and chat's sendRunMail (the one chokepoint every chat send -- invite, wake, relaunch -- funnels through, since an invite alone provisions a run with no message and would only ever see a still-null principal). apps/hub's agent-authored-workflow deployer mints a run with no message of its own and its first trigger runs entirely through Interchange's native route with no Workbench hook to record afterward, so it's left alone. Session teardown reuses the same read: chat's relaunch marks the outgoing run's session ended before repointing the room, and the one-shot prompt's teardown ends its own session once its single turn settles.
…t-against-interchange-main-before-the' into cl-7479-collector
…t-against-interchange-main-before-the' into cl-7479-rebase # Conflicts: # packages/agent-directory/src/one-shot-prompt.ts # packages/chat/src/platform-adapter.ts # packages/chat/src/send-run-mail.ts # packages/webhook-triggers/src/launch.ts # packages/webhook-triggers/test/launch.test.ts # packages/workflows/src/deliver-when-routable.ts # packages/workflows/src/index.ts # packages/workflows/src/launch/agent-session.ts # packages/workflows/test/agent-session.test.ts
Replaces the post-send recordAgentSessionForRun (CL-7477/CL-7479) with one idempotent ensureRunSession, called lazily at the hub seams that actually see a run become mail-routable: the mailbox persist wrapper and the event-collector dispatch path, plus chat's pre-send mail dispatch. Pre-creating the run's principal to dodge the timing isn't an option -- Interchange's grant materialization treats a conflict on that insert as grants already committed and throws.
…t-against-interchange-main-before-the' into cl-7486-wake-binding # Conflicts: # packages/agent-directory/src/one-shot-prompt.ts # packages/chat/src/platform-adapter.ts # packages/chat/src/send-run-mail.ts # packages/webhook-triggers/src/launch.ts # packages/webhook-triggers/test/launch.test.ts # packages/workflows/src/index.ts # packages/workflows/src/launch/agent-session.ts # packages/workflows/test/agent-session.test.ts
…uplicate key Merging cl-7107's repin/re-vendor audit into this branch left packages/workflows/src/index.ts re-exporting the pre-CL-7481 agent-session API (recordAgentSessionForRun) that no longer exists, its matching test file testing that same stale API, and a duplicated eventCollectors key in apps/hub/src/index.ts's webhook-trigger launch wiring. Restores the current exports (recordAgentSessionAtProvision, ensureRunSession) and their test, and drops the duplicate object key.
sendRunMailWithReclaimRetry retried every attempt against the address it started with. Proves the reproduction of the production failure: a participant already one relaunch generation in that dies again mid-retry must have its next retry chase the run that replaced it, not keep hammering the address that just died.
sendRunMailWithReclaimRetry retried every delivery attempt, and woke, against the address it captured at the top of the loop. A wake in between retries can relaunch the run (its dead-run branch repoints the room's stable participant at a fresh address without telling this loop), and a participant already one relaunch generation in that dies again mid-retry leaves this loop holding an address that is no longer either the stable participant id or the current run id — invisible to readBindingByAddressAnyTenant. The next wake for it then threw "No workbench_launch binding" instead of finding the run that replaced it, which is what chat e2e's mention-fanout and invite-drives-a-turn cases were hitting. Re-resolves the live address (and its session) off the stable participant id after each wake, so the retry loop always chases whichever run is actually current. Also hardens publishCorbitsToolsRegistry's create-then-list-on-conflict ensure: two overlapping seed runs racing the same tenant's package registry asset can still throw a raw asset_tenant_kind_name violation if the substrate's own 409 mapping is bypassed; the ensure now also recognizes that violation by name and resolves the winner's row instead of throwing.
…486-wake-binding # Conflicts: # packages/webhook-triggers/test/launch.test.ts
…thReclaimRetry Drives the CL-7486 reclaim-retry test off routing-table transitions instead of fixed backoff timing, and updates the CL-7214 coalescing test's fake sidecar router to model a redeploy actually registering.
…ving up on first-turn delivery sendRunMailWithReclaimRetry used to exhaust a ~7.75s backoff and report the invited agent unreachable while its sidecar was still booting and registering with the hub (CL-7476) -- fast enough to make the fan-out relaunch a perfectly healthy run. It now shares deliverWhenRoutable with the webhook trigger's ingress delivery: wake once on the first unreachable failure (still chasing a mid-wait relaunch by re-resolving the live address inside the retry's send closure), then wait on the hub's routing table up to the same boot-and-register budget (DEFAULT_ROUTABLE_DEADLINE_MS, now exported from @corbits/workflows) before retrying exactly once more.
TheGreatAxios
changed the base branch from
cl-7486-wake-binding
to
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the
September 6, 2026 21:56
TheGreatAxios
changed the title
TheGreatAxios
merged commit Sep 6, 2026
983d1fb
into
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the
21 of 24 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #675.
Resolves CL-7487
Root cause (CI run 34060624470, hub-17311.log)
Inviting the echo agent provisions a per-allocation sidecar that registers 6-9s later (provisioned 21:23:53.040, deployed 21:23:59.294). Chat's
sendRunMailWithReclaimRetryinpackages/chat/src/platform-adapter.tslogsexhausted every reclaim retry; giving upat 21:23:56.301 (~3s in) — its fixed backoff (250/500/1000/2000/4000ms, ~7.75s total) reports the agent unreachable well before the sidecar has a chance to register, so fan-out treats the run as dead, the room relaunches, and it repeats.The webhook path (
packages/webhook-triggers/src/launch.ts) already handles this exact race withdeliverWhenRoutablefrompackages/workflows/src/deliver-when-routable.ts: send once, and only on an "agent is unreachable" failure, poll the hub's routing table until the address is routable (20s default, "a touch past the process provisioner's own boot-and-register time") before sending exactly once more.Fix
sendRunMailWithReclaimRetrynow usesdeliverWhenRoutableinstead of its own fixed-delay backoff loop. On the first unreachable failure it wakes the address once (wakeByAddressBounded, unchanged — this is what actually relaunches a genuinely dead run) and re-resolves the live address from the stable participant id, then hands off todeliverWhenRoutableto wait on the routing table. The re-resolve lives inside thesendclosuredeliverWhenRoutablecalls, so a relaunch that happens during the routable-wait is chased on the retried send, not just the one right after the wake.DEFAULT_ROUTABLE_DEADLINE_MS(20s) is now exported from@corbits/workflows'sdeliver-when-routable.tsso chat shares the exact same boot-and-register budget the webhook path already waits on, instead of declaring a new literal. Deleted the oldRECLAIM_RETRY_DELAYS_MSconstant,reclaimRetryDelaysMsdep, and the bespokesleep/isAgentUnreachablehelpers (the latter replaced by@corbits/workflows'sisAgentUnreachableError).routableWaitDeadlineMsas the equivalent test-injection knob (mirrors the existingmailDeliveryTimeoutMs).isRoutableport needed inapps/hub/src/index.ts—createHubChatPlatformalready receives the fullsidecarRouter, and chat's existingisRoutable()helper (readingsidecarRouter.getRoutableAddresses()) is exactly the same routing-table read the webhook launch wires in separately.Checked the other two symptoms named in the ticket:
packages/chat/src/routes.ts) callsplatform.ensureAwake, which already routes through the samewakeByAddressBounded/lifecycle.ensureAwakepath as everything else — it has no separate literal wait to align.wake … did not settle within 30000mstimeout is@corbits/agent-lifecycle'sDEFAULT_WAKE_TIMEOUT_MS(30s), which bounds theprepareProvisionedDeploymentcall itself, not the sidecar's subsequent boot-and-register — a separate, later step this PR's routable-wait now covers. It comfortably exceeds both the observed 6-9s registration time and the new 20s routable-wait, so it isn't racing the same boot and needed no change.Tests
Updated the CL-7486 unit test (
sendRunMailWithReclaimRetry re-resolves the live address) to drive the fake sidecar router's routable-address transitions instead of asserting on backoff timing, and updated the CL-7214 coalescing test's fake allocation service to mark a redeploy's address routable only once its gated deploy actually resolves. Deleted nothing shape-only; no new tests beyond what the behavior change required.Local:
bunx tsc --noEmit -p packages/chat -p apps/hubclean.bun test packages/chat/src packages/chat/test/platform-adapter.test.ts apps/hub/src: 415 pass, 0 fail (16 DB-gated skips, noDATABASE_URLlocally).