Repository navigation
Chat: a booting run is waited on, not relaunched (CL-7489) - #679
Merged
TheGreatAxios merged 60 commits intoSep 6, 2026
Conversation
The native workflow deploy route now requires sourceOfferingIds/ defaultSourceOfferingId resolved against the tenant's own catalog, rather than a raw provider/baseURL/apiKey source chain. Update workflowDeployBody accordingly, and extract the zero-cost noop catalog chain smoke-webhook.test.ts already built (an anthropic- plugin provider pointed at the hub's noop-inference endpoint) into a shared seedNoopCatalogOffering harness helper. Every deploying e2e suite now seeds or resolves a real catalog offering before deploying: walking-skeleton keeps a never-real placeholder host (assertNeverRealProvider still guards it), chat resolves the offering seedCatalog already planted rather than seeding a second chain, and heartbeat/workbench-digest/smoke-webhook seed the noop chain via the shared helper. cl-6324-launch-proof's section-mode deploy now reuses the offering id the proof already resolved when it narrowed the bench catalog to its pinned model.
seedTenant currently deploys every default workflow with a bare provider/baseURL/apiKey source, but the hub's deploy route now requires sourceOfferingIds/defaultSourceOfferingId resolved from the tenant's catalog. Update the seeding, onboarding, and hub config test suites to expect the new offering-id based deploy body and provider/model-only ModelSource shape ahead of the implementation change.
ensureDeployment posted a bare {provider, baseURL, apiKey, model}
source, but the hub's deploy route requires sourceOfferingIds and a
defaultSourceOfferingId resolved from the tenant's catalog, so seeding
a connect flow's "assistant" workflow 400'd. seedTenant now resolves
the tenant's real catalog offerings (sorted by priority, lowest wins
the default) for a normal workflow, and plants a dedicated noop
catalog offering for a modelSource-pinned one (heartbeat), pointed at
the hub's own noop-inference endpoint. ModelSource shrinks to a bare
provider/model pair — the baseURL/apiKey it used to carry only mattered
for the old deploy body, and callers that still need a key for seeding
a catalog (system-seed's root tenant, config's env-key auto-plant) keep
it on their own local type instead.
Another workflowDeployBody caller CI's typecheck job caught: this stack seeds one catalog offering per real inference target already, so track each one's id and pass it through to the deploy body instead of the raw provider/baseURL/apiKey shape.
…' into cl-7471-generation-zero
The allocation service's initial sidecar_allocation and workflow_probe rows start at generation 0 (@intx/db enforces generation >= 0, not > 0), so the first ensure() for a freshly allocated sidecar must be accepted at generation 0. These tests cover that acceptance, later generations, non-integer/negative rejection, and that stale-generation fencing still rejects an older generation once a newer one has been observed.
@intx/db's sidecar_allocation and workflow_probe tables both check generation >= 0 and the allocation service records the first row for an allocation at generation 0, but validateEnsureRequest() and validateDestroyRequest() in packages/sandbox-sidecar rejected anything that wasn't strictly positive, and the on-disk fencing schema in state-store.ts required generation > 0. The first ensure() for any newly allocated sidecar was rejected with 'generation must be a positive integer', failing every deploy. Accept generation >= 0 in both validators and the state-store schema; stale-generation fencing in observeEnsure()/observeDestroy() already compares against the stored generation rather than assuming positivity, so it is unaffected.
…t-against-interchange-main-before-the' into cl-7461-local
… signature Covers the new GET .../catalog/resolved-offerings route (denied grant, and both directly-owned and ancestor-inherited offerings sorted by priority) and updates seeding/onboarding test fixtures to the route's response shape ahead of the implementation change.
…owned ones resolveRealSourceOfferingIds listed only offerings a tenant owns directly, missing anything inherited from an ancestor tenant — the hub's own workflow deployer already resolves that wider, ancestor-inheriting view via listVisibleOfferings. Adds GET /api/tenants/:tenantId/catalog/resolved-offerings (mounted from apps/hub, implemented in @corbits/inference-catalog so the deployer and this route share one source of truth) and points seed at it instead. Also drops the two long-dead unused parameters from onboarding's modelSourceFor, and exports ensureNoopCatalogOffering from @corbits/seeding for the e2e harness to reuse.
…' into cl-7472-sidecar-ws
At the interchange pin this branch stacks on (692c3106), the hub authenticates a sidecar's register/reconnect frame only against a sidecar row that resolves to a live sidecar_allocation (or workflow_probe) row. Upstream dropped the older "shared" credential scope this repo's e2e harness used to provision a single, standalone sidecar identity for, so a provisionSidecar-only row now connects and is rejected with "invalid token" in a loop rather than ever dialing in. This test pins that negative so a future re-vendor that silently restores (or relies on) the old scope surfaces here first.
Every e2e suite dial-in relied on a single, harness-started sidecar process registered against a bare `provisionSidecar` identity row. At pin 692c3106, the hub only accepts a register/reconnect frame from a sidecar whose identity resolves to a live sidecar_allocation (or workflow_probe) row; a standalone identity with neither now fails the handshake and gets rejected with "invalid token" every ~3s, forever (chat mint 500s, deploys never dial in — CI run 34042907014). The hub's own process-provisioner already spawns a dedicated, correctly-allocated sidecar process per deployment on demand — the same path production already takes for a real workflow run or a minted agent chat. Every affected suite (walking skeleton, chat, heartbeat, workbench-digest, smoke-chat, smoke-webhook, greeting-delivery, channel-reconnect, local-rip, and the isolation suite) now lets that happen instead of dialing in its own sidecar: the existing 502/500 retry-until-ready loops already cover the wait, and diagnostics that used to read the one sidecar process's output now read the hub's — the process-provisioner backend spawns each sidecar with its stdio inherited from the hub, so the hub's own captured output already carries every spawned sidecar's log lines. `provisionSidecar`/`startSidecar` stay in the harness (documented as non-authenticating at this pin) for the handful of standalone proof scripts not yet migrated off a directly-controlled sidecar process.
The e2e job's hub/sidecar output was never captured, so the CI run that surfaced the sidecar disconnect loop (34042907014) had no sidecar-side log to read past the transcript. Point E2E_LOG_DIR at a runner-temp directory and upload it on every run, pass or fail.
…t-against-interchange-main-before-the' into cl-7474-cross-tenant-launch # Conflicts: # scripts/e2e/harness.ts # scripts/e2e/heartbeat.test.ts # scripts/e2e/smoke-webhook.test.ts # scripts/e2e/walking-skeleton.test.ts # scripts/e2e/workbench-digest.test.ts
readBindingByAddress and resolveLiveByStableId took an address or
stable id and resolved whichever workbench_launch row matched it,
regardless of which tenant that row belonged to. Every caller that
already knows its own tenant (sendMail's requireLive, and
refreshAgentInstanceFromDefinition) now passes it through and gets
undefined instead of another tenant's launch if the two ever
disagree. The one legitimate tenant-agnostic caller — the event
stream handler discovering which tenant an inbound address even
belongs to before it has one to check against — keeps working by
simply omitting the new optional parameter.
Also fixes offeringDigest: it joined catalog offering ids with a
NUL byte, which is not valid UTF-8 to Postgres the moment a tenant's
catalog carries more than one visible offering ("invalid byte
sequence for encoding \"UTF8\": 0x00"), permanently failing the
workbench_launch insert for every such launch. This is what was
failing 4 of scripts/e2e/chat.test.ts's cases under per-allocation
sidecars, where seedCatalog now produces more than one offering.
…t-against-interchange-main-before-the' into cl-7476-run-turns # Conflicts: # scripts/e2e/harness.test.ts # scripts/e2e/smoke-webhook.test.ts # scripts/e2e/walking-skeleton.test.ts
…o cl-7476-run-turns
…t-against-interchange-main-before-the' into cl-7479-rebase # Conflicts: # packages/agent-directory/src/one-shot-prompt.ts # packages/chat/src/platform-adapter.ts # packages/chat/src/send-run-mail.ts # packages/webhook-triggers/src/launch.ts # packages/webhook-triggers/test/launch.test.ts # packages/workflows/src/deliver-when-routable.ts # packages/workflows/src/index.ts # packages/workflows/src/launch/agent-session.ts # packages/workflows/test/agent-session.test.ts
Replaces the post-send recordAgentSessionForRun (CL-7477/CL-7479) with one idempotent ensureRunSession, called lazily at the hub seams that actually see a run become mail-routable: the mailbox persist wrapper and the event-collector dispatch path, plus chat's pre-send mail dispatch. Pre-creating the run's principal to dodge the timing isn't an option -- Interchange's grant materialization treats a conflict on that insert as grants already committed and throws.
…t-against-interchange-main-before-the' into cl-7486-wake-binding # Conflicts: # packages/agent-directory/src/one-shot-prompt.ts # packages/chat/src/platform-adapter.ts # packages/chat/src/send-run-mail.ts # packages/webhook-triggers/src/launch.ts # packages/webhook-triggers/test/launch.test.ts # packages/workflows/src/index.ts # packages/workflows/src/launch/agent-session.ts # packages/workflows/test/agent-session.test.ts
…uplicate key Merging cl-7107's repin/re-vendor audit into this branch left packages/workflows/src/index.ts re-exporting the pre-CL-7481 agent-session API (recordAgentSessionForRun) that no longer exists, its matching test file testing that same stale API, and a duplicated eventCollectors key in apps/hub/src/index.ts's webhook-trigger launch wiring. Restores the current exports (recordAgentSessionAtProvision, ensureRunSession) and their test, and drops the duplicate object key.
sendRunMailWithReclaimRetry retried every attempt against the address it started with. Proves the reproduction of the production failure: a participant already one relaunch generation in that dies again mid-retry must have its next retry chase the run that replaced it, not keep hammering the address that just died.
sendRunMailWithReclaimRetry retried every delivery attempt, and woke, against the address it captured at the top of the loop. A wake in between retries can relaunch the run (its dead-run branch repoints the room's stable participant at a fresh address without telling this loop), and a participant already one relaunch generation in that dies again mid-retry leaves this loop holding an address that is no longer either the stable participant id or the current run id — invisible to readBindingByAddressAnyTenant. The next wake for it then threw "No workbench_launch binding" instead of finding the run that replaced it, which is what chat e2e's mention-fanout and invite-drives-a-turn cases were hitting. Re-resolves the live address (and its session) off the stable participant id after each wake, so the retry loop always chases whichever run is actually current. Also hardens publishCorbitsToolsRegistry's create-then-list-on-conflict ensure: two overlapping seed runs racing the same tenant's package registry asset can still throw a raw asset_tenant_kind_name violation if the substrate's own 409 mapping is bypassed; the ensure now also recognizes that violation by name and resolves the winner's row instead of throwing.
…486-wake-binding # Conflicts: # packages/webhook-triggers/test/launch.test.ts
…thReclaimRetry Drives the CL-7486 reclaim-retry test off routing-table transitions instead of fixed backoff timing, and updates the CL-7214 coalescing test's fake sidecar router to model a redeploy actually registering.
…ving up on first-turn delivery sendRunMailWithReclaimRetry used to exhaust a ~7.75s backoff and report the invited agent unreachable while its sidecar was still booting and registering with the hub (CL-7476) -- fast enough to make the fan-out relaunch a perfectly healthy run. It now shares deliverWhenRoutable with the webhook trigger's ingress delivery: wake once on the first unreachable failure (still chasing a mid-wait relaunch by re-resolving the live address inside the retry's send closure), then wait on the hub's routing table up to the same boot-and-register budget (DEFAULT_ROUTABLE_DEADLINE_MS, now exported from @corbits/workflows) before retrying exactly once more.
…s missing Runs deployed straight through Interchange's own deployments route never pass through a Workbench launcher, so recordAgentSessionAtProvision never runs for them and no agent_session row exists for the run. By the time persist/dispatch call ensureRunSession the run is anchored with a principal, so the seam has everything needed to create the row itself instead of treating the missing row as a bug.
isBeyondWake's completed-run test assumed status alone marks a run terminal; a deployment settled "completed" before its first trigger is still booting, not dead, so the predicate now needs a run that took a turn to treat "completed" as terminal. Update platform-adapter's lifecycle/ensureAwake/sendMail coverage to match: a live run that is merely not yet routable is waited on rather than redeployed, so those tests now exercise a genuinely terminal run instead of an offline one.
A provisioned anchor stays "deployed" until its first trigger, so a run mid-boot is indistinguishable from an offline one by routability alone. wakeByAddress's fallback branch treated the two the same and redeployed either — minting a fresh run and address out from under the sidecar's own in-flight registration, which supersedes its allocation and restarts the boot. Repeated on every wake attempt, this produced an endless reconnect loop with no run ever finishing its boot. wakeByAddress now only relaunches a run whose status is genuinely terminal; a live run that is not yet routable is left alone; the caller's own routable wait (deliverWhenRoutable, via sendRunMailWithReclaimRetry) is what actually resolves it once the sidecar finishes registering. isBeyondWake also no longer treats every "completed" status as terminal outright: a deployment settled complete before its first trigger ever landed is still indistinguishable from one mid-boot, so it is only terminal once the store can tell it took a turn (a principal was ever anchored to it).
TheGreatAxios
changed the base branch from
cl-7489-ensure-upsert
to
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the
September 6, 2026 22:41
TheGreatAxios
changed the title
TheGreatAxios
merged commit Sep 6, 2026
178d81f
into
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the
22 of 24 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves CL-7489.
Evidence
CI run 34062656347, artifact
e2e-logs-chat,hub-2647.log:Deployed agentlines in the entire chat-suite hub log.Provisioned sidecar … registered for allocation … generation 0, materializes the echo closure, thenDisconnected from hub~2.3s later. A new sidecar/allocation appears every ~5s.chat·wake:mail to run_0056… hit "agent is unreachable"; waiting for it to become routablethenmail to run_6f7b… exhausted the routable-wait budget; giving up— a different run id, so the participant was relaunched during the wait. Earlier runs loggedchat·wake: relaunching run_…: run is terminal ('completed')for runs that had never taken a turn.Diagnosis
A provisioned anchor stays
deployeduntil its first trigger (vendor/intx/db/src/schema/workflow-run.ts,vendor/intx/db/src/workflow-run-store.ts'sanchorWithPrincipal).wakeByAddress's fallback branch (not terminal, not routable) treated a booting run exactly like an offline one and redeployed it — minting a fresh run/address and repointing the room's binding, superseding the sidecar's in-flight allocation and restarting its boot. Repeated on every wake, this produced the endless reconnect loop with no run ever finishing its boot. Separately,isBeyondWaketreated every"completed"status as terminal, but a deploymentmarkTerminal'd before its first trigger ever landed also settles"completed"— indistinguishable from a run mid-boot without also checking whether it ever took a turn.Fix
wakeByAddress(packages/chat/src/platform-adapter.ts) now only relaunches a run whose status is genuinely terminal (isBeyondWake). A live run (deployed/running) that is not yet routable is left alone — the caller's own routable wait (deliverWhenRoutable, wired throughsendRunMailWithReclaimRetry) is what resolves it once the sidecar finishes registering. This is the one predicate every wake path (wakeByAddress, the invite pre-warm viaensureAwake,reconcileDriftedRun,reconcilePinnedToolPackages) already shared, so no second heuristic needed removing — the bug was inwakeByAddress's own fallback branch reaching for a redeploy without consulting it.isBeyondWake(packages/chat/src/agent-binding.ts) treats"failed"/"cancelled"/"canceled"as unconditionally terminal, and"completed"as terminal only once a principal was ever anchored to the run (principalId !== null— the same signalanchorWithPrincipalsets on a run's first trigger). A"completed"run with no principal is indistinguishable from one mid-boot and is left alone.wakeByAddressBounded's 30sDEFAULT_WAKE_TIMEOUT_MSdoes not interact: sincewakeByAddressis now a cheap no-op for a live-but-not-yet-routable run, it settles well within that budget every time, and nothing relaunches on timeout anywhere in the wake path.Tests
packages/chat/src/agent-binding.test.ts:isBeyondWakenow covers a"completed"run with and without a principal.packages/chat/test/platform-adapter.test.ts: rewrote the tests that pinned the old "redeploy any non-routable run" behavior (sendMail/ensureAwakewiring, the CL-7214 reclaim-retry coalescing test, and the refresh-on-wake test) to exercise a genuinely terminal run instead, and added coverage that adeployed, not-yet-routable run is waited on and delivered once routable — with no relaunch (prepareCallsstays empty).Verify
bunx tsc --noEmit -p packages/chat,-p packages/agent-lifecycle,-p apps/hub— all clean.bun test packages/chat/src packages/chat/test packages/agent-lifecycle— 925 pass, 0 fail (65 DB-gated suites skip locally withoutDATABASE_URL, as expected).scripts/e2e/chat.test.tsagainst a local Postgres: 7/10 pass, 3 fail onwaitForRunProgresstimeouts with zeroDeployed agentlines in the hub log. Confirmed this is pre-existing and environmental, not a regression: the identical 3 failures with the identical zero-Deployed agentsymptom reproduce on an unmodifiedorigin/cl-7489-ensure-upsertcheckout in the same local environment. The hub never sends anagent.deployframe to the sidecar at all in this environment — a different, deeper gap than the wake/relaunch storm this ticket fixes (that storm is absent from both runs' logs; the sidecar here stabilizes after ~2 reconnects and then holds its connection for the rest of the run). Left for a follow-up; CI (with its own sidecar/process-provisioner setup) is the authoritative signal for the actualDeployed agent+ 10/10 bar.Stack
Based on
cl-7489-ensure-upsert(built oncl-7488-chat-routable, with the re-pin/re-vendor branch merged in).