Skip to content

Native launch creates the run's event collector so turns are recorded (CL-7479) - #669

Merged
TheGreatAxios merged 38 commits into
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-thefrom
cl-7479-collector
Sep 6, 2026
Merged

TheGreatAxios merged 38 commits into
cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-thefrom
cl-7479-collector

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Root cause

inference_turn/turn_part rows are written by the per-run event collector (vendor/intx/hub-sessions/src/event-collector.ts), created only by eventCollectors.create(agentAddress, tenantId, sessionId, runId) (vendor/intx/hub-sessions/src/event-collector-registry.ts). Vendored code never calls create; the hub session orchestrator only dispatches (which logs/drops when no collector exists) and abandons. The only caller was the deleted packages/folded-runs/src/launch.ts, which called create right after inserting the agent_session row. Since folded-runs was removed, nothing in this repo ever creates a collector, so no run's turns/turn-parts (or turn-latency, which piggybacks on the same calls) are ever recorded.

Fix

  • recordAgentSessionForRun (packages/workflows/src/launch/agent-session.ts) now takes an EventCollectorPort (create/abandon, Pick<EventCollectorRegistry, ...>) and calls create with the run's address/tenantId/sessionId/id at the same point it records the agent_session row.
  • endAgentSessionForRun now also calls abandon on the outgoing run's address through the same port — one mechanism for ending a run's session (DB status + collector teardown together).
  • The port is threaded from apps/hub/src/index.ts's existing wrapped eventCollectors (the same instance already passed to createHubSessionOrchestrator and createHubChatPlatform) into every caller: packages/chat (sendRunMail, relaunchTerminalRun), packages/webhook-triggers (launchWebhookTrigger), and packages/agent-directory (runOneShotPrompt). No second registry is constructed anywhere.

Follow-up (not in this PR)

No sweeper and no boot-time restore of collectors for already-live runs was added. If a hub restart needs to reattach collectors for runs that were mid-flight, that is a separate, deliberate follow-up.

Resolves CL-7479. Stacks on #668.

The native workflow deploy route now requires sourceOfferingIds/
defaultSourceOfferingId resolved against the tenant's own catalog,
rather than a raw provider/baseURL/apiKey source chain. Update
workflowDeployBody accordingly, and extract the zero-cost noop
catalog chain smoke-webhook.test.ts already built (an anthropic-
plugin provider pointed at the hub's noop-inference endpoint) into
a shared seedNoopCatalogOffering harness helper.

Every deploying e2e suite now seeds or resolves a real catalog
offering before deploying: walking-skeleton keeps a never-real
placeholder host (assertNeverRealProvider still guards it), chat
resolves the offering seedCatalog already planted rather than
seeding a second chain, and heartbeat/workbench-digest/smoke-webhook
seed the noop chain via the shared helper. cl-6324-launch-proof's
section-mode deploy now reuses the offering id the proof already
resolved when it narrowed the bench catalog to its pinned model.
seedTenant currently deploys every default workflow with a bare
provider/baseURL/apiKey source, but the hub's deploy route now requires
sourceOfferingIds/defaultSourceOfferingId resolved from the tenant's
catalog. Update the seeding, onboarding, and hub config test suites to
expect the new offering-id based deploy body and provider/model-only
ModelSource shape ahead of the implementation change.
ensureDeployment posted a bare {provider, baseURL, apiKey, model}
source, but the hub's deploy route requires sourceOfferingIds and a
defaultSourceOfferingId resolved from the tenant's catalog, so seeding
a connect flow's "assistant" workflow 400'd. seedTenant now resolves
the tenant's real catalog offerings (sorted by priority, lowest wins
the default) for a normal workflow, and plants a dedicated noop
catalog offering for a modelSource-pinned one (heartbeat), pointed at
the hub's own noop-inference endpoint. ModelSource shrinks to a bare
provider/model pair — the baseURL/apiKey it used to carry only mattered
for the old deploy body, and callers that still need a key for seeding
a catalog (system-seed's root tenant, config's env-key auto-plant) keep
it on their own local type instead.
Another workflowDeployBody caller CI's typecheck job caught: this
stack seeds one catalog offering per real inference target already,
so track each one's id and pass it through to the deploy body
instead of the raw provider/baseURL/apiKey shape.
The allocation service's initial sidecar_allocation and workflow_probe
rows start at generation 0 (@intx/db enforces generation >= 0, not >
0), so the first ensure() for a freshly allocated sidecar must be
accepted at generation 0. These tests cover that acceptance, later
generations, non-integer/negative rejection, and that stale-generation
fencing still rejects an older generation once a newer one has been
observed.
@intx/db's sidecar_allocation and workflow_probe tables both check
generation >= 0 and the allocation service records the first row for
an allocation at generation 0, but validateEnsureRequest() and
validateDestroyRequest() in packages/sandbox-sidecar rejected anything
that wasn't strictly positive, and the on-disk fencing schema in
state-store.ts required generation > 0. The first ensure() for any
newly allocated sidecar was rejected with 'generation must be a
positive integer', failing every deploy.

Accept generation >= 0 in both validators and the state-store schema;
stale-generation fencing in observeEnsure()/observeDestroy() already
compares against the stored generation rather than assuming positivity,
so it is unaffected.
…t-against-interchange-main-before-the' into cl-7461-local
… signature

Covers the new GET .../catalog/resolved-offerings route (denied grant,
and both directly-owned and ancestor-inherited offerings sorted by
priority) and updates seeding/onboarding test fixtures to the route's
response shape ahead of the implementation change.
…owned ones

resolveRealSourceOfferingIds listed only offerings a tenant owns
directly, missing anything inherited from an ancestor tenant — the
hub's own workflow deployer already resolves that wider,
ancestor-inheriting view via listVisibleOfferings. Adds
GET /api/tenants/:tenantId/catalog/resolved-offerings (mounted from
apps/hub, implemented in @corbits/inference-catalog so the deployer and
this route share one source of truth) and points seed at it instead.

Also drops the two long-dead unused parameters from
onboarding's modelSourceFor, and exports ensureNoopCatalogOffering from
@corbits/seeding for the e2e harness to reuse.
At the interchange pin this branch stacks on (692c3106), the hub
authenticates a sidecar's register/reconnect frame only against a
sidecar row that resolves to a live sidecar_allocation (or
workflow_probe) row. Upstream dropped the older "shared" credential
scope this repo's e2e harness used to provision a single, standalone
sidecar identity for, so a provisionSidecar-only row now connects and
is rejected with "invalid token" in a loop rather than ever dialing
in. This test pins that negative so a future re-vendor that silently
restores (or relies on) the old scope surfaces here first.
Every e2e suite dial-in relied on a single, harness-started sidecar
process registered against a bare `provisionSidecar` identity row.
At pin 692c3106, the hub only accepts a register/reconnect frame from
a sidecar whose identity resolves to a live sidecar_allocation (or
workflow_probe) row; a standalone identity with neither now fails the
handshake and gets rejected with "invalid token" every ~3s, forever
(chat mint 500s, deploys never dial in — CI run 34042907014).

The hub's own process-provisioner already spawns a dedicated,
correctly-allocated sidecar process per deployment on demand — the
same path production already takes for a real workflow run or a
minted agent chat. Every affected suite (walking skeleton, chat,
heartbeat, workbench-digest, smoke-chat, smoke-webhook,
greeting-delivery, channel-reconnect, local-rip, and the isolation
suite) now lets that happen instead of dialing in its own sidecar:
the existing 502/500 retry-until-ready loops already cover the wait,
and diagnostics that used to read the one sidecar process's output
now read the hub's — the process-provisioner backend spawns each
sidecar with its stdio inherited from the hub, so the hub's own
captured output already carries every spawned sidecar's log lines.

`provisionSidecar`/`startSidecar` stay in the harness (documented as
non-authenticating at this pin) for the handful of standalone proof
scripts not yet migrated off a directly-controlled sidecar process.
The e2e job's hub/sidecar output was never captured, so the CI run
that surfaced the sidecar disconnect loop (34042907014) had no
sidecar-side log to read past the transcript. Point E2E_LOG_DIR at a
runner-temp directory and upload it on every run, pass or fail.
…t-against-interchange-main-before-the' into cl-7474-cross-tenant-launch

# Conflicts:
#	scripts/e2e/harness.ts
#	scripts/e2e/heartbeat.test.ts
#	scripts/e2e/smoke-webhook.test.ts
#	scripts/e2e/walking-skeleton.test.ts
#	scripts/e2e/workbench-digest.test.ts
readBindingByAddress and resolveLiveByStableId took an address or
stable id and resolved whichever workbench_launch row matched it,
regardless of which tenant that row belonged to. Every caller that
already knows its own tenant (sendMail's requireLive, and
refreshAgentInstanceFromDefinition) now passes it through and gets
undefined instead of another tenant's launch if the two ever
disagree. The one legitimate tenant-agnostic caller — the event
stream handler discovering which tenant an inbound address even
belongs to before it has one to check against — keeps working by
simply omitting the new optional parameter.

Also fixes offeringDigest: it joined catalog offering ids with a
NUL byte, which is not valid UTF-8 to Postgres the moment a tenant's
catalog carries more than one visible offering ("invalid byte
sequence for encoding \"UTF8\": 0x00"), permanently failing the
workbench_launch insert for every such launch. This is what was
failing 4 of scripts/e2e/chat.test.ts's cases under per-allocation
sidecars, where seedCatalog now produces more than one offering.
…t-against-interchange-main-before-the' into cl-7476-run-turns

# Conflicts:
#	scripts/e2e/harness.test.ts
#	scripts/e2e/smoke-webhook.test.ts
#	scripts/e2e/walking-skeleton.test.ts
…nd 'origin/cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the' into cl-7477-agent-session
resolveRunSessionId (Interchange's mail-routing lookup) reads agent_session
keyed by a run's principal, but nothing writes that table for a native
launch since folded-runs was deleted -- a run's first turn silently fails
to persist mail. Proves recordAgentSessionForRun writes a row
resolveRunSessionId reads back, and updates the webhook launch fake DB to
model workflow_run/agent_session the way the fix now touches them.
…st (CL-7477)

Before #632 deleted folded-runs, mintFoldedRun was the only writer of
agent_session, the table resolveRunSessionId (Interchange's mail-routing
lookup, vendor/intx/hub-sessions/src/hub-session-lookups.ts) reads by a
run's principal to find its session. Every current native launch path
mints its own sessionId and hands it to prepareProvisionedDeployment, but
nothing ever wrote it into agent_session -- a run's first turn logged
"Endpoint ... has no session for address" and silently dropped its mail.

workflow_run.principal_id is still null the instant
prepareProvisionedDeployment returns: a provisioned anchor is born
"deployed" with no principal, and only the run's first trigger reconciles
one onto it (Interchange's anchorWithPrincipal), through the exact same
trigger path every sendUserMessage drives. So the write has to land after
that first send, not right after provisioning.

@corbits/workflows now exports recordAgentSessionForRun, a shared helper
that reads a run's tenant/definition/principal fresh off workflow_run and
inserts the session row (idempotent, since every send after the first is a
no-op). Every call site that sends a message wires it in right after that
send: webhook-triggers' launch, agent-directory's one-shot prompt, and
chat's sendRunMail (the one chokepoint every chat send -- invite, wake,
relaunch -- funnels through, since an invite alone provisions a run with
no message and would only ever see a still-null principal). apps/hub's
agent-authored-workflow deployer mints a run with no message of its own
and its first trigger runs entirely through Interchange's native route
with no Workbench hook to record afterward, so it's left alone.

Session teardown reuses the same read: chat's relaunch marks the outgoing
run's session ended before repointing the room, and the one-shot prompt's
teardown ends its own session once its single turn settles.
…t-against-interchange-main-before-the' into cl-7479-collector
@linear-code

linear-code Bot commented Sep 6, 2026

Copy link
Copy Markdown

CL-7479

…t-against-interchange-main-before-the' into cl-7479-rebase

# Conflicts:
#	packages/agent-directory/src/one-shot-prompt.ts
#	packages/chat/src/platform-adapter.ts
#	packages/chat/src/send-run-mail.ts
#	packages/webhook-triggers/src/launch.ts
#	packages/webhook-triggers/test/launch.test.ts
#	packages/workflows/src/deliver-when-routable.ts
#	packages/workflows/src/index.ts
#	packages/workflows/src/launch/agent-session.ts
#	packages/workflows/test/agent-session.test.ts
@TheGreatAxios
TheGreatAxios changed the base branch from cl-7477-agent-session to cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the September 6, 2026 19:20
@TheGreatAxios
TheGreatAxios merged commit d4aeba1 into cl-7107-repin-and-re-vendor-audit-against-interchange-main-before-the Sep 6, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant