diff --git a/VENDORED.md b/VENDORED.md index e08445d592..58dc233d96 100644 --- a/VENDORED.md +++ b/VENDORED.md @@ -23,26 +23,26 @@ never a convenience. ## Ledger Pinned to [faremeter/interchange](https://github.com/faremeter/interchange) -@ `5453d0b0` (origin/main, v0.4.0). +@ `9febf69` (origin/main, v0.4.0). | Vendored path | What it is | Owner | Kill date | | ------------------------------ | ------------------------------------------------------------------------ | ------ | ---------- | -| `apps/sidecar` | Copy of upstream's sidecar app; apps are never npm-published. Two local deltas: `workflow-substrate-factory.ts` registers `@corbits/credential-header`'s and `@corbits/credential-mcp`'s providers alongside `builtinCredentialProviders()`; `bin/workflow-child` and `bin/workflow-probe-child` run under plain `bun` instead of upstream's `intx-src` resolve condition; `createSidecarStepBuildEnv` resolves the step's director registry from the closure package (`interchange.directors`) instead of the built-ins only, so a workflow's own director runs and not just probes (kill condition: upstream's step env loads the closure registry) | maintainers | 2026-11-03 | +| `apps/sidecar` | Copy of upstream's sidecar app; apps are never npm-published. Three local deltas: `workflow-substrate-factory.ts` registers `@corbits/credential-header`'s and `@corbits/credential-mcp`'s providers alongside `builtinCredentialProviders()`; `bin/workflow-child` and `bin/workflow-probe-child` run under plain `bun` instead of upstream's `intx-src` resolve condition; `createSidecarStepBuildEnv` resolves the step's director registry from the closure package (`interchange.directors`) instead of the built-ins only, so a workflow's own director runs and not just probes (kill condition: upstream's step env loads the closure registry) | maintainers | 2026-11-03 | | `vendor/intx/agent` | `@intx/agent` at a newer commit than npm has published | maintainers | 2026-11-03 | -| `vendor/intx/db` | `@intx/db`, with one local serving-time credential-refresh delta | maintainers | 2026-11-03 | +| `vendor/intx/db` | `@intx/db`, with local fork migrations (wire_projection, launch origin, tolerate rewrite, tenant slug check), a serving-time credential-refresh seam, and an openai-responses catalog id | maintainers | 2026-11-03 | | `vendor/intx/harness` | `@intx/harness` at a newer commit than npm has published | maintainers | 2026-11-03 | -| `vendor/intx/hub-agent` | `@intx/hub-agent`, with a local OAuth-login-over-websocket delta | maintainers | 2026-11-03 | -| `vendor/intx/hub-api` | `@intx/hub-api`, with the local workflow-deploy-bearer route | maintainers | 2026-11-03 | -| `vendor/intx/hub-sessions` | `@intx/hub-sessions`, with several local run/pack-acceptance fixes | maintainers | 2026-11-03 | +| `vendor/intx/hub-agent` | `@intx/hub-agent`, with local childRunId event threading and an OAuth-login-over-websocket delta | maintainers | 2026-11-03 | +| `vendor/intx/hub-api` | `@intx/hub-api`, with the local workflow-deploy-bearer route and a serving-time credential-refresh hook | maintainers | 2026-11-03 | +| `vendor/intx/hub-sessions` | `@intx/hub-sessions`, with local childRunId event threading, serving-time credential refresh, loopback OAuth login, and prepared-deploy step staging | maintainers | 2026-11-03 | | `vendor/intx/inference` | `@intx/inference`, with one local Google file-upload compile fix | maintainers | 2026-11-03 | | `vendor/intx/mail-memory` | `@intx/mail-memory` at a newer commit than npm has published | maintainers | 2026-11-03 | | `vendor/intx/mailbox` | `@intx/mailbox`, never yet published to npm | maintainers | 2026-11-03 | | `vendor/intx/mime` | `@intx/mime` at a newer commit than npm has published | maintainers | 2026-11-03 | -| `vendor/intx/types` | `@intx/types`, with one local model-provider-plugin enum addition | maintainers | 2026-11-03 | +| `vendor/intx/types` | `@intx/types`, with local childRunId, openai-responses plugin id, and OAuth-login token shapes | maintainers | 2026-11-03 | | `vendor/intx/hub-common` | `@intx/hub-common` at a newer commit than npm has published | maintainers | 2026-11-03 | | `vendor/intx/workflow` | `@intx/workflow`, with one local step-timeout-budget delta | maintainers | 2026-11-03 | | `vendor/intx/workflow-deploy` | `@intx/workflow-deploy` at a newer commit than npm has published | maintainers | 2026-11-03 | -| `vendor/intx/workflow-host` | `@intx/workflow-host`, with a local step-grants-collapse delta and a temporary delta omitting an empty inbound `Subject` header instead of passing it through empty (kill condition: upstream tolerates an empty Subject, INTR-577) | maintainers | 2026-11-03 | +| `vendor/intx/workflow-host` | `@intx/workflow-host`, with local credential-wiring/childRunId threading, a step-grants-collapse delta, and a temporary delta omitting an empty inbound `Subject` header instead of passing it through empty (kill condition: upstream tolerates an empty Subject, INTR-577) | maintainers | 2026-11-03 | Every row above is re-vendored at the same commit rather than mixed pins (npm's `0.4.0` is this commit; rows with no local delta are un-vendoring diff --git a/apps/sidecar/src/conversation-state.ts b/apps/sidecar/src/conversation-state.ts index fe0b19f045..3f89d3f41d 100644 --- a/apps/sidecar/src/conversation-state.ts +++ b/apps/sidecar/src/conversation-state.ts @@ -137,7 +137,16 @@ export interface DurableConversationStore { restoreFromSubstrate(): Promise; /** Commit new turns as an O(1) WAL append, compacting into a checkpoint at the compaction interval. */ mirrorToSubstrate(): Promise; - /** Advances connector thread state from an inbound message so composeReply can address a threaded reply. */ + /** + * Advance the connector router from a received inbound message so the + * warm agent's reply path has thread state. Runs the router's pure + * `route()` then `commit()`; the router owns what each decision does to + * the thread. The advanced connector state is flushed into the local + * store's metadata so the run-boundary mirror persists it and a respawn + * restore re-seeds the router. Called before the warm agent's send so + * `composeReply()` can compose a threaded reply. A metadata write + * failure surfaces. + */ seedInbound(message: InboundMessage): Promise; /** Threading headers for a reply on the active connector thread; throws NoActiveConnectorThreadError with none seeded. */ composeReply(): ConnectorReplyParts; diff --git a/bun.lock b/bun.lock index 615331e0ec..19724bf5eb 100644 --- a/bun.lock +++ b/bun.lock @@ -369,7 +369,6 @@ "@intx/crypto": "0.4.0", "@intx/db": "workspace:*", "@intx/hub-common": "0.4.0", - "@intx/inference": "workspace:*", "@intx/log": "0.4.0", "@intx/mime": "workspace:*", "@intx/pack-transport": "0.4.0", @@ -422,6 +421,7 @@ "version": "0.4.0", "dependencies": { "@intx/crypto": "0.4.0", + "@intx/log": "0.4.0", "@intx/mime": "workspace:*", "@intx/types": "workspace:*", "arktype": "catalog:", diff --git a/e2e/hub/mailbox-persist.test.ts b/e2e/hub/mailbox-persist.test.ts index dd296f5a05..ee654c247d 100644 --- a/e2e/hub/mailbox-persist.test.ts +++ b/e2e/hub/mailbox-persist.test.ts @@ -81,7 +81,7 @@ async function setup(opts: { withSession?: boolean } = {}) { await db.insert(tenantTable).values({ id: tenantId, name: "Mailbox Persist Wrap Tenant", - slug: uid("mbxpw"), + slug: uid("mbxpw").replaceAll("_", "-"), domain, }); await db.insert(workflowDefinition).values({ diff --git a/e2e/workflows/agent-session.test.ts b/e2e/workflows/agent-session.test.ts index d9a26aaae4..5fefdbd36a 100644 --- a/e2e/workflows/agent-session.test.ts +++ b/e2e/workflows/agent-session.test.ts @@ -54,7 +54,7 @@ describeIfDb("ensureRunSession", () => { await db.db.insert(schema.tenant).values({ id: tenantId, name: "Agent Session Test Tenant", - slug: `agent-session-${tenantId}`, + slug: `agent-session-${tenantId.replaceAll("_", "-")}`, domain, parentId: null, config: null, diff --git a/vendor/intx/agent/VENDORED-FROM b/vendor/intx/agent/VENDORED-FROM index ec8336b8aa..7443e8840f 100644 --- a/vendor/intx/agent/VENDORED-FROM +++ b/vendor/intx/agent/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/agent) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. CL-7190: ToolBundle.beforeToolExtension (tool.ts) and its composition into ResolvedTools.beforeToolExtensions (agent.ts), so a tool package can contribute its own suspend-capable extension without the reactor or director special-casing it by name. +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). No source delta (the CL-7190 ToolBundle.beforeToolExtension seam retired at the prior re-vendor: neither that pin nor HEAD agent/src carries it and nothing in the workbench references it). diff --git a/vendor/intx/db/VENDORED-FROM b/vendor/intx/db/VENDORED-FROM index 44aa6c3113..da273b63f4 100644 --- a/vendor/intx/db/VENDORED-FROM +++ b/vendor/intx/db/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/db) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. CL-6324: workflow_definition_version gains a wire_projection jsonb column (migration 0089_workflow_definition_version_wire_projection.sql -- renumbered from 0086 as upstream took 0086/0087/0088; the journal keeps our original `when` values, which still sort after upstream's), schema/workflow-definitions.ts declares it, and workflow-definition-store.ts adds loadFrozenWireProjection (exported from src/index.ts) to read it back beside the approved wire hash. CL-6452: workflow_definition gains an origin column (migration 0090_workflow_definition_origin.sql; declared in schema/workflow-definitions.ts) marking the per-run record a folded run's deploy mints for itself, so launch and refresh resolve the hub-authored definition instead of the newest same-named row. Migration 0091_workflow_definition_version_tolerate_body_failure.sql rewrites the retired `onBodyFailure: "continue"` literal to upstream's `"tolerate"` inside stored wire projections. CL-4455: workflow_definition gains schedule_claimed_minute (migration 0092_workflow_definition_schedule_claimed_minute.sql; declared in schema/workflow-definitions.ts) so the hub's native ScheduleTrigger poller can CAS each UTC minute. CL-8162: the hub-side cron tick is deleted, so schedule_claimed_minute is dropped again (migration 0094_drop_workflow_definition_schedule_claimed_minute.sql, journal idx 94) and schema/workflow-definitions.ts no longer declares it. Workbench applies these files by sorted filename through scripts/db-setup.ts (replay-from-scratch with a filename ledger); the drizzle journal exists for drizzle-kit only. CL-7495 re-pin to c1b5bb73: upstream's 0089_tense_selene.sql (principal_key table) is renumbered to 0093_principal_keys.sql behind our four delta migrations, with journal idx 93 and a 0093_snapshot.json built from upstream's 0089 snapshot reconciled with the wire_projection/origin/schedule_claimed_minute columns; upstream 447ebb08's snapshot-chain repair (relinked 0087 prevId, workflow_run.credential_refs restored) is applied to 0087/0088. Our fork migrations 0089-0092 ship no drizzle snapshots (hand-written SQL, journal only), so there was no fork snapshot chain to repair. CL-8425: the CL-8206 re-pin to 79adc433 moved src/schema (tenant domain lower-unique index, workflow_run_dispatch.sender_address, sidecar_allocation destroy_failed status) without its SQL; upstream's 0090_tenant_domain_lower_unique, 0091_workflow_run_dispatch_sender_address and 0092_sidecar_destroy_failed are vendored verbatim as 0095/0096/0097 (journal idx 95-97, no snapshots). Re-vendor to 5453d0b0: upstream's 0093_sidecar_initialization.sql is vendored verbatim as 0098 (journal idx 98, no snapshot). +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). CL-6324: workflow_definition_version gains a wire_projection jsonb column (migration 0089_workflow_definition_version_wire_projection.sql -- renumbered from 0086 as upstream took 0086/0087/0088; the journal keeps our original `when` values, which still sort after upstream's), schema/workflow-definitions.ts declares it, and workflow-definition-store.ts adds loadFrozenWireProjection (exported from src/index.ts) to read it back beside the approved wire hash. CL-6452: workflow_definition gains an origin column (migration 0090_workflow_definition_origin.sql; declared in schema/workflow-definitions.ts) marking the per-run record a folded run's deploy mints for itself, so launch and refresh resolve the hub-authored definition instead of the newest same-named row. Migration 0091_workflow_definition_version_tolerate_body_failure.sql rewrites the retired `onBodyFailure: "continue"` literal to upstream's `"tolerate"` inside stored wire projections. CL-4455: workflow_definition gains schedule_claimed_minute (migration 0092_workflow_definition_schedule_claimed_minute.sql; declared in schema/workflow-definitions.ts) so the hub's native ScheduleTrigger poller can CAS each UTC minute. CL-8162: the hub-side cron tick is deleted, so schedule_claimed_minute is dropped again (migration 0094_drop_workflow_definition_schedule_claimed_minute.sql, journal idx 94) and schema/workflow-definitions.ts no longer declares it. Workbench applies these files by sorted filename through scripts/db-setup.ts (replay-from-scratch with a filename ledger); the drizzle journal exists for drizzle-kit only. CL-7495 re-pin to c1b5bb73: upstream's 0089_tense_selene.sql (principal_key table) is renumbered to 0093_principal_keys.sql behind our four delta migrations, with journal idx 93 and a 0093_snapshot.json built from upstream's 0089 snapshot reconciled with the wire_projection/origin/schedule_claimed_minute columns; upstream 447ebb08's snapshot-chain repair (relinked 0087 prevId, workflow_run.credential_refs restored) is applied to 0087/0088. Our fork migrations 0089-0092 ship no drizzle snapshots (hand-written SQL, journal only), so there was no fork snapshot chain to repair. CL-8425: the CL-8206 re-pin to 79adc433 moved src/schema (tenant domain lower-unique index, workflow_run_dispatch.sender_address, sidecar_allocation destroy_failed status) without its SQL; upstream's 0090_tenant_domain_lower_unique, 0091_workflow_run_dispatch_sender_address and 0092_sidecar_destroy_failed are vendored verbatim as 0095/0096/0097 (journal idx 95-97, no snapshots). At the prior re-vendor, upstream's 0093_sidecar_initialization.sql was vendored verbatim as 0098 (journal idx 98, no snapshot). Re-vendor to 9febf69: upstream's 0094_dark_old_lace.sql (tenant slug DNS-label check) is vendored verbatim as 0099 (journal idx 99, no snapshot). CL-7505: model-source-resolution.ts gains the serving-time refresh seam (ServingRefresh/ServingRefreshResult, credential_needs_reauth reason; index.ts re-exports the types) so an expiring oauth_token secret can be refreshed in place before serving. CL-7510: schema/catalog.ts admits openai-responses (the workbench's loopback-OAuth providers ride the Responses protocol), in sync with @intx/types modelProviderPlugins. diff --git a/vendor/intx/db/migrations/0099_dark_old_lace.sql b/vendor/intx/db/migrations/0099_dark_old_lace.sql new file mode 100644 index 0000000000..3663f851ce --- /dev/null +++ b/vendor/intx/db/migrations/0099_dark_old_lace.sql @@ -0,0 +1,85 @@ +-- Constrain the tenant slug to a single DNS label. The create route derives +-- tenant.domain from the lowercased slug, and that domain is both the sender +-- stamp on outbound mail and the recipient run address, so a slug that is not a +-- legal label gives that route a corrupt admission policy key and routing key. +-- The derivation is the route's own convention, not a relation between the +-- columns: this constrains what the route derives a domain from, and leaves +-- tenant.domain itself under no grammar constraint. +-- +-- Guard first. A database holding a non-conforming slug fails this migration +-- either way, because the constraint below cannot be validated against such a +-- row -- but the bare constraint violation names no row, so the operator cannot +-- tell which tenant broke the deploy. The guard changes only the quality of +-- that failure. The production path applies migrations under `drizzle-kit +-- migrate`, which wraps the pending set in a transaction, and PostgreSQL DDL is +-- transactional, so the abort rolls the whole set back. +-- +-- The guard locks the table before it counts, and its predicate is the +-- constraint predicate below under NOT and nothing else. Predicate equality +-- alone does not stop a row from clearing the guard and then failing the +-- constraint -- which would leave the operator with the bare violation after +-- being told the table was clean. The guard's SELECT and the constraint's +-- validation scan are separate statements, so under READ COMMITTED each takes +-- its own snapshot, and a row committed between the two is invisible to the +-- first and visible to the second. Whether ACCESS EXCLUSIVE closes that window +-- is the caller's to decide, not this file's, because the lock lasts only as +-- long as the transaction holding it. Under a caller that wraps the pending set +-- in one transaction -- `drizzle-kit migrate`, the production path -- the lock +-- is held across both statements, so a writer either commits before it is +-- granted, and is therefore counted, or waits until the migration ends. Under a +-- caller that executes each statement in autocommit, the lock is released when +-- this DO block commits and the ALTER TABLE re-acquires it, so the window +-- between them stays open and the guard narrows it rather than closing it. The +-- ALTER TABLE below takes that same mode, so the lock is acquired earlier rather +-- than newly introduced, and the added hold is the guard's own scan of a table +-- holding one row per tenant. Taking it up front also avoids upgrading ACCESS +-- SHARE to ACCESS EXCLUSIVE mid-transaction, which deadlocks against a +-- conflicting waiter that queues between the two. +-- +-- The column is NOT NULL, so the predicate is never null; were it ever null the +-- two would still agree, because a check admits a row whose predicate is null +-- and a WHERE does not select one. +-- +-- The list is capped. Nothing bounds how many rows a column that accepted any +-- string can have accumulated, and an unbounded message risks a log pipeline +-- truncating away the instructions, which are the part the operator needs, so +-- the instructions come first and the sample last. The total states whether the +-- sample is the whole set, and ordering by slug makes the sample stable, so +-- fixing the listed rows and re-running walks the set rather than resampling it. +-- +-- Each value is rendered as a JSON string rather than a SQL literal. The column +-- accepted any string, so a slug can hold a newline or a tab, and a SQL literal +-- reproduces those verbatim: the message would break across lines, which a log +-- pipeline reads as separate entries and may keep only the first of. JSON +-- escapes every control character, so the message stays one line and an +-- otherwise invisible slug is still legible. The operator keys the fix on the +-- tenant id rather than on the slug, so nothing here needs to be a SQL literal. +DO $$ +DECLARE + offending_total bigint; + offending_sample text; +BEGIN + LOCK TABLE "tenant" IN ACCESS EXCLUSIVE MODE; + + SELECT + count(*), + string_agg( + to_json(offending.slug)::text || ' (tenant ' || to_json(offending.id)::text || ')', + ', ' ORDER BY offending.slug + ) FILTER (WHERE offending.seq <= 25) + INTO offending_total, offending_sample + FROM ( + SELECT + "tenant"."id" AS id, + "tenant"."slug" AS slug, + row_number() OVER (ORDER BY "tenant"."slug") AS seq + FROM "tenant" + WHERE NOT ("tenant"."slug" ~ '^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?$' and length("tenant"."slug") <= 63) + ) AS offending; + + IF offending_total > 0 THEN + RAISE EXCEPTION '% tenant row(s) hold a slug that is not a single DNS label, so tenant_slug_dns_label_check cannot be added. Rename each slug to a single label -- a letter or a digit first and last, letters, digits and hyphens between, at most 63 characters, per RFC 1035 section 2.3.1 as relaxed by RFC 1123 section 2.1 -- then re-run the migration. Renaming a slug does not move the mail identity of a tenant: that is tenant.domain, a separate column this migration neither reads nor changes, and it is what carries the sender stamp on outbound mail and the recipient run address. The create route derives the domain of a new tenant as the lowercased slug followed by .localhost, so read each row before touching its domain: a domain still derived that way from the old slug wants moving too, and moving one that was not breaks whatever addresses it. Offending rows, lowest slug first, at most 25 shown -- fix these and re-run to see the next: %', offending_total, offending_sample; + END IF; +END $$; +--> statement-breakpoint +ALTER TABLE "tenant" ADD CONSTRAINT "tenant_slug_dns_label_check" CHECK ("tenant"."slug" ~ '^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?$' and length("tenant"."slug") <= 63); diff --git a/vendor/intx/db/migrations/meta/_journal.json b/vendor/intx/db/migrations/meta/_journal.json index e7157a2d3b..851cbb894a 100644 --- a/vendor/intx/db/migrations/meta/_journal.json +++ b/vendor/intx/db/migrations/meta/_journal.json @@ -687,6 +687,13 @@ "when": 1789466440782, "tag": "0098_sidecar_initialization", "breakpoints": true + }, + { + "idx": 99, + "version": "7", + "when": 1790350152048, + "tag": "0099_dark_old_lace", + "breakpoints": true } ] } diff --git a/vendor/intx/db/src/schema/tenants.ts b/vendor/intx/db/src/schema/tenants.ts index 61d53de377..52c7829ac5 100644 --- a/vendor/intx/db/src/schema/tenants.ts +++ b/vendor/intx/db/src/schema/tenants.ts @@ -1,5 +1,6 @@ import { sql } from "drizzle-orm"; import { + check, foreignKey, jsonb, pgTable, @@ -30,6 +31,15 @@ export const tenant = pgTable( (t) => [ foreignKey({ columns: [t.parentId], foreignColumns: [t.id] }), uniqueIndex("tenant_domain_lower_idx").on(sql`lower(${t.domain})`), + // Constrains `slug`, and only `slug`, to a single DNS label: RFC 1035 + // section 2.3.1 as relaxed by RFC 1123 section 2.1, the same grammar the + // `TenantSlug` validator applies at the create route, enforced here for + // every write path including the ones that bypass that route. `domain` + // carries no grammar constraint and nothing here relates the two columns. + check( + "tenant_slug_dns_label_check", + sql`${t.slug} ~ '^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?$' and length(${t.slug}) <= 63`, + ), ], ); diff --git a/vendor/intx/harness/VENDORED-FROM b/vendor/intx/harness/VENDORED-FROM index 3473f00e64..1b518c3534 100644 --- a/vendor/intx/harness/VENDORED-FROM +++ b/vendor/intx/harness/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/harness) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. No source delta. diff --git a/vendor/intx/harness/src/connector-router.ts b/vendor/intx/harness/src/connector-router.ts index b7f8009a27..db4f32087b 100644 --- a/vendor/intx/harness/src/connector-router.ts +++ b/vendor/intx/harness/src/connector-router.ts @@ -1,10 +1,13 @@ // Connector-thread routing for the agent harness. // -// The connector is one durable thread per agent. Participants accumulate -// as they speak; no one is displaced. `replyTo` tracks the most recent +// The connector is one durable thread per agent. On a thread holding an +// anchor -- a thread root or a last message id -- participants accumulate +// as they speak and no one is displaced: `replyTo` tracks the most recent // speaker (the primary recipient on the next outbound reply) and `cc` // tracks every other participant who has spoken (carried on outbound so -// everyone stays in the loop). +// everyone stays in the loop). A thread holding neither anchor offers +// nothing for a later message to continue, so the next arrival opens a +// fresh thread and the accumulated list starts over. // // Two-phase decision: route() is pure and returns a discriminated kind // plus an opaque carrier of the next state; commit() advances router @@ -30,7 +33,11 @@ export type RouteDecision = export type ConnectorReplyParts = { to: string; cc: string[]; - inReplyTo: string; + /** + * The nearest identified ancestor: the last message id, or the thread + * root when the last message named no id. Absent when neither exists. + */ + inReplyTo?: string; subject?: string; }; @@ -67,13 +74,16 @@ export interface ConnectorRouter { * does not mutate router state. The returned decision must be passed to * `commit()` to take effect. * - * Throws when `message.headers.from` is not a parseable bare addr-spec - * (per `extractAddrSpec` from `@intx/mime`). The production fetch path - * copies the wire `From:` header verbatim, so a malformed sender is a - * normal-shape runtime concern, not a programmer error. Callers should - * treat the throw as passthrough — deliver the message to the reactor - * but do not advance router state or consume the message from the - * INBOX. + * Returns `passthrough` when `message.headers.from` is absent: the + * message names nobody to reply to. + * + * Throws when it is present but is not a parseable bare addr-spec. + * Callers should treat the throw as passthrough — deliver the message + * but do not advance router state or consume it from the INBOX. + * + * Does not read `Interchange-Type`. A structured payload that names a + * `From` starts or continues the thread exactly as a conversation + * message does. */ route(message: InboundMessage): RouteDecision; @@ -170,7 +180,11 @@ export function createConnectorRouter( const { inReplyTo, references } = message.headers; - if (references !== undefined && references.includes(state.threadRoot)) { + if ( + state.threadRoot !== undefined && + references !== undefined && + references.includes(state.threadRoot) + ) { return true; } @@ -189,12 +203,39 @@ export function createConnectorRouter( return [...existing, value]; } + function speakerOf(message: InboundMessage): string | null { + const { from } = message.headers; + if (from === undefined) return null; + return extractAddrSpec(from); + } + + function passthroughWithNoReplyAddress( + message: InboundMessage, + ): RouteDecision { + logger.debug`Message ${message.headers.messageId} carries no From header, so it has no reply address; routing it as passthrough and leaving the connector thread unadvanced`; + return { kind: "passthrough" }; + } + + // A thread opened by a message that named no id holds neither anchor, so + // `isContinuation` can never match it again. The next message starts a + // fresh thread rather than being absorbed by it forever. + function isAnchored(s: ConnectorThreadState): boolean { + return s.threadRoot !== undefined || s.lastMessageId !== undefined; + } + function route(message: InboundMessage): RouteDecision { - if (state === null) { + if (state === null || !isAnchored(state)) { + const replyTo = speakerOf(message); + if (replyTo === null) return passthroughWithNoReplyAddress(message); + const nextState: ConnectorThreadState = { - threadRoot: message.headers.messageId, - lastMessageId: message.headers.messageId, - replyTo: extractAddrSpec(message.headers.from), + ...(message.headers.messageId !== undefined + ? { + threadRoot: message.headers.messageId, + lastMessageId: message.headers.messageId, + } + : {}), + replyTo, cc: [], ...(message.headers.subject !== undefined ? { subject: message.headers.subject } @@ -206,7 +247,9 @@ export function createConnectorRouter( } if (isContinuation(message)) { - const nextSpeaker = extractAddrSpec(message.headers.from); + const nextSpeaker = speakerOf(message); + if (nextSpeaker === null) return passthroughWithNoReplyAddress(message); + // The previous most-recent speaker moves into the cc list; the // new speaker becomes replyTo. Dedup so a sender returning after // others have spoken doesn't appear twice. @@ -214,8 +257,12 @@ export function createConnectorRouter( (addr) => addr !== nextSpeaker, ); const nextState: ConnectorThreadState = { - threadRoot: state.threadRoot, - lastMessageId: message.headers.messageId, + ...(state.threadRoot !== undefined + ? { threadRoot: state.threadRoot } + : {}), + ...(message.headers.messageId !== undefined + ? { lastMessageId: message.headers.messageId } + : {}), replyTo: nextSpeaker, cc: carriedCc, ...(state.subject !== undefined ? { subject: state.subject } : {}), @@ -247,10 +294,15 @@ export function createConnectorRouter( throw new NoActiveConnectorThreadError(); } + // RFC 5322 3.6.4: a parent that named no Message-ID leaves the root as + // the nearest identified ancestor, and the References chain built from + // it is what keeps the reply on the thread. + const parent = state.lastMessageId ?? state.threadRoot; + return { to: state.replyTo, cc: [...state.cc], - inReplyTo: state.lastMessageId, + ...(parent !== undefined ? { inReplyTo: parent } : {}), ...(state.subject !== undefined ? { subject: state.subject } : {}), }; } @@ -260,7 +312,9 @@ export function createConnectorRouter( throw new NoActiveConnectorThreadError(); } applyState({ - threadRoot: state.threadRoot, + ...(state.threadRoot !== undefined + ? { threadRoot: state.threadRoot } + : {}), lastMessageId: receipt.messageId, replyTo: state.replyTo, cc: [...state.cc], @@ -271,8 +325,12 @@ export function createConnectorRouter( function snapshot(): ConnectorThreadState | null { if (state === null) return null; return { - threadRoot: state.threadRoot, - lastMessageId: state.lastMessageId, + ...(state.threadRoot !== undefined + ? { threadRoot: state.threadRoot } + : {}), + ...(state.lastMessageId !== undefined + ? { lastMessageId: state.lastMessageId } + : {}), replyTo: state.replyTo, cc: [...state.cc], ...(state.subject !== undefined ? { subject: state.subject } : {}), @@ -284,8 +342,12 @@ export function createConnectorRouter( next === null ? null : { - threadRoot: next.threadRoot, - lastMessageId: next.lastMessageId, + ...(next.threadRoot !== undefined + ? { threadRoot: next.threadRoot } + : {}), + ...(next.lastMessageId !== undefined + ? { lastMessageId: next.lastMessageId } + : {}), replyTo: next.replyTo, cc: [...next.cc], ...(next.subject !== undefined ? { subject: next.subject } : {}), diff --git a/vendor/intx/harness/src/harness.ts b/vendor/intx/harness/src/harness.ts index 773dfa3cfd..0d9cceece1 100644 --- a/vendor/intx/harness/src/harness.ts +++ b/vendor/intx/harness/src/harness.ts @@ -29,6 +29,7 @@ import type { ContextStore, InboundMessage, InferenceSource, + MessageRef, MessageTransport, Unsubscribe, } from "@intx/types/runtime"; @@ -38,6 +39,23 @@ import { driveConnectorReplies } from "./reply-drain"; const logger = getLogger(["interchange", "harness"]); +/** + * Interchange keyword flag the INBOX watch puts on a message whose + * `transport.fetchFull` threw. The message stays in the INBOX rather than being + * consumed, because part of what makes a fetch throw is an envelope the sender + * chose and the rest is local (a uid a concurrent expunge removed, a faulting + * read), so deleting on it would put a peer in charge of whether its own mail + * survives. + * + * The keyword is what keeps the watch from attempting the same uid again in the + * run that flagged it, on the premise that a transport delivers a second + * arrival event for a uid already resident in the mailbox -- which no transport + * here does. Nothing in this package re-attempts a flagged message either, so + * the flag preserves the mail and surfaces it to a search for + * `hasFlags: [MAIL_FETCH_FAILED_FLAG]`, but gives it no path back to delivery. + */ +export const MAIL_FETCH_FAILED_FLAG = "$FetchFailed"; + /** * Env extension the composition layer requires beyond `BaseEnv`. Tools * shipped by this package declare the matching `requires` so @@ -327,27 +345,40 @@ export async function createHarness( : {}), }); - // Delete a message from the INBOX after it has been delivered to the - // reactor. - // - // A failure here is logged and swallowed: the router state has - // already been committed and `agent.deliver` has accepted the - // message, so re-raising would unwind a half-applied delivery. The - // message stays in the INBOX and a future startup (or watch firing) - // re-fetches it, re-routes it, and re-delivers it. The router's - // persisted state makes that benign on the routing side: the sender - // is already a thread participant, so `route()` returns either a - // `continue` (which is a no-op state mutation since the sender is - // unchanged) or a `passthrough` (no headers match). The agent's - // director sees a duplicate `message.received`; idempotent - // directors are unaffected, and the audit trail records the - // duplicate for post-hoc reconciliation. - async function consumeFromInbox(message: InboundMessage): Promise { + // A failure here is logged and swallowed: the router state is already + // committed and `agent.deliver` has accepted the message, so re-raising + // would unwind a half-applied delivery. A redelivery shows the director + // a duplicate `message.received`. + async function consumeFromInbox(ref: MessageRef): Promise { try { - await transport.setFlags(message.ref, ["\\Deleted"]); + await transport.setFlags(ref, ["\\Deleted"]); await transport.expunge("INBOX"); } catch (cause) { - logger.warn`Failed to consume message uid=${message.ref.uid} from INBOX: ${cause}`; + logger.warn`Failed to consume message uid=${ref.uid} from INBOX: ${cause}`; + } + } + + // Whether an earlier attempt on this uid already failed its fetch and + // said so on the message. The transport is asked rather than an + // in-process uid set because that is where the record lives: on the + // message, which outlives the process that wrote it. That record is + // what a search on `MAIL_FETCH_FAILED_FLAG` surfaces to an operator. + async function fetchAlreadyFailed(ref: MessageRef): Promise { + const flagged = await transport.search("INBOX", { + hasFlags: [MAIL_FETCH_FAILED_FLAG], + }); + return flagged.some((candidate) => candidate.uid === ref.uid); + } + + // A flag write that fails leaves the message unflagged, so a later + // arrival event for the same uid attempts the fetch again. That is the + // safe direction to fail in: the message is still in the INBOX, and the + // attempt that comes after a transient fault may well deliver it. + async function markFetchFailed(ref: MessageRef): Promise { + try { + await transport.setFlags(ref, [MAIL_FETCH_FAILED_FLAG]); + } catch (cause) { + logger.warn`Failed to flag unfetchable message uid=${ref.uid} in INBOX: ${cause}`; } } @@ -363,11 +394,33 @@ export async function createHarness( void (async () => { try { + if (await fetchAlreadyFailed(ref)) { + // An earlier attempt on this uid failed and said so on the + // message. Part of what makes a fetch fail is permanent, and the + // watch cannot tell that part from the rest, so it leaves the + // message flagged in the INBOX and delivers nothing. + logger.debug`Skipping message uid=${event.uid}: an earlier fetch failed and flagged it ${MAIL_FETCH_FAILED_FLAG}`; + return; + } + if (stopped) return; + let message: InboundMessage; try { message = await transport.fetchFull(ref); } catch (cause) { - logger.error`Failed to fetch message uid=${event.uid}: ${cause}`; + // What reaches here is not only a message the transport could not + // assemble. The same throw covers a uid a concurrent expunge + // already removed, where there is nothing left to consume, and a + // faulting read of sound bytes, where a later attempt could still + // succeed. Only the assembly case is input a peer chooses, and + // consuming on it would let a peer delete its own mail out of the + // INBOX by malforming one header -- so the message is flagged and + // left where it is. Inventing the flags and signature status the + // transport never read would vouch for a message it did not, so + // the message is not delivered either. + logger.error`Failed to fetch message uid=${event.uid}; flagging it ${MAIL_FETCH_FAILED_FLAG} and leaving it in the INBOX: ${cause}`; + if (stopped) return; + await markFetchFailed(ref); return; } @@ -403,7 +456,7 @@ export async function createHarness( connectorRouter.commit(decision); if (stopped) return; agent.deliver(message); - await consumeFromInbox(message); + await consumeFromInbox(message.ref); } catch (cause) { // `agent.deliver` throws `AgentClosedError` synchronously when // called after the agent has closed. The `if (stopped) return` diff --git a/vendor/intx/harness/src/index.ts b/vendor/intx/harness/src/index.ts index 6ea215ab46..4d82eb160e 100644 --- a/vendor/intx/harness/src/index.ts +++ b/vendor/intx/harness/src/index.ts @@ -1,6 +1,7 @@ export { createHarness, defineMailTools, + MAIL_FETCH_FAILED_FLAG, type Harness, type MailEnv, type MailToolWrapper, diff --git a/vendor/intx/harness/src/reply-drain.ts b/vendor/intx/harness/src/reply-drain.ts index 8c604a0649..6cf031a653 100644 --- a/vendor/intx/harness/src/reply-drain.ts +++ b/vendor/intx/harness/src/reply-drain.ts @@ -245,7 +245,8 @@ export function driveConnectorReplies( // (parent absent, malformed id) yields `undefined`, and the // transport derives `[inReplyTo]` as before. const references = - opts.resolveReferences !== undefined + opts.resolveReferences !== undefined && + parts.inReplyTo !== undefined ? await opts.resolveReferences(parts.inReplyTo) : undefined; const receipt = await opts.send({ diff --git a/vendor/intx/hub-agent/VENDORED-FROM b/vendor/intx/hub-agent/VENDORED-FROM index 02adf38a51..473ab79149 100644 --- a/vendor/intx/hub-agent/VENDORED-FROM +++ b/vendor/intx/hub-agent/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/hub-agent) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. CL-6396: SessionEventSink and createSidecarOrchestrator's dispatchEvent/publishWorkflowInferenceEvent take an optional childRunId, and createHubLink's sendEvent stamps it on agent.event so overlapping turns close the right occurrence row. +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). CL-6396: SessionEventSink and createSidecarOrchestrator's dispatchEvent/publishWorkflowInferenceEvent take an optional childRunId, and createHubLink's sendEvent stamps it on agent.event so overlapping turns close the right occurrence row. CL-7508: createSidecarOrchestrator takes an optional oauthLoginExecutor and forwards it to createHubLink, which routes every inbound oauth.login.start through it (started arm on authorize URL, exactly one terminal arm); omitted, a fail-closed placeholder answers with an error instead of hanging. Absorbed upstream at this re-vendor: the inbound-mail-policy-registry mismatchedFrom/absentFrom reject defaults, the inbound-signature AuthorControllableOutcome findings, and the sender-key-cache load-fault reporting. diff --git a/vendor/intx/hub-agent/src/sender-crypto.ts b/vendor/intx/hub-agent/src/sender-crypto.ts index 1e0c9b5bfb..d13e996eb6 100644 --- a/vendor/intx/hub-agent/src/sender-crypto.ts +++ b/vendor/intx/hub-agent/src/sender-crypto.ts @@ -46,6 +46,12 @@ export function createPublicKeyCrypto(publicKey: Uint8Array): CryptoProvider { * signature matches the mailbox verify seam's `getCrypto`, so the resolver is a * drop-in source of the sender's verification key. `undefined` is the seam's * defined "no key for this address" sentinel, not a swallowed failure. + * + * A cache read that THROWS -- the address's on-disk entry failed to load -- is + * left to propagate, because that is a fault about material the cache was given + * and cannot serve, not the absence `undefined` reports. The inbound-verify + * caller turns it into its refusing verdict; flattening it to `undefined` here + * would hand the caller a sender condition instead. */ export function createSenderCryptoResolver( cache: SenderKeyCache, diff --git a/vendor/intx/hub-agent/src/sender-key-cache.ts b/vendor/intx/hub-agent/src/sender-key-cache.ts index 224652981c..03aa7f694a 100644 --- a/vendor/intx/hub-agent/src/sender-key-cache.ts +++ b/vendor/intx/hub-agent/src/sender-key-cache.ts @@ -25,18 +25,44 @@ // every previously-cached key. The on-disk filename is the hex-encoded // address bytes: an injective, reversible handle, so two addresses never // collide onto one file the way the lossy `sanitizeAddress` scheme would. +// +// A file that FAILS to load is not the same condition as a sender with no +// cached key, and the cache must not report it as one: the inbound-verify path +// reads an absent address as "no key was available", which a workflow author can +// relax to admit, so dropping the entry would turn an operator's corrupt file +// into mail admitted with no signature check. The load therefore records the +// fault against the address and `get` THROWS for it; the verify path's own fault +// handling is what turns that throw into the verdict no policy relaxes. A fault +// clears when a key is written for the address or the address is evicted, and +// `rotatableAddresses` reports it for the reconnect re-resolve so the hub's +// re-push replaces the unreadable file. import fsp from "node:fs/promises"; import path from "node:path"; import { type } from "arktype"; import { getLogger } from "@intx/log"; -import { hasCode, hexDecode, hexEncode, isRunAddress } from "@intx/types"; +import { + hasCode, + hexDecode, + hexEncode, + isRunAddress, + parseAddress, +} from "@intx/types"; const logger = getLogger(["interchange", "hub-agent", "sender-key-cache"]); // A raw Ed25519 public key is 32 bytes. const ED25519_PUBLIC_KEY_BYTES = 32; +/** + * Length is the whole rule: a 32-byte value that is not the signer's key + * imports fine and verifies to false, which is a check that ran and failed + * rather than unusable material. + */ +export function isUsableSenderKey(publicKey: Uint8Array): boolean { + return publicKey.length === ED25519_PUBLIC_KEY_BYTES; +} + const SENDER_KEYS_DIR_NAME = "sender-keys"; // The on-disk envelope carries the address alongside the key so a loaded @@ -71,13 +97,19 @@ export type SenderKeyCache = { * A synchronous memory read: the keyring is loaded at construction and * every `put` updates memory, so the map is authoritative for this * single-process sidecar. + * + * THROWS when the address's on-disk entry failed to load. `undefined` means + * the cache was never given a key for this sender, which a caller may treat as + * a sender condition; a load fault is an operator condition about material the + * cache was given and cannot serve, so the two must not share one answer. */ get(address: string): Uint8Array | undefined; /** * Cache `publicKey` for `address`, persisting it durably before returning. * Throws if the durable write fails, so a caller that gates a downstream * durable write on this one can rely on "this key is on disk" once it - * resolves. + * resolves. A successful write clears any load fault recorded for `address`: + * the file it replaces is the one that failed to load. */ put(address: string, publicKey: Uint8Array): Promise; /** @@ -86,7 +118,10 @@ export type SenderKeyCache = { * the map is rebuilt from disk on construction, so if the durable removal * fails and throws, both stores still hold the key (consistent, and the next * reconnect retries) rather than a memory-evicted key resurrecting from disk - * on restart. A no-op for an address the cache does not hold. + * on restart. A no-op for an address the cache does not hold. A successful + * removal also clears any load fault recorded for `address`, and for the same + * disk-first reason: the fault outlives a failed removal, because the file + * that caused it is still there. */ evict(address: string): Promise; /** @@ -95,18 +130,21 @@ export type SenderKeyCache = { * change an array a caller is still holding -- the reported set stays stable * across an `await`. The cache reports addresses only; the key values stay * inside because the refresh-on-reconnect flow re-resolves each current key - * hub-side rather than trusting the cached (possibly stale) one. + * hub-side rather than trusting the cached (possibly stale) one. An address + * whose entry failed to load is NOT here: the cache holds no key for it. */ addresses(): string[]; /** - * The cached addresses whose key can rotate: `addresses()` minus run - * addresses. A run address resolves to the immutable - * `workflow_run.public_key`, so its cached key never changes; a user sender - * resolves to a hub principal key that can rotate. The reconnect reporter - * sends exactly this set so the hub re-resolves only the senders whose key - * could have changed while the sidecar was disconnected. This is a - * work-saving filter, not a trust boundary -- the hub resolves every reported - * address on its own. + * The cached addresses the hub should re-resolve on reconnect, minus run + * addresses. Two kinds qualify: an address the cache holds a key for, because + * a user sender's hub principal key can rotate while the sidecar is + * disconnected; and an address whose on-disk entry failed to load, because the + * hub's re-push is what replaces the file the cache cannot read. Run addresses + * are excluded in both cases -- a run sender's key is the immutable + * `workflow_run.public_key`, the hub skips a reported run address whatever this + * returns, and a run sender's next grants barrier re-pushes its key anyway, + * which repairs a faulted file through `put`. This is a work-saving filter, not + * a trust boundary -- the hub resolves every reported address on its own. */ rotatableAddresses(): string[]; }; @@ -117,6 +155,12 @@ export async function createSenderKeyCache( const { dataDir, writeFileDurable, removeFileDurable } = deps; const dir = path.join(dataDir, SENDER_KEYS_DIR_NAME); const keys = new Map(); + // Addresses whose on-disk entry failed to load, mapped to the reason. Disjoint + // from `keys`: a failed load stores no key, and `put` and `evict` clear the + // fault for the address they touch after their disk work lands, so no address + // is ever in both. `rotatableAddresses` relies on that to concatenate this + // map's keys onto `addresses()` without deduplicating them. + const loadFaults = new Map(); function keyPath(address: string): string { return path.join(dir, hexEncode(new TextEncoder().encode(address))); @@ -144,10 +188,12 @@ export async function createSenderKeyCache( ): Promise<{ address: string; publicKey: Uint8Array } | null> { // A crashed atomic write can leave a `.tmp..` file; its // name is not valid hex, so decoding the filename rejects it below. A - // corrupt or mismatched file is logged and skipped rather than failing the - // whole load -- one bad entry must not deny the sidecar its other keys. + // corrupt or mismatched file does not fail the whole load -- one bad entry + // must not deny the sidecar its other keys -- but it is not simply dropped + // either; the catch attributes it to an address so `get` can refuse it. + let filenameAddress: string | null = null; try { - const filenameAddress = new TextDecoder().decode(hexDecode(entry)); + filenameAddress = new TextDecoder().decode(hexDecode(entry)); const raw = await fsp.readFile(path.join(dir, entry), "utf8"); const parsed = StoredSenderKey(JSON.parse(raw)); if (parsed instanceof type.errors) { @@ -159,7 +205,7 @@ export async function createSenderKeyCache( ); } const publicKey = hexDecode(parsed.publicKey); - if (publicKey.length !== ED25519_PUBLIC_KEY_BYTES) { + if (!isUsableSenderKey(publicKey)) { throw new Error( `expected a ${ED25519_PUBLIC_KEY_BYTES}-byte key, got ${publicKey.length}`, ); @@ -167,12 +213,33 @@ export async function createSenderKeyCache( return { address: parsed.address, publicKey }; } catch (cause) { const message = cause instanceof Error ? cause.message : String(cause); - logger.error`Skipping unreadable sender-key file ${entry}: ${message}`; + // Attributed to the address the FILENAME names, whatever the envelope + // claims: that is the address a reader asks for, and the one whose file + // this is. Only the fault is attributed here -- an entry that loads is + // still keyed on the envelope address the cross-check just proved equal. + const faultedAddress = + filenameAddress !== null && parseAddress(filenameAddress) !== null + ? filenameAddress + : null; + if (faultedAddress === null) { + // Debris rather than an entry: no address to refuse reads for, and + // nothing the hub could re-push to repair. + logger.error`Skipping unreadable sender-key file ${entry}: ${message}`; + return null; + } + loadFaults.set(faultedAddress, message); + logger.error`Sender-key file ${entry} for ${faultedAddress} did not load, so this sidecar refuses to verify that sender's mail until the hub re-pushes the key: ${message}`; return null; } } function get(address: string): Uint8Array | undefined { + const fault = loadFaults.get(address); + if (fault !== undefined) { + throw new Error( + `the cached sender key for ${JSON.stringify(address)} failed to load from the keyring: ${fault}`, + ); + } return keys.get(address); } @@ -181,11 +248,13 @@ export async function createSenderKeyCache( } function rotatableAddresses(): string[] { - return addresses().filter((address) => !isRunAddress(address)); + return [...addresses(), ...loadFaults.keys()].filter( + (address) => !isRunAddress(address), + ); } async function put(address: string, publicKey: Uint8Array): Promise { - if (publicKey.length !== ED25519_PUBLIC_KEY_BYTES) { + if (!isUsableSenderKey(publicKey)) { throw new Error( `refusing to cache a ${publicKey.length}-byte key for ${address}; expected ${ED25519_PUBLIC_KEY_BYTES}`, ); @@ -197,11 +266,13 @@ export async function createSenderKeyCache( await fsp.mkdir(dir, { recursive: true }); await writeFileDurable(keyPath(address), contents); keys.set(address, publicKey); + loadFaults.delete(address); } async function evict(address: string): Promise { await removeFileDurable(keyPath(address)); keys.delete(address); + loadFaults.delete(address); } await loadFromDisk(); diff --git a/vendor/intx/hub-agent/src/ws/hub-link.ts b/vendor/intx/hub-agent/src/ws/hub-link.ts index 573aaff3b9..c2a0a684e6 100644 --- a/vendor/intx/hub-agent/src/ws/hub-link.ts +++ b/vendor/intx/hub-agent/src/ws/hub-link.ts @@ -50,6 +50,7 @@ import { import { verifyInboundSignature, outcomeForVerdict, + decideInboundAdmission, type ResolvedInboundMailPolicy, } from "./inbound-signature"; import { base64Decode, base64Encode } from "@intx/types"; @@ -1695,22 +1696,20 @@ export function createHubLink(config: HubLinkConfig): HubLink { }, resolveSenderCrypto, ); - const outcome = outcomeForVerdict(verdict); // The recipient deployment's resolved admission policy. `frame. // agentAddress` is the mail-router registration key, so an address // with no registered deployment resolves to the fully-closed policy - // and rejects every outcome. The map is total, so index it directly. - // The seam admits ONLY an explicitly-admitted outcome and drops - // everything else: the fail direction is closed by construction, so a - // non-`admit` value (today only `reject`, but any future non-admit - // verdict too) drops rather than leaks through. + // and rejects every outcome. const policy = lookupInboundMailPolicy(frame.agentAddress); - if (policy[outcome] !== "admit") { + const admission = decideInboundAdmission(verdict, policy); + if (admission.rejectedBy !== null) { logger.warn( - "Rejecting inbound mail for {agentAddress}: outcome {outcome} is not admitted (authenticatedSender {authenticatedSender}, messageId {messageId})", + "Rejecting inbound mail for {agentAddress}: {rejectedBy} is not admitted (findings {findings}, headline outcome {outcome}, authenticatedSender {authenticatedSender}, messageId {messageId})", { agentAddress: frame.agentAddress, - outcome, + rejectedBy: admission.rejectedBy, + findings: admission.findings, + outcome: outcomeForVerdict(verdict), authenticatedSender: frame.authenticatedSender, messageId: frame.messageId ?? null, }, diff --git a/vendor/intx/hub-agent/src/ws/inbound-mail-policy-registry.ts b/vendor/intx/hub-agent/src/ws/inbound-mail-policy-registry.ts index 1498cc0e2d..171f23581f 100644 --- a/vendor/intx/hub-agent/src/ws/inbound-mail-policy-registry.ts +++ b/vendor/intx/hub-agent/src/ws/inbound-mail-policy-registry.ts @@ -28,6 +28,8 @@ export const FULLY_CLOSED_INBOUND_MAIL_POLICY: ResolvedInboundMailPolicy = { clean: "reject", error: "reject", untrustedFrom: "reject", + mismatchedFrom: "reject", + absentFrom: "reject", invalid: "reject", missing: "reject", unknown: "reject", diff --git a/vendor/intx/hub-agent/src/ws/inbound-signature.ts b/vendor/intx/hub-agent/src/ws/inbound-signature.ts index 82fd8942a4..e19f3e184d 100644 --- a/vendor/intx/hub-agent/src/ws/inbound-signature.ts +++ b/vendor/intx/hub-agent/src/ws/inbound-signature.ts @@ -1,5 +1,6 @@ import { getLogger } from "@intx/log"; import type { + AuthorControllableOutcome, CryptoProvider, InboundMailOutcome, InboundMailPolicy, @@ -7,6 +8,8 @@ import type { import { verifyMimeSignature } from "@intx/mailbox"; import { parseHeaderSection, extractAddrSpec } from "@intx/mime"; +import { isUsableSenderKey } from "../sender-key-cache"; + const logger = getLogger([ "interchange", "hub-agent", @@ -15,56 +18,27 @@ const logger = getLogger([ ]); /** - * The two-axis verdict of verifying one inbound mail frame. - * - * `signature` reuses the `SignatureStatus` vocabulary -- - * `valid | invalid | missing | unknown` -- with an added `error` for a fault in - * the verifier itself (verify throwing, an unparseable sender). It answers: does - * the message's detached signature verify against the key the recipient's local - * cache holds for `authenticatedSender`? `unknown` means the cache holds no key - * for the sender, so there is nothing to verify against. - * - * `fromMatch` is an orthogonal axis over the message's visible `From`, evaluated - * for EVERY non-error signature status (valid, invalid, missing, unknown): - * - `unchecked`: the message carries no `From`, OR it carries a parseable - * `From` but the signature is not `valid`, so a match is not meaningful. - * - `unparseable`: the message carries a `From` that cannot be reduced to one - * addr-spec. This is DISTINCT from `unchecked` (no `From` at all) -- a - * present but unparseable `From` is suspicious, not benign. - * - `match` / `mismatch`: only atop a VALID signature with a parseable `From`, - * whether that `From` binds to `authenticatedSender`. A valid signature over - * a `From` that names a different sender is an identity forgery -- a - * legitimate key signing under a borrowed display identity. + * The two-axis verdict of verifying one inbound mail frame. `absent` is the + * gate looking for a visible `From` and finding none; `notEvaluated` is the + * placeholder an `error` verdict carries for an axis the gate never reached. * * The signature covers only the message's signed content part, NOT its - * top-level `From` header (see `@intx/mime` `assembleMessage`). The binding - * checked here is therefore the hub-stamped sender against the visible envelope - * `From`, not a `From` inside the signed bytes. + * top-level `From` header, so `fromMatch` binds the hub-stamped sender to the + * visible envelope `From` rather than to a `From` inside the signed bytes. */ export type InboundSignatureVerdict = { signature: "valid" | "invalid" | "missing" | "unknown" | "error"; - fromMatch: "match" | "mismatch" | "unchecked" | "unparseable"; + fromMatch: "match" | "mismatch" | "absent" | "notEvaluated" | "unparseable"; authenticatedSender: string; messageFrom: string | null; }; /** * Reduce a two-axis {@link InboundSignatureVerdict} to the single - * {@link InboundMailOutcome} a delivery decision keys on. - * - * The precedence is reject-dominant: when more than one axis is unhappy, the - * outcome is the one that most resists admission. Two outcomes dominate, for - * distinct reasons: - * - `error` outranks everything -- a fault stopped the check from running, so - * we could not check the message and can make no trust claim about it. - * - `untrustedFrom` outranks the signature axis -- a `From` we cannot trust - * (present but unparseable, or a valid signature worn under a mismatched - * identity) is a forgery signal that must not be masked by a merely - * unverifiable signature underneath it. - * - * `match`/`mismatch` only ever occur atop a `valid` signature (a guarantee of - * how the verdict is produced), while `unparseable` can accompany any non-error - * signature status; the ordering below reflects both facts. + * {@link InboundMailOutcome} that best SUMMARIZES it, for the verdict log line. + * This is a headline, NOT the admission decision -- that is + * {@link decideInboundAdmission}, and the two can name different findings. The + * ordering below is not a security precedence and must not be read as one. */ export function outcomeForVerdict( verdict: InboundSignatureVerdict, @@ -72,24 +46,22 @@ export function outcomeForVerdict( const { signature, fromMatch } = verdict; if (signature === "error") return "error"; if (fromMatch === "unparseable") return "untrustedFrom"; - if (signature === "valid" && fromMatch === "mismatch") return "untrustedFrom"; + if (fromMatch === "mismatch") return "mismatchedFrom"; + if (signature === "valid") { + if (fromMatch === "match") return "clean"; + if (fromMatch === "absent") return "absentFrom"; + if (fromMatch === "notEvaluated") return "untrustedFrom"; + const _exhaustiveBinding: never = fromMatch; + return _exhaustiveBinding; + } if (signature === "invalid") return "invalid"; if (signature === "missing") return "missing"; if (signature === "unknown") return "unknown"; - return "clean"; + const _exhaustive: never = signature; + return _exhaustive; } -/** - * A TOTAL admission decision map: for EVERY {@link InboundMailOutcome}, whether a - * message that resolved to that outcome is `reject`ed or `admit`ted. The per-mail - * delivery decision looks this up directly by the message's outcome with no - * fallback -- every key is present, so there is never an absent value for the - * lookup to default. - * - * This is the resolved counterpart of the SPARSE authored `InboundMailPolicy`: - * the sparse policy carries only what an author declared, and - * {@link resolveInboundMailPolicy} expands it into this total map ONCE. - */ +/** The total counterpart of the SPARSE authored `InboundMailPolicy`. */ export type ResolvedInboundMailPolicy = Record< InboundMailOutcome, "reject" | "admit" @@ -97,25 +69,17 @@ export type ResolvedInboundMailPolicy = Record< /** * Resolve the SPARSE authored {@link InboundMailPolicy} into a TOTAL - * {@link ResolvedInboundMailPolicy}, applying every default HERE. This is the - * single place inbound-admission defaults live: the resolution runs once, and - * the per-mail delivery path looks up the resolved map directly -- it must never - * re-derive a default with a `?? "reject"` of its own. + * {@link ResolvedInboundMailPolicy}, applying every default HERE. The per-mail + * delivery path looks up the resolved map directly -- it must never re-derive a + * default with a `?? "reject"` of its own. * - * The two non-author-controllable outcomes are pinned regardless of what the - * author declared: - * - `clean` -> always `admit`: nothing about the message was suspect, so there - * is nothing to relax and no reason to reject. - * - `error` -> always `reject`: a fault stopped the check from running, so we - * could make no trust claim about the message. `error` is not even a key in - * {@link InboundMailPolicy}, so an authored policy cannot relax it -- a - * message we could not check through is never something an author waves past. + * `clean` and `error` are pinned whatever the author declared: nothing about a + * clean message is suspect, and a fault stopped the check from running, so no + * trust claim can be made about it. `error` is not a key in + * {@link InboundMailPolicy} at all, so an authored policy cannot relax it. * - * The four author-controllable outcomes (`untrustedFrom`, `invalid`, `missing`, - * `unknown`) take the authored value where the author set that key, and default - * to `reject` otherwise. `reject` is the secure default: an outcome the author - * did not explicitly choose to admit stays rejected, so an omitted policy (or a - * policy that omits one outcome) fails closed rather than open. + * Every author-controllable outcome defaults to `reject`, so a policy that + * omits one fails closed rather than open. */ export function resolveInboundMailPolicy( authored: InboundMailPolicy | undefined, @@ -124,12 +88,75 @@ export function resolveInboundMailPolicy( clean: "admit", error: "reject", untrustedFrom: authored?.untrustedFrom ?? "reject", + mismatchedFrom: authored?.mismatchedFrom ?? "reject", + absentFrom: authored?.absentFrom ?? "reject", invalid: authored?.invalid ?? "reject", missing: authored?.missing ?? "reject", unknown: authored?.unknown ?? "reject", }; } +export type InboundAdmission = { + findings: AuthorControllableOutcome[]; + rejectedBy: InboundMailOutcome | null; +}; + +/** + * Decide whether one inbound mail frame is admitted, over the SET of findings + * its verdict raised rather than over a single reduced outcome. Each axis + * carries a separate author judgement, so reducing the pair to one outcome and + * keying admission on that discards one of the two judgements -- and it can + * discard it in the ADMITTING direction. + */ +export function decideInboundAdmission( + verdict: InboundSignatureVerdict, + policy: ResolvedInboundMailPolicy, +): InboundAdmission { + const { signature, fromMatch } = verdict; + if (signature === "error") return { findings: [], rejectedBy: "error" }; + + const findings: AuthorControllableOutcome[] = []; + const fromSignature = signatureFinding(signature); + if (fromSignature !== null) findings.push(fromSignature); + const fromBinding = bindingFinding(fromMatch); + if (fromBinding !== null) findings.push(fromBinding); + + for (const finding of findings) { + if (policy[finding] !== "admit") return { findings, rejectedBy: finding }; + } + // A binding that raised nothing without binding was never evaluated, whatever + // the signature axis found. The gate has no trust claim to make for it, and no + // policy relaxes that. + if (fromBinding === null && fromMatch !== "match") { + return { findings, rejectedBy: "error" }; + } + if (policy.clean !== "admit") return { findings, rejectedBy: "clean" }; + return { findings, rejectedBy: null }; +} + +function signatureFinding( + signature: Exclude, +): AuthorControllableOutcome | null { + if (signature === "valid") return null; + if (signature === "invalid") return "invalid"; + if (signature === "missing") return "missing"; + if (signature === "unknown") return "unknown"; + const _exhaustive: never = signature; + return _exhaustive; +} + +function bindingFinding( + fromMatch: InboundSignatureVerdict["fromMatch"], +): AuthorControllableOutcome | null { + if (fromMatch === "match") return null; + if (fromMatch === "notEvaluated") return null; + if (fromMatch === "mismatch") return "mismatchedFrom"; + if (fromMatch === "unparseable") return "untrustedFrom"; + if (fromMatch === "absent") return "absentFrom"; + const _exhaustive: never = fromMatch; + return _exhaustive; +} + export type InboundSignatureInput = { raw: Uint8Array; authenticatedSender: string; @@ -148,14 +175,7 @@ export type InboundSignatureInput = { * This NEVER throws. A fault degrades to an `error` verdict, logged at ERROR * and returned like any other verdict. The enforcement caller relies on this * contract: it awaits this inline on the delivery path with no per-call catch, - * gates admission on the returned outcome, and drops the mail on a reject, so a - * throw that escaped here would wedge the delivery chain. Returns the verdict so - * the caller (or a test) can read it without scraping the log. - * - * A cache miss is a quiet `unknown` (an expected, benign state -- see below), - * not a fault. A genuine fault (the resolver throwing, the cached key being - * unreadable, or the verify throwing) degrades to an `error` verdict logged at - * ERROR -- surfaced loudly and kept distinct from `unknown`. + * so a throw that escaped here would wedge the delivery chain. */ export async function verifyInboundSignature( input: InboundSignatureInput, @@ -165,43 +185,36 @@ export async function verifyInboundSignature( let verdict: InboundSignatureVerdict; try { - // Resolve and read the cached key inside the try so ANY fault -- the - // resolver throwing, `getPublicKey` throwing, or the verify throwing -- - // is contained as a single `error` verdict rather than escaping. This is - // what keeps the "never throws" contract true. + // Parsed above the cache lookup and above every early return below, so the + // fault belongs on every path. + const stampedAddrSpec = parseSenderStamp(authenticatedSender); + // Keyed on the RAW stamp, not on `stampedAddrSpec`: the cache is populated + // under the address the hub sends, and `extractAddrSpec` lowercases. const crypto = resolveSenderCrypto(authenticatedSender); + let signature: InboundSignatureVerdict["signature"]; if (crypto === undefined) { - // Cache miss: the local keyring holds no key for this sender, so there - // is nothing to verify against. Expected for a sender whose key was - // never co-delivered (a run authorized before this shipped, relayed mail - // with no preceding grant co-delivery) or was unresolvable, or a - // rotation the cache has not yet refreshed. The mail is admitted as an - // unverifiable sender -- a quiet `unknown`, keyed by `authenticatedSender` - // in the log so the observation window stays legible. - verdict = { - signature: "unknown", - fromMatch: "unchecked", - authenticatedSender, - messageFrom: null, - }; + signature = "unknown"; } else { - // The cached key is raw bytes, already validated 32-byte Ed25519 at cache - // write/load time, so it feeds `verifyMimeSignature` directly. - const signature = await verifyMimeSignature(raw, crypto.getPublicKey()); - verdict = { - signature, - fromMatch: "unchecked", - authenticatedSender, - messageFrom: null, - }; + const publicKey = crypto.getPublicKey(); + // The cache-backed resolver cannot reach this: that cache refuses a + // wrong-length key at write and THROWS for an entry that failed to load, + // so an address it answers for carries a usable key. The resolver is an + // injected seam, and a composition supplying its own is not covered by + // that, so the check stays. + requireUsableSenderKey(authenticatedSender, publicKey); + signature = await verifyMimeSignature(raw, publicKey); } - // Evaluate the visible From for EVERY non-error status. It records - // `messageFrom` and, atop a VALID signature, resolves the `fromMatch` - // binding; a present-but-unparseable From is marked `unparseable` even - // under invalid/missing/unknown, so a later enforcement precedence can see - // it. `evaluateVisibleFrom` never throws, so it does not reach the `error` - // path below. - evaluateVisibleFrom(verdict, raw); + const from = evaluateVisibleFrom({ + raw, + stampedAddrSpec, + authenticatedSender, + }); + verdict = { + signature, + fromMatch: from.fromMatch, + authenticatedSender, + messageFrom: from.messageFrom, + }; } catch (cause) { logger.error( "inbound mail signature verify FAULTED for {authenticatedSender} (messageId {messageId}, agentAddress {agentAddress}): {cause}", @@ -209,7 +222,7 @@ export async function verifyInboundSignature( // Carry the same `signature`/`fromMatch` keys the clean verdict logs, so // a consumer counting the verdict corpus by `signature` sees faults too. signature: "error", - fromMatch: "unchecked", + fromMatch: "notEvaluated", authenticatedSender, messageId: input.messageId ?? null, agentAddress: input.agentAddress, @@ -218,7 +231,7 @@ export async function verifyInboundSignature( ); return { signature: "error", - fromMatch: "unchecked", + fromMatch: "notEvaluated", authenticatedSender, messageFrom: null, }; @@ -227,57 +240,68 @@ export async function verifyInboundSignature( return logVerdict(verdict, input); } +type VisibleFromArgs = { + raw: Uint8Array; + stampedAddrSpec: string; + authenticatedSender: string; +}; + /** - * Evaluate the message's visible `From` onto `verdict`, for any non-error - * signature status. Uses the tri-state of `readMessageFrom`: - * - no `From` (or empty): leaves the binding `unchecked`, `messageFrom` null. - * - `From` present but unparseable (`readMessageFrom` throws): marks the - * binding `unparseable` -- a present but malformed `From` is a distinct, - * suspicious state, kept separate from the benign no-`From` `unchecked`. - * - `From` present and parsed: records `messageFrom`, and ONLY atop a VALID - * signature compares it to `authenticatedSender` for `match`/`mismatch`. - * - * A stamped `authenticatedSender` that is not a bare addr-spec leaves the - * binding `unchecked` rather than clobbering the standing signature verdict -- - * the signature is the primary signal, and the check must not turn an unparseable - * header into a false verdict. + * The comparison does NOT consult the signature axis, and must not. Nothing + * downstream of the gate sees `authenticatedSender`, so a `From` contradicting + * the stamp is a claim the gate can refuse whether or not a verified signature + * stood behind it. */ function evaluateVisibleFrom( - verdict: InboundSignatureVerdict, - raw: Uint8Array, -): void { + args: VisibleFromArgs, +): Pick { + const { raw, stampedAddrSpec, authenticatedSender } = args; let messageFrom: string | null; try { messageFrom = readMessageFrom(raw); } catch (cause) { - // The message carries a `From` that `extractAddrSpec` refuses -- present - // but unparseable, distinct from no `From` at all. - verdict.fromMatch = "unparseable"; logger.debug( "inbound mail From-binding unparseable for {authenticatedSender}: {cause}", { - authenticatedSender: verdict.authenticatedSender, + authenticatedSender, cause: describeCause(cause), }, ); - return; + return { fromMatch: "unparseable", messageFrom: null }; } - if (messageFrom === null) return; - verdict.messageFrom = messageFrom; - // A From-binding is only meaningful atop a valid signature. - if (verdict.signature !== "valid") return; + if (messageFrom === null) return { fromMatch: "absent", messageFrom: null }; + return { + fromMatch: messageFrom === stampedAddrSpec ? "match" : "mismatch", + messageFrom, + }; +} + +/** + * `verifyMimeSignature` refuses such a key too, so the throw is not what makes + * the fault reach `error`. What this adds is a log line naming the keyring as + * the thing at fault. + */ +function requireUsableSenderKey( + authenticatedSender: string, + publicKey: Uint8Array, +): void { + if (isUsableSenderKey(publicKey)) return; + logger.error( + "inbound mail sender key for {authenticatedSender} is unusable: {keyBytes} bytes of cached material that is not a key, so no signature could be checked against it", + { authenticatedSender, keyBytes: publicKey.length }, + ); + throw new Error( + `cached sender key for ${JSON.stringify(authenticatedSender)} cannot verify a signature`, + ); +} + +function parseSenderStamp(authenticatedSender: string): string { try { - verdict.fromMatch = - messageFrom === extractAddrSpec(verdict.authenticatedSender) - ? "match" - : "mismatch"; + return extractAddrSpec(authenticatedSender); } catch (cause) { - logger.debug( - "inbound mail sender stamp {authenticatedSender} is not a parseable addr-spec; leaving From-binding unchecked: {cause}", - { - authenticatedSender: verdict.authenticatedSender, - cause: describeCause(cause), - }, + throw new Error( + `authenticated sender stamp is not a bare addr-spec: ${JSON.stringify(authenticatedSender)}`, + { cause }, ); } } diff --git a/vendor/intx/hub-api/VENDORED-FROM b/vendor/intx/hub-api/VENDORED-FROM index 5d6dc27d19..9f625e417d 100644 --- a/vendor/intx/hub-api/VENDORED-FROM +++ b/vendor/intx/hub-api/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/hub-api) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. CL-workflow-deploy-bearer: middleware/workflow-run-deploy-auth.ts (+ test) is a bearer-authenticated mirror onto the session-cookie POST/GET .../workflows/deployments route; MountHubRoutesDeps and CreateAppOpts gain an optional workflowRunAuthenticator that mounts it ahead of resolveTenant, and src/index.ts exports the middleware and its types. +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). CL-workflow-deploy-bearer: middleware/workflow-run-deploy-auth.ts is a bearer-authenticated mirror onto the session-cookie POST/GET .../workflows/deployments route; MountHubRoutesDeps and CreateAppOpts gain an optional workflowRunAuthenticator that mounts it ahead of resolveTenant, and src/index.ts exports the middleware and its types. CL-7505: run-source-resolution.ts threads an optional servingRefresh hook into buildSource, so an expiring oauth_token credential refreshes in place (or the offering is skipped) before its secret is served. diff --git a/vendor/intx/hub-common/VENDORED-FROM b/vendor/intx/hub-common/VENDORED-FROM index 34c53693b8..eac425a59b 100644 --- a/vendor/intx/hub-common/VENDORED-FROM +++ b/vendor/intx/hub-common/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/hub-common) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. First vendored in CL-7495: the hub's signing-key series needs generateId's pky_ principal-key prefix (npm 0.3.0 predates it), and @intx/db's principal-key store depends on this package. Only src/index.ts, ids.ts, action-mapping.ts, and glob.ts are consumed; the package depends on @intx/types via workspace: so the vendored types source joins the composite project-reference graph. diff --git a/vendor/intx/hub-sessions/VENDORED-FROM b/vendor/intx/hub-sessions/VENDORED-FROM index 7e46f3b962..430bf7aa71 100644 --- a/vendor/intx/hub-sessions/VENDORED-FROM +++ b/vendor/intx/hub-sessions/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/hub-sessions) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. Terminal-anchor pack acceptance: hub-session-lookups.ts's receiveWorkflowRunPack gates the anchor lookup on the exported pure helper ownsWorkflowRunRepo (self-anchored row with a routable address, no liveness requirement) instead of upstream's live-status filter, so a terminal run can still land the inbox-enqueue and markConsumed-rejection packs that retire mail arriving in its teardown window; upstream's isNotNull(definitionId) tightening is kept. CL-6361 widens the same lookup to peel a per-step pack source address back to its base run's anchor via anchorAddressForPackSource; CL-6379 classifies an accepted pack's newly-terminal runs through decideTerminalRunFlip so a section occurrence's repo-local child run (turn__) is skipped quietly. CL-6324: workflow-probe-gate.ts's PersistFrozenApprovalFn carries the inert projection and createDbFrozenApprovalWriter stamps workflow_definition_version.wire_projection in the same transaction as approved_wire_hash. CL-6478: sanitize-tool-name.ts + event-collector.ts's tool_call case persist only a tool-call name encodeToolName can re-invert (anything else collapses to MALFORMED_TOOL_NAME), so one bad name fails its turn instead of wedging the room; @intx/inference is a dependency for this. CL-6595: workflow-run-kind.ts's validatePush also reports a run sealed from birth (combined events.jsonl with no per-event blobs) as newly terminal, and the new readCommittedWorkflowRunTerminalStatus export backs a same-push markTerminal backfill in hub-session-lookups.ts; both classify through upstream's classifyTerminalEvent. CL-7190: registerSignalCorrelation (hub-session-lookups.ts) now throws for any signal kind other than "approval", failing loud instead of silently mis-persisting a future SignalKind it has no co-write for. CL-6396: sidecar-events.ts's agent.event map and sidecar-handler.ts's agent.event dispatch forward an optional childRunId from the sidecar frame so the hub can close the matching occurrence row. CL-7508: sidecar-handler.ts's oauth.login result handling settles the caller's request promise as an error when a terminal outcome arrives before the started acknowledgement — previously a completed-before-started frame left the HTTP caller hanging until the whole-login timeout; the no-hang pin lives in sidecar-handler-oauth-login.test.ts (only tests covering a workbench delta are vendored). RETIRED AT THIS PIN: the CL-6324 session-service.ts deltas (deployAdoptedCodeSourcedWorkflow / deployAdoptedWorkflowFromSource / AdoptingWorkflowDeployer, the sourceRef threading, and CL-7191's UserMessageParams.correlationId) hung off upstream's deployWorkflowFromSource front, which this pin deletes in favour of the catalog-offering + provisioner-selection deploy path. corbitsdev/workbench#709: session-service.ts's prepared-deploy path threads `deployContent` + `toolPackagePins` through `DeployPreparedCodeSourcedWorkflowParams` and fans every top-level step out to `stageWorkflowStep` before the deployment frame (pure `preparedStepStageArgs` maps one step's address via `resolveStepAddress`, agent id via `deriveStepAgentId`, pins verbatim-or-absent); workflow-allocation-service.ts rehydrates both from the frozen launch spec (`preparedDeployContent` fails closed without a string prompt). The pins threading is pinned in session-service-pins.test.ts (only tests covering a workbench delta are vendored). +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). CL-6396: sidecar-events.ts carries an optional childRunId on the agent.event frame so the hub can close the matching occurrence row. CL-7505: credential-push.ts threads an optional servingRefresh hook into resolveInstanceModelSources, so an expiring oauth_token credential refreshes in place before the push delivers its material. OAuth loopback login: sidecar-handler.ts hosts the loopback login flow (OAuthLoginFinalOutcome/Gate/RequestOutcome, DEFAULT_OAUTH_LOGIN_TIMEOUT_MS; index.ts and ws/index.ts re-export the types). CL-7508: sidecar-handler.ts settles the login caller promise as an error when a terminal outcome arrives before the started acknowledgement, instead of hanging until the whole-login timeout. corbitsdev/workbench#709: session-service.ts fans every top-level prepared-deploy step out through stageWorkflowStep (pure preparedStepStageArgs maps one step address via resolveStepAddress and agent id via deriveStepAgentId); workflow-allocation-service.ts rehydrates both from the frozen launch spec (preparedDeployContent fails closed without a string systemPrompt). Upstream now threads deployContent/toolPackagePins natively; the address/id mapping and the fail-closed narrowing stay local. Retired at earlier pins (prose trimmed at this re-vendor): the terminal-anchor pack acceptance (ownsWorkflowRunRepo), anchorAddressForPackSource, decideTerminalRunFlip, the PersistFrozenApprovalFn wire_projection stamp, the encodeToolName/MALFORMED_TOOL_NAME gate (and the @intx/inference dependency it needed, dropped), the readCommittedWorkflowRunTerminalStatus backfill, and the registerSignalCorrelation kind check. diff --git a/vendor/intx/hub-sessions/package.json b/vendor/intx/hub-sessions/package.json index c353e88509..af030169e8 100644 --- a/vendor/intx/hub-sessions/package.json +++ b/vendor/intx/hub-sessions/package.json @@ -22,7 +22,6 @@ "@intx/crypto": "0.4.0", "@intx/db": "workspace:*", "@intx/hub-common": "0.4.0", - "@intx/inference": "workspace:*", "@intx/log": "0.4.0", "@intx/mime": "workspace:*", "@intx/pack-transport": "0.4.0", diff --git a/vendor/intx/inference/VENDORED-FROM b/vendor/intx/inference/VENDORED-FROM index 98d577621f..10c6a0fa3d 100644 --- a/vendor/intx/inference/VENDORED-FROM +++ b/vendor/intx/inference/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/inference) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. providers/google-genai-files.ts builds its upload body as `new Uint8Array(bytes)`: TS 6's lib.dom `BodyInit` rejects `Uint8Array` and workbench compiles this source under DOM-lib packages (upstream compiles ESNext-only under TS 5.9). CL-7443 retires CL-7190's message_response resume branch in reactor.ts's resumePendingOperation/timeoutMessageFor (and its answer_result resume mode), plus the reactor.test.ts/testing/fakes.ts regression harness CL-7190 added to cover it: `ask_user` posts its question and ends the turn instead of parking a gate, so the reactor is back to zero local delta and zero tests, matching upstream. +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). providers/google-genai-files.ts builds its upload body as `new Uint8Array(bytes)`: TS 6's lib.dom `BodyInit` rejects `Uint8Array` and workbench compiles this source under DOM-lib packages (upstream compiles ESNext-only under TS 5.9). CL-7443 retires CL-7190's message_response resume branch in reactor.ts's resumePendingOperation/timeoutMessageFor (and its answer_result resume mode), plus the reactor.test.ts/testing/fakes.ts regression harness CL-7190 added to cover it: `ask_user` posts its question and ends the turn instead of parking a gate, so the reactor is back to zero local delta and zero tests, matching upstream. diff --git a/vendor/intx/inference/src/correlation.ts b/vendor/intx/inference/src/correlation.ts index 974a03d037..53bababa60 100644 --- a/vendor/intx/inference/src/correlation.ts +++ b/vendor/intx/inference/src/correlation.ts @@ -11,6 +11,10 @@ import type { InboundMessage, PendingOperation } from "@intx/types/runtime"; * Validates whether an inbound message is an authentic response to a * registered pending operation. Consumers provide this at reactor construction * time to enforce sender identity and signature checks. + * + * The reactor performs no sender check of its own, so a composition that + * supplies no validator resolves a pending operation for any message bearing a + * registered correlation ID. See MESSAGE.md (Correlation Security). */ export interface CorrelationValidator { /** diff --git a/vendor/intx/inference/src/reactor.ts b/vendor/intx/inference/src/reactor.ts index c1a6a442cb..bf5efda17c 100644 --- a/vendor/intx/inference/src/reactor.ts +++ b/vendor/intx/inference/src/reactor.ts @@ -356,20 +356,25 @@ export function createReactor(config: ReactorConfig): Reactor { let toolBatchRepeatCount = 0; let lastToolBatchNames: string[] = []; - function openMessageRun(messageId: string): void { + function openMessageRun(messageId: string | undefined): void { currentMessageRunId = crypto.randomUUID(); - currentMessageId = messageId; + currentMessageId = messageId ?? null; lastToolBatchSignature = null; toolBatchRepeatCount = 0; lastToolBatchNames = []; + const data: { + messageId?: string; + messageRunId: string; + receivedAt: number; + } = { + messageRunId: currentMessageRunId, + receivedAt: Date.now(), + }; + if (messageId !== undefined) data.messageId = messageId; emit({ type: "message.run.started", seq: nextSeq(), - data: { - messageId, - messageRunId: currentMessageRunId, - receivedAt: Date.now(), - }, + data, }); } @@ -377,17 +382,19 @@ export function createReactor(config: ReactorConfig): Reactor { status: "completed" | "failed", error?: { message: string; kind?: string }, ): void { - if (currentMessageRunId === null || currentMessageId === null) return; + // Gated on the run id alone: a message that named no id still has to + // close its bracket, or the run stays open for the rest of the session. + if (currentMessageRunId === null) return; const data: { messageRunId: string; - messageId: string; + messageId?: string; status: "completed" | "failed"; error?: { message: string; kind?: string }; } = { messageRunId: currentMessageRunId, - messageId: currentMessageId, status, }; + if (currentMessageId !== null) data.messageId = currentMessageId; if (error !== undefined) data.error = error; emit({ type: "message.run.ended", seq: nextSeq(), data }); currentMessageRunId = null; @@ -897,9 +904,6 @@ export function createReactor(config: ReactorConfig): Reactor { kind: "approval", registeredAt: Date.now(), gateId, - ...(marker.expectedFrom !== undefined - ? { expectedFrom: marker.expectedFrom } - : {}), }; correlations.register(op); stateManager.addPendingOperation(op); diff --git a/vendor/intx/inference/src/turns.ts b/vendor/intx/inference/src/turns.ts index e0a08a62ce..0edb91415b 100644 --- a/vendor/intx/inference/src/turns.ts +++ b/vendor/intx/inference/src/turns.ts @@ -77,7 +77,7 @@ export function createInboundTurn( if (content.length > 0) { const { from, subject } = message.headers; const envelope: string[] = []; - if (from.length > 0) envelope.push(`[From: ${from}]`); + if (from !== undefined && from.length > 0) envelope.push(`[From: ${from}]`); if (subject !== undefined && subject.length > 0) { envelope.push(`[Subject: ${subject}]`); } diff --git a/vendor/intx/mail-memory/VENDORED-FROM b/vendor/intx/mail-memory/VENDORED-FROM index 9f3049781b..ca7c5637de 100644 --- a/vendor/intx/mail-memory/VENDORED-FROM +++ b/vendor/intx/mail-memory/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/mail-memory) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. No source delta. diff --git a/vendor/intx/mail-memory/src/send.ts b/vendor/intx/mail-memory/src/send.ts index 19044a1b0a..d406068844 100644 --- a/vendor/intx/mail-memory/src/send.ts +++ b/vendor/intx/mail-memory/src/send.ts @@ -3,10 +3,12 @@ import type { SendReceipt, MailboxEvent, } from "@intx/types/runtime"; -import { buildMessageHeaders, type StoredEnvelope } from "@intx/mailbox"; +import type { StoredEnvelope } from "@intx/mailbox"; +import { isConversationType } from "@intx/types/runtime"; import { assembleSignedContent, assembleMessage, + buildMessageHeaders, generateMessageId, isMessageId, parseHeaderSection, @@ -17,12 +19,6 @@ import { } from "@intx/mime"; import type { AddressEntry } from "./mailbox"; -const CONVERSATION_TYPES = new Set([ - "conversation.message", - "conversation.join", - "conversation.leave", -]); - /** * Callback for delivering messages to recipients not registered on this * transport. The federation layer provides this to forward messages and the @@ -115,7 +111,7 @@ export async function executeSend( ); } - const isConversation = CONVERSATION_TYPES.has(message.type); + const isConversation = isConversationType(message.type); if (isConversation && message.payload !== undefined) { throw new Error( @@ -126,6 +122,17 @@ export async function executeSend( throw new Error("Structured messages must not carry a text content field"); } + // RFC 5322 section 3.6.4 defines `In-Reply-To` as `1*msg-id`, so a blank + // value names nothing rather than naming a shorter parent. + if ( + message.inReplyTo !== undefined && + message.inReplyTo.trim().length === 0 + ) { + throw new Error( + "OutboundMessage inReplyTo, when provided, must name a message identifier", + ); + } + const messageId = generateMessageId(senderAddress); const now = new Date(); diff --git a/vendor/intx/mail-memory/src/transport.ts b/vendor/intx/mail-memory/src/transport.ts index d920b2a0e1..d406ee520e 100644 --- a/vendor/intx/mail-memory/src/transport.ts +++ b/vendor/intx/mail-memory/src/transport.ts @@ -18,9 +18,9 @@ import type { Unsubscribe, CryptoProvider, } from "@intx/types/runtime"; -import { parseHeaderSection } from "@intx/mime"; +import { MessageTransportError } from "@intx/types/runtime"; +import { buildMessageHeaders, parseHeaderSection } from "@intx/mime"; import { - buildMessageHeaders, createInMemoryMailboxStore, executeSearch, executeThread, @@ -258,7 +258,10 @@ export class InMemoryTransport implements MessageTransport, HubTransport { _knownState: SyncState, _signal?: AbortSignal, ): Promise { - throw new Error("sync() (QRESYNC) is not implemented"); + throw new MessageTransportError( + "CANNOT", + "sync() (QRESYNC) is not implemented", + ); } async createList( @@ -266,14 +269,20 @@ export class InMemoryTransport implements MessageTransport, HubTransport { _name: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } async listMembers( _address: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } async subscribe( @@ -281,7 +290,10 @@ export class InMemoryTransport implements MessageTransport, HubTransport { _subscriberAddress: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } async unsubscribe( @@ -289,7 +301,10 @@ export class InMemoryTransport implements MessageTransport, HubTransport { _subscriberAddress: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } // --------------------------------------------------------------------------- @@ -318,43 +333,38 @@ export class InMemoryTransport implements MessageTransport, HubTransport { } const { headers } = parseHeaderSection(message); + // The stored envelope and the `exists` event below are built from one + // reading of these bytes, so the two cannot disagree about which headers + // the message carried. `buildMessageHeaders` is that reading: it applies + // the RFC rule that a present-but-blank `Date`, `Message-ID`, `From` or + // `In-Reply-To` names nothing, so a blank one arrives here as an absence. + const msgHeaders = buildMessageHeaders(headers); - const messageId = headers.get("message-id"); - const from = headers.get("from"); - const dateRaw = headers.get("date"); - if (messageId === undefined) { + const dateRaw = msgHeaders.date; + if (msgHeaders.messageId === undefined) { throw new Error("Cannot deliver message: missing Message-ID header"); } - if (from === undefined) { + if (msgHeaders.from === undefined) { throw new Error("Cannot deliver message: missing From header"); } if (dateRaw === undefined) { throw new Error("Cannot deliver message: missing Date header"); } - const msgHeaders = buildMessageHeaders(headers); - - const toRaw = headers.get("to") ?? ""; - const to = toRaw - ? toRaw - .split(",") - .map((s) => s.trim()) - .filter(Boolean) - : []; - - const refsRaw = headers.get("references"); - const references = refsRaw ? refsRaw.split(/\s+/).filter(Boolean) : []; - const envelope: StoredEnvelope = { - messageId, - from, - to, - subject: headers.get("subject") ?? "", + messageId: msgHeaders.messageId, + from: msgHeaders.from, + to: msgHeaders.to, + subject: msgHeaders.subject ?? "", date: new Date(dateRaw), - inReplyTo: headers.get("in-reply-to"), - references, + inReplyTo: msgHeaders.inReplyTo, + references: msgHeaders.references ?? [], + // Read off the raw map rather than `msgHeaders`, which keeps this field + // only when it names a declared `InterchangeType`. The envelope mirrors + // what the message carried, so an unrecognized type is stored verbatim + // instead of being erased from the index. interchangeType: headers.get("interchange-type"), - interchangeCorrelationId: headers.get("interchange-correlation-id"), + interchangeCorrelationId: msgHeaders.interchangeCorrelationId, }; const uid = inbox.append(message, envelope, []); @@ -420,7 +430,16 @@ class ScopedMessageTransport implements MessageTransport { get #entry(): AddressEntry { const e = this.#entries.get(this.#address); if (e === undefined) { - throw new Error(`Address "${this.#address}" has been deregistered`); + // `CANNOT` rather than a bare rejection: a rejection naming no condition + // leaves the outcome unknown, and an unknown outcome reads as worth + // retrying, which no retry through this handle clears. The lookup is + // against the live map, so registering the address again does revive the + // handle against the new entry -- but registration belongs to whoever + // owns the transport, not to the holder of a scoped handle. + throw new MessageTransportError( + "CANNOT", + `Address "${this.#address}" has been deregistered`, + ); } return e; } @@ -428,7 +447,8 @@ class ScopedMessageTransport implements MessageTransport { #requireMailbox(name: string) { const store = this.#entry.mailboxes.get(name); if (store === undefined) { - throw new Error( + throw new MessageTransportError( + "NONEXISTENT", `Mailbox "${name}" does not exist for address "${this.#address}"`, ); } @@ -473,13 +493,24 @@ class ScopedMessageTransport implements MessageTransport { // need Uint8Array. We store a minimal representation. // // For now, serialize the InboundMessage as a minimal RFC 2822 message. + // The stored envelope keys the mailbox index on the id and serializes the + // date, so neither can be absent here. `deliver` refuses the same two on + // the same grounds. Checked before serializing so a message that cannot + // be stored is not encoded first and then discarded. + const { messageId, date } = message.headers; + if (messageId === undefined) { + throw new Error("Cannot append message: missing Message-ID header"); + } + if (date === undefined) { + throw new Error("Cannot append message: missing Date header"); + } const raw = inboundMessageToRaw(message); const envelope = { - messageId: message.headers.messageId, + messageId, from: message.headers.from, to: message.headers.to, subject: message.headers.subject ?? "", - date: new Date(message.headers.date), + date: new Date(date), inReplyTo: message.headers.inReplyTo, references: message.headers.references ?? [], interchangeType: message.headers.interchangeType, @@ -507,7 +538,8 @@ class ScopedMessageTransport implements MessageTransport { async deleteMailbox(name: string, _signal?: AbortSignal): Promise { if (!this.#entry.mailboxes.has(name)) { - throw new Error( + throw new MessageTransportError( + "NONEXISTENT", `Mailbox "${name}" does not exist for address "${this.#address}"`, ); } @@ -704,7 +736,10 @@ class ScopedMessageTransport implements MessageTransport { _knownState: SyncState, _signal?: AbortSignal, ): Promise { - throw new Error("sync() (QRESYNC) is not implemented"); + throw new MessageTransportError( + "CANNOT", + "sync() (QRESYNC) is not implemented", + ); } async createList( @@ -712,14 +747,20 @@ class ScopedMessageTransport implements MessageTransport { _name: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } async listMembers( _address: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } async subscribe( @@ -727,7 +768,10 @@ class ScopedMessageTransport implements MessageTransport { _subscriberAddress: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } async unsubscribe( @@ -735,7 +779,10 @@ class ScopedMessageTransport implements MessageTransport { _subscriberAddress: string, _signal?: AbortSignal, ): Promise { - throw new Error("Distribution list management is not implemented"); + throw new MessageTransportError( + "CANNOT", + "Distribution list management is not implemented", + ); } #fireWatchCallbacks(mailbox: string, event: MailboxEvent): void { @@ -757,13 +804,26 @@ function inboundMessageToRaw(message: InboundMessage): Uint8Array { const enc = new TextEncoder(); const CRLF = "\r\n"; let headers = ""; - headers += `From: ${message.headers.from}${CRLF}`; + // A message with no originator gets no From line. Interpolating the absence + // would write the literal `From: undefined` into the RFC 2822 bytes, which + // reads back as an originator named "undefined". + if (message.headers.from !== undefined) { + headers += `From: ${message.headers.from}${CRLF}`; + } headers += `To: ${message.headers.to.join(", ")}${CRLF}`; if (message.headers.cc && message.headers.cc.length > 0) { headers += `Cc: ${message.headers.cc.join(", ")}${CRLF}`; } - headers += `Date: ${message.headers.date}${CRLF}`; - headers += `Message-ID: ${message.headers.messageId}${CRLF}`; + // Omitted for the same reason as From above: interpolating an absent value + // writes the literal `Date: undefined` into the RFC 2822 bytes, which reads + // back as a real header. An omitted line is the honest encoding of a header + // the message never carried. + if (message.headers.date !== undefined) { + headers += `Date: ${message.headers.date}${CRLF}`; + } + if (message.headers.messageId !== undefined) { + headers += `Message-ID: ${message.headers.messageId}${CRLF}`; + } if (message.headers.subject !== undefined) { headers += `Subject: ${message.headers.subject}${CRLF}`; } diff --git a/vendor/intx/mailbox/VENDORED-FROM b/vendor/intx/mailbox/VENDORED-FROM index f55254f0f3..324e4d7395 100644 --- a/vendor/intx/mailbox/VENDORED-FROM +++ b/vendor/intx/mailbox/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/mailbox) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. No source delta. diff --git a/vendor/intx/mailbox/package.json b/vendor/intx/mailbox/package.json index 27aecaa4b9..a30f2a4e46 100644 --- a/vendor/intx/mailbox/package.json +++ b/vendor/intx/mailbox/package.json @@ -15,6 +15,7 @@ }, "dependencies": { "@intx/crypto": "0.4.0", + "@intx/log": "0.4.0", "@intx/mime": "workspace:*", "@intx/types": "workspace:*", "arktype": "catalog:" diff --git a/vendor/intx/mailbox/src/fetch.ts b/vendor/intx/mailbox/src/fetch.ts index 6e9b08fae1..b078549baa 100644 --- a/vendor/intx/mailbox/src/fetch.ts +++ b/vendor/intx/mailbox/src/fetch.ts @@ -1,5 +1,5 @@ -/* eslint-disable @typescript-eslint/no-non-null-assertion -- MIME multipart parsing with bounds checks */ import { type } from "arktype"; +import { getLogger } from "@intx/log"; import type { MessageHeaders, BodyStructure, @@ -10,7 +10,6 @@ import type { MessageRef, } from "@intx/types/runtime"; import { InterchangeType } from "@intx/types/runtime"; -import { base64Decode } from "@intx/types"; import type { MailboxStore } from "./mailbox"; import { requireMessage } from "./mailbox"; import { @@ -18,12 +17,20 @@ import { parseMimePart, extractBoundary, parseMultipart, + extractContentTypeMime, extractPartByPath, extractAttachments, + buildMessageHeaders, + decodePartBytes, + isRecognizedTransferEncoding, + reportedContentType, + transferEncodingMechanism, } from "@intx/mime"; -import { buildMessageHeaders } from "./headers"; +import type { ParsedMimePart } from "@intx/mime"; import { verifyMimeSignature } from "./verify-signature"; +const logger = getLogger(["interchange", "mailbox", "fetch"]); + const MessagePayload = type({ type: InterchangeType, version: "string", @@ -62,7 +69,8 @@ export async function fetchStructure( } /** - * Fetch a single MIME part by dot-separated path. + * Fetch a single MIME part by dot-separated path. `contentType` carries the + * RFC 2045 §6.4 relabel, since an undecodable part arrives undecoded. */ export async function fetchPart( ref: MessageRef, @@ -74,24 +82,68 @@ export async function fetchPart( const partBytes = extractPartByPath(raw, partPath); const part = parseMimePart(partBytes); - const enc = part.headers.get("content-transfer-encoding") ?? "7bit"; - let content: Uint8Array; - - if (enc.toLowerCase() === "base64") { - const b64 = new TextDecoder().decode(part.body).replace(/\s/g, ""); - content = base64Decode(b64); - } else { - content = part.body; - } - const result: MessagePart = { - contentType: part.contentType, - content, + contentType: reportedContentType(part.contentType, part.headers), + content: decodePartBytes(part.body, part.headers), }; - if (enc !== "7bit") result.encoding = enc; + const mechanism = transferEncodingMechanism(part.headers); + if (mechanism !== "7bit") result.encoding = mechanism; return result; } +/** + * Decode a leaf part's body into text, or undefined when the octets are not + * text: an unrecognized mechanism (RFC 2045 §6.4) or encoded data that will not + * decode. The octets stay reachable through `fetchPart`, which reports them + * under the §6.4 relabel. + */ +function decodePartText( + part: ParsedMimePart, + ref: MessageRef, +): string | undefined { + if (!isRecognizedTransferEncoding(transferEncodingMechanism(part.headers))) { + return undefined; + } + let decoded: Uint8Array; + try { + decoded = decodePartBytes(part.body, part.headers); + } catch (cause) { + // Absent text is the answer, but it is not a self-explaining one: a caller + // reading `undefined` cannot tell "not text" from "would not decode", and + // an operator would otherwise never learn that a peer sends bodies its own + // declared encoding does not describe. The record names the message so the + // octets can be read back through `fetchPart`. + logger.warn`Message uid=${ref.uid} in mailbox ${ref.mailbox} carries a ${part.contentType} part whose ${transferEncodingMechanism(part.headers)} body did not decode; delivering it without text: ${cause instanceof Error ? cause.message : String(cause)}`; + return undefined; + } + return new TextDecoder("utf-8", { fatal: false }).decode(decoded); +} + +/** + * The leaf part a message's content lives in: part 1.1 under our assembler's + * multipart/mixed wrapper, part 1 when a peer signed a bare part, and the + * message itself when it is not multipart at the top level. + */ +function resolveContentPart( + raw: Uint8Array, + messageHeaders: Map, + bodyOffset: number, +): ParsedMimePart { + const declared = messageHeaders.get("content-type") ?? "text/plain"; + if (!extractContentTypeMime(declared).startsWith("multipart/")) { + return { + contentType: declared, + headers: messageHeaders, + body: raw.slice(bodyOffset), + }; + } + const part1 = parseMimePart(extractPartByPath(raw, "1")); + if (!extractContentTypeMime(part1.contentType).startsWith("multipart/")) { + return part1; + } + return parseMimePart(extractPartByPath(raw, "1.1")); +} + /** * Fetch a complete message, verify its PGP/MIME signature, and return * a fully parsed InboundMessage. @@ -103,7 +155,7 @@ export async function fetchFull( ): Promise { const msg = requireMessage(store, ref.uid, ref.mailbox); const raw = await store.readRaw(ref.uid); - const { headers } = parseHeaderSection(raw); + const { headers, bodyOffset } = parseHeaderSection(raw); const parsedHeaders = buildMessageHeaders(headers); const rawType = parsedHeaders.interchangeType; @@ -126,48 +178,27 @@ export async function fetchFull( signatureStatus, }; - try { + // A body that did not decode is not text, and both destinations here are + // text; the message is delivered without one rather than refused. + const text = decodePartText( + resolveContentPart(raw, headers, bodyOffset), + ref, + ); + if (text !== undefined) { if (isConversation) { - const part1 = parseMimePart(extractPartByPath(raw, "1")); - const part1Mime = part1.contentType.split(";")[0]!.trim().toLowerCase(); - if (part1Mime.startsWith("multipart/")) { - // Conversation shape: multipart/mixed with the text body at 1.1. - const textPart = parseMimePart(extractPartByPath(raw, "1.1")); - result.content = new TextDecoder("utf-8", { fatal: false }).decode( - textPart.body, - ); - } else { - // A conversation message is "literally a signed email", so a sender - // (e.g. a plain mail client) may sign a bare text/plain part with no - // multipart/mixed wrapper. This branch reads that body directly. Our - // own assembler always emits multipart/mixed; without this branch a - // bare text/plain message would fail the 1.1 lookup and silently lose - // its content to the catch below. - result.content = new TextDecoder("utf-8", { fatal: false }).decode( - part1.body, - ); - } + result.content = text; } else { - // Structured messages carry their JSON payload at 1.1. Attachments on - // structured messages are intentionally not parsed: they have no - // producer today, so parsing them would handle a shape nobody sends. - const part11Bytes = extractPartByPath(raw, "1.1"); - const part11 = parseMimePart(part11Bytes); - const jsonText = new TextDecoder("utf-8", { fatal: false }).decode( - part11.body, - ); - const validated = MessagePayload(JSON.parse(jsonText)); + // Attachments on structured messages are intentionally not parsed: they + // have no producer today. + const validated = MessagePayload(JSON.parse(text)); if (validated instanceof type.errors) { throw new Error(`invalid message payload: ${validated.summary}`); } result.payload = validated; } - } catch { - // If we can't parse the content, return what we have with the signature status. } - // Attachment parsing is deliberately outside the catch above: a malformed - // attachment must surface as a thrown error, not be silently dropped. + // A malformed attachment throws rather than being silently dropped. if (isConversation) { const attachments = extractAttachments(raw); if (attachments.length > 0) { @@ -180,9 +211,14 @@ export async function fetchFull( async function verifyMessageSignature( raw: Uint8Array, - fromAddress: string, + fromAddress: string | undefined, getCrypto: (fromAddress: string) => CryptoProvider | undefined, ): Promise { + // No originator names no key to verify against, which is the same position + // as a sender we hold no key for. + if (fromAddress === undefined) { + return "unknown"; + } const senderCrypto = getCrypto(fromAddress); if (senderCrypto === undefined) { return "unknown"; diff --git a/vendor/intx/mailbox/src/headers.ts b/vendor/intx/mailbox/src/headers.ts deleted file mode 100644 index 0e4021126a..0000000000 --- a/vendor/intx/mailbox/src/headers.ts +++ /dev/null @@ -1,5 +0,0 @@ -// `buildMessageHeaders` now lives in `@intx/mime` alongside the rest of the -// MIME/header parsing (it is also what the `decodeMail` decoder builds its -// typed header subset with). Re-exported here so mail-memory's callers keep -// their existing import path. -export { buildMessageHeaders } from "@intx/mime"; diff --git a/vendor/intx/mailbox/src/index.ts b/vendor/intx/mailbox/src/index.ts index 15d11b1a7b..85c9b78865 100644 --- a/vendor/intx/mailbox/src/index.ts +++ b/vendor/intx/mailbox/src/index.ts @@ -8,5 +8,4 @@ export type { MailboxStore, StoredMessage, StoredEnvelope } from "./mailbox"; export { executeSearch } from "./search"; export { executeThread } from "./thread"; export { fetchHeaders, fetchStructure, fetchPart, fetchFull } from "./fetch"; -export { buildMessageHeaders } from "./headers"; export { verifyMimeSignature } from "./verify-signature"; diff --git a/vendor/intx/mailbox/src/mailbox.ts b/vendor/intx/mailbox/src/mailbox.ts index 05a3e9455d..95b2649997 100644 --- a/vendor/intx/mailbox/src/mailbox.ts +++ b/vendor/intx/mailbox/src/mailbox.ts @@ -4,10 +4,11 @@ */ export type StoredEnvelope = { messageId: string; - from: string; + from: string | undefined; to: string[]; subject: string; - date: Date; + /** The date the sender put on the message, not the instant it arrived. */ + date: Date | undefined; inReplyTo: string | undefined; references: string[]; interchangeType: string | undefined; diff --git a/vendor/intx/mailbox/src/search.ts b/vendor/intx/mailbox/src/search.ts index 4efc54c2ef..f9a36ac9b5 100644 --- a/vendor/intx/mailbox/src/search.ts +++ b/vendor/intx/mailbox/src/search.ts @@ -55,7 +55,11 @@ async function matchMessage( readRaw: () => Promise, ): Promise { if (query.from !== undefined) { - if (!msg.envelope.from.toLowerCase().includes(query.from.toLowerCase())) { + const sender = msg.envelope.from; + if ( + sender === undefined || + !sender.toLowerCase().includes(query.from.toLowerCase()) + ) { return false; } } @@ -93,19 +97,23 @@ async function matchMessage( } } + // A message that named no date placed itself nowhere in time, so it falls + // outside every date window below rather than inside one. + const sent = msg.envelope.date; + if (query.before !== undefined) { - if (msg.envelope.date >= query.before) return false; + if (sent === undefined || sent >= query.before) return false; } if (query.after !== undefined) { - if (msg.envelope.date <= query.after) return false; + if (sent === undefined || sent <= query.after) return false; } if (query.on !== undefined) { - const d = msg.envelope.date; + if (sent === undefined) return false; const q = query.on; if ( - d.getUTCFullYear() !== q.getUTCFullYear() || - d.getUTCMonth() !== q.getUTCMonth() || - d.getUTCDate() !== q.getUTCDate() + sent.getUTCFullYear() !== q.getUTCFullYear() || + sent.getUTCMonth() !== q.getUTCMonth() || + sent.getUTCDate() !== q.getUTCDate() ) { return false; } @@ -113,18 +121,18 @@ async function matchMessage( // Sent date filters use the Date header (same as envelope date here). if (query.sentBefore !== undefined) { - if (msg.envelope.date >= query.sentBefore) return false; + if (sent === undefined || sent >= query.sentBefore) return false; } if (query.sentAfter !== undefined) { - if (msg.envelope.date <= query.sentAfter) return false; + if (sent === undefined || sent <= query.sentAfter) return false; } if (query.sentOn !== undefined) { - const d = msg.envelope.date; + if (sent === undefined) return false; const q = query.sentOn; if ( - d.getUTCFullYear() !== q.getUTCFullYear() || - d.getUTCMonth() !== q.getUTCMonth() || - d.getUTCDate() !== q.getUTCDate() + sent.getUTCFullYear() !== q.getUTCFullYear() || + sent.getUTCMonth() !== q.getUTCMonth() || + sent.getUTCDate() !== q.getUTCDate() ) { return false; } diff --git a/vendor/intx/mailbox/src/thread.ts b/vendor/intx/mailbox/src/thread.ts index 024557901c..d8c2628b8f 100644 --- a/vendor/intx/mailbox/src/thread.ts +++ b/vendor/intx/mailbox/src/thread.ts @@ -76,9 +76,7 @@ function orderedSubjectThread( const threads: Thread[] = []; for (const [, msgs] of bySubject) { - const sorted = msgs.sort( - (a, b) => a.envelope.date.getTime() - b.envelope.date.getTime(), - ); + const sorted = msgs.sort((a, b) => messageDate(a) - messageDate(b)); const root = sorted[0]!; const rootThread: Thread = { ref: { uid: root.uid, mailbox: mailboxName }, @@ -93,7 +91,7 @@ function orderedSubjectThread( return threads.sort((a, b) => { const aMsg = messages.find((m) => m.uid === a.ref.uid)!; const bMsg = messages.find((m) => m.uid === b.ref.uid)!; - return aMsg.envelope.date.getTime() - bMsg.envelope.date.getTime(); + return messageDate(aMsg) - messageDate(bMsg); }); } @@ -195,7 +193,7 @@ function buildRefList(references: string[], inReplyTo?: string): string[] { } } - if (inReplyTo !== undefined && inReplyTo !== "" && !seen.has(inReplyTo)) { + if (inReplyTo !== undefined && !seen.has(inReplyTo)) { result.push(inReplyTo); } @@ -231,17 +229,35 @@ function pruneContainers(containers: Container[]): Container[] { return result; } +/** + * The sort key for a message that named no date. `Number.MAX_SAFE_INTEGER` + * exceeds the largest time value a `Date` can hold (8.64e15), so a message + * carrying this key sorts after every dated message. The epoch would do the + * opposite: it is the earliest representable instant, which would hand the + * root of an ascending-sorted thread to a message that placed itself nowhere + * in time. + */ +const UNDATED_SORT_KEY = Number.MAX_SAFE_INTEGER; + +/** The sort key for a message's position in a thread (RFC 5256 orders by date). */ +function messageDate(msg: StoredMessage): number { + const date = msg.envelope.date; + return date === undefined ? UNDATED_SORT_KEY : date.getTime(); +} + function containerDate(c: Container): number { if (c.message !== null) { - return c.message.envelope.date.getTime(); + return messageDate(c.message); } - // For dummy containers, use the earliest child date. - let earliest = Infinity; + // A dummy container carries no date of its own, so it borrows the earliest + // date among its descendants. With no dated descendant it keeps + // `UNDATED_SORT_KEY` and sorts last, as an undated message does. + let earliest = UNDATED_SORT_KEY; for (const child of c.children) { const d = containerDate(child); if (d < earliest) earliest = d; } - return earliest === Infinity ? 0 : earliest; + return earliest; } function containersToThreads( diff --git a/vendor/intx/mailbox/src/verify-signature.ts b/vendor/intx/mailbox/src/verify-signature.ts index 565387075e..05fbccae6e 100644 --- a/vendor/intx/mailbox/src/verify-signature.ts +++ b/vendor/intx/mailbox/src/verify-signature.ts @@ -5,7 +5,7 @@ import { extractBoundary, parseMultipart, } from "@intx/mime"; -import { verifyDetachedSignature } from "@intx/crypto"; +import { importPublicKeyBytes, verifyDetachedSignature } from "@intx/crypto"; /** * Verify a PGP/MIME `multipart/signed` message against a public key. @@ -17,8 +17,11 @@ import { verifyDetachedSignature } from "@intx/crypto"; * - `valid` — the detached signature verified against `publicKey` * - `invalid` — the signature check failed, or the message could not be * parsed as a signed message - * - `missing` — the message is not `multipart/signed`, or carries no - * `application/pgp-signature` part + * - `missing` — the message is not `multipart/signed`, declares no `boundary=` + * parameter, or carries no `application/pgp-signature` part + * + * Throws when `publicKey` cannot verify anything: a fault in the caller's + * input, not a verdict about the message. * * `raw` must be the original, unmodified message bytes: the signature is * recomputed over the exact canonical bytes of the signed part, so a @@ -29,6 +32,7 @@ export async function verifyMimeSignature( raw: Uint8Array, publicKey: Uint8Array, ): Promise<"valid" | "invalid" | "missing"> { + await requireUsableKey(publicKey); try { const { headers, bodyOffset } = parseHeaderSection(raw); const body = raw.slice(bodyOffset); @@ -65,3 +69,16 @@ export async function verifyMimeSignature( return "invalid"; } } + +/** + * Refuse a public key that cannot verify a signature, ahead of the catch that + * turns every other failure into `invalid` -- inside it, an unusable key would + * read as a check that ran and failed. + */ +async function requireUsableKey(publicKey: Uint8Array): Promise { + try { + await importPublicKeyBytes(publicKey); + } catch (cause) { + throw new Error("public key cannot verify a signature", { cause }); + } +} diff --git a/vendor/intx/mime/VENDORED-FROM b/vendor/intx/mime/VENDORED-FROM index 315c4c0a47..1a208a9742 100644 --- a/vendor/intx/mime/VENDORED-FROM +++ b/vendor/intx/mime/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/mime) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. No source delta. diff --git a/vendor/intx/mime/src/index.ts b/vendor/intx/mime/src/index.ts index a2029561de..b68ae8c2eb 100644 --- a/vendor/intx/mime/src/index.ts +++ b/vendor/intx/mime/src/index.ts @@ -8,11 +8,16 @@ export { parseMimePart, parseMultipart, extractBoundary, + extractContentTypeMime, extractPartByPath, parseMailToEmail, extractAttachments, buildMessageHeaders, decodeMail, + decodePartBytes, + isRecognizedTransferEncoding, + reportedContentType, + transferEncodingMechanism, } from "./mime"; export type { diff --git a/vendor/intx/mime/src/mail-builder.ts b/vendor/intx/mime/src/mail-builder.ts index 65c3ff3880..6c6afad0c8 100644 --- a/vendor/intx/mime/src/mail-builder.ts +++ b/vendor/intx/mime/src/mail-builder.ts @@ -23,7 +23,11 @@ import type { MessageRef, OutboundMessage, } from "@intx/types/runtime"; -import { InterchangeType, SignatureStatus } from "@intx/types/runtime"; +import { + InterchangeType, + SignatureStatus, + isConversationType, +} from "@intx/types/runtime"; import { generateMessageId } from "./mime"; /** @@ -37,8 +41,6 @@ const DEFAULT_PAYLOAD_VERSION = "1"; const MESSAGE_ID_RE = /^<[^<>\s@]+@[^<>\s@]+>$/; const ADDRESS_RE = /^[^@\s]+@[^@\s]+$/; -const CONVERSATION_TYPE_PREFIX = "conversation."; - // --------------------------------------------------------------------------- // InboundMessage builder // --------------------------------------------------------------------------- @@ -54,7 +56,10 @@ export type InboundPayloadInput = { }; export type CreateInboundMessageOpts = { - from: string; + /** Originator; omit it when the message has none. */ + from?: string; + + /** Recipients; pass an empty array when the message names none. */ to: string | string[]; /** Plain-text body. Mutually exclusive with `payload`. */ @@ -111,8 +116,8 @@ export function createInboundMessage( ): InboundMessage { const fn = "createInboundMessage"; - requireAddress(opts.from, "from", fn); - const to = normalizeAndValidateAddressArray(opts.to, "to", fn); + if (opts.from !== undefined) requireAddress(opts.from, "from", fn); + const to = normalizeAndValidateAddressArray(opts.to, "to", fn, true); validateBodyExclusivity(opts.content, opts.payload, fn); @@ -168,7 +173,7 @@ export function createInboundMessage( const cc = opts.cc === undefined ? undefined - : normalizeAndValidateAddressArray(opts.cc, "cc", fn); + : normalizeAndValidateAddressArray(opts.cc, "cc", fn, false); rejectEmptyStringIfPresent(opts.content, "content", fn); rejectEmptyStringIfPresent(opts.subject, "subject", fn); @@ -202,7 +207,8 @@ export function createInboundMessage( const messageId = opts.messageId ?? generateMessageId(opts.from); const derivedInterchangeType = opts.interchangeType ?? opts.payload?.type; - const headers: MessageHeaders = { from: opts.from, to, date, messageId }; + const headers: MessageHeaders = { to, date, messageId }; + if (opts.from !== undefined) headers.from = opts.from; if (cc !== undefined) headers.cc = cc; if (opts.subject !== undefined) headers.subject = opts.subject; if (opts.inReplyTo !== undefined) headers.inReplyTo = opts.inReplyTo; @@ -310,9 +316,9 @@ export function createOutboundMessage( validateInterchangeType(opts.type, "type", fn); // Validate addresses without mutating the source shape; the OutboundMessage // type preserves `string | string[]` and downstream consumers handle both. - normalizeAndValidateAddressArray(opts.to, "to", fn); + normalizeAndValidateAddressArray(opts.to, "to", fn, false); if (opts.cc !== undefined) { - normalizeAndValidateAddressArray(opts.cc, "cc", fn); + normalizeAndValidateAddressArray(opts.cc, "cc", fn, false); } validateBodyExclusivity(opts.content, opts.payload, fn); @@ -410,16 +416,21 @@ function requireAddress(value: unknown, field: string, fn: string): void { } } +/** + * Validate an address list and normalize a bare string into a one-entry array. + * `allowNone` permits an empty list, which only an inbound `to` does. + */ function normalizeAndValidateAddressArray( input: string | string[], field: string, fn: string, + allowNone: boolean, ): string[] { if (typeof input === "string") { requireAddress(input, field, fn); return [input]; } - if (!Array.isArray(input) || input.length === 0) { + if (!Array.isArray(input) || (input.length === 0 && !allowNone)) { throw new Error( `${fn}: \`${field}\` must contain at least one recipient address`, ); @@ -440,10 +451,6 @@ function validatePayloadBody(value: unknown, field: string, fn: string): void { } } -function isConversationType(t: InterchangeType): boolean { - return t.startsWith(CONVERSATION_TYPE_PREFIX); -} - function validateInterchangeType( value: unknown, field: string, diff --git a/vendor/intx/mime/src/mime.ts b/vendor/intx/mime/src/mime.ts index f842b2594d..554be9ec18 100644 --- a/vendor/intx/mime/src/mime.ts +++ b/vendor/intx/mime/src/mime.ts @@ -123,8 +123,11 @@ export type JMAPEmail = { // Message-ID generation // --------------------------------------------------------------------------- -export function generateMessageId(address: string): string { - const domain = address.includes("@") ? address.split("@")[1]! : "local"; +export function generateMessageId(address?: string): string { + const domain = + address !== undefined && address.includes("@") + ? address.split("@")[1]! + : "local"; const uuid = crypto.randomUUID(); return `<${uuid}@${domain}>`; } @@ -133,13 +136,59 @@ export function generateMessageId(address: string): string { // Address normalization // --------------------------------------------------------------------------- +function addressListError(addressLine: string): Error { + return new Error( + `extractAddrSpec: address lists are not supported: ${JSON.stringify(addressLine)}`, + ); +} + +/** + * Refuse a value naming more than one address; report where its `<` opens, or + * -1 for a bare `addr-spec`. A `quoted-string` carries `@`, `<` and `,` as + * text, so the scan exempts what it spans and refuses unpaired quotes. + */ +function scanSingleAddress(trimmed: string, addressLine: string): number { + let quoted = false; + let angleOpen = -1; + let sawUnbracketedAt = false; + let i = 0; + while (i < trimmed.length) { + const ch = trimmed[i]; + if (quoted && ch === "\\") { + i += 2; + continue; + } + if (ch === '"') { + quoted = !quoted; + } else if (!quoted) { + if (ch === ",") { + throw addressListError(addressLine); + } else if (ch === "@" && angleOpen === -1) { + sawUnbracketedAt = true; + } else if (ch === "<") { + if (angleOpen !== -1 || sawUnbracketedAt) { + throw addressListError(addressLine); + } + angleOpen = i; + } + } + i += 1; + } + if (quoted) { + throw new Error( + `extractAddrSpec: unterminated quoted string in ${JSON.stringify(addressLine)}`, + ); + } + return angleOpen; +} + /** * Extract the bare addr-spec (local-part@domain) from a single RFC 5322 * address value. Strips any display name and surrounding angle brackets, * then lowercases the result so case-insensitive comparison falls out * naturally. * - * Accepted inputs (single-address only — do not pass comma-separated lists): + * Accepted inputs (exactly one address): * `"Display Name" ` → `user@host` * `Display Name ` → `user@host` * `` → `user@host` @@ -148,6 +197,8 @@ export function generateMessageId(address: string): string { * * Rejected (throws) inputs: * - empty or whitespace-only + * - a value naming more than one address, however the members are separated + * - an input whose double quotes do not pair up * - input with no `@` * - input that produces an empty local-part or domain * - quoted local-parts (e.g. `"a@b"@host`) — technically valid per RFC @@ -170,8 +221,9 @@ export function extractAddrSpec(addressLine: string): string { throw new Error("extractAddrSpec: address is empty"); } + const angleOpen = scanSingleAddress(trimmed, addressLine); + let candidate: string; - const angleOpen = trimmed.lastIndexOf("<"); if (angleOpen !== -1) { // Angle-bracketed form. Require the `>` to be the trailing // non-whitespace character so that input like `Name (comment)` @@ -182,6 +234,11 @@ export function extractAddrSpec(addressLine: string): string { ); } candidate = trimmed.slice(angleOpen + 1, -1).trim(); + if (candidate.includes(">")) { + throw new Error( + `extractAddrSpec: stray '>' inside angle brackets in ${JSON.stringify(addressLine)}`, + ); + } } else { // Bare form. An unquoted addr-spec carries no internal whitespace, so // treat any as trailing content (e.g. `a@b (comment)`) and refuse rather @@ -284,6 +341,7 @@ function generateBoundary(): string { const CRLF = "\r\n"; function hdr(name: string, value: string): string { + assertNoLineBreaks(value, `${name} header`); return `${name}: ${value}${CRLF}`; } @@ -293,7 +351,7 @@ function serializeMessageHeaders( ): string { let out = ""; out += hdr("From", h.from); - out += hdr("To", Array.isArray(h.to) ? h.to.join(", ") : (h.to as string)); + out += hdr("To", h.to.join(", ")); if (h.cc && h.cc.length > 0) { out += hdr("Cc", h.cc.join(", ")); } @@ -347,19 +405,21 @@ function serializeMessageHeaders( // MIME part assembly // --------------------------------------------------------------------------- -/** - * Reject values that would break out of a MIME header. CR/LF in a header - * value is a header-injection vector; a double quote breaks the quoted - * `filename="..."` / `name="..."` forms the parser relies on. The MIME - * layer owns header well-formedness, so it fails loudly here rather than - * emitting a corrupt envelope. - */ -function assertHeaderSafe(value: string, field: string): void { +/** Reject a value that would break out of its header: a CR or LF ends the field. */ +function assertNoLineBreaks(value: string, field: string): void { if (/[\r\n]/.test(value)) { throw new Error( `${field} must not contain CR or LF: ${JSON.stringify(value)}`, ); } +} + +/** + * A double quote would close the `filename="..."` form, and this serializer + * emits no escape for an inner quote. + */ +function assertAttachmentHeaderSafe(value: string, field: string): void { + assertNoLineBreaks(value, field); if (value.includes('"')) { throw new Error( `${field} must not contain a double quote: ${JSON.stringify(value)}`, @@ -413,8 +473,8 @@ function assembleConversationSignedPart( // Attachment parts (BODY[1.2..N]) for (const att of attachments) { - assertHeaderSafe(att.contentType, "attachment contentType"); - assertHeaderSafe(att.name, "attachment name"); + assertAttachmentHeaderSafe(att.contentType, "attachment contentType"); + assertAttachmentHeaderSafe(att.name, "attachment name"); body += `--${boundary}${CRLF}`; body += `Content-Type: ${att.contentType}${CRLF}`; body += `Content-Transfer-Encoding: base64${CRLF}`; @@ -554,25 +614,90 @@ export function assembleSignedContent( // MIME parsing (for fetchHeaders, fetchStructure, fetchPart, fetchFull) // --------------------------------------------------------------------------- -const CRLF_CRLF = new Uint8Array([0x0d, 0x0a, 0x0d, 0x0a]); -const LF_LF = new Uint8Array([0x0a, 0x0a]); +const CR = 0x0d; +const LF = 0x0a; -function findByteSequence(haystack: Uint8Array, needle: Uint8Array): number { - if (needle.length === 0) return 0; - const limit = haystack.length - needle.length; - outer: for (let i = 0; i <= limit; i++) { - for (let j = 0; j < needle.length; j++) { - if (haystack[i + j] !== needle[j]) continue outer; +/** + * Refuse a header section that breaks a line with anything but CRLF (RFC 5321 + * §2.3.8, §4.1.1.4). The caller passes the section together with the blank line + * that ends it, so neither an LF-only section nor an `LF LF` separator survives. + * + * Neither reading of a bare break is safe. Splitting on it resolves a field + * from a value the sender smuggled inside one it controls; folding it collapses + * every field after it into the value of the field that carries it, suppressing + * each one. A non-conforming peer is owed an error rather than either parse. + */ +function assertNoBareLineBreaks(section: Uint8Array): void { + for (let i = 0; i < section.length; i++) { + const byte = section[i]!; + if (byte === CR && section[i + 1] === LF) { + i += 1; + continue; + } + if (byte === CR || byte === LF) { + throw new Error( + "parseHeaderSection: a header section must break its lines with CRLF", + ); } - return i; } - return -1; +} + +/** The length of the line break starting at `at`, or 0 when none starts there. */ +function lineBreakLength(raw: Uint8Array, at: number): number { + const byte = raw[at]; + if (byte === CR) return raw[at + 1] === LF ? 2 : 1; + if (byte === LF) return 1; + return 0; +} + +/** + * Locate the blank line that ends the header section: the first line break + * immediately followed by another, whichever flavour either uses. Returns null + * for a message carrying no blank line. + * + * Searching for `CRLF CRLF` alone walks past a blank line written with bare + * breaks and takes the section from a later offset, so every field before that + * offset reads as one field body. Admitting the bare blank line as a separator + * is the mirror of that: a sender who controls one field body ends the section + * early and strips the fields after it. This search locates the boundary and + * `assertNoBareLineBreaks` then refuses the section unless CRLF wrote it, + * including the blank line itself. + * + * The pair is the whole test, so a body part that carries no header fields -- + * legal under RFC 2046 §5.1.1, which makes every field of a part optional -- + * offers one leading break, and its content reads as the header section. A + * leading break is not admitted as a separator here because the message path + * relies on this scan to step over a spurious blank line and keep the fields + * behind it; a strict reading of a leading break would take those fields as + * body and erase the message's originator. The two paths want opposite + * readings of the same leading break, and this scan serves the message. + */ +function findHeaderBoundary( + raw: Uint8Array, +): { headerEnd: number; bodyOffset: number } | null { + let i = 0; + while (i < raw.length) { + const first = lineBreakLength(raw, i); + if (first === 0) { + i += 1; + continue; + } + const second = lineBreakLength(raw, i + first); + if (second !== 0) { + return { headerEnd: i, bodyOffset: i + first + second }; + } + i += first; + } + return null; } /** * Parse the header section of a raw RFC 2822 message. * Returns a map of lowercase header names to their values, and the * byte offset where the body starts. + * + * Throws for a header section that CRLF did not write, the blank line ending it + * included. A message with no blank line at all is all header section. */ export function parseHeaderSection(raw: Uint8Array): { headers: Map; @@ -581,22 +706,13 @@ export function parseHeaderSection(raw: Uint8Array): { } { const headers = new Map(); - // Search for the blank line separator in byte space so the returned - // offset is valid for Uint8Array.slice() even when headers contain - // multi-byte UTF-8 characters. - const crlfIdx = findByteSequence(raw, CRLF_CRLF); - const lfIdx = findByteSequence(raw, LF_LF); + // The boundary is located in byte space so the returned offset is valid for + // Uint8Array.slice() even when headers contain multi-byte UTF-8 characters. + const boundary = findHeaderBoundary(raw); + const headerEnd = boundary === null ? raw.length : boundary.headerEnd; + const bodyOffset = boundary === null ? raw.length : boundary.bodyOffset; - let bodyOffset = raw.length; - let headerEnd = raw.length; - - if (crlfIdx !== -1 && (lfIdx === -1 || crlfIdx <= lfIdx)) { - headerEnd = crlfIdx; - bodyOffset = crlfIdx + 4; - } else if (lfIdx !== -1) { - headerEnd = lfIdx; - bodyOffset = lfIdx + 2; - } + assertNoBareLineBreaks(raw.subarray(0, bodyOffset)); const headerText = new TextDecoder("utf-8", { fatal: false }).decode( raw.subarray(0, headerEnd), @@ -606,14 +722,30 @@ export function parseHeaderSection(raw: Uint8Array): { return { headers, bodyOffset, headerEnd }; } +/** + * Fold a bare CR or LF -- one not part of a CRLF -- to a space. Splitting on it + * would let a sender who controls a field body append a field of its own. + * + * Both callers read a region `parseHeaderSection` has already refused a bare + * break in, so nothing reaches this today. It stays because folding is the safe + * outcome for a region that refusal ever stops covering, and splitting is not. + */ +function foldBareLineBreaks(text: string): string { + return text.replace(/\r(?!\n)|(?): void { // Unfold continuation lines (lines starting with whitespace per RFC 2822). - const unfolded = headerSection - .replace(/\r\n[ \t]+/g, " ") - .replace(/\n[ \t]+/g, " "); - const lines = unfolded.split(/\r\n|\n/); + const unfolded = foldBareLineBreaks(headerSection).replace( + /\r\n[ \t]+/g, + " ", + ); + const lines = unfolded.split(CRLF); for (const line of lines) { if (line.trim() === "") continue; + // A field begins with a printable name character (RFC 2822 2.2), so a + // leading-WSP line continues nothing and names no field. + if (line.startsWith(" ") || line.startsWith("\t")) continue; const colon = line.indexOf(":"); if (colon === -1) continue; const name = line.slice(0, colon).trim().toLowerCase(); @@ -859,6 +991,112 @@ function parseDateHeader(value: string | undefined): string | null { return date.toISOString(); } +/** + * Replace every RFC 822 comment in a header value with a single space -- not + * nothing, or `ba(c)se64` composes a token the sender never wrote. Comments + * nest (RFC 822 §3.4.3), and a backslash quotes the next character (§3.4.5). + */ +function replaceCommentsWithSpace(value: string): string { + const kept: string[] = []; + let depth = 0; + for (let i = 0; i < value.length; i++) { + const ch = value.charAt(i); + if (depth > 0 && ch === "\\") { + i++; + continue; + } + if (ch === "(") { + if (depth === 0) kept.push(" "); + depth++; + continue; + } + if (ch === ")" && depth > 0) { + depth--; + continue; + } + if (depth === 0) kept.push(ch); + } + return kept.join(""); +} + +/** + * Resolve the Content-Transfer-Encoding mechanism a part declares. RFC 2045 + * §6.1 makes it a single case-insensitive token admitting no parameters. + */ +export function transferEncodingMechanism( + headers: Map, +): string { + const withoutComments = replaceCommentsWithSpace( + headers.get("content-transfer-encoding") ?? "", + ); + const semicolon = withoutComments.indexOf(";"); + const mechanism = + semicolon === -1 ? withoutComments : withoutComments.slice(0, semicolon); + const named = mechanism.trim().toLowerCase(); + // A field naming no mechanism takes the same default as an absent one. + return named === "" ? "7bit" : named; +} + +const RECOGNIZED_TRANSFER_ENCODINGS = new Set([ + "base64", + "quoted-printable", + "7bit", + "8bit", + "binary", +]); + +export function isRecognizedTransferEncoding(mechanism: string): boolean { + return RECOGNIZED_TRANSFER_ENCODINGS.has(mechanism); +} + +/** + * The content type a leaf part reports, after the RFC 2045 §6.4 rule that an + * entity under an unrecognized transfer encoding is treated as + * application/octet-stream. A `multipart/*` wrapper is exempt: its children + * are located from its declared type. The relabel governs what a part + * reports, not how it is routed. + */ +export function reportedContentType( + declaredContentType: string, + headers: Map, +): string { + if (extractContentTypeMime(declaredContentType).startsWith("multipart/")) { + return declaredContentType; + } + const mechanism = transferEncodingMechanism(headers); + if (isRecognizedTransferEncoding(mechanism)) return declaredContentType; + return "application/octet-stream"; +} + +/** + * The number of bytes widened per `String.fromCharCode` call. Large enough + * that the per-call cost is amortised over the buffer, small enough to stay + * well clear of the engine's limit on argument count. + */ +const BINARY_WIDEN_CHUNK_SIZE = 8192; + +/** + * Widen bytes into a string of one code unit per byte. A UTF-8 decode would + * fold multi-byte sequences and replace any byte that is not valid UTF-8, and + * no `TextDecoder` label widens bytes either: the WHATWG Encoding Standard + * makes "latin1" and "iso-8859-1" labels for windows-1252, which maps 27 of + * the byte values in 0x80-0x9f to other code points. + * + * Widening runs a chunk at a time. One `String.fromCharCode` per byte + * allocates a single-character string per byte plus an array to hold them, + * which on a body of tens of megabytes costs several times the body's own + * size in resident memory, on a caller that runs synchronously. + */ +function bytesToBinaryString(bytes: Uint8Array): string { + let out = ""; + for (let i = 0; i < bytes.length; i += BINARY_WIDEN_CHUNK_SIZE) { + out += String.fromCharCode( + ...bytes.subarray(i, i + BINARY_WIDEN_CHUNK_SIZE), + ); + } + return out; +} + /** * Decode a MIME body part, handling Content-Transfer-Encoding. */ @@ -866,9 +1104,7 @@ function decodeBodyBytes( body: Uint8Array, headers: Map, ): { value: string; isEncodingProblem: boolean } { - const cte = (headers.get("content-transfer-encoding") ?? "7bit") - .trim() - .toLowerCase(); + const cte = transferEncodingMechanism(headers); if (cte === "base64") { try { @@ -889,11 +1125,15 @@ function decodeBodyBytes( return { value: decodeQuotedPrintable(raw), isEncodingProblem: false }; } - // 7bit, 8bit, binary — decode as UTF-8 - return { - value: new TextDecoder("utf-8", { fatal: false }).decode(body), - isEncodingProblem: false, - }; + if (cte === "7bit" || cte === "8bit" || cte === "binary") { + return { + value: new TextDecoder("utf-8", { fatal: false }).decode(body), + isEncodingProblem: false, + }; + } + + // RFC 2045 §6.4: an unrecognized mechanism is a byte stream, not text. + return { value: bytesToBinaryString(body), isEncodingProblem: true }; } function decodeQuotedPrintable(text: string): string { @@ -925,7 +1165,7 @@ function isAttachmentPart( return true; } -function extractContentTypeMime(contentType: string): string { +export function extractContentTypeMime(contentType: string): string { return contentType.split(";")[0]!.trim().toLowerCase(); } @@ -963,9 +1203,9 @@ function walkMimePart( ctx: WalkContext, ): void { const part = parseMimePart(partBytes); - const mime = extractContentTypeMime(part.contentType); + const declaredMime = extractContentTypeMime(part.contentType); - if (mime.startsWith("multipart/")) { + if (declaredMime.startsWith("multipart/")) { const boundary = extractBoundary(part.contentType); if (boundary === undefined) return; const subParts = parseMultipart(part.body, boundary); @@ -975,12 +1215,14 @@ function walkMimePart( return; } + const reportedMime = reportedContentType(declaredMime, part.headers); + if (isAttachmentPart(part.contentType, part.headers)) { const blobId = `blob_${ctx.mailId}_${partPath}`; ctx.attachments.push({ blobId, name: extractFilename(part.headers), - type: mime, + type: reportedMime, size: part.body.length, }); return; @@ -989,10 +1231,12 @@ function walkMimePart( const decoded = decodeBodyBytes(part.body, part.headers); ctx.bodyValues[partPath] = decoded; - if (mime === "text/plain") { - ctx.textBody.push({ partId: partPath, type: mime }); - } else if (mime === "text/html") { - ctx.htmlBody.push({ partId: partPath, type: mime }); + // Listed on the declared type but reported under the relabeled one: a + // consumer walking `textBody` would otherwise never learn the part exists. + if (declaredMime === "text/plain") { + ctx.textBody.push({ partId: partPath, type: reportedMime }); + } else if (declaredMime === "text/html") { + ctx.htmlBody.push({ partId: partPath, type: reportedMime }); } } @@ -1042,13 +1286,18 @@ export function parseMailToEmail(raw: Uint8Array, mailId: string): JMAPEmail { }); } } else { - // Single-part message (e.g. text/plain). - // Reconstruct minimal part bytes with content-type header so parseMimePart works. + // Single-part message. Reconstruct minimal part bytes so parseMimePart + // works; the message's own Content-Transfer-Encoding travels with the type. const enc = new TextEncoder(); - const ctHeader = `Content-Type: ${contentType}\r\n\r\n`; - const partBytes = new Uint8Array(enc.encode(ctHeader).length + body.length); - partBytes.set(enc.encode(ctHeader), 0); - partBytes.set(body, enc.encode(ctHeader).length); + const cte = msgHeaders.get("content-transfer-encoding"); + const partHeaderText = + `Content-Type: ${contentType}\r\n` + + (cte === undefined ? "" : `Content-Transfer-Encoding: ${cte}\r\n`) + + "\r\n"; + const partHeaderBytes = enc.encode(partHeaderText); + const partBytes = new Uint8Array(partHeaderBytes.length + body.length); + partBytes.set(partHeaderBytes, 0); + partBytes.set(body, partHeaderBytes.length); walkMimePart(partBytes, "1", ctx); } @@ -1077,17 +1326,14 @@ export function parseMailToEmail(raw: Uint8Array, mailId: string): JMAPEmail { * Decode a MIME part body into raw bytes, honoring Content-Transfer-Encoding. * * Unlike `decodeBodyBytes` (which produces a JMAP string value), this returns - * the actual bytes for reconstructing a `MessageAttachment`. A malformed - * base64 body surfaces as a thrown error rather than a silent best-effort - * decode — attachment integrity is load-bearing. + * the actual bytes. A malformed base64 body throws; an unrecognized mechanism + * does not — RFC 2045 §6.4 hands back the bytes as they came. */ -function decodeAttachmentBytes( +export function decodePartBytes( body: Uint8Array, headers: Map, ): Uint8Array { - const cte = (headers.get("content-transfer-encoding") ?? "7bit") - .trim() - .toLowerCase(); + const cte = transferEncodingMechanism(headers); if (cte === "base64") { const raw = new TextDecoder("utf-8", { fatal: false }).decode(body); @@ -1104,13 +1350,8 @@ function decodeAttachmentBytes( return out; } - if (cte === "7bit" || cte === "8bit" || cte === "binary") { - return body; - } - - throw new Error( - `decodeAttachmentBytes: unsupported content-transfer-encoding "${cte}"`, - ); + // Identity encoding, including anything unrecognized (RFC 2045 §6.4). + return body; } /** @@ -1152,8 +1393,11 @@ export function extractAttachments(raw: Uint8Array): MessageAttachment[] { if (!isAttachmentPart(subPart.contentType, subPart.headers)) continue; attachments.push({ name: extractFilename(subPart.headers) ?? "attachment", - contentType: extractContentTypeMime(subPart.contentType), - data: decodeAttachmentBytes(subPart.body, subPart.headers), + contentType: reportedContentType( + extractContentTypeMime(subPart.contentType), + subPart.headers, + ), + data: decodePartBytes(subPart.body, subPart.headers), }); } return attachments; @@ -1175,7 +1419,6 @@ function isInterchangeType(s: string): s is InterchangeType { export function buildMessageHeaders( headers: Map, ): ParsedMessageHeaders { - const from = headers.get("from") ?? ""; const toRaw = headers.get("to") ?? ""; const to = toRaw ? toRaw @@ -1184,10 +1427,30 @@ export function buildMessageHeaders( .filter(Boolean) : []; - const date = headers.get("date") ?? ""; - const messageId = headers.get("message-id") ?? ""; + const result: ParsedMessageHeaders = { to }; + + // A blank header below is recorded as an absence, not an empty value: RFC + // 5322 admits no empty `Date`, `Message-ID`, `In-Reply-To` or `From` body. + // + // A `Date` that does not parse is an absence too, which the parse check below + // covers along with the blank one. A consumer builds a `Date` from this string + // and compares it to a window's bounds; every comparison against an Invalid + // Date is false, so a value kept here places the message inside every date + // window, including two that exclude each other. An absence falls outside all + // of them. + const date = headers.get("date"); + if (date !== undefined && !Number.isNaN(new Date(date).getTime())) { + result.date = date; + } + + const messageId = headers.get("message-id"); + if (messageId !== undefined && messageId.trim().length > 0) { + result.messageId = messageId; + } - const result: ParsedMessageHeaders = { from, to, date, messageId }; + // A non-blank value rides through verbatim even when it is not parseable. + const from = headers.get("from"); + if (from !== undefined && from.trim().length > 0) result.from = from; const ccRaw = headers.get("cc"); if (ccRaw !== undefined) { @@ -1205,7 +1468,9 @@ export function buildMessageHeaders( } const inReplyTo = headers.get("in-reply-to"); - if (inReplyTo !== undefined) result.inReplyTo = inReplyTo; + if (inReplyTo !== undefined && inReplyTo.trim().length > 0) { + result.inReplyTo = inReplyTo; + } const subject = headers.get("subject"); if (subject !== undefined) result.subject = subject; @@ -1257,21 +1522,30 @@ function parseRawHeaders( raw: Uint8Array, headerEnd: number, ): Record { - const text = new TextDecoder("utf-8", { fatal: false }).decode( - raw.subarray(0, headerEnd), + const text = foldBareLineBreaks( + new TextDecoder("utf-8", { fatal: false }).decode( + raw.subarray(0, headerEnd), + ), ); - const out: Record = {}; + // `__proto__` and `constructor` are legal RFC 5322 field names that collide + // with Object.prototype members, so the accumulator is a Map. + const out = new Map(); let current: { name: string; value: string } | null = null; const flush = (): void => { if (current === null) return; const key = current.name.trim().toLowerCase(); - (out[key] ??= []).push(current.value.trim()); + const values = out.get(key) ?? []; + values.push(current.value.trim()); + out.set(key, values); current = null; }; - for (const line of text.split(/\r\n|\n/)) { - if (line === "") break; - if ((line.startsWith(" ") || line.startsWith("\t")) && current !== null) { - current.value += ` ${line.trim()}`; + for (const line of text.split(CRLF)) { + // `headerEnd` bounds the section already, so a blank line here is a + // leading one, not the terminator; breaking would drop every field. + if (line === "") continue; + // Nothing to continue means no field (RFC 2822 2.2), not a field of its own. + if (line.startsWith(" ") || line.startsWith("\t")) { + if (current !== null) current.value += ` ${line.trim()}`; continue; } const idx = line.indexOf(":"); @@ -1280,7 +1554,10 @@ function parseRawHeaders( current = { name: line.slice(0, idx), value: line.slice(idx + 1) }; } flush(); - return out; + // Null prototype for the same reason: an absent name must resolve to nothing. + const result: Record = Object.create(null); + for (const [name, values] of out) result[name] = values; + return result; } function parseDisposition( @@ -1316,8 +1593,8 @@ function collectLeafParts(partBytes: Uint8Array): MessagePart[] { return parseMultipart(part.body, boundary).flatMap(collectLeafParts); } const result: MessagePart = { - contentType: mime, - content: decodeAttachmentBytes(part.body, part.headers), + contentType: reportedContentType(mime, part.headers), + content: decodePartBytes(part.body, part.headers), }; const filename = extractFilename(part.headers); if (filename !== null) result.filename = filename; diff --git a/vendor/intx/types/VENDORED-FROM b/vendor/intx/types/VENDORED-FROM index 77c532f935..7c751ad83b 100644 --- a/vendor/intx/types/VENDORED-FROM +++ b/vendor/intx/types/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/types) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. CL-7443 retires CL-7190's local delta ("message_response" added to signals.ts's signalKinds, alongside signalKindToGateType): `ask_user` posts its question and ends the turn instead of parking a signal, so signals.ts is back to upstream's approval-only shape. CL-6396: AgentEventFrame gains an optional childRunId so a sidecar can name the occurrence (turn__) the event belongs to. +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). CL-7443 retires CL-7190's local delta ("message_response" added to signals.ts's signalKinds, alongside signalKindToGateType): `ask_user` posts its question and ends the turn instead of parking a signal, so signals.ts is back to upstream's approval-only shape. CL-6396: AgentEventFrame gains an optional childRunId so a sidecar can name the occurrence (turn__) the event belongs to. CL-7510: catalog.ts gains the "openai-responses" model-provider plugin id, which the workbench loopback-OAuth providers (Codex, xai-oauth) ride. CL-7508: sidecar.ts hosts the sidecar loopback-login token shape (OAuthLoginTokens: access/refresh/expiresAt/idToken/accountId). Absorbed upstream at this re-vendor: the message-id.ts blank-line handling, the runtime.ts CONVERSATION_TYPES/isConversationType enumeration, and the tenants.ts TenantSlug validation. diff --git a/vendor/intx/types/src/message-id.ts b/vendor/intx/types/src/message-id.ts index 523afffe87..cd96d94cfc 100644 --- a/vendor/intx/types/src/message-id.ts +++ b/vendor/intx/types/src/message-id.ts @@ -18,10 +18,15 @@ import { hexEncode } from "./hex"; +const BARE_LINE_BREAK = /\r(?!\n)|(? { const messageIdFromHeader = parseMessageIdHeader(rawMessage); @@ -38,30 +43,32 @@ export async function deriveMessageId(rawMessage: Uint8Array): Promise { /** * Parse the `Message-ID` header value from a raw message, or `null` when - * the message carries no such header. + * the message carries no such header or carries a blank one. * - * The parser walks the message until the headers/body separator - * (`CRLF CRLF` per RFC 2822 §2.1, with the lone-`LF` variant tolerated to - * match common in-memory senders). Header-field unfolding follows RFC - * 2822 §2.2.3: a continuation line begins with whitespace and appends to - * the prior line. Header-name comparison is case-insensitive per RFC 2822 - * §1.2.2. + * The header section ends at the first blank line, whichever line-break flavour + * wrote it, and the whole message is the section when it carries no blank line. + * CRLF is the sole line terminator (RFC 5321 §2.3.8, §4.1.1.4), matching + * `@intx/mime`'s decoder, which refuses a section any other break wrote -- an + * `LF LF` separator included. Such a message names no id here and takes the + * digest in `deriveMessageId` instead, because reading one out of text that + * parser refuses would fold the fields after the break, and the head of the + * body, into this identifier. Header-field unfolding follows RFC 2822 §2.2.3: a + * continuation line begins with whitespace and appends to the prior line. + * Header-name comparison is case-insensitive per RFC 2822 §1.2.2. */ export function parseMessageIdHeader(rawMessage: Uint8Array): string | null { - const text = new TextDecoder("utf-8", { fatal: false }).decode(rawMessage); - // Headers end at the first blank line. RFC 2822 mandates `CRLF CRLF` - // but tolerate `LF LF` for callers that normalize line endings. - let headerSection = text; - const crlfBoundary = text.indexOf("\r\n\r\n"); - const lfBoundary = text.indexOf("\n\n"); - if (crlfBoundary >= 0 && (lfBoundary < 0 || crlfBoundary < lfBoundary)) { - headerSection = text.slice(0, crlfBoundary); - } else if (lfBoundary >= 0) { - headerSection = text.slice(0, lfBoundary); - } + const raw = new TextDecoder("utf-8", { fatal: false }).decode(rawMessage); + // Locating the blank line before checking the breaks is what keeps a `CRLF + // CRLF` the sender left in the body from becoming the separator: the section + // would then be read from that later offset, absorbing every field before it. + const boundary = BLANK_LINE.exec(raw); + const throughBoundary = + boundary === null ? raw.length : boundary.index + boundary[0].length; + if (BARE_LINE_BREAK.test(raw.slice(0, throughBoundary))) return null; + const headerSection = boundary === null ? raw : raw.slice(0, boundary.index); // Unfold continuation lines (a line starting with WSP belongs to // the prior header field). - const lines = headerSection.split(/\r?\n/); + const lines = headerSection.split("\r\n"); const unfolded: string[] = []; for (const line of lines) { if (line.length > 0 && (line[0] === " " || line[0] === "\t")) { @@ -76,7 +83,12 @@ export function parseMessageIdHeader(rawMessage: Uint8Array): string | null { if (colon < 0) continue; const name = line.slice(0, colon).trim().toLowerCase(); if (name !== "message-id") continue; - return line.slice(colon + 1).trim(); + const value = line.slice(colon + 1).trim(); + // A blank header names no id (RFC 2822 §3.6.4 admits no empty `msg-id`). + // Returning `""` would give every such message the same id, and the dedup + // key built on it would discard the second as a redelivery of the first. + if (value.length === 0) return null; + return value; } return null; } diff --git a/vendor/intx/types/src/runtime.ts b/vendor/intx/types/src/runtime.ts index c71574348b..6bb000fcd2 100644 --- a/vendor/intx/types/src/runtime.ts +++ b/vendor/intx/types/src/runtime.ts @@ -128,11 +128,22 @@ export const InterchangeType = type.enumerated( ); export type InterchangeType = typeof InterchangeType.infer; +const CONVERSATION_TYPES: ReadonlySet = new Set([ + "conversation.message", + "conversation.join", + "conversation.leave", +]); + /** - * Attachment for an outbound message. Content is raw bytes; the transport - * handles Content-Transfer-Encoding (base64 for binary, quoted-printable - * for 8-bit text). + * Membership is enumerated rather than derived from the `conversation.` name + * prefix, so a member added to InterchangeType is not a conversation type until + * it is listed in CONVERSATION_TYPES. */ +export function isConversationType(value: InterchangeType): boolean { + return CONVERSATION_TYPES.has(value); +} + +/** Attachment for an outbound message. Content is raw bytes. */ export type MessageAttachment = { name: string; contentType: string; @@ -205,13 +216,16 @@ export type SendReceipt = { /** * Parsed headers from an inbound message. Field names follow RFC 5322 and * the Interchange-specific header conventions from MESSAGE.md § Headers. + * + * `from` is optional because a message can arrive carrying no originator at + * all, and there is no string that honestly stands for one. */ export type MessageHeaders = { - from: string; + from?: string; to: string[]; cc?: string[]; - date: string; - messageId: string; + date?: string; + messageId?: string; inReplyTo?: string; references?: string[]; subject?: string; @@ -253,9 +267,11 @@ export type SignatureStatus = typeof SignatureStatus.infer; * produces it. * * - `clean` — nothing suspect; always admitted - * - `untrustedFrom` — the visible `From` cannot be trusted, either because it is - * present but unparseable or because a valid signature is worn under a - * mismatched sender identity + * - `untrustedFrom` — the visible `From` is present but cannot be reduced to a + * single addr-spec + * - `mismatchedFrom` — the visible `From` names a different identity from the + * sender the hub stamped + * - `absentFrom` — the message carries no usable `From` * - `invalid` — the signature check failed (tampering or the wrong key) * - `missing` — the message carried no signature * - `unknown` — no key was available to verify against @@ -264,6 +280,8 @@ export type SignatureStatus = typeof SignatureStatus.infer; export const InboundMailOutcome = type.enumerated( "clean", "untrustedFrom", + "mismatchedFrom", + "absentFrom", "invalid", "missing", "unknown", @@ -280,6 +298,8 @@ export type InboundMailOutcome = typeof InboundMailOutcome.infer; */ export const AuthorControllableOutcome = type.enumerated( "untrustedFrom", + "mismatchedFrom", + "absentFrom", "invalid", "missing", "unknown", @@ -288,7 +308,7 @@ export type AuthorControllableOutcome = typeof AuthorControllableOutcome.infer; /** * A per-workflow inbound-mail admission policy: for each admission outcome the - * author may control, whether a message that resolved to that outcome is + * author may control, whether a message that raises that outcome as a finding is * `reject`ed or `admit`ted. The key set is exactly the * {@link AuthorControllableOutcome} values -- `clean` (always admitted) and * `error` (pinned to reject) are deliberately not keys. @@ -303,12 +323,34 @@ export type AuthorControllableOutcome = typeof AuthorControllableOutcome.infer; */ export const InboundMailPolicy = type({ "untrustedFrom?": "'reject' | 'admit'", + "mismatchedFrom?": "'reject' | 'admit'", + "absentFrom?": "'reject' | 'admit'", "invalid?": "'reject' | 'admit'", "missing?": "'reject' | 'admit'", "unknown?": "'reject' | 'admit'", }).onUndeclaredKey("reject"); export type InboundMailPolicy = typeof InboundMailPolicy.infer; +type AssertEqual = [A] extends [B] + ? [B] extends [A] + ? true + : false + : false; + +/** + * The three outcome vocabularies must move together, but nothing else connects + * them: each spells the same names out again in its own notation. + */ +const _policyKeysMatchAuthorControllable: AssertEqual< + keyof InboundMailPolicy, + AuthorControllableOutcome +> = true; + +const _authorControllableAreOutcomes: AssertEqual< + Exclude, + never +> = true; + /** * A parsed MIME part. `content` is the DECODED bytes in memory (the * transfer-encoding has already been undone). `filename` and `disposition` are @@ -322,9 +364,9 @@ export type MessagePart = { filename?: string; disposition?: "inline" | "attachment"; /** - * Original Content-Transfer-Encoding, when a producer chooses to record it. - * Not set for a decoded mail part -- `content` is already decoded, so the - * wire encoding is spent transport metadata. + * The Content-Transfer-Encoding the part declared. Not set for a decoded + * mail part -- `content` is already decoded, so the wire encoding is spent + * transport metadata. */ encoding?: string; }; @@ -377,14 +419,17 @@ export type Mail = { }; const MailShape = type({ - // Require the header fields a consumer dereferences unconditionally (the - // sender/recipient a projection reads); other header fields stay optional - // and are carried losslessly in `rawHeaders`. + // Require the recipient list a consumer dereferences unconditionally; other + // header fields stay optional and are carried losslessly in `rawHeaders`. + // `from` is optional here deliberately: a mail with no usable originator is + // still mail, and requiring it makes `isMail` reject one. headers: { - from: "string", + "from?": "string", to: "string[]", }, - rawHeaders: "object", + rawHeaders: { + "[string]": "string[]", + }, parts: type({ contentType: "string", ref: "string", @@ -683,6 +728,42 @@ export interface MessageTransport { ): Promise; } +/** + * The condition a `MessageTransport` operation failed under. A transport that + * grows a condition this union does not carry takes its name from RFC 5530 § 3 + * rather than coining one. + * + * https://www.rfc-editor.org/rfc/rfc5530.html + */ +export const MessageTransportCondition = type( + "'NONEXISTENT' | 'CANNOT' | 'SERVERBUG'", +); +export type MessageTransportCondition = typeof MessageTransportCondition.infer; + +export class MessageTransportError extends Error { + readonly condition: MessageTransportCondition; + + constructor(condition: MessageTransportCondition, message: string) { + super(message); + this.name = "MessageTransportError"; + this.condition = condition; + } +} + +/** + * `instanceof` is not usable for this: a tool package is published as a bundle + * whose workspace imports are inlined (`bin/build-builtins.ts`), so a consumer + * loaded from a bundle holds its own copy of the class above and answers false + * against an error the host's copy constructed. + */ +export function isMessageTransportError( + value: unknown, +): value is Error & { readonly condition: MessageTransportCondition } { + if (!(value instanceof Error)) return false; + if (!("condition" in value)) return false; + return MessageTransportCondition.allows(value.condition); +} + // --------------------------------------------------------------------------- // Tool Execution (ARCHITECTURE.md § Tools, INFERENCE.md § Tool Execution) // --------------------------------------------------------------------------- @@ -803,7 +884,6 @@ export const ToolResult = type({ "pendingMarker?": { status: "'pending'", correlationId: "string", - "expectedFrom?": "string", }, }); export type ToolResult = typeof ToolResult.infer; @@ -1555,7 +1635,7 @@ export const InferenceEvent = type.or( type: "'message.run.started'", seq: "number", data: { - messageId: "string", + "messageId?": "string", messageRunId: "string", receivedAt: "number", }, @@ -1565,7 +1645,7 @@ export const InferenceEvent = type.or( seq: "number", data: { messageRunId: "string", - messageId: "string", + "messageId?": "string", status: type.enumerated("completed", "failed"), "error?": { message: "string", @@ -1802,7 +1882,7 @@ export type InferenceEvent = type: "message.run.started"; seq: number; data: { - messageId: string; + messageId?: string; messageRunId: string; receivedAt: number; }; @@ -1811,7 +1891,8 @@ export type InferenceEvent = /** * Per-message run-bracket close. Pairs with `message.run.started` * by `messageRunId`. `messageId` is carried redundantly so log - * readers can correlate without a join against the open event. + * readers can correlate without a join against the open event; it is + * absent for a message that named no id of its own. * * The `status` enum is `"completed" | "failed"` only. * Cancellation lives in the workflow-runtime's @@ -1833,7 +1914,7 @@ export type InferenceEvent = seq: number; data: { messageRunId: string; - messageId: string; + messageId?: string; status: "completed" | "failed"; error?: { message: string; @@ -1944,7 +2025,6 @@ export function parseInferenceEvent( export type PendingOperation = { correlationId: string; kind: SignalKind; - expectedFrom?: string; registeredAt: number; gateId: string; /** @@ -2737,8 +2817,8 @@ export type ContextCommit = { * re-declaring the shape. */ export const ConnectorThreadState = type({ - threadRoot: "string", - lastMessageId: "string", + "threadRoot?": "string", + "lastMessageId?": "string", replyTo: "string", cc: "string[]", "subject?": "string", diff --git a/vendor/intx/types/src/tenants.ts b/vendor/intx/types/src/tenants.ts index bf1ffa7831..033e266c3f 100644 --- a/vendor/intx/types/src/tenants.ts +++ b/vendor/intx/types/src/tenants.ts @@ -8,9 +8,23 @@ export const TenantConfig = type({ }); export type TenantConfig = typeof TenantConfig.infer; +/** + * A tenant slug, constrained to a single DNS label: the create route derives + * `tenant.domain` from the lowercased slug, and that domain is both the sender + * stamp on outbound mail and the recipient run address. The grammar is RFC 1035 + * section 2.3.1 as relaxed by RFC 1123 section 2.1. + * + * https://www.rfc-editor.org/rfc/rfc1035#section-2.3.1 + * https://www.rfc-editor.org/rfc/rfc1123#section-2.1 + */ +export const TenantSlug = type( + /^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?$/, +).and("string<=63"); +export type TenantSlug = typeof TenantSlug.infer; + export const CreateTenant = type({ name: "string", - slug: "string", + slug: TenantSlug, "parentId?": "string | null", }); diff --git a/vendor/intx/workflow-deploy/README.md b/vendor/intx/workflow-deploy/README.md index 45f99ae078..081a11177f 100644 --- a/vendor/intx/workflow-deploy/README.md +++ b/vendor/intx/workflow-deploy/README.md @@ -25,20 +25,78 @@ Public surface: - `walkCapabilities(workflow, registry, pluginDefs)` — the pure capability walk; reused to populate per-step capability declarations and as the input to the approval gate. +- `createApprovalSet(grants, requirements)` — the edge of the approval + vocabulary. An `ApprovalSet` is one approved surface behind one operator + decision, carrying the two kinds of approved item in two fields: + `grants`, a `Set` of the grant-shape strings the walk surfaces, and + `requirements`, the `GrantRequirement` records a definition declares. + The fields are separate because the two membership tests are different + operations — a string is found by `Set.has`, a record only by structural + comparison — so each kind sits beside the test that can answer for it. + Every requirement is validated here, which is what lets the comparison + below use `node:util`'s `isDeepStrictEqual` safely: that predicate treats + a present-but-undefined optional key as different from an absent one, and + `GrantRequirement` admits an object or `null` for `conditions`, never + `undefined`, so such a record is rejected loudly at construction instead + of silently failing to match at the gate. +- `approvalSetFromItems(items)` / `approvalItemsFromSet(approvals)` — the + boundary between the in-memory `ApprovalSet` and the flat `ApprovalItem` + list a freeze persists (`FrozenApprovalBundle.approvedGrants`). The + persisted form stays flat and the in-memory form does not, because the two + owe different things: a stored row owes compatibility to rows already + written, and a flat array is append-friendly and order-independent, while + the code reading it owes itself a shape where each kind sits beside its own + membership test. `approvalSetFromItems` partitions by kind and routes the + requirement half through `createApprovalSet`, so a rehydrated record gets + the same parse a freshly-built one does. - `createApprovalSetGate(approvals)` / `createApprovalSourceGate(source)` - — operator-approval gating against a flat `ApprovalSet` or an async - source. + — operator-approval gating against an `ApprovalSet` or an async + source. Both gates check only `grants`, because the walk never surfaces + a requirement. The requirement half is gated by + `gateAndFreezeProbeResult` in `@intx/hub-sessions` — the code-sourced + path is where a declared requirement arrives, outside the wire-hash + preimage the walk covers. +- `isApprovedGrantRequirement(approvals, requirement)` — whether the + operator approved a declared requirement, compared as a whole record + (`source`, `resource`, `action`, `effect`, `conditions`) against + `approvals.requirements`. A requirement is multi-axis and has no + grant-string form, so the comparison is structural. - `pickStepInferenceSource(...)` / `pinInertStepSources(...)` / - `buildInertProjectionStepSources(...)` — resolve each step's inference - source against the operator-approved grant set, so an unapproved source - fails the deploy closed. `pinInertStepSources` is the shared walk (step - order, loop-body recursion, flat-map collision rule) parameterized by a - per-step leaf resolver. + `buildInertProjectionStepSources(...)` / `buildInertBodyStepSources(...)` + — resolve each step's inference source against the operator-approved + grant set, so an unapproved source fails the deploy closed. + `pinInertStepSources` owns the flat-map collision rule and is + parameterized by a per-step leaf resolver; it traverses through + `walkStepTree` from `@intx/workflow` under `LOOP_BODY_DESCENT`, the + descent that stays inside one flat step-id namespace. + `buildInertBodyStepSources` pins a lifted body: an + agent-bearing step resolves through the gate, while a step that cannot + invoke inference takes the deploy's default source as an inert + placeholder. +- `collectAgentBearingStepIds(...)` — the ids of the steps that can actually + invoke inference (`agent`, or `map` over one). Every other primitive is + pinned a source to satisfy the wire requirement that each step carry one but + never issues a request through it, so a consumer deciding what a step is + entitled to — credential delivery, notably — asks this rather than reading + the pinned map. - `enumerateInertBodies(...)` — lift each inline trigger body (onTrigger section or childWorkflow child), transitively, out of a frozen inert projection so the hub can stage it and pin its per-step sources. The enumeration is purely structural; each body step's `(provider, model)` - preference is read at pin time by `pinInertStepSources`. + preference is read at pin time by `buildInertBodyStepSources`. +- `inertNestedBodies(step, descent)` — the nested body projections one step of + a frozen inert projection carries, filtered by a `StepWalkDescent`. This is + the inert counterpart of `@intx/workflow`'s live `nestedWorkflowBodies`, so + the inert representation states its descent rule in one place too. Pair it + with `walkStepTree` to walk an inert projection under any descent; + `inertFlatNamespaceStepIds` is that pairing at `LOOP_BODY_DESCENT`. + It dispatches through a table keyed by `Primitive["kind"]` — the same union + the live reader's `never` assignment guards — so a body-bearing primitive + added to one reader and forgotten in the other is a compile error rather + than a container that silently reads as a leaf. A step whose kind is outside + that closed set throws, as does a container kind carrying neither an inline + body nor a `{ ref }`; a `{ ref }` body yields nothing, because the + referenced asset is deployed and walked on its own. - `deriveRunAddress` / `deriveStepAddress` / `resolveStepAddress` / `deriveRunAgentId` / `deriveStepAgentId` / `deriveWorkflowRunRepoId` — the pure address and id derivation helpers. diff --git a/vendor/intx/workflow-deploy/VENDORED-FROM b/vendor/intx/workflow-deploy/VENDORED-FROM index dfd09f1070..ca431d962a 100644 --- a/vendor/intx/workflow-deploy/VENDORED-FROM +++ b/vendor/intx/workflow-deploy/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/workflow-deploy) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. No source delta. diff --git a/vendor/intx/workflow-host/VENDORED-FROM b/vendor/intx/workflow-host/VENDORED-FROM index be45d8f62f..d72193c16a 100644 --- a/vendor/intx/workflow-host/VENDORED-FROM +++ b/vendor/intx/workflow-host/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/workflow-host) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. CL-6448: the suspendable-child (onTrigger body) spawn seam threads the parent child's credentials-backed authorize and MailPartReader: RunSuspendableChild's input and createInMemorySpawnSuspendableChild's opts gain the two optional fields (adapters/spawn-child.ts), and child/run-child.ts hoists `authorize` above the body resolver and passes both when building it, so a tool-bearing body agent gates its tool calls through the same per-step grant snapshot a top-level step does and reads an attachments-only inbound mail's parts instead of throwing. Upstream runs bodies toolless and has no analog. child/run-child.ts also resolves a step's grants entry through findStepGrantsEntry (head collapse: a body step whose own id is absent from a single-step deployment's snapshot resolves to the sole entry). CL-6396: the event-channel envelope, sender, and receiver carry an optional childRunId; run-child stamps spawnInput.childRunId onto body-child events so overlapping turns HMAC-bind the occurrence they belong to. Retired at this pin: the CL-6448 credentialWiring thread (upstream now passes the run's live credential-material cell into every body spawn natively as `credentialMaterial`). Retired at earlier pins: the empty-mail drop (CL-6164) and the action/loop runtime bind (CL-6325). Temporary delta: buildInboundMessageFromMail omits `subject` when the decoded header is empty (a present-but-textless Subject header), since createInboundMessage rejects an empty subject and fails the step; see VENDORED.md's kill condition. +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). CL-6448: the suspendable-child spawn seam threads the parent child's credentials-backed authorize and MailPartReader (adapters/spawn-child.ts opts credentialWiring/mailPartReader, hoisted above the body resolver in child/run-child.ts), so a tool-bearing body agent gates its tool calls through the same per-step grant snapshot a top-level step does and reads an attachments-only inbound mail's parts instead of throwing; the sidecar still passes both (workflow-substrate-factory.ts bridges CredentialWiring onto upstream's newer native credentialMaterial cell). child/run-child.ts resolves a step's grants entry through findStepGrantsEntry (head collapse: a body step whose own id is absent from a single-step deployment's snapshot resolves to the sole entry). CL-6396: the event-channel envelope, sender, and receiver carry an optional childRunId; run-child stamps spawnInput.childRunId onto body-child events so overlapping turns HMAC-bind the occurrence they belong to. Temporary delta: buildInboundMessageFromMail omits `subject` when the decoded header is empty (a present-but-textless Subject header), since createInboundMessage rejects an empty subject and fails the step; see VENDORED.md's kill condition. Absorbed upstream at this re-vendor: the step-invoker describeReplyFailure/usableAddr rewording, the substrate-mailbox-store nullable-header widening, the supervisor-backed-transport MessageTransportError codes, the control-channel optional from/date/messageId, and the supervisor stored-envelope messageId/date narrowing. Retired at earlier pins: the empty-mail drop (CL-6164) and the action/loop runtime bind (CL-6325). diff --git a/vendor/intx/workflow-host/src/adapters/step-invoker.ts b/vendor/intx/workflow-host/src/adapters/step-invoker.ts index 59d2a66d46..063f1605b3 100644 --- a/vendor/intx/workflow-host/src/adapters/step-invoker.ts +++ b/vendor/intx/workflow-host/src/adapters/step-invoker.ts @@ -341,6 +341,20 @@ async function invokeColdStep( ); } +/** + * The connector router lives in `@intx/harness`, which this package does not + * depend on, so its error is recognized by `name` rather than `instanceof`. + */ +function describeReplyFailure(cause: unknown): string { + if (cause instanceof Error && cause.name === "NoActiveConnectorThreadError") { + return ( + "the warm agent has no active connector thread, so its auto-reply " + + "names nobody to reply to and was never sent" + ); + } + return "the warm agent's auto-reply send failed"; +} + /** * Warm-keep path (design §3b). The agent is built once on the first * invocation and cached under the step's identity; every later @@ -449,17 +463,16 @@ async function invokeWarmStep( // the auto-reply reaches the transport. Only a reply turn produces a // `connector.reply`; a suspended/gate turn produces none, so it must NOT // await the barrier (that reply never arrives and the wait would hang). - // A failed send resolves the barrier with `ok: false`: fail the turn so - // the inbound mail is not consumed as replied and the run's claim-check - // replays it (at-least-once via reprocessing) rather than dropping the - // reply. + // Fail the turn rather than return a step result claiming a reply that + // never went out. The mail is still consumed, because a replay would fail + // to compose the same reply again. if (replyDrive !== null && sendResult.type === "reply") { const settlement = await replyDrive.waitForReplyAfter(replySeqBeforeSend); if (!settlement.ok) { throw new Error( - "workflow step invoker: the warm agent's auto-reply send failed; " + - "failing the turn so the inbound mail replays rather than being " + - "consumed with the reply dropped", + `workflow step invoker: ${describeReplyFailure(settlement.cause)}; ` + + "failing the turn, so the inbound mail is consumed without a " + + "reply rather than replayed", { cause: settlement.cause }, ); } @@ -740,15 +753,26 @@ async function buildSendMessage( return { message: synthesizeInputContent(rawInput), mailInbound: null }; } -/** Extract a bare addr-spec from a header value, or fall back to a synthetic - * local address when the value is absent or unparseable. */ -function safeAddr(raw: string | undefined, fallback: string): string { - if (raw === undefined || raw === "") return fallback; +function usableAddr(raw: string | undefined): string | undefined { + if (raw === undefined) return undefined; try { return extractAddrSpec(raw); } catch { - return fallback; + return undefined; + } +} + +/** + * The hub routes on an out-of-band agent address, never on `To`, so an entry + * this projection cannot resolve is dropped rather than treated as a fault. + */ +function usableAddrs(raw: string[]): string[] { + const resolved: string[] = []; + for (const entry of raw) { + const addr = usableAddr(entry); + if (addr !== undefined) resolved.push(addr); } + return resolved; } /** @@ -816,9 +840,12 @@ export async function buildInboundMessageFromMail( // would throw and fail the step. When the messageId is omitted here, // createInboundMessage synthesizes a valid one; such mail cannot thread. const validReferences = mail.headers.references?.filter(isMessageId) ?? []; + const from = usableAddr(mail.headers.from); + // No `correlationId` option: a sender's `Interchange-Correlation-ID` is + // deliberately not forwarded, because resolving one clears a parked gate. return createInboundMessage({ - from: safeAddr(mail.headers.from, "trigger@local"), - to: safeAddr(mail.headers.to[0], "agent@local"), + ...(from !== undefined ? { from } : {}), + to: usableAddrs(mail.headers.to), // A present-but-empty Subject header (e.g. "Subject:" with no text) // decodes to "" and createInboundMessage rejects an empty subject, // failing the step -- so treat it like an absent header. @@ -826,7 +853,8 @@ export async function buildInboundMessageFromMail( mail.headers.subject.trim().length > 0 ? { subject: mail.headers.subject } : {}), - ...(isMessageId(mail.headers.messageId) + ...(mail.headers.messageId !== undefined && + isMessageId(mail.headers.messageId) ? { messageId: mail.headers.messageId } : {}), ...(mail.headers.inReplyTo !== undefined && diff --git a/vendor/intx/workflow-host/src/adapters/substrate-mailbox-store.ts b/vendor/intx/workflow-host/src/adapters/substrate-mailbox-store.ts index 8e75a7df1f..e0aa39436e 100644 --- a/vendor/intx/workflow-host/src/adapters/substrate-mailbox-store.ts +++ b/vendor/intx/workflow-host/src/adapters/substrate-mailbox-store.ts @@ -81,15 +81,18 @@ const encoder = new TextEncoder(); /** * On-disk envelope shape. Mirrors `StoredEnvelope` but serializes `date` as an - * ISO string and the three nullable header fields as `string | null` (JSON has - * no `undefined`); the loader maps `null` back to `undefined`. + * ISO string and the nullable header fields as `string | null` (JSON has no + * `undefined`); the loader maps `null` back to `undefined`. + * + * Widening a field needs no `INDEX_VERSION` bump: the version is matched + * exactly, so a bump rejects every existing mailbox instead of migrating it. */ const StoredEnvelopeJson = type({ messageId: "string", - from: "string", + from: "string | null", to: "string[]", subject: "string", - date: "string", + date: "string | null", inReplyTo: "string | null", references: "string[]", interchangeType: "string | null", @@ -191,10 +194,10 @@ export type SubstrateMailboxStoreOpts = { function serializeEnvelope(envelope: StoredEnvelope) { return { messageId: envelope.messageId, - from: envelope.from, + from: envelope.from ?? null, to: envelope.to, subject: envelope.subject, - date: envelope.date.toISOString(), + date: envelope.date === undefined ? null : envelope.date.toISOString(), inReplyTo: envelope.inReplyTo ?? null, references: envelope.references, interchangeType: envelope.interchangeType ?? null, @@ -207,10 +210,10 @@ function deserializeEnvelope( ): StoredEnvelope { return { messageId: raw.messageId, - from: raw.from, + from: raw.from === null ? undefined : raw.from, to: raw.to, subject: raw.subject, - date: new Date(raw.date), + date: raw.date === null ? undefined : new Date(raw.date), inReplyTo: raw.inReplyTo === null ? undefined : raw.inReplyTo, references: raw.references, interchangeType: diff --git a/vendor/intx/workflow-host/src/child/supervisor-backed-transport.ts b/vendor/intx/workflow-host/src/child/supervisor-backed-transport.ts index 527ef1a0e5..9ff89c3417 100644 --- a/vendor/intx/workflow-host/src/child/supervisor-backed-transport.ts +++ b/vendor/intx/workflow-host/src/child/supervisor-backed-transport.ts @@ -63,6 +63,7 @@ import type { Thread, Unsubscribe, } from "@intx/types/runtime"; +import { MessageTransportError } from "@intx/types/runtime"; import { executeSearch, @@ -138,7 +139,8 @@ export function createSupervisorBackedTransport( inbound?: SupervisorBackedTransportInbound, ): MessageTransport { function unsupported(method: string): never { - throw new Error( + throw new MessageTransportError( + "CANNOT", `supervisor-backed transport: ${method} is not supported for unified-host step agent ${address}; the supervisor owns the mailbox and the agent owns only its own ${MAILBOX_INBOX_DIR}`, ); } @@ -148,7 +150,8 @@ export function createSupervisorBackedTransport( // surface is a wiring error, not a silently-empty result. function requireInbound(method: string): SupervisorBackedTransportInbound { if (inbound === undefined) { - throw new Error( + throw new MessageTransportError( + "SERVERBUG", `supervisor-backed transport: ${method} needs the inbound surface, but it is not wired for unified-host step agent ${address}; the sidecar must construct the transport with its mailbox reader, watch registry, and crypto`, ); } @@ -160,7 +163,8 @@ export function createSupervisorBackedTransport( // it, which would return the wrong mailbox's messages mislabeled. function requireInbox(mailbox: string): void { if (mailbox !== MAILBOX_INBOX_DIR) { - throw new Error( + throw new MessageTransportError( + "NONEXISTENT", `supervisor-backed transport: unified-host step agent ${address} owns only the "${MAILBOX_INBOX_DIR}" mailbox; "${mailbox}" is not available`, ); } diff --git a/vendor/intx/workflow-host/src/ipc/control-channel.ts b/vendor/intx/workflow-host/src/ipc/control-channel.ts index 093a7382c4..e37a12a15d 100644 --- a/vendor/intx/workflow-host/src/ipc/control-channel.ts +++ b/vendor/intx/workflow-host/src/ipc/control-channel.ts @@ -161,8 +161,7 @@ export type OutboundMessagePayload = typeof OutboundMessagePayload.infer; * Wire shape of the parsed `MessageHeaders` the supervisor rides inline on a * `mailbox.notify` frame. Mirrors `@intx/types/runtime`'s `MessageHeaders` * field-for-field so a child watcher gets the arrived message's envelope for - * its `exists` `MailboxEvent` without a substrate round-trip. The four - * unconditionally-dereferenced fields (`from`, `to`, `date`, `messageId`) are + * its `exists` `MailboxEvent` without a substrate round-trip. Only `to` is * required; the rest are optional, matching the runtime type. * * Duplicated here as an arktype validator (rather than importing the TypeScript @@ -170,11 +169,11 @@ export type OutboundMessagePayload = typeof OutboundMessagePayload.infer; * wire boundary, exactly as `OutboundMessagePayload` does for outbound mail. */ export const MailboxNotifyHeaders = type({ - from: "string", + "from?": "string", to: "string[]", "cc?": "string[]", - date: "string", - messageId: "string", + "date?": "string", + "messageId?": "string", "inReplyTo?": "string", "references?": "string[]", "subject?": "string", diff --git a/vendor/intx/workflow-host/src/supervisor/supervisor.ts b/vendor/intx/workflow-host/src/supervisor/supervisor.ts index 58ab4035c3..b94d2eda52 100644 --- a/vendor/intx/workflow-host/src/supervisor/supervisor.ts +++ b/vendor/intx/workflow-host/src/supervisor/supervisor.ts @@ -1165,15 +1165,21 @@ export function createWorkflowSupervisor( function storedEnvelopeFromHeaders( headers: MessageHeaders, - receivedAt: number, + messageId: string, ): StoredEnvelope { - // The Date header is unvalidated external input; fall back to the arrival - // time when it is absent or unparseable so the store's `toISOString` - // serialization cannot throw on an Invalid Date. - const parsed = new Date(headers.date); - const date = Number.isNaN(parsed.getTime()) ? new Date(receivedAt) : parsed; + // No envelope date for a message that named none: the arrival time is + // this process's observation, not a date the sender ever claimed. + const parsed = + headers.date === undefined ? undefined : new Date(headers.date); + const date = + parsed === undefined || Number.isNaN(parsed.getTime()) + ? undefined + : parsed; return { - messageId: headers.messageId, + // The header's own id, so this envelope's id and the + // `inReplyTo`/`references` below come from one parser -- a second + // reader here would split a parent from its reply. + messageId: headers.messageId ?? messageId, from: headers.from, to: headers.to, subject: headers.subject ?? "", @@ -1198,7 +1204,6 @@ export function createWorkflowSupervisor( async function commitInboundToMailbox( messageId: string, rawMessage: Uint8Array, - receivedAt: number, ): Promise { try { await runMailboxExclusive(async () => { @@ -1218,7 +1223,7 @@ export function createWorkflowSupervisor( const store = await getMailboxStore(); const uid = store.append( rawMessage, - storedEnvelopeFromHeaders(decoded.headers, receivedAt), + storedEnvelopeFromHeaders(decoded.headers, messageId), [], ); mailboxUidByMessageId.set(messageId, uid); @@ -1375,7 +1380,7 @@ export function createWorkflowSupervisor( // child BEFORE waking dispatch, so the warm agent's mail_wait can observe // it committed. Non-fatal by contract: the enqueue above already secured // the durable delivery, so this never withholds the ack. - await commitInboundToMailbox(messageId, rawMessage, receivedAt); + await commitInboundToMailbox(messageId, rawMessage); wakeDispatch(); } else { // A redelivery of a message already durably present: the ack still diff --git a/vendor/intx/workflow/README.md b/vendor/intx/workflow/README.md index 8ef4e684b4..7e58932982 100644 --- a/vendor/intx/workflow/README.md +++ b/vendor/intx/workflow/README.md @@ -13,7 +13,9 @@ Multi-entry exports: - `@intx/workflow/definition` — `WorkflowDefinition`, `defineWorkflow`, `hashDefinition`, the `stepId` shape rule. The on-disk form a - workflow lives in. + workflow lives in. It also carries the canonical step walk + (`walkStepTree`, `walkWorkflowSteps`, `executableStepIds`); see + "Walking a definition's steps" below. - `@intx/workflow/state-machine` — the event union, the transition function, the `RunState` projection. Pure functions over the workflow-run log. @@ -29,8 +31,9 @@ Multi-entry exports: For a production host (workflow-run repo backing, scheduler that honors wall-clock fire times, signal channel that observes commits, DI seams for mail bus / signing key / subprocess spawner), see -`@intx/workflow-host`. For deploy-time validation, capability walk, -and the agent-deploy-trivial-workflow dichotomy, see +`@intx/workflow-host`. For the deploy-time capability walk, the +operator-approval gate that consumes it, and the address derivation +and per-step inference-source pinning a deploy needs, see `@intx/workflow-deploy`. ## Tools are available wherever inference runs @@ -109,6 +112,45 @@ tree types, so the inert wire projection — whose step values are `unknown` and are validated as the caller descends — rides the same traversal. +## A selector names something that must exist + +A selector that names a path or a field fails the run rather than +resolving to a hole. A `from` path throws on a missing key, an +out-of-range index, and an in-range index a sparse array leaves +unfilled; the one hole it admits is an own key whose value is +`undefined`. The `step`, `action`, and `loop` runners canonicalize +an input that is wholly `undefined` to `null`, and that check does +not descend, so a hole nested inside a resolved object survives it. +The `childWorkflow` runner does not canonicalize at all: it tests +whether the input selector is present, not what the selector +resolved to, so a hole there spawns the child with an `undefined` +input while the parent's `StepStarted` value loses the key in +serialization. An agent step's two sinks agree because both see the +JSON form, where the key is absent; a handler that receives the input +object by reference sees the key present. A `project` throws when the +source object does not carry a listed field as an own property, and +when the own value of a listed field is `undefined`. Neither shape +reads an inherited member, so a path segment or a field named `toString`, +`constructor`, or `hasOwnProperty` throws even though the member is +reachable on every JSON-derived object. An own key of that name is a +normal key and resolves normally; `merge` and `project` both build +their result by defining keys, so an own `__proto__` key lands as data +and never changes the result's prototype. + +The vocabulary therefore has no optional field. `fields: ["x"]` is a +claim that the source carries a value for `x`, and the run fails at the +selector that made the claim rather than at some later step that +received `undefined`. A field that is genuinely sometimes absent is +modeled by the producing step, which emits the key with an explicit +`null`. + +`merge` inherits this. Later operands override earlier ones for +overlapping keys, so an operand carrying a hole erases a real value an +earlier operand supplied. A `project` operand cannot carry one, because +it throws on an absent field and on a field whose own value is +`undefined`. A `from` operand carries whatever the producing step +emitted, so it is the shape that can. + ## Consuming a real agent step's structured output Structural selectors (`map.over`, `input.from`, `project`, `merge`) do diff --git a/vendor/intx/workflow/VENDORED-FROM b/vendor/intx/workflow/VENDORED-FROM index 7ecaf186ff..9971fbad04 100644 --- a/vendor/intx/workflow/VENDORED-FROM +++ b/vendor/intx/workflow/VENDORED-FROM @@ -1,4 +1,4 @@ Source: https://github.com/faremeter/interchange (packages/workflow) -Commit: 5453d0b01da9d749aa446c4ba34fcf2aeb4549c0 (origin/main, v0.4.0) +Commit: 9febf699e1c8a01fd329927e5a14be7f94e9919e (origin/main, v0.4.0) License: LGPL-2.1-only (see vendor/intx/LICENSE) -Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages); only the tests covering a workbench delta are kept. One source delta (corbitsdev/workbench#709): run.ts's per-step timeout is an execution budget — trigger-idle input parks disarm the step timer across the wait and re-arm a fresh budget on resume, while the approval park stays armed (snapshot-bound, still bounded by `step.timeout`); pinned in runtime/run.test.ts (only tests covering a workbench delta are vendored). +Local modifications: exports map repointed from the upstream intx-src condition to direct TypeScript source resolution (types/default -> ./src/...); dist references removed. Upstream test files are not vendored (they import the unvendored @intx/inference-testing and @intx/test-harness test packages). One source delta (corbitsdev/workbench#709): run.ts's per-step timeout is an execution budget - trigger-idle input parks disarm the step timer across the wait and re-arm a fresh budget on resume, while the approval park stays armed (snapshot-bound, still bounded by `step.timeout`). Absorbed upstream at this re-vendor: the selectors.ts project-selector own-key/undefined guards. diff --git a/vendor/intx/workflow/src/runtime/selectors.ts b/vendor/intx/workflow/src/runtime/selectors.ts index 8373337869..b2da59e1b4 100644 --- a/vendor/intx/workflow/src/runtime/selectors.ts +++ b/vendor/intx/workflow/src/runtime/selectors.ts @@ -43,11 +43,31 @@ export function evaluate(selector: Selector, ctx: SelectorContext): unknown { selector, ); } - const projected: Record = {}; + const entries: [string, unknown][] = []; for (const field of selector.fields) { - projected[field] = source[field]; + // Own keys only: `source[field]` reads through the prototype chain, so + // a field named `toString` resolves on any JSON-derived object. + if (!Object.hasOwn(source, field)) { + throw new SelectorError( + `missing field ${field} in project selector source`, + selector, + ); + } + const value = source[field]; + // `Object.hasOwn` accepts a key whose value is `undefined`, and `fields` + // claims a value rather than a key. A hole admitted here would win the + // key in a `merge`, erasing the value an earlier operand supplied. + if (value === undefined) { + throw new SelectorError( + `field ${field} is undefined in project selector source`, + selector, + ); + } + entries.push([field, value]); } - return projected; + // `Object.fromEntries` defines; assigning onto an accumulator object would + // hit the inherited `__proto__` setter for a field of that name. + return Object.fromEntries(entries); } if (isMergeSelector(selector)) { const merged: Record = {}; @@ -59,7 +79,18 @@ export function evaluate(selector: Selector, ctx: SelectorContext): unknown { selector, ); } - Object.assign(merged, value); + for (const [key, entry] of Object.entries(value)) { + // Define, never `[[Set]]`: assignment sends an operand's own + // `__proto__` key to the accumulator's prototype setter. Spread is + // equally safe and was rejected -- it reads as a cosmetic rewrite of + // the `Object.assign` this replaces, and invites reverting. + Object.defineProperty(merged, key, { + value: entry, + writable: true, + enumerable: true, + configurable: true, + }); + } } return merged; } @@ -87,18 +118,23 @@ function resolvePath( selector, ); } - // An out-of-range index silently returning `undefined` is the - // same failure mode the key branch guards against with `in`: - // the runtime would feed `undefined` to a step or to a - // subsequent path segment as though the array author had - // supplied a hole. Surface the path so the author can see - // which index missed. + // An out-of-range index silently returning `undefined` would feed a + // step as though the author had supplied a hole. if (segment.index < 0 || segment.index >= cursor.length) { throw new SelectorError( `index [${String(segment.index)}] out of range (length ${String(cursor.length)}) in path ${path}`, selector, ); } + // Own elements only, matching the key branch below: an index a sparse + // array leaves unfilled is in range and is not an own property, so it + // yields the same hole the range guard above refuses. + if (!Object.hasOwn(cursor, segment.index)) { + throw new SelectorError( + `missing index [${String(segment.index)}] in path ${path}`, + selector, + ); + } cursor = cursor[segment.index]; } else { if (cursor === null || cursor === undefined) { @@ -113,13 +149,9 @@ function resolvePath( selector, ); } - // Distinguish a missing key from a key whose value is `null` or - // `undefined`. `in` checks the key's presence on the object; - // bracket-indexing alone would silently return `undefined` for - // a typo and let the runtime feed `undefined` into a step as - // though no input were supplied. Surface the missing key with a - // path so the author can spot the typo. - if (!(segment.key in cursor)) { + // Own keys only: `in` walks the prototype chain, so a path ending in + // `toString` or `constructor` resolves on any JSON-derived object. + if (!Object.hasOwn(cursor, segment.key)) { throw new SelectorError( `missing key ${segment.key} in path ${path}`, selector,