From 4b413498473e4ca06f5c648c53c3c7333145fc05 Mon Sep 17 00:00:00 2001 From: Sawyer Cutler Date: Sun, 20 Sep 2026 23:17:02 -0700 Subject: [PATCH] docs: PRODUCT/ARCHITECTURE/IMPLEMENTATION for @corbits/memory (CL-8775) --- ARCHITECTURE.md | 44 +++++++++++++++++++++--- IMPLEMENTATION.md | 86 +++++++++++++++++++++++++++++++++++++---------- PRODUCT.md | 44 ++++++++++++++++-------- 3 files changed, 138 insertions(+), 36 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0eba61f..cfbe8ad 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -29,6 +29,10 @@ add → ingest elements (store/chunk/embed) → process (optional, host) tools / host ingest workflow → /api/tenants/:tenantId/memory/* → Memory plane → DocumentStore ↑ Interchange auth + principal + grants + +sidecar-bundle (deployed agent) → /api/workflow-memory/* → Memory plane → DocumentStore + ↑ + hub credential + x-workflow-run-address (mountWorkflowMemory, no session) ``` Mount is intentionally small. The host already has `app`, grants, and @@ -117,10 +121,42 @@ Returns an in-process `Memory` (`add`, `search`, `list`, `close`, plus the optional retention writes) for host workers and ingestion modules that already resolved identity. -**Agent tools live in this package** as thin HTTP clients -(`@corbits/memory/tools` / `interchange.tools`): `defineTool` factories that -`fetch` the mounted routes with install credentials. They do not import the -in-process plane. OpenAPI→MCP remains an optional host bridge. +**Capture** is the write path inside `add` (raw capture → chunks / edges / +embed on the default store). **Search** is hybrid retrieval on the same +plane, whether the caller arrived via tenant routes or the sidecar mount. + +## Sidecar mount (`mountWorkflowMemory`) + +Deployed agents do not install this package as a git sidecar. They carry +the factory at `@corbits/memory/sidecar-bundle`, which holds no client +code, no base URL, and no token: it resolves the host `hub` credential and +calls the run-scoped routes under `/api/workflow-memory/*`. That mount is +**parallel** to the tenant routes — two auth conventions stay on two +mounts so neither is harder to reason about. + +```ts +import { mountWorkflowMemory } from "@corbits/memory"; + +mountWorkflowMemory(workflowMemoryApp, { + memory, + agentToken: { verify, resolveRun }, +}); +app.route("/api/workflow-memory", workflowMemoryApp); +``` + +Authorization on this mount **is the token itself**: the hub only mints an +agent token for a definition it already authorized, and every call is +confined to the verified run's tenant and principal. The mount runs **no +grant check of its own** and has no tenant override. A bearer minted for +one workbench cannot act on another's run (`verify` tenant must match +`resolveRun` tenant). Unrecognized bearer, unknown run address, and +cross-tenant mismatch all return the same **401**. + +The sidecar factory (`src/sidecar-bundle.ts`) maps `memory_add` / +`memory_search` / `memory_list` / `memory_feed` onto those run-scoped +routes. Relative paths only — a mediated HTTP handle resolves them against +the origin it is pinned to. Capture and search still execute on the same +in-process `Memory` plane as the tenant routes. ## Provenance diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md index e55bdd5..544be8f 100644 --- a/IMPLEMENTATION.md +++ b/IMPLEMENTATION.md @@ -8,23 +8,25 @@ and wire shapes. For the "why standalone" / boundaries story, read ``` src/ - index.ts # createMemory / registerMemoryRoutes + distiller re-exports + index.ts # createMemory / registerMemoryRoutes / mountWorkflowMemory + distiller re-exports mount-config.ts # MemoryConfig + loadMemoryConfig() — the mount config config.ts # EngineConfig — the core vector-plane config (db + embed + rerank) memory.ts # createMemory — add/search/list against store or pgvector grant-tags.ts # resolveAccessTags + canAccessDocument (host grants) + workflow-mount.ts # mountWorkflowMemory — run-scoped /api/workflow-memory/* + sidecar-bundle.ts # @corbits/memory/sidecar-bundle factory (no client, no token) + tools.ts # MEMORY_TOOL_DEFINITIONS — sidecar binds names to run-scoped routes + http-client.ts # host-side HTTP client for tenant routes (imperative distill tick) log.ts # getLogger(["memory"]) from @intx/log migrations.ts # runMemoryMigrations(url) ports/ # DocumentStore / SourceProvider + fakes - routes/ # the mounted routes + routes/ # the mounted tenant routes mount.ts # registerMemoryRoutes (HTTP) deps.ts # RouteDeps, caller(c) (context identity), grantGuard add.ts, search.ts, list.ts, feed.ts - tools/ # Interchange defineTool factories (HTTP clients) - add.ts, search.ts, list.ts, feed.ts, client.ts, install.ts distiller/ # Resident distiller (CL-5869) — workflow + tick helpers index.ts # createResidentDistiller, runDistillTick, buildDistilledClaim workflow.ts # defineWorkflow + defineAgent with memory tools @@ -640,20 +642,68 @@ surface, or a migrating host silently loses them. `registerMemoryRoutes` and `createMemory({ app })` register these seven HTTP routes (add, search, list, feed, forget, purge, retention-class). -Agent tools ship in this package as Interchange `defineTool` factories -(`@corbits/memory/tools` / `interchange.tools`): thin HTTP clients that call the -mounted routes with install env (`memoryBaseUrl`, `memoryTenantId`, -`memoryAuthToken`). They do not import the plane. Host checklist: agent principal -needs `memory:add` and/or `memory:search` grants; Bearer token only (no session -cookie path); tool results are JSON strings; pass `AbortSignal` if you need hang -protection — the client has no default timeout. OpenAPI→MCP remains an optional -host bridge. The plane surface is `add` / `search` / `list` / `close`, plus -optional transform methods when backed by the engine DocumentStore -(`createTransformConfig`, `listTransformConfigs`, `runTransform`, -`promoteGeneration`, `demoteGeneration`) and optional retention methods -(`tombstoneDocument`, `hardDeleteDocument`, `setRetentionClass`, -`sweepEphemeral`, `deprecateVersion`) — see docs/RETENTION.md. Inference stays -on the host. +**Capture** (`services/capture.ts`) is the write path inside `add`. +**Search** (`services/search.ts`) is hybrid retrieval. Deployed agents do +not call these tenant routes with a git-installed client; they use the +sidecar mount below. The plane surface is `add` / `search` / `list` / +`close`, plus optional transform methods when backed by the engine +DocumentStore (`createTransformConfig`, `listTransformConfigs`, +`runTransform`, `promoteGeneration`, `demoteGeneration`) and optional +retention methods (`tombstoneDocument`, `hardDeleteDocument`, +`setRetentionClass`, `sweepEphemeral`, `deprecateVersion`) — see +docs/RETENTION.md. Inference stays on the host. Host workers that already +have HTTP to the tenant tree can use `createMemoryHttpClient` +(`src/http-client.ts`) — that is a host client, not the agent sidecar. + +### Run-scoped sidecar (`mountWorkflowMemory`) + +`src/workflow-mount.ts`, re-exported from the barrel. Parallel to the +tenant tree — do not fold agent-bearer auth into `registerMemoryRoutes`. +`package.json` exports `@corbits/memory/sidecar-bundle` → +`src/sidecar-bundle.ts`. + +```ts +import { mountWorkflowMemory } from "@corbits/memory"; + +mountWorkflowMemory(workflowMemoryApp, { + memory, + agentToken: { verify, resolveRun }, +}); +app.route("/api/workflow-memory", workflowMemoryApp); +``` + +| Constant | Value | +| --- | --- | +| `WORKFLOW_MEMORY_BASE_PATH` | `/api/workflow-memory` | +| `HUB_CREDENTIAL_HANDLE` | `hub` | +| `SIDECAR_BUNDLE_ID` | `@corbits/memory/sidecar-bundle` | + +**Auth.** Every route sits behind middleware: `agentToken.verify(c)` reads +the presented `Authorization`; `agentToken.resolveRun` looks up +`x-workflow-run-address`. Same **401** body whether the bearer is +unrecognized, the address names no run, or the run's tenant is not the +token's tenant. No `requireGrant`. No tenant override. Scope on context: +`workflowRunScope` `{ tenantId, principalId, runId }`. + +**Wire (relative to the mount; sidecar never names a host).** Responses +are `{ data: … }` on success. Sidecar `requires`: `capabilities`, +`address` (run address). Tool results are JSON strings. + +| Tool name | Method + path | Notes | +| --- | --- | --- | +| `memory_add` | `POST /add` | Body same as tenant add. Forces `share.tenant = true` (team share; explicit share only widens). Capture path is `memory.add` → `captureDocument`. | +| `memory_search` | `POST /search` | Body same as tenant search; `visibleTags` = `[tenantTag(tenantId)]`. | +| `memory_list` | `GET /list?limit=` | Same grant-tag visibility as search (`visibleTags` team tag). | +| `memory_feed` | `GET /feed?after=&limit=&exclude_generator=` | **501** if `memory.feed` is undefined on this plane. | + +Unknown tool name → tool error (`isError: true`). Non-HTTP hub credential +kind → error. `callMemoryRoute` sends `x-workflow-run-address` and +optional JSON body; mediated `credential.fetch` injects the bearer and +pins origin. + +`callerResolver` on the **tenant** routes is a different seam (machine +caller through the same grant path). Do not treat it as a replacement for +`mountWorkflowMemory`. ### Share materialization (CL-5873) diff --git a/PRODUCT.md b/PRODUCT.md index 105f3cc..3ce929d 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -3,8 +3,11 @@ Memory for Interchange hubs: durable documents, hybrid search, recent list. **You mount it on the hub (~5 lines). That exposes protected routes. Agents -and ingestion modules call those routes.** Workbench and coding agents are -clients — not owners of auth. Inference stays host-injected. +and ingestion modules call those routes.** Capture (`add`) writes; search +retrieves. Workbench and coding agents are clients — not owners of auth. +Inference stays host-injected. Deployed agents do **not** git-install this +package: they carry `@corbits/memory/sidecar-bundle` and hit the parallel +`mountWorkflowMemory` routes. ## Default pipeline (locked) @@ -34,10 +37,11 @@ never creates one; it mounts onto yours. | Surface | Role | | --- | --- | | `createMemory({ app, … })` | Register `/api/tenants/:tenantId/memory/*` + return the plane | +| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents | | `loadMemoryConfig()` | Config from env | | `runMemoryMigrations(url)` | Apply pgvector schema | | `registerMemoryRoutes` | Low-level HTTP only (optional) | -| `@corbits/memory/tools` | Interchange tools (`memory_add` / `search` / `list` / `feed`) | +| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token | | `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` | ### Verbs @@ -55,9 +59,11 @@ share-grant materialization. Process helpers: `createResidentDistiller` / `runDistillTick` (`docs/DISTILLER.md`) — host injects inference; not the default ingest path. -Identity is always **`principalId` + `tenantId`** on the plane. HTTP routes -never take body identity — they read `c.get("principal")` from Interchange -context. +Identity is always **`principalId` + `tenantId`** on the plane. Tenant HTTP +routes never take body identity — they read `c.get("principal")` from +Interchange context. Run-scoped sidecar routes read the verified workflow +run (`agentToken.verify` + `x-workflow-run-address`); the sidecar factory +never names a host or carries a token. ### How it is used @@ -66,6 +72,10 @@ Agent / host ingest workflow │ tool call or host worker │ → POST|GET /api/tenants/:tenantId/memory/* │ authenticated by Interchange (session | API key | MCP OAuth) + │ +Deployed agent (sidecar-bundle) + │ hub credential + run address + │ → POST|GET /api/workflow-memory/* (mountWorkflowMemory) ▼ ┌──────────────────────────────────────────────┐ │ Host Interchange createApp │ @@ -73,22 +83,26 @@ Agent / host ingest workflow │ + createMemory({ app, grantStore, … }) │ │ grants: memory:add | memory:search │ │ documentStore: pgvector | host | fake │ +│ + mountWorkflowMemory(app, { memory, … }) │ │ │ in-process │ │ ▼ │ -│ Memory plane: add / search / list │ +│ Memory plane: add (capture) / search / list │ │ → DocumentStore (sole durable backend) │ └──────────────────────────────────────────────┘ ``` 1. **Mount** — host passes `app` + the same grant store it already uses. -2. **Tools** — install `@corbits/memory/tools` (`defineTool` factories) on a - workflow with env credentials (`memoryBaseUrl`, `memoryTenantId`, - `memoryAuthToken`). Tools HTTP-call the mounted routes; identity is the - hub-authenticated principal. OpenAPI→MCP remains an optional host bridge. +2. **Sidecar (deployed agents)** — agents carry + `@corbits/memory/sidecar-bundle`. It holds no client code, no base URL, + and no token: it resolves the host `hub` credential and calls + `/api/workflow-memory/*`. Host wires `mountWorkflowMemory` in parallel + with the tenant routes (see README How it works). Not the primary + install — that is still `createMemory` + `loadMemoryConfig`. 3. **Ingestion** — preferred: one host workflow (or module) does - **add → ingest elements → process**. Mechanical ingest is inside `add` on - the default store; process (claims / links) is host-injected inference in - the same pipeline when you want a company brain. + **add → ingest elements (capture) → process**. Mechanical capture is + inside `add` on the default store; process (claims / links) is + host-injected inference in the same pipeline when you want a company + brain. ### Ports @@ -106,6 +120,8 @@ stores, Linear tools. Core never imports vendor SDKs. - No answer/generation endpoint — host owns inference. - Workbench is a client, not required. - Core does not run the ingest workflow process — the host does. +- Deployed agents do not git-install this package as a sidecar and do not + receive a memory base URL or token — that is the sidecar-bundle contract. **Default durable store:** Postgres via `DATABASE_URL`, tables under the **`memory`** schema. When