diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0eba61f..cfbe8ad 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -29,6 +29,10 @@ add → ingest elements (store/chunk/embed) → process (optional, host) tools / host ingest workflow → /api/tenants/:tenantId/memory/* → Memory plane → DocumentStore ↑ Interchange auth + principal + grants + +sidecar-bundle (deployed agent) → /api/workflow-memory/* → Memory plane → DocumentStore + ↑ + hub credential + x-workflow-run-address (mountWorkflowMemory, no session) ``` Mount is intentionally small. The host already has `app`, grants, and @@ -117,10 +121,42 @@ Returns an in-process `Memory` (`add`, `search`, `list`, `close`, plus the optional retention writes) for host workers and ingestion modules that already resolved identity. -**Agent tools live in this package** as thin HTTP clients -(`@corbits/memory/tools` / `interchange.tools`): `defineTool` factories that -`fetch` the mounted routes with install credentials. They do not import the -in-process plane. OpenAPI→MCP remains an optional host bridge. +**Capture** is the write path inside `add` (raw capture → chunks / edges / +embed on the default store). **Search** is hybrid retrieval on the same +plane, whether the caller arrived via tenant routes or the sidecar mount. + +## Sidecar mount (`mountWorkflowMemory`) + +Deployed agents do not install this package as a git sidecar. They carry +the factory at `@corbits/memory/sidecar-bundle`, which holds no client +code, no base URL, and no token: it resolves the host `hub` credential and +calls the run-scoped routes under `/api/workflow-memory/*`. That mount is +**parallel** to the tenant routes — two auth conventions stay on two +mounts so neither is harder to reason about. + +```ts +import { mountWorkflowMemory } from "@corbits/memory"; + +mountWorkflowMemory(workflowMemoryApp, { + memory, + agentToken: { verify, resolveRun }, +}); +app.route("/api/workflow-memory", workflowMemoryApp); +``` + +Authorization on this mount **is the token itself**: the hub only mints an +agent token for a definition it already authorized, and every call is +confined to the verified run's tenant and principal. The mount runs **no +grant check of its own** and has no tenant override. A bearer minted for +one workbench cannot act on another's run (`verify` tenant must match +`resolveRun` tenant). Unrecognized bearer, unknown run address, and +cross-tenant mismatch all return the same **401**. + +The sidecar factory (`src/sidecar-bundle.ts`) maps `memory_add` / +`memory_search` / `memory_list` / `memory_feed` onto those run-scoped +routes. Relative paths only — a mediated HTTP handle resolves them against +the origin it is pinned to. Capture and search still execute on the same +in-process `Memory` plane as the tenant routes. ## Provenance diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md index e55bdd5..544be8f 100644 --- a/IMPLEMENTATION.md +++ b/IMPLEMENTATION.md @@ -8,23 +8,25 @@ and wire shapes. For the "why standalone" / boundaries story, read ``` src/ - index.ts # createMemory / registerMemoryRoutes + distiller re-exports + index.ts # createMemory / registerMemoryRoutes / mountWorkflowMemory + distiller re-exports mount-config.ts # MemoryConfig + loadMemoryConfig() — the mount config config.ts # EngineConfig — the core vector-plane config (db + embed + rerank) memory.ts # createMemory — add/search/list against store or pgvector grant-tags.ts # resolveAccessTags + canAccessDocument (host grants) + workflow-mount.ts # mountWorkflowMemory — run-scoped /api/workflow-memory/* + sidecar-bundle.ts # @corbits/memory/sidecar-bundle factory (no client, no token) + tools.ts # MEMORY_TOOL_DEFINITIONS — sidecar binds names to run-scoped routes + http-client.ts # host-side HTTP client for tenant routes (imperative distill tick) log.ts # getLogger(["memory"]) from @intx/log migrations.ts # runMemoryMigrations(url) ports/ # DocumentStore / SourceProvider + fakes - routes/ # the mounted routes + routes/ # the mounted tenant routes mount.ts # registerMemoryRoutes (HTTP) deps.ts # RouteDeps, caller(c) (context identity), grantGuard add.ts, search.ts, list.ts, feed.ts - tools/ # Interchange defineTool factories (HTTP clients) - add.ts, search.ts, list.ts, feed.ts, client.ts, install.ts distiller/ # Resident distiller (CL-5869) — workflow + tick helpers index.ts # createResidentDistiller, runDistillTick, buildDistilledClaim workflow.ts # defineWorkflow + defineAgent with memory tools @@ -640,20 +642,68 @@ surface, or a migrating host silently loses them. `registerMemoryRoutes` and `createMemory({ app })` register these seven HTTP routes (add, search, list, feed, forget, purge, retention-class). -Agent tools ship in this package as Interchange `defineTool` factories -(`@corbits/memory/tools` / `interchange.tools`): thin HTTP clients that call the -mounted routes with install env (`memoryBaseUrl`, `memoryTenantId`, -`memoryAuthToken`). They do not import the plane. Host checklist: agent principal -needs `memory:add` and/or `memory:search` grants; Bearer token only (no session -cookie path); tool results are JSON strings; pass `AbortSignal` if you need hang -protection — the client has no default timeout. OpenAPI→MCP remains an optional -host bridge. The plane surface is `add` / `search` / `list` / `close`, plus -optional transform methods when backed by the engine DocumentStore -(`createTransformConfig`, `listTransformConfigs`, `runTransform`, -`promoteGeneration`, `demoteGeneration`) and optional retention methods -(`tombstoneDocument`, `hardDeleteDocument`, `setRetentionClass`, -`sweepEphemeral`, `deprecateVersion`) — see docs/RETENTION.md. Inference stays -on the host. +**Capture** (`services/capture.ts`) is the write path inside `add`. +**Search** (`services/search.ts`) is hybrid retrieval. Deployed agents do +not call these tenant routes with a git-installed client; they use the +sidecar mount below. The plane surface is `add` / `search` / `list` / +`close`, plus optional transform methods when backed by the engine +DocumentStore (`createTransformConfig`, `listTransformConfigs`, +`runTransform`, `promoteGeneration`, `demoteGeneration`) and optional +retention methods (`tombstoneDocument`, `hardDeleteDocument`, +`setRetentionClass`, `sweepEphemeral`, `deprecateVersion`) — see +docs/RETENTION.md. Inference stays on the host. Host workers that already +have HTTP to the tenant tree can use `createMemoryHttpClient` +(`src/http-client.ts`) — that is a host client, not the agent sidecar. + +### Run-scoped sidecar (`mountWorkflowMemory`) + +`src/workflow-mount.ts`, re-exported from the barrel. Parallel to the +tenant tree — do not fold agent-bearer auth into `registerMemoryRoutes`. +`package.json` exports `@corbits/memory/sidecar-bundle` → +`src/sidecar-bundle.ts`. + +```ts +import { mountWorkflowMemory } from "@corbits/memory"; + +mountWorkflowMemory(workflowMemoryApp, { + memory, + agentToken: { verify, resolveRun }, +}); +app.route("/api/workflow-memory", workflowMemoryApp); +``` + +| Constant | Value | +| --- | --- | +| `WORKFLOW_MEMORY_BASE_PATH` | `/api/workflow-memory` | +| `HUB_CREDENTIAL_HANDLE` | `hub` | +| `SIDECAR_BUNDLE_ID` | `@corbits/memory/sidecar-bundle` | + +**Auth.** Every route sits behind middleware: `agentToken.verify(c)` reads +the presented `Authorization`; `agentToken.resolveRun` looks up +`x-workflow-run-address`. Same **401** body whether the bearer is +unrecognized, the address names no run, or the run's tenant is not the +token's tenant. No `requireGrant`. No tenant override. Scope on context: +`workflowRunScope` `{ tenantId, principalId, runId }`. + +**Wire (relative to the mount; sidecar never names a host).** Responses +are `{ data: … }` on success. Sidecar `requires`: `capabilities`, +`address` (run address). Tool results are JSON strings. + +| Tool name | Method + path | Notes | +| --- | --- | --- | +| `memory_add` | `POST /add` | Body same as tenant add. Forces `share.tenant = true` (team share; explicit share only widens). Capture path is `memory.add` → `captureDocument`. | +| `memory_search` | `POST /search` | Body same as tenant search; `visibleTags` = `[tenantTag(tenantId)]`. | +| `memory_list` | `GET /list?limit=` | Same grant-tag visibility as search (`visibleTags` team tag). | +| `memory_feed` | `GET /feed?after=&limit=&exclude_generator=` | **501** if `memory.feed` is undefined on this plane. | + +Unknown tool name → tool error (`isError: true`). Non-HTTP hub credential +kind → error. `callMemoryRoute` sends `x-workflow-run-address` and +optional JSON body; mediated `credential.fetch` injects the bearer and +pins origin. + +`callerResolver` on the **tenant** routes is a different seam (machine +caller through the same grant path). Do not treat it as a replacement for +`mountWorkflowMemory`. ### Share materialization (CL-5873) diff --git a/PRODUCT.md b/PRODUCT.md index 105f3cc..3ce929d 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -3,8 +3,11 @@ Memory for Interchange hubs: durable documents, hybrid search, recent list. **You mount it on the hub (~5 lines). That exposes protected routes. Agents -and ingestion modules call those routes.** Workbench and coding agents are -clients — not owners of auth. Inference stays host-injected. +and ingestion modules call those routes.** Capture (`add`) writes; search +retrieves. Workbench and coding agents are clients — not owners of auth. +Inference stays host-injected. Deployed agents do **not** git-install this +package: they carry `@corbits/memory/sidecar-bundle` and hit the parallel +`mountWorkflowMemory` routes. ## Default pipeline (locked) @@ -34,10 +37,11 @@ never creates one; it mounts onto yours. | Surface | Role | | --- | --- | | `createMemory({ app, … })` | Register `/api/tenants/:tenantId/memory/*` + return the plane | +| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents | | `loadMemoryConfig()` | Config from env | | `runMemoryMigrations(url)` | Apply pgvector schema | | `registerMemoryRoutes` | Low-level HTTP only (optional) | -| `@corbits/memory/tools` | Interchange tools (`memory_add` / `search` / `list` / `feed`) | +| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token | | `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` | ### Verbs @@ -55,9 +59,11 @@ share-grant materialization. Process helpers: `createResidentDistiller` / `runDistillTick` (`docs/DISTILLER.md`) — host injects inference; not the default ingest path. -Identity is always **`principalId` + `tenantId`** on the plane. HTTP routes -never take body identity — they read `c.get("principal")` from Interchange -context. +Identity is always **`principalId` + `tenantId`** on the plane. Tenant HTTP +routes never take body identity — they read `c.get("principal")` from +Interchange context. Run-scoped sidecar routes read the verified workflow +run (`agentToken.verify` + `x-workflow-run-address`); the sidecar factory +never names a host or carries a token. ### How it is used @@ -66,6 +72,10 @@ Agent / host ingest workflow │ tool call or host worker │ → POST|GET /api/tenants/:tenantId/memory/* │ authenticated by Interchange (session | API key | MCP OAuth) + │ +Deployed agent (sidecar-bundle) + │ hub credential + run address + │ → POST|GET /api/workflow-memory/* (mountWorkflowMemory) ▼ ┌──────────────────────────────────────────────┐ │ Host Interchange createApp │ @@ -73,22 +83,26 @@ Agent / host ingest workflow │ + createMemory({ app, grantStore, … }) │ │ grants: memory:add | memory:search │ │ documentStore: pgvector | host | fake │ +│ + mountWorkflowMemory(app, { memory, … }) │ │ │ in-process │ │ ▼ │ -│ Memory plane: add / search / list │ +│ Memory plane: add (capture) / search / list │ │ → DocumentStore (sole durable backend) │ └──────────────────────────────────────────────┘ ``` 1. **Mount** — host passes `app` + the same grant store it already uses. -2. **Tools** — install `@corbits/memory/tools` (`defineTool` factories) on a - workflow with env credentials (`memoryBaseUrl`, `memoryTenantId`, - `memoryAuthToken`). Tools HTTP-call the mounted routes; identity is the - hub-authenticated principal. OpenAPI→MCP remains an optional host bridge. +2. **Sidecar (deployed agents)** — agents carry + `@corbits/memory/sidecar-bundle`. It holds no client code, no base URL, + and no token: it resolves the host `hub` credential and calls + `/api/workflow-memory/*`. Host wires `mountWorkflowMemory` in parallel + with the tenant routes (see README How it works). Not the primary + install — that is still `createMemory` + `loadMemoryConfig`. 3. **Ingestion** — preferred: one host workflow (or module) does - **add → ingest elements → process**. Mechanical ingest is inside `add` on - the default store; process (claims / links) is host-injected inference in - the same pipeline when you want a company brain. + **add → ingest elements (capture) → process**. Mechanical capture is + inside `add` on the default store; process (claims / links) is + host-injected inference in the same pipeline when you want a company + brain. ### Ports @@ -106,6 +120,8 @@ stores, Linear tools. Core never imports vendor SDKs. - No answer/generation endpoint — host owns inference. - Workbench is a client, not required. - Core does not run the ingest workflow process — the host does. +- Deployed agents do not git-install this package as a sidecar and do not + receive a memory base URL or token — that is the sidecar-bundle contract. **Default durable store:** Postgres via `DATABASE_URL`, tables under the **`memory`** schema. When diff --git a/README.md b/README.md index e779030..c7632b2 100644 --- a/README.md +++ b/README.md @@ -1,189 +1,109 @@ # @corbits/memory -Memory for [Interchange](https://github.com/corbitsdev) hubs: **add**, **search**, -**list**. +Memory for Interchange hubs: **add**, **search**, **list**. Mount it on the +hub; routes land under `/api/tenants/:tenantId/memory/*` so the hub's +existing tenant middleware supplies principal + tenant. Host workers call +the same plane in-process. Inference stays host-owned — this package does +not ship an answer endpoint. -Mount it on the hub. Routes land under `/api/tenants/:tenantId/memory/*`, so -the hub’s existing `createResolveTenant` middleware supplies principal + tenant -— same as workflows, assets, and agents. Deployed agents carry the sidecar -bundle and call the run-scoped routes under `/api/workflow-memory/*`; -ingestion modules call the tenant routes or the in-process plane. That’s the -product. +## Runtime support -## Requirements +Bun >= 1.2 runs the published TypeScript source (`package.json` `exports`); +there is no `dist` build, so native Node does not load it. `engines.node` is +`>=24` as a floor for Node-side tooling (typecheck, pack). -**Bun-only runtime.** This package ships TypeScript source (`src/*.ts`, see -`package.json` `exports`) and declares `"engines": { "bun": ">=1.2.0" }` — run -it under Bun 1.2+. There is intentionally no `dist` build step. `engines.node` -is `>=24` as a floor for Node-side tooling (typecheck, pack); native Node does -not load this package's extensionless TypeScript source. +Peer stack you already have on an Interchange hub: `@intx/authz`, +`@intx/hub-api`, `hono`. - -## Install - -Not published to npm yet: +## Quickstart ```bash -bun add git+https://github.com/corbitsdev/corbits-memory.git +npm add @corbits/memory +pnpm add @corbits/memory +yarn add @corbits/memory +bun add @corbits/memory ``` -Peer stack you already have on an Interchange hub: `@intx/authz`, `@intx/hub-api`, -`hono`. Agent tools also need `@intx/agent` (declared as a direct dependency). - -## Mount (≈5 lines) - -On a real hub you already have `app` (with session + -`app.use("/api/tenants/:tenantId/*", resolveTenant)`), `grantStore`, and -`conditionRegistry`: - ```ts import { createMemory, loadMemoryConfig } from "@corbits/memory"; const memory = createMemory({ - app, + app, // your Hono app — routes register under /api/tenants/:tenantId/memory/* config: loadMemoryConfig(), // DATABASE_URL + embed env - grantStore, - conditionRegistry, + grantStore, // your Interchange grant store (required for the HTTP mount) + conditionRegistry, // your condition registry }); ``` -That exposes: - -| Method | Path | Grant | -| --- | --- | --- | -| POST | `/api/tenants/:tenantId/memory/add` | `("memory", "add")` | -| POST | `/api/tenants/:tenantId/memory/search` | `("memory", "search")` | -| GET | `/api/tenants/:tenantId/memory/list` | `("memory", "search")` | -| GET | `/api/tenants/:tenantId/memory/feed` | `("memory", "search")` | -| POST | `/api/tenants/:tenantId/memory/documents/:documentId/forget` | `("memory", "forget")` | -| POST | `/api/tenants/:tenantId/memory/documents/:documentId/purge` | `("memory", "purge")` | -| POST | `/api/tenants/:tenantId/memory/versions/:versionId/retention-class` | `("memory", "forget")` | - -Bodies never carry tenant/principal — routes read `c.get("principal")` from -context (set by the hub’s tenant middleware). Missing principal → **401**. -Missing grant → **403**. - -```http -POST /api/tenants/:tenantId/memory/add { "title", "text", "access_tags"?, "share"? } -POST /api/tenants/:tenantId/memory/search { "query", "limit"?, "kinds"?, "entity_ids"?, "sources"?, "includeEvidence"? } -GET /api/tenants/:tenantId/memory/list ?limit= -GET /api/tenants/:tenantId/memory/feed ?after=&limit=&exclude_generator= -POST /api/tenants/:tenantId/memory/documents/:documentId/forget { "reason"? } -POST /api/tenants/:tenantId/memory/documents/:documentId/purge (no body) -POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class { "retention_class": "durable" | "standard" | "ephemeral" | "source_only" } -``` +That registers the tenant routes. Identity is `c.get("principal")` — bodies +never carry tenant or principal. Missing principal → 401. Missing grant → +403. -### Feed - -`GET …/memory/feed` pulls new live versions after a cursor (`after` = last -consumed `feedSeq`, `limit` 1–100, `exclude_generator` skips one -`generator_agent_id`). Grant: `("memory", "search")`, same grant-tag -post-filter as search. Details: [`docs/FEED.md`](docs/FEED.md); the resident -distiller consumes this feed — see `@corbits/memory/distiller` -(`createResidentDistiller`, `runDistillTick`) and -[`docs/DISTILLER.md`](docs/DISTILLER.md). - -### Retention (`forget`, `purge`, `retention-class`) - -- `POST …/documents/:documentId/forget` — tombstone: stops appearing in - search/feed, chunk text redacted, version rows stay for audit. - Grant `("memory", "forget")` **plus** the caller must be the document’s - creator. Plane verb: `memory.tombstoneDocument`. -- `POST …/documents/:documentId/purge` — hard delete, irreversible; refused - while a durable version is untombstoned. Grant `("memory", "purge")` - **plus** creator check. Plane verb: `memory.hardDeleteDocument`. -- `POST …/versions/:versionId/retention-class` — set a version’s retention - class (`setRetentionClass`). Grant `("memory", "forget")` **plus** the - caller must be that version’s creator. - -Details: [`docs/RETENTION.md`](docs/RETENTION.md). Engine config for these -paths comes from `@corbits/memory/config` (`loadMemoryConfig`, -`MemoryConfig`). - -## Agent tools (sidecar bundle) - -A deployed agent carries the memory tools through one factory at -`@corbits/memory/sidecar-bundle`. It holds no client code, no base URL and no -token: it resolves the `hub` credential handle from the host-assembled runtime -capabilities and calls the run-scoped routes through that mediated fetch, -naming its run with the `x-workflow-run-address` header. - -| Tool | Route | -| --- | --- | -| `memory_add` | `POST /api/workflow-memory/add` | -| `memory_search` | `POST /api/workflow-memory/search` | -| `memory_list` | `GET /api/workflow-memory/list` | -| `memory_feed` | `GET /api/workflow-memory/feed` | - -Those routes are a **second, parallel mount** — the tenant routes above keep -their session auth untouched: +In-process, no HTTP and no Postgres — uses the exported fake store. Creator +always sees their own documents. ```ts -import { mountWorkflowMemory } from "@corbits/memory"; +import { createMemory, createFakeDocumentStore } from "@corbits/memory"; -mountWorkflowMemory(workflowMemoryApi, { - memory, - agentToken: { verify, resolveRun }, // host's own token + run lookup +const memory = createMemory({ + documentStore: createFakeDocumentStore(), }); -app.route("/api/workflow-memory", workflowMemoryApi); -``` - -`verify` returns `{ tenantId, definitionId }` for a recognized bearer (or -`undefined`), `resolveRun` maps the run address to `{ tenantId, principalId, -runId }`. A token whose tenant is not the run's tenant is refused with the same -401 as an unknown bearer. Every call is scoped to that run's tenant and -principal; the body never carries identity. -**Host checklist** +await memory.add({ + tenantId: "acme", + principalId: "alice", + content: { + title: "Deploy notes", + text: "Staging deploys run from main.", + }, +}); -1. Mount the tenant routes: `createMemory({ app, grantStore, … })`. -2. Mount `mountWorkflowMemory` and bind the agent's hub credential to the - `hub` handle on its definition. -3. For peer/space share visibility, grant `search` on the relevant document - tags (see `docs/AUTHZ-DOCUMENT-ACCESS.md`). +const { items } = await memory.search({ + tenantId: "acme", + principalId: "alice", + query: "staging", +}); -## Ingestion (in-process) +console.log(items.map((item) => item.title)); +``` -Host workers that already resolved identity can call the plane without HTTP: +On a real hub, omit `documentStore` and pass `config: loadMemoryConfig()` +(needs `DATABASE_URL`, `EMBED_BASE_URL`, `EMBED_MODEL`; see `.env.example`). +Apply migrations first: ```ts -await memory.add({ - tenantId, - principalId, - content: { title, text }, -}); -const { items } = await memory.search({ tenantId, principalId, query }); +import { runMemoryMigrations } from "@corbits/memory/migrations"; + +await runMemoryMigrations(process.env.DATABASE_URL!); ``` -Inference is host-owned: run your model, then `add` / `search`. Core does not -ship an answer endpoint. +## How it works -## Document access +`createMemory` builds the plane. Pass `app` to register +`/api/tenants/:tenantId/memory/*` behind `requireGrant("memory", …)` — +`grantStore` is required for that mount. `loadMemoryConfig` lives on the +barrel and at `@corbits/memory/config`. -Capability grants (`memory:add` / `memory:search`) gate the routes. Per-document -visibility is Interchange **grant tags** on the row (`access_tags`); the creator -always sees their own docs. Details: -[`docs/AUTHZ-DOCUMENT-ACCESS.md`](docs/AUTHZ-DOCUMENT-ACCESS.md). +Capability grants (`memory:add` / `memory:search` / `memory:forget` / +`memory:purge`) gate the routes. Per-document visibility is Interchange +grant tags on the row (`access_tags`); the creator always sees their own +docs. Details: [`docs/AUTHZ-DOCUMENT-ACCESS.md`](docs/AUTHZ-DOCUMENT-ACCESS.md). -## Config +The resident distiller (`createResidentDistiller` / `runDistillTick`) is at +`@corbits/memory/distiller`. -`loadMemoryConfig()` (from `@corbits/memory/config`) reads env (see `.env.example`). For the default pgvector -store you need `DATABASE_URL`, `EMBED_BASE_URL`, `EMBED_MODEL`. +## Development -```ts -import { runMemoryMigrations } from "@corbits/memory/migrations"; -await runMemoryMigrations(process.env.DATABASE_URL!); +```bash +git clone https://github.com/corbitsdev/corbits-memory.git +cd corbits-memory +bun install +bun run typecheck # tsc --noEmit +bun run test # bun test ./src ``` -Inject `documentStore` to use fakes, a host store, or a sibling adapter instead -of Postgres. - -## More - -- Product: [`PRODUCT.md`](PRODUCT.md) -- Architecture: [`ARCHITECTURE.md`](ARCHITECTURE.md) -- Internals: [`IMPLEMENTATION.md`](IMPLEMENTATION.md) +There is no `build` script — the published surface is `src/`. ## License -LGPL-2.1 — see [`LICENSE`](LICENSE). +LGPL-2.1-only — see [`LICENSE`](LICENSE).