Skip to content

Commit a867f83

Browse files
committed
feat(migrations)!: align runMemoryMigrations with runMigrations
runMemoryMigrations(config, { schema, ftsLanguage }) takes the same DBConfig and schema as Interchange runMigrations. schema names the host schema holding the tenant and principal tables, and the foreign-key references in the SQL are rewritten to it; memory's own tables stay in the memory schema. The runner replays every migration file on each run instead of keeping a memory._migrations ledger, which a new migration drops. Foreign keys and CHECK constraints are added only when missing, so a replay does not re-validate tables, a lock timeout keeps a replay from stalling readers, and the one-time temporal_class backfill runs only in the replay that adds the column. ftsLanguage is required, with no default, and the runner no longer reads FTS_LANGUAGE from the environment.
1 parent c108d16 commit a867f83

17 files changed

Lines changed: 307 additions & 140 deletions

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2424
embed model registry, degrade metrics, FTS helpers), the test fakes, and
2525
`resolveGrantConfig` are no longer exported. The distiller stays at
2626
`@corbits/memory/distiller` and migrations at `@corbits/memory/migrations`.
27+
- `runMemoryMigrations(config, { schema, ftsLanguage })` takes the same
28+
`DBConfig` as Interchange `runMigrations` instead of a database URL.
29+
`schema` names the host schema holding Interchange's `tenant` and
30+
`principal` tables (the value passed to `runMigrations`, e.g. `"public"`);
31+
memory's tables stay in the `memory` schema. `ftsLanguage` is required, and
32+
the runner no longer reads `FTS_LANGUAGE` from the environment or accepts a `log` option.
33+
Every migration file is idempotent and replayed on each run. The
34+
`memory._migrations` ledger is dropped.
2735

2836
### Added
2937

‎CONTRIBUTING.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,13 @@ bun run typecheck && bun run test
4444

4545
`bun run typecheck` (`tsc --noEmit`) must be clean before any commit.
4646

47+
## Migrations
48+
49+
`runMemoryMigrations` replays every file in `migrations/` on each run, so
50+
every file must be idempotent. Never change a shipped file's effect on an
51+
existing database: a changed constraint or a new column goes in a new
52+
numbered file, because a guarded `ADD CONSTRAINT` keeps the old definition.
53+
4754
## Branch and PR conventions
4855

4956
- Branch off `main`; open PRs against `main`.

‎IMPLEMENTATION.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ src/
2020
http-client.ts # host-side HTTP client for tenant routes (imperative distill tick)
2121
2222
log.ts # getLogger(["memory"]) from @intx/log
23-
migrations.ts # runMemoryMigrations(url)
23+
migrations.ts # runMemoryMigrations(dbConfig, { schema, ftsLanguage })
2424
ports/ # DocumentStore / SourceProvider + fakes
2525
routes/ # the mounted tenant routes
2626
mount.ts # createMemoryRoutes (HTTP sub-app)
@@ -49,7 +49,7 @@ src/
4949
# @corbits/supermemory-memory-adapter → github.com/corbitsdev/corbits-supermemory-memory-adapter
5050
# @corbits/linear-tools → github.com/corbitsdev/corbits-linear-tools
5151
migrations/ # pgvector schema, applied in filename order by scripts/db-setup.ts
52-
scripts/db-setup.ts # idempotent migration runner, tracked in `_migrations`
52+
scripts/db-setup.ts # runs the idempotent migrations against DATABASE_URL
5353
compose.yml # pgvector + Ollama + reranker for local dev
5454
```
5555

@@ -150,8 +150,8 @@ egress control front the endpoints with an allowlisting proxy.
150150
## Data model (`src/db/schema.ts` + `migrations/*.sql`)
151151

152152
All tables are Drizzle-defined in `db/schema.ts`, DDL'd in `migrations/`
153-
(applied by `scripts/db-setup.ts`, tracked in a `_migrations` ledger table so
154-
re-running is a no-op). No memory table has a foreign key into any
153+
(applied by `runMemoryMigrations`; every file is idempotent and replayed on
154+
each run, so there is no ledger). No memory table has a foreign key into any
155155
control-plane table — `tenant_id`/`principal_id`/source refs are plain `text`.
156156

157157
### `memory_document`
@@ -206,8 +206,8 @@ fresh full insert of its own chunks.
206206

207207
**Changing `FTS_LANGUAGE` on an already-migrated database** (the mismatch
208208
`verifyFtsLanguage` throws on) requires rebuilding the generated column —
209-
`runMemoryMigrations` only applies new files and will not retroactively
210-
alter an existing one. One-time recipe (verified against a live
209+
`runMemoryMigrations` adds the column only if it is missing and will not
210+
retroactively alter an existing one. One-time recipe (verified against a live
211211
`postgres:16` instance):
212212

213213
```sql

‎PRODUCT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ never creates one; it mounts onto yours.
4040
| `createMemoryRoutes({ memory, requireGrant })` | Hono sub-app the host mounts at `/api/tenants/:tenantId/memory` |
4141
| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents |
4242
| `loadMemoryConfig()` | Config from env |
43-
| `runMemoryMigrations(url)` | Apply pgvector schema |
43+
| `runMemoryMigrations(dbConfig, { schema, ftsLanguage })` | Apply pgvector schema |
4444
| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token |
4545
| `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` |
4646

‎README.md‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -65,20 +65,15 @@ Mount `installMemory` below the middleware that sets `principal`/`tenant`
6565
does). Identity comes from `c.get("principal")` — request bodies never
6666
carry tenant or principal. Missing principal → 401, missing grant → 403.
6767

68-
Apply migrations before serving traffic:
68+
Apply migrations before serving traffic, with the same `DBConfig` and
69+
`schema` your hub passes to Interchange `runMigrations`. Memory's tables land
70+
in their own `memory` schema, with foreign keys into that schema's `tenant`
71+
and `principal` tables:
6972

7073
```ts
7174
import { runMemoryMigrations } from "@corbits/memory/migrations";
7275

73-
export async function migrateMemory(databaseUrl: string): Promise<void> {
74-
await runMemoryMigrations(databaseUrl);
75-
}
76-
77-
const databaseUrl = process.env.DATABASE_URL;
78-
if (databaseUrl === undefined) {
79-
throw new Error("DATABASE_URL is required to run memory migrations");
80-
}
81-
await migrateMemory(databaseUrl);
76+
await runMemoryMigrations(dbConfig, { schema: "public", ftsLanguage: "english" });
8277
```
8378

8479
`loadMemoryConfig()` reads `DATABASE_URL` (required — tables live in a

‎bun.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎migrations/0002_memory_baseline.sql‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,8 @@ CREATE TABLE IF NOT EXISTS "memory"."chunk" (
8585
CREATE UNIQUE INDEX IF NOT EXISTS "chunk_version_ordinal_uniq"
8686
ON "memory"."chunk" ("version_id", "ordinal");
8787

88-
-- {{FTS_LANGUAGE}} is substituted by runMemoryMigrations from FTS_LANGUAGE
89-
-- (or opts.ftsLanguage). Must match the language used at query time.
88+
-- {{FTS_LANGUAGE}} is substituted by runMemoryMigrations from its
89+
-- ftsLanguage option. Must match the language used at query time.
9090
ALTER TABLE "memory"."chunk"
9191
ADD COLUMN IF NOT EXISTS "text_fts" tsvector
9292
GENERATED ALWAYS AS (to_tsvector('{{FTS_LANGUAGE}}', "text")) STORED;

‎migrations/0003_claim_bearing.sql‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,14 @@
55
ALTER TABLE "memory"."version"
66
ADD COLUMN IF NOT EXISTS "provenance" text NOT NULL DEFAULT 'unknown';
77

8-
ALTER TABLE "memory"."version"
9-
DROP CONSTRAINT IF EXISTS "version_provenance_check";
10-
11-
ALTER TABLE "memory"."version"
12-
ADD CONSTRAINT "version_provenance_check"
13-
CHECK ("provenance" IN ('stated', 'inferred', 'unknown'));
8+
DO $$
9+
BEGIN
10+
IF NOT EXISTS (
11+
SELECT 1 FROM pg_constraint
12+
WHERE conname = 'version_provenance_check' AND conrelid = '"memory"."version"'::regclass
13+
) THEN
14+
ALTER TABLE "memory"."version"
15+
ADD CONSTRAINT "version_provenance_check"
16+
CHECK ("provenance" IN ('stated', 'inferred', 'unknown'));
17+
END IF;
18+
END $$;

‎migrations/0004_temporal_model.sql‎

Lines changed: 28 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,39 @@
22
-- See docs/TEMPORAL.md. No asserted_at — occurred_at is effective time;
33
-- ingested_at is when the memory plane learned the content.
44

5-
ALTER TABLE "memory"."version"
6-
ADD COLUMN IF NOT EXISTS "temporal_class" text NOT NULL DEFAULT 'event';
7-
8-
ALTER TABLE "memory"."version"
9-
DROP CONSTRAINT IF EXISTS "version_temporal_class_check";
5+
-- The backfill runs only in the replay that adds the column, so distilled
6+
-- claims (inferred provenance) written before this migration default to
7+
-- state ranking exactly once; later inferred `event` rows are left alone.
8+
DO $$
9+
BEGIN
10+
IF NOT EXISTS (
11+
SELECT 1 FROM information_schema.columns
12+
WHERE table_schema = 'memory' AND table_name = 'version'
13+
AND column_name = 'temporal_class'
14+
) THEN
15+
ALTER TABLE "memory"."version"
16+
ADD COLUMN "temporal_class" text NOT NULL DEFAULT 'event';
17+
UPDATE "memory"."version"
18+
SET "temporal_class" = 'state'
19+
WHERE "provenance" = 'inferred';
20+
END IF;
21+
END $$;
1022

11-
ALTER TABLE "memory"."version"
12-
ADD CONSTRAINT "version_temporal_class_check"
13-
CHECK ("temporal_class" IN ('event', 'deadline', 'state', 'lesson'));
23+
DO $$
24+
BEGIN
25+
IF NOT EXISTS (
26+
SELECT 1 FROM pg_constraint
27+
WHERE conname = 'version_temporal_class_check' AND conrelid = '"memory"."version"'::regclass
28+
) THEN
29+
ALTER TABLE "memory"."version"
30+
ADD CONSTRAINT "version_temporal_class_check"
31+
CHECK ("temporal_class" IN ('event', 'deadline', 'state', 'lesson'));
32+
END IF;
33+
END $$;
1434

1535
ALTER TABLE "memory"."version"
1636
ADD COLUMN IF NOT EXISTS "valid_from" timestamp;
1737

1838
ALTER TABLE "memory"."version"
1939
ADD COLUMN IF NOT EXISTS "valid_until" timestamp;
2040

21-
-- Distilled claims (inferred provenance) default to state ranking.
22-
UPDATE "memory"."version"
23-
SET "temporal_class" = 'state'
24-
WHERE "provenance" = 'inferred'
25-
AND "temporal_class" = 'event';

‎migrations/0007_retention.sql‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,17 @@
44
ALTER TABLE "memory"."version"
55
ADD COLUMN IF NOT EXISTS "retention_class" text NOT NULL DEFAULT 'standard';
66

7-
ALTER TABLE "memory"."version"
8-
DROP CONSTRAINT IF EXISTS "version_retention_class_check";
9-
10-
ALTER TABLE "memory"."version"
11-
ADD CONSTRAINT "version_retention_class_check"
12-
CHECK ("retention_class" IN ('durable', 'standard', 'ephemeral', 'source_only'));
7+
DO $$
8+
BEGIN
9+
IF NOT EXISTS (
10+
SELECT 1 FROM pg_constraint
11+
WHERE conname = 'version_retention_class_check' AND conrelid = '"memory"."version"'::regclass
12+
) THEN
13+
ALTER TABLE "memory"."version"
14+
ADD CONSTRAINT "version_retention_class_check"
15+
CHECK ("retention_class" IN ('durable', 'standard', 'ephemeral', 'source_only'));
16+
END IF;
17+
END $$;
1318

1419
CREATE INDEX IF NOT EXISTS "version_retention_ephemeral_idx"
1520
ON "memory"."version" ("tenant_id", "retention_class", "valid_until")

0 commit comments

Comments
 (0)