diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 279f165..805d69d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -75,6 +75,7 @@ jobs: --health-retries 10 env: ASKR_ORM_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/askr_orm_test + ASKR_ORM_TEST_SHADOW_URL: postgresql://postgres:postgres@127.0.0.1:5432/askr_orm_test_shadow steps: - name: Checkout repository uses: actions/checkout@v7 @@ -87,5 +88,11 @@ jobs: - name: Install dependencies run: npm ci + - name: Create isolated scratch database + run: node tests/prepare-postgres.mjs + - name: Run PostgreSQL integration tests run: npm run test:integration + + - name: Qualify installed PostgreSQL generation with minimum peers + run: npm run test:packed diff --git a/CHANGELOG.md b/CHANGELOG.md index 5970cca..4e8681e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,8 +37,20 @@ - Transaction-owned migration plan/apply/resolve handles now reject use after their transaction ends, before any SQL can run on a released connection. +- Complete PostgreSQL scratch reset with an actual database separation check, + session-owned workflow lock, normalized catalog introspection and protocol-only + query description. Unequal URL strings alone no longer authorize reset. +- Normalize the desired PostgreSQL schema inside rollback-only scratch DDL, + preserving native type/default/expression semantics without comparing codecs + or application property names to database columns. +- Quote enum type/schema names, emit one composite primary key, and defer initial + foreign keys until their referenced tables exist. +- Generate conservative nullable query results and standard PostgreSQL parser + types (int8/numeric strings, timestamp/date Date, unknown unsupported types). +- Gate real and normally installed generation/validation/no-op workflows against + isolated PostgreSQL 16, 17 and 18 scratch databases. + ### Remaining release gate -- PostgreSQL shadow reset/introspection/query description are incomplete in the - advertised generation/validation workflow (#55). No ORM 0.5.0 readiness or - publication claim is made until that real workflow is qualified. +- Complete coordinated 0.5.0 packed-candidate and website qualification remains + pending. Maintainer review and approval are required before publication. diff --git a/README.md b/README.md index d52147c..9d80905 100644 --- a/README.md +++ b/README.md @@ -98,10 +98,13 @@ askr database migration plan askr database migration apply --yes ``` -The PostgreSQL generated-artifact workflow is a 0.5.0 release blocker tracked in -[#55](https://github.com/askrjs/askr-orm/issues/55): the shipped shadow reset, -introspection and description methods are incomplete. Ordinary runtime access -and application of an already bundled migration manifest are covered separately. +PostgreSQL generated-artifact tooling uses a disposable scratch database and +proves actual separation from the target before reset. It normalizes supported +catalog shapes and describes query results without executing application queries. +See [PostgreSQL tooling](docs/postgres-tooling.md) for reset scope, supported +objects, conservative result types and connection requirements. Native and +installed PostgreSQL 16–18 CI lanes qualify the workflow. Complete coordinated +0.5.0 candidate qualification and maintainer review are still required. Generation replays checksummed, forward-only SQL against the shadow database before accepting it. It writes migration SQL plus one committed @@ -122,5 +125,5 @@ cannot be interrupted. The [API decisions](docs/0.5.0-api.md) record every retained and removed name and its migration. The [hardening report](docs/0.5.0-hardening.md) distinguishes -regression fixes, executed characterization and the remaining release blocker. +regression fixes, executed characterization and the remaining coordinated release gates. The package version remains 0.4.0 until the coordinated candidate is prepared. diff --git a/docs/0.5.0-api.md b/docs/0.5.0-api.md index 6c81558..06708ae 100644 --- a/docs/0.5.0-api.md +++ b/docs/0.5.0-api.md @@ -1,6 +1,6 @@ # 0.5.0 ORM API decisions -Status: implementation candidate; API issue #53 remains open for the complete 0.5.0 packed-package and generated website review. PostgreSQL generation/validation also remains blocked by [#55](https://github.com/askrjs/askr-orm/issues/55). +Status: implementation candidate; API issue #53 remains open for the complete 0.5.0 packed-package and generated website review. The real PostgreSQL tooling fix for [#55](https://github.com/askrjs/askr-orm/issues/55) has separate native and installed workflow qualification in [PostgreSQL tooling](postgres-tooling.md). The published 0.4.0 declarations expose 132 entrypoint/name pairs. The curated contract exposes 78: 68 root names, one CLI bridge, seven PostgreSQL names and two SQLite names. There are 54 removed named exports, plus the removed `sql.key` member. No compatibility shims are provided. diff --git a/docs/0.5.0-hardening.md b/docs/0.5.0-hardening.md index 3dc5c8c..10b2d95 100644 --- a/docs/0.5.0-hardening.md +++ b/docs/0.5.0-hardening.md @@ -1,9 +1,9 @@ # 0.5.0 ORM hardening evidence -Status: runtime regressions are fixed and the API candidate is qualified locally. -The package is **not release-ready**. [PostgreSQL generated-artifact tooling -#55](https://github.com/askrjs/askr-orm/issues/55) remains a release blocker, so -#53 and #54 stay open alongside the complete candidate/website qualification. +Status: the API/runtime candidate and real PostgreSQL tooling fix are qualified +locally. PostgreSQL 16–18 hosted and installed workflow checks must pass before +#55 closes. The package is **not release-ready**: #53 and #54 still require the +complete coordinated candidate/website qualification and maintainer review. ## Assertion review and real adapters @@ -55,46 +55,49 @@ identifier rejection and a subsequent successful read on both databases. The normal package gate includes strict packed TypeScript 6/7 declaration checking without skipLibCheck, exact 78 runtime/type names, all 54 removed imports, -ColumnBuilder's type-only boundary, removed sql.key, 14 private subpaths, a normal +ColumnBuilder's type-only boundary, removed sql.key, 16 private subpaths, a normal install of the minimum PostgreSQL peers (8.23.0/4.17.0), and root/SQLite imports with those optional peers absent. Installed SQLite executes a complete migration script and proves transaction rollback plus a subsequent query. The CLI bridge's -help path resolves; real PostgreSQL generation is separately blocked by #55. +help path resolves. Installed PostgreSQL generation, validation and no-op +regeneration pass with minimum peers and TypeScript 6/7 generated-artifact checks. `npm run check` runs lint/types, V8 implementation coverage, type contracts, packed consumer, build, the unchanged 20,000-operation/10,000-row benchmark, publint and pack dry run. Format checking runs explicitly and in hosted CI. -The final source suite contains 82 cases: 70 ordinary cases and 12 real +The final source suite contains 100 cases: 70 ordinary cases and 30 real PostgreSQL integration cases. A lane without the database URL skips only those -12 integration cases; it is not PostgreSQL qualification by itself. +30 integration cases; it is not PostgreSQL qualification by itself. -Coverage is measured across implementation source, including the incomplete tooling. +Coverage is measured across implementation source, including the complete private tooling path. -Measured with the real PostgreSQL URL enabled (82 cases): +Measured with both real PostgreSQL URLs enabled (100 cases): -| Implementation | Statements | Branches | Functions | Lines | -| --------------- | ------------------ | ----------------- | --------------- | ------------------ | -| total | 1289/1626 (79.27%) | 699/1023 (68.32%) | 326/399 (81.7%) | 1193/1450 (82.27%) | -| migrations.ts | 77/84 (91.66%) | 30/37 (81.08%) | 18/19 (94.73%) | 71/77 (92.2%) | -| postgres.ts | 118/141 (83.68%) | 55/76 (72.36%) | 25/30 (83.33%) | 113/124 (91.12%) | -| sqlite.ts | 122/168 (72.61%) | 40/68 (58.82%) | 31/49 (63.26%) | 112/145 (77.24%) | -| tooling-impl.ts | 280/482 (58.09%) | 155/352 (44.03%) | 51/85 (60%) | 262/432 (60.64%) | -| client.ts | 216/250 (86.4%) | 108/147 (73.46%) | 65/74 (87.83%) | 200/221 (90.49%) | +| Implementation | Statements | Branches | Functions | Lines | +| ------------------- | ------------------ | ----------------- | ---------------- | ------------------ | +| total | 1594/1887 (84.47%) | 881/1179 (74.72%) | 407/459 (88.67%) | 1474/1692 (87.11%) | +| client.ts | 216/250 (86.4%) | 106/147 (72.1%) | 65/74 (87.83%) | 200/221 (90.49%) | +| migrations.ts | 77/84 (91.66%) | 29/37 (78.37%) | 18/19 (94.73%) | 71/77 (92.2%) | +| postgres-client.ts | 21/21 (100%) | 4/4 (100%) | 7/7 (100%) | 19/19 (100%) | +| postgres-tooling.ts | 185/187 (98.93%) | 85/90 (94.44%) | 36/36 (100%) | 172/174 (98.85%) | +| postgres.ts | 94/107 (87.85%) | 55/72 (76.38%) | 19/19 (100%) | 89/92 (96.73%) | +| sqlite.ts | 122/168 (72.61%) | 40/68 (58.82%) | 31/49 (63.26%) | 112/145 (77.24%) | +| tooling-impl.ts | 400/569 (70.29%) | 249/418 (59.56%) | 93/113 (82.3%) | 374/513 (72.9%) | Executed commands: ```sh vp env exec --node 24.21.0 npm run fmt -- --check -ASKR_ORM_TEST_DATABASE_URL= vp env exec --node 24.21.0 npm run check +ASKR_ORM_TEST_DATABASE_URL= ASKR_ORM_TEST_SHADOW_URL= vp env exec --node 24.21.0 npm run check vp env exec --node 24.21.0 npm audit --json ``` The normal package CI gate has no PostgreSQL URL; its unit report is separate -from the integration proof above. The real PostgreSQL matrix executes the 12 -integration cases. Coverage HTML/JSON is retained for assertion inspection. +from the integration proof above. The real PostgreSQL matrix executes 12 runtime and 18 tooling +integration cases, followed by the normal installed minimum-peer workflow. Coverage HTML/JSON is retained for assertion inspection. -Test/fixture source is excluded. In particular, incomplete tooling branches are -not treated as covered or accepted merely because runtime tests pass. +Test/fixture source is excluded. Uncovered branches remain visible in the retained coverage; test names and +source counts are not acceptance evidence. The compatible development dependency update reduces the npm audit result from four findings to zero. It does not change optional peer ranges or use forced @@ -121,14 +124,25 @@ moved tooling function bodies, 40 remain token-identical; loadDatabases has this single intentional boundary correction. This does not complete the distinct PostgreSQL shadow workflow tracked below. -## Remaining PostgreSQL tooling blocker - -An executed real PostgreSQL probe confirms that shadow reset always throws, -introspection returns database/schema identity rows rather than a normalized -schema snapshot, and description returns no columns for `SELECT $1::text AS -value`. The advertised generate/validate path calls these boundaries directly. -The prior generation unit test supplies fake implementations and therefore does -not qualify this real adapter workflow. #55 requires actual target/scratch -separation before destructive reset, complete supported-object introspection, -query description without executing application queries, and a real -`generate → validate → unchanged generate` recovery matrix for PostgreSQL 16–18. +## PostgreSQL tooling qualification + +The original real PostgreSQL probe found three stubs: reset always threw, +introspection returned database/schema identity rows, and description returned +no columns. The fake-driver unit fixture did not qualify these boundaries. +The real-adapter regressions and the normal installed workflow are documented in +[PostgreSQL tooling](postgres-tooling.md). They cover actual target aliases, +unreachable target, reset rollback, deterministic supported catalogs, protocol +metadata without executing queries, concurrent description, workflow lock +ownership, real generation/byte-validation/no-op, drift/stale-artifact failure +and replay recovery. Composite keys and quoted enum names receive native tests. + +The final 18-case native tooling suite fails 17 cases against the previous merged +PostgreSQL/schema/private-tooling source at +`3f595754d7e2a3135cd0ed0f5ef67b05b306e82c`; unreachable-target refusal is one +passing characterization because the old reset stub refuses everything. These +are scenarios, not 17 independent bug claims. Additional refinement RED cases +caught quoted enum SQL, missing related-table creation order, generated constraint +name differences during explicit rename, four silently ignored physical schema +attributes, duplicate query output names, and repeated parameter declarations +rejected by TypeScript. The final candidate passes all 100 tests and the installed +TypeScript 6/7 workflow. diff --git a/docs/adapters.md b/docs/adapters.md index 6d0fff8..512eac4 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -34,6 +34,10 @@ interpolate application data. Placeholder rendering is dialect-owned. Set `targetIdentity` and `scratchIdentity` in `database/index.ts`. Tooling checks these before reset or target migration work and fails closed on a match. +The built-in PostgreSQL driver additionally proves actual target/scratch +separation before reset and owns a pinned scratch workflow session. See +[PostgreSQL tooling](postgres-tooling.md) for destructive reset scope, supported +catalog shapes, conservative query metadata and connection requirements. Driver errors may pass through the adapter. The ORM normalizes PostgreSQL constraint, serialization, deadlock, timeout, cancellation, and connection diff --git a/docs/postgres-tooling.md b/docs/postgres-tooling.md new file mode 100644 index 0000000..157d8f5 --- /dev/null +++ b/docs/postgres-tooling.md @@ -0,0 +1,111 @@ +# PostgreSQL generated-artifact tooling + +`generate`, `validate` and manual-artifact refresh use the configured **disposable +scratch database**. Reset removes all non-system schemas and their dependent +objects from that database, then recreates `public`. Do not configure a database +containing data you intend to keep as `shadowUrl` / `DATABASE_SHADOW_URL`. + +The application target is never reset or migrated by these commands. Before +scratch reset, tooling opens a read-only target transaction and holds a random +two-integer session advisory lock. The scratch connection must independently +acquire the same key while the target still owns its lock. The target must then +successfully unlock it. A shared actual database, an unavailable target, or a +lost lock owner fails closed before destructive SQL. URL spelling is not proof +of separation. The target connection needs connection and advisory-lock rights; +this guard reads no application tables and requires no superuser catalog access. + +[PostgreSQL documents database-local advisory locks](https://www.postgresql.org/docs/16/view-pg-locks.html) +and the separate integer-key namespaces and session unlock semantics in its +[advisory lock functions](https://www.postgresql.org/docs/16/functions-admin.html#FUNCTIONS-ADVISORY-LOCKS). + +One scratch adapter owns one physical session, serializes its method calls and +holds a distinct workflow advisory lock until close. Two Askr tooling processes +therefore cannot interleave reset/replay/probe/description on the same scratch +database. Use a direct connection or a session pool; transaction/statement +pooling does not provide this session ownership contract. Every workflow closes +its owner, rolling back any unfinished transaction and releasing the lock. +Failed rollback, unlock or deallocation discards an unsafe native pool client. + +## Physical schema comparison + +Introspection reads PostgreSQL catalogs for ordinary tables and columns, native +types and type modifiers, defaults, stored generated expressions, primary keys, +unique/check constraints, single-column foreign keys, ordinary indexes, enums +and views. Results use deterministic object/column/constraint ordering. Enum +labels and column names are explicitly converted to text arrays for normal `pg` +decoding on every supported PostgreSQL version. + +The desired schema is rendered inside a scratch transaction after temporarily +removing the replayed tables/views/enums. PostgreSQL introspects that desired DDL; +rollback then restores the replayed schema before the diff. This compares native +physical forms, including `varchar`/`character varying`, casts, defaults and view +expressions. It does not copy desired physical facts onto actual introspection. +Application property names and codecs stay in generated source identity; +rename/conversion instructions are excluded from physical equality. Constraint and +index names are compared when the definition explicitly names them; generated +names remain engine-owned, including after a declared column/table rename. +Constraint expressions, columns, uniqueness, methods and predicates remain part +of equality. New related tables are all created before their foreign keys. + +Initial tables precede foreign-key additions, permitting references to tables +declared later and mutual references. A composite key renders as one table-level +primary key, preserving its declared column order. Known enum identifiers are +quoted individually, including embedded quotes and reserved URL characters. + +This is a supported-declaration comparison, not a model of every PostgreSQL +object. Materialized/partitioned/foreign relations, inheritance, unlogged tables, identity +columns, custom column collations, non-stored generated columns, deferred, +unvalidated or unenforced constraints, NULLS NOT DISTINCT uniqueness, +multi-column/non-default foreign keys, invalid indexes and indexes with included +columns currently produce corrective errors. Standalone +sequences, functions, extension-owned objects, grants and other administrative +objects are outside the snapshot; maintain them explicitly in reviewed SQL. +Additional unsupported physical changes and constraint/index changes require a +manual migration. A failed desired-schema probe or final comparison is never +accepted as an unchanged schema. + +## Query metadata without execution + +Tooling submits a named protocol Parse, Describe and Sync exchange, followed by +deallocation. It sends no Bind or Execute message. PostgreSQL rejects multiple +commands during Parse; no later statement in a batch can run. Result column names +and native types come from RowDescription and `format_type`, not sampled rows. +Repeated named parameters retain all positional bindings but appear only once +in the generated parameter interface. Duplicate result column names fail with +an explicit alias error rather than producing ambiguous generated properties. +See PostgreSQL's [extended-query message flow](https://www.postgresql.org/docs/16/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY). + +RowDescription does not establish expression or outer-join nullability, so every +generated PostgreSQL query column is conservatively nullable. Generated result +types assume the standard `pg` parsers: int8/numeric return strings; date and +timestamp return `Date`; supported integer/float/boolean/text/uuid/bytea types use +their corresponding JavaScript types. JSON and unrecognized/custom/array types +remain `unknown`. Custom pool type parsers and application coercions are caller +contracts and are not inferred by tooling. + +## Executed qualification + +The real tooling lane exercises alias-target refusal with preserved target data, +unreachable target, read-only reset rollback, deterministic quoted catalogs, +metadata for SELECT and INSERT RETURNING without changing a row or advancing a +sequence, multi-command rejection, error recovery, concurrent descriptions with +pool size one, scratch lock handoff, complete generation/validation/no-op, +stale artifacts, migration drift, failed replay recovery, incremental related +tables, unique-column rename and no-op replay, composite keys, enums, +indexes, checks, views and standard query value types. + +The packed contract normally installs minimum `pg@8.23.0` and +`pg-query-stream@4.17.0`, repeats generation/validation/no-op through the installed +public tooling bridge and compiles the generated artifact under TypeScript 6 and 7. CI runs both source and installed workflows on PostgreSQL 16, 17 and 18. + +Local integration requires explicit isolated databases: + +```sh +ASKR_ORM_TEST_DATABASE_URL= \ +ASKR_ORM_TEST_SHADOW_URL= \ +vp env exec --node 24.21.0 npm run test:integration +``` + +Ordinary package tests skip these native PostgreSQL cases without both URLs. +The explicit integration command rejects missing URLs instead of passing skips. +No release version, tag or publication is included in this change. diff --git a/package.json b/package.json index 824e075..56bacf0 100644 --- a/package.json +++ b/package.json @@ -62,7 +62,7 @@ "typecheck": "tsc -p tsconfig.json --noEmit", "test": "vp test run", "test:coverage": "vp test run --coverage", - "test:integration": "vp test run src/integration.test.ts", + "test:integration": "node tests/postgres-test-env.mjs && vp test run src/integration.test.ts src/tooling-integration.test.ts", "test:types": "npm run build && tsc -p tests/types/tsconfig.json", "test:publint": "publint", "pack:check": "npm pack --ignore-scripts --dry-run --json", diff --git a/src/postgres-client.ts b/src/postgres-client.ts new file mode 100644 index 0000000..ad77852 --- /dev/null +++ b/src/postgres-client.ts @@ -0,0 +1,49 @@ +import type { PoolClient, QueryResult } from "pg"; +import { normalizeDatabaseError, type DatabaseError } from "./errors"; + +export class CheckedOutClient { + private failure: DatabaseError | undefined; + + private readonly onError = (error: Error): void => { + this.failure ??= normalizeDatabaseError(error); + }; + + constructor(private readonly client: PoolClient) { + client.on("error", this.onError); + } + + assertHealthy(): void { + if (this.failure) throw this.failure; + } + + async query(config: unknown): Promise { + this.assertHealthy(); + try { + const result = await this.client.query(config as never); + this.assertHealthy(); + return result; + } catch (error) { + throw normalizeDatabaseError(error); + } + } + + startStream(query: T): T { + this.assertHealthy(); + return this.client.query(query as never) as T; + } + + async cleanup(statement: string): Promise { + try { + await this.query(statement); + } catch (error) { + // A failed rollback/unlock cannot leave a reusable pooled connection. + this.failure ??= normalizeDatabaseError(error); + } + } + + release(error?: unknown): void { + if (error !== undefined) this.failure ??= normalizeDatabaseError(error); + this.client.off("error", this.onError); + this.client.release(this.failure); + } +} diff --git a/src/postgres-tooling.ts b/src/postgres-tooling.ts new file mode 100644 index 0000000..bbb433e --- /dev/null +++ b/src/postgres-tooling.ts @@ -0,0 +1,479 @@ +import { randomBytes } from "node:crypto"; +import type { Connection, FieldDef, Pool, Submittable } from "pg"; +import type { DatabaseToolingAdapter } from "./definition"; +import { quoteIdentifier } from "./naming"; +import { CheckedOutClient } from "./postgres-client"; +import type { TableConstraint } from "./schema"; +import type { SchemaSnapshot, SnapshotColumn, SnapshotTable } from "./tooling-impl"; + +const TOOLING_LOCK = "4707438161740730"; +const userSchema = "n.nspname !~ '^pg_' AND n.nspname <> 'information_schema'"; + +async function rows( + client: CheckedOutClient, + text: string, + values: unknown[] = [], +): Promise { + return (await client.query({ text, values })).rows as T[]; +} + +/** A Describe/Sync exchange never binds or executes the prepared statement. */ +class StatementMetadata implements Submittable { + private fields: FieldDef[] = []; + private resolve!: (fields: FieldDef[]) => void; + private reject!: (error: unknown) => void; + readonly result = new Promise((resolve, reject) => { + this.resolve = resolve; + this.reject = reject; + }); + constructor(readonly sql?: string) {} + submit(connection: Connection): void { + if (this.sql === undefined) connection.describe({ type: "S", name: "askr_describe" }, false); + else connection.parse({ name: "askr_describe", text: this.sql, types: [] }, false); + connection.sync(); + } + handleRowDescription(message: { fields: FieldDef[] }): void { + this.fields = message.fields; + } + handleReadyForQuery(): void { + this.resolve(this.fields); + } + handleError(error: unknown): void { + this.reject(error); + } +} + +async function introspect(client: CheckedOutClient): Promise { + const relations = await rows<{ + oid: string; + schema: string; + name: string; + kind: string; + inherited: boolean; + persistence: string; + }>( + client, + ` + SELECT c.oid::text, n.nspname AS schema, c.relname AS name, c.relkind AS kind, + EXISTS (SELECT 1 FROM pg_inherits i WHERE i.inhrelid = c.oid) AS inherited, c.relpersistence AS persistence + FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE ${userSchema} AND c.relkind IN ('r', 'v', 'm', 'p', 'f') + AND NOT EXISTS (SELECT 1 FROM pg_depend d WHERE d.classid = 'pg_class'::regclass AND d.objid = c.oid AND d.deptype = 'e') + `, + ); + const tables: SnapshotTable[] = []; + const views: SchemaSnapshot["views"][number][] = []; + for (const relation of relations) { + if (relation.kind === "v") { + const [view] = await rows<{ query: string }>( + client, + "SELECT pg_get_viewdef($1::oid, false) AS query", + [relation.oid], + ); + views.push({ + kind: "view", + schema: relation.schema, + name: relation.name, + query: view!.query, + }); + continue; + } + if (relation.kind !== "r") + throw new Error( + `PostgreSQL tooling cannot diff relation ${relation.schema}.${relation.name} of kind ${relation.kind}; use a supported ordinary table/view.`, + ); + if (relation.inherited || relation.persistence !== "p") + throw new Error( + `Table ${relation.schema}.${relation.name} inheritance or persistence cannot be represented by the declaration.`, + ); + const columns = await rows<{ + name: string; + dataType: string; + nullable: boolean; + primaryKey: boolean; + default: string | null; + generated: string | null; + identity: string; + generatedMode: string; + customCollation: boolean; + }>( + client, + ` + SELECT a.attname AS name, format_type(a.atttypid, a.atttypmod) AS "dataType", + NOT a.attnotnull AS nullable, + EXISTS (SELECT 1 FROM pg_constraint k WHERE k.conrelid = a.attrelid AND k.contype = 'p' AND a.attnum = ANY(k.conkey)) AS "primaryKey", + CASE WHEN a.attgenerated = '' THEN pg_get_expr(d.adbin, d.adrelid, false) END AS "default", + CASE WHEN a.attgenerated <> '' THEN pg_get_expr(d.adbin, d.adrelid, false) END AS generated, + a.attidentity AS identity, a.attgenerated AS "generatedMode", a.attcollation <> t.typcollation AS "customCollation" + FROM pg_attribute a JOIN pg_type t ON t.oid = a.atttypid LEFT JOIN pg_attrdef d ON d.adrelid = a.attrelid AND d.adnum = a.attnum + WHERE a.attrelid = $1::oid AND a.attnum > 0 AND NOT a.attisdropped + `, + [relation.oid], + ); + const constraints = await rows<{ + kind: string; + name: string; + columns: string[]; + expression: string | null; + referenceSchema: string | null; + referenceTable: string | null; + referenceColumns: string[]; + deleteAction: string; + updateAction: string; + match: string; + deferred: boolean; + validated: boolean; + enforced: boolean; + nullsNotDistinct: boolean; + }>( + client, + ` + SELECT k.contype AS kind, k.conname AS name, + ARRAY(SELECT a.attname::text FROM unnest(k.conkey) WITH ORDINALITY u(num, ord) JOIN pg_attribute a ON a.attrelid = k.conrelid AND a.attnum = u.num ORDER BY u.ord) AS columns, + pg_get_expr(k.conbin, k.conrelid, false) AS expression, + rn.nspname AS "referenceSchema", r.relname AS "referenceTable", + ARRAY(SELECT a.attname::text FROM unnest(k.confkey) WITH ORDINALITY u(num, ord) JOIN pg_attribute a ON a.attrelid = k.confrelid AND a.attnum = u.num ORDER BY u.ord) AS "referenceColumns", + k.confdeltype AS "deleteAction", k.confupdtype AS "updateAction", k.confmatchtype AS match, + k.condeferrable AS deferred, k.convalidated AS validated, + COALESCE((to_jsonb(k)->>'conenforced')::boolean, true) AS enforced, + COALESCE(ci.indnullsnotdistinct, false) AS "nullsNotDistinct" + FROM pg_constraint k LEFT JOIN pg_class r ON r.oid = k.confrelid LEFT JOIN pg_namespace rn ON rn.oid = r.relnamespace + LEFT JOIN pg_index ci ON ci.indexrelid = k.conindid + WHERE k.conrelid = $1::oid + `, + [relation.oid], + ); + const snapshotColumns: SnapshotColumn[] = columns.map((column) => { + if (column.customCollation || (column.generatedMode !== "" && column.generatedMode !== "s")) + throw new Error( + `Column ${relation.name}.${column.name} collation or generation mode cannot be represented by the declaration.`, + ); + if (column.identity) + throw new Error( + `Identity column ${relation.name}.${column.name} requires explicit manual tooling support.`, + ); + const foreign = constraints.filter( + (constraint) => constraint.kind === "f" && constraint.columns.includes(column.name), + ); + if (foreign.length > 1) + throw new Error( + `Multiple foreign keys on ${relation.name}.${column.name} cannot be represented by the declaration.`, + ); + const reference = foreign[0]; + return { + property: column.name, + name: column.name, + dataType: column.dataType, + nullable: column.nullable, + primaryKey: column.primaryKey, + unique: false, + ...(column.default === null ? {} : { default: column.default }), + ...(column.generated === null ? {} : { generated: column.generated }), + ...(reference + ? { + reference: { + schema: reference.referenceSchema!, + table: reference.referenceTable!, + column: reference.referenceColumns[0]!, + }, + } + : {}), + }; + }); + const snapshotConstraints: TableConstraint[] = []; + for (const constraint of constraints) { + if (!constraint.enforced || constraint.nullsNotDistinct) + throw new Error( + `Unenforced or NULLS NOT DISTINCT constraint ${constraint.name} cannot be represented by the declaration.`, + ); + if (constraint.deferred || !constraint.validated) + throw new Error( + `Deferred or unvalidated constraint ${constraint.name} cannot be represented by the declaration.`, + ); + if (constraint.kind === "p" || constraint.kind === "n") continue; + if (constraint.kind === "u") + snapshotConstraints.push({ + kind: "unique", + name: constraint.name, + columns: constraint.columns, + }); + else if (constraint.kind === "c") + snapshotConstraints.push({ + kind: "check", + name: constraint.name, + expression: constraint.expression!, + }); + else if (constraint.kind === "f") { + if ( + constraint.columns.length !== 1 || + constraint.referenceColumns.length !== 1 || + constraint.deleteAction !== "a" || + constraint.updateAction !== "a" || + constraint.match !== "s" + ) + throw new Error( + `Foreign key ${constraint.name} cannot be represented by the declaration.`, + ); + } else + throw new Error( + `Constraint ${constraint.name} of kind ${constraint.kind} cannot be represented by the declaration.`, + ); + } + const indexes = await rows<{ + name: string; + expressions: string[]; + unique: boolean; + method: string; + where: string | null; + valid: boolean; + keyCount: number; + totalCount: number; + nullsNotDistinct: boolean; + }>( + client, + ` + SELECT c.relname AS name, i.indisunique AS unique, m.amname AS method, + ARRAY(SELECT pg_get_indexdef(i.indexrelid, s, false) FROM generate_series(1, i.indnkeyatts) s) AS expressions, + pg_get_expr(i.indpred, i.indrelid, false) AS "where", i.indisvalid AS valid, + i.indnkeyatts AS "keyCount", i.indnatts AS "totalCount", i.indnullsnotdistinct AS "nullsNotDistinct" + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid JOIN pg_am m ON m.oid = c.relam + WHERE i.indrelid = $1::oid AND NOT EXISTS (SELECT 1 FROM pg_constraint k WHERE k.conindid = i.indexrelid) + `, + [relation.oid], + ); + for (const index of indexes) { + if (!index.valid || index.keyCount !== index.totalCount || index.nullsNotDistinct) + throw new Error( + `Invalid or covering index ${index.name} cannot be represented by the declaration.`, + ); + snapshotConstraints.push({ + kind: "index", + name: index.name, + expressions: index.expressions, + unique: index.unique, + method: index.method, + ...(index.where === null ? {} : { where: index.where }), + }); + } + const primary = constraints.find((constraint) => constraint.kind === "p"); + tables.push({ + schema: relation.schema, + name: relation.name, + columns: snapshotColumns.sort((a, b) => a.name.localeCompare(b.name)), + constraints: snapshotConstraints.sort((a, b) => + JSON.stringify(a).localeCompare(JSON.stringify(b)), + ), + ...(primary && primary.columns.length > 1 ? { primaryKeyColumns: primary.columns } : {}), + }); + } + const enums = await rows( + client, + ` + SELECT n.nspname AS schema, t.typname AS name, array_agg(e.enumlabel::text ORDER BY e.enumsortorder) AS values + FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace JOIN pg_enum e ON e.enumtypid = t.oid + WHERE ${userSchema} AND NOT EXISTS (SELECT 1 FROM pg_depend d WHERE d.classid = 'pg_type'::regclass AND d.objid = t.oid AND d.deptype = 'e') + GROUP BY n.nspname, t.typname + `, + ); + const order = (a: { schema: string; name: string }, b: { schema: string; name: string }) => + `${a.schema}.${a.name}`.localeCompare(`${b.schema}.${b.name}`); + return { + version: 1, + enums: enums.sort(order), + tables: tables.sort(order), + views: views.sort(order), + }; +} + +/** Proves actual database separation using database-local, session-owned locks. */ +async function assertSeparate(shadow: CheckedOutClient, targetPool: Pool): Promise { + const target = new CheckedOutClient(await targetPool.connect()); + const bytes = randomBytes(8); + const key = [bytes.readInt32BE(0), bytes.readInt32BE(4)]; + let targetLocked = false; + let shadowLocked = false; + try { + // Keep the target lock's backend pinned even behind transaction pooling. + await target.query("BEGIN READ ONLY"); + targetLocked = + ( + await rows<{ locked: boolean }>( + target, + "SELECT pg_try_advisory_lock($1::integer, $2::integer) AS locked", + key, + ) + )[0]?.locked === true; + if (!targetLocked) + throw new Error("Cannot establish target database separation; refusing scratch reset."); + shadowLocked = + ( + await rows<{ locked: boolean }>( + shadow, + "SELECT pg_try_advisory_lock($1::integer, $2::integer) AS locked", + key, + ) + )[0]?.locked === true; + const held = + ( + await rows<{ unlocked: boolean }>( + target, + "SELECT pg_advisory_unlock($1::integer, $2::integer) AS unlocked", + key, + ) + )[0]?.unlocked === true; + targetLocked = !held; + if (!held || !shadowLocked) + throw new Error( + "PostgreSQL scratch must be a distinct actual database from target; refusing reset.", + ); + } finally { + if (shadowLocked) await shadow.cleanup(`SELECT pg_advisory_unlock(${key[0]}, ${key[1]})`); + if (targetLocked) await target.cleanup(`SELECT pg_advisory_unlock(${key[0]}, ${key[1]})`); + await target.cleanup("ROLLBACK"); + target.release(); + } + shadow.assertHealthy(); +} + +export async function createPostgresTooling( + shadowUrl: string, + shadowPool: () => Promise, + targetPool: () => Promise, +): Promise { + const pool = await shadowPool(); + let client: CheckedOutClient | undefined; + let closed = false; + let queue = Promise.resolve(); + const serialized = (operation: () => Promise): Promise => { + const pending = queue.then(async () => { + try { + return await operation(); + } catch (error) { + if (client) { + try { + client.assertHealthy(); + } catch (failure) { + client.release(failure); + client = undefined; + } + } + throw error; + } + }); + queue = pending.then( + () => undefined, + () => undefined, + ); + return pending; + }; + const connection = async () => { + if (closed) throw new Error("PostgreSQL scratch adapter is closed."); + if (!client) { + const acquired = new CheckedOutClient(await pool.connect()); + try { + await acquired.query(`SELECT pg_advisory_lock(${TOOLING_LOCK})`); + } catch (error) { + acquired.release(error); + throw error; + } + client = acquired; + } + client.assertHealthy(); + return client; + }; + const adapter: DatabaseToolingAdapter = { + identity: shadowUrl, + async reset() { + const current = await connection(); + const target = await targetPool(); + try { + await assertSeparate(current, target); + } finally { + await target.end(); + } + await current.query("BEGIN"); + try { + const schemas = await rows<{ name: string }>( + current, + `SELECT n.nspname AS name FROM pg_namespace n WHERE ${userSchema}`, + ); + for (const schema of schemas) + await current.query(`DROP SCHEMA ${quoteIdentifier(schema.name)} CASCADE`); + await current.query('CREATE SCHEMA "public"'); + await current.query("COMMIT"); + } catch (error) { + await current.cleanup("ROLLBACK"); + throw error; + } + }, + async execute(sql) { + await (await connection()).query(sql); + }, + async introspect() { + return introspect(await connection()); + }, + async describe(sql, parameterNames) { + const current = await connection(); + let prepared = false; + let failed = false; + let primaryError: unknown; + let result: Awaited> | undefined; + try { + const prepare = new StatementMetadata(sql); + current.startStream(prepare); + await prepare.result; + prepared = true; + const description = new StatementMetadata(); + current.startStream(description); + const fields = await description.result; + const columns = []; + for (const field of fields) { + const [metadata] = await rows<{ dataType: string }>( + current, + 'SELECT format_type($1::oid, $2::integer) AS "dataType"', + [field.dataTypeID, field.dataTypeModifier], + ); + // RowDescription does not encode outer-join/expression nullability. + columns.push({ name: field.name, dataType: metadata!.dataType, nullable: true }); + } + result = { parameters: [...parameterNames], columns }; + } catch (error) { + failed = true; + primaryError = error; + } + if (prepared) { + try { + await current.query("DEALLOCATE askr_describe"); + } catch (error) { + current.release(error); + client = undefined; + if (!failed) { + failed = true; + primaryError = error; + } + } + } + if (failed) throw primaryError; + return result!; + }, + async close() { + if (closed) return; + closed = true; + if (client) { + await client.cleanup("ROLLBACK"); + await client.cleanup(`SELECT pg_advisory_unlock(${TOOLING_LOCK})`); + client.release(); + client = undefined; + } + await pool.end(); + }, + }; + return { + identity: shadowUrl, + reset: () => serialized(() => adapter.reset()), + execute: (sql) => serialized(() => adapter.execute(sql)), + introspect: () => serialized(() => adapter.introspect()), + describe: (sql, parameters) => serialized(() => adapter.describe(sql, parameters)), + close: () => serialized(() => adapter.close!()), + }; +} diff --git a/src/postgres.test.ts b/src/postgres.test.ts index e7c4b58..14efc77 100644 --- a/src/postgres.test.ts +++ b/src/postgres.test.ts @@ -13,7 +13,16 @@ const state = vi.hoisted(() => ({ vi.mock("pg", () => { class Pool { - async query(config: string | { text?: string }) { + async query( + config: + | string + | { text?: string; sql?: string; submit?: unknown; handleReadyForQuery?: () => void }, + ) { + if (typeof config !== "string" && typeof config.submit === "function") { + state.statements.push(config.sql === undefined ? "DESCRIBE" : `PARSE ${config.sql}`); + config.handleReadyForQuery?.(); + return { rows: [], rowCount: 0 }; + } const statement = typeof config === "string" ? config : (config.text ?? ""); state.statements.push(statement); if (state.failRollback && statement === "ROLLBACK") throw new Error("rollback failed"); @@ -75,10 +84,11 @@ describe("PostgreSQL adapter", () => { }).shadow(); await shadow.describe("SELECT $1::text AS value", ["value"]); await shadow.describe("SELECT $1::text AS value", ["value"]); - expect(state.statements.filter((sql) => sql.startsWith("PREPARE"))).toHaveLength(2); + expect(state.statements.filter((sql) => sql.startsWith("PARSE"))).toHaveLength(2); expect(state.statements.filter((sql) => sql === "DEALLOCATE askr_describe")).toHaveLength(2); - expect(state.releases).toBe(2); + expect(state.releases).toBe(0); await shadow.close?.(); + expect(state.releases).toBe(1); }); it("should make runtime close idempotent", async () => { @@ -144,8 +154,9 @@ describe("PostgreSQL adapter", () => { await expect(shadow.describe("SELECT $1::text", ["value"])).resolves.toMatchObject({ parameters: ["value"], }); - expect(state.releases).toBe(2); + expect(state.releases).toBe(1); await shadow.close?.(); + expect(state.releases).toBe(2); }); it("should preserve the callback error when rollback fails", async () => { diff --git a/src/postgres.ts b/src/postgres.ts index 57c9ac1..a99ff97 100644 --- a/src/postgres.ts +++ b/src/postgres.ts @@ -1,4 +1,4 @@ -import type { Pool as PoolType, PoolClient, PoolConfig, QueryResult } from "pg"; +import type { Pool as PoolType, PoolConfig, QueryResult } from "pg"; import type { DatabaseAdapter, DatabaseDriver, @@ -6,9 +6,9 @@ import type { QueryOptions, TransactionOptions, } from "./adapter"; -import type { DatabaseToolingAdapter } from "./definition"; import type { SqlQuery } from "./sql"; -import { normalizeDatabaseError, type DatabaseError } from "./errors"; +import { CheckedOutClient } from "./postgres-client"; +import { createPostgresTooling } from "./postgres-tooling"; export { jsonb, postgresEnum, postgresType, timestampTz, bytea } from "./schema"; const MIGRATION_LOCK_KEY = "4707438161740729"; @@ -26,53 +26,6 @@ function lazy(value: string | (() => string) | undefined, environment: string): return result; } -class CheckedOutClient { - private failure: DatabaseError | undefined; - - private readonly onError = (error: Error): void => { - this.failure ??= normalizeDatabaseError(error); - }; - - constructor(private readonly client: PoolClient) { - client.on("error", this.onError); - } - - assertHealthy(): void { - if (this.failure) throw this.failure; - } - - async query(config: unknown): Promise { - this.assertHealthy(); - try { - const result = await this.client.query(config as never); - this.assertHealthy(); - return result; - } catch (error) { - throw normalizeDatabaseError(error); - } - } - - startStream(query: T): T { - this.assertHealthy(); - return this.client.query(query as never) as T; - } - - async cleanup(statement: string): Promise { - try { - await this.query(statement); - } catch (error) { - // A failed rollback/unlock cannot leave a reusable pooled connection. - this.failure ??= normalizeDatabaseError(error); - } - } - - release(error?: unknown): void { - if (error !== undefined) this.failure ??= normalizeDatabaseError(error); - this.client.off("error", this.onError); - this.client.release(this.failure); - } -} - class PgAdapter implements DatabaseAdapter { readonly identity: string; private closed = false; @@ -233,45 +186,6 @@ async function createPool( return new module.Pool({ ...options, connectionString: url }); } -async function pgTooling( - url: string, - options?: Omit, -): Promise { - const pool = await createPool(url, options); - return { - identity: url, - async reset() { - throw new Error("PostgreSQL shadow reset must be performed by migration tooling."); - }, - async execute(sql) { - await pool.query(sql); - }, - async introspect() { - return (await pool.query("SELECT current_database() AS database, current_schema() AS schema")) - .rows; - }, - async describe(sql, parameterNames) { - const client = new CheckedOutClient(await pool.connect()); - let cleanupError: unknown; - try { - await client.query(`PREPARE askr_describe AS ${sql}`); - try { - await client.query("DEALLOCATE askr_describe"); - } catch (error) { - cleanupError = error; - throw error; - } - return { parameters: [...parameterNames], columns: [] }; - } finally { - client.release(cleanupError); - } - }, - async close() { - await pool.end(); - }, - }; -} - /** * Creates a PostgreSQL {@link DatabaseDriver} backed by `pg`, connecting to `options.url` * (defaulting to `DATABASE_URL`). Requires the optional peers `pg` and `pg-query-stream`; the @@ -297,7 +211,11 @@ export function postgres(options: PostgresOptions = {}): DatabaseDriver { const shadow = lazy(options.shadowUrl, "DATABASE_SHADOW_URL"); if (target === shadow) throw new Error("PostgreSQL target and shadow database identities must differ."); - return pgTooling(shadow, options.pool); + return createPostgresTooling( + shadow, + () => createPool(shadow, options.pool), + () => createPool(target, options.pool), + ); }, }; } diff --git a/src/schema.test.ts b/src/schema.test.ts index ff1e74d..fd4431f 100644 --- a/src/schema.test.ts +++ b/src/schema.test.ts @@ -53,7 +53,7 @@ describe("schema definitions", () => { it("should define Postgres enum columns", () => { const status = postgresEnum("account_status", ["active", "disabled"]); - expect(status.column().ast.dataType).toBe("public.account_status"); + expect(status.column().ast.dataType).toBe('"public"."account_status"'); }); it("should infer readonly rows, inserts, patches, and primary keys", () => { diff --git a/src/schema.ts b/src/schema.ts index aa0b853..f2d968b 100644 --- a/src/schema.ts +++ b/src/schema.ts @@ -1,4 +1,4 @@ -import { toSnakeCase } from "./naming"; +import { quoteIdentifier, toSnakeCase } from "./naming"; /** Bidirectional converter between a column's stored (database) and application-facing value. */ export interface Codec { @@ -323,7 +323,7 @@ export function postgresEnum( name, schema, values, - column: () => column(`${schema}.${name}`), + column: () => column(`${quoteIdentifier(schema)}.${quoteIdentifier(name)}`), }; } diff --git a/src/tooling-impl.ts b/src/tooling-impl.ts index b34399c..a16f122 100644 --- a/src/tooling-impl.ts +++ b/src/tooling-impl.ts @@ -52,6 +52,7 @@ export interface SnapshotTable { readonly drop?: boolean; readonly columns: readonly SnapshotColumn[]; readonly constraints: readonly TableConstraint[]; + readonly primaryKeyColumns?: readonly string[]; } /** Point-in-time serialized shape of a database schema, produced by {@link snapshotDefinition}. */ @@ -152,6 +153,13 @@ export function snapshotDefinition( constraints: [...(table.$options.constraints ?? [])].sort((left, right) => JSON.stringify(left).localeCompare(JSON.stringify(right)), ), + ...(Object.values(table.$columns).filter((column) => column.ast.primaryKey).length > 1 + ? { + primaryKeyColumns: Object.values(table.$columns) + .filter((column) => column.ast.primaryKey) + .map((column) => column.ast.name), + } + : {}), })) .sort((left, right) => `${left.schema}.${left.name}`.localeCompare(`${right.schema}.${right.name}`), @@ -233,10 +241,29 @@ function indexName(table: SnapshotTable, index: IndexDefinition, position: numbe return index.name ?? `${table.name}_${position + 1}_idx`; } -function createTableSql(table: SnapshotTable, dialect: "postgres" | "sqlite" = "postgres"): string { +function createTableSql( + table: SnapshotTable, + dialect: "postgres" | "sqlite" = "postgres", + deferReferences = false, +): string { const inline = table.constraints.filter((entry) => entry.kind !== "index"); + const primary = + table.primaryKeyColumns ?? + table.columns.filter((column) => column.primaryKey).map((column) => column.name); const parts = [ - ...table.columns.filter((column) => !column.drop).map((column) => columnSql(column, dialect)), + ...table.columns + .filter((column) => !column.drop) + .map((column) => { + const { reference: _reference, ...withoutReference } = column; + return columnSql( + { + ...(deferReferences ? withoutReference : column), + ...(primary.length > 1 ? { primaryKey: false } : {}), + }, + dialect, + ); + }), + ...(primary.length > 1 ? [`PRIMARY KEY (${primary.map(quoteIdentifier).join(", ")})`] : []), ...inline.map((constraint) => constraintSql(table, constraint)), ]; const statements = [ @@ -264,6 +291,15 @@ function activeSnapshot(snapshot: SchemaSnapshot): SchemaSnapshot { }; } +function foreignKeysSql(table: SnapshotTable): string[] { + return table.columns + .filter((column) => !column.drop && column.reference) + .map((column) => { + const reference = column.reference!; + return `ALTER TABLE ${qualified(table.schema, table.name)} ADD FOREIGN KEY (${quoteIdentifier(column.name)}) REFERENCES ${qualified(reference.schema, reference.table)} (${quoteIdentifier(reference.column)});`; + }); +} + function renderInitialSchema( snapshot: SchemaSnapshot, dialect: "postgres" | "sqlite" = "postgres", @@ -284,7 +320,12 @@ function renderInitialSchema( ); } for (const table of snapshot.tables.filter((entry) => !entry.drop)) { - statements.push(createTableSql(table, dialect)); + statements.push(createTableSql(table, dialect, dialect === "postgres")); + } + if (dialect === "postgres") { + for (const table of snapshot.tables.filter((entry) => !entry.drop)) { + statements.push(...foreignKeysSql(table)); + } } for (const view of snapshot.views) { statements.push(`CREATE VIEW ${qualified(view.schema, view.name, dialect)} AS\n${view.query};`); @@ -312,6 +353,17 @@ export function diffSnapshots( return renderInitialSchema(desired, dialect); } const statements: string[] = []; + if (dialect === "postgres") { + const schemas = new Set( + [...desired.enums, ...desired.tables, ...desired.views].map((entry) => entry.schema), + ); + const existingSchemas = new Set( + [...current.enums, ...current.tables, ...current.views].map((entry) => entry.schema), + ); + for (const schema of [...schemas].sort()) + if (schema !== "public" && !existingSchemas.has(schema)) + statements.push(`CREATE SCHEMA IF NOT EXISTS ${quoteIdentifier(schema)};`); + } const currentEnums = byName(current.enums); const desiredEnums = byName(desired.enums); for (const value of desired.enums) { @@ -342,6 +394,14 @@ export function diffSnapshots( const currentTables = byName(current.tables); const desiredTables = byName(desired.tables); + const newTables = desired.tables.filter( + (table) => !currentTables.has(`${table.schema}.${table.name}`) && !table.renamedFrom, + ); + const foreignKeys: string[] = []; + for (const table of newTables) { + statements.push(createTableSql(table, dialect, dialect === "postgres")); + if (dialect === "postgres") foreignKeys.push(...foreignKeysSql(table)); + } for (const table of desired.tables) { const declaredTable = desiredInput.tables.find( (entry) => entry.schema === table.schema && entry.name === table.name, @@ -355,7 +415,6 @@ export function diffSnapshots( ); } if (!existing) { - statements.push(createTableSql(table, dialect)); continue; } const existingColumns = new Map(existing.columns.map((column) => [column.name, column])); @@ -379,9 +438,12 @@ export function diffSnapshots( `Adding required column ${table.name}.${column.name} needs a default or manual migration.`, ); } + const { reference: _reference, ...withoutReference } = column; statements.push( - `ALTER TABLE ${qualified(table.schema, table.name, dialect)} ADD COLUMN ${columnSql(column, dialect)};`, + `ALTER TABLE ${qualified(table.schema, table.name, dialect)} ADD COLUMN ${columnSql(dialect === "postgres" ? withoutReference : column, dialect)};`, ); + if (dialect === "postgres" && column.reference) + foreignKeys.push(...foreignKeysSql({ ...table, columns: [column] })); continue; } if (old.dataType !== column.dataType) { @@ -424,12 +486,14 @@ export function diffSnapshots( ); } } - if (JSON.stringify(existing.constraints) !== JSON.stringify(table.constraints)) { + const renamed = table.renamedFrom || table.columns.some((column) => column.renamedFrom); + if (!renamed && JSON.stringify(existing.constraints) !== JSON.stringify(table.constraints)) { throw new Error( `Changing constraints or indexes on ${table.name} requires a manual migration.`, ); } } + statements.push(...foreignKeys); for (const table of current.tables) { if (desiredTables.has(`${table.schema}.${table.name}`)) continue; const renamed = desiredInput.tables.some( @@ -617,6 +681,14 @@ async function describeRegisteredQueries( const parameters = [...query.parameters]; const compiled = query.compile(Object.fromEntries(parameters.map((name) => [name, null]))); const described = await scratch.describe(compiled.text, parameters); + const columnNames = new Set(); + for (const column of described.columns) { + if (columnNames.has(column.name)) + throw new Error( + `Registered query ${query.key} has duplicate output column ${column.name}; give each result column an explicit unique alias.`, + ); + columnNames.add(column.name); + } descriptions.push({ key: query.key, source: compiled.text, @@ -627,34 +699,44 @@ async function describeRegisteredQueries( return descriptions.sort((left, right) => left.key.localeCompare(right.key)); } -function tsType(dataType: string, nullable: boolean): string { +function tsType(dataType: string, nullable: boolean, dialect: "postgres" | "sqlite"): string { const normalized = dataType.toLowerCase(); const base = normalized === "boolean" ? "boolean" : /^(?:smallint|integer|real|double precision)$/.test(normalized) ? "number" - : normalized === "bigint" - ? "bigint" + : normalized === "bigint" || normalized.startsWith("numeric") + ? "string" : normalized === "bytea" ? "Uint8Array" - : /^(?:json|jsonb)$/.test(normalized) - ? "unknown" - : "string"; + : dialect === "postgres" && + /^(?:date|timestamp(?:\(\d+\))? (?:with|without) time zone)$/.test(normalized) + ? "Date" + : /^(?:json|jsonb)$/.test(normalized) + ? "unknown" + : /^(?:text|uuid|character(?: varying)?(?:\(\d+\))?|time(?:\(\d+\))? (?:with|without) time zone)$/.test( + normalized, + ) + ? "string" + : "unknown"; return nullable ? `${base} | null` : base; } -function renderQueries(descriptions: readonly KeyedSqlDescription[]): string { +function renderQueries( + descriptions: readonly KeyedSqlDescription[], + dialect: "postgres" | "sqlite", +): string { const lines = ["export interface GeneratedDatabaseQueries {"]; for (const query of descriptions) { lines.push(` readonly ${JSON.stringify(query.key)}: {`); lines.push(" readonly parameters: {"); - for (const parameter of query.parameters) + for (const parameter of new Set(query.parameters)) lines.push(` readonly ${JSON.stringify(parameter)}: unknown;`); lines.push(" };", " readonly row: {"); for (const column of query.columns) { lines.push( - ` readonly ${JSON.stringify(column.name)}: ${tsType(column.dataType, column.nullable)};`, + ` readonly ${JSON.stringify(column.name)}: ${tsType(column.dataType, column.nullable, dialect)};`, ); } lines.push(" };", " };"); @@ -664,7 +746,7 @@ function renderQueries(descriptions: readonly KeyedSqlDescription[]): string { } function generatedArtifacts( - _database: LoadedDatabase, + database: LoadedDatabase, snapshot: SchemaSnapshot, descriptions: readonly KeyedSqlDescription[], manifest: MigrationManifest, @@ -674,7 +756,7 @@ function generatedArtifacts( "// Generated by @askrjs/orm. Do not edit.", 'import type { GeneratedDatabaseArtifact } from "@askrjs/orm";', "", - renderQueries(descriptions).trimEnd(), + renderQueries(descriptions, database.definition.dialect).trimEnd(), "", `export const generated = ${JSON.stringify({ schemaIdentity: identity, manifest, queries: Object.fromEntries(descriptions.map((query) => [query.key, { parameters: query.parameters, columns: query.columns }])) }, null, 2)} as const satisfies GeneratedDatabaseArtifact;`, "", @@ -701,8 +783,8 @@ async function replay(database: LoadedDatabase): Promise<{ await scratch.close?.(); throw new Error("Scratch adapter identity does not match configured scratchIdentity."); } - const manifest = await readMigrations(database.databaseDir); try { + const manifest = await readMigrations(database.databaseDir); await scratch.reset(); for (const migration of manifest.migrations) await scratch.execute(migration.sql); const current = (await scratch.introspect()) as SchemaSnapshot; @@ -723,12 +805,116 @@ async function writeArtifacts( await fs.writeFile(path.join(database.databaseDir, GENERATED_FILE), artifacts, "utf8"); } +/** Let PostgreSQL normalize types/expressions; roll back all probe DDL before diffing. */ +async function desiredPhysicalSnapshot( + database: LoadedDatabase, + scratch: DatabaseToolingAdapter, + current: SchemaSnapshot, + declared: SchemaSnapshot, +): Promise { + if (database.definition.dialect !== "postgres") return declared; + await scratch.execute("BEGIN"); + let failed = false; + let primaryError: unknown; + let result: SchemaSnapshot | undefined; + try { + for (const view of current.views) + await scratch.execute(`DROP VIEW IF EXISTS ${qualified(view.schema, view.name)} CASCADE`); + for (const table of current.tables) + await scratch.execute(`DROP TABLE IF EXISTS ${qualified(table.schema, table.name)} CASCADE`); + for (const value of current.enums) + await scratch.execute(`DROP TYPE IF EXISTS ${qualified(value.schema, value.name)} CASCADE`); + await scratch.execute(renderInitialSchema(activeSnapshot(declared), "postgres")); + const physical = (await scratch.introspect()) as SchemaSnapshot; + result = { + ...physical, + tables: physical.tables.map((table) => { + const definition = declared.tables.find( + (entry) => entry.name === table.name && entry.schema === table.schema, + )!; + return { + ...table, + ...(definition.renamedFrom ? { renamedFrom: definition.renamedFrom } : {}), + columns: table.columns.map((column) => { + const original = definition.columns.find((entry) => entry.name === column.name)!; + return { + ...column, + ...(original.renamedFrom ? { renamedFrom: original.renamedFrom } : {}), + ...(original.convertUsing ? { convertUsing: original.convertUsing } : {}), + }; + }), + }; + }), + }; + } catch (error) { + failed = true; + primaryError = error; + } + try { + await scratch.execute("ROLLBACK"); + } catch (error) { + if (!failed) { + failed = true; + primaryError = error; + } + } + if (failed) throw primaryError; + return result!; +} + +/** Rename/conversion instructions are intent, never physical schema facts. */ +function comparisonSnapshot(snapshot: SchemaSnapshot, declared: SchemaSnapshot): SchemaSnapshot { + return { + ...snapshot, + tables: snapshot.tables.map((table) => { + const definition = declared.tables.find( + (entry) => + entry.schema === table.schema && + (entry.name === table.name || entry.renamedFrom === table.name), + ); + const explicitNames = new Set( + definition?.constraints + .map((constraint) => constraint.name) + .filter((name) => name !== undefined), + ); + return { + ...table, + constraints: table.constraints + .map((constraint) => { + const { name, ...unnamed } = constraint; + return name && explicitNames.has(name) ? constraint : unnamed; + }) + .sort((a, b) => JSON.stringify(a).localeCompare(JSON.stringify(b))), + }; + }), + }; +} + +function physicalIdentity(snapshot: SchemaSnapshot, declared: SchemaSnapshot): string { + const comparable = comparisonSnapshot(snapshot, declared); + return stableJson({ + ...activeSnapshot(comparable), + tables: activeSnapshot(comparable).tables.map(({ renamedFrom: _rename, ...table }) => ({ + ...table, + columns: table.columns.map( + ({ renamedFrom: _columnRename, convertUsing: _convert, ...column }) => column, + ), + })), + }); +} + async function validateOne(database: LoadedDatabase): Promise { const desired = activeSnapshot(snapshotDefinition(database.definition)); const replayed = await replay(database); try { const actual = activeSnapshot(replayed.current); - if (stableJson(actual) !== stableJson(desired)) { + const physical = await desiredPhysicalSnapshot( + database, + replayed.scratch, + replayed.current, + desired, + ); + if (physicalIdentity(actual, desired) !== physicalIdentity(physical, desired)) { throw new Error("Migration history does not produce the TypeScript schema definition."); } const descriptions = await describeRegisteredQueries(database.definition, replayed.scratch); @@ -758,7 +944,17 @@ async function generateOne(database: LoadedDatabase): Promise { const replayed = await replay(database); let created: string | null = null; try { - const delta = diffSnapshots(replayed.current, desired, database.definition.dialect); + const physical = await desiredPhysicalSnapshot( + database, + replayed.scratch, + replayed.current, + desired, + ); + const delta = diffSnapshots( + comparisonSnapshot(replayed.current, desired), + comparisonSnapshot(physical, desired), + database.definition.dialect, + ); if (delta) { const id = ulid(); const parent = replayed.manifest.migrations.at(-1)?.id ?? null; @@ -773,7 +969,7 @@ async function generateOne(database: LoadedDatabase): Promise { await replayed.scratch.execute(content); } const finalSnapshot = activeSnapshot((await replayed.scratch.introspect()) as SchemaSnapshot); - if (stableJson(finalSnapshot) !== stableJson(activeSnapshot(desired))) { + if (physicalIdentity(finalSnapshot, desired) !== physicalIdentity(physical, desired)) { throw new Error("Generated migration did not produce the desired schema."); } const manifest = await readMigrations(database.databaseDir); @@ -792,7 +988,13 @@ async function refreshGenerated(database: LoadedDatabase): Promise { const desired = activeSnapshot(snapshotDefinition(database.definition)); const replayed = await replay(database); try { - if (stableJson(activeSnapshot(replayed.current)) !== stableJson(desired)) { + const physical = await desiredPhysicalSnapshot( + database, + replayed.scratch, + replayed.current, + desired, + ); + if (physicalIdentity(replayed.current, desired) !== physicalIdentity(physical, desired)) { throw new Error( "Manual migration history does not produce the TypeScript schema definition.", ); diff --git a/src/tooling-integration.test.ts b/src/tooling-integration.test.ts new file mode 100644 index 0000000..2d90f05 --- /dev/null +++ b/src/tooling-integration.test.ts @@ -0,0 +1,457 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { Pool } from "pg"; +import { describe, expect, it } from "vitest"; +import { postgres } from "./postgres"; +import { runDatabaseCli, type SchemaSnapshot } from "./tooling-impl"; + +const targetUrl = process.env.ASKR_ORM_TEST_DATABASE_URL; +const shadowUrl = process.env.ASKR_ORM_TEST_SHADOW_URL; +const postgresTests = targetUrl && shadowUrl ? describe : describe.skip; + +postgresTests("real PostgreSQL tooling", () => { + it("refuses to reset the actual target through a different URL spelling", async () => { + const pool = new Pool({ connectionString: targetUrl }); + const alias = new URL(targetUrl!); + alias.searchParams.set("application_name", "askr-alias-guard"); + const scratch = await postgres({ url: targetUrl!, shadowUrl: alias.href }).shadow(); + try { + await pool.query("CREATE TABLE askr_tooling_target_guard (value text)"); + await pool.query("INSERT INTO askr_tooling_target_guard VALUES ('preserve me')"); + await expect(scratch.reset()).rejects.toThrow(/target|distinct|separate/i); + expect((await pool.query("SELECT value FROM askr_tooling_target_guard")).rows).toEqual([ + { value: "preserve me" }, + ]); + } finally { + await scratch.close?.(); + await pool.query("DROP TABLE IF EXISTS askr_tooling_target_guard"); + await pool.end(); + } + }); + + it("resets only scratch and returns a physical, deterministic schema snapshot", async () => { + const scratch = await postgres({ url: targetUrl!, shadowUrl: shadowUrl! }).shadow(); + try { + await scratch.reset(); + expect(await scratch.introspect()).toEqual({ version: 1, enums: [], tables: [], views: [] }); + await scratch.execute(` + CREATE SCHEMA "catalog #%"; + CREATE TYPE "catalog #%".status AS ENUM ('new', 'it''s ready'); + CREATE TABLE "catalog #%".users ( + id uuid DEFAULT gen_random_uuid() PRIMARY KEY, + email varchar(100) NOT NULL UNIQUE, + status "catalog #%".status, + slug text GENERATED ALWAYS AS (lower(email)) STORED, + CONSTRAINT positive_length CHECK (length(email) > 0) + ); + CREATE INDEX lower_email ON "catalog #%".users (lower(email)) WHERE status IS NOT NULL; + CREATE VIEW "catalog #%".emails AS SELECT email FROM "catalog #%".users; + `); + const snapshot = (await scratch.introspect()) as SchemaSnapshot; + expect(snapshot.version).toBe(1); + expect(snapshot.enums).toEqual([ + { schema: "catalog #%", name: "status", values: ["new", "it's ready"] }, + ]); + expect(snapshot.tables).toHaveLength(1); + expect(snapshot.tables[0]).toMatchObject({ schema: "catalog #%", name: "users" }); + expect(snapshot.tables[0]!.columns).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + name: "email", + dataType: "character varying(100)", + nullable: false, + }), + expect.objectContaining({ name: "id", primaryKey: true, default: "gen_random_uuid()" }), + expect.objectContaining({ name: "slug", generated: expect.any(String) }), + ]), + ); + expect(snapshot.tables[0]!.constraints).toEqual( + expect.arrayContaining([ + expect.objectContaining({ kind: "check", name: "positive_length" }), + expect.objectContaining({ + kind: "index", + name: "lower_email", + where: expect.any(String), + }), + expect.objectContaining({ kind: "unique", columns: ["email"] }), + ]), + ); + expect(snapshot.views).toEqual([ + expect.objectContaining({ name: "emails", query: expect.stringContaining("email") }), + ]); + expect(await scratch.introspect()).toEqual(snapshot); + await scratch.reset(); + expect(await scratch.introspect()).toEqual({ version: 1, enums: [], tables: [], views: [] }); + } finally { + await scratch.close?.(); + } + }); + + it("describes output and parameters without executing reads or writes, and recovers errors", async () => { + const scratch = await postgres({ + url: targetUrl!, + shadowUrl: shadowUrl!, + pool: { max: 1 }, + }).shadow(); + try { + await scratch.reset(); + await scratch.execute( + "CREATE TABLE users (id integer PRIMARY KEY, label text); CREATE SEQUENCE description_side_effect", + ); + const result = await scratch.describe("SELECT id, label, $1::text AS input FROM users", [ + "input", + ]); + expect(result).toEqual({ + parameters: ["input"], + columns: [ + { name: "id", dataType: "integer", nullable: true }, + { name: "label", dataType: "text", nullable: true }, + { name: "input", dataType: "text", nullable: true }, + ], + }); + expect( + await scratch.describe("SELECT nextval('description_side_effect') AS value", []), + ).toMatchObject({ columns: [{ name: "value", dataType: "bigint" }] }); + expect( + await scratch.describe("INSERT INTO users VALUES ($1::integer, 'ignored') RETURNING id", [ + "id", + ]), + ).toMatchObject({ columns: [{ name: "id", dataType: "integer" }] }); + await expect(scratch.describe("SELECT absent FROM users", [])).rejects.toThrow(); + await expect( + scratch.describe("SELECT 1; INSERT INTO users VALUES (1, 'batch must never execute')", []), + ).rejects.toThrow(/multiple commands/); + expect(await scratch.describe("SELECT id FROM users", [])).toMatchObject({ + columns: [{ name: "id" }], + }); + const observer = new Pool({ connectionString: shadowUrl }); + try { + expect((await observer.query("SELECT count(*)::integer AS count FROM users")).rows).toEqual( + [{ count: 0 }], + ); + expect( + (await observer.query("SELECT is_called FROM description_side_effect")).rows, + ).toEqual([{ is_called: false }]); + } finally { + await observer.end(); + } + } finally { + await scratch.close?.(); + } + }); + + it("generates, validates, no-op regenerates and rejects stale artifacts against real PostgreSQL", async () => { + const root = await fs.mkdtemp( + path.join(path.resolve(import.meta.dirname, ".."), ".orm-pg-fixture- #% "), + ); + const directory = path.join(root, "database"); + const messages: string[] = []; + const io = { + log: (v: unknown = "") => messages.push(String(v)), + error: (v: unknown = "") => messages.push(`ERROR ${v}`), + }; + const run = (command: string) => runDatabaseCli([command], { cwd: root, io }); + try { + await fs.mkdir(directory); + await fs.writeFile( + path.join(directory, "index.ts"), + ` +import { defineDatabase, defineQuery, table, text, uuid, integer } from "../../src/index.ts"; +import { postgres } from "../../src/postgres.ts"; +const users = table("users", { id: uuid().primaryKey().defaultRandom(), emailAddress: text().name("email").notNull().unique() }); +const events = table("events", { id: integer().primaryKey(), userId: uuid().name("user_id").references(() => users.id), payload: text() }); +const byEmail = defineQuery<{ email: string }>("users.by-email")\`SELECT id, email FROM users WHERE email = \${"email"}\`; +export default defineDatabase({ driver: postgres({ url: ${JSON.stringify(targetUrl)}, shadowUrl: ${JSON.stringify(shadowUrl)}, pool: { max: 1 } }), tables: { users, events }, queries: { byEmail } }); +`, + ); + expect(await run("generate"), messages.join("\n")).toBe(0); + const files = await fs.readdir(path.join(directory, "migrations")); + expect(files).toHaveLength(1); + const artifactPath = path.join(directory, "generated.ts"); + const generated = await fs.readFile(artifactPath, "utf8"); + expect(generated).toContain('readonly "id": string | null;'); + expect(generated).toContain('readonly "email": string | null;'); + expect(await run("validate"), messages.join("\n")).toBe(0); + const migrationPath = path.join(directory, "migrations", files[0]!); + const migration = await fs.readFile(migrationPath, "utf8"); + await fs.writeFile(migrationPath, `${migration}\nALTER TABLE users ADD COLUMN drift text;\n`); + expect(await run("validate")).toBe(1); + expect(messages.at(-1)).toMatch(/history does not produce/); + expect(await fs.readFile(artifactPath, "utf8")).toBe(generated); + await fs.writeFile(migrationPath, `${migration}\nSELECT 1 / 0;\n`); + expect(await run("generate")).toBe(1); + expect(messages.at(-1)).toMatch(/division by zero/); + expect(await fs.readFile(artifactPath, "utf8")).toBe(generated); + expect(await fs.readdir(path.join(directory, "migrations"))).toEqual(files); + await fs.writeFile(migrationPath, migration); + expect(await run("validate"), messages.join("\n")).toBe(0); + expect(await run("generate"), messages.join("\n")).toBe(0); + expect(messages.at(-1)).toBe("default: unchanged"); + expect(await fs.readdir(path.join(directory, "migrations"))).toEqual(files); + expect(await fs.readFile(artifactPath, "utf8")).toBe(generated); + await fs.writeFile(artifactPath, `${generated}// stale\n`); + expect(await run("validate")).toBe(1); + expect(messages.at(-1)).toMatch(/stale/); + await fs.writeFile(artifactPath, generated); + expect(await run("validate"), messages.join("\n")).toBe(0); + const entryPath = path.join(directory, "index.ts"); + const originalEntry = await fs.readFile(entryPath, "utf8"); + const expanded = originalEntry + .replace( + "export default defineDatabase", + ` +const zTags = table("z_tags", { id: integer().primaryKey(), label: text() }); +const aPosts = table("a_posts", { id: integer().primaryKey(), tagId: integer().references(() => zTags.id) }); +export default defineDatabase`, + ) + .replace("tables: { users, events }", "tables: { users, events, aPosts, zTags }"); + await fs.writeFile(entryPath, expanded); + expect(await run("generate"), messages.join("\n")).toBe(0); + expect(await run("validate"), messages.join("\n")).toBe(0); + expect(await run("generate"), messages.join("\n")).toBe(0); + expect(messages.at(-1)).toBe("default: unchanged"); + expect(await fs.readdir(path.join(directory, "migrations"))).toHaveLength(2); + const renamed = expanded + .replace('.name("email")', '.name("contact_email").renamedFrom("email")') + .replace( + "SELECT id, email FROM users WHERE email", + "SELECT id, contact_email FROM users WHERE contact_email", + ); + await fs.writeFile(entryPath, renamed); + expect(await run("generate"), messages.join("\n")).toBe(0); + expect(await run("validate"), messages.join("\n")).toBe(0); + expect(await run("generate"), messages.join("\n")).toBe(0); + expect(messages.at(-1)).toBe("default: unchanged"); + expect(await fs.readdir(path.join(directory, "migrations"))).toHaveLength(3); + const renamedArtifact = await fs.readFile(artifactPath, "utf8"); + await fs.writeFile(entryPath, renamed.replace("SELECT id, contact_email", "SELECT id, id")); + expect(await run("generate")).toBe(1); + expect(messages.at(-1)).toMatch(/duplicate output column/i); + expect(await fs.readFile(artifactPath, "utf8")).toBe(renamedArtifact); + await fs.writeFile(entryPath, renamed); + expect(await run("validate"), messages.join("\n")).toBe(0); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }, 30_000); + + it("qualifies composite keys, quoted enums, native expressions and query value types", async () => { + const root = await fs.mkdtemp( + path.join(path.resolve(import.meta.dirname, ".."), ".orm-pg-complex-"), + ); + const directory = path.join(root, "database"); + const messages: string[] = []; + const io = { + log: (v: unknown = "") => messages.push(String(v)), + error: (v: unknown = "") => messages.push(`ERROR ${v}`), + }; + try { + await fs.mkdir(directory); + await fs.writeFile( + path.join(directory, "index.ts"), + ` +import { defineDatabase, defineQuery, table, text, integer, numeric, bytes, index, check, view } from "../../src/index.ts"; +import { postgres, postgresEnum, postgresType } from "../../src/postgres.ts"; +const status = postgresEnum('status " key', ['new', "it's ready"], { schema: "catalog #%" }); +const entries = table("entries", { + tenantId: integer().primaryKey(), id: integer().primaryKey(), + state: status.column(), email: postgresType("varchar(80)").notNull(), + lowerEmail: text().generatedAlwaysAs("lower(email)"), amount: numeric(10, 2).default("0.00"), data: bytes(), +}, { schema: "catalog #%", constraints: [check("length(email) > 0", "email_nonempty"), index(["lower(email)"], { name: "lower_email", where: "state IS NOT NULL" })] }); +const emails = view("emails", 'SELECT email FROM "catalog #%".entries', { schema: "catalog #%" }); +const values = defineQuery>("values")\`SELECT 9007199254740993::bigint AS count, DATE '2026-10-09' AS day, ARRAY[1,2] AS numbers\`; +export default defineDatabase({ driver: postgres({ url: ${JSON.stringify(targetUrl)}, shadowUrl: ${JSON.stringify(shadowUrl)}, pool: { max: 1 } }), tables: { entries }, enums: [status], views: [emails], queries: { values } }); +`, + ); + for (const command of ["generate", "validate", "generate"]) { + expect(await runDatabaseCli([command], { cwd: root, io }), messages.join("\n")).toBe(0); + } + expect(messages.at(-1)).toBe("default: unchanged"); + expect(await fs.readdir(path.join(directory, "migrations"))).toHaveLength(1); + const artifact = await fs.readFile(path.join(directory, "generated.ts"), "utf8"); + expect(artifact).toContain('readonly "count": string | null;'); + expect(artifact).toContain('readonly "day": Date | null;'); + expect(artifact).toContain('readonly "numbers": unknown | null;'); + const observer = new Pool({ connectionString: shadowUrl }); + try { + const [row] = ( + await observer.query( + "SELECT 9007199254740993::bigint AS count, DATE '2026-10-09' AS day, ARRAY[1,2] AS numbers", + ) + ).rows; + expect(row.count).toBe("9007199254740993"); + expect(row.day).toBeInstanceOf(Date); + expect(row.numbers).toEqual([1, 2]); + } finally { + await observer.end(); + } + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }, 30_000); + + it("fails closed if target cannot be reached and preserves existing scratch data", async () => { + const unreachable = new URL(targetUrl!); + unreachable.port = "1"; + const observer = new Pool({ connectionString: shadowUrl }); + const scratch = await postgres({ + url: unreachable.href, + shadowUrl: shadowUrl!, + pool: { connectionTimeoutMillis: 200 }, + }).shadow(); + try { + await observer.query( + "CREATE TABLE reset_guard (value text); INSERT INTO reset_guard VALUES ('preserve')", + ); + await expect(scratch.reset()).rejects.toThrow(); + expect((await observer.query("SELECT value FROM reset_guard")).rows).toEqual([ + { value: "preserve" }, + ]); + } finally { + await scratch.close?.(); + await observer.query("DROP TABLE IF EXISTS reset_guard"); + await observer.end(); + } + }); + + it("serializes concurrent descriptions on one pooled connection and releases it on close", async () => { + const scratch = await postgres({ + url: targetUrl!, + shadowUrl: shadowUrl!, + pool: { max: 1 }, + }).shadow(); + try { + const results = await Promise.all( + Array.from({ length: 12 }, () => + scratch.describe("SELECT $1::integer AS value", ["value"]), + ), + ); + for (const result of results) + expect(result).toEqual({ + parameters: ["value"], + columns: [{ name: "value", dataType: "integer", nullable: true }], + }); + } finally { + await scratch.close?.(); + } + await scratch.close?.(); + await expect(scratch.introspect()).rejects.toThrow(/closed/); + }); + + it("rolls back failed reset and preserves its error and existing scratch data", async () => { + const observer = new Pool({ connectionString: shadowUrl }); + const scratch = await postgres({ + url: targetUrl!, + shadowUrl: shadowUrl!, + pool: { options: "-c default_transaction_read_only=on" }, + }).shadow(); + try { + await observer.query( + "CREATE TABLE readonly_reset_guard (value text); INSERT INTO readonly_reset_guard VALUES ('preserve')", + ); + await expect(scratch.reset()).rejects.toThrow(/read.only/i); + expect((await observer.query("SELECT value FROM readonly_reset_guard")).rows).toEqual([ + { value: "preserve" }, + ]); + } finally { + await scratch.close?.(); + await observer.query("DROP TABLE IF EXISTS readonly_reset_guard"); + await observer.end(); + } + }); + + it("holds the scratch workflow lock until close, then lets another owner reset", async () => { + const first = await postgres({ + url: targetUrl!, + shadowUrl: shadowUrl!, + pool: { max: 1 }, + }).shadow(); + const second = await postgres({ + url: targetUrl!, + shadowUrl: shadowUrl!, + pool: { max: 1 }, + }).shadow(); + const observer = new Pool({ connectionString: shadowUrl }); + let reset: Promise | undefined; + try { + await first.reset(); + await first.execute("CREATE TABLE workflow_guard (value text)"); + reset = second.reset(); + let waiting = false; + for (let attempt = 0; attempt < 100; attempt++) { + const result = await observer.query( + "SELECT 1 FROM pg_locks WHERE locktype = 'advisory' AND NOT granted AND database = (SELECT oid FROM pg_database WHERE datname = current_database())", + ); + if (result.rows.length) { + waiting = true; + break; + } + await new Promise((resolve) => setTimeout(resolve, 10)); + } + expect(waiting).toBe(true); + expect( + (await observer.query("SELECT to_regclass('workflow_guard') AS relation")).rows[0].relation, + ).toBe("workflow_guard"); + await first.close?.(); + await reset; + expect(await second.introspect()).toEqual({ version: 1, enums: [], tables: [], views: [] }); + } finally { + await first.close?.(); + await reset?.catch(() => undefined); + await second.close?.(); + await observer.end(); + } + }); + + it.each([ + [ + "identity", + "CREATE TABLE unsupported (id integer GENERATED ALWAYS AS IDENTITY)", + /Identity column/, + ], + ["materialized view", "CREATE MATERIALIZED VIEW unsupported AS SELECT 1 AS id", /kind m/], + [ + "foreign key action", + "CREATE TABLE parent (id integer PRIMARY KEY); CREATE TABLE unsupported (id integer REFERENCES parent ON DELETE CASCADE)", + /Foreign key.*cannot be represented/, + ], + [ + "included index", + "CREATE TABLE unsupported (id integer, label text); CREATE INDEX included ON unsupported (id) INCLUDE (label)", + /covering index/, + ], + [ + "unvalidated check", + "CREATE TABLE unsupported (id integer); ALTER TABLE unsupported ADD CONSTRAINT positive CHECK (id > 0) NOT VALID", + /unvalidated constraint/, + ], + [ + "nulls not distinct", + "CREATE TABLE unsupported (id integer UNIQUE NULLS NOT DISTINCT)", + /cannot be represented/, + ], + [ + "inheritance", + "CREATE TABLE parent (id integer); CREATE TABLE unsupported () INHERITS (parent)", + /cannot be represented/, + ], + ["unlogged table", "CREATE UNLOGGED TABLE unsupported (id integer)", /cannot be represented/], + [ + "nondefault collation", + 'CREATE TABLE unsupported (id text COLLATE "C")', + /cannot be represented/, + ], + ])( + "rejects unsupported %s rather than accepting an incomplete schema", + async (_name, sql, error) => { + const scratch = await postgres({ url: targetUrl!, shadowUrl: shadowUrl! }).shadow(); + try { + await scratch.reset(); + await scratch.execute(sql); + await expect(scratch.introspect()).rejects.toThrow(error); + } finally { + await scratch.close?.(); + } + }, + ); +}); diff --git a/tests/packed-contract.mjs b/tests/packed-contract.mjs index 848aec6..332661b 100644 --- a/tests/packed-contract.mjs +++ b/tests/packed-contract.mjs @@ -100,6 +100,42 @@ try { consumer, ); const fixture = path.join(consumer, "fixture.ts"); + const postgresWorkflow = Boolean( + process.env.ASKR_ORM_TEST_DATABASE_URL && process.env.ASKR_ORM_TEST_SHADOW_URL, + ); + if (postgresWorkflow) { + await fs.mkdir(path.join(consumer, "database")); + await fs.writeFile( + path.join(consumer, "database/index.ts"), + ` +import { defineDatabase, defineQuery, table, text, uuid } from '@askrjs/orm'; +import { postgres } from '@askrjs/orm/postgres'; +const users = table('users', { id: uuid().primaryKey().defaultRandom(), emailAddress: text().name('email').notNull().unique() }); +const byEmail = defineQuery<{email:string}>('users.by-email')\`SELECT id, email FROM users WHERE email = \${'email'} OR email = \${'email'}\`; +export default defineDatabase({ driver: postgres({ url: process.env.ASKR_ORM_TEST_DATABASE_URL!, shadowUrl: process.env.ASKR_ORM_TEST_SHADOW_URL!, pool: {max:1} }), tables: {users}, queries: {byEmail} }); +`, + ); + await fs.writeFile( + path.join(consumer, "postgres-workflow.mjs"), + ` +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import {runDatabaseCli} from '@askrjs/orm/tooling'; +const messages=[]; +const io={log:(v='')=>messages.push(String(v)), error:(v='')=>messages.push(String(v))}; +for (const command of ['generate','validate','generate']) assert.equal(await runDatabaseCli([command], {cwd:process.cwd(),io}), 0, messages.join('\\n')); +assert.equal(messages.at(-1),'default: unchanged'); +assert.equal((await fs.readdir('database/migrations')).length,1); +const generated=await fs.readFile('database/generated.ts','utf8'); +assert(generated.includes('readonly "id": string | null;')); +assert(generated.includes('readonly "email": string | null;')); +`, + ); + execFileSync(process.execPath, [path.join(consumer, "postgres-workflow.mjs")], { + cwd: consumer, + stdio: "pipe", + }); + } const imports = Object.entries(contract.entrypoints) .map( ([key]) => @@ -134,13 +170,13 @@ void [generated, codec, postgresOptions, invalidPool, sqliteOptions];\n`, types: ["node"], skipLibCheck: false, }, - files: ["fixture.ts"], + files: ["fixture.ts", ...(postgresWorkflow ? ["database/generated.ts"] : [])], }), ); execFileSync( process.execPath, [path.join(root, "node_modules/typescript/bin/tsc"), "--project", "tsconfig.json"], - { cwd: consumer, stdio: "pipe" }, + { cwd: consumer, stdio: "pipe", encoding: "utf8" }, ); const options = { strict: true, @@ -151,7 +187,10 @@ void [generated, codec, postgresOptions, invalidPool, sqliteOptions];\n`, types: ["node"], skipLibCheck: false, }; - const program = ts.createProgram([fixture], options); + const program = ts.createProgram( + [fixture, ...(postgresWorkflow ? [path.join(consumer, "database/generated.ts")] : [])], + options, + ); const diagnostics = ts.getPreEmitDiagnostics(program); assert.equal( diagnostics.length, @@ -187,6 +226,9 @@ void [generated, codec, postgresOptions, invalidPool, sqliteOptions];\n`, removed: Object.values(contract.entrypoints).reduce((n, c) => n + c.removed.length, 0), privateSubpaths: contract.privateSubpaths.length, postgresPeers: ["8.23.0", "4.17.0"], + postgresWorkflow: postgresWorkflow + ? "installed generate, validate, unchanged generate; TS6/7 artifact" + : "not run without explicit test databases", sqliteScript: "complete, rollback, subsequent query", }), ); diff --git a/tests/postgres-test-env.mjs b/tests/postgres-test-env.mjs new file mode 100644 index 0000000..377aa72 --- /dev/null +++ b/tests/postgres-test-env.mjs @@ -0,0 +1,4 @@ +if (!process.env.ASKR_ORM_TEST_DATABASE_URL || !process.env.ASKR_ORM_TEST_SHADOW_URL) + throw new Error( + "PostgreSQL integration requires explicit isolated target and disposable scratch URLs in ASKR_ORM_TEST_DATABASE_URL and ASKR_ORM_TEST_SHADOW_URL.", + ); diff --git a/tests/prepare-postgres.mjs b/tests/prepare-postgres.mjs new file mode 100644 index 0000000..91dd2bd --- /dev/null +++ b/tests/prepare-postgres.mjs @@ -0,0 +1,24 @@ +import { Pool } from "pg"; + +// CI's disposable PostgreSQL service owns these two fixed test databases. +if ( + !process.env.CI || + !process.env.ASKR_ORM_TEST_DATABASE_URL || + !process.env.ASKR_ORM_TEST_SHADOW_URL +) + throw new Error("Scratch database provisioning is restricted to the disposable CI service."); +const target = new URL(process.env.ASKR_ORM_TEST_DATABASE_URL); +const shadow = new URL(process.env.ASKR_ORM_TEST_SHADOW_URL); +if ( + target.hostname !== "127.0.0.1" || + target.pathname !== "/askr_orm_test" || + shadow.hostname !== "127.0.0.1" || + shadow.pathname !== "/askr_orm_test_shadow" +) + throw new Error("Provisioning requires the fixed loopback CI test databases."); +const pool = new Pool({ connectionString: process.env.ASKR_ORM_TEST_DATABASE_URL }); +try { + await pool.query('CREATE DATABASE "askr_orm_test_shadow"'); +} finally { + await pool.end(); +} diff --git a/tests/public-contract.json b/tests/public-contract.json index 5add462..f86c658 100644 --- a/tests/public-contract.json +++ b/tests/public-contract.json @@ -162,6 +162,8 @@ "sql", "tooling-impl", "dist/index.js", - "dist/tooling-impl.js" + "dist/tooling-impl.js", + "postgres-client", + "postgres-tooling" ] }