diff --git a/.agents/skills/constructive-cli.zip b/.agents/skills/constructive-cli.zip new file mode 100644 index 000000000..fe4a1b93b Binary files /dev/null and b/.agents/skills/constructive-cli.zip differ diff --git a/.agents/skills/constructive-cli/SKILL.md b/.agents/skills/constructive-cli/SKILL.md new file mode 100644 index 000000000..c894b8c3c --- /dev/null +++ b/.agents/skills/constructive-cli/SKILL.md @@ -0,0 +1,124 @@ +--- +name: constructive-cli +description: "Generated CLI commands and scaffolding — how the CLI is generated from GraphQL schemas, how to use it, codegen options, multi-target unified CLI, and the CLI reference. Use when asked to 'generate a CLI', 'create CLI commands', 'build a command-line client', 'run generated CLI', or when working with @constructive-io/graphql-codegen CLI output." +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Constructive CLI (Generated) + +The Constructive codegen pipeline generates interactive command-line interfaces from GraphQL schemas using inquirerer. The generated CLI provides CRUD commands for each table and custom operations, plus built-in infrastructure commands for authentication and context management. + +## When to Apply + +Use this skill when: +- Generating a CLI tool from a GraphQL schema +- Running or customizing the generated CLI +- Understanding the CLI codegen pipeline +- Building internal tooling or admin scripts from GraphQL APIs +- Configuring multi-target unified CLI output + +## How the CLI Is Generated + +The CLI is generated by `@constructive-io/graphql-codegen` alongside the ORM client: + +```typescript +import { generate } from '@constructive-io/graphql-codegen'; + +await generate({ + schemaFile: './schemas/public.graphql', + output: './generated', + cli: true, // ORM is auto-generated alongside CLI +}); +``` + +### CLI Configuration Options + +```typescript +cli: { + toolName: 'myapp', // Config stored at ~/.myapp/ via appstash + entryPoint: true, // Generate runnable index.ts + builtinNames: { + auth: 'credentials', // Rename infrastructure commands + context: 'env', + }, +} +``` + +## Output Structure + +``` +{output}/cli/ +├── index.ts # Entry point (only if entryPoint: true) +├── executor.ts # CLI executor with command routing +├── command-map.ts # Map of all commands +├── context.ts # Infrastructure: context management +├── auth.ts # Infrastructure: auth/credentials +├── utils.ts # Shared CLI utilities +└── commands/ + ├── users.ts # Generated CRUD commands per table + └── ... +``` + +## Running the Generated CLI + +```bash +# With entry point +npx ts-node generated/cli/index.ts + +# Or compile and run +npx tsc && node dist/generated/cli/index.js +``` + +## Built-in Infrastructure Commands + +### context — Manage API endpoints + +```bash +myapp context create production --endpoint https://api.example.com/graphql +myapp context list +myapp context use production +myapp context current +``` + +### auth — Manage API credentials + +```bash +myapp auth set-token +myapp auth status +myapp auth logout +``` + +### Builtin Name Collision Handling + +If a table name collides with `auth` or `context`, the infrastructure command is automatically renamed (`auth` → `credentials`, `context` → `env`). Override with `builtinNames`. + +## Multi-Target CLI (Unified) + +Combine all targets into a single CLI with namespaced commands: + +```typescript +import { generateMulti } from '@constructive-io/graphql-codegen'; + +await generateMulti({ + configs: { + public: { schemaFile: './schemas/public.graphql', output: './generated/public', cli: true }, + admin: { schemaFile: './schemas/admin.graphql', output: './generated/admin', cli: true }, + }, + unifiedCli: { toolName: 'myapp', entryPoint: true }, +}); +// Commands: myapp public users list, myapp admin users list +``` + +## Reference Guide + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [references/codegen-cli-reference.md](references/codegen-cli-reference.md) | Full CLI codegen reference | Looking up CLI flags, source options, environment variables | + +## Cross-References + +- `constructive-sdk-graphql` skill (in constructive-skills) — Full codegen pipeline (hooks, ORM, CLI generation) +- `pgpm` skill — Database migrations (deploy before generating CLI) +- `constructive-setup` skill — Monorepo setup and local development diff --git a/.agents/skills/constructive-cli/references/codegen-cli-reference.md b/.agents/skills/constructive-cli/references/codegen-cli-reference.md new file mode 100644 index 000000000..fc13628fd --- /dev/null +++ b/.agents/skills/constructive-cli/references/codegen-cli-reference.md @@ -0,0 +1,111 @@ +# CLI Codegen Reference + +Complete reference for `@constructive-io/graphql-codegen` CLI commands. + +## @constructive-io/graphql-codegen generate + +Generate type-safe React Query hooks and/or ORM client from GraphQL schema. + +```bash +npx @constructive-io/graphql-codegen generate [options] +``` + +### Source Options (choose one) + +| Option | Alias | Description | Default | +|--------|-------|-------------|---------| +| `--endpoint ` | `-e` | GraphQL endpoint URL | - | +| `--schema-file ` | `-s` | Path to GraphQL schema file (.graphql) | - | +| `--schemas ` | - | PostgreSQL schemas (comma-separated) | - | +| `--api-names ` | - | API names for auto schema discovery | - | +| `--config ` | `-c` | Path to config file | `graphql-codegen.config.ts` | + +### Generator Options + +| Option | Description | Default | +|--------|-------------|---------| +| `--react-query` | Generate React Query hooks | `false` | +| `--orm` | Generate ORM client | `false` | + +### Output Options + +| Option | Alias | Description | Default | +|--------|-------|-------------|---------| +| `--output ` | `-o` | Output directory | `./generated/graphql` | +| `--target ` | `-t` | Target name (for multi-target configs) | - | + +### Schema Export Options + +| Option | Description | Default | +|--------|-------------|---------| +| `--schema-enabled` | Export GraphQL SDL schema file | `false` | +| `--schema-output ` | Output directory for exported schema | Same as `--output` | +| `--schema-filename ` | Filename for exported schema | `schema.graphql` | + +### Other Options + +| Option | Alias | Description | Default | +|--------|-------|-------------|---------| +| `--authorization ` | `-a` | Authorization header value | - | +| `--verbose` | `-v` | Show detailed output | `false` | +| `--dry-run` | - | Preview without writing files | `false` | + +## Examples + +### From GraphQL Endpoint + +```bash +npx @constructive-io/graphql-codegen generate --react-query --endpoint https://api.example.com/graphql +npx @constructive-io/graphql-codegen generate --orm --endpoint https://api.example.com/graphql +npx @constructive-io/graphql-codegen generate --react-query --orm --endpoint https://api.example.com/graphql +``` + +### From Schema File + +```bash +npx @constructive-io/graphql-codegen generate --react-query --schema-file ./schema.graphql +``` + +### From Database + +```bash +npx @constructive-io/graphql-codegen generate --react-query --schemas public,app_public +npx @constructive-io/graphql-codegen generate --orm --api-names my_api +``` + +### Using Config File + +```bash +npx @constructive-io/graphql-codegen generate +npx @constructive-io/graphql-codegen generate --config ./config/codegen.config.ts +npx @constructive-io/graphql-codegen generate --target production # Multi-target +``` + +## Environment Variables + +| Variable | Description | +|----------|-------------| +| `PGHOST` | PostgreSQL host (for database introspection) | +| `PGPORT` | PostgreSQL port | +| `PGDATABASE` | PostgreSQL database name | +| `PGUSER` | PostgreSQL user | +| `PGPASSWORD` | PostgreSQL password | + +## Exit Codes + +| Code | Meaning | +|------|---------| +| `0` | Success | +| `1` | General error | +| `2` | Configuration error | +| `3` | Network/schema error | + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| No code generated | Add `--react-query` or `--orm` flag | +| "Cannot use both endpoint and schemas" | Choose one schema source | +| No CLI generated | Add `cli: true` to generate options | +| Auth errors | Run `{toolName} auth set-token ` | +| Wrong endpoint | Run `{toolName} context use ` | diff --git a/.agents/skills/constructive-pnpm.zip b/.agents/skills/constructive-pnpm.zip new file mode 100644 index 000000000..927ee03cb Binary files /dev/null and b/.agents/skills/constructive-pnpm.zip differ diff --git a/.agents/skills/constructive-pnpm/SKILL.md b/.agents/skills/constructive-pnpm/SKILL.md new file mode 100644 index 000000000..29e932be6 --- /dev/null +++ b/.agents/skills/constructive-pnpm/SKILL.md @@ -0,0 +1,62 @@ +--- +name: constructive-pnpm +description: "PNPM workspace management — monorepo configuration, dist-folder publishing with makage/lerna, dependency management, deep nested imports, and anti-patterns to avoid. Use when configuring pnpm workspaces, publishing packages to npm, managing monorepo dependencies, or setting up new TypeScript packages." +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Constructive PNPM + +PNPM workspace management, package publishing, and monorepo best practices for Constructive projects. + +## When to Apply + +Use this skill when: +- Configuring pnpm workspaces and monorepo settings +- Publishing TypeScript packages to npm with makage and lerna +- Managing workspace dependencies and cross-package references +- Setting up the dist-folder publishing pattern +- Understanding deep nested imports vs `exports` map (anti-pattern) + +## Key Concepts + +### Dist-Folder Publishing (TypeScript/JS Only) + +Constructive publishes **TypeScript/JS packages** from the `dist/` folder, which becomes the package root on npm. This means: +- `main: "index.js"` points to `dist/index.js` after publish +- Consumers never see `dist/` in their import paths +- No `exports` field needed — standard Node.js resolution works +- Uses `makage build` to compile TypeScript and copy assets to `dist/` +- `publishConfig.directory: "dist"` in package.json + +**This does NOT apply to pgpm SQL modules.** PGPM modules publish from the package root (no `dist/` folder, no makage). They use `pgpm package` to bundle SQL files instead. See the `pgpm` skill ([publishing.md](../pgpm/references/publishing.md)) for the SQL module publishing workflow. + +### Deep Nested Imports (NOT `exports` Map) + +**NEVER use the `exports` map pattern.** Instead, use deep nested imports via file paths: + +```typescript +// Correct: deep nested import (works because dist/ becomes package root) +import { OrmClient } from '@my-org/sdk/api'; +import { generateOrm } from '@constructive-io/graphql-codegen/core/codegen/orm'; + +// WRONG: exports map anti-pattern +// "exports": { "./api": { "import": "./dist/api/index.js" } } +``` + +See [pnpm-publishing.md](./references/pnpm-publishing.md) for the full explanation. + +## Reference Guide + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [pnpm-workspace.md](./references/pnpm-workspace.md) | pnpm workspace overview | Setting up monorepo, workspace configuration | +| [pnpm-monorepo-management.md](./references/pnpm-monorepo-management.md) | Monorepo management | Cross-package dependencies, filtering, CI/CD patterns | +| [pnpm-publishing.md](./references/pnpm-publishing.md) | Publishing packages | Lerna versioning, makage builds, dist-folder pattern | + +## Cross-References + +- `inquirerer-cli` — CLI building with inquirerer, README formatting +- `pgpm` — PGPM workspaces for SQL modules (different from pnpm workspaces) +- `constructive-platform` — Platform core, environment configuration diff --git a/.agents/skills/constructive-pnpm/references/pnpm-monorepo-management.md b/.agents/skills/constructive-pnpm/references/pnpm-monorepo-management.md new file mode 100644 index 000000000..848b7f1c5 --- /dev/null +++ b/.agents/skills/constructive-pnpm/references/pnpm-monorepo-management.md @@ -0,0 +1,401 @@ +--- +name: monorepo-management +description: Best practices for managing large PNPM monorepos. Use when asked to "manage monorepo", "organize packages", "configure workspace dependencies", or when scaling a multi-package repository. +compatibility: pnpm, lerna, Node.js 18+, TypeScript +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Monorepo Management with PNPM + +Best practices for managing large PNPM monorepos following Constructive conventions. Covers workspace configuration, dependency management, selective builds, and package organization. + +## When to Apply + +Use this skill when: +- Managing a multi-package repository +- Configuring workspace dependencies +- Setting up selective builds and filtering +- Organizing packages in a large codebase +- Configuring lerna versioning strategies + +## Workspace Configuration + +### pnpm-workspace.yaml + +Define package locations: + +```yaml +packages: + - 'packages/*' +``` + +For larger projects with multiple directories: + +```yaml +packages: + - 'packages/*' + - 'pgpm/*' + - 'graphql/*' + - 'postgres/*' + - 'apps/*' + - 'libs/*' +``` + +### Root package.json + +```json +{ + "name": "my-workspace", + "version": "0.0.1", + "private": true, + "scripts": { + "build": "pnpm -r run build", + "test": "pnpm -r run test", + "lint": "pnpm -r run lint", + "clean": "pnpm -r run clean", + "deps": "pnpm up -r -i -L" + } +} +``` + +Key points: +- Root is always `private: true` +- Scripts use `pnpm -r` for recursive execution +- `deps` script for interactive dependency updates + +## Workspace Dependencies + +### workspace:* Protocol + +Reference internal packages: + +```json +{ + "dependencies": { + "my-utils": "workspace:*" + } +} +``` + +When published, `workspace:*` is replaced with the actual version. + +### Dependency Types + +| Protocol | Behavior | +|----------|----------| +| `workspace:*` | Latest version in workspace | +| `workspace:^` | Compatible version range | +| `workspace:~` | Patch version range | + +### Adding Workspace Dependencies + +```bash +# Add workspace dependency +pnpm add my-utils --filter my-app --workspace + +# Add external dependency to specific package +pnpm add lodash --filter my-app + +# Add dev dependency to root +pnpm add -D typescript -w +``` + +## Filtering and Selective Builds + +### --filter Flag + +Run commands on specific packages: + +```bash +# Single package +pnpm --filter my-app run build + +# Multiple packages +pnpm --filter my-app --filter my-utils run build + +# Glob patterns +pnpm --filter "my-*" run build + +# Package and its dependencies +pnpm --filter my-app... run build + +# Package and its dependents +pnpm --filter ...my-utils run build + +# Exclude packages +pnpm --filter "!my-legacy" run build +``` + +### Dependency-Aware Builds + +```bash +# Build package and all its dependencies +pnpm --filter my-app... run build + +# Build only dependencies (not the package itself) +pnpm --filter "my-app^..." run build + +# Build dependents (packages that depend on this) +pnpm --filter "...my-utils" run build +``` + +### Changed Packages + +```bash +# Packages changed since main +pnpm --filter "...[origin/main]" run build + +# Packages changed in last commit +pnpm --filter "...[HEAD~1]" run build +``` + +## Package Organization + +### Directory Structure + +Organize by domain or type: + +``` +my-workspace/ +├── packages/ # Shared utilities +│ ├── utils/ +│ ├── types/ +│ └── config/ +├── apps/ # Applications +│ ├── web/ +│ └── api/ +├── libs/ # Domain libraries +│ ├── auth/ +│ └── database/ +├── pgpm/ # PGPM modules (if hybrid) +│ └── migrations/ +├── pnpm-workspace.yaml +├── lerna.json +└── package.json +``` + +### Naming Conventions + +| Type | Convention | Example | +|------|------------|---------| +| Scoped packages | `@org/package-name` | `@myorg/utils` | +| Internal packages | `package-name` | `my-utils` | +| Apps | `app-name` | `web-app` | + +## Lerna Configuration + +### Independent Versioning + +Each package versioned separately: + +```json +{ + "$schema": "node_modules/lerna/schemas/lerna-schema.json", + "version": "independent", + "npmClient": "pnpm", + "command": { + "publish": { + "allowBranch": "main", + "conventionalCommits": true + } + } +} +``` + +Best for: utility libraries with different release cycles. + +### Fixed Versioning + +All packages share same version: + +```json +{ + "$schema": "node_modules/lerna/schemas/lerna-schema.json", + "version": "1.0.0", + "npmClient": "pnpm", + "command": { + "publish": { + "allowBranch": "main", + "conventionalCommits": true + } + } +} +``` + +Best for: tightly coupled packages that release together. + +### Versioning Commands + +```bash +# Version changed packages +pnpm lerna version + +# Version with specific bump +pnpm lerna version patch +pnpm lerna version minor +pnpm lerna version major + +# Preview without changes +pnpm lerna version --no-git-tag-version --no-push +``` + +## Dependency Management + +### Update Dependencies + +```bash +# Interactive update all packages +pnpm up -r -i -L + +# Update specific dependency everywhere +pnpm up lodash -r + +# Update to latest +pnpm up lodash@latest -r +``` + +### Check for Outdated + +```bash +pnpm outdated -r +``` + +### Dedupe Dependencies + +```bash +pnpm dedupe +``` + +## Build Optimization + +### Parallel Builds + +PNPM runs in parallel by default. Control concurrency: + +```bash +# Limit concurrent tasks +pnpm -r --workspace-concurrency=4 run build +``` + +### Topological Order + +Dependencies are built first automatically: + +```bash +# Builds in dependency order +pnpm -r run build +``` + +### Caching + +Use turbo or nx for build caching in large repos: + +```json +{ + "scripts": { + "build": "turbo run build" + } +} +``` + +## CI/CD Patterns + +### Install Dependencies + +```bash +pnpm install --frozen-lockfile +``` + +### Build Changed Packages + +```bash +# Build packages changed since main +pnpm --filter "...[origin/main]" run build +``` + +### Test Changed Packages + +```bash +pnpm --filter "...[origin/main]" run test +``` + +### Publish Workflow + +```bash +# Version and publish +pnpm lerna version --yes +pnpm lerna publish from-package --yes +``` + +## Common Commands Reference + +| Command | Description | +|---------|-------------| +| `pnpm install` | Install all dependencies | +| `pnpm -r run build` | Build all packages | +| `pnpm -r run test` | Test all packages | +| `pnpm --filter run ` | Run command in specific package | +| `pnpm --filter ... run ` | Run in package and dependencies | +| `pnpm add --filter ` | Add dependency to package | +| `pnpm add -w` | Add dependency to root | +| `pnpm up -r -i -L` | Interactive dependency update | +| `pnpm lerna version` | Version packages | +| `pnpm lerna publish` | Publish packages | + +## Hybrid Workspaces + +Some repos (like Constructive) have both PNPM packages and PGPM modules: + +```yaml +# pnpm-workspace.yaml +packages: + - 'packages/*' # TypeScript packages + - 'pgpm/*' # PGPM CLI and tools + - 'postgres/*' # PostgreSQL utilities +``` + +The root may also have a `pgpm.json` for SQL module configuration. + +## Troubleshooting + +### Dependency Resolution Issues + +```bash +# Clear cache and reinstall +pnpm store prune +rm -rf node_modules +pnpm install +``` + +### Workspace Link Issues + +```bash +# Check workspace links +pnpm why +``` + +### Build Order Issues + +```bash +# Verify dependency graph +pnpm list -r --depth=0 +``` + +## Best Practices + +1. **Keep root private**: Never publish the root package +2. **Use workspace protocol**: Always `workspace:*` for internal deps +3. **Consistent structure**: Same directory layout across packages +4. **Shared config**: Extend root tsconfig.json, eslint.config.js +5. **Filter in CI**: Only build/test changed packages +6. **Lock file**: Always commit pnpm-lock.yaml +7. **Dedupe regularly**: Run `pnpm dedupe` periodically +8. **Document dependencies**: Clear README for each package + +## References + +- Related skill: `pnpm-workspace` for workspace setup +- Related skill: `pnpm-publishing` for publishing workflow +- Related skill: `pgpm` (`references/workspace.md`) for SQL module workspaces diff --git a/.agents/skills/constructive-pnpm/references/pnpm-publishing.md b/.agents/skills/constructive-pnpm/references/pnpm-publishing.md new file mode 100644 index 000000000..cff610e34 --- /dev/null +++ b/.agents/skills/constructive-pnpm/references/pnpm-publishing.md @@ -0,0 +1,416 @@ +--- +name: pnpm-publishing +description: Publish TypeScript packages using makage and lerna following Constructive standards. Use when asked to "publish a package", "release to npm", "build for publishing", or when preparing packages for npm distribution. +compatibility: pnpm, makage, lerna, Node.js 18+ +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Publishing TypeScript Packages (Constructive Standard) + +Publish TypeScript packages to npm using makage for builds and lerna for versioning. This covers the dist-folder publishing pattern that prevents tree-shaking into weird import paths. + +## When to Apply + +Use this skill when: +- Building TypeScript packages for npm publishing +- Configuring makage for package builds +- Running lerna version and publish workflows +- Setting up the dist-folder publishing pattern + +## Why Dist-Folder Publishing? + +Constructive publishes from the `dist/` folder to: +- Prevent consumers from importing internal paths (`my-pkg/src/internal`) +- Ensure clean package structure on npm +- Keep source files out of published package +- Maintain consistent import paths + +## Anti-Pattern: ESM-Only with Exports Map + +**NEVER use the `exports` map pattern:** + +```json +{ + "type": "module", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "exports": { + ".": { + "import": "./dist/index.js", + "types": "./dist/index.d.ts" + }, + "./api": { + "import": "./dist/api/index.js", + "types": "./dist/api/index.d.ts" + } + } +} +``` + +**Problems with this approach:** +- Breaks CommonJS consumers +- Exposes `dist/` in import paths +- Incompatible with the dist-folder publishing pattern +- Creates inconsistent import paths between development and published package + +**Instead, use the Constructive standard pattern shown below.** + +## Deep Nested Imports (Recommended for Tree-Shaking) + +Deep nested imports via file path are **fully supported and recommended** for tree-shaking. With dist-folder publishing, the `dist/` folder becomes the package root, so consumers can import directly from subdirectories: + +```typescript +// These imports work correctly with dist-folder publishing: +import { OrmClient } from '@my-org/sdk/api'; +import { AdminClient } from '@my-org/sdk/admin'; +import { AuthClient } from '@my-org/sdk/auth'; +``` + +This works because the published package structure looks like: + +```text +@my-org/sdk (on npm) +├── index.js # Main entry point +├── api/ +│ └── index.js # API-specific code +├── admin/ +│ └── index.js # Admin-specific code +└── auth/ + └── index.js # Auth-specific code +``` + +**Benefits of this approach:** +- Full tree-shaking support (only import what you need) +- Works with both CommonJS and ESM +- No `exports` map needed +- Clean import paths without `dist/` + +**Source structure for nested imports:** + +```text +my-package/ +├── src/ +│ ├── index.ts # Re-exports or shared code +│ ├── api/ +│ │ └── index.ts # API module +│ ├── admin/ +│ │ └── index.ts # Admin module +│ └── auth/ +│ └── index.ts # Auth module +└── package.json +``` + +After `makage build`, the `dist/` folder mirrors this structure and becomes the published package root. + +## Anti-Pattern: Manual Build Scripts Without Makage + +**NEVER use manual build scripts like this:** + +```json +{ + "scripts": { + "clean": "rimraf dist/**", + "copy": "copyfiles -f ../../LICENSE package.json dist", + "build": "npm run clean; tsc -p tsconfig.json; tsc -p tsconfig.esm.json; npm run copy" + }, + "devDependencies": { + "copyfiles": "^2.4.1", + "rimraf": "^6.0.1" + } +} +``` + +**Problems with this approach:** +- Reinvents what makage already does +- Requires multiple devDependencies (copyfiles, rimraf) instead of one (makage) +- Manual tsconfig management for CJS/ESM builds +- Inconsistent build behavior across packages +- Missing features like automatic source map handling + +**Instead, use makage which handles all of this automatically.** + +## Makage Overview + +[makage](https://www.npmjs.com/package/makage) is a tiny build helper that replaces cpy, rimraf, and other build tools: + +| Command | Description | +|---------|-------------| +| `makage build` | Clean, compile TypeScript, copy assets | +| `makage build --dev` | Build with source maps | +| `makage clean` | Remove dist folder | +| `makage assets` | Copy LICENSE, README, package.json to dist | + +## Package Configuration + +### package.json + +```json +{ + "name": "my-package", + "version": "0.1.0", + "description": "Package description", + "author": "Constructive ", + "main": "index.js", + "module": "esm/index.js", + "types": "index.d.ts", + "homepage": "https://github.com/org/my-workspace", + "license": "MIT", + "publishConfig": { + "access": "public", + "directory": "dist" + }, + "repository": { + "type": "git", + "url": "https://github.com/org/my-workspace" + }, + "scripts": { + "copy": "makage assets", + "clean": "makage clean", + "prepublishOnly": "npm run build", + "build": "makage build", + "lint": "eslint . --fix", + "test": "jest", + "test:watch": "jest --watch" + }, + "devDependencies": { + "makage": "0.1.10" + } +} +``` + +**Critical fields:** +- `publishConfig.directory: "dist"` — Publish from dist folder +- `main: "index.js"` — Points to CJS build (in dist) +- `module: "esm/index.js"` — Points to ESM build (in dist) +- `types: "index.d.ts"` — Points to type declarations (in dist) + +## Build Output Structure + +After `makage build`: + +```text +my-package/ +├── src/ +│ └── index.ts +├── dist/ +│ ├── index.js # CJS build +│ ├── index.d.ts # Type declarations +│ ├── esm/ +│ │ └── index.js # ESM build +│ ├── package.json # Copied from root +│ ├── README.md # Copied from root +│ └── LICENSE # Copied from root +└── package.json +``` + +The `dist/` folder is what gets published to npm. + +## Build Workflow + +### Development Build + +```bash +# Build with source maps for debugging +makage build --dev +``` + +### Production Build + +```bash +# Full build: clean, compile, copy assets +makage build +``` + +### Clean + +```bash +# Remove dist folder +makage clean +``` + +## Publishing Workflow + +### 1. Prepare + +```bash +# Install dependencies +pnpm install + +# Build all packages +pnpm -r run build + +# Run tests +pnpm -r run test + +# Run linting +pnpm -r run lint +``` + +### 2. Version + +```bash +# Interactive versioning (independent mode) +pnpm lerna version + +# Or with conventional commits +pnpm lerna version --conventional-commits +``` + +### 3. Publish + +```bash +# Publish to npm +pnpm lerna publish from-package +``` + +**Note:** Use `from-package` to publish packages that have been versioned but not yet published. + +### One-Liner + +```bash +pnpm install && pnpm -r run build && pnpm -r run test && pnpm lerna version && pnpm lerna publish from-package +``` + +## Dry Run Commands + +Test without making changes: + +```bash +# Test versioning (no git operations) +pnpm lerna version --no-git-tag-version --no-push + +# Test publishing +pnpm lerna publish from-package --dry-run +``` + +## Lerna Configuration + +### lerna.json + +```json +{ + "$schema": "node_modules/lerna/schemas/lerna-schema.json", + "version": "independent", + "npmClient": "pnpm", + "registry": "https://registry.npmjs.org", + "command": { + "create": { + "homepage": "https://github.com/org/my-workspace", + "license": "MIT", + "access": "restricted" + }, + "publish": { + "allowBranch": "main", + "message": "chore(release): publish", + "conventionalCommits": true + } + } +} +``` + +## Access Control + +### Public Packages + +```json +{ + "publishConfig": { + "access": "public", + "directory": "dist" + } +} +``` + +### Private/Scoped Packages + +```json +{ + "publishConfig": { + "access": "restricted", + "directory": "dist" + } +} +``` + +## TypeScript Configuration + +### tsconfig.json (package level) + +```json +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "./src", + "declaration": true, + "declarationMap": true + }, + "include": ["src/**/*"] +} +``` + +### ESM Build + +makage handles dual CJS/ESM builds automatically. The ESM output goes to `dist/esm/`. + +## Workspace Dependencies + +When publishing, `workspace:*` references are converted to actual versions: + +**Before publish (package.json):** +```json +{ + "dependencies": { + "my-other-package": "workspace:*" + } +} +``` + +**After publish (in dist/package.json):** +```json +{ + "dependencies": { + "my-other-package": "^0.5.0" + } +} +``` + +## Common Issues + +### Package Not Found After Publish + +Ensure `publishConfig.directory` is set to `"dist"`. + +### Types Not Found + +Ensure `types` field points to declaration file in dist: +```json +{ + "types": "index.d.ts" +} +``` + +### ESM Import Errors + +Ensure `module` field points to ESM build: +```json +{ + "module": "esm/index.js" +} +``` + +## Best Practices + +1. **Always build before publish**: Use `prepublishOnly` script +2. **Test the build**: Run tests against built output +3. **Use dry-run first**: Test versioning and publishing before committing +4. **Keep dist clean**: Run `makage clean` before builds +5. **Conventional commits**: Enable for automatic changelogs + +## References + +- Related skill: `pnpm-workspace` for workspace setup +- Related skill: `pgpm` (`references/publishing.md`) for SQL module publishing +- [makage on npm](https://www.npmjs.com/package/makage) diff --git a/.agents/skills/constructive-pnpm/references/pnpm-workspace.md b/.agents/skills/constructive-pnpm/references/pnpm-workspace.md new file mode 100644 index 000000000..8aa131bcd --- /dev/null +++ b/.agents/skills/constructive-pnpm/references/pnpm-workspace.md @@ -0,0 +1,275 @@ +--- +name: pnpm-workspace +description: Create and manage PNPM workspaces following Constructive standards. Use when asked to "create a monorepo", "set up a workspace", "configure pnpm", or when starting a new TypeScript/JavaScript project with multiple packages. +compatibility: pnpm, Node.js 18+, lerna +metadata: + author: constructive-io + version: "1.0.0" +--- + +# PNPM Workspaces (Constructive Standard) + +Create and manage PNPM monorepo workspaces following Constructive conventions. This covers pure TypeScript/JavaScript workspaces (not pgpm workspaces for SQL modules). + +## When to Apply + +Use this skill when: +- Creating a new TypeScript/JavaScript monorepo +- Setting up a pnpm workspace structure +- Configuring lerna for versioning and publishing +- Managing internal package dependencies + +## Workspace Structure + +A Constructive-standard pnpm workspace: + +```text +my-workspace/ +├── .eslintrc.json +├── .gitignore +├── .prettierrc.json +├── lerna.json +├── package.json +├── packages/ +│ ├── package-a/ +│ │ ├── package.json +│ │ ├── src/ +│ │ └── tsconfig.json +│ └── package-b/ +│ ├── package.json +│ ├── src/ +│ └── tsconfig.json +├── pnpm-lock.yaml +├── pnpm-workspace.yaml +└── tsconfig.json +``` + +## Core Configuration Files + +### pnpm-workspace.yaml + +Defines which directories contain packages: + +```yaml +packages: + - 'packages/*' +``` + +For larger projects with multiple package directories: + +```yaml +packages: + - 'packages/*' + - 'apps/*' + - 'libs/*' +``` + +### Root package.json + +```json +{ + "name": "my-workspace", + "version": "0.0.1", + "private": true, + "repository": { + "type": "git", + "url": "https://github.com/org/my-workspace" + }, + "license": "MIT", + "publishConfig": { + "access": "restricted" + }, + "scripts": { + "build": "pnpm -r run build", + "clean": "pnpm -r run clean", + "test": "pnpm -r run test", + "lint": "pnpm -r run lint", + "deps": "pnpm up -r -i -L" + }, + "devDependencies": { + "@types/jest": "^30.0.0", + "@types/node": "^22.10.2", + "@typescript-eslint/eslint-plugin": "^8.53.1", + "@typescript-eslint/parser": "^8.53.1", + "eslint": "^9.39.2", + "eslint-config-prettier": "^10.1.8", + "jest": "^30.2.0", + "lerna": "^8.2.4", + "prettier": "^3.8.0", + "ts-jest": "^29.4.6", + "ts-node": "^10.9.2", + "typescript": "^5.6.3" + } +} +``` + +**Key points:** +- Root package is `private: true` (never published) +- Scripts use `pnpm -r` to run recursively across packages +- `deps` script for interactive dependency updates + +### lerna.json + +```json +{ + "$schema": "node_modules/lerna/schemas/lerna-schema.json", + "version": "independent", + "npmClient": "pnpm", + "registry": "https://registry.npmjs.org", + "command": { + "create": { + "homepage": "https://github.com/org/my-workspace", + "license": "MIT", + "access": "restricted" + }, + "publish": { + "allowBranch": "main", + "message": "chore(release): publish", + "conventionalCommits": true + } + } +} +``` + +**Versioning modes:** +- `"version": "independent"` — Each package versioned separately (recommended for utility libraries) +- `"version": "0.0.1"` — Fixed versioning, all packages share same version (recommended for tightly coupled packages) + +## Package Configuration + +### Individual package.json + +```json +{ + "name": "my-package", + "version": "0.1.0", + "description": "Package description", + "author": "Constructive ", + "main": "index.js", + "module": "esm/index.js", + "types": "index.d.ts", + "homepage": "https://github.com/org/my-workspace", + "license": "MIT", + "publishConfig": { + "access": "public", + "directory": "dist" + }, + "repository": { + "type": "git", + "url": "https://github.com/org/my-workspace" + }, + "scripts": { + "clean": "makage clean", + "build": "makage build", + "lint": "eslint . --fix", + "test": "jest" + }, + "devDependencies": { + "makage": "0.1.10" + } +} +``` + +**Key patterns:** +- `publishConfig.directory: "dist"` — Publish from dist folder (prevents tree-shaking into weird paths) +- `main` and `module` point to built files (not src) +- Uses makage for build tooling + +## Internal Dependencies + +Reference workspace packages using `workspace:*`: + +```json +{ + "dependencies": { + "my-other-package": "workspace:*" + } +} +``` + +When published, `workspace:*` is replaced with the actual version number. + +## Common Commands + +| Command | Description | +|---------|-------------| +| `pnpm install` | Install all dependencies | +| `pnpm -r run build` | Build all packages | +| `pnpm -r run test` | Test all packages | +| `pnpm --filter run build` | Build specific package | +| `pnpm up -r -i -L` | Interactive dependency update | +| `pnpm lerna version` | Version packages | +| `pnpm lerna publish` | Publish packages | + +## Creating a New Package + +```bash +# Create package directory +mkdir -p packages/my-new-package/src + +# Initialize package.json +cd packages/my-new-package +pnpm init +``` + +Then configure package.json following the pattern above. + +## TypeScript Configuration + +### Root tsconfig.json + +```json +{ + "compilerOptions": { + "target": "ES2020", + "module": "commonjs", + "lib": ["ES2020"], + "declaration": true, + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": true + } +} +``` + +### Package tsconfig.json + +```json +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "./src" + }, + "include": ["src/**/*"] +} +``` + +## PNPM vs PGPM Workspaces + +| Aspect | PNPM Workspace | PGPM Workspace | +|--------|----------------|----------------| +| Purpose | TypeScript/JS packages | SQL database modules | +| Config | pnpm-workspace.yaml | pnpm-workspace.yaml + pgpm.json | +| Build | makage build | pgpm package | +| Output | dist/ folder | SQL bundles | +| Registry | npm | npm (for @pgpm/* packages) | + +Some repos (like constructive) are **hybrid** — they have both pnpm packages and pgpm modules. + +## Best Practices + +1. **Keep root private**: Never publish the root package +2. **Use workspace protocol**: Always use `workspace:*` for internal deps +3. **Consistent structure**: All packages follow same directory layout +4. **Shared config**: Extend root tsconfig.json in packages +5. **Independent versioning**: Use for utility libraries with different release cycles +6. **Fixed versioning**: Use for tightly coupled packages that should release together + +## References + +- Related skill: `pnpm-publishing` for publishing workflow with makage +- Related skill: `pgpm` (`references/workspace.md`) for SQL module workspaces +- Related skill: `pgpm` (`references/publishing.md`) for publishing pgpm modules diff --git a/.agents/skills/constructive-setup.zip b/.agents/skills/constructive-setup.zip new file mode 100644 index 000000000..e6f238a15 Binary files /dev/null and b/.agents/skills/constructive-setup.zip differ diff --git a/.agents/skills/constructive-setup/SKILL.md b/.agents/skills/constructive-setup/SKILL.md new file mode 100644 index 000000000..65fb61442 --- /dev/null +++ b/.agents/skills/constructive-setup/SKILL.md @@ -0,0 +1,127 @@ +--- +name: constructive-setup +description: "Set up the Constructive monorepo for development — install dependencies, start PostgreSQL via pgpm Docker, bootstrap users, build, run tests, and start local email services. Use when asked to 'set up constructive', 'get constructive running', 'set up dev environment', 'bootstrap database', 'start email services', 'test emails locally', or when starting work in the constructive-io/constructive repo." +metadata: + author: constructive-io + version: "1.0.0" + triggers: "user, model" +--- + +# Constructive Monorepo Setup + +Lightweight setup guide for getting the `constructive-io/constructive` monorepo running locally. References detailed skills for each subsystem instead of duplicating their content. + +## When to Apply + +Use this skill when: +- Setting up the Constructive monorepo for the first time +- Starting a new development session that needs a running database +- Troubleshooting a broken local environment + +## Quick Setup + +```bash +# 1. Install dependencies +pnpm install + +# 2. Start PostgreSQL via pgpm Docker +pgpm docker start --image docker.io/constructiveio/postgres-plus:18 --recreate +eval "$(pgpm env)" + +# 3. Bootstrap database users +pgpm admin-users bootstrap --yes +pgpm admin-users add --test --yes + +# 4. Build the monorepo +pnpm build + +# 5. Run tests (from a specific package) +cd packages/yourmodule +pnpm test +``` + +## Step-by-Step Details + +### 1. Install Dependencies + +The monorepo uses pnpm workspaces: + +```bash +pnpm install +``` + +### 2. Start PostgreSQL + +Use pgpm's Docker integration to start a local PostgreSQL container. The `postgres-plus:18` image includes all required extensions (PostGIS, pgvector, uuid-ossp, etc.). + +```bash +pgpm docker start --image docker.io/constructiveio/postgres-plus:18 --recreate +eval "$(pgpm env)" +``` + +> **Important:** `eval "$(pgpm env)"` must be run as a separate command (not chained with `&&`) because the env vars aren't available until the command completes. + +For full Docker options (custom ports, names, passwords), see the **pgpm** skill: [references/docker.md](../pgpm/references/docker.md) + +For environment variable details, see the **pgpm** skill: [references/env.md](../pgpm/references/env.md) + +### 3. Bootstrap Database Users + +Create the required PostgreSQL roles for Constructive's security model: + +```bash +pgpm admin-users bootstrap --yes +pgpm admin-users add --test --yes +``` + +### 4. Build + +```bash +pnpm build +``` + +This builds all packages in the monorepo. Required before running tests or starting servers. + +### 5. Run Tests + +Tests are run per-package: + +```bash +cd packages/yourmodule # or graphile/yourplugin, pgpm/core, etc. +pnpm test # single run +pnpm test:watch # watch mode +``` + +For testing patterns and frameworks, see the **constructive-testing** skill. + +## Local Email Services + +Start Mailpit, Admin GraphQL, send-email-link, and job-service for local email testing. + +See [local-email-services.md](./references/local-email-services.md) for Docker Compose setup, port reference, and troubleshooting. + +## Monorepo Layout + +| Directory | Contents | +|-----------|----------| +| `packages/*` | Constructive CLI, ORM, query-builder, server-utils | +| `pgpm/*` | PGPM engine, CLI, shared types/logger/env | +| `graphql/*` | GraphQL server, explorer, codegen, types, query/react utilities | +| `graphile/*` | Graphile/PostGraphile plugins (postgis, search, etc.) | +| `postgres/*` | PostgreSQL tooling (pg-ast, pg-codegen, introspectron, pgsql-test) | +| `extensions/*` | PGPM extension modules | + +For full navigation, see the repo's `AGENTS.md`. + +### Local Development Environment + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [local-dev-setup.md](references/local-dev-setup.md) | Quick-start local dev | Docker Postgres + GraphQL server startup | +| [local-env.md](references/local-env.md) | Full local environment | Detailed setup, endpoint reference, troubleshooting | + +## Cross-References + +- **pgpm** skill — Database migrations, Docker, environment, CLI commands +- **constructive-testing** skill — Test frameworks (pgsql-test, drizzle-orm-test, supabase-test) +- **constructive-cli** skill — Generated CLI commands and scaffolding diff --git a/.agents/skills/constructive-setup/references/local-dev-setup.md b/.agents/skills/constructive-setup/references/local-dev-setup.md new file mode 100644 index 000000000..93513c28b --- /dev/null +++ b/.agents/skills/constructive-setup/references/local-dev-setup.md @@ -0,0 +1,63 @@ +# Local Dev Setup + +Quick-start for running the Constructive GraphQL server locally with Docker Postgres and pgpm. + +## Prerequisites + +- Docker (for Postgres) +- Node.js v22+ +- pgpm: `npm install -g pgpm` + +## Start PostgreSQL + +```bash +pgpm docker start --image docker.io/constructiveio/postgres-plus:18 --recreate +eval "$(pgpm env)" +pgpm admin-users bootstrap --yes +pgpm admin-users add --test --yes +``` + +> **Important:** `eval "$(pgpm env)"` must be run as a separate command (not chained with `&&`) so the env vars are available for subsequent commands. + +## Deploy Your Database + +Navigate to your pgpm database workspace and deploy: + +```bash +cd /path/to/your-database +pgpm deploy +``` + +This runs all migrations in your `pgpm.plan` and provisions the schema. If your module hasn't been deployed before, add `--createdb` to create the database: + +```bash +pgpm deploy --database myapp --createdb --yes +``` + +For full deploy options, see the **pgpm** skill: [references/deploy.md](../pgpm/references/deploy.md) + +## Start GraphQL Server + +```bash +cd graphql/server +PGDATABASE=myapp pnpm dev +``` + +Set `PGDATABASE` to match the database name you deployed to. + +Health check: `curl -s -o /dev/null -w "%{http_code}" http://api.localhost:3000/graphql` → 405 + +## Endpoint Reference + +| Endpoint | Purpose | +|---|---| +| `http://auth.localhost:3000/graphql` | Main auth | +| `http://api.localhost:3000/graphql` | Main public API | +| `http://auth-.localhost:3000/graphql` | Per-DB auth | +| `http://app-public-.localhost:3000/graphql` | Per-DB app API | +| `http://admin-.localhost:3000/graphql` | Per-DB admin | + +## Related + +- **pgpm** skill — full deploy/revert/verify reference, Docker options, environment variables +- `constructive-sdk` skill — provision a user database after setup diff --git a/.agents/skills/constructive-setup/references/local-email-services.md b/.agents/skills/constructive-setup/references/local-email-services.md new file mode 100644 index 000000000..aede300f7 --- /dev/null +++ b/.agents/skills/constructive-setup/references/local-email-services.md @@ -0,0 +1,159 @@ +# Local Email Services + +Start Mailpit, Admin GraphQL Server, send-email-link, and job-service for local email testing using Docker Compose. + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Local Development │ +│ │ +│ Next.js (3011) ──► Public GraphQL (3000) │ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Email Services (Docker) │ │ +│ │ │ │ +│ │ Admin GraphQL (3002) │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ send-email-link (8082) │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ Mailpit (1025/8025) ◄── job-service (8080) │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ PostgreSQL (5432) ◄── job-service polls app_jobs │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Prerequisites + +1. Docker Desktop running +2. PostgreSQL running with constructive database +3. Database provisioned with `app_jobs` schema +4. `constructive` repo on branch `feat/add-local-email-service-docker-compose` (or main after merged) +5. `constructive:dev` Docker image built + +> ⚠️ **分支要求：** `docker-compose.local-email.yml` 在 `feat/add-local-email-service-docker-compose` 分支，合并到 main 之前需先切换： +> ```bash +> cd /path/to/constructive +> git checkout feat/add-local-email-service-docker-compose +> ``` + +## Quick Start + +### Step 1: Build the Docker Image (first time only) + +```bash +cd /path/to/constructive + +# Switch to required branch (until merged to main) +git checkout feat/add-local-email-service-docker-compose + +# Create network +docker network create constructive-net 2>/dev/null || true + +# Build image (takes ~3-5 minutes) +docker build -t constructive:dev . +``` + +### Step 2: Start Email Services + +```bash +cd /path/to/constructive + +docker-compose -f docker-compose.local-email.yml up -d +``` + +### Step 3: Verify + +```bash +# Check status +docker-compose -f docker-compose.local-email.yml ps + +# Health check +echo "=== Health Check ===" +curl -s http://localhost:8025 >/dev/null && echo "✅ Mailpit (8025)" +curl -s http://localhost:3002/graphql -H "X-Meta-Schema: true" -H "Content-Type: application/json" \ + -d '{"query":"{ __typename }"}' | grep -q __typename && echo "✅ Admin GraphQL (3002)" +lsof -i:8082 >/dev/null && echo "✅ send-email-link (8082)" +lsof -i:8080 >/dev/null && echo "✅ job-service (8080)" +``` + +**Mailpit UI:** http://localhost:8025 + +## Managing Services + +```bash +cd /path/to/constructive + +# Start +docker-compose -f docker-compose.local-email.yml up -d + +# Stop +docker-compose -f docker-compose.local-email.yml down + +# View logs (all services) +docker-compose -f docker-compose.local-email.yml logs -f + +# View logs (specific service) +docker logs -f mailpit +docker logs -f constructive-admin-server +docker logs -f send-email-link +docker logs -f knative-job-service + +# Restart +docker-compose -f docker-compose.local-email.yml restart +``` + +## Port Reference + +| Service | Port | URL | +|---------|------|-----| +| Mailpit SMTP | 1025 | - | +| Mailpit Web UI | 8025 | http://localhost:8025 | +| Admin GraphQL | 3002 | http://localhost:3002/graphql | +| send-email-link | 8082 | http://localhost:8082 | +| job-service | 8080 | http://localhost:8080 | + +## Test Email Flow + +1. Open Mailpit: http://localhost:8025 +2. Trigger an invite/signup from your app (http://localhost:3011) +3. Check Mailpit for the email + +## Troubleshooting + +### Port already in use + +```bash +# Find and kill process on port +lsof -ti:8082 | xargs kill -9 +``` + +### Container name conflict + +```bash +# Remove old containers +docker rm -f mailpit constructive-admin-server send-email-link knative-job-service + +# Restart +docker-compose -f docker-compose.local-email.yml up -d +``` + +### Cannot connect to PostgreSQL + +Check that `PGHOST` in docker-compose uses `host.docker.internal` (not `localhost`): + +```yaml +PGHOST: host.docker.internal +``` + +### View service logs for debugging + +```bash +docker logs constructive-admin-server 2>&1 | tail -30 +docker logs send-email-link 2>&1 | tail -30 +docker logs knative-job-service 2>&1 | tail -30 +``` diff --git a/.agents/skills/constructive-setup/references/local-env.md b/.agents/skills/constructive-setup/references/local-env.md new file mode 100644 index 000000000..ec3eeea55 --- /dev/null +++ b/.agents/skills/constructive-setup/references/local-env.md @@ -0,0 +1,105 @@ +# Constructive Local Environment Setup + +Set up a fully working local Constructive environment with PostgreSQL and the GraphQL server. Enables developing against a real GraphQL API and running tests. + +## When to Apply + +Use this reference when: +- Setting up a local development environment for Constructive +- Needing a running GraphQL server for development +- Troubleshooting local environment issues + +## Prerequisites + +- Docker installed and running +- Node.js 22+ +- pnpm installed + +## Step-by-Step Setup + +### 1. Install pgpm globally + +```bash +npm install -g pgpm +``` + +### 2. Start PostgreSQL via pgpm Docker + +```bash +pgpm docker start --image docker.io/constructiveio/postgres-plus:18 --recreate +``` + +Uses the `postgres-plus:18` container (includes PostGIS, pgvector, and other required extensions). PostgreSQL 17+ is required due to `security_invoker` views. + +### 3. Set environment variables + +```bash +eval "$(pgpm env)" +``` + +Sets `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, and `PGDATABASE` for the local PostgreSQL instance. + +**Important**: Run this as a separate command (not chained with `&&`) so the env vars are available for subsequent commands. + +### 4. Bootstrap database users + +```bash +pgpm admin-users bootstrap --yes +pgpm admin-users add --test --yes +``` + +### 5. Deploy your database + +Navigate to your pgpm database workspace and deploy: + +```bash +cd /path/to/your-database +pgpm deploy --database myapp --createdb --yes +``` + +This runs all migrations in your `pgpm.plan` and provisions the full schema (tables, functions, triggers, RLS policies). For an existing database, omit `--createdb`: + +```bash +pgpm deploy +``` + +For full deploy options, see the **pgpm** skill. + +### 6. Install dependencies and start server + +```bash +cd /path/to/constructive +pnpm install +pnpm build +cd graphql/server +PGDATABASE=myapp pnpm dev +``` + +Set `PGDATABASE` to match the database name you deployed to. + +The server starts with subdomain-based routing: + +| Target | Endpoint | +|--------|----------| +| Public | `http://api.localhost:3000/graphql` | +| Auth | `http://auth.localhost:3000/graphql` | +| Objects | `http://objects.localhost:3000/graphql` | +| Admin | `http://admin.localhost:3000/graphql` | + +## Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `PGHOST` | `localhost` | PostgreSQL host | +| `PGPORT` | `5432` | PostgreSQL port | +| `PGUSER` | `postgres` | PostgreSQL user | +| `PGPASSWORD` | `password` | PostgreSQL password | +| `PGDATABASE` | `constructive` | Target database | + +## Troubleshooting + +| Issue | Fix | +|-------|-----| +| "security_invoker" error | Need PostgreSQL 17+. Use `pgpm docker start --image docker.io/constructiveio/postgres-plus:18` | +| "permission denied for schema" | Token not set or expired. Re-authenticate | +| Server not responding | Verify with `curl -s http://api.localhost:3000/graphql -H 'Content-Type: application/json' -d '{"query":"{ __typename }"}'` | diff --git a/.agents/skills/constructive-testing.zip b/.agents/skills/constructive-testing.zip new file mode 100644 index 000000000..900341383 Binary files /dev/null and b/.agents/skills/constructive-testing.zip differ diff --git a/.agents/skills/constructive-testing/SKILL.md b/.agents/skills/constructive-testing/SKILL.md new file mode 100644 index 000000000..68f7cabbe --- /dev/null +++ b/.agents/skills/constructive-testing/SKILL.md @@ -0,0 +1,106 @@ +--- +name: constructive-testing +description: "PostgreSQL testing frameworks — pgsql-test (RLS, seeding, snapshots, JWT context, scenario setup), drizzle-orm-test (type-safe Drizzle testing), supabase-test (Supabase RLS testing). Use when writing database tests, testing RLS policies, seeding test data, or testing with any Constructive test framework." +compatibility: pgsql-test, drizzle-orm-test, supabase-test, Jest/Vitest, PostgreSQL + triggers: "user, model" +metadata: + author: constructive-io + version: "2.0.0" +--- + +# Constructive Testing + +All database testing frameworks for Constructive. Each framework builds on `pgsql-test` underneath — they all create isolated test databases with proper teardown. + +## When to Apply + +Use this skill when: +- Writing PostgreSQL integration tests +- Testing RLS policies, permissions, multi-tenant security +- Seeding test data (fixtures, JSON, SQL, CSV) +- Testing with Drizzle ORM or Supabase + +## Which Framework to Use + +| Scenario | Framework | Reference | +|----------|-----------|-----------| +| Raw SQL, RLS policies, database functions | `pgsql-test` | [pgsql-test.md](./references/pgsql-test.md) | +| PostGraphile schema, basic GraphQL queries | `graphile-test` | (part of constructive monorepo) | +| GraphQL with Constructive plugins (search, pgvector, etc.) | `@constructive-io/graphql-test` | (part of constructive monorepo) | +| Dynamic codegen + ORM in tests | `@constructive-io/graphql-test` (`runCodegenAndLoad`) | (part of constructive monorepo) | +| HTTP endpoints, auth headers, middleware | `@constructive-io/graphql-server-test` | (part of constructive monorepo) | +| Type-safe Drizzle ORM tests | `drizzle-orm-test` | [drizzle-orm-test.md](./references/drizzle-orm-test.md) | +| Supabase applications, auth.users | `supabase-test` | [supabase-test.md](./references/supabase-test.md) | + +## Quick Start (pgsql-test) + +```typescript +import { getConnections } from 'pgsql-test'; + +let db, teardown; +beforeAll(async () => ({ db, teardown } = await getConnections())); +afterAll(() => teardown()); +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); + +test('example', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': '123' }); + const result = await db.query('SELECT current_user_id()'); + expect(result.rows[0].current_user_id).toBe('123'); +}); +``` + +## The Testing Framework Hierarchy + +Choose the **highest-level framework** that fits your test scenario: + +``` +┌─────────────────────────────────────────────────────┐ +│ @constructive-io/graphql-server-test │ HTTP-level +│ SuperTest against real Express + PostGraphile │ (full stack) +├─────────────────────────────────────────────────────┤ +│ @constructive-io/graphql-test │ GraphQL + Constructive +│ GraphQL queries with all Constructive plugins │ plugins loaded +├─────────────────────────────────────────────────────┤ +│ graphile-test │ GraphQL schema-level +│ GraphQL queries against PostGraphile schema │ (no HTTP) +├─────────────────────────────────────────────────────┤ +│ pgsql-test │ SQL-level +│ Raw SQL queries, RLS, seeding, snapshots │ (database only) +└─────────────────────────────────────────────────────┘ +``` + +## Critical Rules + +1. **Always include `beforeEach`/`afterEach` hooks** — savepoint-based isolation prevents test state leakage +2. **Never create `new pg.Pool()` or `new pg.Client()` in tests** — use `getConnections()` +3. **Never manually create/drop databases** — the framework handles this +4. **Never skip hooks** — tests will leak state + +## Reference Guide + +### Core Test Framework (pgsql-test) + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [pgsql-test.md](./references/pgsql-test.md) | pgsql-test overview and setup | Getting started with PostgreSQL testing | +| [pgsql-test-rls.md](./references/pgsql-test-rls.md) | RLS policy testing | Testing row-level security, user isolation | +| [pgsql-test-seeding.md](./references/pgsql-test-seeding.md) | Test data seeding | loadJson, loadSql, loadCsv fixtures | +| [pgsql-test-exceptions.md](./references/pgsql-test-exceptions.md) | Exception handling | Testing operations that should fail | +| [pgsql-test-snapshot.md](./references/pgsql-test-snapshot.md) | Snapshot testing | pruneIds, pruneDates, deterministic assertions | +| [pgsql-test-helpers.md](./references/pgsql-test-helpers.md) | Helper utilities | Common test helper functions | +| [pgsql-test-jwt-context.md](./references/pgsql-test-jwt-context.md) | JWT context testing | Setting JWT claims, testing authenticated queries | +| [pgsql-test-transactions.md](./references/pgsql-test-transactions.md) | Transaction-local context & roles | `beforeAll` context gotcha, three roles (superuser/administrator/authenticated), `pg` vs `db` | +| [pgsql-test-scenario-setup.md](./references/pgsql-test-scenario-setup.md) | Complex scenario setup | Multi-client scenarios, complex test arrangements | + +### Additional Frameworks + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [drizzle-orm-test.md](./references/drizzle-orm-test.md) | Drizzle ORM testing | Type-safe database tests with Drizzle | +| [supabase-test.md](./references/supabase-test.md) | Supabase testing | Testing Supabase apps, auth.users, anon/authenticated roles | + +## Cross-References + +- `pgpm` — Database migrations (deploy before testing) +- `constructive-setup` — Monorepo setup and local development environment diff --git a/.agents/skills/constructive-testing/references/drizzle-orm-test.md b/.agents/skills/constructive-testing/references/drizzle-orm-test.md new file mode 100644 index 000000000..b118e674d --- /dev/null +++ b/.agents/skills/constructive-testing/references/drizzle-orm-test.md @@ -0,0 +1,409 @@ +--- +name: drizzle-orm-test +description: Test PostgreSQL databases with Drizzle ORM using drizzle-orm-test. Use when asked to "test with Drizzle", "test Drizzle ORM", "write type-safe database tests", or when testing applications using Drizzle ORM. +compatibility: drizzle-orm-test, drizzle-orm, Jest/Vitest, PostgreSQL +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Testing with Drizzle ORM + +Test PostgreSQL databases with Drizzle ORM using drizzle-orm-test. Get type-safe queries, automatic context management, and RLS testing. + +## When to Apply + +Use this skill when: +- Testing applications using Drizzle ORM +- Writing type-safe database tests +- Testing RLS policies with Drizzle +- Migrating from pgsql-test to Drizzle + +## Why drizzle-orm-test? + +drizzle-orm-test is a drop-in replacement for pgsql-test that adds: +- Type-safe queries with Drizzle ORM +- Automatic context management +- Same test isolation patterns +- Compatible with existing pgsql-test workflows + +## Setup + +### Install Dependencies + +```bash +pnpm add -D drizzle-orm-test drizzle-orm +``` + +### Define Drizzle Schema + +Create `src/schema.ts`: + +```typescript +import { pgTable, uuid, text, timestamp } from 'drizzle-orm/pg-core'; + +export const users = pgTable('users', { + id: uuid('id').primaryKey().defaultRandom(), + email: text('email').notNull().unique(), + name: text('name'), + createdAt: timestamp('created_at').defaultNow() +}); + +export const posts = pgTable('posts', { + id: uuid('id').primaryKey().defaultRandom(), + title: text('title').notNull(), + content: text('content'), + ownerId: uuid('owner_id').references(() => users.id), + createdAt: timestamp('created_at').defaultNow() +}); +``` + +## Core Concepts + +### Three Database Clients + +| Client | Purpose | +|--------|---------| +| `pg` | Superuser pgsql-test client (bypasses RLS) | +| `db` | User pgsql-test client (for RLS context) | +| `drizzleDb` | Drizzle ORM client (type-safe queries) | + +### Test Isolation + +Same as pgsql-test: +- `beforeEach()` starts transaction/savepoint +- `afterEach()` rolls back +- Tests are completely isolated + +## Basic Test Structure + +```typescript +import { getConnections, PgTestClient } from 'drizzle-orm-test'; +import { drizzle } from 'drizzle-orm/node-postgres'; +import { users, posts } from '../src/schema'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; +let drizzleDb: ReturnType; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + // Create Drizzle client from pg connection + drizzleDb = drizzle(pg.client); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await pg.beforeEach(); + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); + await pg.afterEach(); +}); +``` + +## Type-Safe Queries + +### Insert + +```typescript +it('inserts a user with Drizzle', async () => { + const [user] = await drizzleDb + .insert(users) + .values({ + email: 'alice@example.com', + name: 'Alice' + }) + .returning(); + + expect(user.email).toBe('alice@example.com'); + expect(user.name).toBe('Alice'); + expect(user.id).toBeDefined(); +}); +``` + +### Select + +```typescript +it('queries users with Drizzle', async () => { + // Insert test data + await drizzleDb.insert(users).values([ + { email: 'alice@example.com', name: 'Alice' }, + { email: 'bob@example.com', name: 'Bob' } + ]); + + // Query with type safety + const result = await drizzleDb + .select() + .from(users) + .where(eq(users.name, 'Alice')); + + expect(result).toHaveLength(1); + expect(result[0].email).toBe('alice@example.com'); +}); +``` + +### Update + +```typescript +import { eq } from 'drizzle-orm'; + +it('updates a user', async () => { + const [user] = await drizzleDb + .insert(users) + .values({ email: 'alice@example.com', name: 'Alice' }) + .returning(); + + const [updated] = await drizzleDb + .update(users) + .set({ name: 'Alice Smith' }) + .where(eq(users.id, user.id)) + .returning(); + + expect(updated.name).toBe('Alice Smith'); +}); +``` + +### Delete + +```typescript +it('deletes a user', async () => { + const [user] = await drizzleDb + .insert(users) + .values({ email: 'alice@example.com' }) + .returning(); + + await drizzleDb + .delete(users) + .where(eq(users.id, user.id)); + + const result = await drizzleDb + .select() + .from(users) + .where(eq(users.id, user.id)); + + expect(result).toHaveLength(0); +}); +``` + +## Testing RLS with Drizzle + +For RLS testing, use `db.setContext()` with the pgsql-test client, then query with Drizzle: + +```typescript +import { getConnections, PgTestClient } from 'drizzle-orm-test'; +import { drizzle } from 'drizzle-orm/node-postgres'; +import { eq } from 'drizzle-orm'; +import { posts } from '../src/schema'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; +let drizzleDb: ReturnType; + +const alice = '550e8400-e29b-41d4-a716-446655440001'; +const bob = '550e8400-e29b-41d4-a716-446655440002'; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + // Create Drizzle client from db connection (respects RLS) + drizzleDb = drizzle(db.client); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await pg.beforeEach(); + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); + await pg.afterEach(); +}); + +it('user only sees own posts', async () => { + // Seed as superuser + await pg.loadJson({ + 'posts': [ + { title: 'Alice Post', owner_id: alice }, + { title: 'Bob Post', owner_id: bob } + ] + }); + + // Set context to Alice + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + // Query with Drizzle - RLS filters results + const result = await drizzleDb + .select() + .from(posts); + + expect(result).toHaveLength(1); + expect(result[0].title).toBe('Alice Post'); +}); +``` + +## Testing INSERT Policies + +```typescript +it('user can insert own post', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + const [post] = await drizzleDb + .insert(posts) + .values({ + title: 'My Post', + ownerId: alice + }) + .returning(); + + expect(post.title).toBe('My Post'); + expect(post.ownerId).toBe(alice); +}); + +it('user cannot insert for another user', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + const point = 'insert_other'; + await db.savepoint(point); + + await expect( + drizzleDb + .insert(posts) + .values({ + title: 'Hacked Post', + ownerId: bob + }) + ).rejects.toThrow(/permission denied|violates row-level security/); + + await db.rollback(point); +}); +``` + +## Testing UPDATE Policies + +```typescript +it('user can update own post', async () => { + // Seed + await pg.loadJson({ + 'posts': [{ id: 'post-1', title: 'Original', owner_id: alice }] + }); + + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + const [updated] = await drizzleDb + .update(posts) + .set({ title: 'Updated' }) + .where(eq(posts.id, 'post-1')) + .returning(); + + expect(updated.title).toBe('Updated'); +}); + +it('user cannot update other user post', async () => { + await pg.loadJson({ + 'posts': [{ id: 'post-1', title: 'Bob Post', owner_id: bob }] + }); + + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + // RLS filters - update affects 0 rows + const result = await drizzleDb + .update(posts) + .set({ title: 'Hacked' }) + .where(eq(posts.id, 'post-1')) + .returning(); + + expect(result).toHaveLength(0); +}); +``` + +## Testing DELETE Policies + +```typescript +it('user can delete own post', async () => { + await pg.loadJson({ + 'posts': [{ id: 'post-1', title: 'My Post', owner_id: alice }] + }); + + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + await drizzleDb + .delete(posts) + .where(eq(posts.id, 'post-1')); + + // Verify as superuser + const result = await pg.query('SELECT * FROM posts WHERE id = $1', ['post-1']); + expect(result.rows).toHaveLength(0); +}); +``` + +## Handling Expected Failures + +Use savepoint pattern with Drizzle: + +```typescript +it('anonymous cannot insert', async () => { + db.setContext({ role: 'anonymous' }); + + const point = 'anon_insert'; + await db.savepoint(point); + + await expect( + drizzleDb + .insert(posts) + .values({ title: 'Hacked' }) + ).rejects.toThrow(/permission denied/); + + await db.rollback(point); +}); +``` + +## Watch Mode + +```bash +pnpm test:watch +``` + +## Best Practices + +1. **Use `pg` for setup**: Bypass RLS when seeding +2. **Use `db` for context**: Set role/user context +3. **Use Drizzle for queries**: Type-safe assertions +4. **Savepoint for failures**: Handle expected errors +5. **Schema in sync**: Keep Drizzle schema matching database + +## References + +- Related skill: `pgsql-test-rls` for RLS testing patterns +- Related skill: `pgsql-test-exceptions` for handling aborted transactions +- Related skill: `pgsql-test-seeding` for seeding strategies diff --git a/.agents/skills/constructive-testing/references/pgsql-test-exceptions.md b/.agents/skills/constructive-testing/references/pgsql-test-exceptions.md new file mode 100644 index 000000000..c8a397783 --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-exceptions.md @@ -0,0 +1,222 @@ +--- +name: pgsql-test-exceptions +description: Handle PostgreSQL aborted transactions when testing operations that should fail. Use when testing RLS policy violations, constraint errors, permission denied errors, or any expected database exceptions. Essential for security testing. +compatibility: Node.js 18+, pgsql-test package, PostgreSQL +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Testing Exceptions and Aborted Transactions + +Handle PostgreSQL's transaction abort behavior when testing operations that should fail. This is essential for security testing where you verify that unauthorized operations are rejected. + +## When to Apply + +Use this skill when: +- Testing RLS policy violations (user can't access other users' data) +- Testing constraint violations (unique, foreign key, check) +- Testing permission denied errors +- Testing any operation that should throw an error +- Verifying database state after a failed operation + +## The Problem + +When PostgreSQL encounters an error inside a transaction, it aborts the entire transaction. The connection rejects all further commands until you explicitly end the transaction: + +``` +current transaction is aborted, commands ignored until end of transaction block +``` + +This breaks naive exception testing: + +```typescript +// THIS WILL FAIL +it('users cannot insert pets for other users', async () => { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': bob + }); + + await expect( + db.query(`INSERT INTO pets (name, owner_id) VALUES ('Fake', $1)`, [alice]) + ).rejects.toThrow(/violates row-level security/); + + // ERROR: transaction is aborted, this query fails! + const count = await db.query(`SELECT COUNT(*) FROM pets WHERE owner_id = $1`, [alice]); +}); +``` + +## The Solution: Savepoints + +Create a savepoint before the failing operation, then roll back to it: + +```typescript +it('users cannot insert pets for other users', async () => { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': bob + }); + + // 1. Create savepoint before expected failure + await db.savepoint('insert_attempt'); + + // 2. Test the operation that should fail + await expect( + db.query(`INSERT INTO pets (name, owner_id) VALUES ('Fake', $1)`, [alice]) + ).rejects.toThrow(/violates row-level security/); + + // 3. Roll back to clear the error state + await db.rollback('insert_attempt'); + + // 4. Continue with verification queries + const count = await db.query(`SELECT COUNT(*) FROM pets WHERE owner_id = $1`, [alice]); + expect(parseInt(count.rows[0].count)).toBe(2); +}); +``` + +## Pattern Template + +```typescript +// 1. Create savepoint +await db.savepoint('my_savepoint_name'); + +// 2. Execute operation that should fail +await expect( + db.query(`...`) +).rejects.toThrow(/expected error pattern/); + +// 3. Roll back to savepoint +await db.rollback('my_savepoint_name'); + +// 4. Continue with additional queries +const result = await db.query(`...`); +``` + +## Common Scenarios + +### RLS Policy Violations + +```typescript +it('users cannot modify other users data', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': bob }); + + await db.savepoint('update_attempt'); + await expect( + db.query(`UPDATE items SET name = 'stolen' WHERE owner_id = $1`, [alice]) + ).rejects.toThrow(/violates row-level security/); + await db.rollback('update_attempt'); + + // Verify data unchanged + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': alice }); + const item = await db.query(`SELECT name FROM items WHERE owner_id = $1`, [alice]); + expect(item.rows[0].name).toBe('original'); +}); +``` + +### Permission Denied + +```typescript +it('anonymous users cannot modify data', async () => { + db.setContext({ role: 'anonymous' }); + + await db.savepoint('anon_insert'); + await expect( + db.query(`INSERT INTO users (name) VALUES ('hacker')`) + ).rejects.toThrow(/permission denied/); + await db.rollback('anon_insert'); + + await db.savepoint('anon_update'); + await expect( + db.query(`UPDATE users SET name = 'hacked'`) + ).rejects.toThrow(/permission denied/); + await db.rollback('anon_update'); + + await db.savepoint('anon_delete'); + await expect( + db.query(`DELETE FROM users`) + ).rejects.toThrow(/permission denied/); + await db.rollback('anon_delete'); +}); +``` + +### Constraint Violations + +```typescript +it('rejects duplicate emails', async () => { + await db.query(`INSERT INTO users (email) VALUES ('test@example.com')`); + + await db.savepoint('duplicate_email'); + await expect( + db.query(`INSERT INTO users (email) VALUES ('test@example.com')`) + ).rejects.toThrow(/duplicate key value violates unique constraint/); + await db.rollback('duplicate_email'); + + // Verify only one user exists + const count = await db.query(`SELECT COUNT(*) FROM users`); + expect(parseInt(count.rows[0].count)).toBe(1); +}); +``` + +### PLPGSQL Function Validation + +```typescript +describe('plpgsql_expr', () => { + it('rejects NULL query', async () => { + await db.savepoint('null_query'); + await expect( + db.any(`SELECT my_function(NULL)`) + ).rejects.toThrow('query cannot be NULL'); + await db.rollback('null_query'); + }); + + it('rejects invalid input', async () => { + await db.savepoint('invalid_input'); + await expect( + db.any(`SELECT my_function('{"invalid": true}'::jsonb)`) + ).rejects.toThrow('invalid input format'); + await db.rollback('invalid_input'); + }); +}); +``` + +### Multiple Failure Tests in Sequence + +```typescript +it('validates all input constraints', async () => { + // Test 1: null name + await db.savepoint('null_name'); + await expect( + db.query(`INSERT INTO products (name, price) VALUES (NULL, 10)`) + ).rejects.toThrow(/null value in column "name"/); + await db.rollback('null_name'); + + // Test 2: negative price + await db.savepoint('negative_price'); + await expect( + db.query(`INSERT INTO products (name, price) VALUES ('item', -5)`) + ).rejects.toThrow(/violates check constraint/); + await db.rollback('negative_price'); + + // Test 3: valid insert works + await db.query(`INSERT INTO products (name, price) VALUES ('item', 10)`); + const result = await db.query(`SELECT * FROM products`); + expect(result.rows).toHaveLength(1); +}); +``` + +## Key Rules + +1. **Always use unique savepoint names** - Avoid conflicts between tests +2. **Roll back immediately after the expected failure** - Before any other queries +3. **Use descriptive savepoint names** - Makes debugging easier +4. **Each failure needs its own savepoint** - Can't reuse savepoints after rollback + +## Why This Matters + +Without savepoints, you cannot verify database state after a failed operation. That verification is often the most important part of a security test - confirming that the malicious operation had no effect. + +## References + +- Related skill: `pgpm` (`references/testing.md`) for general test setup +- Related skill: `pgpm` (`references/env.md`) for environment configuration diff --git a/.agents/skills/constructive-testing/references/pgsql-test-helpers.md b/.agents/skills/constructive-testing/references/pgsql-test-helpers.md new file mode 100644 index 000000000..26083d541 --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-helpers.md @@ -0,0 +1,345 @@ +--- +name: pgsql-test-helpers +description: Creating reusable test helper functions and constants for consistent, maintainable database tests. Use when extracting common test patterns into helpers, defining shared constants, or reducing duplication in pgsql-test suites. +--- + +Creating reusable test helper functions and constants for consistent, maintainable database tests. + +## Overview + +As test suites grow, common patterns emerge: creating users, setting up contexts, querying specific tables. Extracting these into helper functions improves readability, reduces duplication, and makes tests more maintainable. + +## Predefined Constants + +### Test User IDs + +Use predefined UUIDs for test users to ensure consistency across tests: + +```typescript +export const TEST_USER_IDS = { + USER_1: '00000000-0000-0000-0000-000000000001', + USER_2: '00000000-0000-0000-0000-000000000002', + USER_3: '00000000-0000-0000-0000-000000000003', + ADMIN: '00000000-0000-0000-0000-000000000099', +} as const; +``` + +Benefits: +- Easy to identify in database queries and logs +- Consistent across all test files +- Type-safe with `as const` + +### Scope/Role Constants + +```typescript +export const ROLES = { + ANONYMOUS: 'anonymous', + AUTHENTICATED: 'authenticated', + SERVICE: 'service_role', +} as const; + +export const SCOPE = { + APP: 1, + ORG: 2, + GROUP: 3, +} as const; +``` + +## TypeScript Interfaces for Options + +Define interfaces for helper function parameters: + +```typescript +export interface CreateUserOptions { + id: string; + username?: string; + display_name?: string; + email?: string; + is_admin?: boolean; +} + +export interface CreateOrganizationOptions { + name: string; + owner_id: string; +} + +export interface AddMemberOptions { + user_id: string; + org_id: string; + role?: 'member' | 'admin' | 'owner'; +} +``` + +Benefits: +- Self-documenting function signatures +- IDE autocomplete support +- Compile-time validation + +## Helper Function Patterns + +### Creating Test Users + +```typescript +import { PgTestClient } from 'pgsql-test'; + +export async function createTestUser( + pg: PgTestClient, + options: CreateUserOptions +): Promise { + const { + id, + username = `user_${id.slice(0, 8)}`, + display_name = 'Test User', + email, + is_admin = false, + } = options; + + const columns = ['id', 'username', 'display_name', 'is_admin']; + const values: unknown[] = [id, username, display_name, is_admin]; + const placeholders = ['$1', '$2', '$3', '$4']; + + if (email !== undefined) { + columns.push('email'); + values.push(email); + placeholders.push(`$${values.length}`); + } + + await pg.query( + `INSERT INTO users (${columns.join(', ')}) + VALUES (${placeholders.join(', ')}) + ON CONFLICT (id) DO NOTHING`, + values + ); +} +``` + +### Creating Organizations + +```typescript +export async function createOrganization( + client: PgTestClient, + options: CreateOrganizationOptions +): Promise { + const { name, owner_id } = options; + + const result = await client.one<{ id: string }>( + `INSERT INTO organizations (name, owner_id) + VALUES ($1, $2) + RETURNING id`, + [name, owner_id] + ); + + return result.id; +} +``` + +### Querying with Type Safety + +```typescript +export interface UserRecord { + id: string; + username: string; + display_name: string; + is_admin: boolean; + created_at: Date; +} + +export async function getUserById( + client: PgTestClient, + userId: string +): Promise { + return client.oneOrNone( + `SELECT id, username, display_name, is_admin, created_at + FROM users WHERE id = $1`, + [userId] + ); +} + +export async function getUsersByOrg( + client: PgTestClient, + orgId: string +): Promise { + return client.any( + `SELECT u.id, u.username, u.display_name, u.is_admin, u.created_at + FROM users u + JOIN memberships m ON m.user_id = u.id + WHERE m.org_id = $1`, + [orgId] + ); +} +``` + +## Unique Name Generation + +For avoiding collisions in parallel tests: + +```typescript +export function uniqueName(prefix: string): string { + return `${prefix}-${Date.now()}`; +} + +export function uniqueEmail(prefix: string = 'test'): string { + return `${prefix}-${Date.now()}@example.com`; +} +``` + +Usage: + +```typescript +const orgName = uniqueName('test-org'); // 'test-org-1706123456789' +const email = uniqueEmail('alice'); // 'alice-1706123456789@example.com' +``` + +## Context Helper Functions + +### Setting Up Authenticated Context + +```typescript +export function setAuthContext( + db: PgTestClient, + userId: string, + additionalClaims?: Record +): void { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + ...additionalClaims, + }); +} + +export function setOrgContext( + db: PgTestClient, + userId: string, + orgId: string +): void { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + 'jwt.claims.org_id': orgId, + }); +} +``` + +Usage: + +```typescript +it('user can access their data', async () => { + setAuthContext(db, TEST_USER_IDS.USER_1); + const data = await db.any('SELECT * FROM my_table'); + expect(data.length).toBeGreaterThan(0); +}); +``` + +## Organizing Test Utils + +### File Structure + +``` +__tests__/ + test-utils/ + index.ts # Re-exports everything + constants.ts # TEST_USER_IDS, ROLES, etc. + interfaces.ts # TypeScript interfaces + user-helpers.ts # User-related helpers + org-helpers.ts # Organization helpers + context-helpers.ts # Context/auth helpers +``` + +### index.ts + +```typescript +export * from './constants'; +export * from './interfaces'; +export * from './user-helpers'; +export * from './org-helpers'; +export * from './context-helpers'; +``` + +### Usage in Tests + +```typescript +import { + TEST_USER_IDS, + createTestUser, + createOrganization, + setAuthContext, +} from '../test-utils'; + +describe('Organization tests', () => { + beforeAll(async () => { + await createTestUser(pg, { id: TEST_USER_IDS.USER_1 }); + }); + + it('creates organization', async () => { + setAuthContext(db, TEST_USER_IDS.USER_1); + const orgId = await createOrganization(db, { + name: 'Test Org', + owner_id: TEST_USER_IDS.USER_1, + }); + expect(orgId).toBeDefined(); + }); +}); +``` + +## Assertion Helpers + +### Expecting Specific Counts + +```typescript +export async function expectRowCount( + client: PgTestClient, + table: string, + expectedCount: number, + where?: string, + values?: unknown[] +): Promise { + const whereClause = where ? ` WHERE ${where}` : ''; + const result = await client.one<{ count: string }>( + `SELECT COUNT(*) FROM ${table}${whereClause}`, + values + ); + expect(parseInt(result.count)).toBe(expectedCount); +} +``` + +Usage: + +```typescript +await expectRowCount(db, 'users', 2); +await expectRowCount(db, 'memberships', 1, 'org_id = $1', [orgId]); +``` + +### Expecting Access Denied + +```typescript +export async function expectAccessDenied( + client: PgTestClient, + query: string, + values?: unknown[] +): Promise { + const result = await client.any(query, values); + expect(result.length).toBe(0); +} + +export async function expectQueryError( + client: PgTestClient, + query: string, + values?: unknown[], + errorPattern?: RegExp +): Promise { + await expect(client.query(query, values)).rejects.toThrow(errorPattern); +} +``` + +## Best Practices + +1. Keep helpers focused and single-purpose +2. Use TypeScript interfaces for all option objects +3. Provide sensible defaults for optional parameters +4. Use `ON CONFLICT DO NOTHING` for idempotent user creation +5. Return IDs from creation helpers for use in subsequent operations +6. Group related helpers in separate files +7. Re-export everything from a central index.ts +8. Use predefined constants instead of magic strings/UUIDs +9. Document complex helpers with JSDoc comments +10. Keep helpers in a dedicated test-utils directory diff --git a/.agents/skills/constructive-testing/references/pgsql-test-jwt-context.md b/.agents/skills/constructive-testing/references/pgsql-test-jwt-context.md new file mode 100644 index 000000000..d08050e1f --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-jwt-context.md @@ -0,0 +1,211 @@ +--- +name: pgsql-test-jwt-context +description: Setting up JWT claims and role-based context for RLS testing with pgsql-test. Use when testing Row-Level Security policies, simulating authenticated users with JWT claims, or configuring PostgreSQL session variables for RLS. +--- + +Setting up JWT claims and role-based context for RLS testing with pgsql-test. + +## Overview + +When testing Row-Level Security (RLS) policies, you need to simulate authenticated users with JWT claims. The `pgsql-test` library provides the `setContext()` method to configure PostgreSQL session variables that RLS policies can read. + +## The setContext API + +Use `setContext()` to simulate different user roles and JWT claims: + +```typescript +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': '00000000-0000-0000-0000-000000000001', + 'jwt.claims.org_id': 'acme-corp' +}); +``` + +This applies settings using `SET LOCAL` statements, ensuring they persist only for the current transaction and maintain proper isolation between tests. + +## How It Works Internally + +The `setContext()` method generates SQL statements: + +```sql +-- For the 'role' key, uses SET LOCAL ROLE +SET LOCAL ROLE "authenticated"; + +-- For other keys, uses set_config() with transaction-local scope +SELECT set_config('jwt.claims.user_id', '00000000-0000-0000-0000-000000000001', true); +SELECT set_config('jwt.claims.org_id', 'acme-corp', true); +``` + +The third parameter `true` in `set_config()` makes the setting transaction-local, which is essential for test isolation. + +## The auth() Helper Method + +For common authentication patterns, use the `auth()` helper: + +```typescript +// Simple authenticated user +db.auth({ + role: 'authenticated', + userId: '00000000-0000-0000-0000-000000000001' +}); + +// Custom user ID key +db.auth({ + role: 'authenticated', + userId: '123', + userIdKey: 'request.jwt.claims.sub' +}); +``` + +## Common JWT Claim Patterns + +### User Authentication + +```typescript +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId +}); +``` + +### Organization Context + +```typescript +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + 'jwt.claims.org_id': orgId +}); +``` + +### Database Context + +```typescript +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + 'jwt.claims.database_id': databaseId +}); +``` + +### Additional Claims + +```typescript +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + 'jwt.claims.user_agent': 'Mozilla/5.0...', + 'jwt.claims.ip_address': '127.0.0.1' +}); +``` + +## Reading Claims in SQL + +Your RLS policies can read these claims using `current_setting()`: + +```sql +-- In an RLS policy +CREATE POLICY user_isolation ON my_table + FOR ALL + USING (owner_id = current_setting('jwt.claims.user_id', true)::uuid); +``` + +You can also create helper functions: + +```sql +CREATE FUNCTION current_user_id() RETURNS uuid AS $$ + SELECT current_setting('jwt.claims.user_id', true)::uuid; +$$ LANGUAGE sql STABLE; +``` + +## Clearing Context + +To reset context between scenarios: + +```typescript +db.clearContext(); +``` + +This nulls all previously set context variables and resets to the default anonymous role. + +## Testing Different Access Levels + +```typescript +describe('RLS policies', () => { + const USER_1 = '00000000-0000-0000-0000-000000000001'; + const USER_2 = '00000000-0000-0000-0000-000000000002'; + + beforeEach(() => db.beforeEach()); + afterEach(() => db.afterEach()); + + it('user can see their own data', async () => { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': USER_1 + }); + + const rows = await db.any('SELECT * FROM my_table WHERE owner_id = $1', [USER_1]); + expect(rows.length).toBeGreaterThan(0); + }); + + it('user cannot see other users data', async () => { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': USER_2 + }); + + const rows = await db.any('SELECT * FROM my_table WHERE owner_id = $1', [USER_1]); + expect(rows.length).toBe(0); + }); + + it('anonymous users have no access', async () => { + db.setContext({ role: 'anonymous' }); + + const rows = await db.any('SELECT * FROM my_table'); + expect(rows.length).toBe(0); + }); +}); +``` + +## Context Timing + +Call `setContext()` before `beforeEach()` to apply context at the start of each test: + +```typescript +describe('authenticated role', () => { + beforeEach(async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': USER_ID }); + await db.beforeEach(); + }); + + afterEach(() => db.afterEach()); + + it('runs as authenticated', async () => { + const res = await db.query(`SELECT current_setting('role', true) AS role`); + expect(res.rows[0].role).toBe('authenticated'); + }); +}); +``` + +Or set context within individual tests for scenario-specific testing: + +```typescript +it('switches between users', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': USER_1 }); + const user1Data = await db.any('SELECT * FROM my_table'); + + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': USER_2 }); + const user2Data = await db.any('SELECT * FROM my_table'); + + expect(user1Data).not.toEqual(user2Data); +}); +``` + +## Best Practices + +1. Use predefined UUID constants for test user IDs to ensure consistency +2. Set context before `beforeEach()` for describe-level defaults +3. Use `clearContext()` when switching between unrelated scenarios +4. Test both positive cases (user can access) and negative cases (user cannot access) +5. Test anonymous/unauthenticated access explicitly +6. **In `beforeAll()`, wrap context-dependent `db` operations in explicit `db.begin()`/`db.commit()`** — `set_config(..., true)` is transaction-local, so without an active transaction the context is lost between queries. See [pgsql-test-transactions.md](./pgsql-test-transactions.md) for details. diff --git a/.agents/skills/constructive-testing/references/pgsql-test-rls.md b/.agents/skills/constructive-testing/references/pgsql-test-rls.md new file mode 100644 index 000000000..9861d265c --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-rls.md @@ -0,0 +1,339 @@ +--- +name: pgsql-test-rls +description: Test Row-Level Security (RLS) policies with pgsql-test. Use when asked to "test RLS", "test permissions", "test user access", "verify security policies", or when writing tests for multi-tenant applications. +compatibility: pgsql-test, Jest/Vitest, PostgreSQL +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Testing RLS Policies with pgsql-test + +Test Row-Level Security policies by simulating different users and roles. Verify your security policies work correctly with isolated, transactional tests. + +## When to Apply + +Use this skill when: +- Testing RLS policies for multi-tenant applications +- Verifying user isolation (users only see their own data) +- Testing role-based access (anonymous, authenticated, admin) +- Validating INSERT/UPDATE/DELETE policies + +## Setup + +Install pgsql-test: +```bash +pnpm add -D pgsql-test +``` + +Configure Jest/Vitest with the test database. + +## Core Concepts + +### Two Database Clients + +pgsql-test provides two clients: + +| Client | Purpose | +|--------|---------| +| `pg` | Superuser client for setup/teardown (bypasses RLS) | +| `db` | User client for testing with RLS enforcement | + +### Test Isolation + +Each test runs in a transaction with savepoints: +- `beforeEach()` starts a savepoint +- `afterEach()` rolls back to savepoint +- Tests are completely isolated + +## Basic RLS Test Structure + +```typescript +import { getConnections, PgTestClient } from 'pgsql-test'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await pg.beforeEach(); + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); + await pg.afterEach(); +}); +``` + +## Setting User Context + +Use `setContext()` to simulate different users: + +```typescript +// Simulate authenticated user +db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': userId +}); + +// Simulate anonymous user +db.setContext({ role: 'anonymous' }); + +// Simulate admin +db.setContext({ + role: 'administrator', + 'request.jwt.claim.sub': adminId +}); +``` + +## Testing SELECT Policies + +Verify users only see their own data: + +```typescript +it('users only see their own records', async () => { + // Setup: Insert data as superuser + await pg.query(` + INSERT INTO app.posts (id, title, owner_id) VALUES + ('post-1', 'User 1 Post', $1), + ('post-2', 'User 2 Post', $2) + `, [user1Id, user2Id]); + + // Test: User 1 queries + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': user1Id + }); + + const result = await db.query('SELECT * FROM app.posts'); + + expect(result.rows).toHaveLength(1); + expect(result.rows[0].title).toBe('User 1 Post'); +}); +``` + +## Testing INSERT Policies + +Verify users can only insert their own data: + +```typescript +it('user can insert own record', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': userId + }); + + const result = await db.one(` + INSERT INTO app.posts (title, owner_id) + VALUES ('My Post', $1) + RETURNING id, title, owner_id + `, [userId]); + + expect(result.title).toBe('My Post'); + expect(result.owner_id).toBe(userId); +}); + +it('user cannot insert for another user', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': user1Id + }); + + // Use savepoint pattern for expected failures + const point = 'insert_other_user'; + await db.savepoint(point); + + await expect( + db.query(` + INSERT INTO app.posts (title, owner_id) + VALUES ('Hacked Post', $1) + `, [user2Id]) + ).rejects.toThrow(/permission denied|violates row-level security/); + + await db.rollback(point); +}); +``` + +## Testing UPDATE Policies + +```typescript +it('user can update own record', async () => { + // Setup + await pg.query(` + INSERT INTO app.posts (id, title, owner_id) + VALUES ('post-1', 'Original', $1) + `, [userId]); + + // Test + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': userId + }); + + const result = await db.one(` + UPDATE app.posts SET title = 'Updated' + WHERE id = 'post-1' + RETURNING title + `); + + expect(result.title).toBe('Updated'); +}); + +it('user cannot update another user record', async () => { + await pg.query(` + INSERT INTO app.posts (id, title, owner_id) + VALUES ('post-1', 'Original', $1) + `, [user2Id]); + + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': user1Id + }); + + // Update returns no rows (RLS filters it out) + const result = await db.query(` + UPDATE app.posts SET title = 'Hacked' + WHERE id = 'post-1' + RETURNING id + `); + + expect(result.rows).toHaveLength(0); +}); +``` + +## Testing DELETE Policies + +```typescript +it('user can delete own record', async () => { + await pg.query(` + INSERT INTO app.posts (id, title, owner_id) + VALUES ('post-1', 'To Delete', $1) + `, [userId]); + + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': userId + }); + + await db.query(`DELETE FROM app.posts WHERE id = 'post-1'`); + + // Verify as superuser + const result = await pg.query(` + SELECT * FROM app.posts WHERE id = 'post-1' + `); + expect(result.rows).toHaveLength(0); +}); +``` + +## Testing Anonymous Access + +```typescript +it('anonymous users have read-only access', async () => { + await pg.query(` + INSERT INTO app.public_posts (id, title) + VALUES ('post-1', 'Public Post') + `); + + db.setContext({ role: 'anonymous' }); + + // Can read public data + const result = await db.query('SELECT * FROM app.public_posts'); + expect(result.rows).toHaveLength(1); + + // Cannot modify + const point = 'anon_insert'; + await db.savepoint(point); + await expect( + db.query(`INSERT INTO app.public_posts (title) VALUES ('Hacked')`) + ).rejects.toThrow(/permission denied/); + await db.rollback(point); +}); +``` + +## Multi-User Scenarios + +Test interactions between multiple users: + +```typescript +describe('multi-user isolation', () => { + const alice = '550e8400-e29b-41d4-a716-446655440001'; + const bob = '550e8400-e29b-41d4-a716-446655440002'; + + beforeEach(async () => { + // Seed data for both users + await pg.query(` + INSERT INTO app.posts (title, owner_id) VALUES + ('Alice Post 1', $1), + ('Alice Post 2', $1), + ('Bob Post 1', $2) + `, [alice, bob]); + }); + + it('alice sees only her posts', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice + }); + + const result = await db.query('SELECT title FROM app.posts ORDER BY title'); + expect(result.rows).toHaveLength(2); + expect(result.rows.map(r => r.title)).toEqual(['Alice Post 1', 'Alice Post 2']); + }); + + it('bob sees only his posts', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': bob + }); + + const result = await db.query('SELECT title FROM app.posts'); + expect(result.rows).toHaveLength(1); + expect(result.rows[0].title).toBe('Bob Post 1'); + }); +}); +``` + +## Handling Expected Failures + +When testing operations that should fail, use the savepoint pattern to avoid "current transaction is aborted" errors: + +```typescript +it('rejects unauthorized access', async () => { + db.setContext({ role: 'anonymous' }); + + const point = 'unauthorized_access'; + await db.savepoint(point); + + await expect( + db.query('INSERT INTO app.private_data (secret) VALUES ($1)', ['hack']) + ).rejects.toThrow(/permission denied/); + + await db.rollback(point); + + // Can continue using db connection + const result = await db.query('SELECT 1 as ok'); + expect(result.rows[0].ok).toBe(1); +}); +``` + +## Watch Mode + +Run tests in watch mode for rapid feedback: +```bash +pnpm test:watch +``` + +## References + +- Related skill: `pgsql-test-exceptions` for handling aborted transactions +- Related skill: `pgsql-test-seeding` for seeding test data +- Related skill: `pgpm` (`references/testing.md`) for general test setup diff --git a/.agents/skills/constructive-testing/references/pgsql-test-scenario-setup.md b/.agents/skills/constructive-testing/references/pgsql-test-scenario-setup.md new file mode 100644 index 000000000..e19340144 --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-scenario-setup.md @@ -0,0 +1,289 @@ +--- +name: pgsql-test-scenario-setup +description: Structuring complex test scenarios with proper isolation, transaction management, and multi-client patterns. Use when building complex RLS test scenarios, managing test isolation, or implementing multi-client database test patterns. +--- + +Structuring complex test scenarios with proper isolation, transaction management, and multi-client patterns. + +## Overview + +Complex RLS and database tests often require careful setup: creating users, seeding data, and testing access patterns. This skill covers the patterns for structuring these scenarios with proper test isolation. + +## The Two-Client Pattern + +The `getConnections()` function returns multiple clients with different privilege levels: + +```typescript +import { getConnections, PgTestClient } from 'pgsql-test'; + +let db: PgTestClient; // App-level client (RLS-enforced) +let pg: PgTestClient; // Superuser client (bypasses RLS) +let teardown: () => Promise; + +beforeAll(async () => { + ({ db, pg, teardown } = await getConnections()); +}); + +afterAll(() => teardown()); +``` + +### When to Use Each Client + +**Use `pg` (superuser) for:** +- Test setup that needs to bypass RLS +- Creating test users and seed data +- Administrative operations +- Verifying data exists regardless of RLS + +**Use `db` (app-level) for:** +- Testing actual RLS behavior +- Simulating real application queries +- Verifying access control works correctly + +### Example: RLS Test Setup + +```typescript +const TEST_USER_1 = '00000000-0000-0000-0000-000000000001'; +const TEST_USER_2 = '00000000-0000-0000-0000-000000000002'; + +beforeAll(async () => { + ({ db, pg, teardown } = await getConnections()); + + // Use pg (superuser) to create test users - bypasses RLS + await pg.query( + `INSERT INTO users (id, username) VALUES ($1, 'user1') ON CONFLICT DO NOTHING`, + [TEST_USER_1] + ); + await pg.query( + `INSERT INTO users (id, username) VALUES ($1, 'user2') ON CONFLICT DO NOTHING`, + [TEST_USER_2] + ); + + // Create test data owned by user 1 + await pg.query( + `INSERT INTO documents (owner_id, title) VALUES ($1, 'User 1 Doc')`, + [TEST_USER_1] + ); +}); + +// Now test RLS with db client +it('user 1 can see their documents', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': TEST_USER_1 }); + const docs = await db.any('SELECT * FROM documents'); + expect(docs.length).toBe(1); +}); + +it('user 2 cannot see user 1 documents', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': TEST_USER_2 }); + const docs = await db.any('SELECT * FROM documents'); + expect(docs.length).toBe(0); +}); +``` + +## Transaction Management + +### Test Isolation with beforeEach/afterEach + +Each test runs in its own transaction that rolls back after the test: + +```typescript +beforeEach(async () => { + await db.beforeEach(); // BEGIN + SAVEPOINT +}); + +afterEach(async () => { + await db.afterEach(); // ROLLBACK TO SAVEPOINT + COMMIT +}); +``` + +This ensures tests don't affect each other - any data created during a test is rolled back. + +### What beforeEach/afterEach Do + +```typescript +// db.beforeEach() executes: +await this.begin(); // BEGIN transaction +await this.savepoint(); // SAVEPOINT "lqlsavepoint" + +// db.afterEach() executes: +await this.rollback(); // ROLLBACK TO SAVEPOINT "lqlsavepoint" +await this.commit(); // COMMIT (the outer transaction) +``` + +## The publish() Method + +When you need data created in one client to be visible to another client (or to persist beyond the current transaction), use `publish()`: + +```typescript +it('cross-connection visibility', async () => { + // Create data with db client + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': USER_ID }); + await db.query(`INSERT INTO items (name) VALUES ('test item')`); + + // Data is not yet visible to pg client (different connection) + let pgItems = await pg.any('SELECT * FROM items WHERE name = $1', ['test item']); + expect(pgItems.length).toBe(0); + + // Publish makes data visible to other connections + await db.publish(); + + // Now pg can see it + pgItems = await pg.any('SELECT * FROM items WHERE name = $1', ['test item']); + expect(pgItems.length).toBe(1); +}); +``` + +### What publish() Does + +```typescript +// db.publish() executes: +await this.commit(); // Make data visible to other sessions +await this.begin(); // Start fresh transaction +await this.savepoint(); // Maintain rollback harness +await this.ctxQuery(); // Reapply setContext() settings +``` + +## Setup Patterns + +### Pattern 1: Simple Setup in beforeAll + +For tests that share the same seed data: + +```typescript +beforeAll(async () => { + ({ db, pg, teardown } = await getConnections()); + + // Seed data once + await pg.query(`INSERT INTO users (id, name) VALUES ($1, 'Alice')`, [USER_ID]); +}); + +afterAll(() => teardown()); + +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); +``` + +### Pattern 2: Complex Setup with Transactions + +For setup that requires multiple steps with intermediate commits: + +```typescript +beforeAll(async () => { + ({ db, pg, teardown } = await getConnections()); + + // Start transaction on pg for setup + await pg.begin(); + await pg.savepoint(); + + // Create users + await pg.query(`INSERT INTO users (id, name) VALUES ($1, 'Alice')`, [USER_1]); + await pg.query(`INSERT INTO users (id, name) VALUES ($1, 'Bob')`, [USER_2]); + + // Commit pg's work so db can see it + await pg.commit(); + await pg.begin(); + await pg.savepoint(); + + // Now db can work with the users + await db.begin(); + await db.savepoint(); + + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': USER_1 }); + // ... additional setup with RLS context + + await db.commit(); + await db.begin(); + await db.savepoint(); +}); +``` + +### Pattern 3: Per-Describe Setup + +For describe blocks that need their own isolated setup: + +```typescript +describe('Admin scenarios', () => { + let adminId: string; + + beforeAll(async () => { + // Create admin user for this describe block + const result = await pg.one<{ id: string }>( + `INSERT INTO users (name, is_admin) VALUES ('Admin', true) RETURNING id` + ); + adminId = result.id; + }); + + beforeEach(() => db.beforeEach()); + afterEach(() => db.afterEach()); + + it('admin can see all data', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': adminId }); + // ... + }); +}); +``` + +## Scenario Testing Pattern + +For testing complex workflows with multiple actors: + +```typescript +describe('Organization membership scenarios', () => { + const OWNER_ID = '00000000-0000-0000-0000-000000000001'; + const MEMBER_ID = '00000000-0000-0000-0000-000000000002'; + let orgId: string; + + beforeAll(async () => { + ({ db, pg, teardown } = await getConnections()); + + // Create test users + await pg.query(`INSERT INTO users (id, name) VALUES ($1, 'Owner')`, [OWNER_ID]); + await pg.query(`INSERT INTO users (id, name) VALUES ($1, 'Member')`, [MEMBER_ID]); + }); + + afterAll(() => teardown()); + beforeEach(() => db.beforeEach()); + afterEach(() => db.afterEach()); + + it('owner creates organization', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': OWNER_ID }); + + const result = await db.one<{ id: string }>( + `INSERT INTO organizations (name) VALUES ('Acme') RETURNING id` + ); + orgId = result.id; + + expect(orgId).toBeDefined(); + }); + + it('owner can add members', async () => { + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': OWNER_ID }); + + // First recreate the org (previous test rolled back) + const org = await db.one<{ id: string }>( + `INSERT INTO organizations (name) VALUES ('Acme') RETURNING id` + ); + + await db.query( + `INSERT INTO memberships (org_id, user_id) VALUES ($1, $2)`, + [org.id, MEMBER_ID] + ); + + const members = await db.any( + `SELECT * FROM memberships WHERE org_id = $1`, + [org.id] + ); + expect(members.length).toBe(1); + }); +}); +``` + +## Best Practices + +1. Use `pg` for setup, `db` for testing RLS behavior +2. Always call `beforeEach()`/`afterEach()` for test isolation +3. Use `publish()` when data needs to be visible across connections +4. Keep test user IDs as constants for consistency +5. Structure complex scenarios with clear beforeAll setup +6. Remember that each test's changes are rolled back - don't depend on previous test state +7. Use descriptive test names that explain the scenario being tested diff --git a/.agents/skills/constructive-testing/references/pgsql-test-seeding.md b/.agents/skills/constructive-testing/references/pgsql-test-seeding.md new file mode 100644 index 000000000..fc8b25674 --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-seeding.md @@ -0,0 +1,286 @@ +--- +name: pgsql-test-seeding +description: Seed test databases with pgsql-test using loadJson, loadSql, and loadCsv. Use when asked to "seed test data", "load fixtures", "populate test database", or when setting up test data for database tests. +compatibility: pgsql-test, Jest/Vitest, PostgreSQL +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Seeding Test Databases with pgsql-test + +Load test data efficiently using loadJson, loadSql, and loadCsv methods. Create maintainable, realistic test fixtures. + +## When to Apply + +Use this skill when: +- Setting up test data for database tests +- Loading fixtures from JSON, SQL, or CSV files +- Seeding data that respects or bypasses RLS +- Creating per-test or shared test data + +## Seeding Methods Overview + +| Method | Best For | RLS Behavior | +|--------|----------|--------------| +| `loadJson()` | Inline data, small datasets | Respects RLS (use `pg` to bypass) | +| `loadSql()` | Complex data, version-controlled fixtures | Respects RLS (use `pg` to bypass) | +| `loadCsv()` | Large datasets, spreadsheet exports | Bypasses RLS (uses COPY) | + +## Seeding with loadJson() + +Best for inline test data. Clean, readable, and type-safe. + +```typescript +import { getConnections, PgTestClient } from 'pgsql-test'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + // Seed using superuser to bypass RLS + await pg.loadJson({ + 'app.users': [ + { + id: '550e8400-e29b-41d4-a716-446655440001', + email: 'alice@example.com', + name: 'Alice' + }, + { + id: '550e8400-e29b-41d4-a716-446655440002', + email: 'bob@example.com', + name: 'Bob' + } + ], + 'app.posts': [ + { + id: 'post-1', + title: 'First Post', + owner_id: '550e8400-e29b-41d4-a716-446655440001' + }, + { + id: 'post-2', + title: 'Second Post', + owner_id: '550e8400-e29b-41d4-a716-446655440002' + } + ] + }); +}); + +afterAll(async () => { + await teardown(); +}); +``` + +**Key features:** +- Schema-qualified table names: `'app.users'` +- Explicit UUIDs for referential integrity +- Multiple tables in one call +- Order matters for foreign keys + +## Seeding with loadSql() + +Best for complex data or version-controlled fixtures. + +Create `__tests__/fixtures/seed.sql`: +```sql +-- Insert users +INSERT INTO app.users (id, email, name) VALUES + ('550e8400-e29b-41d4-a716-446655440001', 'alice@example.com', 'Alice'), + ('550e8400-e29b-41d4-a716-446655440002', 'bob@example.com', 'Bob'), + ('550e8400-e29b-41d4-a716-446655440003', 'charlie@example.com', 'Charlie'); + +-- Insert posts with foreign key references +INSERT INTO app.posts (id, title, owner_id) VALUES + ('post-1', 'Alice Post 1', '550e8400-e29b-41d4-a716-446655440001'), + ('post-2', 'Alice Post 2', '550e8400-e29b-41d4-a716-446655440001'), + ('post-3', 'Bob Post', '550e8400-e29b-41d4-a716-446655440002'); +``` + +Load in tests: +```typescript +import path from 'path'; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + await pg.loadSql([ + path.join(__dirname, 'fixtures/seed.sql') + ]); +}); +``` + +**Multiple SQL files:** +```typescript +await pg.loadSql([ + path.join(__dirname, 'fixtures/users.sql'), + path.join(__dirname, 'fixtures/posts.sql'), + path.join(__dirname, 'fixtures/comments.sql') +]); +``` + +Files execute in order, so put parent tables first. + +## Seeding with loadCsv() + +Best for large datasets or spreadsheet exports. + +Create `__tests__/fixtures/users.csv`: +```csv +id,email,name +550e8400-e29b-41d4-a716-446655440001,alice@example.com,Alice +550e8400-e29b-41d4-a716-446655440002,bob@example.com,Bob +550e8400-e29b-41d4-a716-446655440003,charlie@example.com,Charlie +``` + +Load in tests: +```typescript +import path from 'path'; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + await pg.loadCsv({ + 'app.users': path.join(__dirname, 'fixtures/users.csv'), + 'app.posts': path.join(__dirname, 'fixtures/posts.csv') + }); +}); +``` + +**Important:** `loadCsv()` uses PostgreSQL's COPY command, which bypasses RLS. Always use `pg` (superuser) client for CSV loading. + +## Combining Seeding Strategies + +Mix methods based on data characteristics: + +```typescript +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + // 1. Load large reference data from CSV + await pg.loadCsv({ + 'app.categories': path.join(__dirname, 'fixtures/categories.csv') + }); + + // 2. Load complex relationships from SQL + await pg.loadSql([ + path.join(__dirname, 'fixtures/users-with-roles.sql') + ]); + + // 3. Add test-specific data inline + await pg.loadJson({ + 'app.posts': [ + { title: 'Test Post', owner_id: testUserId, category_id: 1 } + ] + }); +}); +``` + +## Per-Test Seeding + +When different tests need different data, seed in `beforeEach()`: + +```typescript +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await pg.beforeEach(); + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); + await pg.afterEach(); +}); + +describe('empty state tests', () => { + it('handles no data gracefully', async () => { + const result = await db.query('SELECT COUNT(*) FROM app.posts'); + expect(result.rows[0].count).toBe('0'); + }); +}); + +describe('populated state tests', () => { + beforeEach(async () => { + await pg.loadJson({ + 'app.posts': [ + { title: 'Test Post', owner_id: userId } + ] + }); + }); + + it('finds existing posts', async () => { + const result = await db.query('SELECT COUNT(*) FROM app.posts'); + expect(result.rows[0].count).toBe('1'); + }); +}); +``` + +## RLS-Aware Seeding + +When testing RLS, seed with the appropriate client: + +```typescript +// Bypass RLS for setup (use pg) +await pg.loadJson({ + 'app.posts': [{ title: 'Admin Post', owner_id: adminId }] +}); + +// Respect RLS for user operations (use db with context) +db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': userId +}); + +await db.loadJson({ + 'app.posts': [{ title: 'User Post', owner_id: userId }] +}); +``` + +## Fixture Organization + +Recommended structure: +```text +__tests__/ +├── fixtures/ +│ ├── users.csv +│ ├── posts.csv +│ ├── seed.sql +│ └── complex-scenario.sql +├── users.test.ts +├── posts.test.ts +└── rls.test.ts +``` + +## Best Practices + +1. **Use explicit IDs**: Makes referential integrity predictable +2. **Order by dependencies**: Parent tables before child tables +3. **Keep fixtures minimal**: Only seed what tests need +4. **Use `pg` for setup**: Bypass RLS during seeding +5. **Use `db` for testing**: Enforce RLS during assertions +6. **Version control fixtures**: SQL/CSV files in repo + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| Foreign key violation | Load parent tables first | +| RLS blocking inserts | Use `pg` client instead of `db` | +| CSV format errors | Ensure headers match column names | +| Data persists between tests | Check beforeEach/afterEach hooks | + +## References + +- Related skill: `pgsql-test-rls` for RLS testing patterns +- Related skill: `pgsql-test-exceptions` for handling errors +- Related skill: `pgpm` (`references/testing.md`) for general test setup diff --git a/.agents/skills/constructive-testing/references/pgsql-test-snapshot.md b/.agents/skills/constructive-testing/references/pgsql-test-snapshot.md new file mode 100644 index 000000000..d658ad281 --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-snapshot.md @@ -0,0 +1,357 @@ +--- +name: pgsql-test-snapshot +description: Snapshot testing utilities for PostgreSQL tests. Use when asked to "snapshot test", "prune IDs from snapshots", "deterministic test output", or when writing tests that need stable, reproducible assertions. +compatibility: pgsql-test, drizzle-orm-test, Jest/Vitest, PostgreSQL +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Snapshot Testing with pgsql-test + +Use snapshot utilities from `pgsql-test/utils` to create deterministic, reproducible test assertions. These helpers replace dynamic values (IDs, UUIDs, dates, hashes) with stable placeholders. + +## When to Apply + +Use this skill when: +- Writing snapshot tests for database queries +- Need deterministic output from queries with UUIDs or timestamps +- Testing API responses that include database-generated values +- Comparing query results across test runs + +## Core Utilities + +Import from `pgsql-test/utils` or `drizzle-orm-test/utils`: + +```typescript +import { + snapshot, + prune, + pruneIds, + pruneDates, + pruneUUIDs, + pruneHashes, + pruneTokens, + composePruners, + createSnapshot +} from 'pgsql-test/utils'; +``` + +## Basic Usage + +### snapshot() + +The main utility that applies all default pruners recursively: + +```typescript +import { snapshot } from 'pgsql-test/utils'; + +const result = await db.query('SELECT * FROM users'); +expect(snapshot(result.rows)).toMatchSnapshot(); +``` + +Output transforms dynamic values to stable placeholders: + +```typescript +// Before snapshot() +{ + id: '550e8400-e29b-41d4-a716-446655440000', + name: 'Alice', + created_at: '2024-01-15T10:30:00.000Z', + password_hash: '$2b$10$...' +} + +// After snapshot() +{ + id: '[ID]', + name: 'Alice', + created_at: '[DATE]', + password_hash: '[hash]' +} +``` + +### With Drizzle ORM + +```typescript +import { drizzle } from 'drizzle-orm/node-postgres'; +import { snapshot } from 'drizzle-orm-test/utils'; +import { users } from './schema'; + +const drizzleDb = drizzle(db.client); +const result = await drizzleDb.select().from(users); +expect(snapshot(result)).toMatchSnapshot(); +``` + +## Individual Pruners + +### pruneIds() + +Replaces `id` and `*_id` fields with `[ID]`: + +```typescript +import { pruneIds } from 'pgsql-test/utils'; + +pruneIds({ id: 123, user_id: 'abc-123', name: 'Alice' }); +// { id: '[ID]', user_id: '[ID]', name: 'Alice' } +``` + +### pruneDates() + +Replaces Date objects and ISO date strings in `*_at` or `*At` fields: + +```typescript +import { pruneDates } from 'pgsql-test/utils'; + +pruneDates({ + created_at: '2024-01-15T10:30:00.000Z', + updatedAt: new Date(), + name: 'Alice' +}); +// { created_at: '[DATE]', updatedAt: '[DATE]', name: 'Alice' } +``` + +### pruneUUIDs() + +Replaces UUID values in `uuid` and `queue_name` fields: + +```typescript +import { pruneUUIDs } from 'pgsql-test/utils'; + +pruneUUIDs({ uuid: '550e8400-e29b-41d4-a716-446655440000' }); +// { uuid: '[UUID]' } +``` + +### pruneHashes() + +Replaces `*_hash` fields starting with `$`: + +```typescript +import { pruneHashes } from 'pgsql-test/utils'; + +pruneHashes({ password_hash: '$2b$10$xyz...' }); +// { password_hash: '[hash]' } +``` + +### pruneTokens() + +Replaces `token` and `*_token` fields: + +```typescript +import { pruneTokens } from 'pgsql-test/utils'; + +pruneTokens({ access_token: 'eyJhbGciOiJIUzI1NiIs...' }); +// { access_token: '[token]' } +``` + +### pruneIdArrays() + +Replaces `*_ids` array fields with count placeholder: + +```typescript +import { pruneIdArrays } from 'pgsql-test/utils'; + +pruneIdArrays({ member_ids: ['id1', 'id2', 'id3'] }); +// { member_ids: '[UUIDs-3]' } +``` + +### prunePeoplestamps() + +Replaces `*_by` fields (audit columns): + +```typescript +import { prunePeoplestamps } from 'pgsql-test/utils'; + +prunePeoplestamps({ created_by: 'user-123', updated_by: 'user-456' }); +// { created_by: '[peoplestamp]', updated_by: '[peoplestamp]' } +``` + +### pruneSchemas() + +Replaces schema names starting with `zz-`: + +```typescript +import { pruneSchemas } from 'pgsql-test/utils'; + +pruneSchemas({ schema: 'zz-abc123' }); +// { schema: '[schemahash]' } +``` + +## ID Hash Tracking + +Track ID relationships across snapshots using `IdHash`: + +```typescript +import { snapshot, pruneIds, IdHash } from 'pgsql-test/utils'; + +const idHash: IdHash = { + '550e8400-e29b-41d4-a716-446655440001': 'alice', + '550e8400-e29b-41d4-a716-446655440002': 'bob' +}; + +const result = [ + { id: '550e8400-e29b-41d4-a716-446655440001', name: 'Alice' }, + { id: '550e8400-e29b-41d4-a716-446655440002', name: 'Bob' } +]; + +expect(snapshot(result, idHash)).toMatchSnapshot(); +// [ +// { id: '[ID-alice]', name: 'Alice' }, +// { id: '[ID-bob]', name: 'Bob' } +// ] +``` + +Numeric ID tracking: + +```typescript +const idHash: IdHash = {}; +let counter = 1; + +// Assign IDs as you encounter them +for (const row of result) { + if (!idHash[row.id]) { + idHash[row.id] = counter++; + } +} + +expect(snapshot(result, idHash)).toMatchSnapshot(); +// [ +// { id: '[ID-1]', name: 'Alice' }, +// { id: '[ID-2]', name: 'Bob' } +// ] +``` + +## Custom Pruners + +### composePruners() + +Combine multiple pruners into one: + +```typescript +import { composePruners, pruneDates, pruneIds } from 'pgsql-test/utils'; + +const myPruner = composePruners(pruneDates, pruneIds); +const result = myPruner({ id: 123, created_at: new Date() }); +// { id: '[ID]', created_at: '[DATE]' } +``` + +### createSnapshot() + +Create a custom snapshot function with specific pruners: + +```typescript +import { createSnapshot, pruneDates, pruneIds, pruneHashes } from 'pgsql-test/utils'; + +const mySnapshot = createSnapshot([pruneDates, pruneIds, pruneHashes]); + +const result = await db.query('SELECT * FROM users'); +expect(mySnapshot(result.rows)).toMatchSnapshot(); +``` + +## Default Pruners + +The `snapshot()` function applies these pruners by default: + +1. `pruneTokens` — `token`, `*_token` +2. `prunePeoplestamps` — `*_by` +3. `pruneDates` — `*_at`, `*At`, Date objects +4. `pruneIdArrays` — `*_ids` arrays +5. `pruneUUIDs` — `uuid`, `queue_name` +6. `pruneHashes` — `*_hash` +7. `pruneIds` — `id`, `*_id` + +## Error Code Extraction + +Extract error codes from enhanced error messages: + +```typescript +import { getErrorCode } from 'pgsql-test/utils'; + +try { + await db.query('SELECT * FROM nonexistent'); +} catch (err) { + const code = getErrorCode(err.message); + // Returns first line only, stripping debug context + expect(code).toBe('UNDEFINED_TABLE'); +} +``` + +## PostgreSQL Error Formatting + +Format PostgreSQL errors for readable output: + +```typescript +import { + extractPgErrorFields, + formatPgError, + formatPgErrorFields +} from 'pgsql-test/utils'; + +try { + await db.query('invalid sql'); +} catch (err) { + const fields = extractPgErrorFields(err); + console.log(formatPgError(err)); + // Formatted error with context +} +``` + +## Complete Test Example + +```typescript +import { getConnections, PgTestClient } from 'pgsql-test'; +import { snapshot, IdHash } from 'pgsql-test/utils'; + +let db: PgTestClient; +let teardown: () => Promise; + +beforeAll(async () => { + ({ db, teardown } = await getConnections()); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); +}); + +describe('User queries', () => { + it('returns users with stable snapshot', async () => { + // Seed test data + await db.query(` + INSERT INTO users (email, name) VALUES + ('alice@example.com', 'Alice'), + ('bob@example.com', 'Bob') + `); + + const result = await db.query('SELECT * FROM users ORDER BY email'); + + // Snapshot with ID tracking + const idHash: IdHash = {}; + result.rows.forEach((row, i) => { + idHash[row.id] = i + 1; + }); + + expect(snapshot(result.rows, idHash)).toMatchSnapshot(); + }); +}); +``` + +## Best Practices + +1. **Use snapshot() by default**: Covers most common dynamic fields +2. **Track IDs with IdHash**: When relationships between records matter +3. **Custom pruners for special fields**: Create domain-specific pruners +4. **Order results**: Use ORDER BY for deterministic row order +5. **Prune before comparing**: Apply pruners before any assertions + +## References + +- Related skill: `pgsql-test-seeding` for seeding test data +- Related skill: `pgsql-test-rls` for RLS testing +- Related skill: `drizzle-orm-test` for Drizzle ORM integration diff --git a/.agents/skills/constructive-testing/references/pgsql-test-transactions.md b/.agents/skills/constructive-testing/references/pgsql-test-transactions.md new file mode 100644 index 000000000..600b873cc --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test-transactions.md @@ -0,0 +1,110 @@ +--- +name: pgsql-test-transactions +description: Transaction-local context, the beforeAll gotcha, and PostgreSQL role patterns for RLS testing. Use when debugging "context lost" errors, getting RLS failures in beforeAll, or choosing between pg and db for test operations. +--- + +# Transaction-Local Context & Role Patterns + +## The `set_config(..., true)` Contract + +`pgsql-test`'s `setContext()` uses `set_config('key', 'value', true)` internally. The third parameter `true` makes the setting **transaction-local** — it only persists within the current transaction. + +This works transparently inside test bodies (`it()` blocks) because `beforeEach()` calls `db.begin()` + `db.savepoint()`, creating an active transaction. But in `beforeAll()`, there is no active transaction by default. + +## The `beforeAll` Gotcha + +**Symptom:** You call `setContext()` and then a query, but the query runs without any JWT context — leading to RLS violations or `current_setting()` returning NULL. + +**Root cause:** Without an active transaction, each `db.query()` runs in auto-commit mode. The `ctxQuery()` call (which executes `set_config`) runs in one implicit transaction that immediately commits, and the actual query runs in a separate implicit transaction where the context no longer exists. + +**Fix — wrap `beforeAll` operations in explicit transactions:** + +```typescript +beforeAll(async () => { + ({ db, pg, teardown } = await getConnections()); + + // pg operations auto-commit — no transaction needed + await pg.query(`CREATE TABLE ...`); + await pg.query(`INSERT INTO users ...`); + + // db operations NEED an explicit transaction for context to persist + await db.begin(); + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': ADMIN_ID }); + await db.query(`INSERT INTO app.teams (...) VALUES (...)`); // context persists! + await db.query(`INSERT INTO app.members (...) VALUES (...)`); // still works! + await db.commit(); +}); +``` + +**Without `db.begin()`:** +```typescript +// WRONG — context lost between queries in auto-commit mode +db.setContext({ role: 'authenticated', 'jwt.claims.user_id': ADMIN_ID }); +await db.query(`INSERT INTO app.teams (...) VALUES (...)`); // ERROR: no context +``` + +## Role Patterns for RLS Testing + +PostgreSQL supports multiple roles with different privilege levels. `pgsql-test` gives you `pg` (superuser) and `db` (app-level) to test against them: + +| Role | Client | Bypasses RLS? | Fires Triggers? | When to use | +|------|--------|---------------|-----------------|-------------| +| **superuser** | `pg` | Yes | Yes | Schema setup, DDL, seed data in `beforeAll` | +| **administrator** | `db` + `setContext({ role: 'administrator' })` | Depends on grants | Yes | Elevated data ops that should still exercise triggers | +| **authenticated** | `db` + `setContext({ role: 'authenticated', ... })` | No (full RLS) | Yes | All test queries — this is what real users experience | + +### Why Use `db` with an Elevated Role Instead of `pg` for Data Operations + +The superuser (`pg`) bypasses RLS entirely, but it also means your tests skip the real code path. If your application has triggers that fire on INSERT (e.g., populating audit logs, updating membership tables, or maintaining denormalized data), using `pg` bypasses none of those — triggers still fire for superusers. However, `pg` **does** bypass all RLS policies, which can mask broken policies. + +Using `db` with an elevated role like `administrator`: +- Triggers fire normally +- FK constraints are validated +- The test exercises a more realistic code path +- You can verify that your grant/privilege setup actually works + +```typescript +// Superuser — bypasses RLS, may hide policy bugs +await pg.query(`INSERT INTO app.posts (owner_id, title) VALUES ($1, 'test')`, [ALICE_ID]); + +// Elevated role via db — triggers fire, grants are tested +db.setContext({ role: 'administrator' }); +await db.query(`INSERT INTO app.posts (owner_id, title) VALUES ($1, 'test')`, [ALICE_ID]); +``` + +### Switching Roles Mid-Test + +```typescript +it('admin seeds, user reads', async () => { + // Elevated operation + db.setContext({ role: 'administrator' }); + await db.query(`INSERT INTO app.posts (owner_id, title) VALUES ($1, 'Admin Post')`, [ALICE_ID]); + + // Switch to user — RLS now enforced + db.setContext({ role: 'authenticated', 'jwt.claims.user_id': ALICE_ID }); + const rows = await db.any('SELECT * FROM app.posts'); + expect(rows.length).toBeGreaterThan(0); +}); +``` + +## `pg` vs `db` Quick Reference + +| Operation | Use `pg`? | Use `db`? | Notes | +|-----------|-----------|-----------|-------| +| CREATE TABLE, DDL | ✓ | | Superuser needed for schema changes | +| Seed reference/lookup data | ✓ | | Bootstrap data in `beforeAll` | +| Catalog/information_schema queries | ✓ | | Read-only, no RLS concern | +| Insert data that triggers should process | | ✓ (elevated role) | Ensures triggers fire and side effects happen | +| Grant permissions | | ✓ (elevated role) | Validates grant setup works | +| Test queries (what users see) | | ✓ (authenticated) | RLS must be enforced | + +## Common Pitfalls + +### 1. Forgetting `db.begin()` in `beforeAll` +Context is transaction-local. Without `begin()`, `setContext()` has no effect on subsequent queries. This is the #1 cause of "my context disappeared" bugs. + +### 2. Cross-connection deadlock +Never mix `pg` and `db` in the same test body when both are inside savepoints. Both have separate transactions — data locked by one blocks the other indefinitely. Use `db` with an elevated role for in-test seeding instead. + +### 3. Assuming `pg` tests real behavior +`pg` bypasses RLS, so tests using `pg` for data operations won't catch broken policies. Use `pg` only for setup/DDL, and `db` for everything you want to actually test. diff --git a/.agents/skills/constructive-testing/references/pgsql-test.md b/.agents/skills/constructive-testing/references/pgsql-test.md new file mode 100644 index 000000000..7bf027051 --- /dev/null +++ b/.agents/skills/constructive-testing/references/pgsql-test.md @@ -0,0 +1,214 @@ +--- +name: pgsql-test +description: PostgreSQL integration testing with pgsql-test — RLS policies, seeding, exceptions, snapshots, helpers, JWT context, and complex scenario setup. Use when asked to "test RLS", "test permissions", "seed test data", "snapshot test", "test database", "write integration tests", "test user access", "handle aborted transactions", or when writing any PostgreSQL test with pgsql-test. +compatibility: pgsql-test, Jest/Vitest, PostgreSQL, Node.js 18+ +metadata: + author: constructive-io + version: "2.0.0" +--- + +# pgsql-test (PostgreSQL Integration Testing) + +pgsql-test provides a complete testing toolkit for PostgreSQL — from RLS policy verification and test seeding to snapshot utilities and complex multi-client scenario management. All tests run in transactions with savepoint-based isolation. + +## When to Apply + +Use this skill when: +- **Testing RLS policies:** Verifying user isolation, role-based access, multi-tenant security +- **Seeding test data:** Loading fixtures with loadJson, loadSql, loadCsv +- **Testing exceptions:** Handling aborted transactions when operations should fail +- **Snapshot testing:** Deterministic assertions with pruneIds, pruneDates, etc. +- **Building helpers:** Reusable test functions, constants, assertion utilities +- **JWT context:** Simulating authenticated users with claims for RLS +- **Complex scenarios:** Multi-client patterns, transaction management, cross-connection visibility + +## Quick Start + +```bash +pnpm add -D pgsql-test +``` + +```typescript +import { getConnections, PgTestClient } from 'pgsql-test'; + +let pg: PgTestClient; // Superuser (bypasses RLS) +let db: PgTestClient; // App-level (enforces RLS) +let teardown: () => Promise; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await pg.beforeEach(); + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); + await pg.afterEach(); +}); +``` + +## Core Concepts + +### Two Database Clients + +| Client | Purpose | +|--------|---------| +| `pg` | Superuser — setup/teardown, bypasses RLS | +| `db` | App-level — testing with RLS enforcement | + +### Test Isolation + +Each test runs in a transaction with savepoints: +- `beforeEach()` starts a savepoint +- `afterEach()` rolls back to savepoint +- Tests are completely isolated + +## Testing RLS Policies + +```typescript +// Set user context +db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': userId +}); + +// Users only see their own records +const result = await db.query('SELECT * FROM app.posts'); +expect(result.rows).toHaveLength(1); +``` + +## Handling Expected Failures (Savepoint Pattern) + +When testing operations that should fail, use savepoints to avoid "current transaction is aborted" errors: + +```typescript +it('rejects unauthorized access', async () => { + db.setContext({ role: 'anonymous' }); + + await db.savepoint('unauthorized_access'); + + await expect( + db.query('INSERT INTO app.private_data (secret) VALUES ($1)', ['hack']) + ).rejects.toThrow(/permission denied/); + + await db.rollback('unauthorized_access'); + + // Connection still works + const result = await db.query('SELECT 1 as ok'); + expect(result.rows[0].ok).toBe(1); +}); +``` + +## Seeding Test Data + +```typescript +// Inline JSON (best for small datasets) +await pg.loadJson({ + 'app.users': [ + { id: 'user-1', email: 'alice@example.com', name: 'Alice' } + ] +}); + +// SQL files (best for complex data) +await pg.loadSql([path.join(__dirname, 'fixtures/seed.sql')]); + +// CSV files (best for large datasets, uses COPY) +await pg.loadCsv({ + 'app.categories': path.join(__dirname, 'fixtures/categories.csv') +}); +``` + +## Snapshot Testing + +```typescript +import { snapshot, IdHash } from 'pgsql-test/utils'; + +const result = await db.query('SELECT * FROM users ORDER BY email'); +expect(snapshot(result.rows)).toMatchSnapshot(); + +// With ID tracking +const idHash: IdHash = {}; +result.rows.forEach((row, i) => { idHash[row.id] = i + 1; }); +expect(snapshot(result.rows, idHash)).toMatchSnapshot(); +``` + +Default pruners: `pruneTokens`, `prunePeoplestamps`, `pruneDates`, `pruneIdArrays`, `pruneUUIDs`, `pruneHashes`, `pruneIds`. + +## JWT Context for RLS + +```typescript +// Authenticated user +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId +}); + +// Organization context +db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + 'jwt.claims.org_id': orgId +}); + +// Anonymous +db.setContext({ role: 'anonymous' }); + +// Clear context +db.clearContext(); +``` + +## Reusable Test Helpers + +```typescript +export const TEST_USER_IDS = { + USER_1: '00000000-0000-0000-0000-000000000001', + USER_2: '00000000-0000-0000-0000-000000000002', + ADMIN: '00000000-0000-0000-0000-000000000099', +} as const; + +export function setAuthContext(db: PgTestClient, userId: string): void { + db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': userId, + }); +} +``` + +## Troubleshooting Quick Reference + +| Issue | Quick Fix | +|-------|-----------| +| "current transaction is aborted" | Use savepoint pattern before expected failures | +| Data persists between tests | Ensure `beforeEach`/`afterEach` hooks are set up | +| RLS blocking test inserts | Use `pg` (superuser) for seeding, `db` for testing | +| Foreign key violations in seeding | Load parent tables before child tables | +| Tests interfere with each other | Every test file needs `beforeEach`/`afterEach` hooks | + +## Reference Guide + +Consult these reference files for detailed documentation on specific topics: + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [references/rls.md](references/rls.md) | Testing RLS policies | SELECT/INSERT/UPDATE/DELETE policies, multi-user isolation, anonymous access | +| [references/seeding.md](references/seeding.md) | Seeding test databases | loadJson, loadSql, loadCsv, RLS-aware seeding, fixture organization | +| [references/exceptions.md](references/exceptions.md) | Handling aborted transactions | Savepoint pattern for expected failures, constraint violations, permission errors | +| [references/snapshot.md](references/snapshot.md) | Snapshot testing utilities | pruneIds, pruneDates, IdHash tracking, custom pruners, error formatting | +| [references/helpers.md](references/helpers.md) | Reusable test helpers | Constants, typed helpers, assertion utilities, test-utils organization | +| [references/jwt-context.md](references/jwt-context.md) | JWT claims and role context | setContext API, auth() helper, reading claims in SQL, context timing | +| [references/scenario-setup.md](references/scenario-setup.md) | Complex test scenarios | Two-client pattern, transaction management, publish(), per-describe setup | + +## Cross-References + +Related skills (separate from this skill): +- **`constructive-testing`** — Framework selection guide: which testing framework to use (pgsql-test vs graphile-test vs graphql-test vs server-test) and anti-patterns to avoid +- `pgpm` (`references/testing.md`) — General pgpm test setup and seed adapters +- `drizzle-orm-test` — Testing with Drizzle ORM (uses pgsql-test utilities) +- `constructive-safegres` — Safegres authorization policies that RLS tests validate diff --git a/.agents/skills/constructive-testing/references/supabase-test.md b/.agents/skills/constructive-testing/references/supabase-test.md new file mode 100644 index 000000000..f1e2bb8c6 --- /dev/null +++ b/.agents/skills/constructive-testing/references/supabase-test.md @@ -0,0 +1,383 @@ +--- +name: supabase-test +description: Test Supabase applications with supabase-test. Use when asked to "test Supabase", "test RLS with Supabase", "write Supabase tests", or when testing applications built on Supabase. +compatibility: supabase-test, Jest/Vitest, Supabase, PostgreSQL +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Testing Supabase Applications with supabase-test + +TypeScript-native testing for Supabase with ephemeral databases, RLS testing, and multi-user simulation. + +## When to Apply + +Use this skill when: +- Testing Supabase applications +- Testing RLS policies with Supabase roles (anon, authenticated) +- Simulating authenticated users in tests +- Testing with Supabase's auth.users table + +## Why supabase-test? + +Traditional Supabase testing uses pgTap (SQL-based). supabase-test provides: +- Pure TypeScript tests (Jest/Vitest) +- Multi-user RLS simulation +- Direct Postgres access +- Instant test isolation +- CI-ready ephemeral databases + +## Setup + +### Install Dependencies + +```bash +pnpm add -D supabase-test +``` + +### Initialize Supabase + +```bash +npx supabase init +npx supabase start +``` + +### Configure pgpm (Optional) + +If using pgpm for schema management: +```bash +pgpm init workspace +cd packages/myapp +pgpm init +pgpm install @pgpm/supabase +``` + +## Core Concepts + +### Two Database Clients + +| Client | Purpose | +|--------|---------| +| `pg` | Superuser client for setup/teardown (bypasses RLS) | +| `db` | User client for testing with Supabase roles | + +### Test Isolation + +Each test runs in a transaction: +- `beforeEach()` starts transaction/savepoint +- `afterEach()` rolls back +- Tests are completely isolated + +## Basic Test Structure + +```typescript +import { getConnections, PgTestClient } from 'supabase-test'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); +}); + +it('queries the database', async () => { + const result = await db.query('SELECT 1 + 1 AS sum'); + expect(result.rows[0].sum).toBe(2); +}); +``` + +## Creating Test Users + +Use `insertUser()` to create users in `auth.users`: + +```typescript +import { getConnections, PgTestClient, insertUser } from 'supabase-test'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; + +let alice: any; +let bob: any; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + // Create users in auth.users (requires superuser) + alice = await insertUser(pg, 'alice@example.com', '550e8400-e29b-41d4-a716-446655440001'); + bob = await insertUser(pg, 'bob@example.com', '550e8400-e29b-41d4-a716-446655440002'); +}); +``` + +**Parameters:** +- `pg` - Superuser client (required for auth.users) +- `email` - User's email +- `id` - Optional UUID (auto-generated if omitted) + +## Setting User Context + +Simulate Supabase roles with `setContext()`: + +```typescript +// Authenticated user +db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice.id +}); + +// Anonymous user +db.setContext({ role: 'anon' }); + +// Service role (admin) +db.setContext({ role: 'service_role' }); +``` + +## Testing RLS Policies + +### User Can Access Own Data + +```typescript +it('user can insert own record', async () => { + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice.id + }); + + const result = await db.one(` + INSERT INTO app.posts (title, owner_id) + VALUES ($1, $2) + RETURNING id, title, owner_id + `, ['My Post', alice.id]); + + expect(result.title).toBe('My Post'); + expect(result.owner_id).toBe(alice.id); +}); +``` + +### User Cannot Access Others' Data + +```typescript +it('user cannot see other users data', async () => { + // Bob creates a post + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': bob.id + }); + + await db.one(` + INSERT INTO app.posts (title, owner_id) + VALUES ($1, $2) + RETURNING id + `, ['Bob Post', bob.id]); + + // Alice queries - should not see Bob's post + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice.id + }); + + const result = await db.query('SELECT * FROM app.posts'); + expect(result.rows).toHaveLength(0); +}); +``` + +### Testing Permission Denied + +Use savepoint pattern for expected failures: + +```typescript +it('anonymous cannot insert', async () => { + db.setContext({ role: 'anon' }); + + const point = 'anon_insert'; + await db.savepoint(point); + + await expect( + db.query(`INSERT INTO app.posts (title) VALUES ('Hacked')`) + ).rejects.toThrow(/permission denied/); + + await db.rollback(point); +}); +``` + +## Seeding Test Data + +### With insertUser() + +```typescript +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + alice = await insertUser(pg, 'alice@example.com'); + bob = await insertUser(pg, 'bob@example.com'); +}); +``` + +### With loadJson() + +```typescript +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + alice = await insertUser(pg, 'alice@example.com', '550e8400-e29b-41d4-a716-446655440001'); + + await pg.loadJson({ + 'app.posts': [ + { title: 'Post 1', owner_id: alice.id }, + { title: 'Post 2', owner_id: alice.id } + ] + }); +}); +``` + +### With loadSql() + +```typescript +import path from 'path'; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + await pg.loadSql([ + path.join(__dirname, 'fixtures/seed.sql') + ]); +}); +``` + +### With loadCsv() + +```typescript +import path from 'path'; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + await pg.loadCsv({ + 'app.posts': path.join(__dirname, 'fixtures/posts.csv') + }); +}); +``` + +**Note:** `loadCsv()` bypasses RLS (uses COPY). Always use `pg` client. + +## Query Methods + +| Method | Returns | Use Case | +|--------|---------|----------| +| `db.query(sql, params)` | `{ rows: [...] }` | Multiple rows | +| `db.one(sql, params)` | Single row object | Exactly one row | +| `db.many(sql, params)` | Array of rows | Multiple rows (array) | + +```typescript +// Multiple rows +const result = await db.query('SELECT * FROM app.posts'); +console.log(result.rows); + +// Single row (throws if not exactly one) +const post = await db.one('SELECT * FROM app.posts WHERE id = $1', [postId]); +console.log(post.title); + +// Array of rows +const posts = await db.many('SELECT * FROM app.posts'); +console.log(posts.length); +``` + +## Complete Example + +```typescript +import { getConnections, PgTestClient, insertUser } from 'supabase-test'; + +let pg: PgTestClient; +let db: PgTestClient; +let teardown: () => Promise; + +let alice: any; +let bob: any; + +beforeAll(async () => { + ({ pg, db, teardown } = await getConnections()); + + alice = await insertUser(pg, 'alice@example.com'); + bob = await insertUser(pg, 'bob@example.com'); +}); + +afterAll(async () => { + await teardown(); +}); + +beforeEach(async () => { + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); +}); + +describe('RLS policies', () => { + it('users only see their own posts', async () => { + // Alice creates a post + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice.id + }); + + await db.one(` + INSERT INTO app.posts (title, owner_id) + VALUES ('Alice Post', $1) + RETURNING id + `, [alice.id]); + + // Bob creates a post + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': bob.id + }); + + await db.one(` + INSERT INTO app.posts (title, owner_id) + VALUES ('Bob Post', $1) + RETURNING id + `, [bob.id]); + + // Alice queries - only sees her post + db.setContext({ + role: 'authenticated', + 'request.jwt.claim.sub': alice.id + }); + + const result = await db.many('SELECT title FROM app.posts'); + expect(result).toHaveLength(1); + expect(result[0].title).toBe('Alice Post'); + }); +}); +``` + +## Running Tests + +```bash +# Run all tests +pnpm test + +# Watch mode +pnpm test:watch +``` + +## References + +- Related skill: `pgsql-test-rls` for general RLS testing patterns +- Related skill: `pgsql-test-seeding` for seeding strategies +- Related skill: `pgsql-test-exceptions` for handling aborted transactions diff --git a/.agents/skills/graphile-search.zip b/.agents/skills/graphile-search.zip new file mode 100644 index 000000000..63385959a Binary files /dev/null and b/.agents/skills/graphile-search.zip differ diff --git a/.agents/skills/graphile-search/SKILL.md b/.agents/skills/graphile-search/SKILL.md new file mode 100644 index 000000000..005b66ec4 --- /dev/null +++ b/.agents/skills/graphile-search/SKILL.md @@ -0,0 +1,255 @@ +--- +name: graphile-search +description: Unified PostGraphile v5 search plugin (graphile-search). Consolidates tsvector, BM25, pg_trgm, and pgvector into a single adapter-based architecture with composite searchScore and unifiedSearch fields. Includes codegen SDK query patterns for all search types. Use when asked to "add search to GraphQL", "expose search in PostGraphile", "configure search adapters", "query search via SDK/codegen", or when building search features on a Constructive or PostGraphile v5 stack. +compatibility: PostGraphile v5, graphile-search, graphile-build-pg, graphile-connection-filter +metadata: + author: constructive-io + version: "1.0.0" +--- + +# Graphile Search — Unified Search Plugin + +A single PostGraphile v5 plugin that consolidates tsvector full-text search, BM25 ranked search, pg_trgm fuzzy matching, and pgvector similarity search behind a pluggable adapter architecture. + +**Package:** `graphile-search` ([npm](https://www.npmjs.com/package/graphile-search)) + +## When to Apply + +Use this skill when: +- Adding search capabilities to a PostGraphile v5 / Constructive GraphQL API +- Configuring which search adapters are active +- Understanding how search fields, filters, and scores appear in the GraphQL schema +- Querying search-enabled tables via the generated SDK (after codegen) +- Debugging missing search fields or unexpected trgm behavior +- Building hybrid search combining multiple algorithms + +## Architecture Overview + +Instead of separate plugins per algorithm, `graphile-search` uses a single `UnifiedSearchPlugin` with pluggable **adapters**: + +``` +UnifiedSearchPlugin + ├── TsvectorAdapter (keyword search with stemming) + ├── Bm25Adapter (relevance-ranked document search) + ├── TrgmAdapter (fuzzy matching, typo tolerance) + └── PgvectorAdapter (semantic/embedding similarity + chunk-aware RAG) +``` + +The pgvector adapter also supports **chunk-aware search** via the `@hasChunks` smart tag — transparently querying across parent and chunk embeddings and returning the minimum distance. See `references/pgvector-adapter.md` for details. + +Each adapter implements the `SearchAdapter` interface: +- **`detectColumns()`** — finds searchable columns on a table (e.g. tsvector columns, columns with BM25 indexes) +- **`registerTypes()`** — registers any custom GraphQL types (e.g. filter input types) +- **`applyFilter()`** — generates the SQL WHERE clause when a filter is used +- **`scoreSemantics`** — declares metric name, direction, and range bounds for score normalization + +The plugin also ships codec plugins that teach PostGraphile about custom types: `TsvectorCodecPlugin`, `Bm25CodecPlugin`, `VectorCodecPlugin`. + +## Quick Start + +```typescript +import { UnifiedSearchPreset } from 'graphile-search'; + +const preset: GraphileConfig.Preset = { + extends: [ + // ... your other presets + UnifiedSearchPreset(), // all 4 adapters enabled by default + ], +}; +``` + +This single preset includes: +- All 4 adapter plugins (tsvector, BM25, trgm, pgvector) +- Codec plugins (TsvectorCodecPlugin, Bm25CodecPlugin, VectorCodecPlugin) +- Connection filter operator factories (matches, similarTo, wordSimilarTo) +- Composite `searchScore` and `unifiedSearch` fields + +## Preset Options + +```typescript +UnifiedSearchPreset({ + // Enable/disable individual adapters (all default to true) + tsvector: true, // or { filterPrefix: 'fullText', tsConfig: 'english' } + bm25: true, // or { filterPrefix: 'bm25' } + trgm: true, // or { defaultThreshold: 0.2, filterPrefix: 'trgm' } + pgvector: true, // or { defaultMetric: 'L2', filterPrefix: 'vector' } + + // Composite fields + enableSearchScore: true, // expose searchScore (0..1) on search-enabled tables + enableUnifiedSearch: true, // expose unifiedSearch composite filter + + // Weights for composite searchScore + searchScoreWeights: { + tsv: 0.3, + bm25: 0.4, + trgm: 0.2, + vector: 0.1, + }, + + // Scalar naming + fullTextScalarName: 'FullText', // GraphQL scalar name for tsvector columns + tsConfig: 'english', // PostgreSQL text search configuration +}) +``` + +## What Gets Generated in GraphQL + +When the plugin detects search infrastructure on a table, it adds: + +### Per-Adapter Score Fields (on row types) + +| Adapter | Example Field | Type | Description | +|---------|--------------|------|-------------| +| tsvector | `searchTsvRank` | `Float` | ts_rank score (0..1, higher = better) | +| BM25 | `bodyBm25Score` | `Float` | BM25 relevance score (negative, more negative = better) | +| trgm | `titleTrgmSimilarity` | `Float` | Trigram similarity (0..1, higher = better) | +| pgvector | `embeddingVectorDistance` | `Float` | Vector distance (0..inf, lower = closer) | + +**Naming pattern:** `{camelCase(column)}{Algorithm}{Metric}` + +### Composite searchScore Field + +```graphql +type Article { + # ... regular fields + searchScore: Float # Normalized 0..1, higher = more relevant +} +``` + +Computed by normalizing all active search signals to 0..1 and averaging them. Returns `null` when no search filters are active. + +### Per-Adapter Filter Fields (on connection filters) + +| Adapter | Filter Field | Input Type | +|---------|-------------|------------| +| tsvector | `fullTextSearchTsv` | `String` (search query) | +| BM25 | `bm25Body` | `Bm25SearchInput` (`{ query, indexName }`) | +| trgm | `trgmTitle` | `TrgmSearchInput` (`{ value, threshold? }`) | +| pgvector | `vectorEmbedding` | `VectorSearchInput` (`{ query: [Float!]!, metric? }`) | + +**Naming pattern:** `{filterPrefix}{CamelCase(column)}` + +### unifiedSearch Composite Filter + +Fans out a single text query to all text-compatible adapters (tsvector, BM25, trgm) simultaneously, combining with OR: + +```graphql +query { + allArticles(where: { unifiedSearch: "postgres tutorial" }) { + nodes { + title + searchScore # composite relevance across all text search signals + } + } +} +``` + +### OrderBy Enum Values + +Each adapter adds ASC/DESC ordering for its score metric: + +```graphql +enum ArticlesOrderBy { + SEARCH_TSV_RANK_ASC + SEARCH_TSV_RANK_DESC + BODY_BM25_SCORE_ASC + BODY_BM25_SCORE_DESC + # ... etc +} +``` + +### StringTrgmFilter Type + +For tables that qualify for trgm (see "Trgm Scoping" below), string columns get a `StringTrgmFilter` type instead of the regular `StringFilter`. This adds two operators: + +```graphql +input StringTrgmFilter { + # ... all standard StringFilter operators (equalTo, includes, etc.) + similarTo: TrgmSearchInput # pg_trgm similarity() + wordSimilarTo: TrgmSearchInput # pg_trgm word_similarity() +} + +input TrgmSearchInput { + value: String! + threshold: Float # default 0.3 +} +``` + +## Trgm Scoping — Supplementary Adapter Pattern + +Trgm doesn't activate on every table with text columns. It uses a **supplementary adapter** pattern: + +1. **Primary adapters** (tsvector, BM25, pgvector) run first and detect columns on each table +2. **Supplementary adapters** (trgm) only run if at least one adapter with `isIntentionalSearch: true` found columns on that table +3. pgvector sets `isIntentionalSearch: false` — embeddings alone don't trigger trgm +4. Only tsvector and BM25 count as "intentional search" and trigger trgm activation + +This means: +- Table with tsvector column → trgm activates on its text columns +- Table with BM25 index → trgm activates on its text columns +- Table with only pgvector → trgm does NOT activate +- Table with no search infrastructure → trgm does NOT activate + +### Opt-in via @trgmSearch Smart Tag + +To force trgm on a table that has no intentional search (or only pgvector), use the `@trgmSearch` smart tag: + +```sql +-- Table-level: enable trgm on all text columns +COMMENT ON TABLE app_public.contacts IS E'@trgmSearch'; + +-- Column-level: enable trgm on specific columns +COMMENT ON COLUMN app_public.contacts.name IS E'@trgmSearch'; +``` + +## Adapter Details + +Each adapter is documented in its own reference file: + +- `references/tsvector-adapter.md` — tsvector full-text search adapter +- `references/bm25-adapter.md` — BM25 ranked search adapter +- `references/trgm-adapter.md` — pg_trgm fuzzy matching adapter +- `references/pgvector-adapter.md` — pgvector similarity search adapter +- `references/search-adapter-interface.md` — SearchAdapter interface specification + +## Querying Search via Codegen SDK + +After running `cnc codegen`, the generated SDK client exposes search filters, score fields, and orderBy enums. See `references/codegen-sdk-queries.md` for complete query patterns covering: + +- **Composite fields** — `unifiedSearch` (multi-strategy filter) and `searchScore` (0..1 relevance) +- **TSVector queries** — `fullTextSearchTsv`, `searchTsvRank`, pagination, combined filters +- **BM25 queries** — `bm25Content`, `bm25ContentScore` (negative, sort ASC) +- **Trigram queries** — `similarTo`/`wordSimilarTo` via `StringTrgmFilter`, adapter-level `trgmTitle`, ILIKE +- **pgvector queries** — `vectorEmbedding`, `embeddingDistance`, distance metrics (COSINE/L2/IP) +- **Chunk-aware search** — `includeChunks` toggle for RAG tables with `@hasChunks`, transparent parent + chunk distance +- **Multi-strategy patterns** — fuzzy fallback, autocomplete pipeline, semantic + keyword hybrid + +## Score Semantics + +Each adapter declares how its scores behave for normalization in `searchScore`: + +| Adapter | Metric | Lower is Better? | Range | +|---------|--------|-------------------|-------| +| tsvector | `rank` | No (higher = better) | [0, 1] | +| BM25 | `score` | Yes (more negative = better) | Unbounded | +| trgm | `similarity` | No (higher = better) | [0, 1] | +| pgvector | `distance` | Yes (closer = better) | Unbounded | + +Bounded ranges use linear normalization. Unbounded ranges use sigmoid normalization (`1 / (1 + |score|)`). + +## Common Pitfalls + +| Issue | Cause | Fix | +|---|---|---| +| No search fields on table | No search infrastructure detected | Add tsvector column, BM25 index, or vector column | +| trgm operators missing | Table has no intentional search | Add tsvector/BM25, or use `@trgmSearch` smart tag | +| `searchScore` is null | No search filters active in query | Add a search filter (unifiedSearch, bm25Body, etc.) | +| `includeChunks` field missing | No `@hasChunks` tables in schema | Add `@hasChunks` smart tag to parent table codec | +| `Unknown type "FullText"` | TsvectorCodecPlugin not loaded | Use `UnifiedSearchPreset()` which includes all codecs | +| `Unknown type "Vector"` | VectorCodecPlugin not loaded | Use `UnifiedSearchPreset()` which includes all codecs | +| Duplicate type errors | Multiple search presets | Use only `UnifiedSearchPreset()`, not individual presets | + +## Related Skills + +- `constructive-db-search` (private) — SQL-level search strategies and metaschema integration +- `constructive-graphql-codegen` — code generation from search-enabled schemas diff --git a/.agents/skills/graphile-search/references/bm25-adapter.md b/.agents/skills/graphile-search/references/bm25-adapter.md new file mode 100644 index 000000000..55394f74e --- /dev/null +++ b/.agents/skills/graphile-search/references/bm25-adapter.md @@ -0,0 +1,81 @@ +# BM25 Adapter + +Relevance-ranked search using BM25 scoring via the `pg_textsearch` extension. Best for document search where ranking quality matters. + +## How It Works + +The BM25 adapter: +1. **Gathers** BM25 index information during the Graphile gather phase (via `Bm25CodecPlugin`) +2. **Detects** text columns that have a BM25 index (stored in the `bm25IndexStore`) +3. **Registers** a `Bm25SearchInput` type (`{ query: String!, indexName: String }`) +4. **Generates** BM25 score as a computed field and orderBy enum + +## Adapter Configuration + +```typescript +import { createBm25Adapter } from 'graphile-search'; + +createBm25Adapter({ + filterPrefix: 'bm25', // default: 'bm25' +}) +``` + +## Generated GraphQL + +Given a table with a BM25 index on the `content` column: + +### Filter + +```graphql +query { + allDocuments(where: { bm25Content: { query: "database indexing" } }) { + nodes { + title + contentBm25Score + } + } +} +``` + +### Score Field + +```graphql +type Document { + contentBm25Score: Float # BM25 score (negative, more negative = more relevant) +} +``` + +### OrderBy + +```graphql +enum DocumentsOrderBy { + CONTENT_BM25_SCORE_ASC # most relevant first (most negative) + CONTENT_BM25_SCORE_DESC # least relevant first +} +``` + +## Score Semantics + +| Property | Value | +|----------|-------| +| Metric | `score` | +| Lower is better | Yes (more negative = more relevant) | +| Range | Unbounded (uses sigmoid normalization in searchScore) | + +## Adapter Flags + +| Flag | Value | +|------|-------| +| `isSupplementary` | `false` (primary adapter) | +| `isIntentionalSearch` | `true` (triggers supplementary adapters like trgm) | +| `supportsTextSearch` | `true` (included in unifiedSearch composite filter) | + +## Prerequisites + +- `pg_textsearch` extension enabled (pre-enabled in Constructive stack) +- A BM25 index on a text column (`CREATE INDEX ... USING bm25 ...`) +- `Bm25CodecPlugin` loaded (included automatically by `UnifiedSearchPreset`) + +## Index Discovery + +The adapter reads from `bm25IndexStore`, which is populated during the Graphile gather phase by `Bm25CodecPlugin`. The gather hook introspects `pg_am` and `pg_class` to find indexes with access method `bm25`, then maps them back to table columns. diff --git a/.agents/skills/graphile-search/references/codegen-sdk-queries.md b/.agents/skills/graphile-search/references/codegen-sdk-queries.md new file mode 100644 index 000000000..f691b13b1 --- /dev/null +++ b/.agents/skills/graphile-search/references/codegen-sdk-queries.md @@ -0,0 +1,563 @@ +# Querying Search via Generated SDK (Codegen) + +After running `cnc codegen`, the generated SDK client exposes search filters, score fields, and orderBy enums for every search-enabled table. This reference shows how to query each search type. + +--- + +## Setup + +```typescript +import { createClient } from '@constructive-io/sdk'; + +const db = createClient({ + endpoint: process.env.GRAPHQL_ENDPOINT || 'https://api.constructive.io/graphql', + headers: { + Authorization: `Bearer ${process.env.API_TOKEN}`, + }, +}); +``` + +--- + +## Composite Fields (Unified Search) + +### unifiedSearch — Multi-Strategy Filter + +A single filter that fans the same text query to all text-compatible adapters (tsvector, BM25, trgm) simultaneously: + +```typescript +const result = await db.article.findMany({ + where: { + unifiedSearch: 'postgres tutorial', + }, + select: { + id: true, + title: true, + searchScore: true, // combined 0..1 relevance across all signals + }, +}).execute(); + +if (result.ok) { + result.data.articles.nodes.forEach(a => { + console.log(`${a.title} (score: ${a.searchScore})`); + }); +} +``` + +pgvector is excluded from `unifiedSearch` because it requires a vector array input, not text. + +### searchScore — Composite Relevance + +A normalized 0..1 relevance score that combines all active search signals. Returns `null` when no search filters are active. + +```typescript +const result = await db.article.findMany({ + where: { + fullTextSearchTsv: 'postgres tutorial', + }, + orderBy: 'SEARCH_TSV_RANK_DESC', + select: { + id: true, + title: true, + searchTsvRank: true, // per-adapter score + searchScore: true, // composite: normalized 0..1 + }, +}).execute(); +``` + +The composite score normalizes each adapter's raw score to 0..1 (bounded ranges use linear normalization, unbounded use sigmoid) and averages them. Custom weights can be configured in the preset. + +--- + +## TSVector Queries + +### Basic Full-Text Search + +```typescript +const result = await db.article.findMany({ + where: { + fullTextSearchTsv: 'postgres full text', + }, + orderBy: 'SEARCH_TSV_RANK_DESC', + select: { + id: true, + title: true, + searchTsvRank: true, + }, +}).execute(); + +if (result.ok) { + result.data.articles.nodes.forEach(a => { + console.log(`${a.title} (rank: ${a.searchTsvRank})`); + }); +} +``` + +### Search with Pagination + +```typescript +const result = await db.article.findMany({ + where: { + fullTextSearchTsv: 'database indexing', + }, + orderBy: 'SEARCH_TSV_RANK_DESC', + first: 10, + after: cursor, + select: { + id: true, + title: true, + searchTsvRank: true, + }, +}).execute(); +``` + +### Combining Search with Other Filters + +```typescript +const result = await db.article.findMany({ + where: { + fullTextSearchTsv: 'postgres', + isPublished: { equalTo: true }, + category: { equalTo: 'tech' }, + }, + orderBy: 'SEARCH_TSV_RANK_DESC', + first: 20, + select: { + id: true, + title: true, + category: true, + searchTsvRank: true, + }, +}).execute(); +``` + +### Field Naming Convention + +| DB Column | Filter Field | Score Field | OrderBy | +|-----------|-------------|-------------|---------| +| `search_tsv` | `fullTextSearchTsv` | `searchTsvRank` | `SEARCH_TSV_RANK_ASC/DESC` | +| `body_tsv` | `fullTextBodyTsv` | `bodyTsvRank` | `BODY_TSV_RANK_ASC/DESC` | + +**Pattern:** +- Filter: `fullText` + camelCase(column) +- Score: camelCase(column) + `Rank` (Float, higher = better, null when no filter active) +- OrderBy: SCREAMING_SNAKE(column) + `_RANK_ASC/DESC` + +--- + +## BM25 Queries + +### Basic BM25 Search + +```typescript +const result = await db.document.findMany({ + where: { + bm25Content: { query: 'postgres full text search' }, + }, + orderBy: 'BM25_CONTENT_SCORE_ASC', + select: { + id: true, + title: true, + bm25ContentScore: true, + }, +}).execute(); + +if (result.ok) { + result.data.documents.nodes.forEach(d => { + console.log(`${d.title} (score: ${d.bm25ContentScore})`); + }); +} +``` + +**Important:** BM25 scores are negative — more negative means more relevant. Sort ascending (`_ASC`) to get the best matches first. + +### Search with Pagination + +```typescript +const result = await db.document.findMany({ + where: { + bm25Content: { query: 'machine learning' }, + }, + orderBy: 'BM25_CONTENT_SCORE_ASC', + first: 10, + after: cursor, + select: { + id: true, + title: true, + bm25ContentScore: true, + }, +}).execute(); +``` + +### Combining BM25 with Other Filters + +```typescript +const result = await db.document.findMany({ + where: { + bm25Content: { query: 'kubernetes deployment' }, + isPublished: { equalTo: true }, + category: { equalTo: 'devops' }, + }, + orderBy: 'BM25_CONTENT_SCORE_ASC', + first: 20, + select: { + id: true, + title: true, + category: true, + bm25ContentScore: true, + }, +}).execute(); +``` + +### Field Naming Convention + +| DB Column | Filter Field | Score Field | OrderBy | +|-----------|-------------|-------------|---------| +| `content` | `bm25Content` | `bm25ContentScore` | `BM25_CONTENT_SCORE_ASC/DESC` | +| `body` | `bm25Body` | `bm25BodyScore` | `BM25_BODY_SCORE_ASC/DESC` | + +**Pattern:** +- Filter: `bm25` + CamelCase(column) — accepts `{ query: String }` input +- Score: `bm25` + CamelCase(column) + `Score` (Float, negative, more negative = better, null when no filter active) +- OrderBy: `BM25_` + SCREAMING_SNAKE(column) + `_SCORE_ASC/DESC` + +--- + +## Trigram Queries + +### Similarity Search (via StringTrgmFilter) + +On qualifying tables (those with intentional search infrastructure), string columns get `similarTo` and `wordSimilarTo` operators: + +```typescript +// similarTo: overall trigram similarity +const result = await db.article.findMany({ + where: { + title: { similarTo: { value: 'postgre', threshold: 0.2 } }, + }, + first: 20, + select: { + id: true, + title: true, + titleTrgmSimilarity: true, // 0..1, higher = more similar + }, +}).execute(); + +if (result.ok) { + result.data.articles.nodes.forEach(a => { + console.log(`${a.title} (similarity: ${a.titleTrgmSimilarity})`); + }); +} +``` + +```typescript +// wordSimilarTo: best substring similarity (better for search-as-you-type) +const result = await db.article.findMany({ + where: { + title: { wordSimilarTo: { value: 'postgres', threshold: 0.3 } }, + }, + orderBy: 'TITLE_TRGM_SIMILARITY_DESC', + first: 10, + select: { + id: true, + title: true, + titleTrgmSimilarity: true, + }, +}).execute(); +``` + +### Adapter-Level Filter + +```typescript +const result = await db.article.findMany({ + where: { + trgmTitle: { value: 'postgre' }, + }, + orderBy: 'TITLE_TRGM_SIMILARITY_DESC', + select: { + id: true, + title: true, + titleTrgmSimilarity: true, + }, +}).execute(); +``` + +### ILIKE Search (GIN-Accelerated) + +The GIN trigram index still accelerates `ILIKE` queries via standard filter operators: + +```typescript +const result = await db.article.findMany({ + where: { + title: { likeInsensitive: '%postgres%' }, + }, + first: 20, + select: { + id: true, + title: true, + }, +}).execute(); +``` + +### Field Naming Convention + +| DB Column | Adapter Filter | Score Field | OrderBy | +|-----------|---------------|-------------|---------| +| `title` | `trgmTitle` | `titleTrgmSimilarity` | `TITLE_TRGM_SIMILARITY_ASC/DESC` | +| `name` | `trgmName` | `nameTrgmSimilarity` | `NAME_TRGM_SIMILARITY_ASC/DESC` | + +**Pattern:** +- Adapter filter: `trgm` + CamelCase(column) — accepts `{ value: String, threshold?: Float }` +- Connection filter: column gets `StringTrgmFilter` with `similarTo`/`wordSimilarTo` operators +- Score: camelCase(column) + `TrgmSimilarity` (Float, 0..1, higher = better) +- OrderBy: SCREAMING_SNAKE(column) + `_TRGM_SIMILARITY_ASC/DESC` + +--- + +## pgvector Queries + +### Basic Nearest Neighbor Search + +```typescript +const result = await db.document.findMany({ + where: { + vectorEmbedding: { + vector: queryVector, + metric: 'COSINE', + }, + }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + first: 10, + select: { + id: true, + title: true, + embeddingDistance: true, + }, +}).execute(); + +if (result.ok) { + result.data.documents.nodes.forEach(d => { + console.log(`${d.title} (distance: ${d.embeddingDistance})`); + }); +} +``` + +### Search with Distance Threshold + +```typescript +const result = await db.document.findMany({ + where: { + vectorEmbedding: { + vector: queryVector, + metric: 'COSINE', + distance: 0.5, // filter out results beyond threshold + }, + }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + select: { + id: true, + title: true, + embeddingDistance: true, + }, +}).execute(); +``` + +### Combining Vector Search with Other Filters + +```typescript +const result = await db.document.findMany({ + where: { + vectorEmbedding: { + vector: queryVector, + metric: 'COSINE', + }, + isPublished: { equalTo: true }, + category: { equalTo: 'tech' }, + }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + first: 10, + select: { + id: true, + title: true, + category: true, + embeddingDistance: true, + }, +}).execute(); +``` + +### Distance Metrics + +| Metric | Range | Meaning | Sort Direction | +|--------|-------|---------|----------------| +| `COSINE` | 0 to 2 | 0 = identical, 2 = opposite | ASC = most similar | +| `L2` | 0 to infinity | 0 = identical | ASC = most similar | +| `IP` | -infinity to 0 | More negative = more similar | ASC = most similar | + +### Field Naming Convention + +| DB Column | Filter Field | Distance Field | OrderBy | +|-----------|-------------|----------------|---------| +| `embedding` | `vectorEmbedding` | `embeddingDistance` | `EMBEDDING_DISTANCE_ASC/DESC` | +| `content_vec` | `vectorContentVec` | `contentVecDistance` | `CONTENT_VEC_DISTANCE_ASC/DESC` | + +**Pattern:** +- Filter: `vector` + CamelCase(column) — accepts `{ vector: [Float!]!, metric?: String, distance?: Float, includeChunks?: Boolean }` +- Distance: camelCase(column) + `Distance` (Float, lower = closer, null when no filter active) +- OrderBy: SCREAMING_SNAKE(column) + `_DISTANCE_ASC/DESC` + +### Chunk-Aware Vector Search + +Tables with the `@hasChunks` smart tag automatically get chunk-aware search. The distance returned is `LEAST(parent_distance, MIN(chunk_distance))` — the best match across the document embedding and all chunk embeddings. + +Chunk search is **on by default** for `@hasChunks` tables. No extra code is needed: + +```typescript +// Chunk-aware search — ON by default for @hasChunks tables +// Distance = LEAST(parent embedding distance, closest chunk distance) +const result = await db.document.findMany({ + where: { + vectorEmbedding: { + vector: queryVector, + metric: 'COSINE', + // includeChunks defaults to true when @hasChunks is present + }, + }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + first: 10, + select: { + id: true, + title: true, + embeddingDistance: true, // best distance across parent + all chunks + }, +}).execute(); + +if (result.ok) { + result.data.documents.nodes.forEach(d => { + console.log(`${d.title} (distance: ${d.embeddingDistance})`); + }); +} +``` + +### Opt Out of Chunk Search + +Set `includeChunks: false` to only search the parent embedding: + +```typescript +const result = await db.document.findMany({ + where: { + vectorEmbedding: { + vector: queryVector, + metric: 'COSINE', + includeChunks: false, // only use parent embedding + }, + }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + first: 10, + select: { + id: true, + title: true, + embeddingDistance: true, // parent distance only + }, +}).execute(); +``` + +### Chunk-Aware Search with Distance Threshold + +The distance threshold applies to the combined (chunk-aware) distance: + +```typescript +const result = await db.document.findMany({ + where: { + vectorEmbedding: { + vector: queryVector, + metric: 'COSINE', + distance: 0.3, // threshold applies to LEAST(parent, chunk) + }, + isPublished: { equalTo: true }, + }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + first: 20, + select: { + id: true, + title: true, + embeddingDistance: true, + searchScore: true, // composite 0..1 relevance + }, +}).execute(); +``` + +> **Note:** `includeChunks` only appears on `VectorNearbyInput` when at least one table in the schema has the `@hasChunks` smart tag. For tables without chunks, the field is absent and vector search behaves as standard parent-only search. + +--- + +## Multi-Strategy Patterns + +### Fuzzy Fallback (BM25 + Trigram) + +```typescript +// Primary: BM25 search +const bm25Result = await db.document.findMany({ + where: { bm25Content: { query: userQuery } }, + orderBy: 'BM25_CONTENT_SCORE_ASC', + first: 10, + select: { id: true, title: true, bm25ContentScore: true }, +}).execute(); + +const bm25Docs = bm25Result.ok ? bm25Result.data.documents.nodes : []; + +// Fallback: trigram if BM25 returned few results +if (bm25Docs.length < 3) { + const fuzzyResult = await db.document.findMany({ + where: { + title: { similarTo: { value: userQuery, threshold: 0.15 } }, + }, + orderBy: 'TITLE_TRGM_SIMILARITY_DESC', + first: 10, + select: { id: true, title: true, titleTrgmSimilarity: true }, + }).execute(); +} +``` + +### Autocomplete Pipeline (Trigram + TSVector) + +```typescript +// Stage 1: Autocomplete (on every keystroke) +const autocomplete = await db.article.findMany({ + where: { + title: { similarTo: { value: partialInput, threshold: 0.15 } }, + }, + orderBy: 'TITLE_TRGM_SIMILARITY_DESC', + first: 5, + select: { id: true, title: true, titleTrgmSimilarity: true }, +}).execute(); + +// Stage 2: Full search (on form submit) +const search = await db.article.findMany({ + where: { fullTextSearchTsv: fullQuery }, + orderBy: 'SEARCH_TSV_RANK_DESC', + first: 20, + select: { id: true, title: true, searchTsvRank: true }, +}).execute(); +``` + +### Semantic + Keyword Hybrid (pgvector + TSVector) + +```typescript +// Semantic search +const semanticResult = await db.document.findMany({ + where: { vectorEmbedding: { vector: queryEmbedding, metric: 'COSINE' } }, + orderBy: 'EMBEDDING_DISTANCE_ASC', + first: 20, + select: { id: true, title: true, embeddingDistance: true }, +}).execute(); + +// Keyword search +const keywordResult = await db.document.findMany({ + where: { fullTextSearchTsv: userQuery }, + orderBy: 'SEARCH_TSV_RANK_DESC', + first: 20, + select: { id: true, title: true, searchTsvRank: true }, +}).execute(); + +// Merge by reciprocal rank fusion or custom weighting +``` diff --git a/.agents/skills/graphile-search/references/pgvector-adapter.md b/.agents/skills/graphile-search/references/pgvector-adapter.md new file mode 100644 index 000000000..bc9e40bcf --- /dev/null +++ b/.agents/skills/graphile-search/references/pgvector-adapter.md @@ -0,0 +1,209 @@ +# pgvector Adapter + +Semantic similarity search using the `pgvector` extension. Supports cosine, L2, and inner product distance metrics for embedding-based search and RAG. + +## How It Works + +The pgvector adapter: +1. **Detects** `vector(N)` columns on tables (via `VectorCodecPlugin` codec registration) +2. **Registers** `VectorSearchInput` type (`{ query: [Float!]!, metric: VectorMetric }`) +3. **Generates** distance score as a computed field and orderBy enum + +## Adapter Configuration + +```typescript +import { createPgvectorAdapter } from 'graphile-search'; + +createPgvectorAdapter({ + filterPrefix: 'vector', // default: 'vector' + defaultMetric: 'COSINE', // default: 'COSINE' (COSINE | L2 | IP) +}) +``` + +## Generated GraphQL + +Given a table with an `embedding vector(768)` column: + +### Filter + +```graphql +query { + allDocuments(where: { + vectorEmbedding: { query: [0.1, 0.2, ...], metric: COSINE } + }) { + nodes { + title + embeddingVectorDistance + } + } +} +``` + +### Score Field + +```graphql +type Document { + embeddingVectorDistance: Float # distance score, lower = more similar +} +``` + +### OrderBy + +```graphql +enum DocumentsOrderBy { + EMBEDDING_VECTOR_DISTANCE_ASC # most similar first + EMBEDDING_VECTOR_DISTANCE_DESC # least similar first +} +``` + +## Distance Metrics + +| Metric | Operator | Range | Notes | +|--------|----------|-------|-------| +| `COSINE` | `<=>` | 0-2 | 0 = identical; best for normalized embeddings | +| `L2` | `<->` | 0-inf | Euclidean distance | +| `IP` | `<#>` | -inf to 0 | Negative inner product; less negative = more similar | + +## Score Semantics + +| Property | Value | +|----------|-------| +| Metric | `distance` | +| Lower is better | Yes (closer = more similar) | +| Range | Unbounded (uses sigmoid normalization in searchScore) | + +## Adapter Flags + +| Flag | Value | +|------|-------| +| `isSupplementary` | `false` (primary adapter) | +| `isIntentionalSearch` | `false` (embeddings do NOT trigger trgm) | +| `supportsTextSearch` | `false` (not included in unifiedSearch composite) | + +pgvector is the only primary adapter that sets `isIntentionalSearch: false`. This is because vector embeddings operate on a different domain than text search — a table with only pgvector columns shouldn't get trgm similarity fields on its text columns. + +## Chunk-Aware Search (`@hasChunks`) + +Tables with long-form content (documents, articles, etc.) often split text into **chunks**, each with its own embedding. The pgvector adapter transparently queries across parent and chunk embeddings when the `@hasChunks` smart tag is present on the table's codec. + +### How It Works + +1. The parent table has a `vector(N)` column (the document-level embedding) +2. A separate chunks table stores per-chunk embeddings with a foreign key back to the parent +3. The `@hasChunks` smart tag tells the adapter where to find the chunks table +4. At query time, the adapter computes `LEAST(parent_distance, MIN(chunk_distance))` — the best match across all embeddings + +### Smart Tag Configuration + +Set `@hasChunks` on the parent table codec as a JSON object: + +```json +{ + "chunksTable": "documents_chunks", + "parentFk": "parent_id", + "parentPk": "id", + "embeddingField": "embedding" +} +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `chunksTable` | *(required)* | Name of the chunks table | +| `chunksSchema` | parent table's schema | Schema of the chunks table | +| `parentFk` | `parent_id` | Column in chunks table that references the parent | +| `parentPk` | `id` | Primary key column on the parent table | +| `embeddingField` | `embedding` | Vector column in the chunks table | + +In Constructive, the `SearchVector` node type with chunks enabled automatically creates the chunks table and wires up the relationship. The smart tag is applied via a Graphile plugin or smart tags file. + +### `includeChunks` Filter Field + +When `@hasChunks` is detected, `VectorNearbyInput` gains an `includeChunks` boolean field: + +```graphql +input VectorNearbyInput { + vector: [Float!]! + metric: VectorMetric + distance: Float + includeChunks: Boolean # only present when @hasChunks is on the table +} +``` + +- **`true` (default for `@hasChunks` tables):** Distance = `LEAST(parent_distance, MIN(chunk_distance))` +- **`false`:** Distance = parent embedding distance only + +### Generated SQL + +When `includeChunks` is active, the adapter generates: + +```sql +LEAST( + COALESCE(parent.embedding <=> $query, 'Infinity'::float), + COALESCE( + (SELECT MIN(c.embedding <=> $query) + FROM documents_chunks AS c + WHERE c.parent_id = parent.id), + 'Infinity'::float + ) +) +``` + +`COALESCE` handles cases where the parent or chunks may not have embeddings. + +### GraphQL Query Examples + +```graphql +# Chunk-aware (default) — returns best distance across parent + all chunks +query { + allDocuments(where: { + vectorEmbedding: { vector: [0.1, 0.2, ...], metric: COSINE } + }) { + nodes { + title + embeddingVectorDistance # LEAST(parent, closest chunk) + } + } +} + +# Parent-only — opt out of chunk search +query { + allDocuments(where: { + vectorEmbedding: { vector: [0.1, 0.2, ...], metric: COSINE, includeChunks: false } + }) { + nodes { + title + embeddingVectorDistance # parent distance only + } + } +} +``` + +## VectorCodecPlugin + +The `VectorCodecPlugin` (included in `UnifiedSearchPreset`) teaches PostGraphile about the `vector` PostgreSQL type: + +- **Wire format:** PostgreSQL sends vectors as text `[0.1,0.2,...,0.768]` +- **JavaScript:** `number[]` +- **GraphQL scalar:** `Vector` (serialized as `[Float]`) + +Without this plugin, PostGraphile silently ignores `vector(N)` columns. + +## Codegen — Vector Scalar Mapping + +When generating typed SDK code, the `Vector` scalar maps to `number[]`: + +```typescript +// graphql-codegen.config.ts +scalars: { + Vector: 'number[]', +} +``` + +With `@constructive-io/graphql-codegen >= 4.6.0`, this mapping is built-in. + +## Prerequisites + +- `pgvector` extension enabled (pre-enabled in Constructive stack) +- A `vector(N)` column on the table +- An HNSW or IVFFlat index for performance (`CREATE INDEX ... USING hnsw (embedding vector_cosine_ops)`) +- `VectorCodecPlugin` loaded (included automatically by `UnifiedSearchPreset`) diff --git a/.agents/skills/graphile-search/references/search-adapter-interface.md b/.agents/skills/graphile-search/references/search-adapter-interface.md new file mode 100644 index 000000000..0085d4078 --- /dev/null +++ b/.agents/skills/graphile-search/references/search-adapter-interface.md @@ -0,0 +1,134 @@ +# SearchAdapter Interface + +The `SearchAdapter` interface is the contract each search algorithm implements to plug into the unified search plugin. + +## Interface Definition + +```typescript +interface SearchAdapter { + /** Unique identifier (e.g. 'tsv', 'bm25', 'trgm', 'vector') */ + name: string; + + /** Score semantics for normalization */ + scoreSemantics: ScoreSemantics; + + /** + * When true, only activates on tables that already have at least one + * column detected by an adapter whose isIntentionalSearch is true. + * Prevents trgm from adding fields to every table with text columns. + * @default false + */ + isSupplementary?: boolean; + + /** + * When true, this adapter's presence triggers supplementary adapters. + * tsvector and BM25 set this to true. pgvector sets this to false + * (embeddings are not text search). + * @default true + */ + isIntentionalSearch?: boolean; + + /** Filter prefix for connection filter field names (e.g. 'bm25' -> bm25Body) */ + filterPrefix: string; + + /** + * Whether this adapter supports plain text queries. + * If true, columns are included in the unifiedSearch composite filter. + * pgvector sets this to false (requires vector input, not text). + * @default false + */ + supportsTextSearch?: boolean; + + /** Detect searchable columns on a given codec/table */ + detectColumns(codec: PgCodecWithAttributes, build: any): SearchableColumn[]; + + /** Register any custom GraphQL types during the init hook */ + registerTypes(build: any): void; + + /** Apply a filter and return the SQL expression + score select index */ + applyFilter(args: FilterApplyArgs): FilterApplyResult | null; + + /** Build a text search input value from a plain text query (for unifiedSearch) */ + buildTextSearchInput?(text: string): unknown; +} +``` + +## ScoreSemantics + +```typescript +interface ScoreSemantics { + /** Metric name for field naming (e.g. 'rank', 'score', 'similarity', 'distance') */ + metric: string; + + /** If true, lower values are better (BM25, pgvector distance) */ + lowerIsBetter: boolean; + + /** + * Known range bounds for normalization, or null if unbounded. + * - trgm: [0, 1] + * - tsvector: [0, 1] + * - BM25: null (unbounded negative) + * - pgvector: null (0 to infinity) + */ + range: [number, number] | null; +} +``` + +## SearchableColumn + +```typescript +interface SearchableColumn { + /** The raw PostgreSQL column name (e.g. 'body', 'tsv', 'embedding') */ + attributeName: string; + + /** Optional extra data the adapter needs during SQL generation */ + adapterData?: unknown; +} +``` + +## Creating a Custom Adapter + +```typescript +import type { SearchAdapter } from 'graphile-search'; + +function createMyAdapter(): SearchAdapter { + return { + name: 'myalgo', + scoreSemantics: { + metric: 'relevance', + lowerIsBetter: false, + range: [0, 1], + }, + filterPrefix: 'myalgo', + supportsTextSearch: true, + + detectColumns(codec, build) { + // Return columns that have your search infrastructure + return []; + }, + + registerTypes(build) { + // Register any custom GraphQL input types + }, + + applyFilter(args) { + // Generate SQL WHERE clause and score expression + return null; + }, + + buildTextSearchInput(text) { + return { query: text }; + }, + }; +} +``` + +Then register it: + +```typescript +import { createUnifiedSearchPlugin } from 'graphile-search'; + +const plugin = createUnifiedSearchPlugin({ + adapters: [createMyAdapter()], +}); +``` diff --git a/.agents/skills/graphile-search/references/trgm-adapter.md b/.agents/skills/graphile-search/references/trgm-adapter.md new file mode 100644 index 000000000..4cd275fd0 --- /dev/null +++ b/.agents/skills/graphile-search/references/trgm-adapter.md @@ -0,0 +1,134 @@ +# Trgm Adapter + +Fuzzy text matching using the `pg_trgm` extension. Provides typo tolerance, "did you mean?" suggestions, and trigram similarity scoring. + +## How It Works + +The trgm adapter: +1. **Detects** text/varchar columns on tables (any `text` or `varchar` column is a candidate) +2. **Registers** `TrgmSearchInput` type (`{ value: String!, threshold: Float }`) +3. **Adds** `similarTo` and `wordSimilarTo` operators on `StringTrgmFilter` +4. **Generates** similarity score as a computed field and orderBy enum + +## Supplementary Adapter Pattern + +Trgm is a **supplementary adapter** — it only activates on tables where at least one "intentional search" adapter (tsvector or BM25) has detected columns. This prevents trgm similarity fields from appearing on every table with text columns. + +pgvector alone does NOT trigger trgm activation because it sets `isIntentionalSearch: false`. + +### Override with @trgmSearch + +Force trgm on tables without intentional search: + +```sql +-- Table-level +COMMENT ON TABLE app_public.contacts IS E'@trgmSearch'; + +-- Column-level +COMMENT ON COLUMN app_public.contacts.name IS E'@trgmSearch'; +``` + +## Adapter Configuration + +```typescript +import { createTrgmAdapter } from 'graphile-search'; + +createTrgmAdapter({ + filterPrefix: 'trgm', // default: 'trgm' + defaultThreshold: 0.3, // default: 0.3 + requireIntentionalSearch: true, // default: true (makes it supplementary) +}) +``` + +Setting `requireIntentionalSearch: false` makes trgm activate on ALL tables with text columns (not recommended for large schemas). + +## Generated GraphQL + +Given a table with intentional search (e.g. tsvector) and a `title text` column: + +### Filter (Connection Filter) + +```graphql +query { + allArticles(where: { + title: { similarTo: { value: "postgre", threshold: 0.2 } } + }) { + nodes { title } + } +} +``` + +```graphql +query { + allArticles(where: { + title: { wordSimilarTo: { value: "postgres", threshold: 0.3 } } + }) { + nodes { title } + } +} +``` + +### Adapter-Level Filter + +```graphql +query { + allArticles(where: { + trgmTitle: { value: "postgre", threshold: 0.2 } + }) { + nodes { + title + titleTrgmSimilarity + } + } +} +``` + +### Score Field + +```graphql +type Article { + titleTrgmSimilarity: Float # pg_trgm similarity() score, 0..1 +} +``` + +### OrderBy + +```graphql +enum ArticlesOrderBy { + TITLE_TRGM_SIMILARITY_ASC + TITLE_TRGM_SIMILARITY_DESC +} +``` + +## StringTrgmFilter vs StringFilter + +On tables that qualify for trgm, string columns use `StringTrgmFilter` instead of the standard `StringFilter`. This type inherits all standard string operators and adds: + +| Operator | SQL | Description | +|----------|-----|-------------| +| `similarTo` | `similarity(col, value) > threshold` | Overall trigram similarity | +| `wordSimilarTo` | `word_similarity(value, col) > threshold` | Best substring similarity | + +Both accept `TrgmSearchInput { value: String!, threshold: Float }`. Default threshold is 0.3. + +## Score Semantics + +| Property | Value | +|----------|-------| +| Metric | `similarity` | +| Lower is better | No (higher = more similar) | +| Range | [0, 1] | + +## Adapter Flags + +| Flag | Value | +|------|-------| +| `isSupplementary` | `true` (when `requireIntentionalSearch` is true) | +| `isIntentionalSearch` | N/A (supplementary adapters don't set this) | +| `supportsTextSearch` | `true` (included in unifiedSearch composite filter) | + +## Prerequisites + +- `pg_trgm` extension enabled (pre-enabled in Constructive stack) +- At least one "intentional search" column on the same table (tsvector or BM25), OR `@trgmSearch` smart tag +- For best performance: a GIN trigram index (`gin_trgm_ops`) on text columns you query diff --git a/.agents/skills/graphile-search/references/tsvector-adapter.md b/.agents/skills/graphile-search/references/tsvector-adapter.md new file mode 100644 index 000000000..d26b6c9e5 --- /dev/null +++ b/.agents/skills/graphile-search/references/tsvector-adapter.md @@ -0,0 +1,77 @@ +# TSVector Adapter + +Full-text search using PostgreSQL's built-in `tsvector` type. Provides keyword search with stemming, weighted ranking, and phrase matching. + +## How It Works + +The tsvector adapter: +1. **Detects** columns with `tsvector` type on each table +2. **Registers** a `FullText` scalar (via `TsvectorCodecPlugin`) so tsvector columns appear in GraphQL +3. **Adds** a `matches` filter operator on the `FullText` type (via `createMatchesOperatorFactory`) +4. **Generates** `ts_rank()` score as a computed field and orderBy enum + +## Adapter Configuration + +```typescript +import { createTsvectorAdapter } from 'graphile-search'; + +createTsvectorAdapter({ + filterPrefix: 'fullText', // default: 'fullText' + tsConfig: 'english', // default: 'english' +}) +``` + +## Generated GraphQL + +Given a table with a `search_tsv tsvector` column: + +### Filter + +```graphql +query { + allArticles(where: { fullTextSearchTsv: { matches: "postgres tutorial" } }) { + nodes { ... } + } +} +``` + +The `matches` operator uses `websearch_to_tsquery()` internally — supports natural language queries with AND/OR/NOT. + +### Score Field + +```graphql +type Article { + searchTsvRank: Float # ts_rank() score, 0..1, higher = better +} +``` + +### OrderBy + +```graphql +enum ArticlesOrderBy { + SEARCH_TSV_RANK_ASC + SEARCH_TSV_RANK_DESC +} +``` + +## Score Semantics + +| Property | Value | +|----------|-------| +| Metric | `rank` | +| Lower is better | No (higher = more relevant) | +| Range | [0, 1] | + +## Adapter Flags + +| Flag | Value | +|------|-------| +| `isSupplementary` | `false` (primary adapter) | +| `isIntentionalSearch` | `true` (triggers supplementary adapters like trgm) | +| `supportsTextSearch` | `true` (included in unifiedSearch composite filter) | + +## Prerequisites + +- A `tsvector` column on the table (typically populated via metaschema `full_text_search` triggers) +- A GIN index on the tsvector column (for performance) +- `TsvectorCodecPlugin` loaded (included automatically by `UnifiedSearchPreset`) diff --git a/.agents/skills/pgpm.zip b/.agents/skills/pgpm.zip new file mode 100644 index 000000000..230a42096 Binary files /dev/null and b/.agents/skills/pgpm.zip differ diff --git a/.agents/skills/pgpm/SKILL.md b/.agents/skills/pgpm/SKILL.md new file mode 100644 index 000000000..5ccbab958 --- /dev/null +++ b/.agents/skills/pgpm/SKILL.md @@ -0,0 +1,327 @@ +--- +name: pgpm +description: PostgreSQL Package Manager — deterministic, plan-driven database migrations with dependency management. Use when asked to "deploy database", "run migrations", "manage pgpm modules", "add a table", "create a function", "add a migration", "write database changes", "create a workspace", "set up pgpm", "manage dependencies", "revert a migration", "verify deployments", "tag a release", "start postgres", "run database locally", "set up database environment", "load env vars", "add an extension", "install a module", "publish pgpm module", "test database", "write integration tests", "troubleshoot pgpm", or when working with PostgreSQL package management, .control files, pgpm.plan, or SQL migration scripts. +compatibility: pgpm CLI, PostgreSQL 14+, Node.js 22+, Docker +metadata: + author: constructive-io + version: "2.0.0" +--- + +# pgpm (PostgreSQL Package Manager) + +pgpm provides deterministic, plan-driven database migrations with dependency management and modular packaging. It brings npm-style modularity to PostgreSQL database development — every change is deployed exactly once and reverted exactly once. + +## When to Apply + +Use this skill when: +- **Creating projects:** Setting up workspaces, initializing modules +- **Writing changes:** Adding tables, functions, triggers, indexes, RLS policies +- **Managing dependencies:** Within-module and cross-module references, .control files +- **Deploying:** Running deploy/verify/revert, tagging releases, checking status +- **Testing:** Writing PostgreSQL integration tests with pgsql-test +- **Configuring:** Docker setup, environment variables, connection config +- **Managing extensions:** Adding PostgreSQL extensions or pgpm modules +- **Publishing:** Bundling and publishing @pgpm/* modules to npm +- **Troubleshooting:** Connection issues, deployment failures, testing problems + +## Quick Start + +```bash +# 1. Install pgpm +npm install -g pgpm + +# 2. Start a local PostgreSQL container +pgpm docker start +eval "$(pgpm env)" + +# 3. Create a workspace +pgpm init workspace +# Enter workspace name when prompted +cd my-database-project +pnpm install + +# 4. Create a module +pgpm init +# Enter module name (e.g., "pets") and select extensions + +# 5. Add a change +cd packages/pets +pgpm add schemas/pets +pgpm add schemas/pets/tables/pets --requires schemas/pets + +# 6. Write your SQL (see "Three-File Pattern" below) + +# 7. Deploy +pgpm deploy --createdb --database mydb +``` + +## Core Concepts + +### Three-File Pattern + +Every database change consists of three files: + +| File | Purpose | Header | +|------|---------|--------| +| `deploy/.sql` | Creates the object | `-- Deploy to pg` | +| `revert/.sql` | Removes the object | `-- Revert from pg` | +| `verify/.sql` | Confirms deployment | `-- Verify on pg` | + +### pgpm.plan + +The plan file controls deployment order. Each line: +``` +change_name [dep1 dep2] 2026-01-25T00:00:00Z author +``` + +Dependencies `[...]` must come immediately after the change name, before the timestamp. + +### .control File + +Each module has a `.control` file declaring its name and PostgreSQL extension dependencies: +``` +comment = 'My database module' +default_version = '0.0.1' +requires = 'uuid-ossp,plpgsql' +``` + +The `requires` field uses **control file names** (e.g., `pgpm-base32`), NOT npm names (e.g., `@pgpm/base32`). See [references/module-naming.md](references/module-naming.md) for details. + +### Workspace vs Module + +- **Workspace** = pnpm monorepo containing one or more modules (`pgpm init workspace`) +- **Module** = individual database package with its own .control, pgpm.plan, and deploy/revert/verify directories (`pgpm init`) + +## Critical Rules + +### 1. NEVER Use CREATE OR REPLACE + +pgpm is deterministic. Each change deploys exactly once. To modify an existing object, create a new change that drops and recreates it. + +```sql +-- CORRECT +CREATE FUNCTION app.my_function() ... + +-- WRONG +CREATE OR REPLACE FUNCTION app.my_function() ... +``` + +### 2. NO Transaction Wrapping + +Do NOT add `BEGIN`/`COMMIT` to SQL files. pgpm handles transactions automatically. + +```sql +-- CORRECT — just the raw SQL +CREATE TABLE app.users ( ... ); + +-- WRONG +BEGIN; +CREATE TABLE app.users ( ... ); +COMMIT; +``` + +### 3. NEVER Run CREATE EXTENSION Directly + +pgpm handles extension creation during deploy. Declare extensions in your `.control` file's `requires` field instead. + +## Key Commands Quick Reference + +| Command | Purpose | +|---------|---------| +| `pgpm init workspace` | Create a new pnpm monorepo workspace | +| `pgpm init` | Create a new module in a workspace | +| `pgpm add --requires ` | Add a new change (creates deploy/revert/verify files) | +| `pgpm deploy` | Deploy changes to database | +| `pgpm deploy --createdb --database ` | Create database and deploy | +| `pgpm verify` | Run verification scripts | +| `pgpm revert --to ` | Revert changes back to a specific point | +| `pgpm tag ` | Tag current state for targeted deploys | +| `pgpm install ` | Install a pgpm module dependency | +| `pgpm extension` | Interactive dependency selector | +| `pgpm test-packages` | Test all packages | +| `pgpm test-packages --full-cycle` | Test deploy → verify → revert → redeploy | +| `pgpm docker start` | Start PostgreSQL container | +| `pgpm docker stop` | Stop PostgreSQL container | +| `pgpm env` | Print environment variable exports | +| `pgpm migrate status` | Show deployed vs pending changes | +| `pgpm plan` | Generate plan from SQL `-- requires:` comments | +| `pgpm package` | Bundle module for publishing | + +## Essential Development Workflow + +```bash +# Start PostgreSQL and load environment +pgpm docker start +eval "$(pgpm env)" + +# Bootstrap admin users (first time) +pgpm admin-users bootstrap --yes +pgpm admin-users add --test --yes + +# Add a new change +pgpm add schemas/app/tables/orders --requires schemas/app + +# Edit deploy/revert/verify SQL files + +# Deploy and verify +pgpm deploy --createdb --database mydb +pgpm verify + +# Run tests +pnpm test + +# Tag a release +pgpm tag v1.0.0 +``` + +## Common Workflows + +### Adding a Table + +```bash +# Add the change +pgpm add schemas/app/tables/users --requires schemas/app +``` + +```sql +-- deploy/schemas/app/tables/users.sql +-- Deploy schemas/app/tables/users to pg +-- requires: schemas/app + +CREATE TABLE app.users ( + id uuid PRIMARY KEY DEFAULT uuid_generate_v4(), + email text NOT NULL UNIQUE, + name text NOT NULL, + created_at timestamptz NOT NULL DEFAULT now() +); +``` + +```sql +-- revert/schemas/app/tables/users.sql +-- Revert schemas/app/tables/users from pg + +DROP TABLE IF EXISTS app.users; +``` + +```sql +-- verify/schemas/app/tables/users.sql +-- Verify schemas/app/tables/users on pg + +DO $$ +BEGIN + ASSERT (SELECT EXISTS ( + SELECT FROM information_schema.tables + WHERE table_schema = 'app' AND table_name = 'users' + )), 'Table app.users does not exist'; +END $$; +``` + +### Writing Tests (with pgsql-test) + +```typescript +import { getConnections } from 'pgsql-test'; +import * as seed from 'pgsql-test/seed'; + +let db, teardown; + +beforeAll(async () => { + ({ db, teardown } = await getConnections({}, [ + seed.pgpm({ database: 'mydb' }) + ])); +}); +afterAll(() => teardown()); +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); + +it('creates a user', async () => { + const result = await db.query(` + INSERT INTO app.users (email, name) + VALUES ('test@example.com', 'Test User') + RETURNING * + `); + expect(result.rows[0].email).toBe('test@example.com'); +}); +``` + +### CI/CD Full-Cycle Validation + +```bash +pgpm test-packages --full-cycle +``` + +This proves: deploy → verify → revert → redeploy works for every module. + +### Fixing a Broken Deploy + +```bash +# Check status +pgpm migrate status + +# Revert the bad change +pgpm revert --to + +# Fix the SQL, then redeploy +pgpm deploy +``` + +## Troubleshooting Quick Reference + +| Issue | Quick Fix | +|-------|-----------| +| Can't connect to database | `pgpm docker start && eval "$(pgpm env)"` | +| `PGHOST` not set | `eval "$(pgpm env)"` — must use `eval`, not run in subshell | +| Transaction aborted in tests | Use `db.beforeEach()` / `db.afterEach()` savepoint pattern | +| Tests interfere with each other | Ensure every test file has `beforeEach`/`afterEach` hooks | +| Module not found during deploy | Verify `.control` file exists and workspace structure is correct | +| Dependency not found | Check `.control` `requires` uses control names, not npm names | +| Port 5432 already in use | `lsof -i :5432` then stop conflicting process | +| `Invalid line format` in pgpm.plan | Dependencies `[...]` must come right after change name, before timestamp | +| `CREATE OR REPLACE` error | Remove `OR REPLACE` — pgpm is deterministic | +| Container won't start | `pgpm docker start --recreate` for a fresh container | + +See [references/troubleshooting.md](references/troubleshooting.md) for detailed solutions. + +## Reference Guide + +Consult these reference files for detailed documentation on specific topics: + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [references/cli.md](references/cli.md) | Complete CLI command reference | Looking up command flags, options, or less common commands | +| [references/workspace.md](references/workspace.md) | Creating and managing workspaces | Setting up a new project, understanding workspace structure | +| [references/changes.md](references/changes.md) | Authoring database changes | Writing deploy/revert/verify scripts, using `pgpm add` | +| [references/sql-conventions.md](references/sql-conventions.md) | SQL file format and conventions | Writing SQL files, naming conventions, header format | +| [references/dependencies.md](references/dependencies.md) | Managing module dependencies | Within-module or cross-module dependency references | +| [references/deploy-lifecycle.md](references/deploy-lifecycle.md) | Deploy/verify/revert lifecycle | Understanding deployment process, tagging, status checking | +| [references/docker.md](references/docker.md) | Docker container management | Starting/stopping PostgreSQL, custom container options | +| [references/env.md](references/env.md) | Environment variable management | Loading env vars, profiles, Supabase local development | +| [references/environment-configuration.md](references/environment-configuration.md) | @pgpmjs/env library API | Programmatic configuration, config hierarchy, utility functions | +| [references/extensions.md](references/extensions.md) | PostgreSQL extensions & pgpm modules | Adding extensions, installing @pgpm/* modules, .control requires | +| [references/module-naming.md](references/module-naming.md) | npm names vs control file names | Confused about which identifier to use where | +| [references/plan-format.md](references/plan-format.md) | pgpm.plan file format | Fixing `Invalid line format` errors, editing plan files manually | +| [references/publishing.md](references/publishing.md) | Publishing modules to npm | Bundling, versioning with lerna, publishing @pgpm/* packages | +| [references/testing.md](references/testing.md) | PostgreSQL integration tests | Setting up pgsql-test, seed adapters, test patterns | +| [references/troubleshooting.md](references/troubleshooting.md) | Common issues and solutions | Debugging connection, deployment, testing, or Docker problems | +| [references/ci-cd.md](references/ci-cd.md) | GitHub Actions CI/CD workflows | Setting up CI for pgpm projects, PostgreSQL service containers, test sharding | + +### Project Scaffolding + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [references/starter-kits.md](references/starter-kits.md) | `pgpm init` templates and scaffolding | Creating new workspaces, modules, or Next.js apps | +| [references/template-authoring.md](references/template-authoring.md) | Custom boilerplate authoring | Creating `.boilerplate.json`, placeholder system, question config | +| [references/nextjs-app.md](references/nextjs-app.md) | Constructive Next.js app boilerplate | Setting up frontend app, project structure, auth flows, SDK generation | + +### Database Operations (from constructive-db) + +| Reference | Topic | Consult When | +|-----------|-------|--------------| +| [references/pgpm-tables.md](references/pgpm-tables.md) | Table creation rules | Creating tables in metaschema, deterministic ID triggers | +| [references/pgpm-export.md](references/pgpm-export.md) | DB export to pgpm packages | Exporting a live database to pgpm for deterministic migrations | + +## Cross-References + +Related skills: +- `constructive-testing` — PostgreSQL testing patterns (RLS, seeding, snapshots, JWT context) +- `constructive-setup` — Monorepo setup and local development environment +- `constructive-cli` — Generated CLI commands and scaffolding diff --git a/.agents/skills/pgpm/references/changes.md b/.agents/skills/pgpm/references/changes.md new file mode 100644 index 000000000..1aa948dd3 --- /dev/null +++ b/.agents/skills/pgpm/references/changes.md @@ -0,0 +1,258 @@ + +# Authoring Database Changes with PGPM + +Create safe, reversible database changes using pgpm's three-file pattern. Every change has deploy, revert, and verify scripts. + +## When to Apply + +Use this skill when: +- Adding tables, functions, triggers, or indexes +- Creating database migrations +- Modifying existing schema +- Organizing database changes in a pgpm module + +## The Three-File Pattern + +Every database change consists of three files: + +| File | Purpose | +|------|---------| +| `deploy/.sql` | Creates the object | +| `revert/.sql` | Removes the object | +| `verify/.sql` | Confirms deployment | + +## Adding a Change + +```bash +pgpm add schemas/pets/tables/pets --requires schemas/pets +``` + +This creates: +```text +deploy/schemas/pets/tables/pets.sql +revert/schemas/pets/tables/pets.sql +verify/schemas/pets/tables/pets.sql +``` + +And updates `pgpm.plan`: +```sh +schemas/pets/tables/pets [schemas/pets] 2025-11-14T00:00:00Z Author +``` + +## Writing Deploy Scripts + +Deploy scripts create database objects. Use `CREATE`, not `CREATE OR REPLACE` (pgpm is deterministic). + +**deploy/schemas/pets/tables/pets.sql:** +```sql +-- Deploy: schemas/pets/tables/pets +-- requires: schemas/pets + +CREATE TABLE pets.pets ( + id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), + name TEXT NOT NULL, + breed TEXT, + owner_id UUID, + created_at TIMESTAMPTZ DEFAULT NOW() +); +``` + +**Important:** Never use `CREATE OR REPLACE` unless absolutely necessary. pgpm tracks what's deployed and ensures idempotency through its migration system. + +## Writing Revert Scripts + +Revert scripts undo the deploy. Must leave database in pre-deploy state. + +**revert/schemas/pets/tables/pets.sql:** +```sql +-- Revert: schemas/pets/tables/pets + +DROP TABLE IF EXISTS pets.pets; +``` + +## Writing Verify Scripts + +Verify scripts confirm deployment succeeded. Use `DO` blocks that raise exceptions on failure. + +**verify/schemas/pets/tables/pets.sql:** +```sql +-- Verify: schemas/pets/tables/pets + +DO $$ +BEGIN + PERFORM 1 FROM pg_tables + WHERE schemaname = 'pets' AND tablename = 'pets'; + IF NOT FOUND THEN + RAISE EXCEPTION 'Table pets.pets does not exist'; + END IF; +END $$; +``` + +## Nested Paths + +Organize changes hierarchically using nested paths: + +```text +schemas/ +└── app/ + ├── schema.sql + ├── tables/ + │ └── users/ + │ ├── table.sql + │ └── indexes/ + │ └── email.sql + ├── functions/ + │ └── create_user.sql + └── triggers/ + └── updated_at.sql +``` + +Add changes with full paths: +```bash +pgpm add schemas/app/schema +pgpm add schemas/app/tables/users/table --requires schemas/app/schema +pgpm add schemas/app/tables/users/indexes/email --requires schemas/app/tables/users/table +pgpm add schemas/app/functions/create_user --requires schemas/app/tables/users/table +``` + +**Key insight:** Deployment order follows the plan file, not directory structure. Nested paths are for organization only. + +## Plan File Format + +The `pgpm.plan` file tracks all changes: + +```sh +%syntax-version=1.0.0 +%project=pets +%uri=pets + +schemas/pets 2025-11-14T00:00:00Z Author +schemas/pets/tables/pets [schemas/pets] 2025-11-14T00:00:00Z Author +schemas/pets/tables/pets/indexes/name [schemas/pets/tables/pets] 2025-11-14T00:00:00Z Author +``` + +Format: `change_name [dependencies] timestamp author # optional note` + +## Two Workflows + +### Incremental (Development) + +Add changes one at a time: +```bash +pgpm add schemas/pets --requires uuid-ossp +pgpm add schemas/pets/tables/pets --requires schemas/pets +``` + +Plan file updates automatically with each `pgpm add`. + +### Pre-Production (Batch) + +Write all SQL files first, then generate plan: +```bash +# Write deploy/revert/verify files manually +# Then generate plan from requires comments: +pgpm plan +``` + +`pgpm plan` reads `-- requires:` comments from deploy files and generates the plan. + +## Common Change Types + +### Schema +```bash +pgpm add schemas/app +``` + +```sql +-- deploy/schemas/app.sql +CREATE SCHEMA app; + +-- revert/schemas/app.sql +DROP SCHEMA IF EXISTS app CASCADE; + +-- verify/schemas/app.sql +DO $$ BEGIN + PERFORM 1 FROM information_schema.schemata WHERE schema_name = 'app'; + IF NOT FOUND THEN RAISE EXCEPTION 'Schema app does not exist'; END IF; +END $$; +``` + +### Table +```bash +pgpm add schemas/app/tables/users --requires schemas/app +``` + +```sql +-- deploy/schemas/app/tables/users.sql +CREATE TABLE app.users ( + id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), + email TEXT UNIQUE NOT NULL, + created_at TIMESTAMPTZ DEFAULT NOW() +); + +-- revert/schemas/app/tables/users.sql +DROP TABLE IF EXISTS app.users; + +-- verify/schemas/app/tables/users.sql +DO $$ BEGIN + PERFORM 1 FROM pg_tables WHERE schemaname = 'app' AND tablename = 'users'; + IF NOT FOUND THEN RAISE EXCEPTION 'Table app.users does not exist'; END IF; +END $$; +``` + +### Function +```bash +pgpm add schemas/app/functions/get_user --requires schemas/app/tables/users +``` + +```sql +-- deploy/schemas/app/functions/get_user.sql +CREATE FUNCTION app.get_user(user_id UUID) +RETURNS app.users AS $$ + SELECT * FROM app.users WHERE id = user_id; +$$ LANGUAGE sql STABLE; + +-- revert/schemas/app/functions/get_user.sql +DROP FUNCTION IF EXISTS app.get_user(UUID); + +-- verify/schemas/app/functions/get_user.sql +DO $$ BEGIN + PERFORM 1 FROM pg_proc WHERE proname = 'get_user'; + IF NOT FOUND THEN RAISE EXCEPTION 'Function get_user does not exist'; END IF; +END $$; +``` + +### Index +```bash +pgpm add schemas/app/tables/users/indexes/email --requires schemas/app/tables/users +``` + +```sql +-- deploy/schemas/app/tables/users/indexes/email.sql +CREATE INDEX idx_users_email ON app.users(email); + +-- revert/schemas/app/tables/users/indexes/email.sql +DROP INDEX IF EXISTS app.idx_users_email; + +-- verify/schemas/app/tables/users/indexes/email.sql +DO $$ BEGIN + PERFORM 1 FROM pg_indexes WHERE indexname = 'idx_users_email'; + IF NOT FOUND THEN RAISE EXCEPTION 'Index idx_users_email does not exist'; END IF; +END $$; +``` + +## Deploy and Verify + +```bash +# Deploy to database +pgpm deploy --database myapp_dev --createdb --yes + +# Verify deployment +pgpm verify --database myapp_dev +``` + +## References + +- Related reference: `references/workspace.md` for workspace setup +- Related reference: `references/dependencies.md` for cross-module dependencies +- Related reference: `references/testing.md` for testing database changes diff --git a/.agents/skills/pgpm/references/ci-cd.md b/.agents/skills/pgpm/references/ci-cd.md new file mode 100644 index 000000000..b33bfbb7c --- /dev/null +++ b/.agents/skills/pgpm/references/ci-cd.md @@ -0,0 +1,441 @@ +--- +name: github-workflows-pgpm +description: Configure GitHub Actions workflows for PostgreSQL database testing, PGPM migrations, and CI/CD pipelines in Constructive projects. Use when setting up CI/CD for a PGPM-based project, configuring PostgreSQL service containers in GitHub Actions, or running database tests in CI. +--- + +Configure GitHub Actions workflows for PostgreSQL database testing, PGPM migrations, and CI/CD pipelines in Constructive projects. + +## When to Apply + +Use this skill when: +- Setting up CI/CD for a PGPM-based project +- Configuring PostgreSQL service containers in GitHub Actions +- Running database tests with pgsql-test in CI +- Generating SDKs or types from database schemas in CI +- Building and publishing Docker images for PostgreSQL + +## Core Workflow Pattern + +Every Constructive CI workflow follows this pattern: + +1. **Spin up PostgreSQL service container** with health checks +2. **Install pnpm and Node.js** with caching +3. **Cache and install pgpm CLI** globally +4. **Build the workspace** with `pnpm -r build` +5. **Bootstrap database users** with `pgpm admin-users` +6. **Run tests** per package + +## PostgreSQL Service Container + +Use the Constructive PostgreSQL image with extensions pre-installed: + +```yaml +services: + pg_db: + image: ghcr.io/constructive-io/docker/postgres-plus:17 + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: password + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + ports: + - 5432:5432 +``` + +For simpler setups without custom extensions: + +```yaml +services: + pg_db: + image: docker.io/constructiveio/postgres-plus:18 + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: password + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + ports: + - 5432:5432 +``` + +## Environment Variables + +Standard PostgreSQL environment variables for tests: + +```yaml +env: + PGHOST: localhost + PGPORT: 5432 + PGUSER: postgres + PGPASSWORD: password +``` + +For MinIO/S3 testing (uploads, storage): + +```yaml +env: + MINIO_ENDPOINT: http://localhost:9000 + AWS_ACCESS_KEY: minioadmin + AWS_SECRET_KEY: minioadmin + AWS_REGION: us-east-1 + BUCKET_NAME: test-bucket +``` + +## PGPM CLI Caching + +Cache the pgpm CLI to speed up workflows: + +```yaml +env: + PGPM_VERSION: '2.7.9' + +steps: + - name: Cache pgpm CLI + uses: actions/cache@v4 + with: + path: ~/.npm + key: pgpm-${{ runner.os }}-${{ env.PGPM_VERSION }} + + - name: Install pgpm CLI globally + run: npm install -g pgpm@${{ env.PGPM_VERSION }} +``` + +## Database User Bootstrap + +Before running tests, bootstrap the database users: + +```yaml +- name: Seed pg and app_user + run: | + pgpm admin-users bootstrap --yes + pgpm admin-users add --test --yes +``` + +This creates: +- The `app_user` role for RLS testing +- Test-specific roles and permissions + +## Complete Test Workflow + +Full workflow for running tests across multiple packages: + +```yaml +name: CI tests +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }}-tests + cancel-in-progress: true + +env: + PGPM_VERSION: '2.7.9' + +jobs: + test: + runs-on: ubuntu-latest + + strategy: + fail-fast: false + matrix: + package: + - packages/my-package + - packages/another-package + + env: + PGHOST: localhost + PGPORT: 5432 + PGUSER: postgres + PGPASSWORD: password + + services: + pg_db: + image: ghcr.io/constructive-io/docker/postgres-plus:17 + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: password + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + ports: + - 5432:5432 + + steps: + - name: Configure Git + run: | + git config --global user.name "CI Test User" + git config --global user.email "ci@example.com" + + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup pnpm + uses: pnpm/action-setup@v2 + with: + version: 10 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'pnpm' + + - name: Install dependencies + run: pnpm install + + - name: Cache pgpm CLI + uses: actions/cache@v4 + with: + path: ~/.npm + key: pgpm-${{ runner.os }}-${{ env.PGPM_VERSION }} + + - name: Install pgpm CLI globally + run: npm install -g pgpm@${{ env.PGPM_VERSION }} + + - name: Build + run: pnpm -r build + + - name: Seed pg and app_user + run: | + pgpm admin-users bootstrap --yes + pgpm admin-users add --test --yes + + - name: Test ${{ matrix.package }} + run: cd ./${{ matrix.package }} && pnpm test +``` + +## Integration Test Workflow + +For running pgpm's built-in integration tests: + +```yaml +- name: Run Integration Tests + run: pgpm test-packages +``` + +This runs all package tests defined in the pgpm workspace. + +## SDK Generation Workflow + +Generate typed SDKs from database schemas: + +```yaml +name: generate-sdk +on: + workflow_dispatch: + inputs: + commit_changes: + description: 'Commit and push generated SDK changes' + required: false + default: 'false' + type: boolean + +jobs: + generate-sdk: + runs-on: ubuntu-latest + + # ... services and setup steps ... + + steps: + # ... checkout, pnpm, node, pgpm setup ... + + - name: Build + run: pnpm -r build + + - name: Seed pg and app_user + run: | + pgpm admin-users bootstrap --yes + pgpm admin-users add --test --yes + + - name: Generate SDK + run: | + cd sdk/my-sdk + pnpm run generate + + - name: Check for changes + id: check_changes + run: | + if git diff --quiet sdk/my-sdk/src/generated; then + echo "has_changes=false" >> $GITHUB_OUTPUT + else + echo "has_changes=true" >> $GITHUB_OUTPUT + fi + + - name: Commit and push changes + if: ${{ inputs.commit_changes == 'true' && steps.check_changes.outputs.has_changes == 'true' }} + run: | + git add sdk/my-sdk/src/generated + git commit -m "chore: regenerate SDK types" + git push + + - name: Upload generated SDK as artifact + if: ${{ steps.check_changes.outputs.has_changes == 'true' }} + uses: actions/upload-artifact@v4 + with: + name: generated-sdk + path: sdk/my-sdk/src/generated + retention-days: 7 +``` + +## Test Sharding + +For large test suites, split tests across parallel jobs: + +```yaml +strategy: + fail-fast: false + matrix: + package: [packages/core] + test_pattern: [''] + include: + - package: packages/large-package + test_pattern: 'auth|rls' + shard_name: 'large-package-auth-rls' + - package: packages/large-package + test_pattern: 'permissions|orgs' + shard_name: 'large-package-permissions-orgs' + +steps: + - name: Test ${{ matrix.package }}${{ matrix.shard_name && format(' ({0})', matrix.shard_name) || '' }} + shell: bash + run: | + cd ./${{ matrix.package }} + if [ -n "${{ matrix.test_pattern }}" ]; then + pnpm test -- "${{ matrix.test_pattern }}" + else + pnpm test + fi +``` + +## MinIO Service Container + +For testing uploads and S3-compatible storage: + +```yaml +services: + minio_cdn: + image: minio/minio:edge-cicd + env: + MINIO_ROOT_USER: minioadmin + MINIO_ROOT_PASSWORD: minioadmin + ports: + - 9000:9000 + - 9001:9001 + options: >- + --health-cmd "curl -f http://localhost:9000/minio/health/live || exit 1" + --health-interval 10s + --health-timeout 5s + --health-retries 5 +``` + +## Concurrency Control + +Prevent duplicate workflow runs: + +```yaml +concurrency: + group: ${{ github.workflow }}-${{ github.ref }}-tests + cancel-in-progress: true +``` + +## Docker Build Workflow + +Build and push PostgreSQL images: + +```yaml +name: Docker +on: + workflow_dispatch: + inputs: + process: + description: 'Process to build' + type: choice + options: [pgvector, postgis, pgvector-postgis] + +jobs: + build-push: + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + + env: + REPO: ghcr.io/${{ github.repository_owner }} + PLATFORMS: linux/amd64,linux/arm64 + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up QEMU + uses: docker/setup-qemu-action@v3 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to GHCR + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Build and push + run: | + make \ + PROCESS=${{ inputs.process }} \ + REPO_NAME=$REPO \ + PLATFORMS="$PLATFORMS" \ + build-push-process +``` + +## Per-Package Environment Variables + +Pass package-specific environment variables: + +```yaml +strategy: + matrix: + include: + - package: packages/client + env: + TEST_DATABASE_URL: postgres://postgres:password@localhost:5432/postgres + - package: uploads/s3-streamer + env: + BUCKET_NAME: test-bucket + +steps: + - name: Test ${{ matrix.package }} + run: cd ./${{ matrix.package }} && pnpm test + env: ${{ matrix.env }} +``` + +## Best Practices + +1. **Always use health checks** — Ensure PostgreSQL is ready before tests run +2. **Cache pgpm CLI** — Speeds up workflow execution significantly +3. **Use concurrency control** — Prevent duplicate runs on rapid pushes +4. **Configure Git** — Required for tests that use git operations +5. **Use matrix strategy** — Run tests in parallel across packages +6. **Bootstrap users before tests** — `pgpm admin-users` creates required roles +7. **Use fail-fast: false** — Let all tests complete even if some fail +8. **Pin pgpm version** — Ensure consistent behavior across runs + +## References + +- Related skill: `pgsql-test` for database testing framework +- Related skill: `pgpm` (`references/workspace.md`) for PGPM project setup +- Related skill: `pnpm-workspace` for PNPM monorepo configuration +- [GitHub Actions documentation](https://docs.github.com/en/actions) +- [pnpm/action-setup](https://github.com/pnpm/action-setup) diff --git a/.agents/skills/pgpm/references/cli.md b/.agents/skills/pgpm/references/cli.md new file mode 100644 index 000000000..57d175d46 --- /dev/null +++ b/.agents/skills/pgpm/references/cli.md @@ -0,0 +1,389 @@ + +# pgpm CLI Reference + +Complete reference for the pgpm (PostgreSQL Package Manager) command-line interface. pgpm provides deterministic, plan-driven database migrations with dependency management. + +## When to Apply + +Use this skill when: +- Deploying database changes +- Managing database migrations +- Installing or upgrading pgpm modules +- Testing pgpm packages in CI/CD +- Setting up local PostgreSQL development + +## Quick Start + +```bash +# Install pgpm globally +npm install -g pgpm + +# Ensure PostgreSQL is running and env vars are loaded +# See references/docker.md and references/env.md for setup + +# Create workspace and module +pgpm init workspace +cd my-app +pgpm init +cd packages/your-module + +# Deploy to database +pgpm deploy --createdb --database mydb +``` + +## Core Commands + +### Database Operations + +**pgpm deploy** — Deploy database changes and migrations + +```bash +# Deploy to current database (from PGDATABASE) +pgpm deploy + +# Create database if missing +pgpm deploy --createdb + +# Deploy to specific database +pgpm deploy --database mydb + +# Deploy specific package to a tag +pgpm deploy --package mypackage --to @v1.0.0 +``` + +> **⚠️ WARNING: `--fast` flag (use with extreme caution)** +> +> ```bash +> pgpm deploy --fast --no-tx +> ``` +> +> `--fast` is **NOT idempotent** — it is meant to be run **once only** for +> quick testing on a fresh database. It skips dependency-tracking checks, so +> if your package shares dependencies with anything already deployed (e.g. +> `pgpm-verify`), it will blindly attempt to re-deploy them, causing errors. +> +> **Only use `--fast` on a throwaway database that has nothing else deployed.** +> For all other cases, use the standard `pgpm deploy` command. + +**pgpm verify** — Verify database state matches expected migrations + +```bash +pgpm verify +pgpm verify --package mypackage +``` + +**pgpm revert** — Safely revert database changes + +```bash +pgpm revert +pgpm revert --to @v1.0.0 +``` + +### Migration Management + +**pgpm migrate** — Comprehensive migration management + +```bash +# Initialize migration tracking +pgpm migrate init + +# Check migration status +pgpm migrate status + +# List all changes +pgpm migrate list + +# Show change dependencies +pgpm migrate deps +``` + +### Module Management + +**pgpm install** — Install pgpm modules as dependencies + +```bash +# Install single package +pgpm install @pgpm/faker + +# Install multiple packages +pgpm install @pgpm/base32 @pgpm/faker +``` + +**pgpm upgrade-modules** — Upgrade installed modules to latest versions + +```bash +# Interactive selection +pgpm upgrade-modules + +# Upgrade all without prompting +pgpm upgrade-modules --all + +# Preview without changes +pgpm upgrade-modules --dry-run + +# Upgrade specific modules +pgpm upgrade-modules --modules @pgpm/base32,@pgpm/faker + +# Upgrade across entire workspace +pgpm upgrade-modules --workspace --all +``` + +**pgpm extension** — Interactively manage module dependencies + +```bash +pgpm extension +``` + +### Workspace Initialization + +**pgpm init** — Initialize new module or workspace + +```bash +# Create new workspace +pgpm init workspace + +# Create new module (inside workspace) +pgpm init + +# Use full template path (recommended) +pgpm init --template pnpm/module +pgpm init -t pgpm/workspace + +# Create workspace + module in one command +pgpm init -w +pgpm init --template pnpm/module -w + +# Use custom template repository +pgpm init --repo https://github.com/org/templates.git --template my-template +``` + +### Change Management + +**pgpm add** — Add a new database change + +```bash +pgpm add my_change +``` + +This creates three files in `sql/`: +- `deploy/my_change.sql` — Deploy script +- `revert/my_change.sql` — Revert script +- `verify/my_change.sql` — Verify script + +**pgpm remove** — Remove a database change + +```bash +pgpm remove my_change +``` + +**pgpm rename** — Rename a database change + +```bash +pgpm rename old_name new_name +``` + +### Tagging and Versioning + +**pgpm tag** — Version your changes with tags + +```bash +# Tag latest change +pgpm tag v1.0.0 + +# Tag with comment +pgpm tag v1.0.0 --comment "Initial release" + +# Tag specific change +pgpm tag v1.1.0 --package mypackage --changeName my-change +``` + +### Packaging and Distribution + +**pgpm plan** — Generate deployment plans + +```bash +pgpm plan +``` + +**pgpm package** — Package module for distribution + +```bash +pgpm package +pgpm package --no-plan +``` + +### Testing + +**pgpm test-packages** — Run integration tests on all modules in workspace + +```bash +# Deploy only +pgpm test-packages + +# Full deploy/verify/revert/deploy cycle +pgpm test-packages --full-cycle + +# Continue after failures +pgpm test-packages --continue-on-fail + +# Exclude specific modules +pgpm test-packages --exclude legacy-module + +# Combine options +pgpm test-packages --full-cycle --continue-on-fail --exclude broken-module +``` + +### Docker and Environment + +**pgpm docker** — Manage local PostgreSQL container + +```bash +pgpm docker start +pgpm docker stop +``` + +**pgpm env** — Print PostgreSQL environment variables + +```bash +# Standard PostgreSQL +eval "$(pgpm env)" + +# Supabase local development +eval "$(pgpm env --supabase)" +``` + +### Admin Users + +**pgpm admin-users** — Manage database admin users + +```bash +# Bootstrap admin users from pgpm.json roles config +pgpm admin-users bootstrap + +# Add specific user +pgpm admin-users add myuser + +# Remove user +pgpm admin-users remove myuser +``` + +### Utilities + +**pgpm dump** — Dump database to SQL file + +```bash +# Dump to timestamped file +pgpm dump --database mydb + +# Dump to specific file +pgpm dump --database mydb --out ./backup.sql + +# Dump with pruning (for test fixtures) +pgpm dump --database mydb --database-id +``` + +**pgpm kill** — Clean up database connections + +```bash +# Kill connections and drop databases +pgpm kill + +# Only kill connections +pgpm kill --no-drop +``` + +**pgpm clear** — Clear database state + +```bash +pgpm clear +``` + +**pgpm export** — Export migrations from existing databases + +```bash +pgpm export +``` + +**pgpm analyze** — Analyze database structure + +```bash +pgpm analyze +``` + +### Cache and Updates + +**pgpm cache clean** — Clear cached template repos + +```bash +pgpm cache clean +``` + +**pgpm update** — Install latest pgpm version + +```bash +pgpm update +``` + +## Environment Variables + +pgpm uses standard PostgreSQL environment variables: + +| Variable | Description | +|----------|-------------| +| `PGHOST` | Database host | +| `PGPORT` | Database port | +| `PGDATABASE` | Database name | +| `PGUSER` | Database user | +| `PGPASSWORD` | Database password | + +Quick setup with `eval "$(pgpm env)"` or manual export. + +## Global Options + +Most commands support: + +| Option | Description | +|--------|-------------| +| `--help, -h` | Show help | +| `--version, -v` | Show version | +| `--cwd ` | Set working directory | + +## Common Workflows + +### Starting a New Project + +```bash +pgpm init workspace +cd my-app +pgpm init +cd packages/new-module +pgpm add some_change +# Edit sql/deploy/some_change.sql +pgpm deploy --createdb +``` + +### Installing and Using a Module + +```bash +cd packages/your-module +pgpm install @pgpm/faker +pgpm deploy --createdb --database mydb +psql -d mydb -c "SELECT faker.city('MI');" +``` + +### CI/CD Testing + +```bash +# Bootstrap admin users +pgpm admin-users bootstrap + +# Test all packages +pgpm test-packages --full-cycle --continue-on-fail +``` + +## References + +- Related reference: `references/workspace.md` for workspace structure +- Related reference: `references/changes.md` for authoring changes +- Related reference: `references/dependencies.md` for module dependencies +- Related skill: `github-workflows-pgpm` for CI/CD workflows diff --git a/.agents/skills/pgpm/references/dependencies.md b/.agents/skills/pgpm/references/dependencies.md new file mode 100644 index 000000000..38c9f3069 --- /dev/null +++ b/.agents/skills/pgpm/references/dependencies.md @@ -0,0 +1,209 @@ + +# Managing PGPM Dependencies + +Handle dependencies between database changes and across modules in pgpm workspaces. + +## When to Apply + +Use this skill when: +- Adding dependencies between database changes +- Referencing objects from other modules +- Managing cross-module dependencies +- Resolving dependency order issues + +## Dependency Types + +### Within-Module Dependencies + +Changes within the same module reference each other by path: + +```sql +-- deploy/schemas/pets/tables/pets.sql +-- requires: schemas/pets + +CREATE TABLE pets.pets ( + id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), + name TEXT NOT NULL +); +``` + +Add with `--requires`: +```bash +pgpm add schemas/pets/tables/pets --requires schemas/pets +``` + +### Cross-Module Dependencies + +Reference changes from other modules using `module:path` syntax: + +```sql +-- deploy/schemas/app/tables/user_pets.sql +-- requires: schemas/app/tables/users +-- requires: pets:schemas/pets/tables/pets + +CREATE TABLE app.user_pets ( + user_id UUID REFERENCES app.users(id), + pet_id UUID REFERENCES pets.pets(id), + PRIMARY KEY (user_id, pet_id) +); +``` + +The `pets:schemas/pets/tables/pets` syntax means: +- `pets` = module name (from .control file) +- `schemas/pets/tables/pets` = change path within that module + +## The .control File + +Module metadata and extension dependencies live in the `.control` file: + +```sh +# pets.control +comment = 'Pet management module' +default_version = '0.0.1' +requires = 'uuid-ossp,plpgsql' +``` + +| Field | Purpose | +|-------|---------| +| `comment` | Module description | +| `default_version` | Semantic version | +| `requires` | PostgreSQL extensions needed | + +## Adding Extension Dependencies + +When your module needs PostgreSQL extensions: + +```bash +# Interactive mode +pgpm extension + +# Or edit .control directly +requires = 'uuid-ossp,plpgsql,pgcrypto' +``` + +## Dependency Resolution + +pgpm resolves dependencies recursively: + +1. Reads `pgpm.plan` for change order +2. Parses `-- requires:` comments +3. Resolves cross-module references +4. Deploys in correct topological order + +Example deployment order: +```text +1. uuid-ossp (extension) +2. plpgsql (extension) +3. schemas/pets (schema) +4. schemas/pets/tables/pets (table) +5. schemas/app (schema) +6. schemas/app/tables/users (table) +7. schemas/app/tables/user_pets (references both) +``` + +## Common Patterns + +### Schema Before Tables + +```bash +pgpm add schemas/app +pgpm add schemas/app/tables/users --requires schemas/app +pgpm add schemas/app/tables/posts --requires schemas/app/tables/users +``` + +### Functions After Tables + +```bash +pgpm add schemas/app/functions/create_user --requires schemas/app/tables/users +``` + +### Triggers After Functions + +```bash +pgpm add schemas/app/triggers/user_updated --requires schemas/app/functions/update_timestamp +``` + +### Cross-Module Reference + +Module A (users): +```bash +pgpm add schemas/users/tables/users +``` + +Module B (posts): +```bash +pgpm add schemas/posts/tables/posts --requires users:schemas/users/tables/users +``` + +## Viewing Dependencies + +Check what a change depends on: +```bash +# View plan file +cat pgpm.plan +``` + +Plan shows dependencies in brackets: +```sh +schemas/app/tables/user_pets [schemas/app/tables/users pets:schemas/pets/tables/pets] 2025-11-14T00:00:00Z Author +``` + +## Circular Dependencies + +pgpm prevents circular dependencies. If you see: +```text +Error: Circular dependency detected +``` + +Refactor to break the cycle: +1. Extract shared objects to a base module +2. Have both modules depend on the base +3. Remove direct cross-references + +**Before (circular):** +```text +module-a depends on module-b +module-b depends on module-a +``` + +**After (resolved):** +```text +module-base (shared objects) +module-a depends on module-base +module-b depends on module-base +``` + +## Deploying with Dependencies + +Deploy resolves all dependencies automatically: + +```bash +# Deploy single module (pulls in dependencies) +pgpm deploy --database myapp_dev --createdb --yes + +# Deploy specific module in workspace +cd packages/posts +pgpm deploy --database myapp_dev --yes +``` + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Module not found" | Ensure module is in workspace `packages/` | +| "Change not found" | Check path matches exactly in plan file | +| "Circular dependency" | Refactor to use base module pattern | +| Wrong deploy order | Check `-- requires:` comments in deploy files | + +## Best Practices + +1. **Explicit dependencies**: Always declare what you need +2. **Minimal dependencies**: Only require what's directly used +3. **Consistent naming**: Use same paths in requires and plan +4. **Test deployments**: Verify order with fresh database + +## References + +- Related reference: `references/workspace.md` for workspace setup +- Related reference: `references/changes.md` for authoring changes +- Related reference: `references/testing.md` for testing modules diff --git a/.agents/skills/pgpm/references/deploy-lifecycle.md b/.agents/skills/pgpm/references/deploy-lifecycle.md new file mode 100644 index 000000000..fe190e7fd --- /dev/null +++ b/.agents/skills/pgpm/references/deploy-lifecycle.md @@ -0,0 +1,248 @@ + +# pgpm Deploy Lifecycle + +The complete deploy → verify → revert lifecycle for pgpm database modules. + +## When to Apply + +Use this skill when: +- Deploying database changes with `pgpm deploy` +- Reverting deployments with `pgpm revert` +- Verifying deployed state with `pgpm verify` +- Tagging deployment points with `pgpm tag` +- Checking deployment status with `pgpm migrate status` +- Running full-cycle tests with `pgpm test-packages` + +## Core Concept + +pgpm deployments are **deterministic and plan-driven**. Every change is tracked in `pgpm.plan`, and each change has exactly three scripts: +- **deploy/** — applies the change +- **verify/** — confirms it was applied correctly +- **revert/** — undoes the change + +pgpm handles transactions automatically — you just write the SQL. + +## Deploy + +### Basic Deploy + +```bash +# Deploy all pending changes for the current module +pgpm deploy + +# Deploy with database creation (if database doesn't exist) +pgpm deploy --createdb + +# Deploy a specific module by name +pgpm deploy my-module + +# Deploy all modules in the workspace +pgpm deploy --workspace --all +``` + +### What Happens During Deploy + +1. **Dependency resolution** — reads `.control` file, resolves all required extensions and pgpm modules +2. **Extension creation** — native Postgres extensions get `CREATE EXTENSION IF NOT EXISTS` +3. **Module dependency deploy** — pgpm modules from `extensions/` are deployed first (topological order) +4. **Plan execution** — each change in `pgpm.plan` is executed in order: + - Checks if already deployed (via tracking schema) + - Runs the deploy script + - Records the change in the tracking schema +5. **Automatic verification** — after deploy, verify scripts run to confirm state + +### Deploy to a Tag + +```bash +# Deploy only up to a specific tag +pgpm deploy --to @v1.0.0 +``` + +### Deploy Options + +| Option | Description | +|--------|-------------| +| `--createdb` | Create the target database if it doesn't exist | +| `--workspace` | Operate at workspace level | +| `--all` | Deploy all modules (with `--workspace`) | +| `--to @tag` | Deploy up to a specific tag | +| `--yes` | Skip confirmation prompts | + +## Verify + +Verify checks that deployed changes are actually in the expected state. + +```bash +# Verify all deployed changes +pgpm verify + +# Verify a specific module +pgpm verify my-module +``` + +### What Happens During Verify + +For each deployed change, pgpm runs the corresponding `verify/` script. Verify scripts typically use `SELECT` statements that will fail if the expected objects don't exist: + +```sql +-- verify/schemas/app/tables/users.sql +SELECT id, email, name, created_at +FROM app.users +WHERE FALSE; +``` + +If any verify script fails, pgpm reports which changes are in a bad state. + +## Revert + +Revert undoes deployed changes in reverse order. + +```bash +# Revert the last deployed change +pgpm revert + +# Revert to a specific tag +pgpm revert --to @v1.0.0 + +# Revert all changes +pgpm revert --all + +# Revert with confirmation skip +pgpm revert --yes +``` + +### What Happens During Revert + +1. Changes are reverted in **reverse plan order** (last deployed = first reverted) +2. Each revert script runs (e.g., `DROP TABLE`, `DROP FUNCTION`) +3. The change is removed from the tracking schema +4. Verify scripts run to confirm the revert + +### Revert Options + +| Option | Description | +|--------|-------------| +| `--to @tag` | Revert back to a specific tag (exclusive — the tag itself stays) | +| `--all` | Revert all deployed changes | +| `--yes` | Skip confirmation prompts | + +## Tagging + +Tags mark specific points in the deployment plan for targeted deploy/revert. + +```bash +# Tag the current state +pgpm tag v1.0.0 + +# Tag with a description +pgpm tag v1.0.0 -m "Initial release" +``` + +Tags appear in `pgpm.plan` as: + +``` +@v1.0.0 2024-01-15T10:00:00Z user # Initial release +``` + +### Using Tags + +```bash +# Deploy up to a tag +pgpm deploy --to @v1.0.0 + +# Revert to a tag (keeps the tag, reverts everything after it) +pgpm revert --to @v1.0.0 +``` + +## Status + +Check what's deployed and what's pending. + +```bash +# Show deployment status +pgpm migrate status +``` + +This shows: +- Which changes are deployed +- Which changes are pending (in plan but not yet deployed) +- The current tag (if any) + +## Full-Cycle Testing + +`pgpm test-packages` runs a full deploy → verify → revert → deploy cycle to validate that all scripts work correctly in both directions. + +```bash +# Full cycle test for current module +pgpm test-packages --full-cycle + +# Full cycle test for all workspace modules +pgpm test-packages --full-cycle --workspace --all +``` + +This is the gold standard for validating migrations — it proves: +1. Deploy scripts apply correctly +2. Verify scripts confirm the deployed state +3. Revert scripts cleanly undo everything +4. Re-deploy works (proving revert was complete) + +## Common Workflows + +### First-time workspace deploy + +> **Prerequisite:** Ensure PostgreSQL is running and environment is loaded. See `references/docker.md` and `references/env.md` for setup. + +```bash +pgpm admin-users bootstrap --yes +pgpm deploy --createdb --workspace --all --yes +``` + +### Deploy after adding new changes + +```bash +pgpm deploy +pgpm verify +``` + +### Revert a bad deploy + +```bash +pgpm revert --yes +# Fix the issue, then redeploy +pgpm deploy +``` + +### Tag a release and deploy to that point + +```bash +pgpm tag v1.0.0 +pgpm deploy --to @v1.0.0 +``` + +### Validate all migrations (CI) + +> **Note:** In CI, start Postgres and load env vars first. See `references/docker.md` and `references/env.md`, or `github-workflows-pgpm` for CI-specific patterns. + +```bash +pgpm admin-users bootstrap --yes +pgpm test-packages --full-cycle --workspace --all +``` + +## Tracking Schema + +pgpm tracks deployments in a PostgreSQL schema (typically `pgpm_migrate`). This contains: +- `changes` table — records each deployed change with timestamp and deployer +- `tags` table — records tagged points + +This is how pgpm knows what's already deployed and what's pending. + +## Troubleshooting + +| Issue | Cause | Fix | +|-------|-------|-----| +| `role "authenticated" does not exist` | Missing bootstrap | Run `pgpm admin-users bootstrap --yes` | +| `database "mydb" does not exist` | Database not created | Use `pgpm deploy --createdb` | +| Deploy fails mid-way | SQL error in a deploy script | Fix the script, `pgpm revert` the failed change, redeploy | +| Verify fails after deploy | Deploy script didn't create expected objects | Check deploy script matches verify expectations | +| Revert fails | Revert script references objects that don't exist | Check for dependencies between changes | +| `Already deployed` | Change was previously deployed | Check `pgpm migrate status` — may need `pgpm revert` first | diff --git a/.agents/skills/pgpm/references/docker.md b/.agents/skills/pgpm/references/docker.md new file mode 100644 index 000000000..c5400c75f --- /dev/null +++ b/.agents/skills/pgpm/references/docker.md @@ -0,0 +1,142 @@ + +# PGPM Docker + +Manage PostgreSQL Docker containers for local development using the `pgpm docker` command. + +## When to Apply + +Use this skill when: +- Setting up a local PostgreSQL database for development +- Starting or stopping PostgreSQL containers +- Recreating a fresh database container +- User asks to run tests that need a database +- Troubleshooting database connection issues + +## Quick Start + +### Start PostgreSQL Container + +```bash +pgpm docker start +``` + +This starts a PostgreSQL 17 container with default settings: +- Container name: `postgres` +- Port: `5432` +- User: `postgres` +- Password: `password` + +### Start with Custom Options + +```bash +pgpm docker start --port 5433 --name my-postgres +``` + +### Recreate Container (Fresh Database) + +```bash +pgpm docker start --recreate +``` + +### Stop Container + +```bash +pgpm docker stop +``` + +## Command Reference + +### pgpm docker start + +Start a PostgreSQL Docker container. + +| Option | Description | Default | +|--------|-------------|---------| +| `--name ` | Container name | `postgres` | +| `--image ` | Docker image | `docker.io/constructiveio/postgres-plus:18` | +| `--port ` | Host port mapping | `5432` | +| `--user ` | PostgreSQL user | `postgres` | +| `--password ` | PostgreSQL password | `password` | +| `--recreate` | Remove and recreate container | `false` | + +### pgpm docker stop + +Stop a running PostgreSQL container. + +| Option | Description | Default | +|--------|-------------|---------| +| `--name ` | Container name to stop | `postgres` | + +## Common Workflows + +### Development Setup + +```bash +# Start fresh database +pgpm docker start --recreate + +# Load environment variables +eval "$(pgpm env)" + +# Deploy your PGPM modules +pgpm deploy +``` + +### Running Tests + +```bash +# Ensure database is running +pgpm docker start + +# Run tests with environment +pgpm env pnpm test +``` + +### Multiple Databases + +```bash +# Start main database on default port +pgpm docker start --name main-db + +# Start test database on different port +pgpm docker start --name test-db --port 5433 +``` + +## PostgreSQL Version + +The default image `docker.io/constructiveio/postgres-plus:18` includes PostgreSQL 17 which is required for: +- `security_invoker` views +- Latest PostgreSQL features used by Constructive + +If you see errors like "unrecognized parameter security_invoker", ensure you're using PostgreSQL 17+. + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Docker is not installed" | Install Docker Desktop or Docker Engine | +| "Port already in use" | Use `--port` to specify a different port, or stop the conflicting container | +| Container won't start | Check `docker logs postgres` for errors | +| "Container already exists" | Use `--recreate` to remove and recreate | +| Permission denied | Ensure Docker daemon is running and user has permissions | + +## Environment Variables + +After starting the container, use `pgpm env` to set up environment variables: + +```bash +eval "$(pgpm env)" +``` + +This sets: +- `PGHOST=localhost` +- `PGPORT=5432` +- `PGUSER=postgres` +- `PGPASSWORD=password` +- `PGDATABASE=postgres` + +## References + +For related references: +- Environment management: See `references/env.md` +- Running tests: See `references/testing.md` diff --git a/.agents/skills/pgpm/references/env.md b/.agents/skills/pgpm/references/env.md new file mode 100644 index 000000000..eaaff791d --- /dev/null +++ b/.agents/skills/pgpm/references/env.md @@ -0,0 +1,205 @@ +# PGPM Env + +Manage PostgreSQL environment variables with profile support using the `pgpm env` command. + +## When to Apply + +Use this skill when: +- Setting up environment variables for database connections +- Running commands that need PostgreSQL connection info +- Switching between local Postgres and Supabase profiles +- Deploying PGPM modules with correct database settings +- Running tests or scripts that need database access + +## Quick Start + +### Load Environment Variables + +```bash +eval "$(pgpm env)" +``` + +This sets the following environment variables: +- `PGHOST=localhost` +- `PGPORT=5432` +- `PGUSER=postgres` +- `PGPASSWORD=password` +- `PGDATABASE=postgres` + +### Run Command with Environment + +```bash +pgpm env pgpm deploy --database mydb +``` + +This runs `pgpm deploy --database mydb` with the PostgreSQL environment variables automatically set. + +## Profiles + +### Default Profile (Local Postgres) + +```bash +eval "$(pgpm env)" +``` + +| Variable | Value | +|----------|-------| +| `PGHOST` | `localhost` | +| `PGPORT` | `5432` | +| `PGUSER` | `postgres` | +| `PGPASSWORD` | `password` | +| `PGDATABASE` | `postgres` | + +### Supabase Profile + +```bash +eval "$(pgpm env --supabase)" +``` + +| Variable | Value | +|----------|-------| +| `PGHOST` | `localhost` | +| `PGPORT` | `54322` | +| `PGUSER` | `supabase_admin` | +| `PGPASSWORD` | `postgres` | +| `PGDATABASE` | `postgres` | + +## Command Reference + +### Print Environment Exports + +```bash +pgpm env # Default Postgres profile +pgpm env --supabase # Supabase profile +``` + +Output (for shell evaluation): +```bash +export PGHOST="localhost" +export PGPORT="5432" +export PGUSER="postgres" +export PGPASSWORD="password" +export PGDATABASE="postgres" +``` + +### Execute Command with Environment + +```bash +pgpm env [args...] +pgpm env --supabase [args...] +``` + +Examples: +```bash +pgpm env createdb mydb +pgpm env pgpm deploy --database mydb +pgpm env psql -c "SELECT 1" +pgpm env --supabase pgpm deploy --database mydb +``` + +## Common Workflows + +### Development Setup + +```bash +# Start database container +pgpm docker start + +# Load environment into current shell +eval "$(pgpm env)" + +# Now all commands have database access +createdb myapp +pgpm deploy --database myapp +``` + +### Running Tests + +```bash +# Run tests with database environment +pgpm env pnpm test + +# Or load into shell first +eval "$(pgpm env)" +pnpm test +``` + +### PGPM Deployment + +```bash +# Deploy to a specific database +pgpm env pgpm deploy --database constructive + +# Verify deployment +pgpm env pgpm verify --database constructive +``` + +### Supabase Local Development + +```bash +# Start Supabase locally (using supabase CLI) +supabase start + +# Load Supabase environment +eval "$(pgpm env --supabase)" + +# Deploy modules to Supabase +pgpm deploy --database postgres +``` + +## Shell Integration + +### Bash/Zsh + +Add to your shell profile for automatic loading: + +```bash +# ~/.bashrc or ~/.zshrc +alias pgenv='eval "$(pgpm env)"' +alias pgenv-supa='eval "$(pgpm env --supabase)"' +``` + +Then use: +```bash +pgenv # Load default Postgres env +pgenv-supa # Load Supabase env +``` + +### One-liner Commands + +```bash +# Create database and deploy in one command +pgpm env bash -c "createdb mydb && pgpm deploy --database mydb" +``` + +## Environment Variables Reference + +The `pgpm env` command sets standard PostgreSQL environment variables that are recognized by: +- `psql` and other PostgreSQL CLI tools +- Node.js `pg` library +- PGPM CLI commands +- Any tool using libpq + +| Variable | Description | +|----------|-------------| +| `PGHOST` | Database server hostname | +| `PGPORT` | Database server port | +| `PGUSER` | Database username | +| `PGPASSWORD` | Database password | +| `PGDATABASE` | Default database name | + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Connection refused" | Ensure database container is running with `pgpm docker start` | +| Wrong database | Check `PGDATABASE` or specify `--database` flag | +| Auth failed | Verify password matches container settings | +| Supabase not connecting | Ensure Supabase is running on port 54322 | +| Env vars not persisting | Use `eval "$(pgpm env)"` to load into current shell | + +## References + +For related skills: +- Docker container management: See `references/docker.md` +- Running tests: See `references/testing.md` diff --git a/.agents/skills/pgpm/references/environment-configuration.md b/.agents/skills/pgpm/references/environment-configuration.md new file mode 100644 index 000000000..dcf11b734 --- /dev/null +++ b/.agents/skills/pgpm/references/environment-configuration.md @@ -0,0 +1,359 @@ +# Environment Configuration with @pgpmjs/env + +Unified environment configuration for PGPM and Constructive projects. Provides config file discovery, environment variable parsing, and hierarchical option merging. + +## When to Apply + +Use this skill when: +- Configuring PostgreSQL connections programmatically +- Setting up PGPM environment options +- Managing database configuration across environments +- Writing code that needs consistent environment handling + +## Installation + +```bash +pnpm add @pgpmjs/env +``` + +## Core Concepts + +### Configuration Hierarchy + +Options are merged in this order (later overrides earlier): + +1. **PGPM defaults** — Built-in sensible defaults +2. **Config file** — `pgpm.json` discovered via walkUp +3. **Environment variables** — `PGHOST`, `PGPORT`, etc. +4. **Runtime overrides** — Passed programmatically + +## Basic Usage + +### getEnvOptions() + +Get merged PGPM options: + +```typescript +import { getEnvOptions } from '@pgpmjs/env'; + +const options = getEnvOptions(); +// Returns merged options from defaults + config + env vars + +// With runtime overrides +const options = getEnvOptions({ + pg: { database: 'mydb' } +}); + +// With custom working directory +const options = getEnvOptions({}, '/path/to/project'); +``` + +### getConnEnvOptions() + +Get database connection options specifically: + +```typescript +import { getConnEnvOptions } from '@pgpmjs/env'; + +const connOptions = getConnEnvOptions(); +// Returns db-specific options with roles and connections resolved +``` + +### getDeploymentEnvOptions() + +Get deployment-specific options: + +```typescript +import { getDeploymentEnvOptions } from '@pgpmjs/env'; + +const deployOptions = getDeploymentEnvOptions(); +// Returns deployment options (useTx, fast, usePlan, etc.) +``` + +## Environment Variables + +### PostgreSQL Connection + +| Variable | Description | Default | +|----------|-------------|---------| +| `PGHOST` | Database host | `localhost` | +| `PGPORT` | Database port | `5432` | +| `PGDATABASE` | Database name | — | +| `PGUSER` | Database user | `postgres` | +| `PGPASSWORD` | Database password | — | + +### Database Configuration + +| Variable | Description | +|----------|-------------| +| `PGROOTDATABASE` | Root database for admin operations | +| `PGTEMPLATE` | Template database for createdb | +| `DB_PREFIX` | Prefix for database names | +| `DB_EXTENSIONS` | Comma-separated list of extensions | +| `DB_CWD` | Working directory for database operations | + +### Connection Credentials + +| Variable | Description | +|----------|-------------| +| `DB_CONNECTION_USER` | App connection user | +| `DB_CONNECTION_PASSWORD` | App connection password | +| `DB_CONNECTION_ROLE` | App connection role | +| `DB_CONNECTIONS_APP_USER` | App-level user | +| `DB_CONNECTIONS_APP_PASSWORD` | App-level password | +| `DB_CONNECTIONS_ADMIN_USER` | Admin-level user | +| `DB_CONNECTIONS_ADMIN_PASSWORD` | Admin-level password | + +### Deployment Options + +| Variable | Description | +|----------|-------------| +| `DEPLOYMENT_USE_TX` | Use transactions for deployment | +| `DEPLOYMENT_FAST` | Fast deployment mode | +| `DEPLOYMENT_USE_PLAN` | Use deployment plan | +| `DEPLOYMENT_CACHE` | Enable deployment caching | +| `DEPLOYMENT_TO_CHANGE` | Deploy to specific change | + +### Server Configuration + +| Variable | Description | +|----------|-------------| +| `PORT` | Server port | +| `SERVER_HOST` | Server host | +| `SERVER_TRUST_PROXY` | Trust proxy headers | +| `SERVER_ORIGIN` | Server origin URL | +| `SERVER_STRICT_AUTH` | Strict authentication mode | + +### CDN/Storage + +| Variable | Description | +|----------|-------------| +| `BUCKET_PROVIDER` | Storage provider (s3, minio) | +| `BUCKET_NAME` | Bucket name | +| `AWS_REGION` | AWS region | +| `AWS_ACCESS_KEY_ID` | AWS access key | +| `AWS_SECRET_ACCESS_KEY` | AWS secret key | +| `MINIO_ENDPOINT` | MinIO endpoint URL | + +### Jobs Configuration + +| Variable | Description | +|----------|-------------| +| `JOBS_SCHEMA` | Schema for job tables | +| `JOBS_SUPPORT_ANY` | Support any job type | +| `JOBS_SUPPORTED` | Comma-separated supported job types | +| `INTERNAL_GATEWAY_URL` | Internal gateway URL | +| `INTERNAL_JOBS_CALLBACK_URL` | Jobs callback URL | +| `INTERNAL_JOBS_CALLBACK_PORT` | Jobs callback port | + +### Error Output + +| Variable | Description | +|----------|-------------| +| `PGPM_ERROR_QUERY_HISTORY_LIMIT` | Query history limit in errors | +| `PGPM_ERROR_MAX_LENGTH` | Max error message length | +| `PGPM_ERROR_VERBOSE` | Verbose error output | + +## Config File Discovery + +### loadConfigSync() + +Load `pgpm.json` by walking up directory tree: + +```typescript +import { loadConfigSync } from '@pgpmjs/env'; + +const config = loadConfigSync('/path/to/project'); +// Finds nearest pgpm.json walking up from given path +``` + +### loadConfigSyncFromDir() + +Load config from specific directory: + +```typescript +import { loadConfigSyncFromDir } from '@pgpmjs/env'; + +const config = loadConfigSyncFromDir('/path/to/project'); +``` + +### resolvePgpmPath() + +Find the pgpm.json file path: + +```typescript +import { resolvePgpmPath } from '@pgpmjs/env'; + +const pgpmPath = resolvePgpmPath('/path/to/project'); +// Returns full path to pgpm.json or undefined +``` + +## Workspace Resolution + +### resolvePnpmWorkspace() + +Find pnpm-workspace.yaml: + +```typescript +import { resolvePnpmWorkspace } from '@pgpmjs/env'; + +const workspacePath = resolvePnpmWorkspace('/path/to/project'); +``` + +### resolveLernaWorkspace() + +Find lerna.json: + +```typescript +import { resolveLernaWorkspace } from '@pgpmjs/env'; + +const lernaPath = resolveLernaWorkspace('/path/to/project'); +``` + +### resolveWorkspaceByType() + +Find workspace config by type: + +```typescript +import { resolveWorkspaceByType, WorkspaceType } from '@pgpmjs/env'; + +const path = resolveWorkspaceByType('/path/to/project', 'pnpm'); +// WorkspaceType: 'pnpm' | 'lerna' | 'npm' +``` + +## Utility Functions + +### walkUp() + +Walk up directory tree to find a file: + +```typescript +import { walkUp } from '@pgpmjs/env'; + +const found = walkUp('/start/path', 'pgpm.json'); +// Returns path to file or undefined +``` + +### getEnvVars() + +Parse environment variables into PgpmOptions: + +```typescript +import { getEnvVars } from '@pgpmjs/env'; + +const envOptions = getEnvVars(); +// Or with custom env object +const envOptions = getEnvVars(process.env); +``` + +### getNodeEnv() + +Get normalized NODE_ENV: + +```typescript +import { getNodeEnv } from '@pgpmjs/env'; + +const env = getNodeEnv(); +// Returns 'development' | 'production' | 'test' +``` + +### parseEnvBoolean() + +Parse boolean environment variable: + +```typescript +import { parseEnvBoolean } from '@pgpmjs/env'; + +parseEnvBoolean('true'); // true +parseEnvBoolean('1'); // true +parseEnvBoolean('yes'); // true +parseEnvBoolean('false'); // false +parseEnvBoolean(undefined); // undefined +``` + +### parseEnvNumber() + +Parse numeric environment variable: + +```typescript +import { parseEnvNumber } from '@pgpmjs/env'; + +parseEnvNumber('5432'); // 5432 +parseEnvNumber('invalid'); // undefined +parseEnvNumber(undefined); // undefined +``` + +## pgpm.json Configuration + +Example `pgpm.json` with environment options: + +```json +{ + "name": "my-module", + "version": "1.0.0", + "db": { + "rootDb": "postgres", + "template": "template1", + "prefix": "myapp_", + "extensions": ["uuid-ossp", "pgcrypto"], + "roles": { + "admin": "admin_role", + "app": "app_role", + "anonymous": "anon_role", + "authenticated": "auth_role" + }, + "connections": { + "app": { + "user": "app_user", + "password": "app_password" + }, + "admin": { + "user": "admin_user", + "password": "admin_password" + } + } + }, + "deployment": { + "useTx": true, + "fast": false, + "usePlan": true + } +} +``` + +## Integration with pgsql-test + +```typescript +import { getConnEnvOptions } from '@pgpmjs/env'; +import { getConnections } from 'pgsql-test'; + +const connOptions = getConnEnvOptions(); +const { db, teardown } = await getConnections(connOptions); +``` + +## Integration with pgpm CLI + +The pgpm CLI uses @pgpmjs/env internally. Quick setup: + +```bash +# Export standard PostgreSQL env vars +eval "$(pgpm env)" + +# Now all pgpm commands use these vars +pgpm deploy --createdb +``` + +## Best Practices + +1. **Use getEnvOptions()**: Let the library handle merging +2. **Config file for defaults**: Put project defaults in pgpm.json +3. **Env vars for secrets**: Never commit passwords to pgpm.json +4. **Override at runtime**: Pass overrides for test-specific config +5. **Consistent cwd**: Pass explicit cwd when running from different directories + +## References + +- Related skill: `references/cli.md` for CLI commands +- Related skill: `references/workspace.md` for workspace configuration +- Related skill: `github-workflows-pgpm` for CI/CD environment setup +- Related skill: `constructive-env` — Covers the full two-layer architecture (`@pgpmjs/env` + `@constructive-io/graphql-env`), GraphQL-specific env vars, SMTP config, and the "which package to import" decision guide diff --git a/.agents/skills/pgpm/references/extensions.md b/.agents/skills/pgpm/references/extensions.md new file mode 100644 index 000000000..41b4d0df2 --- /dev/null +++ b/.agents/skills/pgpm/references/extensions.md @@ -0,0 +1,197 @@ +# pgpm Extensions + +How extensions and modules work in pgpm — adding dependencies, installing packages, and understanding the .control file. + +## When to Apply + +Use this skill when: +- Adding a PostgreSQL extension (uuid-ossp, pgcrypto, plpgsql, etc.) to a module +- Installing a pgpm-published module (@pgpm/faker, @pgpm/base32, etc.) +- Editing a `.control` file's `requires` list +- Running `pgpm extension` or `pgpm install` +- Debugging missing extension errors during deploy + +## Critical Rule + +**NEVER run `CREATE EXTENSION` directly in SQL migration files.** pgpm is deterministic — it reads the `.control` file and handles extension creation automatically during `pgpm deploy`. Writing `CREATE EXTENSION` in a deploy script will cause errors or duplicate operations. + +## Two Kinds of Extensions + +### 1. Native PostgreSQL Extensions + +Built into Postgres or installed via OS packages. Examples: + +| Extension | Purpose | +|-----------|---------| +| `uuid-ossp` | UUID generation (`uuid_generate_v4()`) | +| `pgcrypto` | Cryptographic functions (`gen_random_bytes()`) | +| `plpgsql` | PL/pgSQL procedural language | +| `pg_trgm` | Trigram text similarity | +| `citext` | Case-insensitive text | +| `hstore` | Key-value store | + +These are resolved by Postgres itself during deploy. pgpm issues `CREATE EXTENSION IF NOT EXISTS` for them automatically. + +### 2. pgpm Modules + +Published to npm under scoped names (e.g., `@pgpm/faker`, `@pgpm/base32`, `@pgpm/uuid`). These contain their own deploy/revert/verify scripts and are installed into the workspace's `extensions/` directory. + +| npm Name | Control Name | Purpose | +|----------|-------------|---------| +| `@pgpm/base32` | `pgpm-base32` | Base32 encoding | +| `@pgpm/types` | `pgpm-types` | Common types | +| `@pgpm/verify` | `pgpm-verify` | Verification helpers | +| `@pgpm/uuid` | `pgpm-uuid` | UUID utilities | +| `@pgpm/faker` | `pgpm-faker` | Test data generation | + +During deploy, pgpm resolves these from the `extensions/` directory and deploys them before your module (topological dependency order). + +## The .control File + +Every pgpm module has a `.control` file at its root. This declares metadata and dependencies. + +### Anatomy + +``` +# my-module extension +comment = 'My module description' +default_version = '0.0.1' +requires = 'plpgsql, uuid-ossp, pgpm-base32, pgpm-types' +``` + +**Key fields:** +- `comment` — Human-readable description +- `default_version` — Version string (typically `0.0.1`) +- `requires` — Comma-separated list of dependency **control names** (not npm names) + +### Control Names vs npm Names + +The `requires` field uses **control file names**, not npm package names: + +| npm Name (for install) | Control Name (for requires) | +|------------------------|-----------------------------| +| `@pgpm/base32` | `pgpm-base32` | +| `@pgpm/types` | `pgpm-types` | +| `uuid-ossp` | `uuid-ossp` | +| `pgcrypto` | `pgcrypto` | + +See `references/module-naming.md` for the full naming convention. + +## Adding Dependencies + +### Interactive: `pgpm extension` + +Run inside a module directory to interactively select dependencies: + +```bash +cd packages/my-module +pgpm extension +``` + +This shows a checkbox picker of all available modules in the workspace. Selected items are written to the `.control` file's `requires` list. You can also type custom extension names for native Postgres extensions. + +### Installing npm-published pgpm modules: `pgpm install` + +To add an npm-published pgpm module to your workspace: + +```bash +# Install a single module +pgpm install @pgpm/base32 + +# Install multiple modules +pgpm install @pgpm/base32 @pgpm/types @pgpm/uuid + +# Install all missing modules declared in .control requires +pgpm install +``` + +`pgpm install` downloads the module from npm and places it in the workspace's `extensions/` directory (e.g., `extensions/@pgpm/base32/`). + +After installing, use `pgpm extension` to add the installed module to your `.control` file's `requires`. + +### Manual Editing + +You can also edit the `.control` file directly: + +``` +requires = 'plpgsql, uuid-ossp, pgpm-base32' +``` + +Then run `pgpm install` (no arguments) to install any missing modules. + +## The extensions/ Directory + +When you run `pgpm install @pgpm/foo`, it creates: + +``` +extensions/ + @pgpm/ + foo/ + pgpm-foo.control # Module's control file + pgpm.plan # Module's deployment plan + deploy/ # Deploy scripts + revert/ # Revert scripts + verify/ # Verify scripts + package.json # npm metadata +``` + +This directory is typically committed to version control so that `pgpm deploy` can resolve all dependencies without needing npm access. + +## Upgrading Modules + +```bash +# Upgrade a specific module +pgpm upgrade-modules @pgpm/base32 + +# Upgrade all modules in the workspace +pgpm upgrade-modules --workspace --all + +# Preview what would be upgraded +pgpm upgrade-modules --workspace --all --dry-run +``` + +## Dependency Resolution During Deploy + +When you run `pgpm deploy`, pgpm: + +1. Reads the target module's `.control` file for `requires` +2. Resolves native Postgres extensions → queues `CREATE EXTENSION IF NOT EXISTS` +3. Resolves pgpm modules from `extensions/` → deploys them first (recursively resolving their dependencies) +4. Deploys your module's changes in plan order + +This is fully automatic — you never need to manually order extension creation. + +## Common Workflows + +### Add a native Postgres extension to your module + +1. Edit `.control`: + ``` + requires = 'plpgsql, uuid-ossp, pgcrypto' + ``` +2. Deploy — pgpm creates the extensions automatically + +### Add a pgpm module dependency + +1. Install: `pgpm install @pgpm/base32` +2. Add to requires: `pgpm extension` (interactive) or edit `.control` +3. Deploy — pgpm deploys `@pgpm/base32` before your module + +### Check what's installed + +```bash +# List installed modules in the workspace extensions/ dir +ls extensions/ + +# Check a module's dependencies +cat packages/my-module/my-module.control +``` + +## Troubleshooting + +| Issue | Cause | Fix | +|-------|-------|-----| +| `extension "pgpm-foo" is not available` | Module not installed in `extensions/` | Run `pgpm install @pgpm/foo` | +| `extension "uuid-ossp" is not available` | Postgres image missing the extension | Use `docker.io/constructiveio/postgres-plus:18` or `postgres-plus:17` image | +| Deploy creates extension twice | You wrote `CREATE EXTENSION` in a deploy script | Remove it — pgpm handles this automatically | +| Wrong name in requires | Used npm name instead of control name | Use control name (e.g., `pgpm-base32` not `@pgpm/base32`) | diff --git a/.agents/skills/pgpm/references/module-naming.md b/.agents/skills/pgpm/references/module-naming.md new file mode 100644 index 000000000..dbac42747 --- /dev/null +++ b/.agents/skills/pgpm/references/module-naming.md @@ -0,0 +1,175 @@ +# PGPM Module Naming: npm Names vs Control File Names + +pgpm modules have two different identifiers that serve different purposes. Understanding when to use each is critical for correct dependency management. + +## When to Apply + +Use this skill when: +- Creating or editing `.control` files +- Writing `-- requires:` statements in SQL deploy files +- Running `pgpm install` commands +- Referencing dependencies between modules +- Publishing modules to npm + +## The Two Identifiers + +Every pgpm module has two names: + +### 1. npm Package Name (for distribution) + +Defined in `package.json` as the `name` field. Used for npm distribution and the `pgpm install` command. + +**Format:** `@scope/package-name` (scoped) or `package-name` (unscoped) + +**Examples:** +- `@sf-bot/rag-core` +- `@san-francisco/sf-docs-embeddings` +- `@pgpm/base32` + +### 2. Control File Name / Extension Name (for PostgreSQL) + +Defined by the `.control` filename and `%project=` in `pgpm.plan`. Used in PostgreSQL extension system and SQL dependency declarations. + +**Format:** `module-name` (no scope, no @ symbol) + +**Examples:** +- `rag-core` +- `sf-docs-embeddings` +- `pgpm-base32` + +## When to Use Each + +### Use npm Package Name (`@scope/name`) + +**1. pgpm install command:** +```bash +pgpm install @sf-bot/rag-core @sf-bot/rag-functions @sf-bot/rag-indexes +``` + +**2. package.json dependencies:** +```json +{ + "dependencies": { + "@sf-bot/rag-core": "^0.0.3" + } +} +``` + +### Use Control File Name (`name`) + +**1. .control file requires line:** +```sh +# sf-docs-embeddings.control +requires = 'rag-core' +``` + +**2. SQL deploy file requires comments:** +```sql +-- Deploy data/seed_collection to pg +-- requires: rag-core +``` + +**3. pgpm.plan %project declaration:** +```sh +%project=sf-docs-embeddings +``` + +**4. Cross-package references in pgpm.plan:** +```sh +data/seed [rag-core:schemas/rag/schema] 2026-01-25T00:00:00Z Author +``` + +## Real-World Example + +Consider the `sf-docs-embeddings` module: + +**package.json** (npm name for distribution): +```json +{ + "name": "@san-francisco/sf-docs-embeddings", + "version": "0.0.3" +} +``` + +**sf-docs-embeddings.control** (control name for PostgreSQL): +```sh +# sf-docs-embeddings extension +comment = 'San Francisco documentation embeddings' +default_version = '0.0.1' +requires = 'rag-core' +``` + +**pgpm.plan** (control name for project): +```sh +%project=sf-docs-embeddings +``` + +**deploy/data/seed_collection.sql** (control name in requires): +```sql +-- Deploy data/seed_collection to pg +-- requires: rag-core +``` + +## The Mapping + +pgpm maintains an internal mapping between control names and npm names. When you run `pgpm install`, it: + +1. Reads the `.control` file's `requires` list (control names) +2. Maps those to npm package names +3. Installs the npm packages + +For example, if your `.control` has `requires = 'pgpm-base32'`, pgpm knows to install `@pgpm/base32` from npm. + +## Common Mistakes + +### Wrong: Using npm name in .control file +```sh +# WRONG +requires = '@sf-bot/rag-core' + +# CORRECT +requires = 'rag-core' +``` + +### Wrong: Using control name in pgpm install +```bash +# WRONG +pgpm install rag-core + +# CORRECT +pgpm install @sf-bot/rag-core +``` + +### Wrong: Using npm name in SQL requires +```sql +-- WRONG +-- requires: @sf-bot/rag-core + +-- CORRECT +-- requires: rag-core +``` + +## Quick Reference Table + +| Context | Use | Example | +|---------|-----|---------| +| `pgpm install` | npm name | `@sf-bot/rag-core` | +| `package.json` name | npm name | `@sf-bot/rag-core` | +| `package.json` dependencies | npm name | `@sf-bot/rag-core` | +| `.control` requires | control name | `rag-core` | +| SQL `-- requires:` | control name | `rag-core` | +| `pgpm.plan` %project | control name | `rag-core` | +| Cross-package deps | control name | `rag-core:schemas/rag` | + +## Summary + +- **npm names** (`@scope/name`): Used for distribution and installation via npm/pgpm install +- **Control names** (`name`): Used for PostgreSQL extension system, .control files, and SQL dependency declarations + +Think of it this way: npm names are for the JavaScript/npm ecosystem, control names are for the PostgreSQL ecosystem. + +## References + +- Related skill: `references/cli.md` for CLI commands +- Related skill: `references/workspace.md` for workspace structure +- Related skill: `references/changes.md` for authoring database changes diff --git a/.agents/skills/pgpm/references/nextjs-app.md b/.agents/skills/pgpm/references/nextjs-app.md new file mode 100644 index 000000000..8e7520aba --- /dev/null +++ b/.agents/skills/pgpm/references/nextjs-app.md @@ -0,0 +1,100 @@ +# Constructive Next.js App Boilerplate + +A frontend-only Next.js application that connects to a Constructive backend. Provides production-ready authentication flows, organization management, invite handling, member management, and account settings — all powered by a generated GraphQL SDK. + +## Setup + +### 1. Scaffold from Template + +```bash +pgpm init -w \ + --repo constructive-io/sandbox-templates \ + --template nextjs/constructive-app \ + --name \ + --fullName "" \ + --email "" \ + --repoName \ + --username \ + --license MIT \ + --moduleName +``` + +### 2. Install and Configure + +```bash +cd /packages/ +pnpm install +``` + +Create `.env.local`: + +```bash +NEXT_PUBLIC_SCHEMA_BUILDER_GRAPHQL_ENDPOINT=http://api.localhost:3000/graphql +``` + +### 3. Generate SDK and Start + +```bash +pnpm codegen # Generate GraphQL SDK against running backend +pnpm dev # Opens at http://localhost:3001 +``` + +## Backend Requirements + +Requires a running Constructive backend (typically via Constructive Hub): + +| Service | Port | Purpose | +|---------|------|---------| +| PostgreSQL | 5432 | Database with Constructive schema | +| GraphQL Server (Public) | 3000 | API endpoint for app operations | +| GraphQL Server (Private) | 3002 | Admin operations | +| Job Service | 8080 | Background job processing | +| Email Function | 8082 | Email sending via SMTP | +| Mailpit SMTP | 1025 | Email server (development) | +| Mailpit UI | 8025 | View sent emails | + +## Project Structure + +``` +src/ +├── app/ # Next.js App Router pages +│ ├── login/ register/ # Auth flows +│ ├── account/ settings/ # User management +│ └── orgs/[orgId]/ # Org-scoped pages (activity, invites, members, settings) +├── components/ +│ ├── ui/ # shadcn/ui components (43 components) +│ ├── auth/ # Auth forms +│ ├── organizations/ # Org CRUD +│ ├── invites/ members/ # Org management +│ └── app-shell/ # Sidebar, navigation, layout +├── graphql/ +│ └── schema-builder-sdk/ # Generated SDK (via codegen) +└── lib/ + ├── auth/ # Auth utilities and context + ├── gql/ # GraphQL hooks and query factories + └── permissions/ # Permission checking +``` + +## Customization + +### Branding + +Edit `src/config/branding.ts` — app name, tagline, logo paths, legal links. + +### Adding UI Components + +```bash +npx shadcn@latest add @constructive/ +``` + +Registry URL configured in `components.json`. Components use Base UI primitives, Tailwind CSS 4, and cva for variants. + +## Features + +- **Authentication** — Login, register, logout, password reset, email verification +- **Organizations** — Create and manage organizations +- **Invites** — Send and accept organization invites +- **Members** — Manage organization members and roles +- **Account Management** — Profile, email verification, account deletion +- **App Shell** — Sidebar navigation, theme switching, responsive layout +- **Permissions** — Role-based access control for org features diff --git a/.agents/skills/pgpm/references/pgpm-export.md b/.agents/skills/pgpm/references/pgpm-export.md new file mode 100644 index 000000000..33e104588 --- /dev/null +++ b/.agents/skills/pgpm/references/pgpm-export.md @@ -0,0 +1,56 @@ +# pgpm Export + +Export a provisioned Constructive DB back to pgpm packages — two outputs: + +| Package | Contains | +|---|---| +| Extension (`extensionName`) | Raw SQL migrations — tables, RLS, functions, indexes | +| Service (`metaExtensionName`) | Metaschema records — database/table/field/policy rows as INSERTs | + +## CLI Usage (Partial Automation) + +```bash +cd path/to/your-app # your project workspace +eval "$(pgpm env)" + +pgpm export \ + --author "name " \ + --extensionName myapp \ + --metaExtensionName myapp-svc +``` + +Three prompts still require TTY: select database, select database_id, select schemas. + +## Programmatic API + +Key steps: +1. Look up `database_id` from `metaschema_public.database` +2. Look up `schema_names` from `metaschema_public.schema` +3. Call `exportMigrations()` from `@pgpmjs/core` + +```typescript +await exportMigrations({ + project, options, + dbInfo: { dbname: HOST_DB, databaseName: DB_NAME, database_ids: [databaseId] }, + author: AUTHOR, schema_names, + extensionName: EXT_NAME, metaExtensionName: SVC_NAME, + outdir: resolve(WORKSPACE, 'packages'), +}); +``` + +## How It Works + +1. Queries `db_migrate.sql_actions` for raw migration history +2. Applies schema name replacer (internal names → portable `extensionName` prefix) +3. Writes extension package with `pgpm.plan`, `deploy/`, `revert/`, `verify/` +4. Reads metaschema records via `export-meta` +5. Writes service package + +## Re-Running + +- Interactive: prompts to confirm overwrite +- Programmatic: silently overwrites SQL files, preserves `pgpm.json`/`package.json` + +## Related + +- `constructive-sdk` skill — provision before exporting diff --git a/.agents/skills/pgpm/references/pgpm-tables.md b/.agents/skills/pgpm/references/pgpm-tables.md new file mode 100644 index 000000000..a3b662fad --- /dev/null +++ b/.agents/skills/pgpm/references/pgpm-tables.md @@ -0,0 +1,59 @@ +# pgpm Table Creation Rules + +When creating a new table in `metaschema_modules_public`, `metaschema_public`, or `services_public` schemas in the constructive-db repository, follow these steps. + +## Required: Deterministic ID Trigger + +Every table with a `uuid` primary key MUST have a `zzz_set_deterministic_id` trigger. This is critical for deterministic test runs and reproducible deployments. + +### Pattern + +For a table named `my_table` in schema `metaschema_modules_public`: + +1. **Deploy file** at `packages/metaschema/deploy/schemas/metaschema_modules_public/tables/my_table/triggers/set_deterministic_id.sql`: + +```sql +-- Deploy schemas/metaschema_modules_public/tables/my_table/triggers/set_deterministic_id to pg + +-- requires: schemas/metaschema_modules_private/schema +-- requires: schemas/metaschema_modules_public/tables/my_table/table +-- requires: schemas/metaschema_private/procedures/deterministic_id + +BEGIN; + +CREATE FUNCTION metaschema_modules_private.tg_set_my_table_deterministic_id() +RETURNS TRIGGER AS $$ +BEGIN + IF current_setting('metaschema.deterministic_ids', true) = 'true' THEN + NEW.id := metaschema_private.deterministic_id(NEW.table_id, NEW.node_type); + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER zzz_set_deterministic_id +BEFORE INSERT ON metaschema_modules_public.my_table +FOR EACH ROW +EXECUTE FUNCTION metaschema_modules_private.tg_set_my_table_deterministic_id(); + +ALTER TABLE metaschema_modules_public.my_table +ENABLE ALWAYS TRIGGER zzz_set_deterministic_id; + +COMMIT; +``` + +Note: The `deterministic_id()` arguments vary by table. Check existing tables for the correct arguments. + +2. **Verify file** — use `verify_function` and `verify_trigger` +3. **Revert file** — `DROP TRIGGER IF EXISTS` + `DROP FUNCTION IF EXISTS` +4. **pgpm.plan entry** — add with proper dependencies + +## Checklist for New Tables + +- [ ] Table DDL (deploy/verify/revert + pgpm.plan + extension SQL) +- [ ] Insert trigger (deploy/verify/revert + pgpm.plan) +- [ ] **Deterministic ID trigger** (deploy/verify/revert + pgpm.plan + extension SQL) +- [ ] COMMENT ON COLUMN for every column +- [ ] COMMENT ON TABLE +- [ ] Foreign key constraints with `@omit manyToMany` comments +- [ ] Indexes on foreign key columns diff --git a/.agents/skills/pgpm/references/plan-format.md b/.agents/skills/pgpm/references/plan-format.md new file mode 100644 index 000000000..08e81c379 --- /dev/null +++ b/.agents/skills/pgpm/references/plan-format.md @@ -0,0 +1,154 @@ +# PGPM Plan File Format + +Guide to the correct format for pgpm.plan files and common format errors. + +## When to Apply + +Use this skill when: +- Encountering "Invalid line format" errors from pgpm +- Creating new pgpm.plan files +- Adding changes with dependencies to a plan file +- Debugging plan file parse errors + +## Plan File Format + +### Basic Structure + +A pgpm.plan file has the following structure: + +``` +%syntax-version=1.0.0 +%project=module-name +%uri=module-name + +change_name [dependencies] timestamp planner # comment +``` + +### Change Line Format + +The correct format for a change line is: + +``` +change_name [dep1 dep2] 2026-01-25T00:00:00Z planner-name # optional comment +``` + +**Order matters!** The components must appear in this exact order: +1. `change_name` - The name/path of the change (e.g., `schemas/public/tables/users`) +2. `[dependencies]` - Optional, space-separated list of dependencies in square brackets +3. `timestamp` - ISO 8601 format: `YYYY-MM-DDTHH:MM:SSZ` +4. `planner` - Name of the person/entity who planned the change +5. `` - Email in angle brackets +6. `# comment` - Optional comment starting with `#` + +### Common Mistake: Dependencies After Email + +**Wrong:** +``` +data/seed_chunks 2026-01-25T00:00:00Z city-of-san-francisco [data/create_collection] # comment +``` + +**Correct:** +``` +data/seed_chunks [data/create_collection] 2026-01-25T00:00:00Z city-of-san-francisco # comment +``` + +The parser expects dependencies immediately after the change name, not after the email. + +## Error Messages + +### "Line N: Invalid line format" + +**Symptom:** +```text +PgpmError: Failed to parse plan file /path/to/pgpm.plan: Line 6: Invalid line format +``` + +**Cause:** The line doesn't match the expected format. Most commonly: +- Dependencies placed in wrong position +- Missing or malformed timestamp +- Missing angle brackets around email +- Invalid characters in change name + +**Solution:** Check the line format matches: +``` +change_name [deps] timestamp planner # comment +``` + +### Parser Regex + +The pgpm parser uses this regex pattern for change lines: +```javascript +/^(\S+)(?:\s+\[([^\]]*)\])?(?:\s+(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)(?:\s+([^<]+?))?(?:\s+<([^>]+)>)?(?:\s+#\s+(.*))?)?$/ +``` + +This breaks down as: +- `(\S+)` - change name (required) +- `(?:\s+\[([^\]]*)\])?` - dependencies in brackets (optional) +- `(?:\s+(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)` - ISO timestamp +- `(?:\s+([^<]+?))?` - planner name +- `(?:\s+<([^>]+)>)?` - email in angle brackets +- `(?:\s+#\s+(.*))?` - comment + +## Examples + +### Change Without Dependencies + +``` +schemas/public/tables/users 2026-01-25T00:00:00Z dan # create users table +``` + +### Change With Single Dependency + +``` +schemas/public/tables/posts [schemas/public/tables/users] 2026-01-25T00:00:00Z dan # posts table +``` + +### Change With Multiple Dependencies + +``` +schemas/public/views/user_posts [schemas/public/tables/users schemas/public/tables/posts] 2026-01-25T00:00:00Z dan +``` + +### Cross-Module Dependency + +``` +data/seed [other-module:schemas/setup] 2026-01-25T00:00:00Z dan # depends on other module +``` + +## SQL File Dependencies + +Dependencies can also be declared in SQL deploy files using the `-- requires:` comment: + +```sql +-- Deploy module-name:schemas/public/tables/posts to pg +-- requires: schemas/public/tables/users + +CREATE TABLE posts ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid REFERENCES users(id), + content text +); +``` + +**Note:** Do not wrap SQL in `BEGIN`/`COMMIT` transactions - pgpm handles transactions automatically. + +These are used by pgpm for dependency resolution but the plan file format is what gets parsed first. + +## Quick Reference + +| Component | Required | Position | Format | +|-----------|----------|----------|--------| +| change_name | Yes | 1st | No spaces, use `/` for paths | +| [dependencies] | No | 2nd | Space-separated in brackets | +| timestamp | Yes* | 3rd | `YYYY-MM-DDTHH:MM:SSZ` | +| planner | Yes* | 4th | Any text without `<` | +| email | Yes* | 5th | In angle brackets `<...>` | +| comment | No | 6th | After `# ` | + +*Required if any metadata is present + +## References + +- Related skill: `references/troubleshooting.md` for general pgpm issues +- Related skill: `references/dependencies.md` for dependency management +- Related skill: `references/changes.md` for adding changes to modules diff --git a/.agents/skills/pgpm/references/publishing.md b/.agents/skills/pgpm/references/publishing.md new file mode 100644 index 000000000..305cb3bf1 --- /dev/null +++ b/.agents/skills/pgpm/references/publishing.md @@ -0,0 +1,345 @@ + +# Publishing PGPM Modules (Constructive Standard) + +Publish pgpm SQL modules to npm using pgpm package bundling and lerna for versioning. This covers the workflow for @pgpm/* scoped packages. + +## When to Apply + +Use this skill when: +- Publishing SQL database modules to npm +- Bundling pgpm packages for distribution +- Managing @pgpm/* scoped packages +- Working with pgpm-modules or similar repositories + +## PGPM vs PNPM Workspaces + +| Aspect | PGPM Workspace | PNPM Workspace | +|--------|----------------|----------------| +| Purpose | SQL database modules | TypeScript/JS packages | +| Config | pnpm-workspace.yaml + pgpm.json | pnpm-workspace.yaml only | +| Build | `pgpm package` | `makage build` | +| Output | SQL bundles | dist/ folder | +| Versioning | Fixed (recommended) | Independent | + +## Workspace Structure + +```text +pgpm-modules/ +├── .gitignore +├── lerna.json +├── package.json +├── packages/ +│ ├── faker/ +│ │ ├── deploy/ +│ │ ├── revert/ +│ │ ├── verify/ +│ │ ├── package.json +│ │ ├── pgpm-faker.control +│ │ └── pgpm.plan +│ └── utils/ +│ ├── deploy/ +│ ├── revert/ +│ ├── verify/ +│ ├── package.json +│ ├── pgpm-utils.control +│ └── pgpm.plan +├── pgpm.json +├── pnpm-lock.yaml +└── pnpm-workspace.yaml +``` + +## Configuration Files + +### pgpm.json + +Points to packages containing SQL modules: + +```json +{ + "packages": [ + "packages/*" + ] +} +``` + +### pnpm-workspace.yaml + +Same packages directory: + +```yaml +packages: + - packages/* +``` + +### lerna.json (Fixed Versioning) + +For pgpm modules, use **fixed versioning** so all modules release together: + +```json +{ + "$schema": "node_modules/lerna/schemas/lerna-schema.json", + "version": "0.16.6", + "npmClient": "pnpm" +} +``` + +**Note:** Unlike TypeScript packages which often use independent versioning, pgpm modules typically use fixed versioning because they're tightly coupled. + +### Root package.json + +```json +{ + "name": "pgpm-modules", + "version": "0.0.1", + "private": true, + "repository": { + "type": "git", + "url": "https://github.com/constructive-io/pgpm-modules" + }, + "license": "MIT", + "engines": { + "node": ">=20" + }, + "scripts": { + "bundle": "pnpm -r bundle", + "lint": "pnpm -r lint", + "test": "pnpm -r test", + "deps": "pnpm up -r -i -L" + }, + "devDependencies": { + "@types/jest": "^30.0.0", + "jest": "^30.2.0", + "lerna": "^8.2.3", + "pgsql-test": "^2.18.6", + "ts-jest": "^29.4.5", + "typescript": "^5.9.3" + } +} +``` + +## Module Configuration + +### Module package.json + +```json +{ + "name": "@pgpm/faker", + "version": "0.16.0", + "description": "Fake data generation utilities for testing", + "author": "Dan Lynch ", + "keywords": ["postgresql", "pgpm", "faker", "testing"], + "publishConfig": { + "access": "public" + }, + "scripts": { + "bundle": "pgpm package", + "test": "jest", + "test:watch": "jest --watch" + }, + "dependencies": { + "@pgpm/types": "workspace:*", + "@pgpm/verify": "workspace:*" + }, + "devDependencies": { + "pgpm": "^1.3.0" + }, + "repository": { + "type": "git", + "url": "https://github.com/constructive-io/pgpm-modules" + } +} +``` + +**Key differences from TypeScript packages:** +- No `publishConfig.directory` — publishes from package root +- Uses `pgpm package` for bundling instead of makage +- Dependencies on other @pgpm/* modules use `workspace:*` + +### Module .control File + +```sh +# pgpm-faker.control +comment = 'Fake data generation utilities' +default_version = '0.16.0' +requires = 'plpgsql,uuid-ossp' +``` + +### Module pgpm.plan + +```sh +%syntax-version=1.0.0 +%project=pgpm-faker +%uri=pgpm-faker + +schemas/faker 2025-01-01T00:00:00Z Author +schemas/faker/functions/random_name [schemas/faker] 2025-01-01T00:00:00Z Author +``` + +## Build Workflow + +### Bundle a Module + +```bash +cd packages/faker +pgpm package +``` + +Or bundle all modules: + +```bash +pnpm -r bundle +``` + +### Run Tests + +```bash +# All modules +pnpm -r test + +# Specific module +pnpm --filter @pgpm/faker test +``` + +## Publishing Workflow + +### 1. Prepare + +```bash +pnpm install +pnpm -r bundle +pnpm -r test +``` + +### 2. Version + +```bash +# Fixed versioning (all packages get same version) +pnpm lerna version + +# Or with conventional commits +pnpm lerna version --conventional-commits +``` + +### 3. Publish + +```bash +# Use from-package to publish versioned packages +pnpm lerna publish from-package +``` + +### One-Liner + +```bash +pnpm install && pnpm -r bundle && pnpm -r test && pnpm lerna version && pnpm lerna publish from-package +``` + +## Dry Run Commands + +```bash +# Test versioning (no git operations) +pnpm lerna version --no-git-tag-version --no-push + +# Test publishing +pnpm lerna publish from-package --dry-run +``` + +## Module Dependencies + +### Internal Dependencies + +Use `workspace:*` for dependencies on other pgpm modules: + +```json +{ + "dependencies": { + "@pgpm/types": "workspace:*", + "@pgpm/verify": "workspace:*" + } +} +``` + +### SQL Dependencies + +Declare SQL-level dependencies in the .control file: + +```sh +requires = 'plpgsql,uuid-ossp,@pgpm/types' +``` + +And in deploy scripts: + +```sql +-- Deploy: schemas/faker/functions/random_name +-- requires: schemas/faker +-- requires: @pgpm/types:schemas/types + +CREATE FUNCTION faker.random_name() +RETURNS TEXT AS $$ + -- implementation +$$ LANGUAGE plpgsql; +``` + +## Three-File Pattern + +Every SQL change has three files: + +| File | Purpose | +|------|---------| +| `deploy/.sql` | Creates the object | +| `revert/.sql` | Removes the object | +| `verify/.sql` | Confirms deployment | + +**Example:** + +`deploy/schemas/faker/functions/random_name.sql`: +```sql +-- Deploy: schemas/faker/functions/random_name +-- requires: schemas/faker + +BEGIN; +CREATE FUNCTION faker.random_name() +RETURNS TEXT AS $$ +BEGIN + RETURN 'John Doe'; +END; +$$ LANGUAGE plpgsql; +COMMIT; +``` + +`revert/schemas/faker/functions/random_name.sql`: +```sql +-- Revert: schemas/faker/functions/random_name + +BEGIN; +DROP FUNCTION IF EXISTS faker.random_name(); +COMMIT; +``` + +`verify/schemas/faker/functions/random_name.sql`: +```sql +-- Verify: schemas/faker/functions/random_name + +SELECT verify_function('faker.random_name'); +``` + +## Naming Conventions + +- Package name: `@pgpm/` +- Control file: `pgpm-.control` +- SQL uses snake_case for identifiers +- Never use `CREATE OR REPLACE` — pgpm is deterministic + +## Best Practices + +1. **Fixed versioning**: Use for tightly coupled SQL modules +2. **Test before publish**: Run `pnpm -r test` to verify all modules +3. **Bundle before publish**: Run `pnpm -r bundle` to create packages +4. **Use verify helpers**: Leverage @pgpm/verify for consistent verification +5. **Document dependencies**: Keep .control file and SQL requires in sync + +## References + +- Related reference: `references/workspace.md` for workspace setup +- Related reference: `references/changes.md` for authoring SQL changes +- Related reference: `references/dependencies.md` for managing dependencies +- Related skill: `pnpm-publishing` for TypeScript package publishing diff --git a/.agents/skills/pgpm/references/sql-conventions.md b/.agents/skills/pgpm/references/sql-conventions.md new file mode 100644 index 000000000..758604720 --- /dev/null +++ b/.agents/skills/pgpm/references/sql-conventions.md @@ -0,0 +1,309 @@ + +# pgpm SQL Conventions + +Rules and format for writing SQL migration files in pgpm modules. + +## When to Apply + +Use this skill when: +- Writing new deploy/revert/verify SQL files +- Adding database changes to a pgpm module +- Reviewing SQL migration code for correctness +- Debugging deployment failures related to SQL format + +## Critical Rules + +### 1. NEVER Use CREATE OR REPLACE + +pgpm is **deterministic** — each change is deployed exactly once and reverted exactly once. Use `CREATE`, not `CREATE OR REPLACE`: + +```sql +-- CORRECT +CREATE FUNCTION app.my_function() ... + +-- WRONG — never do this in pgpm +CREATE OR REPLACE FUNCTION app.my_function() ... +``` + +If you need to modify an existing function, create a **new change** that drops and recreates it, or use the revert/redeploy cycle. + +### 2. NO Transaction Wrapping + +**Do NOT add `BEGIN`/`COMMIT` or `BEGIN`/`ROLLBACK` to your SQL files.** pgpm handles transactions automatically. Just write the raw SQL: + +```sql +-- CORRECT — just the SQL +-- Deploy schemas/app/tables/users to pg + +CREATE TABLE app.users ( + id uuid PRIMARY KEY DEFAULT uuid_generate_v4(), + email text NOT NULL UNIQUE, + name text NOT NULL, + created_at timestamptz NOT NULL DEFAULT now() +); + +-- WRONG — do not wrap in transactions +BEGIN; +CREATE TABLE app.users ( ... ); +COMMIT; +``` + +### 3. Use snake_case for All Identifiers + +All SQL identifiers must use `snake_case`: + +```sql +-- CORRECT +CREATE TABLE app.user_profiles ( + user_id uuid NOT NULL, + display_name text, + created_at timestamptz NOT NULL DEFAULT now() +); + +-- WRONG +CREATE TABLE app.userProfiles ( + userId uuid NOT NULL, + displayName text, + createdAt timestamptz NOT NULL DEFAULT now() +); +``` + +## File Header Format + +Every SQL file starts with a header comment declaring its purpose and path. + +### Deploy Files + +```sql +-- Deploy schemas/app/tables/users to pg + +-- requires: schemas/app/schema + +CREATE TABLE app.users ( + ... +); +``` + +### Revert Files + +```sql +-- Revert schemas/app/tables/users from pg + +DROP TABLE IF EXISTS app.users; +``` + +### Verify Files + +```sql +-- Verify schemas/app/tables/users on pg + +SELECT id, email, name, created_at +FROM app.users +WHERE FALSE; +``` + +**Header pattern:** +- Deploy: `-- Deploy to pg` +- Revert: `-- Revert from pg` +- Verify: `-- Verify on pg` + +Always check existing files in the same directory for the exact format used in that module. + +## Dependency Declarations + +Use `-- requires:` comments after the header to declare dependencies: + +```sql +-- Deploy schemas/app/tables/user_profiles to pg + +-- requires: schemas/app/schema +-- requires: schemas/app/tables/users + +CREATE TABLE app.user_profiles ( + user_id uuid NOT NULL REFERENCES app.users(id), + bio text, + avatar_url text +); +``` + +### Cross-Module Dependencies + +When depending on a change from another module, prefix with the module name: + +```sql +-- Deploy schemas/app/procedures/get_user to pg + +-- requires: schemas/app/schema +-- requires: other-module:schemas/shared/tables/users + +CREATE FUNCTION app.get_user(user_id uuid) ... +``` + +The format is `module_name:change_path`. + +## Common Change Types + +### Schema + +```sql +-- Deploy schemas/app/schema to pg + +CREATE SCHEMA app; +``` + +Revert: `DROP SCHEMA IF EXISTS app;` +Verify: `SELECT 1/count(*) FROM information_schema.schemata WHERE schema_name = 'app';` + +### Table + +```sql +-- Deploy schemas/app/tables/users to pg + +-- requires: schemas/app/schema + +CREATE TABLE app.users ( + id uuid PRIMARY KEY DEFAULT uuid_generate_v4(), + email text NOT NULL UNIQUE, + name text NOT NULL, + created_at timestamptz NOT NULL DEFAULT now() +); +``` + +Revert: `DROP TABLE IF EXISTS app.users;` +Verify: `SELECT id, email, name, created_at FROM app.users WHERE FALSE;` + +### Function / Procedure + +```sql +-- Deploy schemas/app/procedures/authenticate to pg + +-- requires: schemas/app/schema +-- requires: schemas/app/tables/users + +CREATE FUNCTION app.authenticate(email text, password text) +RETURNS app.users AS $$ +DECLARE + result app.users; +BEGIN + SELECT * INTO result + FROM app.users u + WHERE u.email = authenticate.email; + + IF result IS NULL THEN + RAISE EXCEPTION 'Invalid credentials'; + END IF; + + RETURN result; +END; +$$ LANGUAGE plpgsql STRICT SECURITY DEFINER; +``` + +Revert: `DROP FUNCTION IF EXISTS app.authenticate(text, text);` +Verify: `SELECT has_function_privilege('app.authenticate(text, text)', 'execute');` + +### Index + +```sql +-- Deploy schemas/app/tables/users/indexes/users_email_idx to pg + +-- requires: schemas/app/tables/users + +CREATE INDEX users_email_idx ON app.users (email); +``` + +Revert: `DROP INDEX IF EXISTS app.users_email_idx;` + +### Grant / RLS Policy + +```sql +-- Deploy schemas/app/tables/users/policies/users_select_policy to pg + +-- requires: schemas/app/tables/users + +ALTER TABLE app.users ENABLE ROW LEVEL SECURITY; + +CREATE POLICY users_select_policy ON app.users + FOR SELECT + TO authenticated + USING (id = current_setting('auth.user_id')::uuid); +``` + +Revert: `DROP POLICY IF EXISTS users_select_policy ON app.users;` + +### View (PostgreSQL 17+) + +```sql +-- Deploy schemas/app/views/active_users to pg + +-- requires: schemas/app/tables/users + +CREATE VIEW app.active_users + WITH (security_invoker = true) +AS + SELECT id, email, name + FROM app.users + WHERE active = true; +``` + +Note: `security_invoker` requires PostgreSQL 17+. + +### Trigger + +```sql +-- Deploy schemas/app/tables/users/triggers/update_timestamp to pg + +-- requires: schemas/app/tables/users + +CREATE FUNCTION app.tg_update_timestamp() +RETURNS TRIGGER AS $$ +BEGIN + NEW.updated_at := now(); + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER update_timestamp + BEFORE UPDATE ON app.users + FOR EACH ROW + EXECUTE FUNCTION app.tg_update_timestamp(); +``` + +## Nested Path Organization + +Changes are organized in nested directory paths that mirror the database structure: + +``` +deploy/ + schemas/ + app/ + schema.sql + tables/ + users.sql + posts.sql + posts/ + indexes/ + posts_author_idx.sql + policies/ + posts_select_policy.sql + procedures/ + authenticate.sql + views/ + active_users.sql +``` + +The path in the plan file matches the directory path: +``` +schemas/app/schema [deps] timestamp author # comment +schemas/app/tables/users [schemas/app/schema] timestamp author # comment +``` + +## Checklist for New Changes + +1. Create all three files: `deploy/`, `revert/`, `verify/` +2. Add the correct header to each file (`-- Deploy`, `-- Revert`, `-- Verify`) +3. Add `-- requires:` declarations in the deploy file +4. Add the change to `pgpm.plan` with dependencies +5. Use `CREATE` not `CREATE OR REPLACE` +6. Do NOT wrap in `BEGIN`/`COMMIT` — pgpm handles transactions +7. Use `snake_case` for all identifiers +8. Check existing files in the module for format conventions diff --git a/.agents/skills/pgpm/references/starter-kits.md b/.agents/skills/pgpm/references/starter-kits.md new file mode 100644 index 000000000..51ba7e1f1 --- /dev/null +++ b/.agents/skills/pgpm/references/starter-kits.md @@ -0,0 +1,79 @@ +# Project Scaffolding with pgpm init + +Scaffold new Constructive projects using `pgpm init` — workspace/module templates (PGPM and PNPM variants), Next.js app boilerplate, custom template repositories, and boilerplate authoring. + +## When to Apply + +Use this reference when: +- Scaffolding a new workspace or module with `pgpm init` +- Setting up a Constructive Next.js frontend application +- Using custom template repositories +- Authoring new boilerplate templates +- Setting up non-interactive `pgpm init` for CI/CD + +## Quick Start + +```bash +# Create a PGPM workspace + module +pgpm init -w + +# Create a Next.js app from template +pgpm init -w --repo constructive-io/sandbox-templates --template nextjs/constructive-app + +# Create a pure TypeScript workspace +pgpm init workspace --dir pnpm +``` + +## Available Templates + +| Template | Command | Description | +|----------|---------|-------------| +| PGPM workspace | `pgpm init workspace` | Monorepo with pgpm.json, migrations support | +| PGPM module | `pgpm init` | Database module with pgpm.plan, .control file | +| PNPM workspace | `pgpm init workspace --dir pnpm` | Pure PNPM workspace (no pgpm files) | +| PNPM module | `pgpm init --dir pnpm` | Pure TypeScript package | +| Next.js App | `pgpm init -w --repo constructive-io/sandbox-templates -t nextjs/constructive-app` | Full-stack Constructive frontend | + +## CLI Options + +| Option | Description | +|--------|-------------| +| `--repo ` | Template repository (default: constructive-io/pgpm-boilerplates) | +| `--from-branch ` | Branch/tag to use when cloning repo | +| `--dir ` | Template variant directory (e.g., pnpm, supabase) | +| `--template, -t ` | Full template path (e.g., pnpm/module) — combines dir and type | +| `--boilerplate` | Prompt to select from available boilerplates | +| `--create-workspace, -w` | Create a workspace first, then create the module inside it | +| `--no-tty` | Run in non-interactive mode | + +## Non-Interactive Mode + +For CI/CD pipelines and automation, use `--no-tty` or set `CI=true`: + +```bash +pgpm init workspace --no-tty \ + --name my-workspace \ + --fullName "Your Name" \ + --email "you@example.com" \ + --username your-github-username \ + --license MIT +``` + +### Required Parameters for Non-Interactive Module + +| Parameter | Description | +|-----------|-------------| +| `--moduleName` | Module name | +| `--moduleDesc` | Module description | +| `--fullName` | Author's full name | +| `--email` | Author's email | +| `--username` | GitHub username | +| `--repoName` | Repository name | +| `--license` | License | +| `--access` | npm access level (public/restricted) | +| `--extensions` | PostgreSQL extensions (comma-separated) | + +## Detailed References + +- [template-authoring.md](template-authoring.md) — Creating custom boilerplate templates +- [nextjs-app.md](nextjs-app.md) — Constructive Next.js app boilerplate diff --git a/.agents/skills/pgpm/references/template-authoring.md b/.agents/skills/pgpm/references/template-authoring.md new file mode 100644 index 000000000..273a6b464 --- /dev/null +++ b/.agents/skills/pgpm/references/template-authoring.md @@ -0,0 +1,111 @@ +# Custom Boilerplate Authoring + +Create and customize boilerplate templates for `pgpm init`. + +## Template Repository Structure + +``` +my-boilerplates/ + .boilerplates.json # Root config (points to default directory) + pgpm/ # Default template variant (PGPM) + module/ + .boilerplate.json # Module template config + package.json # Template files with placeholders + pgpm.plan + workspace/ + .boilerplate.json # Workspace template config + pnpm/ # Alternative variant (pure PNPM) + module/ + .boilerplate.json + workspace/ + .boilerplate.json +``` + +## Root Configuration + +`.boilerplates.json` at the repository root specifies the default template directory: + +```json +{ + "dir": "pgpm" +} +``` + +## Template Configuration + +Each template has a `.boilerplate.json` file defining its type, workspace requirements, and questions. + +### Template Types + +| Type | Description | +|------|-------------| +| `workspace` | Creates a new monorepo workspace | +| `module` | Creates a package within a workspace | +| `generic` | Standalone template (no workspace context) | + +### Workspace Requirements + +```json +{ + "type": "module", + "requiresWorkspace": "pgpm" +} +``` + +| Value | Description | +|-------|-------------| +| `"pgpm"` | Requires PGPM workspace (pgpm.json) | +| `"pnpm"` | Requires PNPM workspace (pnpm-workspace.yaml) | +| `"lerna"` | Requires Lerna workspace (lerna.json) | +| `"npm"` | Requires npm workspace (package.json with workspaces) | +| `false` | No workspace required | + +## Placeholder System + +Templates use the `____placeholder____` pattern (4 underscores on each side) for variable substitution: + +```json +{ + "name": "@____username____/____moduleName____", + "version": "0.0.1", + "description": "____moduleDesc____", + "author": "____fullName____ <____email____>" +} +``` + +## Question Configuration + +| Field | Type | Description | +|-------|------|-------------| +| `name` | string | Placeholder name (e.g., `____fullName____`) | +| `message` | string | Prompt shown to user | +| `required` | boolean | Whether the field is required | +| `type` | string | Input type: `text`, `list`, `checkbox` | +| `options` | string[] | Static options for list/checkbox | +| `default` | any | Static default value | +| `defaultFrom` | string | Resolver for dynamic default | +| `setFrom` | string | Auto-set value (skips prompt) | +| `optionsFrom` | string | Resolver for dynamic options | + +### Resolvers + +**defaultFrom:** `git.user.name`, `git.user.email`, `npm.whoami`, `workspace.dirname` + +**setFrom:** `workspace.name`, `workspace.author.name`, `workspace.author.email`, `workspace.license`, `workspace.organization.name` + +**optionsFrom:** `licenses` (SPDX license identifiers) + +## Creating a Custom Repository + +1. Create a new repository with the structure above +2. Add `.boilerplates.json` pointing to your default directory +3. Create template directories with `.boilerplate.json` configs +4. Add template files with `____placeholder____` patterns +5. Use with `pgpm init --repo owner/your-boilerplates` + +## Best Practices + +1. Use `setFrom` for values that inherit from workspace context +2. Use `defaultFrom` for sensible defaults that users can override +3. Keep placeholder names descriptive and consistent +4. Test templates with `--no-tty` to ensure all required fields are defined diff --git a/.agents/skills/pgpm/references/testing.md b/.agents/skills/pgpm/references/testing.md new file mode 100644 index 000000000..a80d977ff --- /dev/null +++ b/.agents/skills/pgpm/references/testing.md @@ -0,0 +1,292 @@ + +# PGPM Testing + +Run PostgreSQL integration tests with isolated databases using the `pgsql-test` package. + +## Testing Framework Standard + +**IMPORTANT**: Constructive projects use **Jest** as the standard testing framework. Do NOT use vitest, mocha, or other test runners unless explicitly approved. Jest provides: +- Consistent testing experience across all packages +- Built-in mocking and assertion libraries +- Snapshot testing support +- Parallel test execution + +## When to Apply + +Use this skill when: +- Writing integration tests that need a database +- Testing PGPM modules or migrations +- Setting up isolated test databases +- Seeding test data from SQL files or PGPM modules +- Running PostGraphile/GraphQL integration tests + +## Quick Start + +### Installation + +```bash +pnpm add -D pgsql-test +``` + +### Basic Test Setup + +```typescript +import { getConnections } from 'pgsql-test'; + +let db: any; +let teardown: () => Promise; + +beforeAll(async () => { + ({ db, teardown } = await getConnections()); +}); + +afterAll(() => teardown()); +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); + +test('database query works', async () => { + const result = await db.query('SELECT 1 as num'); + expect(result.rows[0].num).toBe(1); +}); +``` + +## Core API + +### getConnections() + +Creates an isolated test database and returns clients plus cleanup function. + +```typescript +import { getConnections } from 'pgsql-test'; + +const { db, teardown } = await getConnections( + connectionOptions?, // Optional: custom connection settings + seedAdapters? // Optional: array of seed adapters +); +``` + +Returns: +- `db` - PgTestClient with query methods and transaction helpers +- `teardown` - Cleanup function to drop the test database + +### PgTestClient Methods + +| Method | Description | +|--------|-------------| +| `db.query(sql, params?)` | Execute SQL query | +| `db.beforeEach()` | Start savepoint (call in beforeEach) | +| `db.afterEach()` | Rollback to savepoint (call in afterEach) | +| `db.setContext(key, value)` | Set session context variable | +| `db.getPool()` | Get underlying pg Pool | + +## Seeding Data + +### SQL File Seeding + +```typescript +import { getConnections, seed } from 'pgsql-test'; + +const { db, teardown } = await getConnections({}, [ + seed.sqlfile(['./fixtures/schema.sql', './fixtures/data.sql']) +]); +``` + +### Function Seeding + +```typescript +import { getConnections, seed } from 'pgsql-test'; + +const { db, teardown } = await getConnections({}, [ + seed.fn(async (client) => { + await client.query(` + CREATE TABLE users ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL + ) + `); + await client.query(` + INSERT INTO users (name) VALUES ('Alice'), ('Bob') + `); + }) +]); +``` + +### PGPM Module Seeding + +Deploy a PGPM module into the test database: + +```typescript +import { getConnections, seed } from 'pgsql-test'; + +const { db, teardown } = await getConnections({}, [ + seed.pgpm(process.cwd()) // Deploy module from current directory +]); +``` + +### CSV Seeding + +```typescript +import { getConnections, seed } from 'pgsql-test'; + +const { db, teardown } = await getConnections({}, [ + seed.sqlfile(['./fixtures/schema.sql']), + seed.csv('users', './fixtures/users.csv') +]); +``` + +## Test Patterns + +### Transaction Isolation + +Each test runs in a savepoint that gets rolled back: + +```typescript +beforeEach(() => db.beforeEach()); // Creates savepoint +afterEach(() => db.afterEach()); // Rolls back to savepoint + +test('insert is isolated', async () => { + await db.query("INSERT INTO users (name) VALUES ('Test')"); + // This insert is rolled back after the test +}); + +test('previous insert not visible', async () => { + const result = await db.query("SELECT * FROM users WHERE name = 'Test'"); + expect(result.rows).toHaveLength(0); // Rolled back! +}); +``` + +### Setting User Context + +For RLS (Row Level Security) testing: + +```typescript +test('user can only see own data', async () => { + await db.setContext('user_id', 'user-123'); + + const result = await db.query('SELECT * FROM user_data'); + // Only returns rows where user_id = 'user-123' +}); +``` + +### Multiple Connections + +```typescript +const { db: adminDb, teardown: teardownAdmin } = await getConnections({ + user: 'postgres' +}); + +const { db: appDb, teardown: teardownApp } = await getConnections({ + user: 'app_user' +}); +``` + +## Running Tests + +### Prerequisites + +1. Start PostgreSQL: +```bash +pgpm docker start +``` + +2. Load environment: +```bash +eval "$(pgpm env)" +``` + +3. Run tests: +```bash +pnpm test +``` + +### One-liner + +```bash +pgpm env pnpm test +``` + +### Watch Mode + +```bash +pgpm env pnpm test --watch +``` + +## Common Workflows + +### Testing PGPM Module + +```typescript +import { getConnections, seed } from 'pgsql-test'; + +describe('my-module', () => { + let db: any, teardown: () => Promise; + + beforeAll(async () => { + ({ db, teardown } = await getConnections({}, [ + seed.pgpm(__dirname + '/..') // Deploy parent module + ])); + }); + + afterAll(() => teardown()); + beforeEach(() => db.beforeEach()); + afterEach(() => db.afterEach()); + + test('function works correctly', async () => { + const result = await db.query('SELECT my_function($1)', ['input']); + expect(result.rows[0].my_function).toBe('expected'); + }); +}); +``` + +### Testing with Fixtures + +```typescript +import { getConnections, seed } from 'pgsql-test'; +import path from 'path'; + +const fixtures = path.join(__dirname, '__fixtures__'); + +beforeAll(async () => { + ({ db, teardown } = await getConnections({}, [ + seed.sqlfile([ + path.join(fixtures, 'schema.sql'), + path.join(fixtures, 'seed-data.sql') + ]) + ])); +}); +``` + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Connection refused" | Run `pgpm docker start` first | +| "Database does not exist" | Check PGDATABASE env var or use `pgpm env` | +| Tests hang | Ensure `teardown()` is called in afterAll | +| Data leaking between tests | Add `beforeEach/afterEach` savepoint calls | +| Permission denied | Check database user has CREATE DATABASE permission | +| Slow tests | Use savepoints instead of recreating database per test | + +## File Structure + +Recommended test file organization: + +``` +my-module/ + __tests__/ + __fixtures__/ + schema.sql + seed-data.sql + my-feature.test.ts + deploy/ + revert/ + verify/ + pgpm.plan +``` + +## References + +For related skills: +- Docker container management: See `references/docker.md` +- Environment variables: See `references/env.md` +- GraphQL codegen: See `constructive-graphql-codegen` skill diff --git a/.agents/skills/pgpm/references/troubleshooting.md b/.agents/skills/pgpm/references/troubleshooting.md new file mode 100644 index 000000000..ccc8534d1 --- /dev/null +++ b/.agents/skills/pgpm/references/troubleshooting.md @@ -0,0 +1,313 @@ + +# PGPM Troubleshooting + +Quick fixes for common pgpm, PostgreSQL, and testing issues. + +## When to Apply + +Use this skill when encountering: +- Connection errors to PostgreSQL +- Docker-related issues +- Environment variable problems +- Transaction aborted errors in tests +- Deployment failures + +## PostgreSQL Connection Issues + +### Docker Not Running + +**Symptom:** +```text +Cannot connect to the Docker daemon +``` + +**Solution:** +1. Start Docker Desktop +2. Wait for it to fully initialize +3. Then run pgpm commands + +### PostgreSQL Not Accepting Connections + +**Symptom:** +```text +psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed +``` + +**Solution:** +```bash +# Start PostgreSQL container +pgpm docker start + +# Load environment variables +eval "$(pgpm env)" + +# Verify connection +psql -c "SELECT version();" +``` + +### Wrong Port or Host + +**Symptom:** +```text +connection refused +``` + +**Solution:** +```bash +# Check current environment +echo $PGHOST $PGPORT + +# Reload environment +eval "$(pgpm env)" + +# Verify settings +pgpm env +``` + +## Environment Variable Issues + +### PGHOST Not Set + +**Symptom:** +```text +PGHOST not set +``` +or +```text +could not connect to server: No such file or directory +``` + +**Solution:** +```bash +# Load pgpm environment +eval "$(pgpm env)" +``` + +**Permanent fix** - add to shell config: +```bash +# Add to ~/.bashrc or ~/.zshrc +eval "$(pgpm env)" +``` + +### Environment Not Persisting + +**Symptom:** Environment variables reset after each command + +**Cause:** Running `eval $(pgpm env)` in a subshell or script + +**Solution:** Run in current shell: +```bash +# Correct - runs in current shell +eval "$(pgpm env)" + +# Wrong - runs in subshell +bash -c 'eval "$(pgpm env)"' +``` + +## Testing Issues + +### Tests Fail to Connect + +**Symptom:** Tests time out or fail with connection errors + +**Solution:** +```bash +# 1. Start PostgreSQL +pgpm docker start + +# 2. Load environment +eval "$(pgpm env)" + +# 3. Bootstrap users (run once) +pgpm admin-users bootstrap --yes + +# 4. Run tests +pnpm test +``` + +### Current Transaction Is Aborted + +**Symptom:** +```text +current transaction is aborted, commands ignored until end of transaction block +``` + +**Cause:** An error occurred in the transaction, and PostgreSQL marks the entire transaction as aborted. All subsequent queries fail until the transaction ends. + +**Solution:** Use savepoints when testing operations that should fail: + +```typescript +// Before the expected failure +const point = 'my_savepoint'; +await db.savepoint(point); + +// Operation that should fail +await expect( + db.query('INSERT INTO restricted_table ...') +).rejects.toThrow(/permission denied/); + +// After the failure - rollback to savepoint +await db.rollback(point); + +// Now you can continue using the connection +const result = await db.query('SELECT 1'); +``` + +**Pattern for multiple failures:** +```typescript +it('tests multiple failure scenarios', async () => { + // First failure + const point1 = 'first_failure'; + await db.savepoint(point1); + await expect(db.query('...')).rejects.toThrow(); + await db.rollback(point1); + + // Second failure + const point2 = 'second_failure'; + await db.savepoint(point2); + await expect(db.query('...')).rejects.toThrow(); + await db.rollback(point2); + + // Continue with passing assertions + const result = await db.query('SELECT 1'); + expect(result.rows[0]).toBeDefined(); +}); +``` + +### Tests Interfering with Each Other + +**Symptom:** Tests pass individually but fail when run together + +**Cause:** Missing or incorrect beforeEach/afterEach hooks + +**Solution:** +```typescript +beforeEach(async () => { + await pg.beforeEach(); + await db.beforeEach(); +}); + +afterEach(async () => { + await db.afterEach(); + await pg.afterEach(); +}); +``` + +## Deployment Issues + +### Module Not Found + +**Symptom:** +```text +Error: Module 'mymodule' not found +``` + +**Solution:** +1. Ensure you're in a pgpm workspace (has `pgpm.json`) +2. Check module is in `packages/` directory +3. Verify module has `.control` file + +### Dependency Not Found + +**Symptom:** +```text +Error: Change 'module:path/to/change' not found +``` + +**Solution:** +1. Check the referenced module exists +2. Verify the change path matches exactly +3. Check `-- requires:` comment syntax: + ```sql + -- requires: other_module:schemas/other/tables/table + ``` + +### Deploy Order Wrong + +**Symptom:** Foreign key or reference errors during deployment + +**Solution:** +1. Check `pgpm.plan` for correct order +2. Verify `-- requires:` comments in deploy files +3. Regenerate plan if needed: + ```bash + pgpm plan + ``` + +## Docker Issues + +### Container Won't Start + +**Symptom:** +```text +Error starting container +``` + +**Solution:** +```bash +# Stop any existing containers +pgpm docker stop + +# Remove old containers +docker rm -f pgpm-postgres + +# Start fresh +pgpm docker start +``` + +### Port Already in Use + +**Symptom:** +```text +port 5432 is already in use +``` + +**Solution:** +```bash +# Find what's using the port +lsof -i :5432 + +# Either stop that process or use a different port +# Edit docker-compose.yml to use different port +``` + +### Volume Permission Issues + +**Symptom:** +```text +Permission denied on volume mount +``` + +**Solution:** +```bash +# Remove old volumes +docker volume rm pgpm_data + +# Restart +pgpm docker start +``` + +## Quick Reference + +| Issue | Quick Fix | +|-------|-----------| +| Can't connect | `pgpm docker start && eval "$(pgpm env)"` | +| PGHOST not set | `eval "$(pgpm env)"` | +| Transaction aborted | Use savepoint pattern | +| Tests interfere | Check beforeEach/afterEach hooks | +| Module not found | Verify workspace structure | +| Port in use | `lsof -i :5432` then stop conflicting process | + +## Getting Help + +If issues persist: +1. Check pgpm version: `pgpm --version` +2. Check Docker status: `docker ps` +3. Check PostgreSQL logs: `docker logs pgpm-postgres` +4. Verify environment: `pgpm env` + +## References + +- Related reference: `references/docker.md` for Docker management +- Related reference: `references/env.md` for environment configuration +- Related skill: `pgsql-test-exceptions` for transaction handling diff --git a/.agents/skills/pgpm/references/workspace.md b/.agents/skills/pgpm/references/workspace.md new file mode 100644 index 000000000..9b59de274 --- /dev/null +++ b/.agents/skills/pgpm/references/workspace.md @@ -0,0 +1,188 @@ + +# PGPM Workspaces + +Create and manage pgpm workspaces for modular PostgreSQL development. Workspaces bring npm-style modularity to database development. + +## When to Apply + +Use this skill when: +- Starting a new modular database project +- Creating a pgpm workspace structure +- Initializing database modules +- Setting up a pnpm monorepo for database packages + +## Quick Start + +### Create a Workspace + +```bash +pgpm init workspace +``` + +Enter workspace name when prompted: +```sh +? Enter workspace name: my-database-project +``` + +This creates a complete pnpm monorepo: +```text +my-database-project/ +├── docker-compose.yml +├── pgpm.json +├── lerna.json +├── LICENSE +├── Makefile +├── package.json +├── packages/ +├── pnpm-workspace.yaml +├── README.md +└── tsconfig.json +``` + +### Install Dependencies + +```bash +cd my-database-project +pnpm install +``` + +### Create a Module + +Inside the workspace: +```bash +pgpm init +``` + +Enter module details: +```sh +? Enter module name: pets +? Select extensions: uuid-ossp, plpgsql +``` + +This creates: +```text +packages/pets/ +├── pets.control +├── pgpm.plan +├── deploy/ +├── revert/ +└── verify/ +``` + +## Workspace vs Module + +**Workspace**: Top-level directory containing your entire project. Has `pgpm.json` and `packages/` directory. Like an npm project root. + +**Module**: Self-contained database package inside the workspace. Has its own `pgpm.plan`, `.control` file, and migration directories. Like an individual npm package. + +## Key Files + +### pgpm.json (Workspace Config) + +```json +{ + "packages": ["packages/*"] +} +``` + +Points pgpm to your modules directory. + +### module.control (Module Metadata) + +```sh +# pets.control +comment = 'Pet adoption module' +default_version = '0.0.1' +requires = 'uuid-ossp,plpgsql' +``` + +Declares module name, description, version, and dependencies. + +### pgpm.plan (Migration Plan) + +```sh +%syntax-version=1.0.0 +%project=pets +%uri=pets + +schemas/pets 2025-11-14T00:00:00Z Author +schemas/pets/tables/pets [schemas/pets] 2025-11-14T00:00:00Z Author +``` + +Tracks all changes in deployment order. + +## Common Commands + +| Command | Description | +|---------|-------------| +| `pgpm init workspace` | Create new workspace | +| `pgpm init` | Create new module in workspace | +| `pgpm add ` | Add a database change | +| `pgpm deploy` | Deploy module to database | +| `pgpm verify` | Verify deployment | +| `pgpm revert` | Rollback changes | + +## Environment Setup + +Before deploying, ensure PostgreSQL is running and connection variables are loaded. + +> See `references/docker.md` for starting PostgreSQL and `references/env.md` for loading environment variables. + +```bash +# Verify connection +psql -c "SELECT version();" + +# Bootstrap database users (run once) +pgpm admin-users bootstrap --yes +``` + +## Deploy a Module + +```bash +cd packages/pets +pgpm deploy --database pets_dev --createdb --yes +``` + +pgpm: +1. Creates the database if needed +2. Resolves dependencies +3. Deploys changes in order +4. Tracks deployment in `pgpm_migrate` schema + +## Module Structure Best Practices + +Organize changes hierarchically: +```text +deploy/ +└── schemas/ + └── app/ + ├── schema.sql + ├── tables/ + │ └── users.sql + ├── functions/ + │ └── create_user.sql + └── triggers/ + └── updated_at.sql +``` + +Use nested paths: +```bash +pgpm add schemas/app/schema +pgpm add schemas/app/tables/users --requires schemas/app/schema +pgpm add schemas/app/functions/create_user --requires schemas/app/tables/users +``` + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Cannot connect to Docker" | Start Docker Desktop first | +| "PGHOST not set" | Load PG env vars (see `references/env.md`) | +| "Connection refused" | Ensure PostgreSQL is running (see `references/docker.md`) | +| Module not found | Ensure you're inside a workspace with `pgpm.json` | + +## References + +- Related reference: `references/docker.md` for Docker management +- Related reference: `references/env.md` for environment configuration +- Related reference: `references/changes.md` for authoring database changes diff --git a/AGENTS.md b/AGENTS.md index 81283511f..a647400b2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -137,6 +137,19 @@ afterEach(async () => { - **Never** skip `beforeEach`/`afterEach` hooks — tests will leak state to each other - **Never** construct connection strings manually — use the env configuration system +## Tooling Skills + +The `.agents/skills/` directory contains tooling-focused skills for this monorepo: + +| Skill | Description | When to Use | +|-------|-------------|-------------| +| `pgpm` | PostgreSQL Package Manager — migrations, CLI, Docker, CI/CD, project scaffolding, table creation rules, DB export | Database migrations, workspace/module creation, `pgpm init`, deploy/revert | +| `constructive-pnpm` | PNPM workspace management — monorepo config, dist-folder publishing with makage/lerna, dependency management | Configuring pnpm workspaces, publishing packages, managing monorepo dependencies | +| `constructive-setup` | Monorepo setup — install dependencies, start PostgreSQL, bootstrap users, build, run tests, local email services | Setting up the development environment, local dev, full pipeline | +| `constructive-testing` | PostgreSQL testing frameworks — pgsql-test, drizzle-orm-test, supabase-test | Writing database tests, testing RLS policies, seeding test data | +| `constructive-cli` | Generated CLI commands — how the CLI is generated from GraphQL schemas, codegen options, multi-target CLI | Generating CLI tools, running generated CLI, understanding codegen pipeline | +| `graphile-search` | Unified PostGraphile v5 search plugin — tsvector, BM25, pg_trgm, pgvector adapters, composite searchScore | Adding search to GraphQL, configuring search adapters, querying search via SDK | + ## Tips 1. Start with `pgpm/core/AGENTS.md` to understand the migration and plan model.