Fix CI python version mismatch, address minor review findings, add pg…

.github/workflows/python-test.yml

@@ -14,16 +14,18 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.12"]
+        python-version: ["3.14"]
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           submodules: "recursive"
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+      - name: Ensure podman is installed
+        run: command -v podman || (sudo apt-get update && sudo apt-get install -y podman)
       - name: Install uv
         run: curl -LsSf https://astral.sh/uv/install.sh | sh
       - name: Install project dependencies

.gitignore

@@ -14,3 +14,6 @@ docs/superpowers/
 
 # Local migration checklist (not shipped with the package)
 /MIGRATION_NOTES.md
+
+# Local follow-up tracking (not shipped with the package)
+/FOLLOWUP.md

docs/database-layout.md

@@ -0,0 +1,145 @@
+# `database/` folder layout
+
+The Postgres schema is versioned as plain `.sql` files under a `database/`
+folder in the repo — that folder *is* the source of truth for the schema.
+`pgdb testdb` (see [`pgdevkit/testdb/schema.py`](../pgdevkit/testdb/schema.py))
+applies it to a local test database; a human applies the same files to
+production.
+
+This is the single source for the convention — don't duplicate this table or
+these rules elsewhere; link here instead.
+
+---
+
+## Layer directories
+
+Top-level directories group tables/objects by conceptual layer — one
+directory per Postgres schema, or per logical grouping within one schema.
+Names and count are entirely per-project; a generic example:
+
+```
+database/
+├── schema.sql            # CREATE SCHEMA statements
+├── 0_public/              # shared types/functions usable from anywhere
+├── 1_reference_data/      # dimension / reference tables
+├── 2_transactional/       # fact / transactional tables
+├── 3_app/                 # user-editable, app-owned tables
+├── 4_reporting/           # aggregated / statistics tables
+├── permissions.sql        # grants
+```
+
+The leading number is a **display/sort aid only** — it groups related
+folders together in a file listing so a human can scan them top-to-bottom in
+a sensible order. It does not control apply order: `pgdb testdb` applies
+files by object-type priority (below) and resolves cross-file dependencies
+itself, regardless of which numbered folder a file sits in. Feel free to
+renumber layers for readability without worrying about breaking anything.
+
+---
+
+## Object-type subfolders, in apply order
+
+Within a layer directory, group files by object type. This is what actually
+controls apply order, across all layer directories — cross-file dependencies
+between objects of the same type (e.g. one view selecting from another) are
+resolved automatically; this table only fixes the order *between* types.
+This table must match `_TYPE_ORDER` / `_SCHEMA_QUALIFIED_TYPES` in
+[`pgdevkit/testdb/schema.py`](../pgdevkit/testdb/schema.py) exactly — update
+both together.
+
+| Priority | Directory | Object type |
+|---|---|---|
+| 1 | `schema` | `CREATE SCHEMA` |
+| 2 | `types` | Custom types / enums |
+| 3 | `tables` | Tables |
+| 4 | `scalar_functions` | Scalar functions |
+| 5 | `functions` | Functions |
+| 6 | `views` | Views |
+| 7 | `table_functions` | Table functions |
+| 8 | `procedures` | Procedures |
+| 100 | `permissions` | Grants |
+| 101 | `indexes` | Indexes |
+
+One object per file: `tables/user.sql`, `views/all_edits.sql`,
+`types/measurement_unit.sql`.
+
+---
+
+## File-naming conventions
+
+| Suffix | Meaning |
+|---|---|
+| `<name>.sql` | The object's live definition (`CREATE TABLE`, `CREATE OR REPLACE VIEW`, ...) |
+| `<name>.test_data.json` | Seed rows for a table — a JSON array of row objects, loaded after the table is created |
+| `<name>.init.sql` | One-time setup for an object (e.g. a backfill), run once, kept separate from the reusable definition |
+| `<name>.prod.sql` / `.prod` anywhere in the name | Production-only (real permission grants, real user accounts) — skipped by `pgdb testdb` |
+| `all.sql` | Generated concatenation of the whole tree — not hand-edited, not committed |
+
+---
+
+## Migrations
+
+One-off, non-idempotent changes (rename/drop column, backfill, data fix) go
+in a `migrations/` (or `_migration_scripts/`) folder — one file per change,
+named by date:
+
+```
+database/migrations/2026-07-10_customer_geocode.sql
+```
+
+Rules:
+
+- Skipped by `pgdb testdb` — it only applies the layer directories above.
+- Update the corresponding table/view `.sql` file in the same change so its
+  definition already reflects the new shape — the migration and the
+  source-of-truth file must never drift apart.
+- Applied to production manually, once, by a human, after being verified
+  locally.
+- Never edited after being applied — a further change gets a new dated file.
+
+---
+
+## `COMMENT ON` — document schema in the object's own file
+
+Add a `COMMENT ON` for every table, and for any column whose purpose isn't
+obvious from its name and type (flags, status codes, denormalized fields,
+units). Put it directly in the table's `.sql` file, right after the
+`CREATE TABLE` — not in a migration, wiki, or README. It lives with the
+definition it describes and survives `\d+` / `pg_catalog` inspection.
+
+```sql
+-- database/1_reference_data/tables/user.sql
+create table dim.user (
+    id bigint generated always as identity primary key,
+    email text not null,
+    is_active boolean not null default true
+);
+
+comment on table dim.user is 'End-user accounts; one row per registered person.';
+comment on column dim.user.is_active is 'False once a user is soft-deleted; keep for audit trail.';
+```
+
+---
+
+## Backfilling untracked objects
+
+If a table, scalar function, or table function was created directly on the
+database and never got a `.sql` file, use `pgdb fetch-missing` to find it and
+generate one — see `pgdb fetch-missing --help`. It connects to Postgres,
+diffs the live schema against what's already tracked under `database/`, and
+for each object you select, reverse-engineers its DDL into the matching
+layer folder's `tables/`, `views/`, `scalar_functions/`, or
+`table_functions/` subfolder. The layer folder is matched by schema name
+against the existing top-level directories under `database/`, ignoring their
+leading sort number.
+
+---
+
+## Quick checklist
+
+- [ ] New table/view/function/type gets its own `.sql` file under the right layer + object-type folder
+- [ ] Object-type folder (`tables`, `views`, ...) matches the apply-order table above — that's what governs ordering, not the layer's leading number
+- [ ] One-off changes go in `migrations/`, dated, never edited after applying
+- [ ] The live `.sql` file is updated in the same change as any migration touching that object
+- [ ] `.prod` files are production-only and skipped by `pgdb testdb`
+- [ ] Every table (and non-obvious column) has a `COMMENT ON`, placed in the object's own `.sql` file

pgdevkit/cli.py

@@ -3,6 +3,7 @@
 import os
 from pathlib import Path
 
+import psycopg
 import typer
 from rich.console import Console
 from rich.table import Table
@@ -11,6 +12,7 @@
 from . import testdb
 from .connection import build_conninfo
 from .diff import DiffKind, compute_diff
+from .fetch_missing import SUBFOLDER, find_missing_objects, layer_folder_for, reconstruct_ddl
 from .introspect import introspect_db
 from .parser import parse_directory
 
@@ -67,6 +69,67 @@ def compare(
     raise typer.Exit(1)
 
 
+@app.command("fetch-missing")
+def fetch_missing(
+    scripts_dir: Path = typer.Argument(..., help="The database/ folder to compare against and write into"),
+    url: str = typer.Option(..., "--url", help="PostgreSQL DSN (postgresql://user:pass@host:port/db)"),
+    entra_user: str | None = typer.Option(None, "--entra-user", help="Azure Entra user (triggers token auth)"),
+    write: bool = typer.Option(False, "--write", help="Write the reconstructed .sql files (default: dry run)"),
+    only: list[str] = typer.Option([], "--only", help="Only fetch schema.name (repeatable); default is everything"),
+) -> None:
+    """Find tables/views/functions that exist in the database but aren't
+    tracked under scripts_dir, and reverse-engineer their DDL into new files."""
+    if not scripts_dir.is_dir():
+        err_console.print(f"[red]Error:[/red] {scripts_dir} is not a directory")
+        raise typer.Exit(2)
+
+    conninfo = build_conninfo(url, entra_user)
+
+    with console.status("Comparing database/ against the live schema..."):
+        missing = find_missing_objects(scripts_dir, conninfo)
+
+    if only:
+        wanted = set(only)
+        missing = [m for m in missing if m.qualified_name in wanted]
+
+    if not missing:
+        console.print("[green]No missing objects.[/green]")
+        return
+
+    table = Table(box=box.SIMPLE, show_header=True, header_style="bold")
+    table.add_column("Type", style="magenta")
+    table.add_column("Object")
+    table.add_column("Destination", style="dim")
+    for m in missing:
+        dest = layer_folder_for(scripts_dir, m.schema) / SUBFOLDER[m.object_type] / f"{m.name}.sql"
+        table.add_row(m.object_type, m.qualified_name, str(dest))
+    console.print(table)
+
+    if not write:
+        console.print("\n[yellow]Dry run[/yellow] — pass --write to create these files.")
+        return
+
+    written = 0
+    with psycopg.connect(conninfo) as conn:
+        for m in missing:
+            dest_dir = layer_folder_for(scripts_dir, m.schema) / SUBFOLDER[m.object_type]
+            dest = dest_dir / f"{m.name}.sql"
+            if dest.exists():
+                console.print(f"  [yellow]SKIP[/yellow] {dest} (already exists)")
+                continue
+            try:
+                ddl = reconstruct_ddl(conn, m)
+            except Exception as e:  # noqa: BLE001
+                err_console.print(f"[red]Error[/red] reconstructing {m.qualified_name}: {e}")
+                continue
+            dest_dir.mkdir(parents=True, exist_ok=True)
+            dest.write_text(ddl, encoding="utf-8")
+            console.print(f"  [green]WROTE[/green] {dest}")
+            written += 1
+
+    console.print(f"\nWrote {written} file(s).")
+
+
 @testdb_app.command("up")
 def testdb_up() -> None:
     """Ensure the container is running, the workspace DB exists, and schema is applied."""
@@ -104,7 +167,7 @@ def testdb_run_sql(
         console.print(f"OK — {len(rows)} row(s)")
         return
     if not rows:
-        console.print("(0 rows)")
+        console.print("(0 row(s))")
         return
     table = Table(box=box.SIMPLE, show_header=True, header_style="bold")
     for col in rows[0]:

pgdevkit/db/__init__.py

@@ -0,0 +1,37 @@
+from __future__ import annotations
+
+from .connection import PgPool
+from .crud import (
+    pg_delete,
+    pg_delete_dict,
+    pg_insert,
+    pg_insert_many,
+    pg_retrieve,
+    pg_retrieve_many,
+    pg_update,
+    pg_update_dict,
+    pg_upsert,
+    pg_upsert_dict,
+    pg_upsert_many,
+    pg_upsert_many_dict,
+)
+from .loader import SqlLoader
+from .model import PostgresTableModel
+
+__all__ = [
+    "PgPool",
+    "PostgresTableModel",
+    "SqlLoader",
+    "pg_delete",
+    "pg_delete_dict",
+    "pg_insert",
+    "pg_insert_many",
+    "pg_retrieve",
+    "pg_retrieve_many",
+    "pg_update",
+    "pg_update_dict",
+    "pg_upsert",
+    "pg_upsert_dict",
+    "pg_upsert_many",
+    "pg_upsert_many_dict",
+]

pgdevkit/db/connection.py

@@ -0,0 +1,46 @@
+from __future__ import annotations
+
+import os
+
+from psycopg_pool import AsyncConnectionPool
+
+
+class PgPool:
+    """A connection pool keyed off `{env_prefix}HOST/PORT/DB/USER/PASSWORD`
+    environment variables (e.g. env_prefix="MDM_POSTGRES_")."""
+
+    def __init__(self, env_prefix: str = "POSTGRES_", max_size: int = 40) -> None:
+        self._env_prefix = env_prefix
+        self._max_size = max_size
+        self._pool: AsyncConnectionPool | None = None
+
+    def _dsn(self) -> str:
+        p = self._env_prefix
+        return (
+            f"host={os.environ[p + 'HOST']} "
+            f"port={os.environ[p + 'PORT']} "
+            f"dbname={os.environ[p + 'DB']} "
+            f"user={os.environ[p + 'USER']} "
+            f"password={os.environ[p + 'PASSWORD']}"
+        )
+
+    async def open(self) -> None:
+        if self._pool is None:
+            self._pool = AsyncConnectionPool(
+                conninfo=self._dsn,
+                open=False,
+                max_size=self._max_size,
+                check=AsyncConnectionPool.check_connection,
+            )
+        if not self._pool._opened:
+            await self._pool.open()
+
+    async def close(self) -> None:
+        if self._pool is not None:
+            await self._pool.close()
+
+    def connection(self):
+        """Return an async connection context manager from the pool."""
+        if self._pool is None:
+            raise RuntimeError("Call open() first (e.g. in app startup).")
+        return self._pool.connection()