From 95653c206cb02465f34c686a110eca0db6ca5de6 Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Thu, 3 Sep 2026 17:00:01 +1000 Subject: [PATCH 1/3] docs: document pull and add the baseline export + zero-diff demo step MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Onboarding an existing database is pull → commit → diff = zero; the docs and the demo tour now show that loop end to end against the built binary. --- README.md | 2 +- demo/README.md | 7 ++- demo/tour.sh | 39 +++++++++++++++-- docs/README.md | 1 + docs/cli-output-examples.md | 32 +++++++++++--- docs/pull.md | 86 +++++++++++++++++++++++++++++++++++++ 6 files changed, 155 insertions(+), 12 deletions(-) create mode 100644 docs/pull.md diff --git a/README.md b/README.md index 855421b..880a6b6 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ change — every other command is read-only or fully offline. | `migrate` | required | Resolve the target table, preflight it (privileges, partitioning, size and catalog facts), classify and route the change, then **execute** the routed SQL under bounded budgets | | `migrate --dry-run` | required | The same introspection as a real run — server version, target resolution, table facts — so the printed plan reflects the actual target; executes nothing | | `diff` | required | Introspect the live table (read-only) and materialize the desired-state file on a scratch schema inside a transaction that is always rolled back; prints the plan, changes nothing | -| `pull` | required | Introspect each supported table in a schema and create one desired-state file per table; existing files are never overwritten | +| [`pull`](docs/pull.md) | required | Introspect each supported table in a schema and create one desired-state file per table; existing files are never overwritten, and a zero-change `diff` verifies the baseline | | `status` | required | Read-only view over `pg_stat_activity` for live pg-sprite sessions on the connected database | | `fmt` | none | Canonicalize a schema file — parser only | | `lint` | none | Flag patterns the engine would refuse, rewrite, or gate, from the DDL text alone | diff --git a/demo/README.md b/demo/README.md index f8a019f..1c0d35b 100644 --- a/demo/README.md +++ b/demo/README.md @@ -7,13 +7,14 @@ in check mode, the packaged-binary smoke test CI runs. make demo # build + compose DB up + reseed + run the whole tour ``` -The tour walks four sections, each runnable on its own via +The tour walks five sections, each runnable on its own via `demo/tour.sh
` (with `PGS` and `PG_DSN` set — see the Makefile): | Section | What it shows | Writes? | | --------- | ------------------------------------------------------------------------------------------------------------------- | ------------------- | | `dryrun` | One statement per planner route, reason, and disposition: metadata-only, fast-default, binary-coercible, the safer idioms, a rewrite-required suggestion, type-rewrite, volatile-default, app-breaking-rename, destructive, relocation, and a refusal | no | | `diff` | The declarative front door: a routed convergence plan for an existing table and for a missing one | no | +| `pull` | Existing-database onboarding: export one desired file per demo table, then prove each produces a zero-change `diff` | no | | `offline` | `lint` (gates on error findings), `suggest` (advises), `fmt` (canonicalizes) — no database | no | | `exec` | Real executions: a native add, the concurrent index substitution, the four-step `SET NOT NULL` sequence, and a structured refusal (exit code 2) for a rewrite whose backend is not yet available | yes (seeded tables) | @@ -34,7 +35,9 @@ routes/reasons/destructive flags and `format_version`, verdict outcomes and reasons, the substituted `executed_sql` shape (step count plus a distinguishing fragment, so a regression that drops `CONCURRENTLY` or collapses the `SET NOT NULL` sequence turns the job red), statement counts — -and on exit codes (`0` success, `2` refusal, `1` lint gate). It never +and on exit codes (`0` success, `2` refusal, `1` lint gate). The `pull` +step has no JSON mode, so check mode asserts only its exit code and exported +file count, then uses each `diff --json` report to assert zero statements. It never asserts on human-facing prose, which is free to change. CI runs this as the `demo` job ("smoke test (built pg-sprite artifact)"): the built `bin/pg-sprite` exercised end-to-end against compose PostgreSQL. diff --git a/demo/tour.sh b/demo/tour.sh index c031de8..012a139 100755 --- a/demo/tour.sh +++ b/demo/tour.sh @@ -2,7 +2,7 @@ # # A runnable tour of the pg-sprite CLI: walk one statement through every # planner route and reason, print the declarative diff plans, run the -# offline commands, and finish by executing real schema changes — including +# offline commands, export and verify a declarative baseline, and finish by executing real schema changes — including # the safer-sequence substitutions — against the seeded demo tables. Run it # via `make demo`, which builds the binary, starts the compose database, # and reseeds demo/seed.sql first. @@ -147,6 +147,38 @@ diff_plan() { fi } +run_pull() { + heading "Existing database onboarding: export a baseline and prove zero diff" + local out_dir file_count desired out status + out_dir=$(mktemp -d "${TMPDIR:-/tmp}/pg-sprite-pull.XXXXXX") + + step "pull --schema public --out $out_dir" + status=0 + if [ "$CHECK" = 1 ]; then + "$PGS" pull --url "$PG_DSN" --schema public --out "$out_dir" >/dev/null || status=$? + assert_eq "pull exit" 0 "$status" + file_count=$(find "$out_dir" -maxdepth 1 -type f -name '*.sql' | wc -l | tr -d ' ') + assert_eq "pulled file count" 2 "$file_count" + else + "$PGS" pull --url "$PG_DSN" --schema public --out "$out_dir" || echo "(exit $?)" + fi + + for desired in "$out_dir"/*.sql; do + step "zero-diff proof: $desired" + status=0 + if [ "$CHECK" = 1 ]; then + out=$("$PGS" diff --url "$PG_DSN" --schema public --desired "$desired" --json) || status=$? + assert_eq "diff exit of [$desired]" 0 "$status" + assert_eq "diff format_version of [$desired]" 2 "$(jq -r '.format_version' <<<"$out")" + assert_eq "diff disposition of [$desired]" execute "$(jq -r '.disposition' <<<"$out")" + assert_eq "zero diff of [$desired]" 0 "$(jq -r '.statements | length' <<<"$out")" + else + "$PGS" diff --url "$PG_DSN" --schema public --desired "$desired" || echo "(exit $?)" + fi + done + rm -rf "$out_dir" +} + run_dryrun() { heading "Classification: one statement per route and reason (dry-run, no writes)" # route reason destructive disposition @@ -234,11 +266,12 @@ run_exec() { case "$section" in dryrun) run_dryrun ;; diff) run_diff ;; +pull) run_pull ;; offline) run_offline ;; exec) run_exec ;; -all) run_dryrun; run_diff; run_offline; run_exec ;; +all) run_dryrun; run_diff; run_pull; run_offline; run_exec ;; *) - echo "usage: tour.sh [dryrun|diff|offline|exec|all]" >&2 + echo "usage: tour.sh [dryrun|diff|pull|offline|exec|all]" >&2 exit 64 ;; esac diff --git a/docs/README.md b/docs/README.md index 1cd32f9..0e3e6ee 100644 --- a/docs/README.md +++ b/docs/README.md @@ -35,6 +35,7 @@ Aurora-only. Why that combination is the product is [vision.md](vision.md); star | [invariants.md](invariants.md) | The canonical **invariant registry** — testable runtime MUST-statements (correctness, locking, state/resume, refusals, orchestration), each with its enforcement point and source. Mined from this doc set plus [Spirit](https://github.com/block/spirit)'s stated safety invariants and [SchemaBot](https://github.com/block/schemabot)'s control-plane discipline; the build plan's phases carry per-invariant test obligations. | | [tcb-model.md](tcb-model.md) | The **TCB model** — the trusted-computing-base partition of the engine: which components are the small trusted core that enforces the invariant registry vs the untrusted periphery, the never-trust-callers rule, domain types that make illegal states unrepresentable, the in-TCB engineering rules (from TigerBeetle TIGER_STYLE, s2n-tls, qmail, bitcoin-core), the verification ladder, and the per-side AI-assisted development policy. | | [plan-report.md](plan-report.md) | The **plan report contract** — the versioned JSON shape both front doors emit for dry-run plans: fields, closed vocabularies, the fingerprint identity, required consumer behavior for unknown versions/values, and one generated example per source (pinned by test). | +| [pull.md](pull.md) | **Export an existing schema** — use `pull` to create one desired-state file per table, then prove the baseline round-trips with a zero-change `diff`. | | [cli-output-examples.md](cli-output-examples.md) | **CLI output examples** — one real, captured JSON output per shape the CLI produces: the plan report for every dry-run disposition (execute, safer-sequence substitution, rewrite-required, backend-unavailable, refusal, destructive), the execution verdict, exit codes, the linter, and diff. | | [safer-sequences.md](safer-sequences.md) | **Safer-sequence substitution** (the improve path) — how the planner replaces a native-but-blocking form with the ordered online sequence that reaches the same end state: a worked `ADD CONSTRAINT … UNIQUE` comparison (locking, failure modes, transactionality, cost), what the engine adds over running the idiom by hand, the substitutions made today, and the typed caveats. | | [execution-model.md](execution-model.md) | The **execution model** — why safer sequences run autocommit-each-step with no wrapping transaction (PostgreSQL forbids it for the online forms), the **committed prefix** a mid-sequence failure leaves, how the verdict reports the boundary, and the per-sequence partial-failure contracts with their retry paths. Read this to answer "if a multi-step change fails halfway, what state is my table in?" | diff --git a/docs/cli-output-examples.md b/docs/cli-output-examples.md index d04c743..8df8c75 100644 --- a/docs/cli-output-examples.md +++ b/docs/cli-output-examples.md @@ -1,11 +1,11 @@ # CLI output examples -Representative, real JSON outputs for every shape the CLI produces: the -plan report for each dry-run disposition, the execution verdict, the -linter, and diff. The human text rendering of the same reports is display -only — see the animated demos in [demos/](demos/) for how it reads; the -JSON is the machine contract. The text reports color their diagnostic -labels when stdout is a terminal (`--color=auto|always|never`; +Representative, real outputs for every shape the CLI produces: the plan +report for each dry-run disposition, the execution verdict, the linter, +pull, and diff. Where JSON is available, the human text rendering of the +same reports is display only — see the animated demos in [demos/](demos/) +for how it reads; the JSON is the machine contract. The text reports color +their diagnostic labels when stdout is a terminal (`--color=auto|always|never`; [`NO_COLOR`](https://no-color.org) and `TERM=dumb` disable auto-detection); the JSON and `diff --sql` outputs are never colored. All were captured verbatim from a real session against the compose database (`make db-up`, @@ -48,6 +48,8 @@ a verdict, not a plan report — and exit 2. The JSON report schema is - [Executable but destructive (`destructive`) — exit 0](#executable-but-destructive-destructive--exit-0) - [Lint](#lint) - [Blocking idioms flagged (`blocking-idiom`) — exit 0](#blocking-idioms-flagged-blocking-idiom--exit-0) +- [Pull](#pull) + - [Export a desired-state file per table — exit 0](#export-a-desired-state-file-per-table--exit-0) - [Diff](#diff) - [Converge to the desired state (`metadata-only`) — exit 0](#converge-to-the-desired-state-metadata-only--exit-0) @@ -413,6 +415,24 @@ $ pg-sprite lint /tmp/changes.sql --json } ``` +## Pull + +`pull` exports every renderable table in a schema as a separate desired-state +file. It currently emits a text-only per-table report rather than JSON. This +output was captured from the compose database after loading `demo/seed.sql`: + +### Export a desired-state file per table — exit 0 + +```console +$ pg-sprite pull --schema public --out /tmp/pg-sprite-pull-example +PULLED orders -> /tmp/pg-sprite-pull-example/orders.sql +PULLED users -> /tmp/pg-sprite-pull-example/users.sql +Summary: 2 pulled, 0 refused, 0 errors +``` + +See [pull.md](pull.md) for the refusal and exit-code contract and the +zero-change `diff` verification loop. + ## Diff The declarative front door: point `diff` at a reviewed desired-state file diff --git a/docs/pull.md b/docs/pull.md new file mode 100644 index 0000000..19461ec --- /dev/null +++ b/docs/pull.md @@ -0,0 +1,86 @@ +# Export a declarative baseline with `pull` + +`pg-sprite pull` reads every ordinary, non-partition-child table in one live +schema and creates a desired-state SQL file for each table. Use it to put an +existing database under declarative management, then use `diff` to prove that +the exported files describe the same schema. + +```sh +pg-sprite pull --url "$PG_DSN" --schema public --out schema +``` + +`--url` accepts a PostgreSQL URL or key/value DSN and can instead be supplied +as `PGSPRITE_URL`. `--schema` defaults to `public`; `--out` (short form `-o`) +defaults to `schema`. The shared database flags also provide TLS, timeout, and +debug controls; run `pg-sprite pull --help` for the complete list. + +## Output and exit status + +For a schema containing `accounts` and `events`, the output directory is: + +```text +schema/ +├── accounts.sql +└── events.sql +``` + +Each file contains one canonical `CREATE TABLE`, followed by that table's +`CREATE INDEX` statements. Export is create-only: `pull` creates the output +directory when necessary but never overwrites a file. Move or delete an old +baseline before refreshing it. + +Tables are processed independently rather than fail-fast. The text report has +one `PULLED`, `REFUSED`, or `ERROR` result per table and a final count. A fully +successful export exits 0; one or more model refusals exit 2; an operational +error (including an existing output file, unsafe or case-colliding file names, +or a missing schema) exits 1. If both refusals and operational errors occur, +the operational error takes precedence. `pull` does not currently have JSON +output. + +## Baseline export and zero-diff verification + +Start with an empty destination, export the live schema, and verify every file +against the same database before committing the baseline: + +```sh +export PGSPRITE_URL='postgres://user:password@localhost/database?sslmode=disable' + +rm -rf schema +pg-sprite pull --schema public --out schema + +for desired in schema/*.sql; do + pg-sprite diff --schema public --desired "$desired" --json | + jq -e '.disposition == "execute" and (.statements | length == 0)' >/dev/null +done + +git add schema +git commit -m 'Add declarative schema baseline' +``` + +The loop exits non-zero if any exported table produces a schema change plan. +Run it against the same database and schema used by `pull`; each desired file +is single-table scoped, while `--schema` tells `diff` where to find that live +table. + +This is the command-level form of `schemadiff.Render`'s round-trip guarantee: +**introspect → render → parse → diff = zero changes**. `pull` calls +`schemadiff.Introspect` and `schemadiff.Render` for each table; `Render` parses +its own output as a desired file, and integration tests materialize that output +and prove that its diff from the source model is empty. + +## Refused table shapes + +Export fails closed when the declarative model cannot represent a table +without losing meaning. Current refusals include: + +- partitioned parents (partition children are not independently exported); +- either side of classic `INHERITS` relationships; +- either side of a foreign-key relationship; +- unlogged tables and columns with explicit collations; and +- sequence-backed defaults that cannot be rendered as an owned `serial` form. + +Extension-owned tables are excluded from enumeration. Comments, storage +parameters, and non-table objects are outside the declarative model and are not +exported; manage them separately. See the complete +[declarative model boundaries](limitations.md#declarative-model-boundaries) +and [support matrix](capabilities.md#the-declarative-model-desired-files-diff-pull). From 4982a91def99ccff555ee1ac21cc74b99132d4b9 Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Fri, 4 Sep 2026 14:42:17 +1000 Subject: [PATCH 2/3] test(cli): keep the examples check scoped to --json blocks The pull example is text-only output, so the pipeline-reproduction test must not pick it up as a JSON block to compare. --- internal/cli/docs_test.go | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/internal/cli/docs_test.go b/internal/cli/docs_test.go index 0f7080c..72dc48e 100644 --- a/internal/cli/docs_test.go +++ b/internal/cli/docs_test.go @@ -63,15 +63,16 @@ func alterReport(t *testing.T, sql, schema, table string, facts planner.Facts) p return report } -// The doc's example outputs are captured pipeline output, not prose: +// The doc's --json example outputs are captured pipeline output, not prose: // rebuilding each one through the same classify-and-route pipeline must -// reproduce the published JSON exactly (up to JSON equivalence). If this +// reproduce the published JSON exactly (up to JSON equivalence). Examples +// of text-only commands are prose and stay outside this check. If this // fails, regenerate the examples in docs/cli-output-examples.md. func TestCLIOutputExamplesMatchPipelineOutput(t *testing.T) { raw, err := os.ReadFile(cliOutputExamplesDoc) require.NoError(t, err) - blocks := regexp.MustCompile("(?s)```console\n\\$ pg-sprite [^\n]*\n(.*?)```").FindAllStringSubmatch(string(raw), -1) - require.Len(t, blocks, 9, "the doc publishes nine captured outputs") + blocks := regexp.MustCompile("(?s)```console\n\\$ pg-sprite [^\n]*--json\n(.*?)```").FindAllStringSubmatch(string(raw), -1) + require.Len(t, blocks, 9, "the doc publishes nine captured --json outputs") metadataOnly := alterReport(t, "ALTER TABLE users ADD COLUMN note text", "public", "users", usersFacts()) From add5a88397197d2edd88a2bb8788a10b9e5fb7f3 Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Fri, 4 Sep 2026 19:40:15 +1000 Subject: [PATCH 3/3] docs: make the zero-diff check fail on any divergent table The pasted loop's exit status was that of its last iteration, so a non-final table that did not round-trip would still let the baseline commit proceed. The check now records every divergence and reports it before the commit step; the pull text report is stated as display-only alongside the other text renderings. --- demo/tour.sh | 2 ++ docs/cli-output-examples.md | 7 ++++--- docs/pull.md | 25 ++++++++++++++++++------- 3 files changed, 24 insertions(+), 10 deletions(-) diff --git a/demo/tour.sh b/demo/tour.sh index 012a139..ec21506 100755 --- a/demo/tour.sh +++ b/demo/tour.sh @@ -158,6 +158,7 @@ run_pull() { "$PGS" pull --url "$PG_DSN" --schema public --out "$out_dir" >/dev/null || status=$? assert_eq "pull exit" 0 "$status" file_count=$(find "$out_dir" -maxdepth 1 -type f -name '*.sql' | wc -l | tr -d ' ') + # One file per table in demo/seed.sql (orders, users). assert_eq "pulled file count" 2 "$file_count" else "$PGS" pull --url "$PG_DSN" --schema public --out "$out_dir" || echo "(exit $?)" @@ -169,6 +170,7 @@ run_pull() { if [ "$CHECK" = 1 ]; then out=$("$PGS" diff --url "$PG_DSN" --schema public --desired "$desired" --json) || status=$? assert_eq "diff exit of [$desired]" 0 "$status" + # Plan report contract version (plan.FormatVersion), same pin as diff_plan. assert_eq "diff format_version of [$desired]" 2 "$(jq -r '.format_version' <<<"$out")" assert_eq "diff disposition of [$desired]" execute "$(jq -r '.disposition' <<<"$out")" assert_eq "zero diff of [$desired]" 0 "$(jq -r '.statements | length' <<<"$out")" diff --git a/docs/cli-output-examples.md b/docs/cli-output-examples.md index edd8f9e..1c022cf 100644 --- a/docs/cli-output-examples.md +++ b/docs/cli-output-examples.md @@ -2,9 +2,10 @@ Representative, real outputs for every shape the CLI produces: the plan report for each dry-run disposition, the execution verdict, the linter, -pull, and diff. Where JSON is available, the human text rendering of the -same reports is display only — see the animated demos in [demos/](demos/) -for how it reads; the JSON is the machine contract. The text reports color +pull, and diff. Every human text rendering is display only and unpinned — +including `pull`'s, which has no JSON form yet — see the animated demos in +[demos/](demos/) for how it reads; where JSON exists it is the machine +contract. The text reports color their diagnostic labels when stdout is a terminal (`--color=auto|always|never`; [`NO_COLOR`](https://no-color.org) and `TERM=dumb` disable auto-detection); the JSON and `diff --sql` outputs are never colored. All were captured diff --git a/docs/pull.md b/docs/pull.md index 19461ec..b16364f 100644 --- a/docs/pull.md +++ b/docs/pull.md @@ -39,28 +39,39 @@ output. ## Baseline export and zero-diff verification -Start with an empty destination, export the live schema, and verify every file +Start with a destination directory that does not exist yet (`pull` never +overwrites an existing file), export the live schema, and verify every file against the same database before committing the baseline: ```sh export PGSPRITE_URL='postgres://user:password@localhost/database?sslmode=disable' -rm -rf schema pg-sprite pull --schema public --out schema +divergent=0 for desired in schema/*.sql; do pg-sprite diff --schema public --desired "$desired" --json | - jq -e '.disposition == "execute" and (.statements | length == 0)' >/dev/null + jq -e '.disposition == "execute" and (.statements | length == 0)' >/dev/null || + { echo "$desired does not round-trip"; divergent=1; } done +test "$divergent" -eq 0 +``` + +The check exits non-zero, and names each offending file, if any exported table +produces a schema change plan — the result of the loop alone would only reflect +the last file, so the loop records every divergence and the final `test` reports +it. Commit the baseline only after the check exits 0: +```sh git add schema git commit -m 'Add declarative schema baseline' ``` -The loop exits non-zero if any exported table produces a schema change plan. -Run it against the same database and schema used by `pull`; each desired file -is single-table scoped, while `--schema` tells `diff` where to find that live -table. +Run the check against the same database and schema used by `pull`; each desired +file is single-table scoped, while `--schema` tells `diff` where to find that +live table. The per-table `diff` reports are JSON; `pull` itself has no JSON +output yet, so a CI consumer that wants a machine-readable per-table pull result +runs this loop rather than parsing `pull`'s text report. This is the command-level form of `schemadiff.Render`'s round-trip guarantee: **introspect → render → parse → diff = zero changes**. `pull` calls