Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ change — every other command is read-only or fully offline.
| `migrate` | required | Resolve the target table, preflight it (privileges, partitioning, size and catalog facts), classify and route the change, then **execute** the routed SQL under bounded budgets |
| `migrate --dry-run` | required | The same introspection as a real run — server version, target resolution, table facts — so the printed plan reflects the actual target; executes nothing |
| `diff` | required | Introspect the live table (read-only) and materialize the desired-state file on a scratch schema inside a transaction that is always rolled back; prints the plan, changes nothing |
| `pull` | required | Introspect each supported table in a schema and create one desired-state file per table; existing files are never overwritten |
| [`pull`](docs/pull.md) | required | Introspect each supported table in a schema and create one desired-state file per table; existing files are never overwritten, and a zero-change `diff` verifies the baseline |
| `status` | required | Read-only view over `pg_stat_activity` for live pg-sprite sessions on the connected database |
| `fmt` | none | Canonicalize a schema file — parser only |
| `lint` | none | Flag patterns the engine would refuse, rewrite, or gate, from the DDL text alone |
Expand Down
7 changes: 5 additions & 2 deletions demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,14 @@ in check mode, the packaged-binary smoke test CI runs.
make demo # build + compose DB up + reseed + run the whole tour
```

The tour walks four sections, each runnable on its own via
The tour walks five sections, each runnable on its own via
`demo/tour.sh <section>` (with `PGS` and `PG_DSN` set — see the Makefile):

| Section | What it shows | Writes? |
| --------- | ------------------------------------------------------------------------------------------------------------------- | ------------------- |
| `dryrun` | One statement per planner route, reason, and disposition: metadata-only, fast-default, binary-coercible, the safer idioms, a rewrite-required suggestion, type-rewrite, volatile-default, app-breaking-rename, destructive, relocation, and a refusal | no |
| `diff` | The declarative front door: a routed convergence plan for an existing table and for a missing one | no |
| `pull` | Existing-database onboarding: export one desired file per demo table, then prove each produces a zero-change `diff` | no |
| `offline` | `lint` (gates on error findings), `suggest` (advises), `fmt` (canonicalizes) — no database | no |
| `exec` | Real executions: a native add, the concurrent index substitution, the four-step `SET NOT NULL` sequence, and a structured refusal (exit code 2) for a rewrite whose backend is not yet available | yes (seeded tables) |

Expand All @@ -34,7 +35,9 @@ routes/reasons/destructive flags and `format_version`, verdict outcomes and
reasons, the substituted `executed_sql` shape (step count plus a
distinguishing fragment, so a regression that drops `CONCURRENTLY` or
collapses the `SET NOT NULL` sequence turns the job red), statement counts —
and on exit codes (`0` success, `2` refusal, `1` lint gate). It never
and on exit codes (`0` success, `2` refusal, `1` lint gate). The `pull`
step has no JSON mode, so check mode asserts only its exit code and exported
file count, then uses each `diff --json` report to assert zero statements. It never
asserts on human-facing prose, which is free to change. CI runs
this as the `demo` job ("smoke test (built pg-sprite artifact)"): the
built `bin/pg-sprite` exercised end-to-end against compose PostgreSQL.
Expand Down
39 changes: 36 additions & 3 deletions demo/tour.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# A runnable tour of the pg-sprite CLI: walk one statement through every
# planner route and reason, print the declarative diff plans, run the
# offline commands, and finish by executing real schema changes — including
# offline commands, export and verify a declarative baseline, and finish by executing real schema changes — including
# the safer-sequence substitutions — against the seeded demo tables. Run it
# via `make demo`, which builds the binary, starts the compose database,
# and reseeds demo/seed.sql first.
Expand Down Expand Up @@ -147,6 +147,38 @@ diff_plan() {
fi
}

run_pull() {
heading "Existing database onboarding: export a baseline and prove zero diff"
local out_dir file_count desired out status
out_dir=$(mktemp -d "${TMPDIR:-/tmp}/pg-sprite-pull.XXXXXX")

step "pull --schema public --out $out_dir"
status=0
if [ "$CHECK" = 1 ]; then
"$PGS" pull --url "$PG_DSN" --schema public --out "$out_dir" >/dev/null || status=$?
assert_eq "pull exit" 0 "$status"
file_count=$(find "$out_dir" -maxdepth 1 -type f -name '*.sql' | wc -l | tr -d ' ')
assert_eq "pulled file count" 2 "$file_count"
else
"$PGS" pull --url "$PG_DSN" --schema public --out "$out_dir" || echo "(exit $?)"
fi

for desired in "$out_dir"/*.sql; do
step "zero-diff proof: $desired"
status=0
if [ "$CHECK" = 1 ]; then
out=$("$PGS" diff --url "$PG_DSN" --schema public --desired "$desired" --json) || status=$?
assert_eq "diff exit of [$desired]" 0 "$status"
assert_eq "diff format_version of [$desired]" 2 "$(jq -r '.format_version' <<<"$out")"
assert_eq "diff disposition of [$desired]" execute "$(jq -r '.disposition' <<<"$out")"
assert_eq "zero diff of [$desired]" 0 "$(jq -r '.statements | length' <<<"$out")"
else
"$PGS" diff --url "$PG_DSN" --schema public --desired "$desired" || echo "(exit $?)"
fi
done
rm -rf "$out_dir"
}

run_dryrun() {
heading "Classification: one statement per route and reason (dry-run, no writes)"
# route reason destructive disposition
Expand Down Expand Up @@ -234,11 +266,12 @@ run_exec() {
case "$section" in
dryrun) run_dryrun ;;
diff) run_diff ;;
pull) run_pull ;;
offline) run_offline ;;
exec) run_exec ;;
all) run_dryrun; run_diff; run_offline; run_exec ;;
all) run_dryrun; run_diff; run_pull; run_offline; run_exec ;;
*)
echo "usage: tour.sh [dryrun|diff|offline|exec|all]" >&2
echo "usage: tour.sh [dryrun|diff|pull|offline|exec|all]" >&2
exit 64
;;
esac
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Aurora-only. Why that combination is the product is [vision.md](vision.md); star
| [invariants.md](invariants.md) | The canonical **invariant registry** — testable runtime MUST-statements (correctness, locking, state/resume, refusals, orchestration), each with its enforcement point and source. Mined from this doc set plus [Spirit](https://github.com/block/spirit)'s stated safety invariants and [SchemaBot](https://github.com/block/schemabot)'s control-plane discipline; the build plan's phases carry per-invariant test obligations. |
| [tcb-model.md](tcb-model.md) | The **TCB model** — the trusted-computing-base partition of the engine: which components are the small trusted core that enforces the invariant registry vs the untrusted periphery, the never-trust-callers rule, domain types that make illegal states unrepresentable, the in-TCB engineering rules (from TigerBeetle TIGER_STYLE, s2n-tls, qmail, bitcoin-core), the verification ladder, and the per-side AI-assisted development policy. |
| [plan-report.md](plan-report.md) | The **plan report contract** — the versioned JSON shape both front doors emit for dry-run plans: fields, closed vocabularies, the fingerprint identity, required consumer behavior for unknown versions/values, and one generated example per source (pinned by test). |
| [pull.md](pull.md) | **Export an existing schema** — use `pull` to create one desired-state file per table, then prove the baseline round-trips with a zero-change `diff`. |
| [cli-output-examples.md](cli-output-examples.md) | **CLI output examples** — one real, captured JSON output per shape the CLI produces: the plan report for every dry-run disposition (execute, safer-sequence substitution, rewrite-required, backend-unavailable, refusal, destructive), the execution verdict, exit codes, the linter, and diff. |
| [safer-sequences.md](safer-sequences.md) | **Safer-sequence substitution** (the improve path) — how the planner replaces a native-but-blocking form with the ordered online sequence that reaches the same end state: a worked `ADD CONSTRAINT … UNIQUE` comparison (locking, failure modes, transactionality, cost), what the engine adds over running the idiom by hand, the substitutions made today, and the typed caveats. |
| [execution-model.md](execution-model.md) | The **execution model** — why safer sequences run autocommit-each-step with no wrapping transaction (PostgreSQL forbids it for the online forms), the **committed prefix** a mid-sequence failure leaves, how the verdict reports the boundary, and the per-sequence partial-failure contracts with their retry paths. Read this to answer "if a multi-step change fails halfway, what state is my table in?" |
Expand Down
32 changes: 26 additions & 6 deletions docs/cli-output-examples.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# CLI output examples

Representative, real JSON outputs for every shape the CLI produces: the
plan report for each dry-run disposition, the execution verdict, the
linter, and diff. The human text rendering of the same reports is display
only — see the animated demos in [demos/](demos/) for how it reads; the
JSON is the machine contract. The text reports color their diagnostic
labels when stdout is a terminal (`--color=auto|always|never`;
Representative, real outputs for every shape the CLI produces: the plan
report for each dry-run disposition, the execution verdict, the linter,
pull, and diff. Where JSON is available, the human text rendering of the
same reports is display only — see the animated demos in [demos/](demos/)
for how it reads; the JSON is the machine contract. The text reports color
their diagnostic labels when stdout is a terminal (`--color=auto|always|never`;
[`NO_COLOR`](https://no-color.org) and `TERM=dumb` disable auto-detection);
the JSON and `diff --sql` outputs are never colored. All were captured
verbatim from a real session against the compose database (`make db-up`,
Expand Down Expand Up @@ -48,6 +48,8 @@ a verdict, not a plan report — and exit 2. The JSON report schema is
- [Executable but destructive (`destructive`) — exit 0](#executable-but-destructive-destructive--exit-0)
- [Lint](#lint)
- [Blocking idioms flagged (`blocking-idiom`) — exit 0](#blocking-idioms-flagged-blocking-idiom--exit-0)
- [Pull](#pull)
- [Export a desired-state file per table — exit 0](#export-a-desired-state-file-per-table--exit-0)
- [Diff](#diff)
- [Converge to the desired state (`metadata-only`) — exit 0](#converge-to-the-desired-state-metadata-only--exit-0)

Expand Down Expand Up @@ -413,6 +415,24 @@ $ pg-sprite lint /tmp/changes.sql --json
}
```

## Pull

`pull` exports every renderable table in a schema as a separate desired-state
file. It currently emits a text-only per-table report rather than JSON. This
output was captured from the compose database after loading `demo/seed.sql`:

### Export a desired-state file per table — exit 0

```console
$ pg-sprite pull --schema public --out /tmp/pg-sprite-pull-example
PULLED orders -> /tmp/pg-sprite-pull-example/orders.sql
PULLED users -> /tmp/pg-sprite-pull-example/users.sql
Summary: 2 pulled, 0 refused, 0 errors
```

See [pull.md](pull.md) for the refusal and exit-code contract and the
zero-change `diff` verification loop.

## Diff

The declarative front door: point `diff` at a reviewed desired-state file
Expand Down
86 changes: 86 additions & 0 deletions docs/pull.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Export a declarative baseline with `pull`

`pg-sprite pull` reads every ordinary, non-partition-child table in one live
schema and creates a desired-state SQL file for each table. Use it to put an
existing database under declarative management, then use `diff` to prove that
the exported files describe the same schema.

```sh
pg-sprite pull --url "$PG_DSN" --schema public --out schema
```

`--url` accepts a PostgreSQL URL or key/value DSN and can instead be supplied
as `PGSPRITE_URL`. `--schema` defaults to `public`; `--out` (short form `-o`)
defaults to `schema`. The shared database flags also provide TLS, timeout, and
debug controls; run `pg-sprite pull --help` for the complete list.

## Output and exit status

For a schema containing `accounts` and `events`, the output directory is:

```text
schema/
├── accounts.sql
└── events.sql
```

Each file contains one canonical `CREATE TABLE`, followed by that table's
`CREATE INDEX` statements. Export is create-only: `pull` creates the output
directory when necessary but never overwrites a file. Move or delete an old
baseline before refreshing it.

Tables are processed independently rather than fail-fast. The text report has
one `PULLED`, `REFUSED`, or `ERROR` result per table and a final count. A fully
successful export exits 0; one or more model refusals exit 2; an operational
error (including an existing output file, unsafe or case-colliding file names,
or a missing schema) exits 1. If both refusals and operational errors occur,
the operational error takes precedence. `pull` does not currently have JSON
output.

## Baseline export and zero-diff verification

Start with an empty destination, export the live schema, and verify every file
against the same database before committing the baseline:

```sh
export PGSPRITE_URL='postgres://user:password@localhost/database?sslmode=disable'

rm -rf schema
pg-sprite pull --schema public --out schema

for desired in schema/*.sql; do
pg-sprite diff --schema public --desired "$desired" --json |
jq -e '.disposition == "execute" and (.statements | length == 0)' >/dev/null
done

git add schema
git commit -m 'Add declarative schema baseline'
```

The loop exits non-zero if any exported table produces a schema change plan.
Run it against the same database and schema used by `pull`; each desired file
is single-table scoped, while `--schema` tells `diff` where to find that live
table.

This is the command-level form of `schemadiff.Render`'s round-trip guarantee:
**introspect → render → parse → diff = zero changes**. `pull` calls
`schemadiff.Introspect` and `schemadiff.Render` for each table; `Render` parses
its own output as a desired file, and integration tests materialize that output
and prove that its diff from the source model is empty.

## Refused table shapes

Export fails closed when the declarative model cannot represent a table
without losing meaning. Current refusals include:

- partitioned parents (partition children are not independently exported);
- either side of classic `INHERITS` relationships;
- either side of a foreign-key relationship;
- unlogged tables and columns with explicit collations; and
- sequence-backed defaults that cannot be rendered as an owned `serial` form.

Extension-owned tables are excluded from enumeration. Comments, storage
parameters, and non-table objects are outside the declarative model and are not
exported; manage them separately. See the complete
[declarative model boundaries](limitations.md#declarative-model-boundaries)
and [support matrix](capabilities.md#the-declarative-model-desired-files-diff-pull).
9 changes: 5 additions & 4 deletions internal/cli/docs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,16 @@ func alterReport(t *testing.T, sql, schema, table string, facts planner.Facts) p
return report
}

// The doc's example outputs are captured pipeline output, not prose:
// The doc's --json example outputs are captured pipeline output, not prose:
// rebuilding each one through the same classify-and-route pipeline must
// reproduce the published JSON exactly (up to JSON equivalence). If this
// reproduce the published JSON exactly (up to JSON equivalence). Examples
// of text-only commands are prose and stay outside this check. If this
// fails, regenerate the examples in docs/cli-output-examples.md.
func TestCLIOutputExamplesMatchPipelineOutput(t *testing.T) {
raw, err := os.ReadFile(cliOutputExamplesDoc)
require.NoError(t, err)
blocks := regexp.MustCompile("(?s)```console\n\\$ pg-sprite [^\n]*\n(.*?)```").FindAllStringSubmatch(string(raw), -1)
require.Len(t, blocks, 9, "the doc publishes nine captured outputs")
blocks := regexp.MustCompile("(?s)```console\n\\$ pg-sprite [^\n]*--json\n(.*?)```").FindAllStringSubmatch(string(raw), -1)
require.Len(t, blocks, 9, "the doc publishes nine captured --json outputs")

metadataOnly := alterReport(t, "ALTER TABLE users ADD COLUMN note text",
"public", "users", usersFacts())
Expand Down
Loading