Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ Aurora-only. Why that combination is the product is [vision.md](vision.md); star
| --- | --- |
| [vision.md](vision.md) | The **vision statement** — what pg-sprite is (the reliable execution engine under a GitOps front-end like [SchemaBot](https://github.com/block/schemabot), as [Spirit](https://github.com/block/spirit) is for MySQL) and what it deliberately is not. Five pillars, success criteria, and explicit non-goals. Start here for the why. |
| [architecture.md](architecture.md) | The **one-screen codebase map** — the three layers, the package map with build status, the copy-and-swap lifecycle, and where to read more. Start here for orientation. |
| [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md) | The PostgreSQL equivalent of MySQL's [InnoDB Online DDL Operations](https://dev.mysql.com/doc/refman/8.4/en/innodb-online-ddl-operations.html) reference — lock levels, rewrite/scan behaviour, and concurrent-DML safety per operation. |
| [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md) | The PostgreSQL equivalent of MySQL's [InnoDB Online DDL Operations](https://dev.mysql.com/doc/refman/8.4/en/innodb-online-ddl-operations.html) reference — the **three buckets** (catalog-only / full scan / full rewrite) MySQL's `ALGORITHM` states map to, then lock levels, rewrite/scan behaviour, and concurrent-DML safety per operation. |
| [binary-coercible-type-changes.md](binary-coercible-type-changes.md) | **Type changes without a rewrite** — how PostgreSQL decides whether `ALTER COLUMN TYPE` is a catalog relabel or a full rewrite (the structural test, never a data scan), the four shapes that skip the rewrite and why, the six categories of reason a rewrite is forced, the changes that look free but are not (shortening `varchar`, `numeric` scale, `char(n)` → `text`), what "no rewrite" still costs, how to check ahead of time, and the exact rules pg-sprite's `binary-coercible` verdict accepts. |
| [mysql-vs-postgresql.md](mysql-vs-postgresql.md) | The **MySQL ↔ PostgreSQL comparison reference** — how each engine expresses online DDL (`ALGORITHM=`/`LOCK=` vs per-operation idioms), the lock-mode → MDL mapping, **why DDL is dangerous** (the lock-queue pile-up, the same failure mode in both engines, and its mitigations), and the per-primitive [Spirit](https://github.com/block/spirit) (MySQL) → PostgreSQL translation the copy-and-swap executor is built on. |
| [high-level-design.md](high-level-design.md) | The **high-level design** — the conceptual overview: the problem, the planner → router → executor philosophy, the execution patterns and when each is chosen, and coverage at a glance. No package/interface detail. Start here for the architecture. |
| [low-level-design.md](low-level-design.md) | The **low-level design** — the detailed engineering design: package layout, the `Executor` interface, library choices, copy-and-swap lifecycle internals, the full coverage matrix, table requirements, and the decisions remaining for later execution phases. Read this when designing the interfaces and packages. |
| [design-principles.md](design-principles.md) | The canonical **design principles** that govern the engine — safety over speed, decisions-not-options, classify-first, mandatory checksum gate, log-based CDC, and the PostgreSQL/Aurora-specific rules everything else traces back to. |
Expand Down
458 changes: 458 additions & 0 deletions docs/binary-coercible-type-changes.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/design-principles.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ the phased build plan should be traceable back to one of these.
- **Bound every exclusive lock.** Every `ACCESS EXCLUSIVE` (only the cutover swap in the happy
path) and every catalog-flip runs under `lock_timeout` + bounded retry/backoff, so the
engine never sits at the head of the lock queue and amplifies one slow transaction into an
outage (see 12-mysql-vs-postgresql.md § Why DDL is dangerous: the lock queue).
outage (see [mysql-vs-postgresql.md § Why DDL is dangerous: the lock queue](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue)).
- **Refuse the unsafe rather than guess.** Lossy conversions, PK changes, FK/trigger tables,
and ambiguous renames are rejected up front with a clear reason — never silently attempted
(see [low-level-design's requirements](low-level-design.md#table-requirements-and-unsupported-operations-postgresql-analogs)).
Expand Down
4 changes: 2 additions & 2 deletions docs/high-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,8 +285,8 @@ Two things define this path and distinguish it from existing PostgreSQL tools:
continuous re-verification loop while it waits.

The mechanism (logical decoding, chunking, the transactional swap, checkpoint/resume) and the
MySQL→PostgreSQL primitive mapping are in the low-level design and
mysql-vs-postgresql.md.
MySQL→PostgreSQL primitive mapping are in the [low-level design](low-level-design.md#copy-and-swap-executor-lifecycle) and
[mysql-vs-postgresql.md](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping).

## What it covers (and what it deliberately does not)

Expand Down
6 changes: 3 additions & 3 deletions docs/invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ order-preserving apply within the batch, per-row retry on unique violation, or d
pairs — and prove convergence under test. The checksum (CO-1) backstops, but the applier must
converge without it. *Enforced:* applier batch semantics (design work, Phase 6). *Source:* Spirit
`pkg/change/README.md` (the REPLACE rationale) — the PG translation in
mysql-vs-postgresql
[mysql-vs-postgresql](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping)
is incomplete without this.

### CO-7 — Every statement parses, or it is an error
Expand Down Expand Up @@ -138,7 +138,7 @@ mutual-exclusion gap called out in the validation review.
The cutover swap is the only `ACCESS EXCLUSIVE` acquisition in the happy path, and **every**
strong-lock acquisition (swap, catalog flips, trigger install in fallback mode) runs under
`lock_timeout` + bounded retry/backoff so the engine never sits at the head of the lock queue
(mysql-vs-postgresql § the lock queue).
([mysql-vs-postgresql § the lock queue](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue)).
**Exception policy required:** `CREATE INDEX CONCURRENTLY` and `REINDEX CONCURRENTLY` wait on
other transactions via lock waits that a naive `lock_timeout` cancels — leaving an `INVALID`
index — so they get their own wait policy (no per-lock timeout, one overall statement deadline)
Expand All @@ -148,7 +148,7 @@ executor's validate class deliberately keeps a bounded per-lock timeout — queu
conflicting lock holder must not stall a sequence for the whole scan budget — while the scan
itself runs under its own generous overall budget. *Enforced:* every DDL execution path in the
native and copy-and-swap executors.
*Source:* [design-principles](design-principles.md#correctness-and-safety), mysql-vs-postgresql;
*Source:* [design-principles](design-principles.md#correctness-and-safety), [mysql-vs-postgresql](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue);
CIC exception from the validation review.

### LK-3 — Pending work is claimed exactly once, and Wait means finished
Expand Down
2 changes: 1 addition & 1 deletion docs/low-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ idle and a plain rewrite is acceptable); it is an escape hatch, not a shortcut,
> executors are described in the architecture section and in
> tool-pgroll.md. The per-primitive **Spirit (MySQL) →
> PostgreSQL mapping** this executor is built on lives in
> 12-mysql-vs-postgresql.md § primitive mapping.
> [mysql-vs-postgresql.md § primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping).

```
+----------------------------------------------------------------------+
Expand Down
Loading
Loading