Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions docs/binary-coercible-type-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,8 @@ table.
| # | Category | Why the bytes cannot be reused | Example |
| --- | --- | --- | --- |
| **A** | **Different on-disk representation** | The two types encode values differently (width, layout, or structure), so a function must produce every new datum. | `integer → bigint` (4 → 8 bytes); `real → double precision`; `integer → numeric`; `json → jsonb` (text → binary tree); `uuid → text` |
| **B** | **Value-transforming conversion** | The types are "similar", but the cast function changes the bytes of at least some values — padding, masking, normalisation. | `char(10) → text` (`rtrim1()` strips the padding); `inet → cidr` (masks host bits); `char(10) → char(20)` (the padding is stored, so every datum grows) |
| **C** | **A modifier stored inside the datum changes** | The typmod is not just a check — part of it is written into each value's header, so a different typmod means a different datum even when the numeric value is equal. | `numeric(10,2) → numeric(10,3)` (display scale lives in the datum — see [below](#numeric102--numeric103-the-value-survives-the-datum-does-not)); `timestamp(6) → timestamp(3)` (rounds fractional seconds); `interval` field/precision reduction |
| **B** | **Value-transforming conversion** | The types are "similar", but the cast or coercion function changes the bytes of at least some values — padding, masking, normalisation, rounding. | `char(10) → text` (`rtrim1()` strips the padding); `inet → cidr` (masks host bits); `char(10) → char(20)` (the padding is stored, so every datum grows); `timestamp(6) → timestamp(3)` (rounds fractional seconds — `TemporalSimplify` only relabels when precision does not shrink); `interval` precision reduction (same, via `interval_support`) |
| **C** | **A modifier stored inside the datum changes** | The typmod is not just a check — part of it is written into each value's header, so a different typmod means a different datum even when the value is equal. `numeric` is the one built-in type that does this. | `numeric(10,2) → numeric(10,3)` (display scale lives in the datum — see [below](#numeric102--numeric103-the-value-survives-the-datum-does-not)) |
| **D** | **A tightening the type system cannot prove safe** | The new type or modifier could reject a value the old one accepted. PostgreSQL does not scan to see whether any row *would* be rejected; it rewrites, and fails if one is. | `varchar(100) → varchar(50)`; `text → varchar(255)`; `numeric(12,2) → numeric(10,2)`; `text → xml` (validation); `text → email` (domain with `CHECK` or `NOT NULL`); `varbit(16) → varbit(8)` |
| **E** | **Session-dependent conversion** | The result depends on runtime state, so the bytes are only reusable under one specific setting. | `timestamp → timestamptz` under any session time zone other than a fixed +00:00 |
| **F** | **Wrapped in a node the rewrite test does not recognise** | The conversion may be harmless, but it is expressed as a node type the test has no case for, so it falls through to "rewrite". | `varchar(50)[] → varchar(100)[]` (`ArrayCoerceExpr` — the scalar rule is not lifted to arrays); `integer → text` (`CoerceViaIO`); any `USING` clause containing a real function call, e.g. `USING lower(col)` or `USING col || ''` |
Expand All @@ -259,9 +259,9 @@ Common assumptions that turn out to be wrong in the safe direction:
[No rewrite is not no cost](#no-rewrite-is-not-no-cost).
- **Constraints referencing the column** (`CHECK`, `UNIQUE`, foreign keys). Same: re-created
in the statement, not a heap rewrite.
- **A stored generated column that references the altered column.** Its expression is
re-recorded as a dependency; the rewrite decision is made solely on the altered column's
own transform expression.
- **An `interval` field-only reduction** (`DAY TO SECOND` → `HOUR TO SECOND`, for example).
The transform is a bare relabel with no heap rewrite, and existing values keep components
the new declared type excludes — a day component can survive in an `interval HOUR TO SECOND`.
- **A collation change alone** (`ALTER COLUMN name TYPE text COLLATE "C"` on a `text`
column). The transform is a bare relabel — no heap rewrite — but every index on the column
is rebuilt because its sort order may differ.
Expand Down Expand Up @@ -363,6 +363,12 @@ statement rolls back; the time and lock are spent regardless).
- **A view or rule referencing the column blocks the statement outright** —
`cannot alter type of a column used by a view or rule` — whether or not the change would
have been a relabel. The view has to be dropped and recreated around the change.
- **A stored generated column whose expression references the altered column blocks the
statement the same way** — `cannot alter type of a column used by a generated column`
(`ERRCODE_FEATURE_NOT_SUPPORTED`), again regardless of whether the change would have been a
relabel. `RememberAllDependentForRebuilding` refuses rather than re-plans the generation
expression. The generated column has to be dropped before the change and re-added after
it — and re-adding a `STORED` generated column is itself a full rewrite.

## Checking before you run it

Expand Down Expand Up @@ -454,5 +460,5 @@ across the 14–18 range.
| Typmod no-op rules | `varchar_support()` (`varchar.c`), `numeric_support()` (`numeric.c`), `varbit_support()` (`varbit.c`), `timestamp_support()` → `TemporalSimplify()` (`timestamp.c`, `datetime.c`), `interval_support()` (`timestamp.c`) |
| Display scale stored in the `numeric` datum | `NumericShort` / `NumericLong` headers and `NUMERIC_DSCALE()` — `src/backend/utils/adt/numeric.c` |
| Index storage reuse after a no-rewrite change | `ATPostAlterTypeParse()` → `TryReuseIndex()` → `CheckIndexCompatible()` — `tablecmds.c`, `indexcmds.c` |
| View / rule dependency refusal | `RememberAllDependentForRebuilding()` — `tablecmds.c` |
| View / rule and generated-column dependency refusals | `RememberAllDependentForRebuilding()` — `tablecmds.c` |
| Cast methods | `CoercionMethod` enum — `src/include/catalog/pg_cast.h` |
2 changes: 1 addition & 1 deletion docs/capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ Status legend: ✅ T1 (supported today) · 🟡 T2 (planned; typed refusal today
| `ADD COLUMN ... GENERATED ... STORED` | 🟡 | copy-and-swap | Yes | Table rewrite; copy-and-swap route. The copy engine must **recompute, never copy,** generated columns on the shadow table |
| `ADD COLUMN` with inline `UNIQUE`/`PRIMARY KEY`/`REFERENCES`/`CHECK` | 🟡 | native, planned flow | Yes | The inline constraint does its index build or validation scan under the `ADD COLUMN`'s `ACCESS EXCLUSIVE` lock; refused with guidance to add the column first, then build the constraint online |
| `DROP COLUMN` | ✅ | native, as-is | Yes | Metadata-only; flagged **destructive** in the plan report |
| `ALTER COLUMN TYPE`, binary-coercible (proven against live column facts) | ✅ | native, as-is | Yes | Catalog relabel, e.g. `varchar(50)` → `varchar(100)`, `varchar` → `text` |
| `ALTER COLUMN TYPE`, binary-coercible (proven against live column facts) | ✅ | native, as-is | Yes | Catalog relabel, e.g. `varchar(50)` → `varchar(100)`, `varchar` → `text`; PostgreSQL itself refuses the change when a view, rule, or `STORED` generated column depends on the column — see [binary-coercible-type-changes.md](binary-coercible-type-changes.md#no-rewrite-is-not-no-cost) |
| `ALTER COLUMN TYPE`, general (or with `USING`) | 🟡 | copy-and-swap | Yes | Table rewrite; copy-and-swap route, refused today |
| `SET DEFAULT` / `DROP DEFAULT` / `DROP NOT NULL` | ✅ | native, as-is | Yes | Metadata-only |
| `SET NOT NULL` | ✅ | native, safer sequence | Yes | Executed as the native four-step pattern: `ADD CONSTRAINT ... CHECK (col IS NOT NULL) NOT VALID` → online `VALIDATE` → `SET NOT NULL` (catalog flip, PG 12+) → drop the scaffold check |
Expand Down
6 changes: 3 additions & 3 deletions docs/design-principles.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Design principles

The canonical list of principles that govern the engine. They are distilled from Spirit's
philosophy (see spirit-architecture-notes.md) and
philosophy (see the [Spirit README](https://github.com/block/spirit)) and
adapted for PostgreSQL. Everything in [low-level-design.md](low-level-design.md) and
the phased build plan should be traceable back to one of these.

Expand Down Expand Up @@ -184,5 +184,5 @@ enforcement mechanics) live in [tcb-model](tcb-model.md); the repo-process versi
checksum, lock behaviour).
- **Each increment is independently useful.** The build is sequenced so that early phases
(classify/print, then native-path execution) ship value on their own, and the highest-risk
components (CDC, cutover) are added last on a proven foundation (see
build-plan.md).
components (CDC, cutover) are added last on a proven foundation (see the
[current implementation status and next increment](low-level-design.md#next-step)).
6 changes: 4 additions & 2 deletions docs/high-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,5 +337,7 @@ The choices that shape everything else (the full categorized list is in
**[schemabot-integration.md](schemabot-integration.md)**.
- The detailed interfaces, package layout, libraries, lifecycle internals, full coverage matrix,
and later-phase decisions → **[low-level-design.md](low-level-design.md)**.
- How Spirit (the inspiration) works → spirit-architecture-notes.md.
- The phased plan to build it → build-plan.md.
- How Spirit (the inspiration) works → the [Spirit README](https://github.com/block/spirit) and
[PostgreSQL primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping).
- Current implementation status and the next increment →
**[low-level-design.md](low-level-design.md#next-step)**.
2 changes: 1 addition & 1 deletion docs/invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ did or didn't commit. PostgreSQL's transactional DDL makes the swap itself atomi
*client's knowledge* of the outcome is not. Retries of the cutover must be written against this
ambiguity. *Enforced:* cutover retry loop. *Source:* Spirit's cutover
(`information_schema` inspection on dropped connection,
spirit-architecture-notes).
[Spirit README](https://github.com/block/spirit#cut-over-and-cleanup)).

## State, checkpoint, and resume (ST)

Expand Down
17 changes: 11 additions & 6 deletions docs/low-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,11 @@ MySQL-only — too many MySQL-isms to retrofit cleanly). The goal is a **decoupl
executor** engine in which **copy-and-swap** is only one of several execution strategies. pg-sprite
**derives design practices from several tools**: the copy-and-swap lifecycle and operator model
from Spirit, the shadow-table approach from pg-osc/pg_repack, and the expand/contract executor
from pgroll. See spirit-architecture-notes.md
for how the Spirit original works and tool-pgroll.md for pgroll.
from pgroll. See [Spirit's README](https://github.com/block/spirit) for how the original works
(its per-primitive translation to PostgreSQL is in
[mysql-vs-postgresql.md](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping))
and [high-level-design.md § execution patterns](high-level-design.md#the-execution-patterns-and-when-each-is-chosen)
for where pgroll's expand/contract fits.

## Table of contents

Expand Down Expand Up @@ -176,7 +179,8 @@ pattern *per migration*:

- **native** for the majority (the ➖/❌ rows in [postgres-online-ddl-reference](postgres-online-ddl-reference.md));
- **log-based copy-and-swap** for transparent, heavy physical rewrites (`int→bigint`, repack)
where the change is invisible to the app — see tool-pgroll's comparison;
where the change is invisible to the app — see the pattern comparison in
[high-level-design.md](high-level-design.md#the-execution-patterns-and-when-each-is-chosen);
- **expand/contract via pgroll** for prod-critical breaking changes where **instant
reversibility** and **two live schema versions** matter more than transparency.

Expand Down Expand Up @@ -381,7 +385,8 @@ idle and a plain rewrite is acceptable); it is an escape hatch, not a shortcut,
> strategy for genuine table rewrites. Until it lands, those rewrites receive a **not
> native-safe** refusal. The `native` and `expand/contract`
> executors are described in the architecture section and in
> tool-pgroll.md. The per-primitive **Spirit (MySQL) →
> [high-level-design.md § execution patterns](high-level-design.md#the-execution-patterns-and-when-each-is-chosen).
> The per-primitive **Spirit (MySQL) →
> PostgreSQL mapping** this executor is built on lives in
> [mysql-vs-postgresql.md § primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping).

Expand Down Expand Up @@ -539,7 +544,7 @@ covered. The unsupported rows are explicit non-goals for v1.

Spirit publishes a short, deliberate list of things it **requires** of a table and things it
**refuses to do** (see [its README](https://github.com/block/spirit#unsupported-features) and
spirit-architecture-notes.md). These are not arbitrary —
[the PostgreSQL primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping)). These are not arbitrary —
each maps to a property the copy/CDC/cutover machinery depends on. Below is the faithful
translation of each constraint to PostgreSQL, **with the Postgres-specific reason**
(not just "because Spirit does it"). The coverage matrix above states *what* is supported;
Expand Down Expand Up @@ -760,7 +765,7 @@ Target the highest-value rewrite cases first: general `ALTER COLUMN TYPE`, volat
routes natively-safe operations to direct DDL. PK-required, no-FK-on-migrated-table for v1.

Both the declarative and imperative front ends exist and share classify → route, matching the
[README TL;DR](README.md#tldr-recommendation),
[README § the decided shape](README.md#the-decided-shape),
[high-level-design](high-level-design.md#two-front-ends-declarative-and-imperative), and
build-plan Phase 2. Declarative performs introspection, diff, and ordering; the imperative
`--alter` path uses the same classifier with the diff step skipped. Phase 3 makes their classified
Expand Down
2 changes: 1 addition & 1 deletion docs/plan-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ consumer rendering either into a shared surface must clamp and escape them.
| `generated-stored` | Adding a stored generated column computes every row — a full rewrite. |
| `type-rewrite` | A type conversion PostgreSQL cannot relabel — rewrite plus reindex. |
| `relocation` | SET TABLESPACE moves the heap — a rewrite-scale copy. |
| `partition-parent-lock` | Partition attach/detach in its lock-taking form. |
| `partition-parent-lock` | Creating a partition (`CREATE TABLE … PARTITION OF`): a brief `ACCESS EXCLUSIVE` on the parent, no scan. |
| `unsupported-operation` | The planner does not recognize the operation or knows no safe path for it. |

### Target-dependent refusal reasons (`reason`, `statements[].reason`)
Expand Down
23 changes: 21 additions & 2 deletions docs/postgres-online-ddl-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,21 @@ a copy-and-swap avoids the long exclusive lock. The
plan report's `decisions[].reason` vocabulary
([plan-report.md](plan-report.md#planner-decision-reasons-decisionsreason)) is this table
made machine-readable: `metadata-only` / `fast-default` / `binary-coercible` are the first
bucket, `safer-idiom` the second, `type-rewrite` / `volatile-default` / `generated-stored` /
`relocation` the third.
bucket; `safer-idiom` is the second; `online-idiom` spans the first and second buckets —
`NOT VALID`, `DROP INDEX CONCURRENTLY`, and `DETACH PARTITION CONCURRENTLY` read no rows,
while `CREATE INDEX CONCURRENTLY` and `VALIDATE CONSTRAINT` read every row; `ADD CONSTRAINT …
USING INDEX` is catalog-only when adopting an existing unique index as `UNIQUE`, but scans the
full heap under `ACCESS EXCLUSIVE` when adopting it as a `PRIMARY KEY` on a nullable column
because PostgreSQL validates `NOT NULL` — reach `NOT NULL` first as described in
[safer sequences](safer-sequences.md#the-substitutions-the-planner-makes-today). See the per-operation table.
`type-rewrite` / `volatile-default` / `generated-stored` / `relocation` are the third bucket.
The remaining reasons are first-bucket by cost but carry a warning the cost alone does not:
`partition-parent-lock` (a new partition takes a brief `ACCESS EXCLUSIVE` on the parent, so
the lock queue below applies to every query on the parent) and `app-breaking-rename` (the
catalog change is instant; the running application code that still uses the old name is
what breaks). `unsupported-operation` means the planner does not recognize the operation or
knows no safe path for it. The exclusion-constraint carve-out above is a known full-scan cost
with no online path, not an operation with no cost bucket.

The MySQL-side detail — how `ALGORITHM=` and `LOCK=` are asserted and how the two engines'
lock models line up — is in
Expand Down Expand Up @@ -353,6 +366,12 @@ the catalog instead of rewriting the table. Catalog-only; runs as written.
|---|---|---|---|
| runs as written | brief `ACCESS EXCLUSIVE` | none — column relabeled in place | 0 |

PostgreSQL refuses the statement outright when a view or rule, or a `STORED` generated
column, depends on the altered column (`cannot alter type of a column used by a generated
column`). The planner cannot see these dependencies because `Facts` carries only column
types, so dry run still reports `binary-coercible`; see
[No rewrite is not no cost](binary-coercible-type-changes.md#no-rewrite-is-not-no-cost).

The type change is binary-coercible (for example `varchar(50)` → `varchar(100)`,
or `varchar` → `text`), so PostgreSQL relabels the column in place without a
table rewrite. Runs as written. How PostgreSQL decides, which changes only
Expand Down
6 changes: 3 additions & 3 deletions docs/postgresql-version-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,6 @@ What this *excludes* and why it's fine:
exactly PG 14; the planned native and copy-and-swap executors would be unaffected, and PG 15+ removes the
limitation entirely.

See why-build-this-engine.md for why we reuse these tools as
executors rather than replace them, and build-plan.md for how the
version floor feeds the phased build.
See [vision.md](vision.md) for why we reuse these tools as executors rather than replace
them. The version floor feeds the later execution work summarized under the
[current implementation status and next increment](low-level-design.md#next-step).
Loading