diff --git a/docs/binary-coercible-type-changes.md b/docs/binary-coercible-type-changes.md index ef3c455..d47f048 100644 --- a/docs/binary-coercible-type-changes.md +++ b/docs/binary-coercible-type-changes.md @@ -238,8 +238,8 @@ table. | # | Category | Why the bytes cannot be reused | Example | | --- | --- | --- | --- | | **A** | **Different on-disk representation** | The two types encode values differently (width, layout, or structure), so a function must produce every new datum. | `integer → bigint` (4 → 8 bytes); `real → double precision`; `integer → numeric`; `json → jsonb` (text → binary tree); `uuid → text` | -| **B** | **Value-transforming conversion** | The types are "similar", but the cast function changes the bytes of at least some values — padding, masking, normalisation. | `char(10) → text` (`rtrim1()` strips the padding); `inet → cidr` (masks host bits); `char(10) → char(20)` (the padding is stored, so every datum grows) | -| **C** | **A modifier stored inside the datum changes** | The typmod is not just a check — part of it is written into each value's header, so a different typmod means a different datum even when the numeric value is equal. | `numeric(10,2) → numeric(10,3)` (display scale lives in the datum — see [below](#numeric102--numeric103-the-value-survives-the-datum-does-not)); `timestamp(6) → timestamp(3)` (rounds fractional seconds); `interval` field/precision reduction | +| **B** | **Value-transforming conversion** | The types are "similar", but the cast or coercion function changes the bytes of at least some values — padding, masking, normalisation, rounding. | `char(10) → text` (`rtrim1()` strips the padding); `inet → cidr` (masks host bits); `char(10) → char(20)` (the padding is stored, so every datum grows); `timestamp(6) → timestamp(3)` (rounds fractional seconds — `TemporalSimplify` only relabels when precision does not shrink); `interval` precision reduction (same, via `interval_support`) | +| **C** | **A modifier stored inside the datum changes** | The typmod is not just a check — part of it is written into each value's header, so a different typmod means a different datum even when the value is equal. `numeric` is the one built-in type that does this. | `numeric(10,2) → numeric(10,3)` (display scale lives in the datum — see [below](#numeric102--numeric103-the-value-survives-the-datum-does-not)) | | **D** | **A tightening the type system cannot prove safe** | The new type or modifier could reject a value the old one accepted. PostgreSQL does not scan to see whether any row *would* be rejected; it rewrites, and fails if one is. | `varchar(100) → varchar(50)`; `text → varchar(255)`; `numeric(12,2) → numeric(10,2)`; `text → xml` (validation); `text → email` (domain with `CHECK` or `NOT NULL`); `varbit(16) → varbit(8)` | | **E** | **Session-dependent conversion** | The result depends on runtime state, so the bytes are only reusable under one specific setting. | `timestamp → timestamptz` under any session time zone other than a fixed +00:00 | | **F** | **Wrapped in a node the rewrite test does not recognise** | The conversion may be harmless, but it is expressed as a node type the test has no case for, so it falls through to "rewrite". | `varchar(50)[] → varchar(100)[]` (`ArrayCoerceExpr` — the scalar rule is not lifted to arrays); `integer → text` (`CoerceViaIO`); any `USING` clause containing a real function call, e.g. `USING lower(col)` or `USING col || ''` | @@ -259,9 +259,9 @@ Common assumptions that turn out to be wrong in the safe direction: [No rewrite is not no cost](#no-rewrite-is-not-no-cost). - **Constraints referencing the column** (`CHECK`, `UNIQUE`, foreign keys). Same: re-created in the statement, not a heap rewrite. -- **A stored generated column that references the altered column.** Its expression is - re-recorded as a dependency; the rewrite decision is made solely on the altered column's - own transform expression. +- **An `interval` field-only reduction** (`DAY TO SECOND` → `HOUR TO SECOND`, for example). + The transform is a bare relabel with no heap rewrite, and existing values keep components + the new declared type excludes — a day component can survive in an `interval HOUR TO SECOND`. - **A collation change alone** (`ALTER COLUMN name TYPE text COLLATE "C"` on a `text` column). The transform is a bare relabel — no heap rewrite — but every index on the column is rebuilt because its sort order may differ. @@ -363,6 +363,12 @@ statement rolls back; the time and lock are spent regardless). - **A view or rule referencing the column blocks the statement outright** — `cannot alter type of a column used by a view or rule` — whether or not the change would have been a relabel. The view has to be dropped and recreated around the change. +- **A stored generated column whose expression references the altered column blocks the + statement the same way** — `cannot alter type of a column used by a generated column` + (`ERRCODE_FEATURE_NOT_SUPPORTED`), again regardless of whether the change would have been a + relabel. `RememberAllDependentForRebuilding` refuses rather than re-plans the generation + expression. The generated column has to be dropped before the change and re-added after + it — and re-adding a `STORED` generated column is itself a full rewrite. ## Checking before you run it @@ -454,5 +460,5 @@ across the 14–18 range. | Typmod no-op rules | `varchar_support()` (`varchar.c`), `numeric_support()` (`numeric.c`), `varbit_support()` (`varbit.c`), `timestamp_support()` → `TemporalSimplify()` (`timestamp.c`, `datetime.c`), `interval_support()` (`timestamp.c`) | | Display scale stored in the `numeric` datum | `NumericShort` / `NumericLong` headers and `NUMERIC_DSCALE()` — `src/backend/utils/adt/numeric.c` | | Index storage reuse after a no-rewrite change | `ATPostAlterTypeParse()` → `TryReuseIndex()` → `CheckIndexCompatible()` — `tablecmds.c`, `indexcmds.c` | -| View / rule dependency refusal | `RememberAllDependentForRebuilding()` — `tablecmds.c` | +| View / rule and generated-column dependency refusals | `RememberAllDependentForRebuilding()` — `tablecmds.c` | | Cast methods | `CoercionMethod` enum — `src/include/catalog/pg_cast.h` | diff --git a/docs/capabilities.md b/docs/capabilities.md index 238fd64..7179857 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -150,7 +150,7 @@ Status legend: ✅ T1 (supported today) · 🟡 T2 (planned; typed refusal today | `ADD COLUMN ... GENERATED ... STORED` | 🟡 | copy-and-swap | Yes | Table rewrite; copy-and-swap route. The copy engine must **recompute, never copy,** generated columns on the shadow table | | `ADD COLUMN` with inline `UNIQUE`/`PRIMARY KEY`/`REFERENCES`/`CHECK` | 🟡 | native, planned flow | Yes | The inline constraint does its index build or validation scan under the `ADD COLUMN`'s `ACCESS EXCLUSIVE` lock; refused with guidance to add the column first, then build the constraint online | | `DROP COLUMN` | ✅ | native, as-is | Yes | Metadata-only; flagged **destructive** in the plan report | -| `ALTER COLUMN TYPE`, binary-coercible (proven against live column facts) | ✅ | native, as-is | Yes | Catalog relabel, e.g. `varchar(50)` → `varchar(100)`, `varchar` → `text` | +| `ALTER COLUMN TYPE`, binary-coercible (proven against live column facts) | ✅ | native, as-is | Yes | Catalog relabel, e.g. `varchar(50)` → `varchar(100)`, `varchar` → `text`; PostgreSQL itself refuses the change when a view, rule, or `STORED` generated column depends on the column — see [binary-coercible-type-changes.md](binary-coercible-type-changes.md#no-rewrite-is-not-no-cost) | | `ALTER COLUMN TYPE`, general (or with `USING`) | 🟡 | copy-and-swap | Yes | Table rewrite; copy-and-swap route, refused today | | `SET DEFAULT` / `DROP DEFAULT` / `DROP NOT NULL` | ✅ | native, as-is | Yes | Metadata-only | | `SET NOT NULL` | ✅ | native, safer sequence | Yes | Executed as the native four-step pattern: `ADD CONSTRAINT ... CHECK (col IS NOT NULL) NOT VALID` → online `VALIDATE` → `SET NOT NULL` (catalog flip, PG 12+) → drop the scaffold check | diff --git a/docs/design-principles.md b/docs/design-principles.md index c9129ac..066749d 100644 --- a/docs/design-principles.md +++ b/docs/design-principles.md @@ -1,7 +1,7 @@ # Design principles The canonical list of principles that govern the engine. They are distilled from Spirit's -philosophy (see spirit-architecture-notes.md) and +philosophy (see the [Spirit README](https://github.com/block/spirit)) and adapted for PostgreSQL. Everything in [low-level-design.md](low-level-design.md) and the phased build plan should be traceable back to one of these. @@ -184,5 +184,5 @@ enforcement mechanics) live in [tcb-model](tcb-model.md); the repo-process versi checksum, lock behaviour). - **Each increment is independently useful.** The build is sequenced so that early phases (classify/print, then native-path execution) ship value on their own, and the highest-risk - components (CDC, cutover) are added last on a proven foundation (see - build-plan.md). + components (CDC, cutover) are added last on a proven foundation (see the + [current implementation status and next increment](low-level-design.md#next-step)). diff --git a/docs/high-level-design.md b/docs/high-level-design.md index 603a5f6..2b7bbfa 100644 --- a/docs/high-level-design.md +++ b/docs/high-level-design.md @@ -337,5 +337,7 @@ The choices that shape everything else (the full categorized list is in **[schemabot-integration.md](schemabot-integration.md)**. - The detailed interfaces, package layout, libraries, lifecycle internals, full coverage matrix, and later-phase decisions → **[low-level-design.md](low-level-design.md)**. -- How Spirit (the inspiration) works → spirit-architecture-notes.md. -- The phased plan to build it → build-plan.md. +- How Spirit (the inspiration) works → the [Spirit README](https://github.com/block/spirit) and + [PostgreSQL primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping). +- Current implementation status and the next increment → + **[low-level-design.md](low-level-design.md#next-step)**. diff --git a/docs/invariants.md b/docs/invariants.md index 021ba1f..b6df6b4 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -170,7 +170,7 @@ did or didn't commit. PostgreSQL's transactional DDL makes the swap itself atomi *client's knowledge* of the outcome is not. Retries of the cutover must be written against this ambiguity. *Enforced:* cutover retry loop. *Source:* Spirit's cutover (`information_schema` inspection on dropped connection, -spirit-architecture-notes). +[Spirit README](https://github.com/block/spirit#cut-over-and-cleanup)). ## State, checkpoint, and resume (ST) diff --git a/docs/low-level-design.md b/docs/low-level-design.md index 2e66fd8..1296456 100644 --- a/docs/low-level-design.md +++ b/docs/low-level-design.md @@ -13,8 +13,11 @@ MySQL-only — too many MySQL-isms to retrofit cleanly). The goal is a **decoupl executor** engine in which **copy-and-swap** is only one of several execution strategies. pg-sprite **derives design practices from several tools**: the copy-and-swap lifecycle and operator model from Spirit, the shadow-table approach from pg-osc/pg_repack, and the expand/contract executor -from pgroll. See spirit-architecture-notes.md -for how the Spirit original works and tool-pgroll.md for pgroll. +from pgroll. See [Spirit's README](https://github.com/block/spirit) for how the original works +(its per-primitive translation to PostgreSQL is in +[mysql-vs-postgresql.md](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping)) +and [high-level-design.md § execution patterns](high-level-design.md#the-execution-patterns-and-when-each-is-chosen) +for where pgroll's expand/contract fits. ## Table of contents @@ -176,7 +179,8 @@ pattern *per migration*: - **native** for the majority (the ➖/❌ rows in [postgres-online-ddl-reference](postgres-online-ddl-reference.md)); - **log-based copy-and-swap** for transparent, heavy physical rewrites (`int→bigint`, repack) - where the change is invisible to the app — see tool-pgroll's comparison; + where the change is invisible to the app — see the pattern comparison in + [high-level-design.md](high-level-design.md#the-execution-patterns-and-when-each-is-chosen); - **expand/contract via pgroll** for prod-critical breaking changes where **instant reversibility** and **two live schema versions** matter more than transparency. @@ -381,7 +385,8 @@ idle and a plain rewrite is acceptable); it is an escape hatch, not a shortcut, > strategy for genuine table rewrites. Until it lands, those rewrites receive a **not > native-safe** refusal. The `native` and `expand/contract` > executors are described in the architecture section and in -> tool-pgroll.md. The per-primitive **Spirit (MySQL) → +> [high-level-design.md § execution patterns](high-level-design.md#the-execution-patterns-and-when-each-is-chosen). +> The per-primitive **Spirit (MySQL) → > PostgreSQL mapping** this executor is built on lives in > [mysql-vs-postgresql.md § primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping). @@ -539,7 +544,7 @@ covered. The unsupported rows are explicit non-goals for v1. Spirit publishes a short, deliberate list of things it **requires** of a table and things it **refuses to do** (see [its README](https://github.com/block/spirit#unsupported-features) and -spirit-architecture-notes.md). These are not arbitrary — +[the PostgreSQL primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping)). These are not arbitrary — each maps to a property the copy/CDC/cutover machinery depends on. Below is the faithful translation of each constraint to PostgreSQL, **with the Postgres-specific reason** (not just "because Spirit does it"). The coverage matrix above states *what* is supported; @@ -760,7 +765,7 @@ Target the highest-value rewrite cases first: general `ALTER COLUMN TYPE`, volat routes natively-safe operations to direct DDL. PK-required, no-FK-on-migrated-table for v1. Both the declarative and imperative front ends exist and share classify → route, matching the -[README TL;DR](README.md#tldr-recommendation), +[README § the decided shape](README.md#the-decided-shape), [high-level-design](high-level-design.md#two-front-ends-declarative-and-imperative), and build-plan Phase 2. Declarative performs introspection, diff, and ordering; the imperative `--alter` path uses the same classifier with the diff step skipped. Phase 3 makes their classified diff --git a/docs/plan-report.md b/docs/plan-report.md index 0f8f79f..e1f1036 100644 --- a/docs/plan-report.md +++ b/docs/plan-report.md @@ -111,7 +111,7 @@ consumer rendering either into a shared surface must clamp and escape them. | `generated-stored` | Adding a stored generated column computes every row — a full rewrite. | | `type-rewrite` | A type conversion PostgreSQL cannot relabel — rewrite plus reindex. | | `relocation` | SET TABLESPACE moves the heap — a rewrite-scale copy. | -| `partition-parent-lock` | Partition attach/detach in its lock-taking form. | +| `partition-parent-lock` | Creating a partition (`CREATE TABLE … PARTITION OF`): a brief `ACCESS EXCLUSIVE` on the parent, no scan. | | `unsupported-operation` | The planner does not recognize the operation or knows no safe path for it. | ### Target-dependent refusal reasons (`reason`, `statements[].reason`) diff --git a/docs/postgres-online-ddl-reference.md b/docs/postgres-online-ddl-reference.md index 38e968d..afd5700 100644 --- a/docs/postgres-online-ddl-reference.md +++ b/docs/postgres-online-ddl-reference.md @@ -41,8 +41,21 @@ a copy-and-swap avoids the long exclusive lock. The plan report's `decisions[].reason` vocabulary ([plan-report.md](plan-report.md#planner-decision-reasons-decisionsreason)) is this table made machine-readable: `metadata-only` / `fast-default` / `binary-coercible` are the first -bucket, `safer-idiom` the second, `type-rewrite` / `volatile-default` / `generated-stored` / -`relocation` the third. +bucket; `safer-idiom` is the second; `online-idiom` spans the first and second buckets — +`NOT VALID`, `DROP INDEX CONCURRENTLY`, and `DETACH PARTITION CONCURRENTLY` read no rows, +while `CREATE INDEX CONCURRENTLY` and `VALIDATE CONSTRAINT` read every row; `ADD CONSTRAINT … +USING INDEX` is catalog-only when adopting an existing unique index as `UNIQUE`, but scans the +full heap under `ACCESS EXCLUSIVE` when adopting it as a `PRIMARY KEY` on a nullable column +because PostgreSQL validates `NOT NULL` — reach `NOT NULL` first as described in +[safer sequences](safer-sequences.md#the-substitutions-the-planner-makes-today). See the per-operation table. +`type-rewrite` / `volatile-default` / `generated-stored` / `relocation` are the third bucket. +The remaining reasons are first-bucket by cost but carry a warning the cost alone does not: +`partition-parent-lock` (a new partition takes a brief `ACCESS EXCLUSIVE` on the parent, so +the lock queue below applies to every query on the parent) and `app-breaking-rename` (the +catalog change is instant; the running application code that still uses the old name is +what breaks). `unsupported-operation` means the planner does not recognize the operation or +knows no safe path for it. The exclusion-constraint carve-out above is a known full-scan cost +with no online path, not an operation with no cost bucket. The MySQL-side detail — how `ALGORITHM=` and `LOCK=` are asserted and how the two engines' lock models line up — is in @@ -353,6 +366,12 @@ the catalog instead of rewriting the table. Catalog-only; runs as written. |---|---|---|---| | runs as written | brief `ACCESS EXCLUSIVE` | none — column relabeled in place | 0 | +PostgreSQL refuses the statement outright when a view or rule, or a `STORED` generated +column, depends on the altered column (`cannot alter type of a column used by a generated +column`). The planner cannot see these dependencies because `Facts` carries only column +types, so dry run still reports `binary-coercible`; see +[No rewrite is not no cost](binary-coercible-type-changes.md#no-rewrite-is-not-no-cost). + The type change is binary-coercible (for example `varchar(50)` → `varchar(100)`, or `varchar` → `text`), so PostgreSQL relabels the column in place without a table rewrite. Runs as written. How PostgreSQL decides, which changes only diff --git a/docs/postgresql-version-support.md b/docs/postgresql-version-support.md index 83ac79d..8356ec4 100644 --- a/docs/postgresql-version-support.md +++ b/docs/postgresql-version-support.md @@ -107,6 +107,6 @@ What this *excludes* and why it's fine: exactly PG 14; the planned native and copy-and-swap executors would be unaffected, and PG 15+ removes the limitation entirely. -See why-build-this-engine.md for why we reuse these tools as -executors rather than replace them, and build-plan.md for how the -version floor feeds the phased build. +See [vision.md](vision.md) for why we reuse these tools as executors rather than replace +them. The version floor feeds the later execution work summarized under the +[current implementation status and next increment](low-level-design.md#next-step).