diff --git a/docs/README.md b/docs/README.md index e17e6b8..1cd32f9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -24,7 +24,9 @@ Aurora-only. Why that combination is the product is [vision.md](vision.md); star | --- | --- | | [vision.md](vision.md) | The **vision statement** — what pg-sprite is (the reliable execution engine under a GitOps front-end like [SchemaBot](https://github.com/block/schemabot), as [Spirit](https://github.com/block/spirit) is for MySQL) and what it deliberately is not. Five pillars, success criteria, and explicit non-goals. Start here for the why. | | [architecture.md](architecture.md) | The **one-screen codebase map** — the three layers, the package map with build status, the copy-and-swap lifecycle, and where to read more. Start here for orientation. | -| [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md) | The PostgreSQL equivalent of MySQL's [InnoDB Online DDL Operations](https://dev.mysql.com/doc/refman/8.4/en/innodb-online-ddl-operations.html) reference — lock levels, rewrite/scan behaviour, and concurrent-DML safety per operation. | +| [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md) | The PostgreSQL equivalent of MySQL's [InnoDB Online DDL Operations](https://dev.mysql.com/doc/refman/8.4/en/innodb-online-ddl-operations.html) reference — the **three buckets** (catalog-only / full scan / full rewrite) MySQL's `ALGORITHM` states map to, then lock levels, rewrite/scan behaviour, and concurrent-DML safety per operation. | +| [binary-coercible-type-changes.md](binary-coercible-type-changes.md) | **Type changes without a rewrite** — how PostgreSQL decides whether `ALTER COLUMN TYPE` is a catalog relabel or a full rewrite (the structural test, never a data scan), the four shapes that skip the rewrite and why, the six categories of reason a rewrite is forced, the changes that look free but are not (shortening `varchar`, `numeric` scale, `char(n)` → `text`), what "no rewrite" still costs, how to check ahead of time, and the exact rules pg-sprite's `binary-coercible` verdict accepts. | +| [mysql-vs-postgresql.md](mysql-vs-postgresql.md) | The **MySQL ↔ PostgreSQL comparison reference** — how each engine expresses online DDL (`ALGORITHM=`/`LOCK=` vs per-operation idioms), the lock-mode → MDL mapping, **why DDL is dangerous** (the lock-queue pile-up, the same failure mode in both engines, and its mitigations), and the per-primitive [Spirit](https://github.com/block/spirit) (MySQL) → PostgreSQL translation the copy-and-swap executor is built on. | | [high-level-design.md](high-level-design.md) | The **high-level design** — the conceptual overview: the problem, the planner → router → executor philosophy, the execution patterns and when each is chosen, and coverage at a glance. No package/interface detail. Start here for the architecture. | | [low-level-design.md](low-level-design.md) | The **low-level design** — the detailed engineering design: package layout, the `Executor` interface, library choices, copy-and-swap lifecycle internals, the full coverage matrix, table requirements, and the decisions remaining for later execution phases. Read this when designing the interfaces and packages. | | [design-principles.md](design-principles.md) | The canonical **design principles** that govern the engine — safety over speed, decisions-not-options, classify-first, mandatory checksum gate, log-based CDC, and the PostgreSQL/Aurora-specific rules everything else traces back to. | diff --git a/docs/binary-coercible-type-changes.md b/docs/binary-coercible-type-changes.md new file mode 100644 index 0000000..ef3c455 --- /dev/null +++ b/docs/binary-coercible-type-changes.md @@ -0,0 +1,458 @@ +# Type changes without a rewrite: binary coercibility in PostgreSQL + +An `ALTER TABLE … ALTER COLUMN … TYPE` is either a catalog relabel that finishes in +milliseconds or a full table rewrite under `ACCESS EXCLUSIVE` — same syntax, opposite cost. +This document explains how PostgreSQL makes that decision, the four shapes of change that +skip the rewrite and *why* they are safe to skip, every category of reason a rewrite is +forced, the changes that look free but are not, and what "no rewrite" still costs you. + +```sql +-- varchar(20) → text: same on-disk bytes, catalog-only, milliseconds +ALTER TABLE orders ALTER COLUMN sku TYPE text; + +-- integer → bigint: 4 bytes → 8 bytes, every row re-encoded, every index +-- on the column rebuilt, all under ACCESS EXCLUSIVE +ALTER TABLE orders ALTER COLUMN amount TYPE bigint; +``` + +Everything here applies to PostgreSQL 14 and later (the versions pg-sprite supports). The +mechanics are cited to the PostgreSQL source in [Source pointers](#source-pointers) so a +claim can be re-checked rather than trusted. + +## Table of contents + +- [Terms used in this document](#terms-used-in-this-document) +- [The rule in one sentence](#the-rule-in-one-sentence) +- [How PostgreSQL decides](#how-postgresql-decides) + - [What a function cast is](#what-a-function-cast-is) +- [The four shapes that skip the rewrite](#the-four-shapes-that-skip-the-rewrite) + - [1. Binary-coercible casts (`pg_cast.castmethod = 'b'`)](#1-binary-coercible-casts-pg_castcastmethod--b) + - [2. Typmod relaxation (same type, looser modifier)](#2-typmod-relaxation-same-type-looser-modifier) + - [3. Unconstrained domains over the same base type](#3-unconstrained-domains-over-the-same-base-type) + - [4. `timestamp` ↔ `timestamptz` under a UTC session](#4-timestamp--timestamptz-under-a-utc-session) +- [Why rewrites happen: the six categories](#why-rewrites-happen-the-six-categories) + - [Things that are *not* rewrite reasons](#things-that-are-not-rewrite-reasons) +- [Looks free, but rewrites](#looks-free-but-rewrites) + - [Shortening `varchar(n)`](#shortening-varcharn) + - [`numeric(10,2)` → `numeric(10,3)`: the value survives, the datum does not](#numeric102--numeric103-the-value-survives-the-datum-does-not) + - [`numeric(12,2)` → `numeric(10,2)`: the rows fit, PostgreSQL cannot know](#numeric122--numeric102-the-rows-fit-postgresql-cannot-know) +- [No rewrite is not no cost](#no-rewrite-is-not-no-cost) +- [Checking before you run it](#checking-before-you-run-it) + - [Failing closed with a `table_rewrite` event trigger](#failing-closed-with-a-table_rewrite-event-trigger) +- [How pg-sprite classifies type changes](#how-pg-sprite-classifies-type-changes) +- [Source pointers](#source-pointers) + +## Terms used in this document + +| Term | Meaning | +| --- | --- | +| **Type** | The declared data type of a column: `integer`, `text`, `numeric`, … Determines the on-disk encoding of each value. | +| **Type modifier (typmod)** | The parenthesised part of a type: the `50` in `varchar(50)`, the `(10,2)` in `numeric(10,2)`, the `3` in `timestamp(3)`. It constrains values; it never changes the storage format. `-1` means "unconstrained". | +| **Datum** | The bytes of one stored value. A **rewrite** exists to produce new datums for every row. | +| **Cast** | A conversion from one type to another, described by a row in `pg_cast`. | +| **`castmethod`** | How the cast is performed: `b` = *binary-compatible* (no function, the bytes are reinterpreted), `f` = *function* (a C or SQL function transforms each value), `i` = *I/O conversion* (the value is rendered to text and re-parsed by the target type). | +| **Relabel** | The planner node (`RelabelType`) that changes a value's type label without touching its bytes. The output of a binary-compatible cast. | +| **Transform expression** | The expression PostgreSQL builds to turn the old column value into the new one. Its *shape* after simplification is what decides whether a rewrite happens. | +| **Planner support function** | A per-type hook (`varchar_support`, `numeric_support`, …) the planner consults to simplify calls of that type's functions — including proving a length check is a no-op. | +| **Domain** | A named type built on a base type, optionally with `CHECK` / `NOT NULL` constraints: `CREATE DOMAIN email AS text CHECK (VALUE ~ '@')`. Stored exactly like its base type. | +| **Rewrite** | PostgreSQL copies every row into a new physical file (a new *relfilenode*), rebuilding all indexes, while holding `ACCESS EXCLUSIVE` on the table. | + +## The rule in one sentence + +PostgreSQL rewrites the table unless it can prove, **from the shape of the transform +expression alone**, that every byte already on disk is a valid datum of the new type. It +never scans the data to find out. A change that would be harmless for the rows you happen to +have still rewrites if the *type system* cannot prove it harmless for every possible row. + +## How PostgreSQL decides + +`ATPrepAlterColumnType` (`src/backend/commands/tablecmds.c`) builds the transform expression +— the `USING` clause if you gave one, otherwise a bare reference to the column — and coerces +it to the target type with assignment-cast rules. It then runs the expression through the +planner's simplifier (`expression_planner`), which is where the type-specific planner support +functions get a chance to prove a length coercion is a no-op and replace it with a relabel. + +`ATColumnChangeRequiresRewrite` then walks the simplified expression: + +``` +loop over the expression: + a plain reference to the column → NO REWRITE (stop) + RelabelType → strip it, continue + CoerceToDomain with no domain constraints → strip it, continue + with CHECK / NOT NULL → REWRITE + FuncExpr timestamp ↔ timestamptz, and the session time zone is a fixed +00:00 + → strip it, continue + any other function → REWRITE + anything else (ArrayCoerceExpr, CoerceViaIO, …) → REWRITE +``` + +Two consequences follow. First, the decision is **structural**: if a real function call +survives simplification, the table rewrites, however cheap that function would be per row. +Second, the decision depends on the column's **current** type, which is not in the DDL text +— `ALTER COLUMN sku TYPE text` is free if `sku` is `varchar(20)` and a rewrite if it is +`integer`. Anything that classifies these statements ahead of time has to introspect the +catalog first. + +### What a function cast is + +The value `42` stored as `integer` is four bytes; stored as `bigint` it is eight: + +``` +integer 42 → 2A 00 00 00 +bigint 42 → 2A 00 00 00 00 00 00 00 +``` + +There is no way to read the four-byte datum as an eight-byte one — every row must pass +through the C function `int8(integer)`, which reads four bytes and writes eight, and the +row layout of every tuple grows. That function call is the `FuncExpr` the rewrite test +refuses to strip. Contrast `varchar → text`: both are stored as `[length header][bytes]`, +so the planner only changes the type label on the same datum — a `RelabelType`. + +The catalog says which is which: + +```sql +SELECT castsource::regtype, casttarget::regtype, castfunc::regproc, castmethod +FROM pg_cast +WHERE (castsource, casttarget) IN (('integer'::regtype, 'bigint'::regtype), + ('varchar'::regtype, 'text'::regtype)); + +-- castsource | casttarget | castfunc | castmethod +-- -------------------+------------+----------+----------- +-- integer | bigint | int8 | f ← function: rewrite +-- character varying | text | - | b ← binary: relabel +``` + +Casts to and from string types that have no `pg_cast` row at all (`integer → text`, +`text → uuid`) are performed by *I/O conversion*: render with the source type's output +function, parse with the target's input function. The planner represents these as +`CoerceViaIO`, which the rewrite test also refuses to strip. + +## The four shapes that skip the rewrite + +### 1. Binary-coercible casts (`pg_cast.castmethod = 'b'`) + +Two distinct types whose values share one on-disk representation. The catalog keeps them +separate because they differ in semantics, I/O functions, or operators — but the stored +bytes are identical, so the planner emits a `RelabelType` and the loop strips it. + +| Cast | Why the bytes are identical | +| --- | --- | +| `varchar` → `text`, `text` → `varchar` (unbounded) | Both are variable-length strings; the `varchar` length limit is a check, not a storage format | +| `xml` → `text` | `xml` is stored as text; only the input validation differs | +| `cidr` → `inet` | Same struct; `cidr` is `inet` with a stricter invariant (no host bits) | +| `oid` ↔ `regclass`, `regtype`, `regproc`, … | All 4-byte object identifiers with different display functions | + +Ask the catalog rather than memorising the list: + +```sql +SELECT castsource::regtype, casttarget::regtype +FROM pg_cast +WHERE castmethod = 'b'; +``` + +Binary coercibility is **directional**. `cidr → inet` is a relabel; `inet → cidr` masks +the host bits and is a function cast. `xml → text` is a relabel; `text → xml` validates +the document and is a function cast. Check the row for the direction you are changing. + +### 2. Typmod relaxation (same type, looser modifier) + +`varchar(50) → varchar(100)` is not a cast at all — the type is `varchar` on both sides +and only the modifier changes. The coercion still starts life as a function call +(`varchar(value, 100)`, the length check), but during simplification the type's planner +support function proves the check can never reject or alter a value the old modifier +admitted, and replaces the call with a `RelabelType`. + +Each type with a support function encodes its own "cannot lose information" rule: + +| Type | Support function | Relabel (no rewrite) when | +| --- | --- | --- | +| `varchar(n)` | `varchar_support` | new limit is unbounded, **or** old was bounded and new ≥ old | +| `numeric(p,s)` | `numeric_support` | new is unconstrained, **or** both constrained with the **same scale** and new precision ≥ old | +| `varbit(n)` | `varbit_support` | new limit is unbounded, **or** old was bounded and new ≥ old | +| `timestamp(p)`, `timestamptz(p)`, `time(p)`, `timetz(p)` | `timestamp_support` etc. | new precision is unspecified or the type's maximum, **or** old was specified and new ≥ old | +| `interval` fields/precision | `interval_support` | new range keeps every field the old one allowed and (if seconds are in range) fractional precision does not shrink | + +Strictly, this is not binary coercion in PostgreSQL's own vocabulary — that term is for the +`castmethod = 'b'` rows above — it is a **no-op typmod coercion**. The operational outcome +is the same (catalog-only, no scan, no rewrite), which is why pg-sprite reports both under +one verdict reason. + +### 3. Unconstrained domains over the same base type + +A domain is a named type layered on a base type: + +```sql +CREATE DOMAIN sku_code AS text; -- unconstrained +CREATE DOMAIN email AS text CHECK (VALUE ~ '@'); -- constrained +``` + +A domain has **no storage of its own**. A `sku_code` value is stored byte-for-byte as a +`text` value; the domain only adds checks that run when a value is *assigned*. So changing +a `text` column to `sku_code` produces a `CoerceToDomain` node with nothing to enforce, and +the rewrite test strips it — the existing bytes are already valid `sku_code` datums. + +```sql +ALTER TABLE products ALTER COLUMN code TYPE sku_code; -- text → unconstrained domain: relabel +ALTER TABLE users ALTER COLUMN mail TYPE email; -- text → constrained domain: REWRITE +``` + +The second statement rewrites because every existing row would have to be checked against +`VALUE ~ '@'`, and PostgreSQL has no "scan without rewrite" path for `ALTER COLUMN TYPE` +— its only tool for "check every row" is to re-create every row. `NOT NULL` on the domain +counts as a constraint too; a `DEFAULT` does not (defaults never affect existing rows). + +Three related facts: + +- **Domain → its base type** (or dropping a domain in favour of `text`) is always a + relabel. There is nothing to check when you *remove* constraints. +- **The base type may itself be relabelled first.** `varchar(20) → sku_code` is + `varchar → text` (binary-coercible) followed by `CoerceToDomain` (unconstrained): both + strip, no rewrite. +- **Stacked domains** behave the same way: each `CoerceToDomain` layer is stripped if + that domain has no constraints, and forces a rewrite if it has any. + +### 4. `timestamp` ↔ `timestamptz` under a UTC session + +Converting between `timestamp` and `timestamptz` normally depends on the session +`TimeZone` and is a function cast. PostgreSQL 12+ special-cases exactly these two functions +in the rewrite test: if the session time zone is a fixed zero offset (`UTC`, `Etc/UTC`, +`+00`), the conversion is the identity on the stored 8-byte value, so it is treated as a +relabel. + +```sql +SET timezone = 'UTC'; -- the same ALTER under 'Australia/Sydney' rewrites the table +ALTER TABLE events ALTER COLUMN created_at TYPE timestamptz; +``` + +This is the one case where the rewrite decision depends on **session state**, not on the +catalog. A tool that classifies the change ahead of time has to know which time zone the +executing session will run under. + +## Why rewrites happen: the six categories + +A rewrite happens whenever a node the test cannot strip survives simplification. Grouping +by *why* the node survives gives six categories; every rewriting type change falls into at +least one. The letter tags are used in the [Looks free, but rewrites](#looks-free-but-rewrites) +table. + +| # | Category | Why the bytes cannot be reused | Example | +| --- | --- | --- | --- | +| **A** | **Different on-disk representation** | The two types encode values differently (width, layout, or structure), so a function must produce every new datum. | `integer → bigint` (4 → 8 bytes); `real → double precision`; `integer → numeric`; `json → jsonb` (text → binary tree); `uuid → text` | +| **B** | **Value-transforming conversion** | The types are "similar", but the cast function changes the bytes of at least some values — padding, masking, normalisation. | `char(10) → text` (`rtrim1()` strips the padding); `inet → cidr` (masks host bits); `char(10) → char(20)` (the padding is stored, so every datum grows) | +| **C** | **A modifier stored inside the datum changes** | The typmod is not just a check — part of it is written into each value's header, so a different typmod means a different datum even when the numeric value is equal. | `numeric(10,2) → numeric(10,3)` (display scale lives in the datum — see [below](#numeric102--numeric103-the-value-survives-the-datum-does-not)); `timestamp(6) → timestamp(3)` (rounds fractional seconds); `interval` field/precision reduction | +| **D** | **A tightening the type system cannot prove safe** | The new type or modifier could reject a value the old one accepted. PostgreSQL does not scan to see whether any row *would* be rejected; it rewrites, and fails if one is. | `varchar(100) → varchar(50)`; `text → varchar(255)`; `numeric(12,2) → numeric(10,2)`; `text → xml` (validation); `text → email` (domain with `CHECK` or `NOT NULL`); `varbit(16) → varbit(8)` | +| **E** | **Session-dependent conversion** | The result depends on runtime state, so the bytes are only reusable under one specific setting. | `timestamp → timestamptz` under any session time zone other than a fixed +00:00 | +| **F** | **Wrapped in a node the rewrite test does not recognise** | The conversion may be harmless, but it is expressed as a node type the test has no case for, so it falls through to "rewrite". | `varchar(50)[] → varchar(100)[]` (`ArrayCoerceExpr` — the scalar rule is not lifted to arrays); `integer → text` (`CoerceViaIO`); any `USING` clause containing a real function call, e.g. `USING lower(col)` or `USING col || ''` | + +Categories A–C are *genuine*: the datums really must change. Category D is *conservative*: +the datums might all be fine, but PostgreSQL will not look. Categories E and F are +*mechanical*: the bytes are reusable in principle, but the decision procedure cannot see it. +Knowing which category a change falls into tells you whether there is an online alternative +(D often has one — see the `varchar` shortening idiom below; A never does). + +### Things that are *not* rewrite reasons + +Common assumptions that turn out to be wrong in the safe direction: + +- **Indexes on the column.** They are dropped and re-created by the statement, and rebuilt + if incompatible, but they never cause a *heap* rewrite. See + [No rewrite is not no cost](#no-rewrite-is-not-no-cost). +- **Constraints referencing the column** (`CHECK`, `UNIQUE`, foreign keys). Same: re-created + in the statement, not a heap rewrite. +- **A stored generated column that references the altered column.** Its expression is + re-recorded as a dependency; the rewrite decision is made solely on the altered column's + own transform expression. +- **A collation change alone** (`ALTER COLUMN name TYPE text COLLATE "C"` on a `text` + column). The transform is a bare relabel — no heap rewrite — but every index on the column + is rebuilt because its sort order may differ. + +## Looks free, but rewrites + +Each of these is a reasonable guess that fails the structural test. The tag refers to the +[category table](#why-rewrites-happen-the-six-categories). + +| Change | Why it looks free | What actually happens | Why | Tag | +| --- | --- | --- | --- | --- | +| `varchar(100)` → `varchar(50)` | "every value already fits" | full rewrite + reindex under `ACCESS EXCLUSIVE`; errors if any row is too long | `varchar_support` only simplifies when new ≥ old; the length check survives and PostgreSQL does not scan to see whether it would pass — see [below](#shortening-varcharn) | D | +| `text` → `varchar(255)`, `varchar` → `varchar(255)` | "just adding a limit" | full rewrite | the old typmod is unbounded, so the "old was bounded" guard fails and the check survives | D | +| `numeric(10,2)` → `numeric(10,3)` | "wider" | full rewrite | the display scale is stored in every datum; a scale change alters every stored value — see [below](#numeric102--numeric103-the-value-survives-the-datum-does-not) | C | +| `numeric(12,2)` → `numeric(10,2)` | "same scale, my values fit" | full rewrite; errors if any row overflows | precision shrinks; PostgreSQL will not look at the rows — see [below](#numeric122--numeric102-the-rows-fit-postgresql-cannot-know) | D | +| `integer` → `bigint`, `smallint` → `integer`, `real` → `double precision`, `integer` → `numeric` | "widening within the family" | full rewrite + every index rebuilt | different width or encoding; the cast is a real function (`int8(integer)`) | A | +| `char(n)` → `text` / `varchar` | "all text types" | full rewrite | `bpchar → text` is `rtrim1()` — it strips the padding — a function cast, not a relabel. `char(n) → char(m)` rewrites too: the padding is stored | B | +| `varchar(50)[]` → `varchar(100)[]` | "element widening is free" | full rewrite | the coercion is an `ArrayCoerceExpr`; the rewrite test has no case for it, and support-function simplification applies to scalars only | F | +| `text` → domain with `CHECK` or `NOT NULL` | "only adds validation" | full **rewrite**, not a validation scan | `DomainHasConstraints` → rewrite; there is no scan-only path for `ALTER COLUMN TYPE` | D | +| `timestamp` → `timestamptz` under a non-UTC session | "PG 12 made this free" | full rewrite | the exemption applies only when the session zone is a fixed +00:00 | E | +| `inet` → `cidr`, `text` → `xml` | "the reverse direction is a relabel" | full rewrite | binary coercibility is directional; these directions are function casts | B, D | +| `json` → `jsonb`, `uuid` ↔ `text`, `integer` → `text` | "lossless" | full rewrite | different storage formats; the casts go through functions or I/O conversion | A, F | +| same type, only `COLLATE` changes | "nothing about the data changes" | **no heap rewrite**, but every index on the column is rebuilt | the transform is a bare relabel, but the index's collation changes and its storage cannot be reused — the sort order may differ | — | + +### Shortening `varchar(n)` + +The most common surprise. There is little practical reason to shorten a `varchar` limit, +but for completeness: `varchar(100) → varchar(50)` is a **full table rewrite**, not a +check. PostgreSQL re-encodes every row through the length-coercion function, rebuilds every +index on the column, and holds `ACCESS EXCLUSIVE` for the whole duration; if any row is +longer than 50 characters the statement fails after doing that work (`value too long for +type character varying(50)`). Widening is free because the type system can prove nothing is +lost; shortening cannot be proven from the type alone, and PostgreSQL's answer to "cannot +prove" is always "rewrite", never "scan and see". + +If the goal is to *enforce* a shorter limit rather than to change the declared type, the +online idiom is a constraint: + +```sql +ALTER TABLE t ADD CONSTRAINT t_col_len CHECK (length(col) <= 50) NOT VALID; -- brief lock +ALTER TABLE t VALIDATE CONSTRAINT t_col_len; -- SHARE UPDATE EXCLUSIVE, concurrent DML OK +``` + +Same guarantee for new writes and, after `VALIDATE`, for existing rows — with the declared +type left as `varchar(100)`. If the declared type itself must change, that is a genuine +rewrite and belongs to the copy-and-swap path. + +### `numeric(10,2)` → `numeric(10,3)`: the value survives, the datum does not + +Increasing the scale looks like pure widening — every `numeric(10,2)` value is +representable as `numeric(10,3)`. But `numeric` stores the **display scale** in each +datum's header, so equal values at different scales are different bytes: + +| Row | Stored as `numeric(10,2)` | Stored as `numeric(10,3)` | Same value? | Same datum? | +| --- | --- | --- | --- | --- | +| 1 | `123.45` (dscale 2) | `123.450` (dscale 3) | yes | **no** | +| 2 | `7.10` (dscale 2) | `7.100` (dscale 3) | yes | **no** | +| 3 | `0.05` (dscale 2) | `0.050` (dscale 3) | yes | **no** | + +After the change, `SELECT` returns `123.450`, not `123.45` — the stored value itself has +changed, and every row had to be re-encoded to make it so. `numeric_support` therefore +requires the scale to be *equal* before it will simplify; only precision may grow. + +The reverse, `numeric(10,3) → numeric(10,2)`, is more obviously a rewrite: `123.456` +rounds to `123.46` — information is lost, and the values may fail the new precision. + +### `numeric(12,2)` → `numeric(10,2)`: the rows fit, PostgreSQL cannot know + +Shrinking precision at the same scale is the `numeric` twin of shortening `varchar`. +`numeric(10,2)` allows eight integer digits (10 − 2); `numeric(12,2)` allows ten. + +| Row | Stored as `numeric(12,2)` | Under `numeric(10,2)` | Outcome | +| --- | --- | --- | --- | +| 1 | `12345678.90` (8 integer digits) | `12345678.90` | fits — bytes unchanged | +| 2 | `999.99` | `999.99` | fits — bytes unchanged | +| 3 | `1234567890.12` (10 integer digits) | — | `ERROR: numeric field overflow` — `A field with precision 10, scale 2 must round to an absolute value less than 10^8.` | + +If row 3 does not exist, every datum is already a valid `numeric(10,2)` — and PostgreSQL +still rewrites the table, because `numeric_support` cannot prove from the typmods alone +that no such row exists, and the rewrite test never looks at data. If row 3 *does* exist, +the statement fails on reaching it, after rewriting every row before it (the whole +statement rolls back; the time and lock are spent regardless). + +## No rewrite is not no cost + +- **`ACCESS EXCLUSIVE` is still taken** for the duration of the statement. Milliseconds of + work, but the lock request queues behind every in-flight query on the table and every + later query queues behind it — the lock-queue pile-up in + [mysql-vs-postgresql.md](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue). + Always run under `lock_timeout`. +- **Indexes are dropped and recreated** in the same statement, even for a relabel. Their + physical storage is *reused* only when `CheckIndexCompatible` passes: same operator + class, same collation, and same opclass options on every key column; no expression + columns; no predicate; index currently valid. `varchar → text` passes (a `varchar` + index already uses `text_ops`). An expression index or a partial index on the column is + rebuilt regardless. +- **Constraints referencing the column** are likewise dropped and re-added in the same + statement. +- **A view or rule referencing the column blocks the statement outright** — + `cannot alter type of a column used by a view or rule` — whether or not the change would + have been a relabel. The view has to be dropped and recreated around the change. + +## Checking before you run it + +Ask the catalog for the current type and modifier (the part the DDL text does not tell +you): + +```sql +SELECT format_type(atttypid, atttypmod), attcollation::regcollation +FROM pg_attribute +WHERE attrelid = 't'::regclass AND attname = 'col'; +``` + +Then test empirically inside a transaction and roll back. A table rewrite allocates a new +relfilenode; a relabel does not: + +```sql +BEGIN; +SELECT pg_relation_filenode('t'::regclass); +ALTER TABLE t ALTER COLUMN col TYPE varchar(100); +SELECT pg_relation_filenode('t'::regclass); -- unchanged ⇒ no rewrite +ROLLBACK; +``` + +Run this against a copy or a scratch database when the table is large: the `ALTER` inside +the transaction performs the full rewrite before you roll it back. + +### Failing closed with a `table_rewrite` event trigger + +PostgreSQL can refuse, database-wide, any `ALTER TABLE` (or `ALTER TYPE`) that would +rewrite a table. The `table_rewrite` event fires *before* the rewrite starts: + +```sql +CREATE FUNCTION refuse_rewrite() RETURNS event_trigger LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'ALTER TABLE on % would rewrite the table', + pg_event_trigger_table_rewrite_oid()::regclass; +END $$; + +CREATE EVENT TRIGGER no_table_rewrite ON table_rewrite EXECUTE FUNCTION refuse_rewrite(); +``` + +This is **PostgreSQL-specific**; MySQL has no equivalent hook. The two engines protect +against an accidental table copy in opposite ways: + +| | PostgreSQL | MySQL 8.0 | +| --- | --- | --- | +| Mechanism | A database-wide **event trigger** on `table_rewrite` | A **per-statement assertion**: `ALTER TABLE … ALGORITHM=INSTANT` (or `ALGORITHM=INPLACE, LOCK=NONE`) | +| Who opts in | The DBA, once, for every statement anyone runs | The author of each statement | +| On violation | The trigger raises; the statement fails before any row is copied | The server refuses: `ALGORITHM=INSTANT is not supported` (or the equivalent for `INPLACE`) | +| Scope | `ALTER TABLE` and `ALTER TYPE` rewrites only — not `CLUSTER` or `VACUUM FULL` | The statement it is attached to | + +Creating an event trigger requires superuser (managed services expose an equivalent, such +as `rds_superuser` on Amazon RDS and Aurora). The trigger function can be selective — +`pg_event_trigger_table_rewrite_reason()` returns why the rewrite is happening, and the +function can consult the table's size before deciding to raise. + +## How pg-sprite classifies type changes + +The planner's `binary-coercible` verdict reason +([plan-report.md](plan-report.md#planner-decision-reasons-decisionsreason)) is deliberately narrower than what +PostgreSQL can prove. It accepts exactly: + +- the same type and modifier (a no-op relabel); +- `varchar(n) → varchar(m)` with `m ≥ n`, `varchar(n) → varchar`, and `varchar → text`; +- `numeric(p,s) → numeric(p',s)` with `p' ≥ p`, and `numeric(p,s) → numeric`. + +Everything else — including changes PostgreSQL would relabel, such as `varbit` or temporal +precision widening, unconstrained domains, `xml → text`, and the UTC `timestamp ↔ +timestamptz` case — is treated as `type-rewrite` and routed to the copy-and-swap path (a +typed refusal until that executor lands). The bias is intentional: a false "free" verdict +would run a blocking rewrite on a hot table; a false "rewrite" verdict costs a slower but +safe path. The rule set grows only when a new row has been proven against the reference +table in [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md). + +Because the decision needs the column's current type, the classifier takes the introspected +catalog shape as input. Without it, no type change can be proven free, and the planner +routes the statement to the rewrite path rather than gamble. + +## Source pointers + +All in the PostgreSQL repository, `master` at time of writing; the behaviour is unchanged +across the 14–18 range. + +| What | Where | +| --- | --- | +| The rewrite decision loop | `ATColumnChangeRequiresRewrite()` — `src/backend/commands/tablecmds.c` | +| Building and simplifying the transform expression | `ATPrepAlterColumnType()` — `src/backend/commands/tablecmds.c` (calls `coerce_to_target_type()` then `expression_planner()`) | +| `timestamp ↔ timestamptz` UTC exemption | `TimestampTimestampTzRequiresRewrite()` — `src/backend/utils/adt/timestamp.c` | +| Typmod no-op rules | `varchar_support()` (`varchar.c`), `numeric_support()` (`numeric.c`), `varbit_support()` (`varbit.c`), `timestamp_support()` → `TemporalSimplify()` (`timestamp.c`, `datetime.c`), `interval_support()` (`timestamp.c`) | +| Display scale stored in the `numeric` datum | `NumericShort` / `NumericLong` headers and `NUMERIC_DSCALE()` — `src/backend/utils/adt/numeric.c` | +| Index storage reuse after a no-rewrite change | `ATPostAlterTypeParse()` → `TryReuseIndex()` → `CheckIndexCompatible()` — `tablecmds.c`, `indexcmds.c` | +| View / rule dependency refusal | `RememberAllDependentForRebuilding()` — `tablecmds.c` | +| Cast methods | `CoercionMethod` enum — `src/include/catalog/pg_cast.h` | diff --git a/docs/design-principles.md b/docs/design-principles.md index eacbebf..c9129ac 100644 --- a/docs/design-principles.md +++ b/docs/design-principles.md @@ -47,7 +47,7 @@ the phased build plan should be traceable back to one of these. - **Bound every exclusive lock.** Every `ACCESS EXCLUSIVE` (only the cutover swap in the happy path) and every catalog-flip runs under `lock_timeout` + bounded retry/backoff, so the engine never sits at the head of the lock queue and amplifies one slow transaction into an - outage (see 12-mysql-vs-postgresql.md § Why DDL is dangerous: the lock queue). + outage (see [mysql-vs-postgresql.md § Why DDL is dangerous: the lock queue](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue)). - **Refuse the unsafe rather than guess.** Lossy conversions, PK changes, FK/trigger tables, and ambiguous renames are rejected up front with a clear reason — never silently attempted (see [low-level-design's requirements](low-level-design.md#table-requirements-and-unsupported-operations-postgresql-analogs)). diff --git a/docs/high-level-design.md b/docs/high-level-design.md index 58a2315..603a5f6 100644 --- a/docs/high-level-design.md +++ b/docs/high-level-design.md @@ -285,8 +285,8 @@ Two things define this path and distinguish it from existing PostgreSQL tools: continuous re-verification loop while it waits. The mechanism (logical decoding, chunking, the transactional swap, checkpoint/resume) and the -MySQL→PostgreSQL primitive mapping are in the low-level design and -mysql-vs-postgresql.md. +MySQL→PostgreSQL primitive mapping are in the [low-level design](low-level-design.md#copy-and-swap-executor-lifecycle) and +[mysql-vs-postgresql.md](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping). ## What it covers (and what it deliberately does not) diff --git a/docs/invariants.md b/docs/invariants.md index 590538d..7880c56 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -99,7 +99,7 @@ order-preserving apply within the batch, per-row retry on unique violation, or d pairs — and prove convergence under test. The checksum (CO-1) backstops, but the applier must converge without it. *Enforced:* applier batch semantics (design work, Phase 6). *Source:* Spirit `pkg/change/README.md` (the REPLACE rationale) — the PG translation in -mysql-vs-postgresql +[mysql-vs-postgresql](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping) is incomplete without this. ### CO-7 — Every statement parses, or it is an error @@ -138,7 +138,7 @@ mutual-exclusion gap called out in the validation review. The cutover swap is the only `ACCESS EXCLUSIVE` acquisition in the happy path, and **every** strong-lock acquisition (swap, catalog flips, trigger install in fallback mode) runs under `lock_timeout` + bounded retry/backoff so the engine never sits at the head of the lock queue -(mysql-vs-postgresql § the lock queue). +([mysql-vs-postgresql § the lock queue](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue)). **Exception policy required:** `CREATE INDEX CONCURRENTLY` and `REINDEX CONCURRENTLY` wait on other transactions via lock waits that a naive `lock_timeout` cancels — leaving an `INVALID` index — so they get their own wait policy (no per-lock timeout, one overall statement deadline) @@ -148,7 +148,7 @@ executor's validate class deliberately keeps a bounded per-lock timeout — queu conflicting lock holder must not stall a sequence for the whole scan budget — while the scan itself runs under its own generous overall budget. *Enforced:* every DDL execution path in the native and copy-and-swap executors. -*Source:* [design-principles](design-principles.md#correctness-and-safety), mysql-vs-postgresql; +*Source:* [design-principles](design-principles.md#correctness-and-safety), [mysql-vs-postgresql](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue); CIC exception from the validation review. ### LK-3 — Pending work is claimed exactly once, and Wait means finished diff --git a/docs/low-level-design.md b/docs/low-level-design.md index 5454d1b..2e66fd8 100644 --- a/docs/low-level-design.md +++ b/docs/low-level-design.md @@ -383,7 +383,7 @@ idle and a plain rewrite is acceptable); it is an escape hatch, not a shortcut, > executors are described in the architecture section and in > tool-pgroll.md. The per-primitive **Spirit (MySQL) → > PostgreSQL mapping** this executor is built on lives in -> 12-mysql-vs-postgresql.md § primitive mapping. +> [mysql-vs-postgresql.md § primitive mapping](mysql-vs-postgresql.md#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping). ``` +----------------------------------------------------------------------+ diff --git a/docs/mysql-vs-postgresql.md b/docs/mysql-vs-postgresql.md new file mode 100644 index 0000000..7851046 --- /dev/null +++ b/docs/mysql-vs-postgresql.md @@ -0,0 +1,181 @@ +# MySQL vs PostgreSQL: the comparison reference + +One place for every MySQL ↔ PostgreSQL comparison pg-sprite relies on: how each engine +expresses online DDL, how their lock models map, **why DDL is dangerous (the lock-queue +pile-up — the same failure mode in both engines)**, and how +[Spirit](https://github.com/block/spirit)'s MySQL primitives translate to the PostgreSQL +copy-and-swap executor. The other docs link here instead of repeating it. If you are new to +why an apparently instant `ALTER TABLE` can take an app down, start with +[Why DDL is dangerous: the lock queue](#why-ddl-is-dangerous-the-lock-queue). + +## Table of contents + +- [How online DDL is expressed](#how-online-ddl-is-expressed) +- [Lock model comparison](#lock-model-comparison) +- [Why DDL is dangerous: the lock queue](#why-ddl-is-dangerous-the-lock-queue) + - [The mechanism: a three-step pile-up](#the-mechanism-a-three-step-pile-up) + - [The DDL is the catalyst, not the long query](#the-ddl-is-the-catalyst-not-the-long-query) + - [How long does the impact last?](#how-long-does-the-impact-last) + - [Mitigations the engine and operators rely on](#mitigations-the-engine-and-operators-rely-on) + - [The lock queue is the same in both engines](#the-lock-queue-is-the-same-in-both-engines) +- [Copy-and-swap executor: Spirit (MySQL) → PostgreSQL primitive mapping](#copy-and-swap-executor-spirit-mysql--postgresql-primitive-mapping) + +## How online DDL is expressed + +The two engines reach "online schema change" by **different routes**, which is why a tool +designed for one does not port mechanically to the other: + +| Aspect | MySQL 8.0 / InnoDB | PostgreSQL | +| --- | --- | --- | +| How you ask for online behaviour | Explicit `ALTER … ALGORITHM={INSTANT\|INPLACE\|COPY}, LOCK={NONE\|SHARED\|EXCLUSIVE}` | **No such knob.** Each DDL has a *fixed* lock level and rewrite behaviour you cannot lower | +| What "online" means | The server runs the change in place and lets you request `LOCK=NONE` | Some operations are inherently online (metadata-only, `CONCURRENTLY`, `NOT VALID`); others always take `ACCESS EXCLUSIVE` and/or rewrite | +| How much work the change does | Three server-asserted states: `INSTANT` (metadata only), `INPLACE` (rebuilt in place, often a full scan), `COPY` (full table copy) | The same three buckets exist — catalog-only, full scan without rewrite, full rewrite — but nothing asserts them; you infer them per operation. See [the three buckets](postgres-online-ddl-reference.md#the-three-buckets-what-mysqls-algorithm-states-map-to) | +| Avoiding a rewrite | `ALGORITHM=INSTANT`/`INPLACE` | Use the **native-safe pattern** (`CONCURRENTLY`, `NOT VALID`+`VALIDATE`, fast default, `ADD PK USING INDEX`, [binary-coercible type change](binary-coercible-type-changes.md)) | +| When neither is possible | `ALGORITHM=COPY` (server-side table rebuild) or an OSC tool (gh-ost / Spirit) | An **OSC tool** (pg_osc, pg_repack, or pg-sprite's copy-and-swap executor) | +| Where the engine decides | Spirit attempts INSTANT/INPLACE, else copies | pg-sprite **classifies** the change → native-safe, copy-and-swap, or refuse (see [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md)) | + +The practical upshot: in MySQL the *server* exposes the online machinery and you opt into it; in +PostgreSQL the online machinery is a **set of per-operation idioms**, and the value of an engine +is knowing which idiom applies (or that a copy is unavoidable). + +## Lock model comparison + +The two engines model locking differently, so the mapping is by **what concurrent access is +allowed**, not a 1:1 equivalence: + +- **PostgreSQL** has a single explicit **table-lock hierarchy** (8 modes). Each DDL statement + acquires a *fixed* mode you cannot lower — you can only bound how long it *waits* with + `lock_timeout`. (The full weakest→strongest list lives in + [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md#lock-levels-weakest--strongest).) +- **MySQL 8.0** splits the concern: **metadata locks (MDL)** protect the schema, **InnoDB row + locks** protect DML, and for online DDL you *request* the concurrency you want via the + `ALGORITHM=` / `LOCK=` clause (`LOCK=NONE | SHARED | EXCLUSIVE`). + +| PostgreSQL lock mode | Typically acquired by | Closest MySQL 8.0 concept | Concurrent access | +| --- | --- | --- | --- | +| `ACCESS SHARE` | `SELECT` | `MDL_SHARED_READ` held by a query | reads | +| `ROW SHARE` | `SELECT ... FOR UPDATE/SHARE` | MDL read + InnoDB shared row locks | reads + locking reads | +| `ROW EXCLUSIVE` | `INSERT` / `UPDATE` / `DELETE` | `MDL_SHARED_WRITE` + InnoDB `IX`/row locks | reads + writes | +| `SHARE UPDATE EXCLUSIVE` | `CREATE INDEX CONCURRENTLY`, `VALIDATE CONSTRAINT`, `VACUUM`, `ANALYZE`, online `ALTER`s | online DDL `ALGORITHM=INPLACE/INSTANT, LOCK=NONE` | **reads + writes** (the "online" threshold) | +| `SHARE` | `CREATE INDEX` (non-concurrent) | online DDL `LOCK=SHARED` | reads only (writes blocked) | +| `SHARE ROW EXCLUSIVE` | `CREATE TRIGGER`, some `ALTER`s | `LOCK=SHARED` (no exact analog) | reads only | +| `EXCLUSIVE` | `REFRESH MATERIALIZED VIEW CONCURRENTLY` | between `LOCK=SHARED` and `LOCK=EXCLUSIVE` | plain reads only (no locking reads/writes) | +| `ACCESS EXCLUSIVE` | most `ALTER TABLE`, `DROP`, `TRUNCATE`, `REINDEX`, `VACUUM FULL`, the cutover swap | online DDL `LOCK=EXCLUSIVE` / the table lock of `ALGORITHM=COPY` | **nothing** (reads + writes blocked) | + +**Shared parallel worth noting:** even a fully "online" change needs a *brief* exclusive lock +at the metadata boundaries in both engines — PostgreSQL takes `ACCESS EXCLUSIVE` momentarily to +publish the catalog change, and MySQL takes a brief exclusive MDL at the start/end of a +`LOCK=NONE` operation. In both, that brief exclusive lock is what can still get stuck behind a +long-running transaction. + +## Why DDL is dangerous: the lock queue + +This is the prerequisite background for the whole design set: *why* an apparently instant +`ALTER TABLE` can still take an application down, and why online-schema-change tools exist at +all on both engines. It lives here because the failure mode and its mitigations are best +understood as a direct consequence of the [lock model comparison](#lock-model-comparison) +above. The per-operation lock/rewrite details are in +[postgres-online-ddl-reference.md](postgres-online-ddl-reference.md); the engine that +works around this is in [low-level-design.md](low-level-design.md). + +### The mechanism: a three-step pile-up + +The risk in PostgreSQL DDL is usually not the DDL's own work — it is the **lock queue**. The +mechanism has three steps: + +1. An `ALTER TABLE` needs `ACCESS EXCLUSIVE`, which conflicts with the `ACCESS SHARE` that + every `SELECT` already holds. So the `ALTER` cannot start until the in-flight queries on + that table finish; it **waits**. +2. PostgreSQL queues lock requests roughly in **arrival order** — the order in which each + statement requests the lock — and it does not let a later request jump ahead of an + already-waiting one, even if the later request would be compatible with the current holder. + So once the `ALTER` is waiting, **every statement that requests the lock after it lines up + behind it** — including plain `SELECT`s that would otherwise run fine alongside the + in-flight queries. +3. Each blocked query keeps holding its client connection. If enough pile up, they exhaust the + connection pool / `max_connections`, at which point **even queries on unrelated tables** + can't get a connection. That is how a single table's lock contention becomes an app-wide + outage. + +``` +time ──▶ + long query (holds ACCESS SHARE) ═══════════════════════════════╗ still running + ALTER TABLE (wants ACCESS EXCLUSIVE) ░░░░░░░░░░░░░░░░░░░║ waiting at head of queue + later SELECTs (want ACCESS SHARE) ░░░░░░░░░░░░░░░║ blocked behind the ALTER + └─ backlog grows; pool may exhaust +``` + +### The DDL is the catalyst, not the long query + +Without the `ALTER`, the long-running `SELECT` would **not** block the later `SELECT`s at all: +they all take `ACCESS SHARE`, which is compatible with itself, so any number of readers run +concurrently regardless of how long one of them takes (`ACCESS SHARE` only conflicts with +`ACCESS EXCLUSIVE`). The pile-up exists purely because the `ALTER`'s pending `ACCESS EXCLUSIVE` +request sits in the queue and the later readers will not jump ahead of it. Remove the DDL and +there is no queue. + +### How long does the impact last? + +It is **not** a fixed number — it is roughly *"how long the blocking transaction keeps +running"* plus the time to drain the backlog afterward. The `ALTER` clears the instant the +conflicting transaction(s) commit/abort and it acquires the lock (then its own work is +milliseconds for a metadata-only change). So the worst cases are driven by **how long +something holds a conflicting lock**: + +- a long analytics `SELECT` → impact lasts about as long as that query runs; +- an **idle-in-transaction** session that `SELECT`ed the table and never committed → impact + lasts until that session is closed or `idle_in_transaction_session_timeout` fires, which + can be effectively unbounded; +- without `lock_timeout`, the `ALTER` itself waits indefinitely, so the backlog keeps growing + the whole time. + +The takeaway is the *causal chain*, not a specific duration: a metadata-only `ALTER` can +amplify one slow or stuck transaction into widespread blocking. + +### Mitigations the engine and operators rely on + +- **Always set `lock_timeout`** (e.g. `SET lock_timeout = '3s'`) before DDL on a hot table, + so the `ALTER` gives up quickly instead of sitting at the head of the queue and growing a + backlog. Retry with backoff rather than waiting indefinitely. +- Keep `ACCESS EXCLUSIVE` windows as short as possible — this is exactly why the cutover swap + is the only `ACCESS EXCLUSIVE` step in the engine's design. +- Avoid running blocking DDL while long analytics queries or idle-in-transaction sessions hold + locks on the table; consider `idle_in_transaction_session_timeout` to bound the worst case. + +### The lock queue is the same in both engines + +MySQL has the **same dynamic**, via **metadata locks (MDL)** rather than table locks. A DDL +needs an exclusive MDL; it waits behind any open transaction still holding a shared MDL on the +table (the familiar `Waiting for table metadata lock` state), and subsequent queries queue +behind the waiting DDL — same three-step pile-up. The MySQL equivalents of the mitigations are +`lock_wait_timeout` (bound how long the DDL waits) and avoiding long/abandoned transactions. + +This is precisely why online-schema-change tools exist on **both** engines and why their +**cutover** is the delicate step: Spirit/gh-ost take a brief, bounded `RENAME TABLE` under a +metadata lock; pg-sprite takes a brief, bounded `ACCESS EXCLUSIVE` swap. The whole point of +the copy-and-swap approach is to replace one long lock-holding `ALTER` with a short, retryable +locked window. + +## Copy-and-swap executor: Spirit (MySQL) → PostgreSQL primitive mapping + +Spirit's copy-and-swap lifecycle maps cleanly onto PostgreSQL, but **every database-specific +primitive must be swapped**. This is the per-primitive translation the +[copy-and-swap executor](low-level-design.md#copy-and-swap-executor-lifecycle) is built on +(see the [Spirit repository](https://github.com/block/spirit) for how the MySQL original +works). Rows marked *Aurora* name the managed-service signal; the vanilla-PostgreSQL +equivalent is given alongside. + +| Spirit (MySQL) | PostgreSQL equivalent | +| --- | --- | +| Binlog (`go-mysql` `BinlogSyncer`) | **Logical decoding** via a replication slot (`pgoutput` / `wal2json` / `test_decoding`) — the faithful, low-overhead analog of the binlog. Trigger-based capture is the fallback. | +| `binlog_row_image=FULL` | `REPLICA IDENTITY FULL` (or default = PK) on the source table so updates/deletes carry enough identity | +| `SHOW BINARY LOG STATUS` → file:offset position | LSN + slot `confirmed_flush_lsn` | +| `REPLACE INTO target VALUES (...)` (apply) | `INSERT ... ON CONFLICT (pk) DO UPDATE SET ...` + explicit delete handling | +| `INSERT IGNORE ... SELECT` (copy) | `INSERT INTO shadow SELECT ... FROM src WHERE ON CONFLICT DO NOTHING` | +| `CRC32(CONCAT(col,...))` checksum | `md5(row::text)` aggregated per chunk, or `sum(hashtext(...))` / count compare | +| `RENAME TABLE old→_old, new→old` under `LOCK TABLES` (needs MySQL 8.0.13+) | `BEGIN; LOCK TABLE src IN ACCESS EXCLUSIVE MODE; ; ALTER TABLE src RENAME TO src_old; ALTER TABLE shadow RENAME TO src; COMMIT;` — **PostgreSQL's transactional DDL makes this cleaner than MySQL** | +| Force-kill via `performance_schema` | `pg_terminate_backend()` + `lock_timeout`/`statement_timeout` to bound the cutover wait | +| TiDB SQL parser (`pkg/statement`) | [`wasilibs/go-pgquery`](https://github.com/wasilibs/go-pgquery) (libpg_query compiled to Wasm — the real PostgreSQL grammar, no cgo) for parsing `ALTER` / `CREATE TABLE` | +| Aurora MySQL throttling (active threads, replica lag) | Replication **slot lag** (`pg_replication_slots`), WAL generation rate, replica lag (`pg_stat_replication`; *Aurora:* `aurora_replica_status()`, CloudWatch) | +| `AUTO_INCREMENT` optimistic chunker | `bigint`/`identity`/`serial` PK range chunker; composite-PK chunker otherwise | +| TLS / RDS CA auto-detection | same idea, the RDS/Aurora CA bundle for `pgx` when the target is a managed service | diff --git a/docs/postgres-online-ddl-reference.md b/docs/postgres-online-ddl-reference.md index 7310c25..bde4e72 100644 --- a/docs/postgres-online-ddl-reference.md +++ b/docs/postgres-online-ddl-reference.md @@ -14,10 +14,43 @@ These are exactly the two dimensions MySQL lets authors *assert* with `ALGORITHM pg-sprite's `diff` and `migrate --dry-run` are that missing declaration today: they classify and route the change. Routed execution lands with the Phase 3 executor. -The rest of this document breaks down both dimensions per operation. +The rest of this document breaks down both dimensions per operation. + +## The three buckets: what MySQL's `ALGORITHM` states map to + +MySQL 8.0 asserts one of three work-done states per `ALTER`: `INSTANT` (metadata only), +`INPLACE` (rebuilt in place — usually a full scan, sometimes a rebuild — while DML continues), +and `COPY` (a full table copy). In practice MySQL tooling treats this as binary — INSTANT, or +a copy — because INPLACE on a large table costs about what COPY does. + +PostgreSQL has the same three states; it just does not name them. Crossing the two axes +above (lock × work done) gives exactly three buckets every DDL in this reference falls into: + +| Bucket | Work done | Lock | MySQL analog | Examples | pg-sprite route | +| --- | --- | --- | --- | --- | --- | +| **Catalog-only** | none — a catalog entry changes; no row is read or written | brief `ACCESS EXCLUSIVE` (milliseconds once acquired) | `ALGORITHM=INSTANT` | `ADD COLUMN` (nullable or constant default), `DROP COLUMN`, `SET/DROP DEFAULT`, `DROP NOT NULL`, renames, [binary-coercible type changes](binary-coercible-type-changes.md) | native, as written | +| **Full scan, no rewrite** | every row is *read* (to validate or to build an index); the heap is not rewritten | native online forms hold `SHARE UPDATE EXCLUSIVE` for the scan; the as-written forms hold a blocking lock for the whole scan | `ALGORITHM=INPLACE` | `SET NOT NULL`, `ADD CONSTRAINT … CHECK/FOREIGN KEY`, `ADD CONSTRAINT … UNIQUE`, `CREATE INDEX` | native, **safer sequence** substituted (`NOT VALID` + `VALIDATE`, `CONCURRENTLY` + `USING INDEX`) | +| **Full rewrite** | every row is *written* into a new heap; all indexes rebuilt | `ACCESS EXCLUSIVE` for the whole rewrite | `ALGORITHM=COPY` | `ALTER COLUMN TYPE` (non-coercible), `ADD COLUMN … DEFAULT `, `ADD COLUMN … GENERATED … STORED`, `SET TABLESPACE`, `VACUUM FULL`/`CLUSTER` | **copy-and-swap** (typed refusal until the executor lands) | + +Two things fall out of this. For a coarse "does this cost scale with table size?" question, +the last two buckets answer alike — only the first is free. For deciding *how* to run the +change, the split between the second and third bucket is what matters: the second bucket +nearly always has a native online form that keeps DML flowing (the lock, not the scan, was +the problem — exclusion constraints are the notable exception); the third has none, and only +a copy-and-swap avoids the long exclusive lock. The +plan report's `decisions[].reason` vocabulary +([plan-report.md](plan-report.md#planner-decision-reasons-decisionsreason)) is this table +made machine-readable: `metadata-only` / `fast-default` / `binary-coercible` are the first +bucket, `safer-idiom` the second, `type-rewrite` / `volatile-default` / `generated-stored` / +`relocation` the third. + +The MySQL-side detail — how `ALGORITHM=` and `LOCK=` are asserted and how the two engines' +lock models line up — is in +[mysql-vs-postgresql.md](mysql-vs-postgresql.md#how-online-ddl-is-expressed). ## Table of contents +- [The three buckets: what MySQL's `ALGORITHM` states map to](#the-three-buckets-what-mysqls-algorithm-states-map-to) - [Lock levels (weakest → strongest)](#lock-levels-weakest--strongest) - [Why DDL is dangerous: the lock queue](#why-ddl-is-dangerous-the-lock-queue) - [Column operations](#column-operations) @@ -44,13 +77,13 @@ ACCESS SHARE < ROW SHARE < ROW EXCLUSIVE < SHARE UPDATE EXCLUSIVE ### Comparison with MySQL / InnoDB The PostgreSQL lock-mode → MySQL 8.0 (MDL + InnoDB) mapping, the `ALGORITHM=`/`LOCK=` contrast, -and the brief-exclusive-lock parallel now live in the dedicated comparison doc: -**12-mysql-vs-postgresql.md § Lock model comparison**. +and the brief-exclusive-lock parallel live in the dedicated comparison doc: +[mysql-vs-postgresql.md § Lock model comparison](mysql-vs-postgresql.md#lock-model-comparison). ## Why DDL is dangerous: the lock queue This is covered in the comparison reference: -**12-mysql-vs-postgresql.md § Why DDL is dangerous: the lock queue**. +[mysql-vs-postgresql.md § Why DDL is dangerous: the lock queue](mysql-vs-postgresql.md#why-ddl-is-dangerous-the-lock-queue). It explains the three-step lock-queue pile-up, why the DDL (not the long query) is the catalyst, how long the impact lasts, the mitigations the engine relies on, and why MySQL has the same dynamic via metadata locks. Read it first if any of that is unfamiliar. @@ -313,7 +346,10 @@ the catalog instead of rewriting the table. Catalog-only; runs as written. The type change is binary-coercible (for example `varchar(50)` → `varchar(100)`, or `varchar` → `text`), so PostgreSQL relabels the column in place without a -table rewrite. Runs as written. +table rewrite. Runs as written. How PostgreSQL decides, which changes only +*look* free (shortening `varchar`, changing `numeric` scale, `char(n)` → `text`), +and the exact rules pg-sprite accepts are in +[binary-coercible-type-changes.md](binary-coercible-type-changes.md). ### `safer-idiom`