feat: adapt the coldfront extension and pg_duckdb to PostgreSQL 19

.github/workflows/release.yml

@@ -198,21 +198,21 @@ jobs:
                   git clone --depth 1 \
                       "https://x-access-token:${TOKEN}@github.com/pgEdge/pgedge-detect-build-matrix.git" \
                       ".github/actions/pgedge-detect-build-matrix"
-            - name: Detect — pg_duckdb (PG 16/17/18)
+            - name: Detect — pg_duckdb (PG 16/17/18/19)
               id: pgduckdb
               uses: ./.github/actions/pgedge-detect-build-matrix
               with:
                   component_name: packaging/pg_duckdb
-                  pg_versions: '16,17,18'
+                  pg_versions: '16,17,18,19'
                   archs: '["amd64","arm64"]'
                   rpm_images: ${{ env.RPM_IMAGES }}
                   deb_images: ${{ env.DEB_IMAGES }}
-            - name: Detect — coldfront extension (PG 16/17/18)
+            - name: Detect — coldfront extension (PG 16/17/18/19)
               id: coldfront
               uses: ./.github/actions/pgedge-detect-build-matrix
               with:
                   component_name: packaging/coldfront
-                  pg_versions: '16,17,18'
+                  pg_versions: '16,17,18,19'
                   archs: '["amd64","arm64"]'
                   rpm_images: ${{ env.RPM_IMAGES }}
                   deb_images: ${{ env.DEB_IMAGES }}

DUCKDB_1.5_PATCHED.md

@@ -19,7 +19,28 @@ and verified*. The cold-tier compactor's own story lives in
 | **Bakery-aware commit-refresh** | `docker/iceberg-bakery-aware-commit-refresh-v15.patch` | makes the async parquet-upload ordering safe → the **no-409** guarantee for concurrent cold writers, at contended-upload throughput | cold writes still work and still never 409 — they fall back to serialized (claim-first) uploads (see [DUCKDB_1.5_UNPATCHED.md](DUCKDB_1.5_UNPATCHED.md)) |
 | **Strict-reader interop** (upstreamable) | `docker/iceberg-manifest-list-format-version-v15.patch`, `docker/iceberg-data-file-format-v15.patch` | make the manifests duckdb-iceberg *writes* readable by strict Apache readers (apache/iceberg-go) | the cold-tier **compactor cannot read the table** - see [docs/compaction.md](docs/compaction.md). pg_duckdb's own reads/writes are unaffected. |
 
-All three patches apply cleanly to a **pristine** `duckdb-iceberg` @ `5edc45f0`
+There is also one patch outside the iceberg families, and outside the image:
+**PG-19 deparser re-sync** (`packaging/patches/pg_duckdb-pg19-ruleutils.patch`).
+pg_duckdb's vendored PG-19 `ruleutils.c` predates two PostgreSQL changes late in
+the 19 beta cycle — the `GROUP BY ALL` revert (postgres `372b8d1`) and
+`ForPortionOfExpr.range_name` → `rangeVar` (postgres `2a9541d`) — so it
+references fields that no longer exist, and without the patch the extension does
+not compile against PG 19 at all. The file is `#if`'d to PG 19, so it is
+irrelevant to PG 16/17/18. `packaging/patches/apply-pg19-ruleutils-patch.sh` —
+shared by `packaging/pg_duckdb/build-rpm.sh` and `build-deb.sh` — is the one
+place that decides: it applies the patch only for PG >= 19, skips silently when
+the change is already in the tree (a re-run, or upstream having refreshed its
+vendored copy), and fails the build on anything else, so patch rot is loud but a
+stale patch never blocks a PG 16/17/18 build. PG 19 also needs `PG_MAX_VER=19`
+passed to `make` - upstream's Makefile.global defaults it to 18. Drop both once
+upstream refreshes the vendored copy; the helper's skip message says when that
+has happened.
+
+This is a packaging-only path. The base image is unchanged: it stays on
+pg_duckdb `c04e6a2` and builds PG 16/17/18, because there is no
+`pgedge-postgres:19-*` base to build 19 from. Only the RPM/DEB cells build 19.
+
+All three iceberg patches apply cleanly to a **pristine** `duckdb-iceberg` @ `5edc45f0`
 (branch `v1.5-variegata`); `docker/Dockerfile.duckdb15-base` `git apply --check`s
 each before applying, failing the build loudly on patch rot.
 
@@ -144,8 +165,9 @@ patches only.
 
 | Component | Pin | Notes |
 |---|---|---|
-| pg_duckdb | **merged PR #1025** (`c04e6a2`) | no released tag carries 1.5.x; `git checkout c04e6a2`. Its duckdb submodule is the v1.5.4 tag (`08e34c4`). |
-| DuckDB | **v1.5.4 tag** (`08e34c4`) | pinned by pg_duckdb @ `c04e6a2`; the iceberg build re-pins ITS duckdb submodule to the same tag so the extension ABI matches the engine. The `duckdb.*` GUCs + PRE_COMMIT iceberg-commit deferral ColdFront relies on are unchanged. |
+| pg_duckdb (packages) | **PR #983** (`ee7aaeb`) | PG 16-19 support, three commits past `c04e6a2`; no released tag carries 1.5.x. Pinned in `packaging/pg_duckdb/common.sh`. Its duckdb submodule is still the v1.5.4 tag (`08e34c4`). |
+| pg_duckdb (base image) | **merged PR #1025** (`c04e6a2`) | the image builds PG 16/17/18 only (no `pgedge-postgres:19-*` base), so it stays on the pin it shipped with rather than rebuilding three majors for a 19 it cannot use. Pinned in `docker/Dockerfile.duckdb15-base`. |
+| DuckDB | **v1.5.4 tag** (`08e34c4`) | pinned by both pg_duckdb commits; the iceberg build re-pins ITS duckdb submodule to the same tag so the extension ABI matches the engine. The `duckdb.*` GUCs + PRE_COMMIT iceberg-commit deferral ColdFront relies on are unchanged. |
 | duckdb-iceberg | **`v1.5-variegata` @ `5edc45f0`** | extension code the three patches target — kept fixed, so the patches apply unchanged. The build re-pins its duckdb submodule to the v1.5.4 tag (the branch tracks duckdb `main`, which drifts off the release; verified: `5edc45f0` compiles clean against v1.5.4). Transaction code lives in `src/catalog/rest/transaction/`. |
 | avro | **`7f423d69`** | the pin `v1.5-variegata` uses. |
 | azure | **`v1.5-variegata` @ `563589b2`** | the ABI-matched sibling of iceberg's branch. **NOT `main`** — azure `main` collides at link (`multiple definition of duckdb::FileFlags::FILE_FLAGS_NULL_IF_NOT_EXISTS`). |

docs/installation.md

@@ -90,7 +90,8 @@ Then follow [usage.md → One-time setup](usage.md#one-time-setup)
 
 > **pg_duckdb pin.** The base pins pg_duckdb to the merged PR #1025 commit
 > `c04e6a2` (DuckDB 1.5.4 — its duckdb submodule is the v1.5.4 tag), a fixed
-> commit for reproducible builds rather than a moving PR head.
+> commit for reproducible builds rather than a moving PR head. The RPM/DEB
+> packages pin a later commit — see [PostgreSQL 19 (beta)](#postgresql-19-beta).
 >
 > **Base foundation.** The base is
 > `FROM ghcr.io/pgedge/pgedge-postgres:<pg>-spock5-minimal`; you need pull
@@ -190,6 +191,21 @@ coldfront.local_pg_dsn        = 'host=/var/run/postgresql dbname=<db> user=<role
 (See the README for the full GUC set and the optional turnkey
 non-superuser role.)
 
+### PostgreSQL 19 (beta)
+
+PG 19 is a packages-only target. The RPM and DEB cells build it alongside
+16/17/18 - the pgEdge `dnf`/`apt` repositories carry PG 19 beta, and both the
+coldfront extension and pg_duckdb compile against it. Those cells pin pg_duckdb
+to `ee7aaeb` (PR #983, PG 16-19 support) in `packaging/pg_duckdb/common.sh`, and
+need two ColdFront-side adjustments for 19: `PG_MAX_VER=19` (upstream's
+Makefile.global defaults it to 18) and
+`packaging/patches/pg_duckdb-pg19-ruleutils.patch`, which re-syncs its vendored
+PG-19 deparser with PG 19 as released. Both are no-ops below 19.
+
+The container images stay on PG 16/17/18 and on pg_duckdb `c04e6a2`: there is no
+`pgedge-postgres:19-*` base to build from and Spock has no 19 build, so the
+`ci/matrix.sh` cells - and therefore mesh topologies - remain 16/17/18.
+
 ## Testing & CI
 
 One canonical user journey ([ci/journey.sh](https://github.com/pgEdge/ColdFront/blob/main/ci/journey.sh)) runs identically in

extension/coldfront/src/coldfront.c

@@ -3108,8 +3108,14 @@ cf_dispatch_emit(Query *query, RangeTblEntry *rte, TieredViewInfo *info,
  * rewrite, so do nothing — the same guard the DDL hook already applies.
  */
 static void
+#if PG_VERSION_NUM >= 190000
+/* PG 19+ passes a const JumbleState. */
+coldfront_post_parse_analyze(ParseState *pstate, Query *query,
+                              const JumbleState *jstate)
+#else
 coldfront_post_parse_analyze(ParseState *pstate, Query *query,
                               JumbleState *jstate)
+#endif
 {
     TieredViewInfo  info;
     char           *new_sql;

extension/coldfront/test/expected/cte_on_insert_2.out

@@ -0,0 +1,74 @@
+-- Regression guard: a leading WITH (CTE) clause on INSERT into a tiered view.
+-- emit_tiered_insert splits the row source across the PG hot half and the DuckDB
+-- cold half; the CTE must reach both. The rewrite folds the WITH into the source
+-- subquery so its CTEs scope to the derived table on each engine. White-box:
+-- EXPLAIN VERBOSE shows the rewritten split; we do NOT execute.
+CREATE EXTENSION IF NOT EXISTS pg_duckdb;
+NOTICE:  extension "pg_duckdb" already exists, skipping
+CREATE EXTENSION IF NOT EXISTS coldfront;
+NOTICE:  extension "coldfront" already exists, skipping
+SET TIME ZONE 'UTC';
+SET coldfront.warehouse = '';
+SET coldfront.lakekeeper_endpoint = '';
+SET coldfront.local_pg_dsn = '';
+CREATE TABLE public._events (id int, ts timestamptz, status text);
+CREATE VIEW public.events AS SELECT * FROM public._events;
+INSERT INTO coldfront.tiered_views(schema_name, relname, hot_table, iceberg_table, partition_col)
+VALUES ('public', 'events', 'public._events', 'ice.default.events', 'ts');
+INSERT INTO coldfront.archive_watermark(schema_name, table_name, cutoff_time)
+VALUES ('public', 'events', '2026-03-01'::timestamptz);
+-- INSERT ... SELECT FROM a CTE into the tiered view: the WITH must reach both the
+-- hot (PG) and cold (DuckDB) halves, folded into the source derived table.
+EXPLAIN (COSTS OFF, VERBOSE)
+  WITH s AS (SELECT 7 AS id, '2026-05-01 00:00:00+00'::timestamptz AS ts, 'new' AS status)
+  INSERT INTO public.events SELECT id, ts, status FROM s;
+                                                                                                                                                                                                                            QUERY PLAN                                                                                                                                                                                                                            
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Result
+   Output: (InitPlan expr_1).col1, (InitPlan expr_2).col1
+   CTE hot_ins
+     ->  Insert on public._events
+           Output: 1
+           ->  Result
+                 Output: 7, 'Fri May 01 00:00:00 2026 UTC'::timestamp with time zone, 'new'::text
+   CTE cold_call
+     ->  Result
+           Output: _exec_iceberg_with_claim('ice.default.events'::text, 'INSERT INTO ice.default.events SELECT id, ts, status FROM (WITH s AS (          SELECT 7 AS id,             ''Fri May 01 00:00:00 2026 UTC''::timestamptz AS ts,             ''new''::text AS status         )  SELECT s.id,             s.ts,             s.status            FROM s) AS coldfront_src(id, ts, status) WHERE ts < ''Sun Mar 01 00:00:00 2026 UTC''::timestamptz'::text)
+   InitPlan expr_1
+     ->  Aggregate
+           Output: count(*)
+           ->  CTE Scan on hot_ins
+                 Output: hot_ins."?column?"
+   InitPlan expr_2
+     ->  Aggregate
+           Output: count(*)
+           ->  CTE Scan on cold_call
+                 Output: cold_call._exec_iceberg_with_claim
+(20 rows)
+
+-- Cleanup.
+DELETE FROM coldfront.tiered_views;
+DELETE FROM coldfront.archive_watermark;
+DROP VIEW public.events;
+DROP TABLE public._events;
+-- The clustered iceberg-only INSERT is re-emitted so the cluster is derived in
+-- the same statement, and the CTE must survive that re-emission too: the WITH
+-- folds into the derived table the assignment reads from, the only scope its
+-- CTEs are visible from.
+CREATE TABLE public._vec_base (id int, ts timestamptz, embedding real[]);
+CREATE VIEW public.icevec AS SELECT * FROM public._vec_base;
+INSERT INTO coldfront.tiered_views(schema_name, relname, iceberg_table, is_iceberg_only, vec_columns)
+VALUES ('public', 'icevec', 'ice.default.icevec', true, ARRAY['embedding']);
+EXPLAIN (COSTS OFF, VERBOSE)
+  WITH s AS (SELECT 7 AS id, '2026-05-01 00:00:00+00'::timestamptz AS ts, ARRAY[1,0,0]::real[] AS embedding)
+  INSERT INTO public.icevec SELECT id, ts, embedding FROM s;
+                                                                                                                                                                                                                                                                                                                                                                                                                                                  QUERY PLAN                                                                                                                                                                                                                                                                                                                                                                                                                                                   
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Result
+   Output: _exec_iceberg_with_claim('ice.default.icevec'::text, 'INSERT INTO ice.default.icevec (_cf_vec_list_embedding, id, ts, embedding) SELECT (SELECT arg_min(c.centroid_id, list_cosine_distance(c.centroid, coldfront_src.embedding)) FROM pglocal.coldfront.vector_centroids c WHERE c.schema_name = ''public'' AND c.table_name = ''icevec'' AND c.column_name = ''embedding'' AND c.generation = (SELECT vc.generation FROM pglocal.coldfront.vector_config vc WHERE vc.schema_name = ''public'' AND vc.table_name = ''icevec'' AND vc.column_name = ''embedding'')), id, ts, embedding FROM (WITH s AS (          SELECT 7 AS id,             ''Fri May 01 00:00:00 2026 UTC''::timestamptz AS ts,             ARRAY[(1)::real, (0)::real, (0)::real] AS embedding         )  SELECT s.id,             s.ts,             s.embedding            FROM s) AS coldfront_src(id, ts, embedding)'::text)
+(2 rows)
+
+-- Cleanup.
+DELETE FROM coldfront.tiered_views WHERE relname = 'icevec';
+DROP VIEW public.icevec;
+DROP TABLE public._vec_base;