diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a2b88cb..0e401e5 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -198,21 +198,21 @@ jobs: git clone --depth 1 \ "https://x-access-token:${TOKEN}@github.com/pgEdge/pgedge-detect-build-matrix.git" \ ".github/actions/pgedge-detect-build-matrix" - - name: Detect — pg_duckdb (PG 16/17/18) + - name: Detect — pg_duckdb (PG 16/17/18/19) id: pgduckdb uses: ./.github/actions/pgedge-detect-build-matrix with: component_name: packaging/pg_duckdb - pg_versions: '16,17,18' + pg_versions: '16,17,18,19' archs: '["amd64","arm64"]' rpm_images: ${{ env.RPM_IMAGES }} deb_images: ${{ env.DEB_IMAGES }} - - name: Detect — coldfront extension (PG 16/17/18) + - name: Detect — coldfront extension (PG 16/17/18/19) id: coldfront uses: ./.github/actions/pgedge-detect-build-matrix with: component_name: packaging/coldfront - pg_versions: '16,17,18' + pg_versions: '16,17,18,19' archs: '["amd64","arm64"]' rpm_images: ${{ env.RPM_IMAGES }} deb_images: ${{ env.DEB_IMAGES }} diff --git a/DUCKDB_1.5_PATCHED.md b/DUCKDB_1.5_PATCHED.md index 78d29ae..85ca608 100644 --- a/DUCKDB_1.5_PATCHED.md +++ b/DUCKDB_1.5_PATCHED.md @@ -19,7 +19,28 @@ and verified*. The cold-tier compactor's own story lives in | **Bakery-aware commit-refresh** | `docker/iceberg-bakery-aware-commit-refresh-v15.patch` | makes the async parquet-upload ordering safe → the **no-409** guarantee for concurrent cold writers, at contended-upload throughput | cold writes still work and still never 409 — they fall back to serialized (claim-first) uploads (see [DUCKDB_1.5_UNPATCHED.md](DUCKDB_1.5_UNPATCHED.md)) | | **Strict-reader interop** (upstreamable) | `docker/iceberg-manifest-list-format-version-v15.patch`, `docker/iceberg-data-file-format-v15.patch` | make the manifests duckdb-iceberg *writes* readable by strict Apache readers (apache/iceberg-go) | the cold-tier **compactor cannot read the table** - see [docs/compaction.md](docs/compaction.md). pg_duckdb's own reads/writes are unaffected. | -All three patches apply cleanly to a **pristine** `duckdb-iceberg` @ `5edc45f0` +There is also one patch outside the iceberg families, and outside the image: +**PG-19 deparser re-sync** (`packaging/patches/pg_duckdb-pg19-ruleutils.patch`). +pg_duckdb's vendored PG-19 `ruleutils.c` predates two PostgreSQL changes late in +the 19 beta cycle — the `GROUP BY ALL` revert (postgres `372b8d1`) and +`ForPortionOfExpr.range_name` → `rangeVar` (postgres `2a9541d`) — so it +references fields that no longer exist, and without the patch the extension does +not compile against PG 19 at all. The file is `#if`'d to PG 19, so it is +irrelevant to PG 16/17/18. `packaging/patches/apply-pg19-ruleutils-patch.sh` — +shared by `packaging/pg_duckdb/build-rpm.sh` and `build-deb.sh` — is the one +place that decides: it applies the patch only for PG >= 19, skips silently when +the change is already in the tree (a re-run, or upstream having refreshed its +vendored copy), and fails the build on anything else, so patch rot is loud but a +stale patch never blocks a PG 16/17/18 build. PG 19 also needs `PG_MAX_VER=19` +passed to `make` - upstream's Makefile.global defaults it to 18. Drop both once +upstream refreshes the vendored copy; the helper's skip message says when that +has happened. + +This is a packaging-only path. The base image is unchanged: it stays on +pg_duckdb `c04e6a2` and builds PG 16/17/18, because there is no +`pgedge-postgres:19-*` base to build 19 from. Only the RPM/DEB cells build 19. + +All three iceberg patches apply cleanly to a **pristine** `duckdb-iceberg` @ `5edc45f0` (branch `v1.5-variegata`); `docker/Dockerfile.duckdb15-base` `git apply --check`s each before applying, failing the build loudly on patch rot. @@ -144,8 +165,9 @@ patches only. | Component | Pin | Notes | |---|---|---| -| pg_duckdb | **merged PR #1025** (`c04e6a2`) | no released tag carries 1.5.x; `git checkout c04e6a2`. Its duckdb submodule is the v1.5.4 tag (`08e34c4`). | -| DuckDB | **v1.5.4 tag** (`08e34c4`) | pinned by pg_duckdb @ `c04e6a2`; the iceberg build re-pins ITS duckdb submodule to the same tag so the extension ABI matches the engine. The `duckdb.*` GUCs + PRE_COMMIT iceberg-commit deferral ColdFront relies on are unchanged. | +| pg_duckdb (packages) | **PR #983** (`ee7aaeb`) | PG 16-19 support, three commits past `c04e6a2`; no released tag carries 1.5.x. Pinned in `packaging/pg_duckdb/common.sh`. Its duckdb submodule is still the v1.5.4 tag (`08e34c4`). | +| pg_duckdb (base image) | **merged PR #1025** (`c04e6a2`) | the image builds PG 16/17/18 only (no `pgedge-postgres:19-*` base), so it stays on the pin it shipped with rather than rebuilding three majors for a 19 it cannot use. Pinned in `docker/Dockerfile.duckdb15-base`. | +| DuckDB | **v1.5.4 tag** (`08e34c4`) | pinned by both pg_duckdb commits; the iceberg build re-pins ITS duckdb submodule to the same tag so the extension ABI matches the engine. The `duckdb.*` GUCs + PRE_COMMIT iceberg-commit deferral ColdFront relies on are unchanged. | | duckdb-iceberg | **`v1.5-variegata` @ `5edc45f0`** | extension code the three patches target — kept fixed, so the patches apply unchanged. The build re-pins its duckdb submodule to the v1.5.4 tag (the branch tracks duckdb `main`, which drifts off the release; verified: `5edc45f0` compiles clean against v1.5.4). Transaction code lives in `src/catalog/rest/transaction/`. | | avro | **`7f423d69`** | the pin `v1.5-variegata` uses. | | azure | **`v1.5-variegata` @ `563589b2`** | the ABI-matched sibling of iceberg's branch. **NOT `main`** — azure `main` collides at link (`multiple definition of duckdb::FileFlags::FILE_FLAGS_NULL_IF_NOT_EXISTS`). | diff --git a/docs/installation.md b/docs/installation.md index 40b1cf8..e852fb2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -90,7 +90,8 @@ Then follow [usage.md → One-time setup](usage.md#one-time-setup) > **pg_duckdb pin.** The base pins pg_duckdb to the merged PR #1025 commit > `c04e6a2` (DuckDB 1.5.4 — its duckdb submodule is the v1.5.4 tag), a fixed -> commit for reproducible builds rather than a moving PR head. +> commit for reproducible builds rather than a moving PR head. The RPM/DEB +> packages pin a later commit — see [PostgreSQL 19 (beta)](#postgresql-19-beta). > > **Base foundation.** The base is > `FROM ghcr.io/pgedge/pgedge-postgres:-spock5-minimal`; you need pull @@ -190,6 +191,21 @@ coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user== 190000 +/* PG 19+ passes a const JumbleState. */ +coldfront_post_parse_analyze(ParseState *pstate, Query *query, + const JumbleState *jstate) +#else coldfront_post_parse_analyze(ParseState *pstate, Query *query, JumbleState *jstate) +#endif { TieredViewInfo info; char *new_sql; diff --git a/extension/coldfront/test/expected/cte_on_insert_2.out b/extension/coldfront/test/expected/cte_on_insert_2.out new file mode 100644 index 0000000..8622c8d --- /dev/null +++ b/extension/coldfront/test/expected/cte_on_insert_2.out @@ -0,0 +1,74 @@ +-- Regression guard: a leading WITH (CTE) clause on INSERT into a tiered view. +-- emit_tiered_insert splits the row source across the PG hot half and the DuckDB +-- cold half; the CTE must reach both. The rewrite folds the WITH into the source +-- subquery so its CTEs scope to the derived table on each engine. White-box: +-- EXPLAIN VERBOSE shows the rewritten split; we do NOT execute. +CREATE EXTENSION IF NOT EXISTS pg_duckdb; +NOTICE: extension "pg_duckdb" already exists, skipping +CREATE EXTENSION IF NOT EXISTS coldfront; +NOTICE: extension "coldfront" already exists, skipping +SET TIME ZONE 'UTC'; +SET coldfront.warehouse = ''; +SET coldfront.lakekeeper_endpoint = ''; +SET coldfront.local_pg_dsn = ''; +CREATE TABLE public._events (id int, ts timestamptz, status text); +CREATE VIEW public.events AS SELECT * FROM public._events; +INSERT INTO coldfront.tiered_views(schema_name, relname, hot_table, iceberg_table, partition_col) +VALUES ('public', 'events', 'public._events', 'ice.default.events', 'ts'); +INSERT INTO coldfront.archive_watermark(schema_name, table_name, cutoff_time) +VALUES ('public', 'events', '2026-03-01'::timestamptz); +-- INSERT ... SELECT FROM a CTE into the tiered view: the WITH must reach both the +-- hot (PG) and cold (DuckDB) halves, folded into the source derived table. +EXPLAIN (COSTS OFF, VERBOSE) + WITH s AS (SELECT 7 AS id, '2026-05-01 00:00:00+00'::timestamptz AS ts, 'new' AS status) + INSERT INTO public.events SELECT id, ts, status FROM s; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ + Result + Output: (InitPlan expr_1).col1, (InitPlan expr_2).col1 + CTE hot_ins + -> Insert on public._events + Output: 1 + -> Result + Output: 7, 'Fri May 01 00:00:00 2026 UTC'::timestamp with time zone, 'new'::text + CTE cold_call + -> Result + Output: _exec_iceberg_with_claim('ice.default.events'::text, 'INSERT INTO ice.default.events SELECT id, ts, status FROM (WITH s AS ( SELECT 7 AS id, ''Fri May 01 00:00:00 2026 UTC''::timestamptz AS ts, ''new''::text AS status ) SELECT s.id, s.ts, s.status FROM s) AS coldfront_src(id, ts, status) WHERE ts < ''Sun Mar 01 00:00:00 2026 UTC''::timestamptz'::text) + InitPlan expr_1 + -> Aggregate + Output: count(*) + -> CTE Scan on hot_ins + Output: hot_ins."?column?" + InitPlan expr_2 + -> Aggregate + Output: count(*) + -> CTE Scan on cold_call + Output: cold_call._exec_iceberg_with_claim +(20 rows) + +-- Cleanup. +DELETE FROM coldfront.tiered_views; +DELETE FROM coldfront.archive_watermark; +DROP VIEW public.events; +DROP TABLE public._events; +-- The clustered iceberg-only INSERT is re-emitted so the cluster is derived in +-- the same statement, and the CTE must survive that re-emission too: the WITH +-- folds into the derived table the assignment reads from, the only scope its +-- CTEs are visible from. +CREATE TABLE public._vec_base (id int, ts timestamptz, embedding real[]); +CREATE VIEW public.icevec AS SELECT * FROM public._vec_base; +INSERT INTO coldfront.tiered_views(schema_name, relname, iceberg_table, is_iceberg_only, vec_columns) +VALUES ('public', 'icevec', 'ice.default.icevec', true, ARRAY['embedding']); +EXPLAIN (COSTS OFF, VERBOSE) + WITH s AS (SELECT 7 AS id, '2026-05-01 00:00:00+00'::timestamptz AS ts, ARRAY[1,0,0]::real[] AS embedding) + INSERT INTO public.icevec SELECT id, ts, embedding FROM s; + QUERY PLAN +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Result + Output: _exec_iceberg_with_claim('ice.default.icevec'::text, 'INSERT INTO ice.default.icevec (_cf_vec_list_embedding, id, ts, embedding) SELECT (SELECT arg_min(c.centroid_id, list_cosine_distance(c.centroid, coldfront_src.embedding)) FROM pglocal.coldfront.vector_centroids c WHERE c.schema_name = ''public'' AND c.table_name = ''icevec'' AND c.column_name = ''embedding'' AND c.generation = (SELECT vc.generation FROM pglocal.coldfront.vector_config vc WHERE vc.schema_name = ''public'' AND vc.table_name = ''icevec'' AND vc.column_name = ''embedding'')), id, ts, embedding FROM (WITH s AS ( SELECT 7 AS id, ''Fri May 01 00:00:00 2026 UTC''::timestamptz AS ts, ARRAY[(1)::real, (0)::real, (0)::real] AS embedding ) SELECT s.id, s.ts, s.embedding FROM s) AS coldfront_src(id, ts, embedding)'::text) +(2 rows) + +-- Cleanup. +DELETE FROM coldfront.tiered_views WHERE relname = 'icevec'; +DROP VIEW public.icevec; +DROP TABLE public._vec_base; diff --git a/extension/coldfront/test/expected/param_cold_via_plpgsql_2.out b/extension/coldfront/test/expected/param_cold_via_plpgsql_2.out new file mode 100644 index 0000000..a063729 --- /dev/null +++ b/extension/coldfront/test/expected/param_cold_via_plpgsql_2.out @@ -0,0 +1,170 @@ +-- Cause 1 (params -> DuckDB): a cold-tier write carrying bound parameters ($N) +-- — from PREPARE/EXECUTE, the extended protocol, or a plpgsql variable — must +-- keep those params LIVE. Before the fix the deparsed cold SQL baked "$N" into a +-- string literal DuckDB could not bind ("Expected N parameters, but none were +-- supplied"). The fix emits the cold SQL as a runtime format(