Repository navigation
Bump to DuckDB 2.0 - #2
Open
leo-altertable wants to merge 153 commits into
Open
leo-altertable wants to merge 153 commits into
leo-altertable wants to merge 153 commits into
Conversation
DuckDB 1.5 introduced CRS metadata on the GEOMETRY type (e.g.,
GEOMETRY('EPSG:4326')). This change makes that CRS survive a full
round trip through PostgreSQL/PostGIS via postgres_scanner.
Write side (DuckDB → PostGIS):
- WriteGeometry detects CRS on the LogicalType and writes EWKB
instead of plain WKB so PostGIS receives the SRID:
WKB: [byte_order:1] [type:4] [payload...]
EWKB: [byte_order:1] [type|0x20000000:4] [srid:4] [payload...]
Falls back to plain WKB when no CRS is set, preserving the
previous behavior. (Uses GeoType::HasCRS / GeoType::GetCRS — the
earlier draft mistakenly referenced LogicalType::GetCRS, which
does not exist on DuckDB 1.5.)
Read side (PostGIS → DuckDB):
- TypeToLogicalType decodes the SRID from PostGIS's column-level
typmod for `geometry(TYPE, SRID)` columns and returns
LogicalType::GEOMETRY("EPSG:N") instead of plain GEOMETRY().
The typmod layout is:
bits 0 : has-M
bits 1 : has-Z
bits 2-7 : geometry type code (POINT=1, LINESTRING=2, ...)
bits 8-29 : SRID
Untyped `geometry` columns carry typmod -1 and fall back to plain
GEOMETRY (no CRS), matching the previous behavior.
Before: DuckDB GEOMETRY('EPSG:4326') → PostGIS geometry(SRID=0)
PostGIS geometry(POINT,4326) → DuckDB GEOMETRY (no CRS)
After: DuckDB GEOMETRY('EPSG:4326') → PostGIS geometry(SRID=4326)
PostGIS geometry(POINT,4326) → DuckDB GEOMETRY('EPSG:4326')
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added the EWKB write path and typmod read decode,
but on CREATE TABLE AS the CRS was being stripped before reaching
either side, so the resulting PostGIS column landed as plain
`geometry` (typmod -1) and the read-back lost the SRID.
Two strip points in this file:
1. ToPostgresType (called by AddCastToPostgresTypes during CTAS
planning) returned LogicalType::GEOMETRY() unconditionally,
dropping CRS before the projection that feeds the new physical
plan. Now returns the input unchanged for GEOMETRY so CRS
survives the cast/projection.
2. TypeToString returned plain "GEOMETRY" for any GEOMETRY type and
would have been short-circuited anyway by the HasAlias() early
return (GEOMETRY('EPSG:N') carries an alias of "GEOMETRY"). The
new branch runs *before* HasAlias, detects CRS via
GeoType::HasCRS, and emits `geometry(Geometry, N)` so PostGIS
records the SRID in pg_attribute.atttypmod and the existing
read-side typmod decode picks it up.
Verified end-to-end through the streamability harness in
duckdb-warehouse-transfer: postgres -> oracle and postgres -> snowflake
now round-trip GEOMETRY('EPSG:3857') with byte-perfect WKB and CRS
preserved at the destination, with no plan-level materialization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix error message for Postgres connection failure
first attempt to enable use_information_schema for ATTACH
…alog-load-race fix: catalog cache loading race
This is a follow-up to duckdb#476, it extends the `VARCHAR` fallback logic for types, that are defined in a schema that is different from the attached schema. Testing: new test added that covers inaccessible schema type. Ref: duckdb#497
…alogset-scan-uaf fix: prevent catalog scan use-after-free
…exes-constraint-type-assertion fix: avoid crash by setting default index constraint type
This PR extends the `SCHEMA` option for `ATTACH` allowing it additionally to take a list of schemas: ```sql ATTACH '...' AS p (TYPE POSTGRES, SCHEMA ['s1', 's2', 's3']) ``` Testing: new test added that covers visibility for multiple schemas. Closes: duckdb#498
…-return-type.patch
CONNECT on Redshift failed unless the user manually did SET pg_use_text_protocol=true, while ATTACH + USE worked. DuckDB rewrites a CONNECT passthrough statement into postgres_query() (PostgresCatalog::RemoteExecute). Its bind, PGQueryBind, never called PostgresScanFunction::PrepareBind - which is where every server capability decision is made, including the Redshift opt-out from the binary COPY protocol. As a result the CONNECT path always wrapped the query in COPY (...) TO STDOUT (FORMAT "binary"), which Redshift does not support. Instead of duplicating the Redshift check on each bind path, make the protocol a property of the attached database: - add a use_text_protocol ATTACH option, accepting TRUE / FALSE / 'auto' (the default), stored on the PostgresCatalog - PostgresCatalog::UseTextProtocol() is now the single place resolving the protocol for a database: an explicit option wins, 'auto' falls back to the pg_use_text_protocol setting or to Redshift auto-detection - PGQueryBind calls PrepareBind, so the CONNECT path resolves it as well Also resolve the protocol at the start of PrepareBind: it was set after the ctid scan and max_threads decisions that depend on it, which happened to be harmless only because Redshift reports major version 8.
Apply the text protocol opt-out on the postgres_query (CONNECT) path
Since duckdb/duckdb#23539: ```sql SELECT ARRAY['a', NULL] = ARRAY['a', NULL]; -- Postgres: t (true) -- DuckDB, post-#23539: NULL ``` so removing the NULL carrying value
…alog-cache-staleness fix: detect Postgres table catalog staleness
Add plain query binary reader
…pers Inline the binary COPY parser's length and bounds checks
The binary COPY parser sets the output chunk's cardinality after every row. That used to be cheap: DataChunk::SetCardinality only stored the count. Since DuckDB 2.0 vectors carry their own size, and the update to it (573bcfd) replaced the call with DataChunk::SetChildCardinality, which visits every vector of the chunk (and, through VectorBuffer::SetVectorSize, the children of struct and array vectors). A 2048-row chunk now paid for that 2048 times. The parser goes back to storing only the count per row, with SetCardinalityUnsafe (what the deprecated SetCardinality does), and the readers sync the vector sizes with SetChildCardinality once, before handing the chunk back. It can't happen at the end of the parser's loop: libpq hands over COPY data one row per PQgetCopyData call, so that would still be once per row. Nothing reads a vector's size while a chunk is being filled: values are written by row offset, bounded by the vector's capacity, and list child sizes are tracked separately by ListVector. Single-threaded user-mode cycles, median of 9 interleaved runs: table scan of one int column (10M rows) -14%, five columns -4%, a 12-type table (3M rows) -12%, postgres_query -13 to -14%, read_postgres_binary -18 to -25%. Wall clock is unchanged on scans bounded by the server and 16-19% lower on read_postgres_binary.
`ORDER` psuhdown optimizer was added on early stage of 2.0 development in incomplete state. It is intended to be replaced by the [RemotePushdown](duckdb/duckdb#23135) in upcoming updates. For 2.0.0 the plain `LIMIT` optimizer (the same as in 1.5.x) is restored instread. Fixes: duckdb#596, duckdb#598
…er-chunk Copy cardinality once per chunk
Preserve PostgreSQL base-10000 digits in a scientific-notation string and convert once with DuckDB floating-point parsing. Avoid repeated rounding and intermediate overflow from large display scales. Add exact comparisons against PostgreSQL float8 across binary COPY, binary query, and text query modes, including halfway values, subnormals, high-scale decimals, and arrays. Fixes duckdb#604.
Fix libpq Meson detection for clang-cl
Fix binary NUMERIC-to-DOUBLE rounding
This PR adds 2 new scalar functions:
- `postgres_pin_connection`(`<attached_database_name>`: `VARCHAR`) -> `UBIGINT`
- `postgres_close_pinned_connection`(`<attached_database_name>`: `VARCHAR`, `<connection_id>`: `UBIGINT`) -> BOOL
`postgres_pin_connection` acquires a connection from the connection pool
of the specified attached database and "pins" that connection returning
its ID as a handle. This handle can be specified to the
`postgres_query()` function to make it run on that specific connection.
This allows to make multiple subsequent `postgres_query()` and
`postgres_execute()` calls on the same connection, thus keeping
consistent server session state.
Pinned connection remains "active" and takes a slot in the connection
pool until it is closed by calling `postgres_close_pinned_connection`.
Usage example:
```sql
ATTACH '...' AS s (TYPE postgres);
SET VARIABLE conn = postgres_pin_connection('s');
CALL postgres_execute('s', 'CREATE TEMPORARY TABLE temp1(col1 INT)', connection=getvariable('conn'));
CALL postgres_execute('s', 'INSERT INTO temp1 values(42)', connection=getvariable('conn'));
FROM postgres_query('s', 'SELECT * FROM temp1', connection=getvariable('conn'));
----
00:42:00
SELECT postgres_close_pinned_connection('s', getvariable('conn'));
```
Testing: `query_pin_connection.test` added.
… conjunctions, and test both
…njunction Pushdown timestamptz conjunction
This PR sets the [application_name](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-APPLICATION-NAME) connection parameter to `DuckDB` when it is not specified by user in connection string or in `APPLICATION_NAME` or `URI` options of the `SECRET`, that used to `ATTACH` the Postgres database. Fixes: duckdb#613
Pass DuckDB label as a default application_name
…of in front of them
Put the default application_name into URI connection strings instead of in front of them
The query passed to `postgres_query()` function was being wrapped into a subquery with an explicit list of columns. This wrapping is necessary in some cases when a Postgres-side `CAST` is required for particular column. But it can be omitted in most of the usual cases. This PR enables the wrapping of the user query only when necessry. Ref: duckdblabs/duckdb-internal#11178
Queries that return columns with same names currently cannot be run
through `postgres_query()`, because DuckDB requires table functions to
return distinct column names:
```sql
postgres=# SELECT 41, 42;
?column? | ?column?
----------+----------
41 | 42
(1 row)
```
This PR adds deduplication to the `postgres_query()` column names, the
example above is changed to: `?column?`, `?column?_1`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v8 calls apt-key, which the image no longer ships. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
action-setup-postgres v9 no longer installs Postgres 14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs this repo's CI on
altertable-v2.0.0, the branch altertable-duckdb pins for the DuckDB 2.0 bump.🤖 Generated with Claude Code