Repository navigation
Conversation
…project#12123) ## Description of changes `upsert`'s `on` parameter now accepts an iterable of column names in addition to a single string, so upserting on a composite key (e.g. `tenant_id, entity_id`) is possible. The `WHEN MATCHED` update clause excludes all key columns, and the `ON` join condition ANDs an equality per key column. Single-column `on` behavior is unchanged. When there are no non-key columns to update, the `WHEN MATCHED` clause is now omitted instead of being emitted as a bare `UPDATE`. This fixes two existing problems that composite keys make much easier to hit: - A source with only the key columns used to raise (e.g. `syntax error at or near "WHEN"` on Postgres, `ORA-00971` on Oracle); it now inserts the missing rows and leaves matching rows unchanged. - On DuckDB, a bare `UPDATE` means a positional `UPDATE SET *`, so upserting a table made up only of key columns could silently swap values when the source columns were in a different order. ## Issues closed * Resolves ibis-project#11996 ## Test plan - [x] Parametrized `test_upsert_from_dataframe` with a multi-column case that includes a row matching on both keys (updated) and a row matching only `first_name` (inserted), so dropping a key from `ON` fails the test - [x] Added `test_upsert_on_all_columns` for a table where every column is a key and the source columns are in a different order - [x] Updated `test_upsert_from_memtable` so a source with only the `on` column succeeds instead of raising - [x] Added `test_upsert_empty_on_raises` for the `on=[]` edge case (raises a clear `IbisInputError` before anything runs on the backend) - [x] Verified the new tests fail without the fix and pass with it - [x] Inspected generated `MERGE` SQL directly; confirmed single-column `on` produces unchanged SQL - [x] Validated on BigQuery and Athena in ibis-project#12127 (Snowflake and Databricks not tested) --------- Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
- Update the Impala backend to use SQLGlot's own `INVERSE_TIME_MAPPING` and `INVERSE_TIME_TRIE` instead of manually inverting `TIME_MAPPING`. Manual inversion silently drops keys since tobymao/sqlglot#7773, breaking format strings like `%m`/`%d`. `INVERSE_TIME_MAPPING` already existed, so this is backwards compatible with older SQLGlot versions. - Upgrade the lock to SQLGlot 30.13.0, which contains the actual fix for ibis-project#12004 on Spark/Databricks. - ClickHouse's `arrayMap`/`arrayFilter` with an index now hit an arity cap in SQLGlot's generic expression classes; route through `self.f.anon` to bypass it (output SQL unchanged, backwards compatible with older SQLGlot versions). - MSSQL got its `##` global temp table prefix by writing one `#` and letting SQLGlot's tsql generator prepend the second. That double-prefixing was a SQLGlot bug, fixed in 30.9.0 by tobymao/sqlglot@a40dde9d3, after which our temp tables were created session-local and went invisible to the next pooled connection. Write `##` out in full and drop the `TemporaryProperty` (output SQL unchanged on both sides of that fix). - Athena bitwise shifts are now generated as `BITWISE_LEFT_SHIFT` and `BITWISE_RIGHT_SHIFT` (since SQLGlot 30.10.0), which Athena supports, so remove the `notyet` markers from the shift tests. - Add regression tests for single-digit month/day parsing, and drop the Spark legacy time-parser config, so the strict parser actually exercises the fix. Resolves ibis-project#12004
sqlglot 30.18 renamed `Drop`'s `this` argument to `tables`, and silently ignores the old keyword, so every `DROP` that ibis built lost its target (e.g., `DROP TABLE IF EXISTS` with no table name). Add a `Drop` helper to `_compat.py` that passes the target under whichever argument name the installed sqlglot accepts, and use it everywhere. Bump the locked sqlglot from 30.13.0 to 30.18.0, and fix the Postgres SQL that changed since 30.13.0: - Compile `MapContains` and `MapGet` to the `?` key existence operator via `JSONBContainsTopKey`. The `jsonb_contains` function now generates `JSONB_CONTAINS(m, 'k')`, which tests containment of a whole jsonb value. - Call `json_extract_path_text` as an anonymous function, because sqlglot otherwise rewrites it into `->> VARIADIC ARRAY[]::TEXT[]`, which is invalid. Closes ibis-project#12101 Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description of changes
Move `numpy`/`pandas`/`pyarrow`/`pyarrow-hotfix` into a `pyarrow`
dependency group and select it from the matrix, so they resolve into the
environment the tests actually run in and their versions come from
`uv.lock`. A dependency group rather than an extra keeps this out of the
published metadata.
Also aligns the group's `pandas` bound (`<4`) with the one every backend
extra already uses, so the matrix leg tests what users installing a
backend actually get.
## Issues closed
The `pyarrow: [true, false]` axis of the `test_core` matrix does
nothing; both variations run the same tests.
The install step runs `pip install numpy pandas pyarrow pyarrow-hotfix`.
The tests run under `uv run`, via `just ci-check`, which uses uv's own
environment. `pip` installs into the interpreter `actions/setup-python`
provides, so nothing that step installs is on the path when the tests
run. The step succeeds, and has no effect.
Two consequences:
1. Every module guarded on pyarrow skips in both legs:
```
ibis/backends/tests/test_dataframe_interchange.py
ibis/backends/tests/test_expr_caching.py
ibis/expr/datatypes/tests/test_core.py
ibis/formats/tests/test_pandas.py
ibis/formats/tests/test_pyarrow.py
```
2. The version was never pinned: the most recent run on `main` installed
pyarrow 25.0.0, while `uv.lock` pins 23.0.0.
## Verification
Before, [run
30753287253](https://github.com/ibis-project/ibis/actions/runs/30753287253)
on `main`, ubuntu-latest / python-3.10:
| variation | result |
| --- | --- |
| `pyarrow-true` | 4179 passed, 247 skipped, 3 xfailed |
| `pyarrow-false` | 4179 passed, 247 skipped, 3 xfailed |
After, [run
31230096112](https://github.com/ibis-project/ibis/actions/runs/31230096112)
on this branch, same job:
| variation | result |
| --- | --- |
| `pyarrow-true` | 4509 passed, 150 skipped, 4 xfailed |
| `pyarrow-false` | 4179 passed, 247 skipped, 3 xfailed |
All eight `test_core` jobs pass, on both operating systems and both
Python versions.
## Alternatives considered
**A published `pyarrow` extra that every backend extra reuses**, as in
an earlier revision of this branch
(1106e98). It fixes the bug equally
well and additionally lets external backend authors pin the same bounds
via `ibis-framework[pyarrow]`, which a dependency group cannot do. But
an extra is published metadata and a permanent commitment, so the name
has to hold up on its own — and `pyarrow` installs numpy and pandas too,
and would lose `pyarrow-hotfix` if ibis-project#11977 lands. That trade is worth its
own PR rather than riding along with a CI fix, so this PR takes the
narrower shape.
**`uv run --with`** does not work here. `uv run` invokes the project
environment's `pytest` entry point, and the project environment's copy
of a package shadows one layered into the ephemeral overlay, so the
override never takes effect. `--with` packages are not covered by the
lock file either.
**Extras or groups for version *combinations*** — relevant to ibis-project#11977,
which wants `none`/`latest`/`old` legs. This is not expressible in
`pyproject.toml` under either shape: uv's universal resolution collapses
compatible ranges to one version, so a second, old-pinned extra silently
downgrades the normal one; making the ranges disjoint to force a fork
fails to resolve, since every backend extra pulls the shared one; and
declaring `[tool.uv] conflicts` breaks the `uv sync --all-extras` calls
in the justfile and in `ibis-main.yml`. Version overrides have to happen
at install time instead, which composes fine with this PR:
```sh
uv sync --group tests --group pyarrow
uv pip install "numpy<2" "pyarrow==10.0.1" pyarrow-hotfix # old leg only
just ci-check "--no-sync" -m "'core or benchmark'"
```
The expression-level to_csv, to_parquet, to_parquet_dir and to_delta dispatch to the backend, so the keyword arguments they accept depend on it: DuckDB and PySpark override some of these with their own writers, and only the default implementation goes through PyArrow or deltalake. Describe kwargs as backend-specific, as to_json already does, and point to the backend's own method, which documents what it accepts. Resolves ibis-project#11769 Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
- to_json's parameter descriptions were copied from to_delta, and misspelled "backend-specific" - the output paths of the other writers were described as data sources - the PyArrow links after `kwargs` were bare URLs, which the numpydoc parser read as a parameter named `https`, adding an empty row to the parameter tables of every backend that inherits these methods Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
The `experimental`, `deprecated` and `backend_sensitive` decorators add their admonitions by assigning to `func.__doc__` at import time. That worked while the docs were collected by `pytkdocs`, which reads `__doc__`, but 7baea15 replaced it with griffe-based `mkdocstrings-python`, which parses source, and quartodoc does the same. Apart from the few `dynamic: true` entries, none of these admonitions have rendered since. The docs build does see the decorators, so the renderer now applies them to a stand-in function carrying the parsed docstring and uses the result. This also fixes two bugs from porting the admonitions to quarto in 9a3ff39 that never surfaced because nothing rendered: - `backend_sensitive` used `callout-info`, which is not a quarto callout type; it is now `callout-note`, as is the Snowflake blog post using it - the `why=` body was indented after the closing fence, which quarto renders as a code block; it now goes inside the fence Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
sma-2004
force-pushed
the
db2-backend-ibis-clean
branch
from
October 8, 2026 12:30
418f1ba to
de4284a
Compare
…to int/None (ibis-project#12122) ## Summary Some backends (PySpark over Spark Connect, per the linked issue) return array columns as numpy arrays. When an array contains a null, numpy upcasts it to `float64` with `NaN` standing in for null, since numpy has no native null-int representation. Pandas nullable-integer extension arrays instead yield `pd.NA` for missing elements. `PandasData.get_element_converter` had no converter for integer element types, so these raw floats/NaN/`pd.NA` values passed straight through instead of being normalized to the target `int64` element type. This adds `convert_Integer_element`, used for any integer dtype without a more specific converter, which normalizes `NaN`/`pd.NA`/`None` to `None` (via `pd.isna`) and casts everything else to `int`. ## Issues closed * Resolves ibis-project#11860 ## Test plan - [x] Added a regression test exercising `PandasData.convert_table` with the exact numpy float/NaN shapes produced by Spark Connect - [x] Added a second regression test for the `pd.NA` case (pandas nullable-integer arrays) - [x] Verified both tests fail without the fix and pass with it - [x] `pytest ibis/formats/` (189 passed) - [x] `pytest ibis/backends/tests/test_array.py -k duckdb` (182 passed, no regression)
The `mcr.microsoft.com/vscode/devcontainers/python:3.13` base image moved
to Debian trixie, which does not package `openjdk-17-jdk`. Every dev
container build has failed since then:
E: Unable to locate package openjdk-17-jdk
Reported in ibis-project#11715, where the fix was identified but never committed.
Take `default-jdk` rather than pinning a version, as suggested by
@JonAnCla in ibis-project#12029, so this does not break again the next time the base
image moves to a new Debian release.
That also keeps the JDK usable. It is only in the image for pyspark
(6af1dbf), we lock pyspark 4.0.1, and Spark 4.0 supports Java 17 and
21. `default-jdk` is openjdk-21 on trixie; openjdk-25 builds fine but
fails at runtime, since JDK 24 dropped the Security Manager that Hadoop's
UserGroupInformation relies on:
java.lang.UnsupportedOperationException: getSubject is not supported
Assisted-by: Claude Opus 5
The script is dead code: `devcontainer.json` defines only a `postStartCommand`, there is no `postCreateCommand`, and nothing else in the repo references the file. It is also stale. It reads `.tool.poetry.version` out of `pyproject.toml` to set `POETRY_DYNAMIC_VERSIONING_BYPASS`, but poetry was dropped in dad2acc in favor of uv and hatchling, so that lookup now yields `null`. The `pip install -e` it guards duplicates the `just sync duckdb` the Dockerfile already runs. Assisted-by: Claude Opus 5
## Description of changes `ibis/formats/tests/test_pyarrow.py` guards itself on `pyarrow.tests.strategies`. **conda-forge splits that into a separate `pyarrow-tests` package**, which neither conda environment file asks for — so the module-level `importorskip` fires and takes all 19 tests in the file with it, silently, since a skipped module reports nothing. `docs/contribute/01_environment.qmd` points conda, mamba and pixi users at `conda/environment.yml` (pixi via `pixi init --import conda/environment.yml`), so every contributor who set up the documented way has been losing that coverage. ## History Regression of ibis-project#7383, which reported this in October 2023 — back when it was a hard collection error: | PR | Effect | |---|---| | ibis-project#7390 | Closed ibis-project#7383 by adding `pyarrow-tests` to the conda **lock files** only. | | ibis-project#8242 | Removed the conda lock files entirely, taking the fix with them. | | ibis-project#8383 | Hand-added `pyarrow-tests` to the then hand-written `conda/environment-arm64.yml`. | | ibis-project#9552 | Rewrote the import as `pytest.importorskip("pyarrow.tests.strategies")`, converting the collection error into a silent skip. | | ibis-project#11878 | Converted both env files from hand-written YAML to `pyproject2conda` output plus manual fixups; `pyarrow-tests` did not make it onto the fixup list. (Mine.) | `pyproject2conda` derives conda dependencies from `pyproject.toml`, and `pyarrow-tests` is conda-only — it has no PyPI existence — so the generator structurally cannot emit it. It survives only by being on the manual fixup list that these files carry on top of generated output, alongside `channels`, `just`, `gcsfs`, the `trino-python-client` and `PyAthena` conda-forge name mappings, and the `pip:` block for packages conda-forge doesn't have. That list is recorded nowhere but the files themselves, under a `# This file is manually edited.` header. Two consequences. `conda/environment.yml` never carried the dependency at any point in its history, hand-written era included — only the arm64 file and the since-deleted lock files ever had it, so the documented x86 setup was never actually fixed. And since ibis-project#9552 the failure has emitted no signal at all, which is why it survived two years after someone reported the loud version of it within days. ## Testing Verified `pyarrow-tests` resolves alongside `pyarrow>=10.0.1` on conda-forge and provides the module: ```console $ pixi exec --spec 'python=3.12' --spec 'pyarrow>=10.0.1' --spec pyarrow-tests --spec pytest --spec hypothesis \ python -c "import pyarrow, pyarrow.tests.strategies as past; print(pyarrow.__version__, bool(past.schemas))" 25.0.0 True ``` Not an issue for PyPI wheels, which bundle `pyarrow/tests/`. ## Notes - A more durable fix would add `-d pyarrow-tests` to the `pyproject2conda` invocation recorded in each file's header, so the dependency survives the next regeneration (verified: `-d` emits the entry and the generated header records the flag). Left out here because the same argument applies to every other entry on the fixup list, and making regeneration lossless is a much larger change than restoring one line. - An earlier revision of this PR claimed PyArrow had stopped shipping `pyarrow/tests/` in its wheels as of 23.0.0, and worked around it in the test file. That was wrong — the wheel does ship it (89 `pyarrow/tests/*` entries, `strategies.py` included); the check had been contaminated by an ambient conda environment. The cause is narrower and this is the honest fix. - This does **not** make those tests run in CI — the core CI job installs no pyarrow at all, so the module skips one line earlier. Separate bug, fixed in ibis-project#12071. - The `importorskip` masking is arguably worth addressing on its own; cf. ibis-project#3480. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
The backend hierarchy table was stale: it listed the removed pandas backend, omitted supported backends, and used inaccurate terminology for several systems. - **Coverage** - Added Athena, Databricks, Exasol, Materialize, and SingleStoreDB. - Removed pandas. - **Terminology** - Corrected verified native hierarchy names: BigQuery datasets, PySpark catalogs/databases, and Oracle's `database`/`schema` levels. - Replaced hierarchy levels a backend does not have with `N/A`. - Removed the Druid datasource terms, which do not represent catalog/database hierarchy; Druid's database level is its SQL `schema`. Assisted-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
## Description of changes
`as_date`/`as_timestamp` silently returned a wrong result on Flink:
```pycon
>>> con.execute(ibis.literal("2021-01-02").as_date("%Y-%m-%d"))
datetime.date(5471, 9, 27)
```
The format string was going through sqlglot with `dialect=Flink`, which re-parses it as if it were already a Java pattern. Translate it to a Java pattern explicitly instead, the same way the Impala compiler does. Two `never` markers started XPASSing as a result and are dropped.
The `notyet(["flink"])` marker ibis-project#12021 added for this bug now XPASSes, so it is dropped too.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: hdm-db2-eco-system <Hdm-dev-persona-db2-eco-system@ibm.com>
…formatting cleanup
This test was added to main after the db2 branch was originally cut (commit 5f1dff0). DB2 has no STR_TO_DATE function and requires a FROM clause for scalar SELECT, so both param variants fail at runtime. Mark them notimpl with IbmDb2Error so CI skips rather than fails.
1.0.0 had sqlglot>=30.8.0,<30.10.0 which caps sqlglot for the entire project. 1.1.0 drops the upper bound. Set the minimum to 1.1.0 so the broken version can never be resolved.
sma-2004
force-pushed
the
db2-backend-ibis-clean
branch
from
October 10, 2026 13:27
9d96f26 to
fe0a281
Compare
DB2 has no STR_TO_DATE function so both as_date and as_timestamp with format string fail at runtime with SQL0440N.
…rmat DB2 supports as_timestamp with format (passes), but not as_date (STR_TO_DATE doesn't exist). Scope the notimpl marker to date only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Issues closed