feat(python/sedonadb-geopandas): add column assignment by jiayuasu · Pull Request #1195 · apache/sedona-db

jiayuasu · 2026-08-27T04:22:45Z

What changes

This is the first slice of the column assignment / arithmetic / dissolve work drafted in #1184, which grew too large to review as one change. It is being split into four focused PRs — assignment, dissolve, arithmetic, temporal support — that build on each other, so each can be reviewed on its own terms.

This PR adds gdf[key] = value to the experimental sedonadb-geopandas package:

Same-frame Series assignment. A Series read from the same frame can be assigned back (renamed or replacing a column). A Series from a different frame, a bare SedonaDB expression, and array-likes are rejected with actionable messages: none of them records a usable row provenance, and each previously failed obscurely or would silently write wrong values. Because assignment rebinds the frame, a Series read before an assignment is stale and is rejected the same way.
Scalar broadcasting. Python scalars, NumPy scalars (including 0-d arrays), Arrow scalars, masked values, and pandas.NA broadcast to every row, via a shared scalar classifier/normalizer (is_scalar / normalize_scalar).
Geometry assignment with CRS bookkeeping. Assigning a geometry value keeps its type and CRS; a destination CRS is stamped only onto CRS-less geometry. A newly assigned geometry column becomes the active geometry only when the frame had none, and replacing the active geometry column with a non-geometry value clears the active geometry, matching GeoPandas.
Temporal scalars. NumPy datetime64/timedelta64 values are converted at a lossless Arrow unit (ambiguous or sub-nanosecond units are rejected the way pandas rejects them, with an overflow check); pandas Timestamp/Timedelta keep their nanoseconds, zone-aware timestamps keep their zone, and NaT assigns as a datetime missing value.
Series stay usable across column-adding assignments. A Series read from a frame remains valid while assignments only add columns, so one captured geometry can supply several derived columns; replacing a column or filtering invalidates earlier reads, which then raise instead of silently resolving to different values.
Raises the sedonadb floor to 0.4.1, the first released version with DataFrame.mutate, which assignment is built on.
Declares shapely>=2 as a dependency (geometry scalars are Shapely objects; Shapely 2 dropped the sequence protocol on multipart geometries).

Testing

118 tests (96 new, including direct tests of the temporal scalar normalization), run with warnings promoted to errors against both a source-built current main and the released sedonadb 0.4.1 in a clean environment (covering pandas 2.x and 3.x). Ruff lint/format clean.

Support gdf[key] = value for same-frame Series, broadcastable scalars, and geometry values. Assignment validates its input — a Series from another frame, a bare expression, and array-likes are rejected with actionable messages, since none records a usable row provenance — and geometry assignment preserves the active-geometry bookkeeping: a geometry column becomes active only when the frame had none, replacing the active geometry with a non-geometry clears it, and destination CRS is stamped only onto CRS-less geometry. The scalar surface accepts Python scalars, NumPy scalars (0-d arrays included), Arrow scalars, masked values, and pandas NA; NumPy and pandas temporal scalars are rejected for now because representing them faithfully needs dedicated unit and timezone handling, which arrives as its own change. Raises the sedonadb floor to 0.4.1 for DataFrame.mutate.

…lization gaps Read any geometry-typed column back as a GeoSeries rather than only the active one, so a freshly assigned geometry column supports .area and .buffer() immediately. Keep an explicitly cleared active geometry cleared through materialization by applying GeoPandas' own constructor rule instead of the engine heuristic. Inherit the destination CRS when a CRS-less same-frame geometry Series replaces a column that has one, mirroring the scalar path, and construct replacements for geography columns with the destination's spatial kind so they stay geography. Preserve NumPy scalar dtypes (and uint64 range) by rebuilding them as typed Arrow scalars instead of .item() promotion, classify Arrow-wrapped nulls and NaN as missing exactly like their bare payloads, broadcast typed-null nested Arrow scalars through one-element typed arrays, reject pandas NaT with the same temporal deferral as Timestamp and Timedelta (and disclose the deferral in the README), and recognize Shapely 1.x multipart geometries as single values before the sequence check.

…CRS, and scalar edge cases Preserve the no-active-geometry marker through materialization whatever the remaining columns are named: the GeoDataFrame constructor auto-activates a geometry column literally named 'geometry', and there is no public spelling for a frame with geometry columns but none active, so the marker is cleared directly. Apply the destination CRS unconditionally for missing and bare-Shapely geography replacements — the geography constructors synthesize CRS84, which the CRS-absence guard mistook for a CRS the value carried. Exempt np.void from the typed Arrow scalar conversion, which broke its previously working bytes broadcast. Classify the one-element-null Arrow array spelling of a typed-null nested scalar as missing explicitly, instead of through a pandas coincidence that a pandas-less install would lose.

…lls, and structured scalars Route a typed spatial null (an invalid GeoArrow scalar carrying its own kind and CRS) through the value path so its metadata survives, instead of synthesizing a destination-kind null stamped with the destination CRS; typed non-spatial nulls still mean missing geometry. Strip the CRS84 that geography constructors synthesize when the destination column is explicitly CRS-less, so replacements do not silently gain a CRS. Clear the no-active-geometry marker in place during materialization rather than reconstructing the frame, which let the GeoDataFrame constructor coerce an unrelated all-null column named geometry to geometry dtype. Broadcast structured NumPy scalars as typed Arrow structs preserving field names and dtypes (rejecting dtypes with no Arrow mapping) instead of flattening them to lossy lists, keeping the bytes fallback for plain voids only. The pandas-free nested-null classification now has a test that actually blocks the pandas import.

…s, honor GeoArrow metadata Strip a synthesized CRS with the value-preserving st_setsrid(0) instead of st_setcrs(NULL), which null-propagates its CRS argument and erased every row while leaving the schema looking correct; the regression test now asserts materialized values, not just the schema. Recognize GeoArrow-typed scalars from their extension name rather than resolving them — the scalar resolver drops the planar/spherical edge type and rejects non-WKB storage outright — and resolve WKB-typed scalars, valid and null alike, through their one-element-array spelling, which honors the complete extension metadata; a null of non-WKB storage is rebuilt from its own kind and CRS, and a valid one keeps the resolver's error. Handle structured masked records before the generic mask check, which itself raises on the mask dtype: masked fields become struct nulls and a fully masked record is missing. The no-pandas regression test now builds its Arrow inputs before blocking the import (PyArrow's lazy pandas shim otherwise fails first) and was mutation-checked.

…o every destination Recognize GeoArrow-typed scalars before the new-column fast path, so a spherical WKB scalar assigned to a fresh or non-geometry column keeps its geography kind (and activates a geometry-less frame with the right semantics) and non-WKB nulls resolve there too; destination CRS inheritance still applies only when replacing an existing geometry column, since a new destination has no metadata to inherit. Serialize a non-WKB GeoArrow null's own CRS with its canonical to_json() form — stringifying the wrapper produced StringCrs(...), which ST_SetCRS rejects as PROJJSON. Unwrap 0-d structured masked containers to their record form before the generic mask check, which itself raises on structured dtypes, covering direct and Literal-wrapped assignment for all three mask states.

…askedRecords recursion Rebuild a large-WKB GeoArrow scalar's type on Binary storage — keeping its CRS and edge metadata — before the one-element-array resolution, since SedonaDB's WKB importer rejects LargeBinary storage outright; the typed routing had regressed what previously normalized through the plain scalar path. Unwrap 0-d structured masked containers through the base MaskedArray view: MaskedRecords' own [()] returns another 0-d MaskedRecords and recursed until RecursionError, while the base view yields the record form for the field/mask normalizer.

paleolimbot

A few things worth handling here, but in general this looks great!

Unless the timezone unit stuff is truly untenable, probably worth handling it while you're here.

paleolimbot · 2026-09-01T02:43:06Z

+        from sedonadb.expr import Literal, lit
+
+        from sedonadb_geopandas._series import normalize_scalar


It probably makes more sense to have these be top level imports?

Done: sedonadb.expr, pyarrow, and the _series helpers are module-level now (shapely too, since it's a declared dependency as of this round), and the ImportError guards that only wrapped pyarrow are gone. numpy and pandas stay function-local because they're optional.

paleolimbot · 2026-09-01T21:02:56Z

+            # Any geometry-typed column reads back as a GeoSeries — not just
+            # the active one — so a freshly assigned geometry column supports
+            # .area and .buffer() immediately, as it does in GeoPandas.
+            if key == self._geometry_name or key in _geometry_column_names(self._df):


Do you need to check these both or is the second one sufficient?

The second one is sufficient. The active name is validated to be a geometry column at construction and cleared whenever the column stops being one, so it's just the schema check now.

paleolimbot · 2026-09-01T21:07:10Z

+                raise ValueError(
+                    "Cannot assign a Series that comes from a different "
+                    "DataFrame: there is no row alignment, so the result would "
+                    "be silently wrong. Note that assigning to this frame "
+                    "rebinds it, so a Series read before an earlier assignment "
+                    "is already stale; re-read it as gdf[...] and try again."
+                )


I can see how this might be annoying...you might want to assign a few different transformations of a geometry into the dataframe like perimeter, area, etc. I don't have a great solution...UUID that bumps on everything except a projection?

Agreed, and it turned out to be tractable without tracking column references inside expressions. A Series read from a frame now stays valid across assignments that only add columns: the rows and every column it could reference are unchanged, so its expression resolves by name to the same values. That covers g = gdf.geometry; gdf['area'] = g.area; gdf['len'] = g.length; .... Replacing a column or filtering still invalidates earlier reads, since a Series captured before gdf['v'] = ... would otherwise resolve to the new v. It's implemented as a list of ancestor frames on the GeoDataFrame rather than a token on the Series, so Series didn't need to change. README updated to say this.

paleolimbot · 2026-09-01T21:11:55Z

+                    # SedonaDB's WKB importer requires Binary storage; the
+                    # type is rebuilt on Binary with the same CRS and edge


We should probably fix to also accept LargeBinary. I ran into this issue in Iceberg (LargeBinary is what they return BLOBs as).

In general it would be great to know what needs to change in geoarrow-pyarrow or sedonadb to make this easier (it feels like it should be easier than it is).

Filed #1215 for the importer. For the broader question, #1214 collects everything the wrapper currently has to normalize before lit() can take it. The two that matter most here: _lit_from_geoarrow_scalar rebuilds the type as ga.wkb().with_crs(crs), which drops the edge type (spherical scalars come back planar), and only WkbScalar is registered, so WKT and native-point scalars fail in the pa.array([obj]) fallback. With those two fixed this whole GeoArrow branch collapses to ctx.lit(raw). The rebuild stays for now so the package keeps working on 0.4.1.

paleolimbot · 2026-09-01T21:15:32Z

+        elif strips_crs and _expr_crs(self._df, expr) is not None:
+            # SRID 0 means "no CRS" and keeps the value; st_setcrs(NULL)
+            # would null-propagate and erase every row.
+            expr = expr.funcs.st_setsrid(ctx.lit(0))


SetSRID to 0 is probably best (ST_SetCRS("0") is the totally ungraceful way to do this with setcrs)

Yes, SetSRID(0) is what's used here. It's the value-preserving spelling; ST_SetCRS(NULL) null-propagated and erased the rows, which is how this branch came to exist.

paleolimbot · 2026-09-01T21:16:59Z

+        # Shapely 1.x multipart geometries implement __len__ and __iter__;
+        # they are still single values. (Shapely 2 removed the sequence
+        # protocol, but the package floor does not require Shapely 2.)


You could probably require shapely 2 (I think it is fairly old at this point)

Done: shapely>=2 is now a declared dependency. It wasn't declared at all before and only came in through the geopandas extra. The 1.x multipart guard and its test are gone.

paleolimbot · 2026-09-01T21:24:05Z

+def normalize_scalar(value):
+    """Normalize an accepted scalar into something a literal can hold.
+
+    Passing the classifier is not the same as being constructible: a 0-d NumPy
+    array is a scalar but `lit()` cannot take it, and `pandas.NA` is a missing
+    sentinel `lit()` does not recognize. Unwrap the former to its Python value
+    and convert missing sentinels to `None` (SQL null). Callers apply this only
+    after `is_scalar` has accepted the value.


Can we update lit() to handle some of these values to make it more useful for non-geopandas? I know that is slightly tricky because you can't import pandas there (but special casing the class names like we do for some other literal types might work).

Agreed, that's #1214. It lists each case with today's behavior (pd.NA and NaT unrecognized, Timestamp/Timedelta silently truncated to microseconds, numpy day/week/year units rejected, np.ma.masked, 0-d arrays, structured voids, typed-null nested scalars) and notes that the class-name table already handles the pandas cases without importing pandas. Each one that lands upstream lets normalize_scalar shrink to a pass-through.

paleolimbot · 2026-09-01T21:26:07Z

+    # .item() promoted np.int8/np.float32 to int64/float64 columns and
+    # overflowed np.uint64 past int64, which the engine supports natively.
+    import numpy as np


These and other can be top level imports (we always test with numpy/pyarrow)

Done: numpy, pyarrow, pandas, shapely, geoarrow and lit are module-level in the test file now.

…s across column additions Handle NumPy and pandas temporal scalars instead of deferring them: a new _temporal module chooses a lossless Arrow unit for NumPy datetimes and timedeltas (rejecting ambiguous or sub-nanosecond units the way pandas does, with an overflow check), routes pandas Timestamp and Timedelta through their numpy form so nanoseconds survive, rebuilds a zone-aware Timestamp at nanosecond ticks with its zone, assigns NaT as a datetime missing value, and treats an Arrow temporal scalar holding the INT64_MIN sentinel as missing. Let a Series read from a frame stay valid across assignments that only add columns — so one captured geometry can supply several derived columns — while replacing a column or filtering still invalidates earlier reads; the frame tracks the ancestor frames whose columns it contains unchanged. Require Shapely 2 and drop the Shapely 1.x multipart guard. Read any geometry-typed column as a GeoSeries by schema alone (the active name is always one). Hoist the module-level imports the packages' declared dependencies allow, in the package and in the tests.

jiayuasu · 2026-09-02T07:50:41Z

Thanks for the review. Pushed 888a8dbd9 with everything addressed; the decisions in one place:

Temporal scalars are handled here rather than deferred. A new _temporal module picks a lossless Arrow unit for numpy.datetime64/timedelta64 (rejecting ambiguous or sub-nanosecond units the way pandas does, with an overflow check), keeps nanoseconds for pandas.Timestamp/Timedelta (including zone-aware timestamps), and assigns NaT as a datetime missing value. The "temporarily rejected" note is gone from the README. Duration arithmetic and temporal group keys come with the arithmetic and dissolve follow-ups.
A Series now survives assignments that only add columns, so one captured geometry can supply several derived columns. Replacing a column or filtering still invalidates earlier reads.
shapely>=2 is a declared dependency; the Shapely 1.x compatibility code is gone.
Upstream: lit() should resolve GeoArrow scalars with their edge type, and common missing/temporal scalars #1214 (what lit() could resolve on its own: GeoArrow scalars with their edge type, and the missing/temporal scalar cases) and WKB import rejects LargeBinary storage #1215 (WKB import rejecting LargeBinary storage). The wrapper keeps its workarounds until those land so it stays usable on 0.4.1.

92 tests, run with warnings promoted to errors against both current main and the released 0.4.1.

paleolimbot

Thank you!

I didn't check test coverage here...the temporal stuff in general might benefit from some tests of its internals if they aren't already tested.

paleolimbot · 2026-09-04T21:41:51Z

+    assert gdf._geometry_name is None
+
+
+# -- regressions from review ------------------------------------------------


I think you can remove these headers

Done, all three are gone.

paleolimbot · 2026-09-04T21:43:54Z

+@pytest.fixture
+def points():
+    """A small frame in a projected CRS, shared across the tests."""
+    return gpd.GeoDataFrame(
+        {"name": ["A", "B", "C"], "v": [1, 2, 3]},
+        geometry=gpd.GeoSeries.from_wkt(["POINT (0 0)", "POINT (5 5)", "POINT (9 9)"]),
+        crs="EPSG:3857",
+    )


I know it sounds dumb, but these tests are better off inlining this (or whatever subset of it they need). Saves both humans and LLMs a lot of tokens and scrolling 🙂

Not dumb at all. Done: each test that used it now builds the frame inline (the same three points, via points_from_xy, so nothing changed in what the tests assert). It also turned up one dissolve test that took the fixture and never used it. cities and con_free_geom_frame predate this PR, so I left those alone.

…al internals directly Build the three-point frame inside each test that uses it instead of through a fixture, so a test reads without scrolling to its setup, and drop the section headers. Add direct tests of the temporal scalar normalization — the unit table, zone handling, the missing-value forms, and the error classes — alongside the assignment-level coverage.

jiayuasu · 2026-09-05T06:29:09Z

Thanks! On the temporal internals: the compatibility tests exercise them end to end (assignment, then materialization), but nothing pinned the helper itself, so 0b14e05a5 adds tests/test_temporal.py testing normalize_temporal_scalar and nat_scalar directly. It covers the unit table (Arrow-native units kept; week/day/hour/minute and datetime year/month converted exactly to seconds, including values outside the nanosecond range; exact sub-nanosecond values narrowed to nanoseconds), the zone-aware Timestamp path, every missing-value form (numpy NaT, pd.NaT, and the INT64_MIN sentinel as duration and timestamp), Arrow scalar pass-through, and the error classes for ambiguous units, lossy values, and overflow. That last one runs under both NumPy 2.3 and 2.5, which differ only in which layer raises the OverflowError.

jiayuasu mentioned this pull request Aug 27, 2026

feat(python/sedonadb-geopandas): add column assignment, arithmetic, and dissolve #1184

Closed

github-actions Bot requested a review from prantogg August 27, 2026 04:23

jiayuasu added 6 commits August 26, 2026 22:05

paleolimbot reviewed Sep 1, 2026

View reviewed changes

This was referenced Sep 2, 2026

lit() should resolve GeoArrow scalars with their edge type, and common missing/temporal scalars #1214

Closed

WKB import rejects LargeBinary storage #1215

Open

jiayuasu mentioned this pull request Sep 3, 2026

fix(python/sedonadb): resolve GeoArrow scalars with their edge type, and common scalar values #1229

Merged

paleolimbot approved these changes Sep 4, 2026

View reviewed changes

		from sedonadb.expr import Literal, lit

		from sedonadb_geopandas._series import normalize_scalar

		# SedonaDB's WKB importer requires Binary storage; the
		# type is rebuilt on Binary with the same CRS and edge

		assert gdf._geometry_name is None


		# -- regressions from review ------------------------------------------------

Uh oh!

Conversation

jiayuasu commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Testing

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiayuasu commented Sep 2, 2026

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiayuasu commented Sep 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jiayuasu commented Aug 27, 2026 •

edited

Loading