feat(python/sedonadb-geopandas): add column assignment, arithmetic, and dissolve - #1184
Closed
jiayuasu wants to merge 8 commits into
Closed
feat(python/sedonadb-geopandas): add column assignment, arithmetic, and dissolve#1184jiayuasu wants to merge 8 commits into
jiayuasu wants to merge 8 commits into
Conversation
…nd dissolve The unblocked half of the sjoin/dissolve slice, split out so it can land while the spatial-join engine issues (apache#1165, apache#1166) are being fixed; sjoin follows in its own PR. - __setitem__: assign a Series from the same frame, or a scalar (a geometry included). Frame provenance is enforced — a Series from another frame, a bare expression (which records no origin), and multi-element values are rejected with focused errors. Replacing the active geometry with a non-geometry clears it; giving a geometry to a frame without one activates it; a geometry scalar or missing sentinel (None, NaN, pandas.NA) keeps the column's type and CRS, and a literal that carries its own CRS keeps that instead of being relabeled. - Arithmetic on Series (+ - * / and reflected forms), with `/` as true division: integer-typed expressions are cast to double (decided from the projected schema) so SQL integer division does not truncate, while decimals and durations are left untouched. NumPy ufunc dispatch is opted out of so an array operand is rejected whole rather than broadcast element-by-element. - dissolve(by, aggfunc="first", dropna=True): unions each group's geometry via collect + unary union, which handles every geometry type (ST_Union_Agg only initializes for polygonal input, apache#1093); dropna covers IEEE NaN as well as SQL null, including dictionary-encoded float keys; an all-null group yields an empty geometry collection with the source CRS, matching GeoPandas. - One scalar classification shared by operators and assignment, with normalization (0-d arrays unwrap, pandas.NA becomes SQL null). - Boolean indexing rejects masks from another frame, including masks captured before an assignment rebound this one. 73 tests, asserted against GeoPandas where a comparison exists.
… and honor released floors Nine issues from review. Division resolves both operands' logical types before deciding on the double cast: a dictionary<int64> column is unwrapped (it used to skip the cast and truncate), and an integer divided by a Decimal stays in decimal arithmetic instead of being forced through double. The cast now applies only to integer/integer division. Durations support * and / by numeric scalars, matching pandas: the value goes through int64 ticks and back to the column's own duration type. Reusing the source type rather than assuming nanoseconds matters — verified against released 0.4.1, which ingests durations as microseconds, where a hard-coded ns cast silently scaled results by 1000. Scalar handling: NumPy temporal scalars are no longer flattened to integer ticks (datetime64 passes through, timedelta64 converts via pandas); a masked value normalizes to missing rather than exposing its hidden payload; and Arrow scalars are classified as scalars despite implementing __len__, so a ListScalar or StructScalar broadcasts. Geometry-ness in assignment is decided from the resolved literal's schema rather than duck-typing for __geo_interface__, so a CRS-less GeoArrow WKB scalar now inherits the column's CRS instead of clearing it. An explicitly deactivated geometry column (geometry=None) stays inactive across a reassignment of an existing geometry column; activation now requires the assignment to have created a new geometry column. dissolve(by=[]) raises "No group keys passed!" as GeoPandas does, instead of silently collapsing the frame; by=None still dissolves all. Categorical grouping is documented as observed-only, with a test pinning the divergence. Dependency floors drop to sedonadb>=0.4.1 / sedonadb-expr>=0.4.0: mutate shipped in the released 0.4.1 and expr is unchanged since 0.4.0, so the prerelease floor was excluding a compatible release. The whole suite passes against released 0.4.1 and against current main. Also removes two tests that duplicated newer parametrized coverage.
…ithmetic exact Six issues from review, two of them silent data corruption. NumPy temporal scalars are now rebuilt as typed Arrow scalars instead of being routed through .item() or pandas: a 0-D nanosecond datetime array was stored as raw integer ticks, a day-unit timedelta materialized as a zero duration, day-unit datetimes failed literal construction, and temporal NaT raised. All now materialize as the correct timestamp/duration values, and NaT becomes a typed null of the right family. The tests assert materialization end to end rather than helper equality, which is what let the zeroed duration hide. Duration division by an integral operand uses exact integer tick division; the float64 path rounded 2**53 + 1 ticks and could fail on cast-back. Fractional operands keep floating arithmetic. Division by zero and non-finite operands yield typed NaT, as a pandas Series does, instead of failing the int64 cast. Type decisions look through operand wrappers: series / lit(2) and series / pa.scalar(2) are integer division exactly as series / 2 is (they used to truncate), and duration * lit(2) is accepted (it used to be rejected as non-numeric). One shared resolver unwraps Literal and Arrow-scalar payloads for both decisions. Corrects the dependency comment: sedonadb-expr's generated surface has changed since 0.4.0; what is true is that the APIs this wrapper calls are available in 0.4.0. Verified against released sedonadb 0.4.1 and against current main.
Four issues from review, one of them silent corruption centuries wide.
Temporal conversion no longer forces nanoseconds. The ns range covers only
1677-2262, so an unchecked astype silently wrapped coarse-unit values —
np.datetime64("2500-01-01", "D") materialized as 1915-06-14. Arrow-native
resolutions (s, ms, us, ns) are kept exactly; whole-second units (weeks, days,
hours, minutes — plus calendar year/month positions for datetimes) convert
exactly to seconds; ambiguous timedelta months/years and sub-nanosecond units
are rejected as pandas rejects them; and a round-trip guard turns any remaining
overflow into an error rather than a wrong value.
Division of a duration by infinity is zero for valid rows with source nulls
preserved — computed as ticks * 0 — matching pandas, instead of the blanket
typed-NaT broadcast that the non-finite handling introduced. Division by zero
or NaN, and multiplication by non-finite values, still yield NaT.
The numeric resolver looks through one-element containers: lit(pa.array([2])),
lit(pd.Series([2])), and one-cell frames are single-value literals to SedonaDB,
so they are resolved via the literal's Arrow value rather than judged by their
raw Python payload. Integer division by them no longer truncates and duration
arithmetic accepts them.
Test NaT values carry explicit units, ahead of NumPy deprecating the generic
form.
Verified against released sedonadb 0.4.1 and against current main.
…s, and overflow Accept sub-nanosecond datetimes that convert exactly to nanoseconds (GeoPandas accepts these; lossy values are still rejected rather than silently truncated). Raise ValueError like pandas for ambiguous or unsupported timedelta units. Unwrap zero-dimensional object arrays before temporal normalization so a wrapped numpy temporal is handled. Let the literal resolver's precise errors propagate instead of replacing them with a generic type error, and validate multi-value literals explicitly. Gate integral duration multiplication so int64 tick overflow produces null instead of silently wrapping (pandas 3 raises; a lazy expression cannot raise per row), and rewrite the coarse-unit regression test against independent second-resolution constants so it fails against the implementation it guards.
Raise OverflowError, as pandas does, for an integer operand that itself exceeds the signed 64-bit tick range instead of failing literal construction with an unrelated error. Route integral-valued floats through the exact integer path so identities like '* 1.0' stay exact at every tick, and range-gate non-integral float results before the cast back to ticks so out-of-range rows become null instead of aborting the whole query. pandas clamps finite positive float overflow to Timedelta.max while negative overflow lands on the NaT sentinel; that asymmetric casting artifact is deliberately not copied — both directions are missing values, and the README says so. Also test that an exactly representable sub-nanosecond timedelta (1000 ps) is still rejected, since the rejection is per unit rather than per value.
…d keep float parity Nullify INT64_MIN duration ticks where ticks are derived: the value is a representable Arrow tick but the pandas missing sentinel, and treating it as data aborted division by -1 on arithmetic overflow, let negation wrap back onto the sentinel where a chained multiply resurrected it as zero, and divided it into real-looking results. Restrict the exact integer path to the float identities 1.0 and -1.0 so non-identity floats keep pandas' float64 semantics above 2**53 instead of silently improving on them. Report an oversized plain-integer Literal payload as OverflowError before the Arrow resolver fails on it with a misleading message. Strengthen the negative float overflow test with a value that genuinely needs the lower gate plus the exact boundary the gate must keep, and version-qualify the README's description of pandas overflow behavior.
…he column boundary Nullifying INT64_MIN only inside multiplication and division left addition, subtraction, and comparisons treating the pandas NaT sentinel as data: self-subtraction returned zero, adding a tick produced a finite Timedelta.min, subtracting one aborted the query on arithmetic overflow, and equality matched, so filters retained rows that materialize as NaT. Sanitize duration and timestamp columns where they are read instead, so every operator inherits the invariant, and treat a sentinel-valued Arrow temporal scalar operand the same way. Timestamps get the identical handling because numpy-backed pandas stores datetime NaT as INT64_MIN too. Along the way the requested regression tests exposed that pandas Timestamp and Timedelta scalars resolved to microsecond literals — assignment silently zeroed nanoseconds and duration arithmetic lost them behind an interval coercion — so they now route through their numpy form and its lossless unit handling.
Member
Author
|
Closing this in favor of a split into focused PRs, since the combined change grew too large to review well. The first slice (column assignment) is #1195; dissolve, arithmetic, and temporal support follow as their predecessors merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The unblocked half of #1142, split out so it can move forward while the spatial-join engine issues are being fixed.
sjoinstays in #1142, which is blocked on #1165 (ST_Touches/ST_Withinboundary misclassification) and #1166 (empty preserved side fails in the spatial join executor); nothing here touches those code paths.What is added
__setitem__— assign aSeriesfrom the same frame, or a scalar (a geometry included). Frame provenance is enforced: aSeriesfrom another frame, a bare expression (which records no origin), and multi-element values are rejected with focused errors. Replacing the active geometry with a non-geometry clears it; assigning geometry to a frame without one activates it; a geometry scalar or missing sentinel (None, NaN,pandas.NA) keeps the column's type and CRS, and a literal carrying its own CRS keeps that rather than being relabeled.Series—+,-,*,/and the reflected forms./is true division as in pandas: integer-typed expressions are cast to double (decided from the projected schema, a plan build) so SQL integer division does not truncate, while decimals stay exact and durations stay durations.//is deliberately not implemented, since SQL division truncates toward zero where Python floors.Seriesopts out of NumPy ufunc dispatch so an array operand is rejected whole instead of being broadcast element-by-element into an object array.dissolve(by, aggfunc="first", dropna=True)— unions each group's geometry via collect + unary union, which handles every geometry type (ST_Union_Aggonly initializes for polygonal input, ST_Union_Agg returns NULL for point inputs #1093).dropnacovers IEEE NaN as well as SQL null, including dictionary-encoded float keys. A group whose geometries are all null yields an empty geometry collection carrying the source CRS, matching GeoPandas.pandas.NAbecomes SQL null, NaN stays a float value.Documented differences from GeoPandas
aggfunc="first"is an unordered aggregate: it returns some value from the group and does not skip missing values.Tests
73 tests, asserted against GeoPandas where a comparison exists (dissolve geometry equality across point/line/polygon groups, aggregation values, CRS propagation, division exactness for decimals, and the full assignment matrix including every missing-value sentinel).
ruff formatandruff checkclean.