fix(python/sedonadb): resolve GeoArrow scalars with their edge type, and common scalar values - #1229
Conversation
…and common scalar values The literal resolver rebuilt a GeoArrow scalar's type from its CRS alone, so a spherical (geography) scalar came back as planar geometry, and it registered only the WKB scalar class, so WKT and native-encoded scalars fell through to pa.array([obj]) and failed. Carry the edge type through and route every GeoArrow scalar class through the same WKB path. Also resolve values that pa.array([obj]) rejects or handles lossily: pandas.NA and numpy.ma.masked become NULL, pandas.NaT a timestamp NULL, pandas Timestamp and Timedelta keep their full resolution and time zone instead of being truncated to microseconds, NumPy datetime64/timedelta64 values in any unit convert at a lossless Arrow resolution (ambiguous or sub-nanosecond units are rejected the way pandas rejects them, with an overflow check), 0-d arrays resolve as their typed scalar, structured numpy.void values become structs, and null Arrow scalars of nested or extension types resolve through their typed one-element array. pandas 3 class names are registered alongside the pandas 2 ones. Closes apache#1214.
|
|
||
|
|
||
| def _lit_from_wkb_and_crs(wkb, crs): | ||
| def _lit_from_wkb_and_crs(wkb, crs, edge_type=None): |
There was a problem hiding this comment.
function name probably needs an update
There was a problem hiding this comment.
Done, it's _lit_from_wkb now (named for what it builds from rather than its argument list, since the edge type joined the CRS).
| # than an untyped NULL. | ||
| import pyarrow as pa | ||
|
|
||
| return pa.array([None], pa.timestamp("ns")) |
There was a problem hiding this comment.
how do we know it should be ns? I supposed it doesn't matter since the system can cast it to other TS types as needed in the query?
There was a problem hiding this comment.
Right on both counts. NaT itself carries no unit (it's a unit-less singleton), so some unit has to be picked, and nanoseconds is where pandas stores it and the finest one, so it can never be a lossy choice. And it does coerce: coalesce(us_col, lit(pd.NaT)) resolves to timestamp(µs) and coalesce(s_tz_col, lit(pd.NaT)) to timestamp(s, UTC), so the null takes the surrounding expression's unit and zone. Added that as a comment on the function.
… NaT unit The helper now takes an edge type as well as a CRS, so its name names the value it builds from rather than its arguments. Note why NaT resolves at nanoseconds: it carries no unit of its own, nanoseconds is where pandas stores it, and as a null it coerces to the unit and zone the surrounding expression needs.
2 participants
What changes
Closes #1214.
lit()resolves Python values through a class-name → handler table with apa.array([obj])fallback. Two gaps in the GeoArrow handling and a set of common scalar values that the fallback rejects (or handles lossily) came up while building thesedonadb-geopandasassignment surface (#1195), where they are currently worked around before the value reacheslit(). Handling them here benefits every SedonaDB user, and lets that wrapper code shrink to a pass-through later.GeoArrow scalars
pa.array([obj])and failed. Every GeoArrow scalar exposes its WKB, so one handler now serves them all, valid or null.Scalar values
pandas.NA,numpy.ma.maskedValueErrorpandas.NaTValueErrorpandas.Timestamp/Timedeltanumpy.datetime64/timedelta64in non-Arrow units (D,W,Y, …)ValueErrornumpy.ndarrayValueErrornumpy.voidValueErrorValueErrorpandas 3 renamed these classes (
pandas.api.typing.NAType,pandas.Timestamp, …), so both sets are registered, the way the table already handlesDataFrameandSeries. Existing paths (valid Arrow scalars, arrays with dimensions, shapely, GeoSeries, pandas frames) are unchanged.Testing
23 new tests in
tests/expr/test_literal.py, run with warnings promoted to errors against pandas 2.3 / NumPy 2.3 and pandas 3.0 / NumPy 2.5. The fullexpr/,test_dataframe.py, andtest_context.pysuites pass unchanged.