test(integration): RS_ZonalStats and RS_ZonalStatsAll parity - #1233
Merged
Conversation
Probed first: every statistic — including the float-accumulating mean, variance, and stddev — matches numpy's exact double bit-for-bit on both engines (sample ddof=1 variance/stddev; even-count median averages the middle pair; mode ties break toward the higher value), so every anchor is exact with no tolerance anywhere. Coverage: all nine statistics plus the avg/average/sd aliases, band addressing, the band-less form on a single-band raster, exclude_no_data on and off, all_touched, the lenient NULL for a disjoint roi, and parity-on-refusal for strict-disjoint, unknown statistics, and out-of-range bands. RS_ZonalStatsAll is compared field by field (both dialects parse s['field']); its whole-struct schemas differ (SedonaDB Int64 count vs Spark all-double) and the harness does not stringify structs. Two xfails: the band-less forms on a multi-band raster (SedonaDB requires the band, Sedona Spark defaults to band 1 — a divergence the kernel documents as deliberate) and a CRS-carrying roi on a CRS-less raster (SedonaDB refuses; Sedona Spark computes as if they matched).
Ported from apache/sedona's GH-3118 traversal tests (verbatim grids and WKTs — the near-corner coordinates are float-tuned to unit pixels) plus the coastline multipolygon whose 1738/1842 counts Sedona pins to GDAL/rasterio; both engines land exactly on them. Every line runs forwards and reversed against the same anchor, pinning GH-3118's direction independence along with parity, and the bottom-up (positive scaleY) grid agrees under both flags. Probing surfaced two Spark line-roi divergences, xfail-cataloged with apache/sedona#3322: all_touched is ignored for line geometries (the count never changes with the flag, where SedonaDB matches rasterio), and the default rule burns every traversed cell where GDAL burns only centre/diamond crossings — corner-crossing segments mask this because their traversal and diamond counts coincide. One corner-tangent case is anchored to the engines' shared 13 rather than this suite's rasterio wheel (which reports 14 — a GDAL-version-sensitive corner-touch tie).
james-willis
marked this pull request as ready for review
September 4, 2026 19:53
paleolimbot
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parity coverage for
RS_ZonalStatsandRS_ZonalStatsAll, probe-first, one module per function. Branches offmain, independent of the other open parity PRs.What agrees (32 passing tests, every anchor exact)
The headline probe finding: every statistic matches numpy's exact double bit-for-bit on both engines — including mean, variance, and stddev — so there is no tolerance anywhere in the module. The shared conventions the anchors pin:
avg/average/sdaliases accepted by both;exclude_no_datadefaults true (a planted in-roi nodata pixel drops from the count) and can be disabled;all_touchedselects a centre-free sliver that the default centre-in rule ignores;lenient=false), unknown statistics, and out-of-range bands (0 and beyond) — contrast theRS_BandNoDataValuegetter, where SedonaDB returns NULL for out-of-range bands.RS_ZonalStatsAllis compared field by field —s['field']parses in both dialects — because the whole-struct schemas differ (SedonaDB typescountas nullable Int64; Sedona Spark returns non-nullable Doubles throughout) and the harness stringifies scalar columns, not structs.What diverges (3 xfails)
Line and diagonal rasterization (ported from apache/sedona GH-3118)
The traversal-rewrite fixtures from
RasterizationTestsare ported verbatim (unit-pixel grids and WKTs kept as-is — the near-corner coordinates are float-tuned to them), expressed asRS_ZonalStats(count)over LINESTRING rois so the burned-cell set is what's compared. Every line runs forwards and reversed against one anchor, pinning GH-3118's direction independence along with parity; the bottom-up (positive scaleY) grid agrees under both flags; and the coastline multipolygon lands exactly on the 1738/1842 counts Sedona pins to GDAL/rasterio.Probing these surfaced two new Spark line-roi divergences, xfail-cataloged with apache/sedona#3322:
all_touchedis ignored for line geometries — the count never changes with the flag, where SedonaDB includes every touched cell, matching rasterio;One corner-tangent case is anchored to the engines' shared 13 rather than this suite's rasterio wheel (which says 14): whether a cell merely touched at a lattice corner counts under all_touched is a GDAL-version-sensitive tie, noted in the test.
Verification
integration/spark-paritylocally: the two modules run41 passed, 7 xfailed(pyspark 4.0.4, Sedona 1.9.1); CI runs the parity lane on this branch.