Skip to content

test(integration): RS_ZonalStats and RS_ZonalStatsAll parity - #1233

Merged
james-willis merged 2 commits into
apache:mainfrom
james-willis:jw/parity-zonalstats
Sep 4, 2026
Merged

test(integration): RS_ZonalStats and RS_ZonalStatsAll parity#1233
james-willis merged 2 commits into
apache:mainfrom
james-willis:jw/parity-zonalstats

Conversation

@james-willis

@james-willis james-willis commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Parity coverage for RS_ZonalStats and RS_ZonalStatsAll, probe-first, one module per function. Branches off main, independent of the other open parity PRs.

What agrees (32 passing tests, every anchor exact)

The headline probe finding: every statistic matches numpy's exact double bit-for-bit on both engines — including mean, variance, and stddev — so there is no tolerance anywhere in the module. The shared conventions the anchors pin:

  • sample (ddof=1) variance and stddev;
  • even-count median averages the two middle values; odd-count is the exact middle element;
  • mode ties break toward the higher value (probed with planted 2-vs-2 ties);
  • avg/average/sd aliases accepted by both;
  • exclude_no_data defaults true (a planted in-roi nodata pixel drops from the count) and can be disabled;
  • all_touched selects a centre-free sliver that the default centre-in rule ignores;
  • a disjoint roi yields NULL under the default lenient behavior;
  • both engines refuse strict-disjoint (lenient=false), unknown statistics, and out-of-range bands (0 and beyond) — contrast the RS_BandNoDataValue getter, where SedonaDB returns NULL for out-of-range bands.

RS_ZonalStatsAll is compared field by field — s['field'] parses in both dialects — because the whole-struct schemas differ (SedonaDB types count as nullable Int64; Sedona Spark returns non-nullable Doubles throughout) and the harness stringifies scalar columns, not structs.

What diverges (3 xfails)

  • Band-less forms on a multi-band raster (both functions): SedonaDB requires the band argument (documented as deliberate in the kernel); Sedona Spark defaults to band 1.
  • CRS-carrying roi on a CRS-less raster: SedonaDB refuses ("geometry has a CRS but the raster does not"); Sedona Spark computes as if they matched.

Line and diagonal rasterization (ported from apache/sedona GH-3118)

The traversal-rewrite fixtures from RasterizationTests are ported verbatim (unit-pixel grids and WKTs kept as-is — the near-corner coordinates are float-tuned to them), expressed as RS_ZonalStats(count) over LINESTRING rois so the burned-cell set is what's compared. Every line runs forwards and reversed against one anchor, pinning GH-3118's direction independence along with parity; the bottom-up (positive scaleY) grid agrees under both flags; and the coastline multipolygon lands exactly on the 1738/1842 counts Sedona pins to GDAL/rasterio.

Probing these surfaced two new Spark line-roi divergences, xfail-cataloged with apache/sedona#3322:

  • all_touched is ignored for line geometries — the count never changes with the flag, where SedonaDB includes every touched cell, matching rasterio;
  • the default rule burns every traversed cell where GDAL burns only centre/diamond crossings — corner-crossing segments mask this because their traversal and diamond counts coincide, which is why GH-3118's own fixtures agree by default.

One corner-tangent case is anchored to the engines' shared 13 rather than this suite's rasterio wheel (which says 14): whether a cell merely touched at a lattice corner counts under all_touched is a GDAL-version-sensitive tie, noted in the test.

Verification

integration/spark-parity locally: the two modules run 41 passed, 7 xfailed (pyspark 4.0.4, Sedona 1.9.1); CI runs the parity lane on this branch.

Probed first: every statistic — including the float-accumulating mean,
variance, and stddev — matches numpy's exact double bit-for-bit on both
engines (sample ddof=1 variance/stddev; even-count median averages the
middle pair; mode ties break toward the higher value), so every anchor is
exact with no tolerance anywhere. Coverage: all nine statistics plus the
avg/average/sd aliases, band addressing, the band-less form on a
single-band raster, exclude_no_data on and off, all_touched, the lenient
NULL for a disjoint roi, and parity-on-refusal for strict-disjoint,
unknown statistics, and out-of-range bands. RS_ZonalStatsAll is compared
field by field (both dialects parse s['field']); its whole-struct schemas
differ (SedonaDB Int64 count vs Spark all-double) and the harness does
not stringify structs.

Two xfails: the band-less forms on a multi-band raster (SedonaDB requires
the band, Sedona Spark defaults to band 1 — a divergence the kernel
documents as deliberate) and a CRS-carrying roi on a CRS-less raster
(SedonaDB refuses; Sedona Spark computes as if they matched).
@github-actions
github-actions Bot requested a review from paleolimbot September 4, 2026 18:49
Ported from apache/sedona's GH-3118 traversal tests (verbatim grids and
WKTs — the near-corner coordinates are float-tuned to unit pixels) plus
the coastline multipolygon whose 1738/1842 counts Sedona pins to
GDAL/rasterio; both engines land exactly on them. Every line runs
forwards and reversed against the same anchor, pinning GH-3118's
direction independence along with parity, and the bottom-up (positive
scaleY) grid agrees under both flags.

Probing surfaced two Spark line-roi divergences, xfail-cataloged with
apache/sedona#3322: all_touched is ignored for line geometries (the
count never changes with the flag, where SedonaDB matches rasterio),
and the default rule burns every traversed cell where GDAL burns only
centre/diamond crossings — corner-crossing segments mask this because
their traversal and diamond counts coincide. One corner-tangent case is
anchored to the engines' shared 13 rather than this suite's rasterio
wheel (which reports 14 — a GDAL-version-sensitive corner-touch tie).
@james-willis
james-willis marked this pull request as ready for review September 4, 2026 19:53
@james-willis
james-willis merged commit d976cc6 into apache:main Sep 4, 2026
5 checks passed
@james-willis
james-willis deleted the jw/parity-zonalstats branch September 4, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants