Skip to content

fix(rust): aggregate size() audit — count all per-group heap in ItemCrs, union/intersection, and envelope accumulators - #1227

Merged
james-willis merged 3 commits into
apache:mainfrom
james-willis:jw/agg-size-audit
Sep 4, 2026
Merged

fix(rust): aggregate size() audit — count all per-group heap in ItemCrs, union/intersection, and envelope accumulators#1227
james-willis merged 3 commits into
apache:mainfrom
james-willis:jw/agg-size-audit

Conversation

@james-willis

@james-willis james-willis commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #1225: after fixing ST_Collect_Agg's memory accounting, we audited every other Accumulator/GroupsAccumulator size() implementation in the repo (enumerated via the trait-required fn size(&self) -> usize, cross-checked with a repo-wide sweep for per-group set/collection state). Three more accounting gaps turned up, each small; this PR fixes all three as separate commits. No behavior changes — only what gets reported to the memory pool.

1. ItemCrsAccumulator (sedona-expr)

size() returned inner.size() + size_of::<ItemCrsAccumulator>(), omitting the per-group crs: Option<String> heap. Since this wrapper multiplies across every aggregate over item-CRS arguments, a high-cardinality GROUP BY leaked one uncounted String allocation per group. Fix adds the string's capacity.

2. ST_Union_Agg / ST_Intersection_Agg (sedona-geo)

These accumulators hold a growing MultiPolygon (the running union/intersection), yet size() counted only the raw coordinate bytes inside it — and by ring .len(), not .capacity(). Every container that holds those coordinates together went uncounted: the MultiPolygon's own Vec<Polygon> buffer, each Polygon's interior-rings Vec<LineString> buffer, the per-ring LineString headers, and any capacity slack in the coordinate vectors.

For the workload these accumulators exist to serve — dissolving many small polygons into one accumulating MultiPolygon — the Vec<Polygon> buffer alone scales with the polygon count (each geo::Polygon is a ~48-byte struct), so on small features the uncounted structural overhead is comparable to the coordinate bytes that were counted: roughly a 40–50% under-report that grows as the union does.

The fix replaces the hand-rolled walk with a shared geometry_heap_size() (new crate-private geometry_mem module, with tests) that charges for every owned buffer — coordinates by capacity, the polygon and ring vectors, and recursion through nested geometries. As a side benefit the new walk is variant-exhaustive, so it can no longer silently report zero if the held geometry is ever something other than a MultiPolygon; that path isn't reachable today (the accumulators maintain a MultiPolygon-or-None invariant), but the walk is robust either way.

3. BoundsGroupsAccumulator2D (sedona-functions)

The groups accumulator summed per-bounder mem_used() over the live bounders, missing the bounders: Vec<T> doubling slack (capacity beyond len). Smallest of the three — added the slack term.

Not in scope

ST_ConvexHull_Agg and ST_Analyze_Agg were audited and are already honest (capacity-based counting and the u32 bitset respectively). The larger ST_Collect_Agg follow-ups discussed in #1225 (native GroupsAccumulator, DataFusion-side adapter accounting) remain separate work.

@github-actions
github-actions Bot requested a review from paleolimbot September 2, 2026 23:27
…n size()

The previous size() only counted coordinates, and only when the held
geometry was the MultiPolygon variant: any other variant contributed
zero, and container overhead (the polygon buffer, per-ring LineString
headers, interior-ring buffers) was never counted. Replace the
hand-rolled walk with a variant-exhaustive geometry_heap_size() shared
by both accumulators. The MultiPolygon-only invariant these
accumulators maintain still holds; the total walk makes size() robust
to future changes rather than silently reporting zero.
@james-willis
james-willis marked this pull request as ready for review September 3, 2026 22:46

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

There is also a place in sedona-spatial-join where I believe we approximate or ignore the memory used by geo geometries that might be worth updating here. That may be more performance sensitive because we're estimating the usage of a Vec of geometries rather than just one.

@james-willis
james-willis merged commit c88bd72 into apache:main Sep 4, 2026
17 checks passed
@james-willis
james-willis deleted the jw/agg-size-audit branch September 4, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants