Skip to content

perf: Theta, Tuple and AoD aggregate states in DuckDB's arena, so they count toward memory_limit and spill - #28

Merged
rustyconover merged 1 commit into
v2.0from
arena/theta
Oct 28, 2026
Merged

rustyconover merged 1 commit into
v2.0from
arena/theta

Conversation

@rustyconover

Copy link
Copy Markdown
Collaborator

This moves all seven Theta, Tuple and AoD aggregates onto arena states with no destructor, which export and import their state as BLOBs. That's the same model #27 introduced for KLL and HLL. The shared code is in src/include/datasketches_theta_arena.hpp.

20M rows, 2M groups, 1 GB limit, 8 threads before after
datasketch_theta(v) 22.8 GB, 22.4 s 1.01 GB, 16.0 s
datasketch_theta(s) (VARCHAR) 22.8 GB, 21.6 s 1.08 GB, 14.3 s
datasketch_tuple(v, d) 43.4 GB, 33.9 s 1.08 GB, 16.1 s
datasketch_aod(v, [d, d]) out of memory at a 60 GB cap 1.09 GB, 18.3 s

How the states work

  • Builds: keep their first 32 raw rows (up to 1 KB) and replay them, in order, into an update sketch when more arrive.
  • Unions: keep up to min(32, k) distinct keys with their summaries already combined. That's what the library's union would hold anyway.
  • Intersections: keep their current result and overwrite it in place. The library's intersection allocates a new table on every row.
  • Temporaries: input sketches, results and serialized bytes go to a per-thread scratch arena that is reset after each use. Every sketch, including AoD's value arrays, allocates in arena mode.
  • Spilling: buffered states export as raw rows or keys, so importing them doesn't inflate them.

Tests

  • Results are byte-identical to the old code across 133 compared queries: 1, 4 and 8 threads, group sizes from 3 to 200k, seed/p/mode options, merges and intersections.
  • New test/sql/datasketch_spill_theta.test runs under 128 MB with 4 threads. Every result's hash must equal the same query at 4 GB.
  • The full suite passes (33 test cases, 1987 assertions).

Known limits

  • finalize(agg(...) EXPORT_STATE) can't re-bind aggregates that take keyword-only arguments (seed :=, p :=, mode :=), because DuckDB re-binds positionally. Spilling doesn't re-bind, so it's unaffected.
  • The library allocates a new hash table each time a sketch rebuilds in estimation mode, and in an arena the old one isn't freed. That waste is counted toward the limit, so it can still spill.

🤖 Generated with Claude Code

… they can spill

The set-operation sketches' aggregates (datasketch_theta, datasketch_theta_blob,
datasketch_theta_intersect_agg, datasketch_tuple, datasketch_tuple_intersect_agg,
datasketch_aod, datasketch_aod_intersect_agg: builds and merges) kept heap states
with a destructor, so DuckDB neither counted nor spilled them. They now live in the
aggregate arena, have no destructor, and export and import themselves as BLOBs.

DuckDB creates a state per row when grouping reduces little, and an update sketch
or union allocates a hash table up front, so the states buffer small inputs:
- builds keep their first 32 raw rows (up to 1 KB) and replay them, in order, into
  an update sketch when more arrive, or into a temporary one for a result;
- unions keep up to 32 distinct keys with their combined summaries while that is
  no more than k, which is exactly what the library's union would hold;
- intersections keep their current result and overwrite it in place (the
  library's intersection reallocates its table on every update).
Results are identical to unbuffered building; buffered states export as raw rows
or keys, so importing them keeps them small.

Temporaries (input sketches, results, serialized bytes) go to a per-thread scratch
arena (DSScratchScope). Every sketch object, AoD arrays included, allocates in
arena mode, so the library's copies, which take the source's allocator, land in
whichever arena is current, never on the heap. That avoids the allocator bug class
of hll_union without a library patch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rustyconover
rustyconover merged commit 6ab7116 into v2.0 Oct 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant