Repository navigation
perf: Theta, Tuple and AoD aggregate states in DuckDB's arena, so they count toward memory_limit and spill - #28
Merged
Merged
Conversation
… they can spill The set-operation sketches' aggregates (datasketch_theta, datasketch_theta_blob, datasketch_theta_intersect_agg, datasketch_tuple, datasketch_tuple_intersect_agg, datasketch_aod, datasketch_aod_intersect_agg: builds and merges) kept heap states with a destructor, so DuckDB neither counted nor spilled them. They now live in the aggregate arena, have no destructor, and export and import themselves as BLOBs. DuckDB creates a state per row when grouping reduces little, and an update sketch or union allocates a hash table up front, so the states buffer small inputs: - builds keep their first 32 raw rows (up to 1 KB) and replay them, in order, into an update sketch when more arrive, or into a temporary one for a result; - unions keep up to 32 distinct keys with their combined summaries while that is no more than k, which is exactly what the library's union would hold; - intersections keep their current result and overwrite it in place (the library's intersection reallocates its table on every update). Results are identical to unbuffered building; buffered states export as raw rows or keys, so importing them keeps them small. Temporaries (input sketches, results, serialized bytes) go to a per-thread scratch arena (DSScratchScope). Every sketch object, AoD arrays included, allocates in arena mode, so the library's copies, which take the source's allocator, land in whichever arena is current, never on the heap. That avoids the allocator bug class of hll_union without a library patch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This moves all seven Theta, Tuple and AoD aggregates onto arena states with no destructor, which export and import their state as BLOBs. That's the same model #27 introduced for KLL and HLL. The shared code is in
src/include/datasketches_theta_arena.hpp.datasketch_theta(v)datasketch_theta(s)(VARCHAR)datasketch_tuple(v, d)datasketch_aod(v, [d, d])How the states work
Tests
seed/p/modeoptions, merges and intersections.test/sql/datasketch_spill_theta.testruns under 128 MB with 4 threads. Every result's hash must equal the same query at 4 GB.Known limits
finalize(agg(...) EXPORT_STATE)can't re-bind aggregates that take keyword-only arguments (seed :=,p :=,mode :=), because DuckDB re-binds positionally. Spilling doesn't re-bind, so it's unaffected.🤖 Generated with Claude Code