Repository navigation
perf: Quantiles, REQ and TDigest aggregate states in DuckDB's arena - #31
Merged
Merged
Conversation
rustyconover
force-pushed
the
arena/quantiles
branch
from
October 28, 2026 17:54
74ecd12 to
5d33ba3
Compare
KLL's arena state is generalized to every item sketch: datasketch_quantiles, datasketch_req and datasketch_tdigest (and their merge overloads) now keep their states in the aggregate arena, without a destructor, and export and import them, so they count toward memory_limit and spill. VARCHAR Quantiles stays on the heap. States buffer their first items (at most 32, and fewer than a new sketch holds before it compacts) and small exact input sketches. REQ and TDigest merge an exact sketch as a sorted run rather than item by item, so their states remember which buffered items came from which merged sketch and replay them as merges; REQ also keeps the first input's rank accuracy mode. Quantiles' merge takes over a copy of an estimation-mode input, allocator included, so such a heap input is copied into the arena first. A REQ state exported with an explicit hra re-binds with it. TDigest exports keep their buffered values, so a spilled state resumes unchanged. DSAllocator's arena mode now takes back the arena's last allocation when it is freed, so the temporaries t-digest allocates in compress() and merge() no longer accumulate in the arena. The new overlay patch tdigest-lazy-buffers grows t-digest's centroids and buffer as they fill instead of reserving them (about 10 KB at K = 100) in the constructor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ates, at half the size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rustyconover
force-pushed
the
arena/quantiles
branch
from
October 28, 2026 18:06
5d33ba3 to
fdb8eae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #27: KLL's arena state is generalized to every item sketch the code generator produces. VARCHAR Quantiles stays on the heap.
datasketch_quantiles(128, v)datasketch_req(12, v)datasketch_tdigest(100, v)How it works
hraargument is now recorded in exported states.DSAllocator::deallocatenow gives back the arena's most recent allocation. Without it, the temporary buffers TDigest allocates on every compress and merge piled up in the arena.tdigest-lazy-buffers.patch(port-version 4). TDigest reserved about 10 KB at K=100 in its constructor and on every deserialize; its buffers now grow as they fill.Results
Single-threaded, the new code gives byte-identical results to the old on nine deterministic cases: TDigest at K=100 and K=10, built from values and merged, and small exact Quantiles and REQ in both HRA modes. All tests pass.
Known issue: slower at moderate group counts
At 200k groups of 100 rows each, these aggregates are 2–3x slower than with heap states. DuckDB 2.0 exports arena-backed states based on its current memory reservation rather than
memory_limit, so it spills even when memory is plentiful. Its ownlist()does the same. An experimental DuckDB change that grows the reservation first removes the slowdown; we're preparing an upstream report.🤖 Generated with Claude Code