Skip to content

perf: Quantiles, REQ and TDigest aggregate states in DuckDB's arena - #31

Merged
rustyconover merged 2 commits into
v2.0from
arena/quantiles
Oct 28, 2026
Merged

rustyconover merged 2 commits into
v2.0from
arena/quantiles

Conversation

@rustyconover

Copy link
Copy Markdown
Collaborator

Builds on #27: KLL's arena state is generalized to every item sketch the code generator produces. VARCHAR Quantiles stays on the heap.

20M rows, 2M groups, 1 GB limit, 8 threads before after
datasketch_quantiles(128, v) 4.3 GB, 17.1 s 0.77 GB, 7.1 s
datasketch_req(12, v) 29.3 GB, 21.4 s 1.07 GB, 9.2 s
datasketch_tdigest(100, v) filled the box's 92 GB; killed after 540 s 1.05 GB, 9.2 s

How it works

  • Buffer: states buffer up to 32 items, and always fewer than a new sketch holds before it first compacts. They also buffer small exact input sketches. REQ and TDigest merge an exact input as a sorted run, and their buffers replay inputs the same way.
  • REQ: keeps the first input's HRA mode and rejects mixing modes, even while buffering. The hra argument is now recorded in exported states.
  • Quantiles: the library's merge takes over a copy of an estimation-mode input, allocator included. Such inputs are copied into the arena first.
  • Shared allocator change: in arena mode, DSAllocator::deallocate now gives back the arena's most recent allocation. Without it, the temporary buffers TDigest allocates on every compress and merge piled up in the arena.
  • Library patch: tdigest-lazy-buffers.patch (port-version 4). TDigest reserved about 10 KB at K=100 in its constructor and on every deserialize; its buffers now grow as they fill.

Results

Single-threaded, the new code gives byte-identical results to the old on nine deterministic cases: TDigest at K=100 and K=10, built from values and merged, and small exact Quantiles and REQ in both HRA modes. All tests pass.

Known issue: slower at moderate group counts

At 200k groups of 100 rows each, these aggregates are 2–3x slower than with heap states. DuckDB 2.0 exports arena-backed states based on its current memory reservation rather than memory_limit, so it spills even when memory is plentiful. Its own list() does the same. An experimental DuckDB change that grows the reservation first removes the slowdown; we're preparing an upstream report.

🤖 Generated with Claude Code

rustyconover and others added 2 commits October 28, 2026 14:00
KLL's arena state is generalized to every item sketch: datasketch_quantiles,
datasketch_req and datasketch_tdigest (and their merge overloads) now keep
their states in the aggregate arena, without a destructor, and export and
import them, so they count toward memory_limit and spill. VARCHAR Quantiles
stays on the heap.

States buffer their first items (at most 32, and fewer than a new sketch
holds before it compacts) and small exact input sketches. REQ and TDigest
merge an exact sketch as a sorted run rather than item by item, so their
states remember which buffered items came from which merged sketch and
replay them as merges; REQ also keeps the first input's rank accuracy mode.
Quantiles' merge takes over a copy of an estimation-mode input, allocator
included, so such a heap input is copied into the arena first. A REQ state
exported with an explicit hra re-binds with it. TDigest exports keep their
buffered values, so a spilled state resumes unchanged.

DSAllocator's arena mode now takes back the arena's last allocation when it
is freed, so the temporaries t-digest allocates in compress() and merge()
no longer accumulate in the arena. The new overlay patch
tdigest-lazy-buffers grows t-digest's centroids and buffer as they fill
instead of reserving them (about 10 KB at K = 100) in the constructor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ates, at half the size

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rustyconover
rustyconover merged commit 81d2e1a into v2.0 Oct 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant