Repository navigation
perf: tdigest grows its buffers as they fill - #561
Conversation
The constructor reserved room for 2k + fudge centroids and four times as many buffered values (about 10 KB for k = 100 with doubles), also when deserializing. Code that holds many small t-digests paid for all of it. Grow the buffer geometrically up to its limit on update, and the centroids before a compress or merge needs them. Results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3dee23d to
15024ac
Compare
|
Thanks, Rusty. Same situation as #558: we'd like one allocation convention across the sketches and languages before merging either. The dev@ proposal, a lazy consensus ending Nov 7, 01:00 UTC, is here: https://lists.apache.org/thread/rv730qo61qqkd99dcy4wnglf3hz1s0fz I'm marking this PR as a draft until that resolves. If it passes, this PR fits it well. The geometric growth up to the existing limits is the behavior the proposal describes, and we'd only add a |
|
@leerho thanks for outlining the remaining work. The proposal's deadline has now passed, so I've followed up on #558 to confirm the outcome, when both PRs can leave draft, and who will handle All 21 checks on this PR's current commit ( |
leerho
left a comment
There was a problem hiding this comment.
The lazy-allocation proposal passed (https://lists.apache.org/thread/d9pzln7m5q1of82ghhotrr3h0j6p514m), so I'm taking this out of draft. As with #558 (see my comment there), I'll add reserve(), the allocation-failure tests and the documentation on top of this PR, in C++ and Java. Marking changes requested so it isn't merged before those are in.
|
Thanks @leerho you're awesome! |
…g storage reserve() allocates the full storage for centroids and buffered values right after construction, for callers who want allocation failures there or don't want updates to pay for growing the storage. merge() now allocates before it widens min and max, so a failed merge leaves the t-Digest unchanged. The new tests use an allocator that fails on purpose. A failure at any allocation of reserve(), a stream of updates or a merge leaves the t-Digest exactly as it was, and it stays usable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I've pushed the |
Changes I submitted are in and there are no prerequisite PRs.
tisonkun
left a comment
There was a problem hiding this comment.
The allocation changes look sound. Compression still follows the existing logical buffer limit, and the additional reservations happen before the digest contents are modified. I found no blocking correctness issues.
Two non-blocking suggestions:
-
In the
reserve()documentation (tdigest.hpp, lines 131–135), clarify that it only preallocates the persistent centroid and value buffers.compress()andmerge()still allocate temporary storage, so anupdate()that triggers compression can still throwstd::bad_allocafterreserve()succeeds. The wording "if allocation failures should happen here" could otherwise suggest that all allocation failures are moved to this call. -
In
tdigest_custom_allocator_test.cpp, lines 57–65, bothgrownandreferenceuse the modified implementation; only their allocators differ. This checks allocator independence, rather than equivalence with the previous implementation. Please consider adjusting the comment and the corresponding claim in the PR description. The new allocation test already compares serialized contents with and withoutreserve(), so that coverage does not need to be duplicated.
Validation: the complete tdigest_test suite passed with AppleClang 21, C++11, ASan and UBSan. I also compared the base and head across 36 combinations of value type, k and input distribution, including compression, copying, merging, self-merging and updates after both stream and byte-buffer deserialization; the final serialized-output hashes matched in every case.
tdigest's constructor reserves room for2k + fudgecentroids andBUFFER_MULTIPLIERtimes as many buffered values. Fork = 100withdoublethat's about 10 KB per digest before anything is added, and it happens again on everydeserialize().This is the same kind of change as #558 for
kll_sketch. Instead of reserving everything up front:update()grows the buffer geometrically, starting at 8 values and doubling up to its existing limit.compress()andmerge()grow the centroids before they need them, also up to the existing limit.Results and the serialized format are unchanged.
Test
Added
tdigest custom allocator: buffers grow as they filltotdigest_custom_allocator_test.cpp, usingtest_allocator:REQUIRE( test_allocator_total_bytes == 0 ).std::allocatorwithout the change.All 17 test suites pass under CTest (GCC 14, aarch64).
We found this in the DuckDB DataSketches extension. For GROUP BY queries with many groups, DuckDB creates a state for each input row and merges them later, so the up-front reservation multiplied by millions of states. DuckDB made the same change to its own t-digest.
🤖 Generated with Claude Code