Skip to content

perf: kll_sketch grows its items buffer by 8x up to k - #569

Merged
proost merged 8 commits into
apache:masterfrom
stojkomilos:stojkomilos/kll-lazy-growth
Nov 18, 2026
Merged

proost merged 8 commits into
apache:masterfrom
stojkomilos:stojkomilos/kll-lazy-growth

Conversation

@stojkomilos

@stojkomilos stojkomilos commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What this adds on top of #558

On master, a sketch allocates its k items in the constructor, and #558 moves that allocation to the first update or merge. Either way a sketch with one item holds 512 KiB at k = 65535 with doubles. This PR makes that allocation room for min(k, 200) items (200 is DEFAULT_K), and grows a full buffer by 8x up to k: 200, 1600, 12800, then k.

  • Compaction still starts only when the buffer holds k items, so the compaction points and random bits do not change.

  • Once the buffer holds k items, both serializers run the same code as master. While it is still growing, with one level, they write the start of level 0 as an offset into a buffer of k items, so the serialized bytes and to_string() do not change. A deserialized sketch still gets the full buffer of k items.

  • After the first compaction nothing changes.

Bytes held by a kll_sketch<double> with k = 65535 (items buffer and levels), measured with a counting allocator on master (98c6b74), this PR (0ad21a2), #558 (1fcf432) and #558 + this PR:

Items master this PR #558 #558 + this PR
0 524,288 1,608 8 8
1 to 200 524,288 1,608 524,288 1,608
201 to 1600 524,288 12,808 524,288 12,808
1601 to 12800 524,288 102,408 524,288 102,408
more 524,288 524,288 524,288 524,288

At the default k = 200 nothing changes, because the first allocation is already k items.

Tests

Added

kll sketch: lazy growth of the items buffer (10 sections). Every section also requires that all test_allocator bytes are freed at the end.

  • Start size (1 section): with k = DEFAULT_K / 2, the test_allocator bytes stay at k items, not DEFAULT_K items.
  • Growth by 8x (1 section): at k = 65535, the bytes follow 200, 1600, 12800 and 65535 items, and compaction starts only after 65535 items.
  • Merge growth (1 section): at k = 65535, merging 1000 items into an empty sketch grows its buffer to 1600 items, as updates do.
  • reserve() (3 sections): before any update, it allocates k items, and the buffer never grows. After updates, it grows the buffer to k items in one step. A reserved sketch holds the same items as a growing one.

The next 4 bullets run a growing sketch next to a copy with a full buffer (a deserialized sketch), with the same random_utils::random_bit state. They require the same items, weights, min, max, quantiles, ranks, PMF, CDF, both serializations and to_string().

  • Updates (3 k = 3 runs): k = 201, 5000 and 20000, up to 10 times k items, compared at each growth step and around compactions.
  • Merges (6 inputs x 2 orders = 12 merges): compared after each merge and after 50000 more updates.
  • Copies (6 sketches): the original, a copy, a copy assignment, a move, a move assignment and a float to double conversion, compared after 1000, 20000 and 200000 more updates.
  • Strings (1 run): k = 5000, compared around each growth step and at 10001 and 30000 items.

kll sketch: reserve() after a query (1 test): a string sketch caches its sorted view with get_rank(), then reserve() grows the buffer. The rank and the median must not change.

kll_sketch_serialize_for_java.cpp (4 types x 2 k x 7 n = 56 files): float, double, long and string sketches. With k = 1000, n = 0, 1, 10, 100, 200, 201 and 1000. With k = 65535, n = 0, 1, 10, 100, 1000, 10000 and 12801. The buffer is still growing at the smaller n, and n = 201 and 12801 cap its growth at k.

Test strength: without the k_ - items_size_ shift in either serializer, 4 checks fail. Without the min in the start size, 1 check fails. Without reset_sorted_view() in reserve(), the new test fails, and ASan reports a heap-use-after-free.

All 17 CTest suites pass with GCC 10.5 and C++11, and kll_test also passes with ASan and UBSan. A separate harness (not in this PR) wrote bitwise-identical serializations with master, with this PR, and with this PR on top of #558. It covers double, float, int64 and string items, 15 values of k, n around every buffer size, 4 input orders, 2 seeds, merges by copy and by move, and the type-converting constructor. With RESIZE_FACTOR set to X1, X2 or X4, it also wrote the same bytes as master. The 56 new cross-language files are byte-identical to master's. datasketches-java (main, ef440a81) reads all 56, and each one equals the bytes of a Java sketch of the same items.

Benchmarks

A Google Benchmark program on kll_sketch<double>, with items from a fixed-seed 64-bit LCG (integers spread uniformly over [0, 2^53)). I built it against master (c22f2b1) and this PR (3d1461c), with GCC 10.5 in Release, on an x86_64 VM (Xeon 8375C). The binaries ran in 7 rounds in random order on one pinned core, with 5 repetitions per round. The VM is shared, so the times are the fastest of the 35 runs. The ratios of the median times are within 6% of these.

The last column is master / this PR: 5 means this PR is 5 times faster, and 0.2 means 5 times slower.

Case k master this PR master / this PR
1000 live sketches, 10 items each 5000, 20000, 65535 1199, 1538, 1684 us 74, 74, 74 us 16.2, 20.8, 22.8
One sketch, 10^6 items 800 35.6 ms 36.1 ms 0.99
One sketch, 10^7 items 65535 650 ms 654 ms 0.99

This pull request and its description were written by Isaac.

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For safe changes, can you add cross language compatible test case too?

Comment thread kll/include/kll_sketch_impl.hpp Resolved
Comment thread kll/include/kll_sketch_impl.hpp Resolved
@stojkomilos
stojkomilos force-pushed the stojkomilos/kll-lazy-growth branch from 72066ad to ea028f7 Compare November 10, 2026 09:57
@stojkomilos

Copy link
Copy Markdown
Contributor Author

For safe changes, can you add cross language compatible test case too?

@proost

I think I did that, but I wasn't 100% sure if this is what you meant. It's in the last commit if you want to take a look.

@stojkomilos
stojkomilos requested a review from proost November 10, 2026 11:25

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding cases. But can you add k=65535 and n=12801 case too?

This is growth capped case

Comment thread kll/include/kll_sketch_impl.hpp Outdated
// Grows the items buffer of a one-level sketch when it is full and smaller than k.
// Compaction starts only when the buffer is full at k items, so growing does not change the result.
template<typename T, typename C, typename A>
void kll_sketch<T, C, A>::grow_items() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes use-after-free problem when using with string and update after grow_items called.

Consider this case:

TEST_CASE("kll sketch: update with aliased item during buffer growth", "[kll_sketch]") {
  const uint16_t k = 5000;
  kll_sketch<std::string> sketch(k);

  for (int i = 0; i < kll_constants::DEFAULT_K; ++i) {
    sketch.update(std::string(64, 'x') + std::to_string(i));
  }

  REQUIRE(sketch.get_n() == kll_constants::DEFAULT_K);

  const std::string& item = sketch.begin()->first;
  const std::string expected = item;

  // Triggers grow_items()
  sketch.update(item);

  REQUIRE(sketch.get_n() == kll_constants::DEFAULT_K + 1);

  uint32_t count = 0;
  for (const auto& entry : sketch) {
    if (entry.first == expected) {
      ++count;
    }
  }
  REQUIRE(count == 2);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, valid.

Although I don't think I should do this in this PR but in a followup. Reason: this bug is also present on master right now, although it's triggered on compaction rather than grow_items. See this example that fails on master

TEST_CASE("kll sketch: update with aliased item during compaction", "[kll_sketch]") {
    const uint16_t k = 200;
    kll_sketch<std::string> sketch(k);


    const std::string& item = sketch.begin()->first;
    const std::string expected = item;

    // Triggers compaction
    sketch.update(item);

    REQUIRE(sketch.get_n() == kll_constants::DEFAULT_K + 1);

    uint32_t count = 0;
    for (const auto& entry : sketch) {
      if (entry.first == expected) {
        ++count;
      }
    }
    REQUIRE(count == 2);
  }

I already have a draft PR for this (fixes both master bug and current bug, not ready for review yet).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, Thank you. you are right. It happens master branch also.

@stojkomilos stojkomilos Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, then if you agree can we merge this?

The PR that fixes this for master (compact) and this PR (regrowth) is now ready for review - #573

stojkomilos and others added 5 commits November 11, 2026 06:31
A new sketch has room for min(k, 200) items instead of k, and a full
buffer smaller than k grows by RESIZE_FACTOR (X8) up to k. Compaction
still starts only when the buffer holds k items, and while there is one
level the serialized levels are offsets into a buffer of k items, so
results and serialized bytes do not change.

Co-authored-by: Isaac <no-reply@databricks.com>
Write the growing-buffer level offset only for a one-level sketch, and
generate float, double, long and string files with k = 65535 and n up to
10000, while the items buffer still grows.

Co-authored-by: Isaac <no-reply@databricks.com>
…ss-language files

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@stojkomilos
stojkomilos force-pushed the stojkomilos/kll-lazy-growth branch from 4384c32 to 0ad21a2 Compare November 11, 2026 06:42
@stojkomilos
stojkomilos requested a review from proost November 11, 2026 07:45
@stojkomilos

Copy link
Copy Markdown
Contributor Author

Thanks for adding cases. But can you add k=65535 and n=12801 case too?

This is growth capped case

@proost

Done! :D

stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 12, 2026
This is proost's test from apache#569. It passes on master, which allocates k
items at the start, and guards the lazy growth in apache#569.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 12, 2026
…e it

If the item refers to an item of the same sketch and level 0 is full,
internal_update() compacts first, which can destroy that item or free
items_. update() then read freed memory.

Adds a test for this compaction case, and proost's test from apache#569 for
the buffer growth case, which passes on master.

Co-authored-by: Isaac <no-reply@databricks.com>
@leerho

leerho commented Nov 12, 2026

Copy link
Copy Markdown
Member

@stojkomilos thanks for the cross-language cases. Merge order for the open KLL PRs: #573 first, then #558, then this one. #558 moves the item allocation to the first update or merge, and your growth changes the same code, so this will need a rebase onto master after #558 is merged. We'll comment here when it is. @proost's change request also still needs to be resolved.

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@leerho

leerho commented Nov 15, 2026

Copy link
Copy Markdown
Member

@stojkomilos thanks, and thanks @proost for the approval. #573 is merged; apache/datasketches-cpp#558 · perf: kll_sketch allocates its… is next. Once apache/datasketches-cpp#558 · perf: kll_sketch allocates its… is merged this needs a rebase, and it will conflict in kll_sketch.hpp and kll_sketch_impl.hpp as well as the tests, because apache/datasketches-cpp#558 · perf: kll_sketch allocates its… makes the item storage lazy and adds reserve(). In the rebase, allocate_items() should allocate your smaller starting buffer, and reserve() should keep its documented meaning: allocate the storage needed before the first compaction, which with your growth means growing to the full size. Happy to do that rebase here with maintainer edits if you prefer; just say so.

Resolves the conflicts with apache#558 (lazy allocation of the items buffer):
- the constructor sets the levels to the starting buffer size and allocates nothing;
  the first update or merge allocates min(k, DEFAULT_K) items, which then grow by 8x up to k
- reserve() allocates the full k items, so a reserved sketch never grows its buffer
- keeps the aliased-item tests from apache#573 and the lazy growth tests, with the
  construction-time allocation check updated and reserve() tests added

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho

leerho commented Nov 15, 2026

Copy link
Copy Markdown
Member

I merged master into this branch to resolve the conflicts with #558. The constructor no longer allocates; the first update or merge allocates the starting buffer of min(k, DEFAULT_K) items, which then grows by 8x up to k as before. reserve() now allocates the full k items, so a reserved sketch never grows. I kept the aliased-item tests from #573, changed the one lazy-growth check that expected an allocation at construction, and added tests for reserve(). @stojkomilos and @proost, please take a look, since this changed after the approval.

@leerho
leerho requested a review from proost November 15, 2026 19:28
@stojkomilos

stojkomilos commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

I merged master into this branch to resolve the conflicts with #558. The constructor no longer allocates; the first update or merge allocates the starting buffer of min(k, DEFAULT_K) items, which then grows by 8x up to k as before. reserve() now allocates the full k items, so a reserved sketch never grows. I kept the aliased-item tests from #573, changed the one lazy-growth check that expected an allocation at construction, and added tests for reserve(). @stojkomilos and @proost, please take a look, since this changed after the approval.

LGTM. Please do note this behaviour, I don't know if it's intentional or not, decide if you want it - experiments show (and AI confirms in code):

  • .reserve() if called before the first update (when the buffer is empty), will allocate the full k items (so the buffer never grows)
  • However if you call .reserve() before the first update is called, it only allocates the min(k, 200).

Perhaps you want these 2 to behave the same? Perhaps not.

Before the first compaction, a buffer that is still growing is grown to
k items in one step, so reserve() means the same before and after the
first update: the buffer does not need to grow before the first
compaction. As noted in review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho

leerho commented Nov 16, 2026

Copy link
Copy Markdown
Member

Thanks, good catch (I read your second point as "after the first update"). That was the documented behavior, but you're right that the two should match: reserve() should mean that the buffer does not need to grow before the first compaction, whenever it is called. t-digest's reserve() already works that way, and REQ's first allocation is already full size, so it has no partly grown buffer.

So now, if the buffer is still growing, reserve() grows it to k items in one step, keeping the items already in it. Once the buffer has k items, or after the first compaction, it does nothing. grow_items() now takes the new size, so reserve() and the growth on update share it. The test "reserve() after updates grows the buffer to k items" replaces "reserve() after the first update does nothing": after 10 updates, reserve() grows the buffer to k, and updates up to k items allocate nothing more. All tests pass under ASan and UBSan.

@stojkomilos

Copy link
Copy Markdown
Contributor Author

Looks like ready to merge IMO, if you agree, @leerho

@leerho

leerho commented Nov 16, 2026

Copy link
Copy Markdown
Member

Waiting for @proost to do a final review. Since I contributed code to this, I can no longer do the final approval.

items_size_ = k_;
levels_[0] = levels_[1] = k_;
} else if (num_levels_ == 1 && items_size_ < k_) {
grow_items(k_); // the buffer is still growing: grow it to its full size in one step

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this, we should reset sorted view.

Considering get_rank -> reserve -> get_rank case. At the first get_rank, we cached view. But at reserve call we destroy. And second get_rank derefer dangling pointer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, also added test that would have caught this.

For non-arithmetic items, the cached sorted view holds pointers into the
items buffer, and growing the buffer frees it. A query after reserve() then
read freed memory.

Co-authored-by: Isaac <no-reply@databricks.com>
@stojkomilos
stojkomilos requested a review from proost November 17, 2026 19:00

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your effort!

@proost
proost merged commit 5492764 into apache:master Nov 18, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants