Skip to content

KLL: weighted update, matching datasketches-java - #582

Open
leerho wants to merge 6 commits into
masterfrom
kll-weighted-update
Open

leerho wants to merge 6 commits into
masterfrom
kll-weighted-update

Conversation

@leerho

@leerho leerho commented Nov 15, 2026

Copy link
Copy Markdown
Member

Adds kll_sketch::update(item, weight), equivalent to calling update(item) weight times. It follows the weighted update of the KLL sketches in datasketches-java.

This builds on #542 by Shani Elharrar, whose two commits are kept here unchanged, with fixes on top.

Behavior

  • If weight is smaller than the free space in level zero, the item is inserted weight times as plain updates. No compaction can happen on this path.
  • Otherwise, the update builds an exact sketch with one copy of the item at each level whose bit is set in weight, and merges it in. This costs one merge with a sketch of at most 61 items, however large the weight.
  • A weight of 1 is a plain update. A weight of 0, or above 2^61 - 1 (kll_constants::MAX_WEIGHT), throws std::invalid_argument. These are the same rules as in Java, which cannot read an image with n >= 2^63.
  • NaN items are ignored, as in update(item).
  • Serialization is unchanged.

Fixes on top of #542

  • The small-weight path wrote through a null pointer on a new sketch: since perf: kll_sketch allocates its items on first use #558, the items storage is allocated on first use. The weighted update now allocates first, like update(item), so a failed allocation also leaves the sketch unchanged.
  • Each copy is constructed before it is counted, so a failed copy leaves a valid sketch.
  • Weights of 2^61 and above are rejected, as described above, instead of folding the high bits into level 60.

Also included

  • kll_helper::floor_of_log2_of_fraction looped forever once numer >= 2^63, because doubling denom overflowed to zero. Any merge that reached n >= 2^63 hung. This is the first commit from Add weighted update to kll_sketch #542. Java computes the same bound without a loop, so it is not affected.
  • The KLL iterator started at level 0 without skipping empty levels. When level zero was empty, which a merge can leave, every item was reported with weight 1. Weighted updates with an even weight leave level zero empty every time, but plain merges hit it too: in 2,000 random merge trials on master, 384 had wrong iterator weights. Quantile queries were not affected, since the sorted view reads the levels directly. Java's iterator skips empty levels and is correct.

Tests

  • Weighted update sections in kll_sketch_test.cpp: zero weight, NaN, same as repeated updates in exact mode, one item with a large weight, the maximum weight with a serialization round trip, weights above the maximum, weight 1, the first weighted update of a new sketch on both paths, large weights into a full sketch, estimation mode within the k=200 error bound, and strings.
  • Allocation failures in kll_sketch_allocation_test.cpp: a failed first weighted update leaves the sketch unchanged on both paths, and failing each allocation of a stream of weighted updates in turn leaves a valid, usable sketch.
  • A test that iterator weights sum to n after merges that leave level zero empty. It fails on master.

All tests pass under ASan and UBSan.

🤖 Generated with Claude Code

shanielh and others added 4 commits November 15, 2026 10:53
…numerators

Doubling denom until it exceeds numer overflows to zero once numer >= 2^63,
so the loop never terminated. ub_on_num_levels(n) calls it with the merged
stream weight, which made a merge reaching n >= 2^63 hang. Compare against
numer / 2 instead, which is equivalent and cannot overflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update(item, weight) is equivalent to calling update(item) weight times.
A weight that fits into the free space of level zero is applied as plain
updates; a larger one builds an exact sketch holding one copy of the item
at each level whose bit is set in the weight and merges it, so the cost is
logarithmic in the weight. This mirrors the weighted update in
datasketches-java.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The const_iterator started at level 0 without skipping empty levels.
When level 0 is empty, which a merge can leave, every item was reported
with weight 1 instead of the weight of its level. Quantile queries were
not affected: the sorted view reads the levels directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Reject weights above 2^61 - 1 (kll_constants::MAX_WEIGHT), as
  datasketches-java does, instead of folding the high bits into level 60.
  Java cannot read an image with n >= 2^63.
- Allocate the items storage before writing into it. Since items are
  allocated lazily, the small-weight path wrote through a null pointer
  on a new sketch. Allocating first also leaves the sketch unchanged
  when the allocation fails.
- Construct each copy before counting it, so that a failed copy leaves
  a valid sketch.
- A weight of 1 is a plain update, as in Java.
- Check the item before the weight, as in Java.
- Tests: the maximum weight, weights above it, weight 1, the first
  weighted update of a new sketch on both paths, and allocation failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coveralls

coveralls commented Nov 15, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 44436712044

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.1%) to 96.974%

Details

  • Coverage increased (+0.1%) from the base build.
  • Patch coverage: 47 of 47 lines across 2 files are fully covered (100%).
  • 92 coverage regressions across 13 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

92 previously-covered lines in 13 files lost coverage.

Top 10 Files by Coverage Loss Lines Losing Coverage Coverage
kll/include/kll_sketch_impl.hpp 34 95.51%
tdigest/include/tdigest_impl.hpp 20 95.83%
filters/include/bloom_filter_impl.hpp 10 93.28%
hll/include/CouponHashSet-internal.hpp 5 96.45%
hll/include/CouponList-internal.hpp 5 97.17%
count/include/count_min_impl.hpp 4 98.25%
hll/include/Hll4Array-internal.hpp 4 97.4%
hll/include/HllSketch-internal.hpp 4 97.14%
hll/include/AuxHashMap-internal.hpp 2 98.71%
common/include/quantiles_sorted_view_impl.hpp 1 98.59%

Coverage Stats

Coverage Status
Relevant Lines: 17083
Covered Lines: 16566
Line Coverage: 96.97%
Coverage Strength: 1907459.12 hits per line

💛 - Coveralls

Both deserializers, from bytes and from a stream, must allocate only what
a newly built empty sketch does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread kll/include/kll_sketch_impl.hpp Outdated
}
items_size_ = levels_[num_levels_];
items_ = allocator_.allocate(items_size_);
for (uint32_t i = 0; i < items_size_; ++i) new (&items_[i]) T(item);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of this logic, If bad_alloc happens, we should deallocate in L248. If not, this is memory leak.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks. If a copy of the item throws, the destructor doesn't run, so the storage and the copies already made leaked. Fixed in 0ab4ab6: the constructor now destroys the copies it made and frees the storage before rethrowing.

Comment thread kll/include/kll_sketch_impl.hpp Resolved
The constructor that builds the weighted sketch now destroys the copies it
made and frees its storage if a copy throws. The weighted update resets the
sorted view before changing anything. merge() now forwards an rvalue source
to merge_higher_levels(), so its higher levels are moved rather than copied,
and the merge the weighted update makes copies no items.

A new test makes each item copy of a weighted update throw in turn, and
checks for leaks and a stale sorted view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested a review from proost November 19, 2026 01:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants