Repository navigation
fix: kll_sketch::update() copies the item before a compaction can free it - #573
Merged
leerho merged 1 commit intoNov 15, 2026
Merged
Conversation
stojkomilos
commented
Nov 12, 2026
| update_min_max(static_cast<const T&>(item)); // min and max are always copies | ||
| const uint32_t index = internal_update(); | ||
| new (&items_[index]) T(std::forward<FwdT>(item)); | ||
| if (levels_[0] == 0) { |
Contributor
Author
There was a problem hiding this comment.
Compaction and regrowth can only happen when level 0 is full. So we use that fact to handle this situation.
…e it If the item refers to an item of the same sketch and level 0 is full, internal_update() compacts first, which can destroy that item or free items_. update() then read freed memory. Adds a test for this compaction case, and proost's test from apache#569 for the buffer growth case, which passes on master. Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos
force-pushed
the
stojkomilos/kll-update-aliased-item
branch
from
November 12, 2026 10:13
0c1dd75 to
eabcc54
Compare
stojkomilos
marked this pull request as ready for review
November 12, 2026 10:15
Member
This was referenced Nov 12, 2026
proost
approved these changes
Nov 13, 2026
Contributor
Author
Okay, then this one gets merged first, it's ready. I can't seem to merge it as I don't have permission. Can one of you guys do it? @proost @leerho |
leerho
added a commit
to stojkomilos/datasketches-cpp
that referenced
this pull request
Nov 15, 2026
Resolves the conflicts with apache#558 (lazy allocation of the items buffer): - the constructor sets the levels to the starting buffer size and allocates nothing; the first update or merge allocates min(k, DEFAULT_K) items, which then grow by 8x up to k - reserve() allocates the full k items, so a reserved sketch never grows its buffer - keeps the aliased-item tests from apache#573 and the lazy growth tests, with the construction-time allocation check updated and reserve() tests added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kll_sketch::update(item)copiesiteminto the sketch afterinternal_update(). Ifitemrefers to an item of the same sketch compaction is triggered, those items move. This then leads toitembecoming invalid in the middle of the callkll_sketch::update(item).Note: on master, this is triggered only by compaction, but after #569 merged, it will also be triggered by regrowth. This is not too important for understanding this PR, as the fix is identical for both situations.
Master
This is @proost 's test from #569 (see the next section), with
k = 200instead of 5000, so that the 201st update triggers the first compaction:On master (98c6b74), with GCC 10.5 and 10 runs per build:
-O3: fails 10/10 withstd::bad_alloc.-O1 -fsanitize=address:heap-use-after-free10/10. The read is inupdate()atkll_sketch_impl.hpp:185. The memory was freed inadd_empty_top_level_to_completely_full_sketch()(:742), called fromcompress_while_updating()(:662) andinternal_update()(:202).sketch.update(sketch.get_quantile(0.5))on akll_sketch<std::string>hits the same bug, becauseget_quantile()returns a reference into the sketch. On master it reportsheap-use-after-freeunder ASan. With this PR, the added item equals the median.Tests
Added 2 tests that would catch this bug. One is for regrowth and one is for compaction.