Skip to content

Return the exact min and max at ranks 0 and 1 in KLL sketches - #567

Merged
leerho merged 1 commit into
apache:masterfrom
stojkomilos:stojkomilos/quantiles-include-min-max
Nov 16, 2026
Merged

leerho merged 1 commit into
apache:masterfrom
stojkomilos:stojkomilos/quantiles-include-min-max

Conversation

@stojkomilos

@stojkomilos stojkomilos commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What changed

kll_sketch keeps the exact min and max of the stream. But get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Then ranks 0 and 1 give a retained item, not the exact min and max:

kll_sketch<float> sketch(8);
sketch.update(0);
sketch.update(999);
for (int i = 1; i < 999; ++i) sketch.update(i);
sketch.get_min_item();   // 0
sketch.get_quantile(0);  // 23 (changes from run to run)
sketch.get_max_item();   // 999
sketch.get_quantile(1);  // 998

Since 6.0.0, datasketches-java returns the exact min and max here (apache/datasketches-java#547, IncludeMinMax). Its sorted views add the min and max back when the retained items do not hold them. So for the same data, the two libraries give different answers at ranks 0 and 1.

This PR ports that step to the C++ KLL sketch:

  • kll_sketch::get_sorted_view() adds a missing min and max, with the same arithmetic as Java. A missing min goes first, and the entry after it gives 1 of its weight to the min. A missing max goes last, and the entry before it gives 1 of its weight to the max. The total weight does not change. So get_quantile(), get_rank(), get_PMF(), get_CDF() and the sorted view iterator all see the min and max, as in Java.
  • The levels are sorted, so the sketch finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with the new quantiles_sorted_view::add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. The new convert_to_cummulative(max_item) takes the weight for the min and appends a missing max. The view reserves room for both, but only for a non-empty sketch.
  • quantiles_sketch and req_sketch do not change. They still call convert_to_cummulative(), so their sorted views are the same as on master.

Behavior change: only ranks within 1/n of 0 or 1 give a different quantile, and a rank changes by at most 1/n. For example, the inclusive rank of a dropped min is now 1/n, not 0. These are the values that Java returns for the same data.

For non-arithmetic types, the view and the reference that get_quantile() returns now can point to the min and max in the sketch object, not only to its item buffer. So a move of the sketch now also makes them invalid. The docs of kll_sketch::get_sorted_view(), kll_sketch::get_quantile() and quantiles_sorted_view now say that, for non-arithmetic types, they must not be used after the sketch is updated, merged, assigned to, moved or destroyed.

Performance

Benchmarks of kll_sketch::get_sorted_view() against master measured no meaningful performance decrease.

How tested

Added

quantiles_sorted_view_test (6 tests): each checks the entries and their cumulative weights.

  • Min and max (5 separate tests): both missing, both retained, one retained item (with the data of the Java IncludeMinMaxTest), only the min missing, and only the max missing.
  • add() (1 test): empty ranges first and between two ranges.

kll_sketch_test (3 tests): each builds 1000 sketches, adds the min and max first so compaction drops them, and requires at least one sketch that dropped one.

  • Built by update (float and string = 2 tests): ranks 0 and 1 give the exact min and max. The float test also checks the Java ranks of a dropped min and max.
  • After deserialize and merge (1 test): the same checks on a deserialized sketch and on its merge.

Test strength: with get_sorted_view() as on master, the 3 KLL tests fail. With add_min() and the max step of convert_to_cummulative(max_item) changed to do nothing, the 3 KLL tests and the 4 sorted view tests that need the min or max fail. "Both retained" and the add() test pass, as expected.

This pull request and its description were written by Isaac.

@stojkomilos
stojkomilos marked this pull request as ready for review November 6, 2026 13:06

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Matching Java at ranks 0 and 1 is worthwhile, and the arithmetic in include_min_max() matches IncludeMinMax. CI passes, and the common, KLL and classic quantiles tests also pass for me locally under ASan and UBSan.

Timing: this changes the results users get, so it belongs in a minor release, not in the bug-fix release we're finishing now. We're holding other PRs until that release is out.

There are two things to settle before merging.

  1. Sorted view lifetime. As you noted, for non-arithmetic types the view now points at the min and max inside the sketch object, so a view held across a move of the sketch no longer works. That fits the existing rule, as in Java, that a sorted view shouldn't be used after the sketch changes, but in C++ that rule isn't written down, and the failure is silent. Could you add it to the get_sorted_view() documentation in both sketches, saying that for non-arithmetic types the view refers to the sketch's items and must not be used after the sketch is updated, merged, moved or destroyed?

  2. REQ. With this PR, KLL and classic quantiles match Java, but REQ doesn't. I'd like to see the REQ follow-up you offered planned alongside this, so all three sketches change in the same release. Moving REQ get_rank() to the sorted view is a reasonable way to do it.

@stojkomilos stojkomilos changed the title Return the exact min and max at ranks 0 and 1 in KLL and quantiles sketches Return the exact min and max at ranks 0 and 1 in KLL, quantiles and REQ sketches Nov 7, 2026
@stojkomilos
stojkomilos force-pushed the stojkomilos/quantiles-include-min-max branch from e714c8d to 8cd37cc Compare November 7, 2026 23:29
T* begin();
T* end();

uint64_t compute_weight(const T& item, bool inclusive) const;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed as it's now unused.

@stojkomilos
stojkomilos marked this pull request as draft November 9, 2026 12:35
@stojkomilos
stojkomilos force-pushed the stojkomilos/quantiles-include-min-max branch 6 times, most recently from 0a1bef4 to 305ee6c Compare November 11, 2026 05:54
@stojkomilos
stojkomilos force-pushed the stojkomilos/quantiles-include-min-max branch from 305ee6c to 8af7451 Compare November 13, 2026 09:57
@stojkomilos stojkomilos changed the title Return the exact min and max at ranks 0 and 1 in KLL, quantiles and REQ sketches Return the exact min and max at ranks 0 and 1 in KLL sketches Nov 13, 2026
@stojkomilos
stojkomilos force-pushed the stojkomilos/quantiles-include-min-max branch 4 times, most recently from 4bbf29f to dc81617 Compare November 13, 2026 13:07
@stojkomilos

Copy link
Copy Markdown
Contributor Author

@leerho

Timing: this changes the results users get, so it belongs in a minor release, not in the bug-fix release we're finishing now. We're holding other PRs until that release is out.

Do you know approximately when this bug-fix release and this minor release are coming out? Order of magnitude

  1. Sorted view lifetime. As you noted, for non-arithmetic types the view now points at the min and max inside the sketch object, so a view held across a move of the sketch no longer works. That fits the existing rule, as in Java, that a sorted view shouldn't be used after the sketch changes, but in C++ that rule isn't written down, and the failure is silent. Could you add it to the get_sorted_view() documentation in both sketches, saying that for non-arithmetic types the view refers to the sketch's items and must not be used after the sketch is updated, merged, moved or destroyed?

Done.

2. REQ. With this PR, KLL and classic quantiles match Java, but REQ doesn't. I'd like to see the REQ follow-up you offered planned alongside this, so all three sketches change in the same release. Moving REQ get_rank() to the sorted view is a reasonable way to do it.

Actually I decided against this, and doing only for KLL. Let me know if you disagree. The reason I changed my mind, is that i reworked most of the PR (due to the performance issues), and now it's a bit more complex, so I think it's a bit easier to review it this way after the rework.

@stojkomilos

Copy link
Copy Markdown
Contributor Author

@leerho I reworked the PR a bit, due to performance overheads the previous version was adding. It's a bit more complex now but adds no meaningfull overhead, so I think this is a lot better.

@leerho

leerho commented Nov 16, 2026

Copy link
Copy Markdown
Member

Thanks for the rework, and for adding the lifetime notes.

On timing: the bug-fix release (5.3.1) went out on Nov 13. The next one will be a minor release, probably 5.4.0, once we've landed the open PRs (we may have to draw a line somewhere). It comes before the larger refactor that will become 6.0.0. I can't give a firm date, because it depends on how fast those PRs land.

I checked the new version against Java's IncludeMinMax: on 7,500 random KLL sketches (floats and strings, merged sketches, many duplicates), the sorted view's items and cumulative weights matched Java's in every case, including the 4,795 that needed an adjustment. I also merged it into current master locally and ran all tests under ASan and UBSan: they pass. There is one small conflict with master, in the get_sorted_view() doc comment in kll_sketch.hpp, where #558 added "An empty sketch returns an empty view without allocating item storage." Keeping both lines resolves it. Could you merge master into the branch, or would you like us to do it?

On scope: KLL only is fine for this PR, since it keeps the review manageable. But Java includes the exact min and max in classic quantiles and REQ too, and I don't want a release where C++ KLL does this and the other two don't. So I'd like classic quantiles and REQ to follow as separate PRs before 5.4.0. If you can't do those, please let us know and we'll take them on.

kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
@stojkomilos
stojkomilos force-pushed the stojkomilos/quantiles-include-min-max branch from dc81617 to 7121fb3 Compare November 16, 2026 09:40
@stojkomilos

Copy link
Copy Markdown
Contributor Author

@leerho

On scope: KLL only is fine for this PR, since it keeps the review manageable. But Java includes the exact min and max in classic quantiles and REQ too, and I don't want a release where C++ KLL does this and the other two don't. So I'd like classic quantiles and REQ to follow as separate PRs before 5.4.0. If you can't do those, please let us know and we'll take them on.

I don't really use these 2 sketches (I only use KLL), so I would have to learn how they work and other details about them. So I would prefer if you take it on, that would be great :D

here is one small conflict with master, in the get_sorted_view() doc comment in kll_sketch.hpp, where #558 added "An empty sketch returns an empty view without allocating item storage." Keeping both lines resolves it. Could you merge master into the branch, or would you like us to do it?

This is pushed now.

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rebase. The new head (7121fb3) has exactly the same tree as the merge I tested locally under ASan and UBSan.

And no problem: we’ll take on classic quantiles and REQ ourselves, as separate PRs before 5.4.0.

Approving. Thanks again for this.

@leerho
leerho merged commit 727e37c into apache:master Nov 16, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants