Repository navigation
Return the exact min and max at ranks 0 and 1 in KLL sketches - #567
Conversation
leerho
left a comment
There was a problem hiding this comment.
Thanks for this. Matching Java at ranks 0 and 1 is worthwhile, and the arithmetic in include_min_max() matches IncludeMinMax. CI passes, and the common, KLL and classic quantiles tests also pass for me locally under ASan and UBSan.
Timing: this changes the results users get, so it belongs in a minor release, not in the bug-fix release we're finishing now. We're holding other PRs until that release is out.
There are two things to settle before merging.
-
Sorted view lifetime. As you noted, for non-arithmetic types the view now points at the min and max inside the sketch object, so a view held across a move of the sketch no longer works. That fits the existing rule, as in Java, that a sorted view shouldn't be used after the sketch changes, but in C++ that rule isn't written down, and the failure is silent. Could you add it to the
get_sorted_view()documentation in both sketches, saying that for non-arithmetic types the view refers to the sketch's items and must not be used after the sketch is updated, merged, moved or destroyed? -
REQ. With this PR, KLL and classic quantiles match Java, but REQ doesn't. I'd like to see the REQ follow-up you offered planned alongside this, so all three sketches change in the same release. Moving REQ
get_rank()to the sorted view is a reasonable way to do it.
e714c8d to
8cd37cc
Compare
| T* begin(); | ||
| T* end(); | ||
|
|
||
| uint64_t compute_weight(const T& item, bool inclusive) const; |
There was a problem hiding this comment.
Removed as it's now unused.
0a1bef4 to
305ee6c
Compare
305ee6c to
8af7451
Compare
4bbf29f to
dc81617
Compare
Do you know approximately when this bug-fix release and this minor release are coming out? Order of magnitude
Done.
Actually I decided against this, and doing only for KLL. Let me know if you disagree. The reason I changed my mind, is that i reworked most of the PR (due to the performance issues), and now it's a bit more complex, so I think it's a bit easier to review it this way after the rework. |
|
@leerho I reworked the PR a bit, due to performance overheads the previous version was adding. It's a bit more complex now but adds no meaningfull overhead, so I think this is a lot better. |
|
Thanks for the rework, and for adding the lifetime notes. On timing: the bug-fix release (5.3.1) went out on Nov 13. The next one will be a minor release, probably 5.4.0, once we've landed the open PRs (we may have to draw a line somewhere). It comes before the larger refactor that will become 6.0.0. I can't give a firm date, because it depends on how fast those PRs land. I checked the new version against Java's On scope: KLL only is fine for this PR, since it keeps the review manageable. But Java includes the exact min and max in classic quantiles and REQ too, and I don't want a release where C++ KLL does this and the other two don't. So I'd like classic quantiles and REQ to follow as separate PRs before 5.4.0. If you can't do those, please let us know and we'll take them on. |
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
dc81617 to
7121fb3
Compare
I don't really use these 2 sketches (I only use KLL), so I would have to learn how they work and other details about them. So I would prefer if you take it on, that would be great :D
This is pushed now. |
leerho
left a comment
There was a problem hiding this comment.
Thanks for the rebase. The new head (7121fb3) has exactly the same tree as the merge I tested locally under ASan and UBSan.
And no problem: we’ll take on classic quantiles and REQ ourselves, as separate PRs before 5.4.0.
Approving. Thanks again for this.
What changed
kll_sketchkeeps the exact min and max of the stream. Butget_quantile(0)andget_quantile(1)read only the retained items, and compaction can drop the min and max from them. Then ranks 0 and 1 give a retained item, not the exact min and max:Since 6.0.0, datasketches-java returns the exact min and max here (apache/datasketches-java#547,
IncludeMinMax). Its sorted views add the min and max back when the retained items do not hold them. So for the same data, the two libraries give different answers at ranks 0 and 1.This PR ports that step to the C++ KLL sketch:
kll_sketch::get_sorted_view()adds a missing min and max, with the same arithmetic as Java. A missing min goes first, and the entry after it gives 1 of its weight to the min. A missing max goes last, and the entry before it gives 1 of its weight to the max. The total weight does not change. Soget_quantile(),get_rank(),get_PMF(),get_CDF()and the sorted view iterator all see the min and max, as in Java.quantiles_sorted_view::add_min()before the levels, so no entry has to move, andadd()does not merge the first level into the lone min. The newconvert_to_cummulative(max_item)takes the weight for the min and appends a missing max. The view reserves room for both, but only for a non-empty sketch.quantiles_sketchandreq_sketchdo not change. They still callconvert_to_cummulative(), so their sorted views are the same as on master.Behavior change: only ranks within 1/n of 0 or 1 give a different quantile, and a rank changes by at most 1/n. For example, the inclusive rank of a dropped min is now 1/n, not 0. These are the values that Java returns for the same data.
For non-arithmetic types, the view and the reference that
get_quantile()returns now can point to the min and max in the sketch object, not only to its item buffer. So a move of the sketch now also makes them invalid. The docs ofkll_sketch::get_sorted_view(),kll_sketch::get_quantile()andquantiles_sorted_viewnow say that, for non-arithmetic types, they must not be used after the sketch is updated, merged, assigned to, moved or destroyed.Performance
Benchmarks of
kll_sketch::get_sorted_view()against master measured no meaningful performance decrease.How tested
Added
quantiles_sorted_view_test(6 tests): each checks the entries and their cumulative weights.IncludeMinMaxTest), only the min missing, and only the max missing.add()(1 test): empty ranges first and between two ranges.kll_sketch_test(3 tests): each builds 1000 sketches, adds the min and max first so compaction drops them, and requires at least one sketch that dropped one.update(float and string = 2 tests): ranks 0 and 1 give the exact min and max. The float test also checks the Java ranks of a dropped min and max.deserializeandmerge(1 test): the same checks on a deserialized sketch and on its merge.Test strength: with
get_sorted_view()as on master, the 3 KLL tests fail. Withadd_min()and the max step ofconvert_to_cummulative(max_item)changed to do nothing, the 3 KLL tests and the 4 sorted view tests that need the min or max fail. "Both retained" and theadd()test pass, as expected.This pull request and its description were written by Isaac.