Repository navigation
fix: ebpps_sketch::merge() keeps the larger maximum weight - #581
Conversation
internal_merge() computed new_wt_max and used it for every rho step but never stored it, so after merging a sketch that held the heaviest item the target kept its old wt_max. Later updates then used a stale maximum, which breaks proportional sampling, and serialize() wrote the stale value. Store new_wt_max with the final cumulative weight and n, and add a test that compares C against a single sketch for all four merge paths.
leerho
left a comment
There was a problem hiding this comment.
Thanks for the clear write-up. The fix is right: internal_merge() uses new_wt_max for every rho step but never stores it, so after a merge wt_max_ keeps the target's own maximum, and the next update() can raise rho without scaling up the items already in the sample. Storing it next to the final cumulative weight and n covers every merge() overload, including the swap paths, since they all end in internal_merge().
I merged this into current master locally and ran the EBPPS tests under ASan and UBSan: all 15 test cases pass. Against master's code, with the four checks made non-fatal, all four cases fail with C = 3.00666 against 3.002, so the test catches the bug in each merge path.
I agree that Java's EbppsItemsSketch.internalMerge has the same omission. We'll follow up on that in datasketches-java.
internalMerge() computed the maximum weight of the two sketches and used it for every rho step, but never stored it. After a merge, a later update computed rho with the target's own, smaller maximum, so the sample stopped being proportional to weight, and toByteArray() wrote the stale value. The swap path in merge() copied the same stale value. This is the Java counterpart of apache/datasketches-cpp#581, and the new test is a port of the C++ one. Also correct the profile name in a UtilityIO javadoc example (check_cpp_files, not check-cpp-files) and add its missing closing quote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ebpps_sketch::internal_merge()computesnew_wt_max = max(wt_max_, sk.wt_max_)and uses it for every rho step of the merge, but never stores it. After the merge,wt_max_is still the value of whichever sketch ends up as the target, so if the heavier item came from the other sketch the sketch forgets it.The next
update()then computesrho = min(1 / wt_max_, k / cum_wt)with the stale maximum, so rho can go up.ebpps_sample::downsample()ignores ratios at or above 1, so the items already in the sample are not scaled up to match, and from then on the sample is no longer proportional to weight. The stale value is also written byserialize(), so a deserialized sketch carries it forward.Example with k = 10: one sketch gets 1000 items of weight 1, another gets one item of weight 500, and the second is merged into the first. Then 2000 more items of weight 1 are added. A single sketch fed the same 3001 items always keeps the weight-500 item (its inclusion probability is min(1, rho * 500) = 1) and keeps every weight-1 item with probability 0.002. In one set of 2000 runs of the merged version, the weight-500 item was kept in 44% of the runs, the weight-1 items from before the merge were kept at a rate of 0.00086 each, and the ones added after the merge at 0.00285.
The same thing happens when the heavy item is in the target sketch and the other sketch has the larger cumulative weight, because the two sketches are swapped before
internal_merge()runs. Both the lvalue and rvalue overloads are affected.The fix stores
new_wt_maxtogether with the final cumulative weight and n. Since everymerge()overload goes throughinternal_merge(), this one line covers all of them.The new test merges a weight-500 item with 1000 weight-1 items in all four ways (lvalue and rvalue, with and without the swap), adds one more weight-1 item, and checks C against a single sketch fed the same items. C depends only on the sequence of rho values and not on the random choices, so the check is deterministic. Before the fix C is 3.00666 in all four cases. The single-sketch value, and the value after the fix, is 3.002.
The Java implementation (
EbppsItemsSketch.internalMerge) has the same omission. There are no cross-language EBPPS fixtures, so no generated files change.This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.