Skip to content

fix: ebpps_sketch::merge() keeps the larger maximum weight - #581

Merged
leerho merged 1 commit into
apache:masterfrom
0z5a:fix/ebpps-merge-wt-max
Nov 15, 2026
Merged

leerho merged 1 commit into
apache:masterfrom
0z5a:fix/ebpps-merge-wt-max

Conversation

@0z5a

@0z5a 0z5a commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

ebpps_sketch::internal_merge() computes new_wt_max = max(wt_max_, sk.wt_max_) and uses it for every rho step of the merge, but never stores it. After the merge, wt_max_ is still the value of whichever sketch ends up as the target, so if the heavier item came from the other sketch the sketch forgets it.

The next update() then computes rho = min(1 / wt_max_, k / cum_wt) with the stale maximum, so rho can go up. ebpps_sample::downsample() ignores ratios at or above 1, so the items already in the sample are not scaled up to match, and from then on the sample is no longer proportional to weight. The stale value is also written by serialize(), so a deserialized sketch carries it forward.

Example with k = 10: one sketch gets 1000 items of weight 1, another gets one item of weight 500, and the second is merged into the first. Then 2000 more items of weight 1 are added. A single sketch fed the same 3001 items always keeps the weight-500 item (its inclusion probability is min(1, rho * 500) = 1) and keeps every weight-1 item with probability 0.002. In one set of 2000 runs of the merged version, the weight-500 item was kept in 44% of the runs, the weight-1 items from before the merge were kept at a rate of 0.00086 each, and the ones added after the merge at 0.00285.

The same thing happens when the heavy item is in the target sketch and the other sketch has the larger cumulative weight, because the two sketches are swapped before internal_merge() runs. Both the lvalue and rvalue overloads are affected.

The fix stores new_wt_max together with the final cumulative weight and n. Since every merge() overload goes through internal_merge(), this one line covers all of them.

The new test merges a weight-500 item with 1000 weight-1 items in all four ways (lvalue and rvalue, with and without the swap), adds one more weight-1 item, and checks C against a single sketch fed the same items. C depends only on the sequence of rho values and not on the random choices, so the check is deterministic. Before the fix C is 3.00666 in all four cases. The single-sketch value, and the value after the fix, is 3.002.

The Java implementation (EbppsItemsSketch.internalMerge) has the same omission. There are no cross-language EBPPS fixtures, so no generated files change.

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

internal_merge() computed new_wt_max and used it for every rho step but
never stored it, so after merging a sketch that held the heaviest item
the target kept its old wt_max. Later updates then used a stale maximum,
which breaks proportional sampling, and serialize() wrote the stale value.
Store new_wt_max with the final cumulative weight and n, and add a test
that compares C against a single sketch for all four merge paths.

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clear write-up. The fix is right: internal_merge() uses new_wt_max for every rho step but never stores it, so after a merge wt_max_ keeps the target's own maximum, and the next update() can raise rho without scaling up the items already in the sample. Storing it next to the final cumulative weight and n covers every merge() overload, including the swap paths, since they all end in internal_merge().

I merged this into current master locally and ran the EBPPS tests under ASan and UBSan: all 15 test cases pass. Against master's code, with the four checks made non-fatal, all four cases fail with C = 3.00666 against 3.002, so the test catches the bug in each merge path.

I agree that Java's EbppsItemsSketch.internalMerge has the same omission. We'll follow up on that in datasketches-java.

@leerho
leerho merged commit 103e1e2 into apache:master Nov 15, 2026
21 checks passed
proost pushed a commit to proost/datasketches-java that referenced this pull request Nov 19, 2026
internalMerge() computed the maximum weight of the two sketches and used
it for every rho step, but never stored it. After a merge, a later update
computed rho with the target's own, smaller maximum, so the sample stopped
being proportional to weight, and toByteArray() wrote the stale value.
The swap path in merge() copied the same stale value.

This is the Java counterpart of apache/datasketches-cpp#581, and the new
test is a port of the C++ one.

Also correct the profile name in a UtilityIO javadoc example
(check_cpp_files, not check-cpp-files) and add its missing closing quote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants