Skip to content

EBPPS: keep the larger maximum weight after a merge - #803

Merged
proost merged 2 commits into
mainfrom
ebpps-merge-wtmax
Nov 18, 2026
Merged

proost merged 2 commits into
mainfrom
ebpps-merge-wtmax

Conversation

@leerho

@leerho leerho commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

EbppsItemsSketch.internalMerge() computed newWtMax = Math.max(wtMax_, other.wtMax_) and used it for every rho step, but never stored it. This is the Java counterpart of apache/datasketches-cpp#581.

The bug

After a merge, wtMax_ was still the target's own maximum. That had three effects:

  • Proportionality: the next update() computed rho = min(1/wtMax, k/cumWt) with the smaller, stale maximum, so rho could rise. downsample() ignores ratios ≥ 1, so the items already in the sample weren't rescaled and the sample stopped being proportional to weight.
  • Serialization: toByteArray() wrote the stale maximum, and a heapified sketch carried it forward.
  • Swap path: when the argument has the larger cumulative weight, merge() merges into a copy and then takes copy.wtMax_, which was stale too.

The fix

internalMerge() now ends with wtMax_ = newWtMax;. Nothing in the serialized format changes, and there are no EBPPS cross-language files.

Test

The new EbppsItemsSketchTest.mergeKeepsMaxWeight is a port of the C++ test: k = 10, 1,000 items of weight 1 and one item of weight 500. The reference is a single sketch fed all of those items followed by one more weight-1 item; its C is (1000 + 1 + 500) / 500 = 3.002. Each variant below is followed by one weight-1 update and must match the reference's n, cumulative weight and C:

  1. The 1,000-item sketch merges the heavy sketch.
  2. The heavy sketch merges the 1,000-item sketch (the swap path).
  3. Variants 1 and 2 again, each after a serialize and heapify round trip.

C depends only on the sequence of rho values, not on the random choices, so the test is deterministic. Without the fix, every variant gives C = 3.00666, the same value C++ saw.

Also

A javadoc example in the test utility UtilityIO used the hyphenated profile name -Pcheck-cpp-files. Profile names use underscores, so it now says -Pcheck_cpp_files. The example's missing closing quote is fixed too. Another UtilityIO javadoc linked to #javaPath, which doesn't exist; it now links to getJavaPath().

The full suite passes (2,405 tests). javac -Xlint:all -Xdoclint:all,-missing reports nothing in the changed files.

🤖 Generated with Claude Code

internalMerge() computed the maximum weight of the two sketches and used
it for every rho step, but never stored it. After a merge, a later update
computed rho with the target's own, smaller maximum, so the sample stopped
being proportional to weight, and toByteArray() wrote the stale value.
The swap path in merge() copied the same stale value.

This is the Java counterpart of apache/datasketches-cpp#581, and the new
test is a port of the C++ one.

Also correct the profile name in a UtilityIO javadoc example
(check_cpp_files, not check-cpp-files) and add its missing closing quote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The link pointed to a javaPath member that does not exist; it now links
to getJavaPath().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@proost
proost merged commit 34f997f into main Nov 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants