Repository navigation
EBPPS: keep the larger maximum weight after a merge - #803
Merged
Merged
Conversation
internalMerge() computed the maximum weight of the two sketches and used it for every rho step, but never stored it. After a merge, a later update computed rho with the target's own, smaller maximum, so the sample stopped being proportional to weight, and toByteArray() wrote the stale value. The swap path in merge() copied the same stale value. This is the Java counterpart of apache/datasketches-cpp#581, and the new test is a port of the C++ one. Also correct the profile name in a UtilityIO javadoc example (check_cpp_files, not check-cpp-files) and add its missing closing quote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The link pointed to a javaPath member that does not exist; it now links to getJavaPath(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EbppsItemsSketch.internalMerge()computednewWtMax = Math.max(wtMax_, other.wtMax_)and used it for every rho step, but never stored it. This is the Java counterpart of apache/datasketches-cpp#581.The bug
After a merge,
wtMax_was still the target's own maximum. That had three effects:update()computedrho = min(1/wtMax, k/cumWt)with the smaller, stale maximum, so rho could rise.downsample()ignores ratios ≥ 1, so the items already in the sample weren't rescaled and the sample stopped being proportional to weight.toByteArray()wrote the stale maximum, and a heapified sketch carried it forward.merge()merges into a copy and then takescopy.wtMax_, which was stale too.The fix
internalMerge()now ends withwtMax_ = newWtMax;. Nothing in the serialized format changes, and there are no EBPPS cross-language files.Test
The new
EbppsItemsSketchTest.mergeKeepsMaxWeightis a port of the C++ test: k = 10, 1,000 items of weight 1 and one item of weight 500. The reference is a single sketch fed all of those items followed by one more weight-1 item; its C is (1000 + 1 + 500) / 500 = 3.002. Each variant below is followed by one weight-1 update and must match the reference's n, cumulative weight and C:C depends only on the sequence of rho values, not on the random choices, so the test is deterministic. Without the fix, every variant gives C = 3.00666, the same value C++ saw.
Also
A javadoc example in the test utility
UtilityIOused the hyphenated profile name-Pcheck-cpp-files. Profile names use underscores, so it now says-Pcheck_cpp_files. The example's missing closing quote is fixed too. AnotherUtilityIOjavadoc linked to#javaPath, which doesn't exist; it now links togetJavaPath().The full suite passes (2,405 tests).
javac -Xlint:all -Xdoclint:all,-missingreports nothing in the changed files.🤖 Generated with Claude Code