Skip to content

Sampling: use a bounded-long draw for reservoir updates - #804

Open
hutiefang76 wants to merge 2 commits into
apache:mainfrom
hutiefang76:frank/sampling647_single_draw_20261118
Open

hutiefang76 wants to merge 2 commits into
apache:mainfrom
hutiefang76:frank/sampling647_single_draw_20261118

Conversation

@hutiefang76

@hutiefang76 hutiefang76 commented Nov 18, 2026 •

Copy link
Copy Markdown

Part of #647: use one bounded-long draw in the steady-state update of both ReservoirItemsSketch and ReservoirLongsSketch. A value below k both accepts the item and selects the replacement slot, giving each slot replacement probability 1/n.

The 48-bit item-count limit, serialized format, initial fill, and capacity/null-check ordering are unchanged. The weighted union algorithms are outside this change. Thirteen boundary cases for ReservoirItemsSketch cover counts around 32 bits and the 48-bit limit, sample membership, serialization round-trips, and unchanged state after a rejected update. The null-at-capacity assertion pins the current check ordering.

Validation: OpenJDK 25.0.5 and Maven 3.10.0. Full clean verify passed with 2,442 tests, zero failures, errors, or skips. Javadoc and Apache RAT also passed. The default POM excludes the special cross-language groups. The linked checkout used the native Git backend via -Dmaven.gitcommitid.nativegit=true.

Generated with assistance from DeepSeek (deepseek-v4-pro) and OpenAI Codex for the code, tests, review, and PR text.

Reuse the accepted bounded-long value as the replacement index.
Cover long item-count boundaries and preserved serialization/capacity behavior.

Generated-by: DeepSeek (deepseek-v4-pro)
Generated-by: OpenAI Codex

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The change is correct: a single nextLong(n) draw in [0, n) accepts the item with probability exactly k/n and picks a uniform slot when it does, so this is still standard reservoir sampling (Algorithm R). It halves the random draws for accepted items and removes the small rounding error of the old double-based comparison. The item count limit and the serialized format are unchanged.

I ran the reservoir tests on this branch locally, and they pass. I also ran a quick uniformity check (k = 5, n = 40, 400,000 trials). The chi-square statistic was 39.8 with 39 degrees of freedom, as expected for a uniform sample.

Requested change

  • ReservoirLongsSketch.update() has the same two-draw code (ReservoirLongsSketch.java, lines 325-326). Please apply the same change there in this PR, so the two reservoir sketches stay consistent. The weighted draws in ReservoirItemsUnion and ReservoirLongsUnion are different and can stay as they are.

Comments, no change required

  • The new tests check invariants (the item count, the sample size, no duplicates, the serialization round trip, the capacity limit), so they would also pass on main. That's fine for a change that keeps the same distribution. A small statistical check of uniform inclusion would be a welcome addition, but it isn't required.
  • checkNearMaxItemSeenCapacity asserts that update(null) throws once the sketch is at capacity. That is the current behavior, since the capacity check comes before the null check, but the test makes it look intended. Please add a short comment saying it pins the current behavior, so it can change later without looking like a regression.
  • This covers only the first point of #647. The issue should stay open for the rest.

Match the item reservoir update and clarify the null-at-capacity test.

Generated-by: DeepSeek (deepseek-v4-pro)
Generated-by: OpenAI Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants