Repository navigation
KLL: keep the sketch consistent when a single-item update fails - #806
Merged
Merged
Conversation
- A direct doubles, floats or longs sketch wrote numLevels + 1 into its segment before asking for a larger one. If that request failed, the segment was left with one level too many. The early write (incNumLevels) is removed; setNumLevels at the end already writes the right value, so incNumLevels is gone. - A single-item update set min and max before compaction, which can fail. They are now set after it. - An items sketch whose comparator throws during compaction now removes the top level it had just added, so the sketch is unchanged. - Merge allocates the new levels array before it may overwrite the current items array. Results and the serialized form are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes a failed single-item update leave a KLL sketch unchanged. It's the KLL part of the same review that produced #805 for classic quantiles. A REQ PR will follow.
Fixes
addEmptyTopLevelToCompletelyFullSketchfirst wrote numLevels + 1 into the current segment (incNumLevels()), then asked theMemorySegmentRequestfor a larger segment. If that request failed, the segment was left with one level too many over the old levels and items.setNumLevels()at the end of the same method already writes the right value.incNumLevels()had no other callers, so it's removed fromKllSketchand its subclasses. Three read-only tests that called it now callsetNumLevels()instead.OutOfMemoryError, a refused segment request, or a throwing comparator). They could then include an item that was never counted. They're now set after compaction.The
KllSketchclass javadoc now says which failures leave the sketch unchanged: a single-item update that throws because storage can't be allocated, the segment request fails, or the comparator throws. It also says that weighted and vector updates, and merges, may have applied part of their input when they throw.Results and the serialized form don't change. A probe over many sizes, values of k and merges gave byte-identical images on main and on this branch. It covered heap doubles, floats, longs and items sketches, and a direct doubles sketch.
Tests
The new
KllUpdateFailureTest:MemorySegmentRequestthat refuses to grow. The update that needs a new level throws. The segment is then byte-identical to before that update, min and max are unchanged, and heapifying it gives one level.All four tests fail on main.
The full suite passes (2,433 tests).
javac -Xlint:all -Xdoclint:all,-missingreports nothing in the changed files.🤖 Generated with Claude Code