Repository navigation
Quantiles items: keep the sketch consistent when an update throws - #805
Merged
Merged
Conversation
update() counted the item in n and the base buffer count before processing a full base buffer. If processing threw, the base buffer count stayed at 2k: later updates then either threw ArrayIndexOutOfBoundsException or silently wrote over level 0. update() also stored a new maximum before the minimum comparison, which could throw. update() now compares first and commits n, the base buffer count, min and max only after processing succeeds, as HeapUpdateDoublesSketch does. The full base buffer is sorted as a copy, so a comparator that throws during the sort leaves the base buffer as it was. The merge code now allocates all the space it needs before it changes the target. Results and the serialized form are unchanged. The class javadoc states which failures leave the sketch unchanged; a comparator that throws while levels are merged, or is inconsistent, leaves it undefined. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QuantilesItemsSketch.update()could leave the sketch broken if processing a full base buffer threw. This fixes that, and makes the merge code allocate before it changes the target.The problem
update()added 1 ton_andbaseBufferCount_before callingItemsUtil.processFullBaseBuffer(). That call can throw:OutOfMemoryErrorwhile growing the levels;ClassCastException,NullPointerException, or TimSort's "Comparison method violates its general contract!".The base buffer count then stayed at 2k. If the levels hadn't grown, every later update threw
ArrayIndexOutOfBoundsException. If they had, later updates silently wrote over level 0.update()also stored a new maximum before comparing the item with the minimum, so a comparator that threw on that second comparison left a maximum that was never counted.The fix
update()compares the item with the min and max first, and commitsn_, the base buffer count, min and max only after processing succeeds. This followsHeapUpdateDoublesSketch.update().processFullBaseBuffer()grows the levels first, then sorts a copy of the base buffer and propagates from that copy. A comparator that throws during the sort therefore leaves the base buffer as it was. The copy is 2k references, allocated once per full base buffer.ItemsMergeImpl.mergeIntoanddownSamplingMergeInto) now allocates the target's space and its work buffers before changing the target, so anOutOfMemoryErrorthere leaves the target as it was.QuantilesItemsSketchclass javadoc says which failures leave the sketch unchanged. If the comparator throws while sorted levels are being merged, or doesn't impose a consistent total order, the sketch's state is undefined. Making that case safe too would need scratch space for the whole carry chain on every propagation.Results and the serialized form don't change. A probe over many sketch sizes, values of k, and unions (including down-sampling ones) gave byte-identical images on main and on this branch.
Tests
The new
ItemsUpdateFailureTest:Both tests fail on main.
The full suite passes (2,431 tests).
javac -Xlint:all -Xdoclint:all,-missingreports nothing in the changed files.C++ classic quantiles has the same flaw; DS-CPP plans a fix there separately.
🤖 Generated with Claude Code