Skip to content

KLL: keep the sketch consistent when a single-item update fails - #806

Merged
proost merged 1 commit into
mainfrom
kll-update-failure-safety
Nov 19, 2026
Merged

proost merged 1 commit into
mainfrom
kll-update-failure-safety

Conversation

@leerho

@leerho leerho commented Nov 19, 2026

Copy link
Copy Markdown
Member

This makes a failed single-item update leave a KLL sketch unchanged. It's the KLL part of the same review that produced #805 for classic quantiles. A REQ PR will follow.

Fixes

  • Direct sketches could corrupt their segment. When a direct doubles, floats or longs sketch needed a new level, addEmptyTopLevelToCompletelyFullSketch first wrote numLevels + 1 into the current segment (incNumLevels()), then asked the MemorySegmentRequest for a larger segment. If that request failed, the segment was left with one level too many over the old levels and items.
    • The early write is removed. setNumLevels() at the end of the same method already writes the right value.
    • incNumLevels() had no other callers, so it's removed from KllSketch and its subclasses. Three read-only tests that called it now call setNumLevels() instead.
  • Min and max were set too early. A single-item update set them before compaction, which can fail (an OutOfMemoryError, a refused segment request, or a throwing comparator). They could then include an item that was never counted. They're now set after compaction.
    • The items sketch compares first and stores the new min and max only at the end.
    • The weighted items update no longer sets them before its merge, because the merge already does that.
  • Items comparator during compaction. The compaction already sorts and merges a copy of the items array, so a throwing comparator left the items and levels as they were. However, if a new top level had just been added, it stayed. The sketch was still valid but compacted differently from then on. That level is now removed again. The pre-growth arrays are saved only when a level is added, which happens O(log n) times.
  • Merge order. The heap merge could overwrite the current items array in place before allocating the new levels array. The levels array is now allocated first, in all four helpers.

The KllSketch class javadoc now says which failures leave the sketch unchanged: a single-item update that throws because storage can't be allocated, the segment request fails, or the comparator throws. It also says that weighted and vector updates, and merges, may have applied part of their input when they throw.

Results and the serialized form don't change. A probe over many sizes, values of k and merges gave byte-identical images on main and on this branch. It covered heap doubles, floats, longs and items sketches, and a direct doubles sketch.

Tests

The new KllUpdateFailureTest:

  • For direct doubles, floats and longs sketches, uses a MemorySegmentRequest that refuses to grow. The update that needs a new level throws. The segment is then byte-identical to before that update, min and max are unchanged, and heapifying it gives one level.
  • For the items sketch, makes the comparator throw while level 0 is sorted for compaction. N, min and max are unchanged. After many more updates, the sketch is byte-identical to one that never saw that item.

All four tests fail on main.

The full suite passes (2,433 tests). javac -Xlint:all -Xdoclint:all,-missing reports nothing in the changed files.

🤖 Generated with Claude Code

- A direct doubles, floats or longs sketch wrote numLevels + 1 into its
  segment before asking for a larger one. If that request failed, the
  segment was left with one level too many. The early write
  (incNumLevels) is removed; setNumLevels at the end already writes the
  right value, so incNumLevels is gone.
- A single-item update set min and max before compaction, which can
  fail. They are now set after it.
- An items sketch whose comparator throws during compaction now removes
  the top level it had just added, so the sketch is unchanged.
- Merge allocates the new levels array before it may overwrite the
  current items array.

Results and the serialized form are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@proost
proost merged commit c6e809e into main Nov 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants