Skip to content

REQ: allocate compactor storage on first use, and add reserve() - #575

Merged
proost merged 6 commits into
masterfrom
req-lazy-storage
Nov 19, 2026
Merged

proost merged 6 commits into
masterfrom
req-lazy-storage

Conversation

@leerho

@leerho leerho commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

This is the REQ part of the change agreed on dev@: sketches allocate their item storage on first use, and reserve() allocates it up front (https://lists.apache.org/thread/d9pzln7m5q1of82ghhotrr3h0j6p514m). The KLL and t-digest parts are in #558 and #561.

Allocation on first use

  • A req_compactor no longer allocates its item storage in its constructor. It allocates the same 2 * nominal capacity on its first append, merge or compaction into it. Once storage is in use, the memory footprint is the same as before. A REQ sketch that stays empty, or that is only merged from empty sketches, holds no item storage.
  • New req_sketch::reserve() allocates the storage of the first compactor right after construction.
  • update() reserves that storage before it touches min and max, so if the first update fails to allocate, the sketch is unchanged.
  • The class documentation, the constructor, update() and merge() describe when the allocation happens.
  • Results, the serialized format and cross-language compatibility are unchanged. The REQ tests pass with SERDE_COMPAT against the Java-generated req_* files.

Existing problems found by the new tests

The allocation-failure tests found four problems that are already on master. They are fixed here.

  1. The iterator didn't skip levels without items. begin() always started at level 0. For an empty sketch, begin() != end(), and the loop ran past the item storage without ever reaching end(). The iterator now skips levels that have no items.

  2. A failed allocation could leave the counters out of step with the compactors.

    • A compaction moves items and can then grow the compactor for its new sections.
    • A merge combines compactors one level at a time and only adds to n at the end.

    If an allocation failed in between, n, the number of retained items and the nominal size no longer matched the items. They are now recomputed from the compactors when an exception passes through. This relies on n always equaling the total weight of the retained items. The sorted view is now reset before anything changes.

  3. update() compressed only when the number of retained items was exactly the nominal size. After a failed compression, the count could pass that size, and the sketch then never compressed again. It now compresses at or above that size, as merge() already did. Without a failure, the two conditions behave the same.

  4. optional::emplace() didn't destroy the current value. Unlike std::optional::emplace, the library's optional built the new value on top of the old one. The sketches only call emplace() for min and max when the sketch is empty, so this never mattered before. After a failed merge into an empty sketch, though, min and max are already set, and the next update emplaced them again, leaking the old values. LeakSanitizer found this in CI. emplace() now destroys the current value first, and optional_test.cpp has a test for it.

Tests

The new req_sketch_allocation_test.cpp uses an allocator that fails on purpose. Each test runs for both HRA and LRA.

  • reserve() allocates the expected storage, and updates up to the nominal capacity allocate nothing more.
  • An empty sketch, a copy of it, and a merge from it hold no item storage.
  • A failed reserve() or first update leaves the sketch unchanged and usable.
  • Failing each allocation of 5,000 updates, in turn, leaves the sketch consistent: the retained weights add up to n. It also stays usable and keeps compressing as updates continue. With the old == condition, this test fails with 20,072 retained items, against about 950 normally.
  • Failing each allocation of a merge, in turn, for several target and source sizes, leaves the target consistent and usable and the source unchanged.
  • Iterating an empty sketch, or a merged sketch, visits exactly the retained items.

The REQ suite passes under ASan and UBSan. This PR merges cleanly with all open PRs, including #567, which also changes REQ.

🤖 Generated with Claude Code

A new compactor no longer allocates its item storage in its constructor.
It allocates the same 2 * nominal capacity on its first append, merge or
compaction into it, so a REQ sketch that stays empty holds no item storage.
req_sketch::reserve() allocates the storage of the first compactor up front.
update() reserves it before it touches min and max, so a failed first
update leaves the sketch unchanged.

Allocation-failure tests found three existing problems, fixed here:
- The iterator did not skip levels without items. Iterating an empty
  sketch never reached end() and read past the item storage.
- An allocation that failed after a compaction or partway through a merge
  left n, the number of retained items and the nominal size out of step
  with the compactors. They are now recomputed from the compactors, and
  the sorted view is reset before anything changes.
- update() compressed only when the number of retained items was exactly
  the nominal size. After a failed compression it could pass that size and
  never compress again. It now compresses at or above it, as merge() does.

The new tests use an allocator that fails on purpose, at each allocation
of reserve(), a stream of updates and a merge, and check that the sketch
stays consistent and usable and keeps compressing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
emplace() constructed the new value over the current one without
destroying it. The sketches call emplace() for min and max only when
they are empty, so this didn't matter until a REQ merge into an empty
sketch failed after setting min and max: the next update emplaced them
again and leaked the old values. LeakSanitizer found it in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coveralls

coveralls commented Nov 12, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 44437582621

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Warning

No base build found for commit 386f335 on master.
Coverage changes can't be calculated without a base build.
If a base build is processing, this comment will update automatically when it completes.

Coverage: 96.874%

Details

  • Patch coverage: 55 of 55 lines across 3 files are fully covered (100%).

Uncovered Changes

No uncovered changes found.

Coverage Regressions

Requires a base build to compare against. How to fix this →


Coverage Stats

Coverage Status
Relevant Lines: 17117
Covered Lines: 16582
Line Coverage: 96.87%
Coverage Strength: 1884997.25 hits per line

💛 - Coveralls

Comment thread req/include/req_compactor_impl.hpp Resolved
ensure_space(0) returns before reserve(), as suggested in review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested a review from proost November 15, 2026 23:37
Both deserializers, from bytes and from a stream, must allocate only what
a newly built empty sketch does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compactors_[i].merge(conditional_forward<FwdSk>(other.compactors_[i]));
}
} catch (...) {
restore_counts(); // some levels may be merged already

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In here, we should remove last compactor. When grow is success and merge failed, then sketch's state is invalid.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks. If grow() succeeded and a compactor merge then failed, the added levels stayed behind empty. The sketch then reported estimation mode with nothing above level 0, and its rank bounds used too many levels. For example, an empty target merging a 1,000-item source and failing at its second allocation was left with n = 0 but in estimation mode.

Fixed in ccbcf6a: on failure, merge() now removes the levels it added that received no items. It also grows before it widens min and max, so a failure to grow leaves the sketch unchanged. The test "failure at each allocation of a merge" now checks that a failed merge which merged no items leaves the target's estimation mode as it was; it fails without the change.

If merge() grew the sketch and then failed, the new levels stayed behind
empty, so a sketch could report estimation mode with no items above level
0, and its rank bounds used too many levels. Remove the added levels that
received no items. Grow before widening min and max, so that a failure to
grow changes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested a review from proost November 17, 2026 22:42
try {
compress_levels();
} catch (...) {
restore_counts(); // a compactor can fail to grow after a compaction moved its items

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In here we should also reset compactors. Because compress_levels call grow

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, same problem as in merge(): compress_levels() can add a level and then fail to compact into it, when the new level fails to allocate space for the promoted items. That left the sketch in estimation mode with an empty top level.

Fixed in 1267c75: on failure compress() now removes the levels it added that hold no items, the same way merge() does. Both use a new helper, remove_empty_levels_above(). The allocation tests now also check that a sketch in estimation mode retains an item with weight above 1. "failure at each allocation of the updates" fails that check without the change.

compress() can add a level and then fail to compact into it, which left
the sketch in estimation mode with an empty top level. On failure it now
removes the levels it added that hold no items, like merge() does; both
use a new helper, remove_empty_levels_above().

The allocation tests now check that a sketch in estimation mode retains
an item of weight above 1, which fails without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested a review from proost November 19, 2026 01:31

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@proost
proost merged commit c3ba705 into master Nov 19, 2026
35 checks passed
@proost
proost deleted the req-lazy-storage branch November 19, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants