Repository navigation
REQ: allocate compactor storage on first use, and add reserve() - #575
Conversation
A new compactor no longer allocates its item storage in its constructor. It allocates the same 2 * nominal capacity on its first append, merge or compaction into it, so a REQ sketch that stays empty holds no item storage. req_sketch::reserve() allocates the storage of the first compactor up front. update() reserves it before it touches min and max, so a failed first update leaves the sketch unchanged. Allocation-failure tests found three existing problems, fixed here: - The iterator did not skip levels without items. Iterating an empty sketch never reached end() and read past the item storage. - An allocation that failed after a compaction or partway through a merge left n, the number of retained items and the nominal size out of step with the compactors. They are now recomputed from the compactors, and the sorted view is reset before anything changes. - update() compressed only when the number of retained items was exactly the nominal size. After a failed compression it could pass that size and never compress again. It now compresses at or above it, as merge() does. The new tests use an allocator that fails on purpose, at each allocation of reserve(), a stream of updates and a merge, and check that the sketch stays consistent and usable and keeps compressing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
emplace() constructed the new value over the current one without destroying it. The sketches call emplace() for min and max only when they are empty, so this didn't matter until a REQ merge into an empty sketch failed after setting min and max: the next update emplaced them again and leaked the old values. LeakSanitizer found it in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage Report for CI Build 44437582621Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Warning No base build found for commit Coverage: 96.874%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
ensure_space(0) returns before reserve(), as suggested in review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both deserializers, from bytes and from a stream, must allocate only what a newly built empty sketch does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| compactors_[i].merge(conditional_forward<FwdSk>(other.compactors_[i])); | ||
| } | ||
| } catch (...) { | ||
| restore_counts(); // some levels may be merged already |
There was a problem hiding this comment.
In here, we should remove last compactor. When grow is success and merge failed, then sketch's state is invalid.
There was a problem hiding this comment.
You're right, thanks. If grow() succeeded and a compactor merge then failed, the added levels stayed behind empty. The sketch then reported estimation mode with nothing above level 0, and its rank bounds used too many levels. For example, an empty target merging a 1,000-item source and failing at its second allocation was left with n = 0 but in estimation mode.
Fixed in ccbcf6a: on failure, merge() now removes the levels it added that received no items. It also grows before it widens min and max, so a failure to grow leaves the sketch unchanged. The test "failure at each allocation of a merge" now checks that a failed merge which merged no items leaves the target's estimation mode as it was; it fails without the change.
If merge() grew the sketch and then failed, the new levels stayed behind empty, so a sketch could report estimation mode with no items above level 0, and its rank bounds used too many levels. Remove the added levels that received no items. Grow before widening min and max, so that a failure to grow changes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| try { | ||
| compress_levels(); | ||
| } catch (...) { | ||
| restore_counts(); // a compactor can fail to grow after a compaction moved its items |
There was a problem hiding this comment.
In here we should also reset compactors. Because compress_levels call grow
There was a problem hiding this comment.
Right, same problem as in merge(): compress_levels() can add a level and then fail to compact into it, when the new level fails to allocate space for the promoted items. That left the sketch in estimation mode with an empty top level.
Fixed in 1267c75: on failure compress() now removes the levels it added that hold no items, the same way merge() does. Both use a new helper, remove_empty_levels_above(). The allocation tests now also check that a sketch in estimation mode retains an item with weight above 1. "failure at each allocation of the updates" fails that check without the change.
compress() can add a level and then fail to compact into it, which left the sketch in estimation mode with an empty top level. On failure it now removes the levels it added that hold no items, like merge() does; both use a new helper, remove_empty_levels_above(). The allocation tests now check that a sketch in estimation mode retains an item of weight above 1, which fails without this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This is the REQ part of the change agreed on dev@: sketches allocate their item storage on first use, and
reserve()allocates it up front (https://lists.apache.org/thread/d9pzln7m5q1of82ghhotrr3h0j6p514m). The KLL and t-digest parts are in #558 and #561.Allocation on first use
req_compactorno longer allocates its item storage in its constructor. It allocates the same2 * nominal capacityon its first append, merge or compaction into it. Once storage is in use, the memory footprint is the same as before. A REQ sketch that stays empty, or that is only merged from empty sketches, holds no item storage.req_sketch::reserve()allocates the storage of the first compactor right after construction.update()reserves that storage before it touches min and max, so if the first update fails to allocate, the sketch is unchanged.update()andmerge()describe when the allocation happens.SERDE_COMPATagainst the Java-generatedreq_*files.Existing problems found by the new tests
The allocation-failure tests found four problems that are already on master. They are fixed here.
The iterator didn't skip levels without items.
begin()always started at level 0. For an empty sketch,begin() != end(), and the loop ran past the item storage without ever reachingend(). The iterator now skips levels that have no items.A failed allocation could leave the counters out of step with the compactors.
nat the end.If an allocation failed in between,
n, the number of retained items and the nominal size no longer matched the items. They are now recomputed from the compactors when an exception passes through. This relies onnalways equaling the total weight of the retained items. The sorted view is now reset before anything changes.update()compressed only when the number of retained items was exactly the nominal size. After a failed compression, the count could pass that size, and the sketch then never compressed again. It now compresses at or above that size, asmerge()already did. Without a failure, the two conditions behave the same.optional::emplace()didn't destroy the current value. Unlikestd::optional::emplace, the library'soptionalbuilt the new value on top of the old one. The sketches only callemplace()for min and max when the sketch is empty, so this never mattered before. After a failed merge into an empty sketch, though, min and max are already set, and the next update emplaced them again, leaking the old values. LeakSanitizer found this in CI.emplace()now destroys the current value first, andoptional_test.cpphas a test for it.Tests
The new
req_sketch_allocation_test.cppuses an allocator that fails on purpose. Each test runs for both HRA and LRA.reserve()allocates the expected storage, and updates up to the nominal capacity allocate nothing more.reserve()or first update leaves the sketch unchanged and usable.n. It also stays usable and keeps compressing as updates continue. With the old==condition, this test fails with 20,072 retained items, against about 950 normally.The REQ suite passes under ASan and UBSan. This PR merges cleanly with all open PRs, including #567, which also changes REQ.
🤖 Generated with Claude Code