Skip to content

Make HLL sketch and union exception-safe on allocation failure - #565

Merged
leerho merged 4 commits into
apache:masterfrom
clee704:hll-exception-safety
Nov 18, 2026
Merged

leerho merged 4 commits into
apache:masterfrom
clee704:hll-exception-safety

Conversation

@clee704

@clee704 clee704 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #551.

Prevent allocation failures from leaking HLL storage or leaving sketches and
unions unusable. Guard unfinished objects, roll back failed insertions, and
commit HLL_4 auxiliary-map changes only after allocation succeeds.

Sketch updates, copy assignment and reset leave the sketch unchanged on
allocation failure. Merging LIST/SET sketches into LIST/SET unions may retain
coupons already inserted; the union remains usable and the merge can be
retried. Other union updates, copy assignment and reset leave it unchanged.

Also fix self-assignment and malformed auxiliary-map deserialization cleanup.
Reject overfull compact SET images and orphan HLL_4 exception tokens.

Tests inject allocation failures across construction, updates, copying,
conversion, deserialization and union merges, checking cleanup, rollback and
allocator ownership. Reported sanitizer and C++/Java serialization comparisons
pass.

Performance: Previously measured shared-host timing increases:

Workload GCC 13.3 Clang 18.1
LIST/SET copy assignment, aggregate medians 2-7% 6.5-17.5%
Selected union workloads Up to 2.8% Up to 10.5%
HLL_8 stream, 20 million updates, lg12 Not reported About 3%

Some Clang SET-assignment batches reached about 19%. These observations are
not bounds; timing causes remain unverified.

This change was developed with assistance from GitHub Copilot.

clee704 and others added 2 commits November 3, 2026 00:42
An allocation failure inside an HLL sketch or union could leak memory,
leave a sketch pointing to freed memory, or leave the sketch or union
changed or internally inconsistent.

Allocate objects through a helper that keeps the allocator's pointer
type until construction succeeds and frees storage if the constructor
throws. Use it for construction, copies, conversions, promotions and
deserialization. Use copy-and-swap for sketch and union assignment,
including self-assignment. Roll back LIST, SET and AuxHashMap insertions
when growth fails, and update the auxiliary table size only after the
larger table is in place.

Copy auxiliary maps with an explicit source allocator for both the
object and its entries. Preserve table order even when container copy
selection would choose an unequal allocator.

Move LIST and SET growth and promotion, with their rollback, into
non-inlined functions so the ordinary coupon path stays small. Mark the
sketch constructor and object helper inline to retain inexpensive LIST
construction.

In HLL_4, create and fill the auxiliary map before writing AUX_TOKEN.
Keep auxiliary updates out of the inlined register path. Check for the
last minimum register before changing it. On the ordinary non-auxiliary
update path, update the estimator before the non-allocating register
writes. Validate and decrement registers in one pass per curMin step,
restoring the written registers if a consistency check or allocation
throws. Keep replacement auxiliary maps owned until they are installed.
During conversion, use the same update path with the iterator's slot
directly; the delegating constructor cleans up if conversion throws.

Merge sparse source coupons through a local implementation pointer,
updating the union's owning pointer after each promotion.
Retain the union's allocator for inputs with unequal allocators,
including guarded HLL copies and downsampling at the smaller precision.
Sweep those paths with per-allocator accounting and check source-pool
lifetimes.

The sketch documentation now states: "Allocation failures propagate
without leaking storage. Updates, copy assignment and reset leave the
sketch unchanged if an allocation fails."

The union documentation now states: "Allocation failures propagate
without leaking storage. If an allocation fails while a LIST or SET
sketch is merged into a union in LIST or SET mode, the union stays
valid and usable, holding the coupons inserted before the failure.
Repeating the merge gives the result of an uninterrupted merge. All
other updates, copy assignment and reset leave the union unchanged if
an allocation fails."

This also fixes a heap buffer overflow when serializing after failed
auxiliary growth, and a use-after-free when stream deserialization finds
the wrong number of auxiliary entries.

Reject overfull compact SET images before replaying their coupons, so
replay cannot lose a promoted implementation. Check the SET precision
before computing the compact count limit.

Add allocation-failure sweeps that check cleanup, unchanged destinations
where guaranteed, continued use and merge retries. Compare allocator
pointer and raw-pointer results, and check construction-failure cleanup.
Check null construction, assignment and comparisons in both operand
orders. Exercise selecting allocators on curMin shifts, conversions and
failures, recording mismatched releases and balancing each pool.
Check that rejected curMin shifts on malformed HLL_4 images leave the
sketch unchanged.
Preserve the original consistency exceptions where both implementations
reject an image. A curMin shift also rejects HLL_4 images that mark a
slot as an exception without an auxiliary entry; the previous code
accepted them.

Fixes apache#551

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5852c4a7-7589-49ef-bf80-8f2050501a9d
## Summary
Remove two unreachable HLL_4 branches and two duplicate allocation-failure
test cases. Explain the minimum-shift entry condition and map lifetime.

## Prompting Intent
Reduce unnecessary changes while preserving the allocation-failure fix,
its guarantees and distinct allocator coverage.

## Linked Sources
- apache#565
- apache#551

## Rationale
The auxiliary update caller excludes ordinary values, and the shift scan
already rejects exception tokens without a map. Existing sketch-update
sweeps cover union scalar forwarding; failed-mutation checks retain their
serialization assertions. Keep the outlined first-map helper and the
multi-shift transaction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b7960e30-8d3c-4554-82c5-881b981ab3f4
@clee704
clee704 marked this pull request as draft November 5, 2026 00:22
@clee704
clee704 marked this pull request as ready for review November 5, 2026 01:08
Comment thread hll/include/CouponHashSet-internal.hpp Outdated
std::memcpy(&couponCount, data + hll_constants::HASH_SET_COUNT_INT, sizeof(couponCount));
const uint32_t maxCouponCount = hll_constants::RESIZE_NUMER * (uint32_t(1) << (lgK - 3))
/ hll_constants::RESIZE_DENOM;
if (compactFlag && couponCount > maxCouponCount) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this guard only for compact?

@leerho leerho Nov 11, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a compact image the reader re-inserts each coupon with couponUpdate(). A count above 3/4·2^(lgK−3) would make the set promote to HLL partway through deserialization, and newSet() would discard the promoted object. No valid writer produces such a count, because a set promotes as soon as it passes that limit, so the check only rejects corrupt images. An updatable image is copied into the table as a block, so its count can't trigger a promotion during deserialization. My review comment asks for hardening on that path.

@leerho leerho left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Chungmin. I reviewed this against current master (98c6b74), which includes the 5.3.1 fixes from #568.

What I checked:

  • It merges cleanly into master, and hll_test passes under ASan and UBSan.

  • Serialized bytes are unchanged. I ran the same inputs through master and through this PR, covering:

    • HLL_4, HLL_6 and HLL_8, with lg_k 4, 5, 7, 8, 10, 12 and 14
    • n from 0 to 10^6, in two input orders
    • deserialize-then-update, conversion to HLL_4, and unions with mixed lg_k and types

    At all 1,890 checkpoints, the compact, updatable and stream serializations, the estimates and the bounds were identical. So the rewritten HLL_4 update path stays compatible with existing images and with Java.

  • Update speed is unchanged. With 3·10^7 updates, best of 5, Apple clang -O3 on arm64:

    • HLL_4 lg_k 12: 86 ms with both
    • HLL_4 lg_k 21: 245 ms on master, 235 ms with this PR
    • HLL_8 lg_k 12: 94–95 ms with both

The approach looks right to me:

  • HllUtil::newObject returns the storage if the constructor throws.
  • The promotions hold the new object in a unique_ptr until it is complete.
  • growAuxSpace() now updates lgAuxArrInts only after the new table is built.
  • Copy assignment uses copy-and-swap.
  • Rolling back the last insertion by writing EMPTY is safe in these open-addressing tables, because nothing was inserted after it.

Requested hardening: the updatable SET path doesn’t validate its header fields the way the compact path now does. Please add these checks for updatable images, in both the bytes and stream readers:

  • lgArrInts is in [LG_INIT_SET_SIZE, lgK − 3]
  • couponCount equals the number of non-empty slots in the table

We’ll add the same checks to the Java reader, so both languages accept and reject the same images.

One question: what do the noinline attributes buy? If they help code size or update speed, please add a sentence with the measurement to the comment or the PR description. Otherwise I'd drop them.

5.3.1 is in its vote now. We'll merge after it closes.

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the updatable SET hardening described in my review above (lgArrInts range and couponCount checks in both readers). It is required for this PR, not a follow-up.

@leerho

leerho commented Nov 12, 2026

Copy link
Copy Markdown
Member

@clee704 one more check for the updatable SET path, in addition to the two I requested above: the table must be no more than 3/4 full, that is 4·couponCount <= 3·2^lgArrInts. A valid writer always grows or promotes the table before it gets fuller than that, and the Java reader now has the same check (apache/datasketches-java#799). Without it, a table can be completely full with a matching count, and the first update of a new coupon after deserializing throws "Key not found and no empty slots!".

To save you guessing, here is all of the updatable-path hardening as one diff against your branch (563bbef), the two earlier checks included. It is the same in both newSet overloads in hll/include/CouponHashSet-internal.hpp:

  • The lgArrInts range and the 3/4-full checks go right after the existing if (lgArrInts < hll_constants::LG_INIT_SET_SIZE) recompute, before anything uses lgArrInts. In the bytes reader that's before couponsInArray is computed, so 1 << lgArrInts is never evaluated with an out-of-range shift. In the stream reader it's before the sketch is allocated. The lower bound needs no check, because the recompute already guarantees lgArrInts >= LG_INIT_SET_SIZE.
  • The 3/4-full product is computed in uint64_t, so a large couponCount can't overflow it.
  • The count check goes after the table is copied in. In the stream reader it goes after a stream-state check, so a short read is reported as a read error rather than as a count mismatch.
--- a/hll/include/CouponHashSet-internal.hpp
+++ b/hll/include/CouponHashSet-internal.hpp
@@ -23,4 +23,5 @@
 #include "CouponHashSet.hpp"
 
+#include <algorithm>
 #include <cstring>
 #include <exception>
@@ -101,4 +102,17 @@ CouponHashSet<A>* CouponHashSet<A>::newSet(const void* bytes, size_t len, const
     lgArrInts = HllUtil<>::computeLgArrInts(SET, couponCount, lgK);
   }
+  if (!compactFlag) {
+    // lgArrInts >= LG_INIT_SET_SIZE here, because of the recompute above
+    if (lgArrInts > lgK - 3) {
+      throw std::invalid_argument("Attempt to deserialize invalid updatable CouponHashSet with lgArrInts > lgK - 3. Found lgArrInts: "
+                                  + std::to_string(lgArrInts) + ", lgK: " + std::to_string(lgK));
+    }
+    // a valid writer grows or promotes the table before it is more than 3/4 full
+    if (static_cast<uint64_t>(hll_constants::RESIZE_DENOM) * couponCount
+        > static_cast<uint64_t>(hll_constants::RESIZE_NUMER) << lgArrInts) {
+      throw std::invalid_argument("Attempt to deserialize invalid updatable CouponHashSet that is more than 3/4 full. Found couponCount: "
+                                  + std::to_string(couponCount) + ", lgArrInts: " + std::to_string(lgArrInts));
+    }
+  }
   // Don't set couponCount in sketch here;
   // we'll set later if updatable, and increment with updates if compact
@@ -126,4 +140,10 @@ CouponHashSet<A>* CouponHashSet<A>::newSet(const void* bytes, size_t len, const
                 data + hll_constants::HASH_SET_INT_ARR_START,
                 couponsInArray * sizeof(uint32_t));
+    const auto nonEmpty = std::count_if(sketch->coupons_.begin(), sketch->coupons_.end(),
+                                        [](uint32_t c) { return c != hll_constants::EMPTY; });
+    if (static_cast<uint64_t>(nonEmpty) != couponCount) {
+      throw std::invalid_argument("Attempt to deserialize invalid updatable CouponHashSet with couponCount != number of coupons. Found couponCount: "
+                                  + std::to_string(couponCount) + ", coupons: " + std::to_string(nonEmpty));
+    }
   }
 
@@ -173,4 +193,17 @@ CouponHashSet<A>* CouponHashSet<A>::newSet(std::istream& is, const A& allocator)
     lgArrInts = HllUtil<>::computeLgArrInts(SET, couponCount, lgK);
   }
+  if (!compactFlag) {
+    // lgArrInts >= LG_INIT_SET_SIZE here, because of the recompute above
+    if (lgArrInts > lgK - 3) {
+      throw std::invalid_argument("Attempt to deserialize invalid updatable CouponHashSet with lgArrInts > lgK - 3. Found lgArrInts: "
+                                  + std::to_string(lgArrInts) + ", lgK: " + std::to_string(lgK));
+    }
+    // a valid writer grows or promotes the table before it is more than 3/4 full
+    if (static_cast<uint64_t>(hll_constants::RESIZE_DENOM) * couponCount
+        > static_cast<uint64_t>(hll_constants::RESIZE_NUMER) << lgArrInts) {
+      throw std::invalid_argument("Attempt to deserialize invalid updatable CouponHashSet that is more than 3/4 full. Found couponCount: "
+                                  + std::to_string(couponCount) + ", lgArrInts: " + std::to_string(lgArrInts));
+    }
+  }
 
   auto sketch = HllUtil<A>::template newObject<CouponHashSet<A>>(allocator, lgK, tgtHllType, allocator);
@@ -190,4 +223,11 @@ CouponHashSet<A>* CouponHashSet<A>::newSet(std::istream& is, const A& allocator)
     // for stream processing, read entire list so read pointer ends up set correctly
     read(is, sketch->coupons_.data(), sketch->coupons_.size() * sizeof(uint32_t));
+    if (!is.good()) { throw std::runtime_error("error reading from std::istream"); }
+    const auto nonEmpty = std::count_if(sketch->coupons_.begin(), sketch->coupons_.end(),
+                                        [](uint32_t c) { return c != hll_constants::EMPTY; });
+    if (static_cast<uint64_t>(nonEmpty) != couponCount) {
+      throw std::invalid_argument("Attempt to deserialize invalid updatable CouponHashSet with couponCount != number of coupons. Found couponCount: "
+                                  + std::to_string(couponCount) + ", coupons: " + std::to_string(nonEmpty));
+    }
   }

I applied this to your branch locally. hll_test passes under ASan and UBSan. I also patched a valid updatable image (lgK 12, HLL_8, 200 items, lgArrInts 9) in several ways, and both readers behave as expected:

  • The unpatched image is accepted.
  • lgArrInts set to 10 or 33 is rejected.
  • couponCount off by ±1 is rejected.
  • A completely full table with a matching count is rejected.
  • A table at exactly 3/4 full (384 coupons) is accepted and can still be updated.

You don't need to add tests for these. We'll add corrupt-image tests that match the Java ones after this is merged.

@leerho

leerho commented Nov 16, 2026

Copy link
Copy Markdown
Member

@clee704 just checking in on this one. 5.3.1 is out now, so this PR is unblocked apart from the updatable SET checks in my review. The diff in my comment of Nov 12 has all three checks, in both newSet overloads, so it should only need to be applied and tested.

We'd like to get this into the next minor release. If you're busy, we're happy to apply that diff to your branch ourselves (maintainer edits are enabled) and you can review it. Just let us know which you'd prefer. The question about the noinline attributes is still open too, but it doesn't block the merge.

Thanks again for this work.

@clee704

clee704 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@clee704 just checking in on this one. 5.3.1 is out now, so this PR is unblocked apart from the updatable SET checks in my review. The diff in my comment of Nov 12 has all three checks, in both newSet overloads, so it should only need to be applied and tested.

We'd like to get this into the next minor release. If you're busy, we're happy to apply that diff to your branch ourselves (maintainer edits are enabled) and you can review it. Just let us know which you'd prefer. The question about the noinline attributes is still open too, but it doesn't block the merge.

Thanks again for this work.

Sorry for the delay. I'm working on it now

clee704 and others added 2 commits November 17, 2026 15:29
## Summary
Check the size, load and occupied-slot count of updatable HLL SET
images in both readers. Reject incomplete stream reads before using
the data. Keep the existing size recovery for legacy images.

## Prompting Intent
Address the maintainer's SET validation feedback without changing the
serialized format or the allocation-failure behavior.

## Linked Sources
- Review feedback: apache#565 (comment)
- Java reader checks: apache/datasketches-java#799

## Rationale
Bound the coupon count before recomputing a missing size. Check the
table size and declared load before allocation, then compare the count
with the occupied slots once the complete table is available.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 19f3e046-71c3-4824-a22b-af059e50ecda
## Summary
Truncate an updatable SET image after one table slot. The reader should
report a short read before comparing the declared and occupied counts.

## Prompting Intent
Verify the order of the stream-state and count checks in the SET reader.

## Linked Sources
- Review feedback: apache#565 (comment)

## Rationale
Dropping only the last byte can leave the occupied count unchanged.
Leaving one slot makes the test fail if the reader checks the count
first.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 19f3e046-71c3-4824-a22b-af059e50ecda

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Chungmin, this looks good. All three updatable SET checks are in both readers: the lgArrInts range, the 3/4 load limit (checked before the sketch is allocated in the stream reader), and the count against the occupied slots. Moving the count limit ahead of the lgArrInts recompute, and checking the stream after each read, are good additions too. Thanks also for the tests, which we hadn’t asked for.

I checked 6432718: it merges cleanly into master, hll_test passes under ASan and UBSan, and CI is green.

On the noinline attributes: compilers are generally good at deciding what to inline, and we haven’t seen a measurement showing these help, so we’ll likely remove them later to keep the headers simple. If you do have numbers that show a benefit, please post them here and we’ll keep them. Either way, this doesn’t block the merge.

Approving now. Thanks again for the careful work on this.

@leerho
leerho merged commit 641c59c into apache:master Nov 18, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hll_sketch is not exception-safe when the allocator throws

3 participants