Repository navigation
Remove redundant nullptr checks after allocate() in bloom_filter - #578
Conversation
Thanks for the review and clarification! I'll check allocate(0) and update the patch accordingly. |
|
@P92Z thanks for taking on the As far as I can see, The branch also conflicts with master now, because #580 changed the stream Thanks again for this. |
|
Thanks @leerho, I'm still on it. Your analysis matches mine, and I'll push the update shortly: rejecting a non-empty image with |
|
Thanks @leerho, and sorry for the wait. I pushed the update:
On rebasing onto master after #580: I checked it locally. There are two small conflicts: the removed |
proost
left a comment
There was a problem hiding this comment.
Can you rebase? Conflict happens
Standard-conforming allocators report failure by throwing from allocate(), never by returning nullptr, so these checks can never fire. One of them was also placed after std::fill_n had already written to the buffer. Other sketches in the library don't perform such checks either. Add a test that a failing (throwing) allocator propagates std::bad_alloc from the constructor and both deserialize paths.
The length of the bit array is read from the serialized image, and 0 was rejected only for empty images. For a non-empty image with 0 longs, deserialize() called allocate(0), and deserialize(), wrap() and writable_wrap() all created a filter with a capacity of 0. update() on that filter then divided by zero. Reject such images as corrupt, as other invalid header fields are. The tests cover non-empty and empty images with 0 longs, and that the capacity of a new filter is rounded up to a multiple of 64 bits, so its bit array is never shorter than 8 bytes.
So that it can't clash with the failing allocators that other tests define in namespace datasketches.
e18b781 to
1b1e931
Compare
proost
left a comment
There was a problem hiding this comment.
Not num_longs == 0. We should check num_bytes directly. Because overflow can happens in shifts
num_longs is read from the image as a uint32_t, and num_longs << 3 and num_longs << 6 were computed in 32 bits before being stored in uint64_t. They could overflow: a non-empty image with 2^29 longs got a bit array of 0 bytes and a capacity of 0, so it passed the check for 0 longs, called allocate(0), and update() then divided by zero. Even a valid filter of 2^32 bits or more was read back with its capacity modulo 2^32. Widen num_longs before shifting, and check the size of the bit array itself, num_bytes, instead of num_longs, so that allocate() is never called with 0. In the tests, give the filter type with the throwing allocator a short name, throwing_filter, and name the parameters of the bad_alloc test.
Thanks @proost, you're right. I reproduced it: a non-empty image with The same overflow in The new commit widens One more thing, not in this PR: neither deserialize path checks |
Problem
bloom_filterchecks the result ofallocate()fornullptrin four places and throwsstd::bad_allocif it is null. A standard-conforming allocator never returnsnullptrfromallocate(n)forn > 0— it throwsstd::bad_allocitself, so these checks can never fire.The other sketches in the library don't make such checks, and neither do the standard containers that the library uses.
Misplaced check
One of the checks was also in the wrong place. In the main constructor,
std::fill_nwrote to the buffer before the check:With an allocator that returns
nullptr, this writes through a null pointer before the check runs. UBSan reports store to null pointer, and ASan reports a SEGV instd::fill_n.Changes
bloom_filter_impl.hpp: removed thenullptrchecks afterallocate()in:num_bitsdeserialize()from a streamdeserialize()from memory (the copying branch ofinternal_deserialize_or_wrap(); thewrap()branch never allocated).Behavioral impact
std::bad_alloc, now fromallocate()itself.nullptrinstead of throwing: these do not meet the Allocator requirements and aren't supported elsewhere in the library. With such an allocator, three of these paths used to throwstd::bad_alloc; they no longer do.Alternative
If you'd rather keep the checks, the misplaced one can instead be moved before
std::fill_n, as the copy constructor and bothdeserialize()paths already do.Tests
bloom_filter_allocation_test.cpphas a new test with an allocator that fails in the standard way — by throwingstd::bad_alloc. It checks that the exception reaches the caller from:builder::create_by_size()deserialize()from memorydeserialize()from a streamThe Bloom filter tests pass under ASan and UBSan.
Scope
This PR touches only
filters/.Update after review
deserialize()paths, soallocate()is never called with 0. The check also coverswrap()andwritable_wrap(), so no filter with a capacity of 0 can be created. Tests cover each path, and that the capacity of a new filter is rounded up to a multiple of 64 bits.