fix(bloomfilter): bound read/write to prevent BinaryArray i32 offset overflow#7753
Open
zhangyue19921010 wants to merge 6 commits into
Open
fix(bloomfilter): bound read/write to prevent BinaryArray i32 offset overflow#7753zhangyue19921010 wants to merge 6 commits into
zhangyue19921010 wants to merge 6 commits into
Conversation
github-actions
Bot
added
A-index
📝 WalkthroughWalkthroughChangesThe Bloom-filter scalar index now reads and writes serialized bloom-filter data in size-bounded record-batch chunks, rejects oversized payloads, preserves optional null-row metadata, and tests these behaviors. Bloom-filter payload chunking
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant BloomFilterIndex
participant read_range_stream
participant try_from_serialized
participant BloomFilterBatchWriter
BloomFilterIndex->>read_range_stream: read bounded serialized chunks
read_range_stream-->>BloomFilterIndex: record-batch chunks
BloomFilterIndex->>try_from_serialized: deserialize chunks with max_array_length
try_from_serialized-->>BloomFilterIndex: zone statistics
BloomFilterBatchWriter->>BloomFilterIndex: flush bounded record batches
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance-index/src/scalar/bloomfilter.rs`:
- Line 52: Document MAX_BLOOMFILTER_ARRAY_LENGTH with the rationale for
reserving a 1 MiB safety margin below the i32 limit. Add rustdoc for the public
write_index API, including an example that demonstrates linking IndexStore and
IndexFile.
- Around line 2379-2434: Update test_bloomfilter_chunked_write_and_load to build
nullable data containing null values, ensuring the chunked write triggers
add_global_buffer across multiple flushes. After loading, assert the preserved
null_rows metadata or verify equivalent IsNull query results while retaining the
existing chunk and zone assertions.
- Around line 2387-2399: Replace the manual Int32 schema, array, and RecordBatch
construction in the test setup with the standard gen_batch() or record_batch!()
builder, preserving the existing VALUE_COLUMN_NAME data and row count. Apply the
same change to the repeated setup near the second referenced test block, while
leaving the stream and add_row_addr flow unchanged.
- Around line 2367-2375: Update the related BloomFilterIndex error tests,
including the cases around read_batch_size, to assert that the returned error
matches the expected InvalidInput variant before checking its message content.
Preserve the existing “exceeds max supported batch bytes” message assertions
while adding variant-level validation for each applicable case.
- Line 142: Update the zones allocation in the surrounding bloom-filter
construction logic to use Vec::with_capacity() with the known row count instead
of Vec::new(), while preserving the existing element type and subsequent
population behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 46b2c204-8d4b-4a88-9d8b-bc30c306c2d2
📒 Files selected for processing (1)
rust/lance-index/src/scalar/bloomfilter.rs
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rust/lance-index/src/scalar/bloomfilter.rs (1)
112-161: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
read_batch_sizeis computed eagerly even when the index has zero rows.
load_with_max_array_lengthcallsSelf::read_batch_size(number_of_items, probability, max_array_length)(Line 143-144) before checking whetherindex_file.num_rows()is0. If a builder was configured withnumber_of_items/probabilitylarge enough that a single serialized filter would exceedmax_array_length,read_batch_sizereturns an error (pertest_bloomfilter_read_batch_size_rejects_oversized_filter). For a genuinely empty index (no zones were ever written, e.g. an empty input stream), this needlessly failsloadeven though no bloom-filter bytes need to be read at all.🐛 Proposed fix
+ if index_file.num_rows() == 0 { + return Ok(Arc::new(Self { + zones: Vec::new(), + number_of_items, + probability, + null_rows, + })); + } + let read_batch_size = Self::read_batch_size(number_of_items, probability, max_array_length)?; let mut zones = Vec::with_capacity(index_file.num_rows());🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/bloomfilter.rs` around lines 112 - 161, Update load_with_max_array_length to handle an index with zero rows before calling Self::read_batch_size. Return a valid BloomFilter with empty zones while still loading applicable metadata and null_rows, and preserve the existing batch-size validation and reading path for non-empty indexes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@rust/lance-index/src/scalar/bloomfilter.rs`:
- Around line 112-161: Update load_with_max_array_length to handle an index with
zero rows before calling Self::read_batch_size. Return a valid BloomFilter with
empty zones while still loading applicable metadata and null_rows, and preserve
the existing batch-size validation and reading path for non-empty indexes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 166b7639-4a9a-43ce-a736-e4cb757bd9a2
📒 Files selected for processing (1)
rust/lance-index/src/scalar/bloomfilter.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When create bloom filter index on a column with more than 1.07B rows, we might ran into arrow byte array offset overflow error. This pr fix it by writing index in batch.
Summary by CodeRabbit
Bug Fixes
Compatibility