Skip to content

feat: expose row address allowlist/blocklist prefilters in C and C++ - #96

Merged
jja725 merged 1 commit into
lance-format:mainfrom
wanglun:row-addr-prefilter
Oct 14, 2026
Merged

jja725 merged 1 commit into
lance-format:mainfrom
wanglun:row-addr-prefilter

Conversation

@wanglun

@wanglun wanglun commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

C and C++ callers cannot restrict a scan or search to a known set of rows before ranking: a _rowid IN (...) filter is evaluated by a full row-id read plus an IN-list parse, which costs tens of milliseconds even for a few ids and seconds for large lists. Lance's Scanner::with_row_addr_prefilter does this natively; pylance exposes it as row_addr_allowlist / row_addr_blocklist.

Add lance_scanner_set_row_addr_allowlist and
lance_scanner_set_row_addr_blocklist, which take arrays of _rowid values (stable row ids when enabled, otherwise row addresses), and the C++ Scanner::row_addr_allowlist / row_addr_blocklist wrappers. The lists are combined like RowAddrMask::from_serialized_parts (allowed AND NOT blocked) and ANDed with any filter expression. An empty allowlist allows no rows.

A prepared FTS query context rebuilds its Match/Phrase execs around the shared global scorer, and the rebuilt execs did not carry the external mask, so the mask was silently dropped. The rewrite now re-attaches it, so prepared Match/Phrase searches score only the allowed rows with unchanged BM25 scores. Combining a row list with a segment-scoped scalar scan, which installs its own row mask, is rejected.

Tests cover plain scans (allow, block, both, empty, replacement, AND with a SQL filter) on a two-fragment dataset with stable row ids, invalid inputs (NULL with len > 0, setting after the scan starts, scalar_index_segment), plain and prepared full-text search (OR and AND operators, scores equal to the unmasked search), and nearest-neighbor top-k over allowed rows. C and C++ runtime tests exercise the exported symbols and wrappers.

Validation:

  • cargo fmt -- --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test
  • cargo test --test compile_and_run_test -- --ignored --test-threads=1

C and C++ callers cannot restrict a scan or search to a known set of rows
before ranking: a `_rowid IN (...)` filter is evaluated by a full row-id
read plus an IN-list parse, which costs tens of milliseconds even for a few
ids and seconds for large lists. Lance's `Scanner::with_row_addr_prefilter`
does this natively; pylance exposes it as `row_addr_allowlist` /
`row_addr_blocklist`.

Add `lance_scanner_set_row_addr_allowlist` and
`lance_scanner_set_row_addr_blocklist`, which take arrays of `_rowid`
values (stable row ids when enabled, otherwise row addresses), and the C++
`Scanner::row_addr_allowlist` / `row_addr_blocklist` wrappers. The lists are
combined like `RowAddrMask::from_serialized_parts` (allowed AND NOT blocked)
and ANDed with any filter expression. An empty allowlist allows no rows.

A prepared FTS query context rebuilds its Match/Phrase execs around the
shared global scorer, and the rebuilt execs did not carry the external
mask, so the mask was silently dropped. The rewrite now re-attaches it, so
prepared Match/Phrase searches score only the allowed rows with unchanged
BM25 scores. Combining a row list with a segment-scoped scalar scan, which
installs its own row mask, is rejected.

Tests cover plain scans (allow, block, both, empty, replacement, AND with a
SQL filter) on a two-fragment dataset with stable row ids, invalid inputs
(NULL with len > 0, setting after the scan starts, scalar_index_segment),
plain and prepared full-text search (OR and AND operators, scores equal to
the unmasked search), and nearest-neighbor top-k over allowed rows. C and C++
runtime tests exercise the exported symbols and wrappers.

Validation:
- `cargo fmt -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo test`
- `cargo test --test compile_and_run_test -- --ignored --test-threads=1`

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The native row-mask approach preserves selection before ranking and shared BM25 scoring while keeping the C and C++ bindings thin. The new Rust tests, prepared-FTS regressions, additional phrase/top-k and indexed-vector checks, and both C/C++ runtime suites passed.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 13, 2026
@wanglun

wanglun commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

@jja725 thanks for the review, it seems I cannot merge it, can you help? thanks!

@jja725
jja725 merged commit 2089f6f into lance-format:main Oct 14, 2026
10 checks passed
@wanglun
wanglun deleted the row-addr-prefilter branch October 15, 2026 00:55
Gabriel39 added a commit to apache/doris that referenced this pull request Oct 21, 2026
…69332)

### What problem does this PR solve?

Related PRs: lance-format/lance-c#97,
lance-format/lance-c#96

Update the pinned lance-c revision from
`8e9e09ecc795c91ce7f97513db94c2d91eec799d` to
`4fafcd64525b274b113208390bbcca86bbbc093b`, the latest upstream `main`
commit when this PR was prepared, and update the archive checksum.

The new revision includes the upstream fix for cold-session vector
prewarm when an index name matches a different schema column. It
resolves the indexed field from segment metadata while preserving
concurrent segment opening/prewarming. The revision also includes the
row-address allowlist/blocklist prefilter APIs from lance-c #96.

Refresh one C++ test context in the existing Foyer patch to accommodate
the test added by #96. The patch's added/deleted code lines are
unchanged; no local prewarm patch is introduced.

### Release note

Fix Lance index prewarm failures when a vector index name matches a
different schema column.

### Check List (For Author)

- Test
    - [x] Manual test
- Downloaded the exact pinned archive and verified its checksum through
the Doris downloader.
- `bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src`:
passed all scenarios, including fresh extraction, idempotence, cache
invalidation, and patch-failure handling.
- `cargo metadata --locked --offline --format-version 1` on the patched
source: passed dependency resolution, including Foyer.
- Compiled the patched upstream C and C++ test sources with
`-fsyntax-only`: passed.
- `bash -n` on the dependency scripts and `git diff --check`: passed.
- Full Doris build and runtime regression tests were not run locally;
requested through `run buildall`.
- Behavior changed:
    - [x] Yes. Includes the upstream prewarm fix and new prefilter APIs.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants