perf(index): vectorized merge kernels and frequency-bound pruning for FTS conjunctions by BubbleCal · Pull Request #7625 · lance-format/lance

BubbleCal · 2026-07-04T12:32:21Z

Stacked on #7624.

Performance issue

#7624 sends compressed top-k AND / phrase queries through a bulk conjunction path. This PR specializes the 2- and 3-clause merges; 4+ clauses use the generic merge. On Lu's random 5-term AND workload, forcing that generic bulk path was slower than the classic loop, so enabling bulk for every clause count was not a safe default.

What changed

Keep clause-count-specialized scalar and AVX2 merge kernels for 2 and 3 clauses. Four or more clauses use the generic kernel when bulk is explicitly forced.
Keep the per-clause frequency-bucketed score-bound LUT. It skips a lead doc before follower advances when the monotone score bound cannot beat the threshold; emitted results are unchanged.
Replace the boolean LANCE_FTS_BULK_AND behavior with three modes:
- auto (default): use bulk only for 2/3 effective clauses; use classic for all other widths.
- on / legacy 1: force bulk for every eligible compressed AND / phrase query. Four or more clauses use the generic kernel.
- off / legacy 0: always use the classic conjunction loop.
Count effective posting clauses after tokenization, deduplication, and expansion (lead.len()), not raw input words. Invalid values warn and fall back to auto.

The AVX2 find_next_geq uses an 8-wide compare + movemask catch-up scan, with runtime dispatch and a scalar fallback. #[target_feature(enable = "avx2")] covers each whole specialized kernel so the vector scan can inline.

Benchmark

Existing 2/3-clause benefit vs #7624

Per-branch-tip wheels, 1000 queries × 8 concurrent. AND used the warm, prewarmed 200M-doc V3/256 index; phrase used the 50M-doc V3/256 positions index.

query	#7624	SIMD/frequency-bound kernels
AND 3w k10 @200M	136 qps	172 qps (+27%)
AND 3w k100 @200M	69 qps	86 qps (+25%)
phrase 3w k10 @50m	34 qps	35 qps (+4%)
phrase 2w k10 @50m	100 qps	102 qps (~flat)

4/5-clause follow-up

AWS r7i.24xlarge, 200M English-only MMLB, V3/256 index with 338 partitions, k=100, c32, 600 GB cache, synchronous full prewarm. The same fixed query set was used across modes within each clause count.

clauses	tier	classic	generic	experimental specialized	specialized vs generic	specialized vs classic
4	500-600	1,330	1,536	1,594	+3.79%	+19.80%
4	2000-2100	2,889	2,577	2,576	-0.04%	-10.82%
4	9000-10000	3,271	3,184	3,182	-0.06%	-2.72%
5	500-600	3,065	2,631	2,630	-0.02%	-14.19%
5	2000-2100	3,260	2,996	3,004	+0.28%	-7.84%
5	9000-10000	3,328	3,336	3,284	-1.56%	-1.34%

Four-clause results use two runs for every mode. Its only repeatable kernel gain was in the tier with the most hits in this experiment (avg_rows≈5.09): +4.17% / +3.40% in the two runs. The other tiers showed no kernel gain, and bulk itself was slower than classic in two of three tiers. Five-clause classic and specialized results use two runs; generic uses one, and no meaningful specialized-kernel gain was observed. The 5-clause 2000-2100 and 9000-10000 tiers returned zero rows, so they primarily measure the empty-result path.

Therefore auto remains limited to 2/3 clauses, and neither the experimental 4-clause nor 5-clause specialized kernel is retained. Under forced on, 4+ clauses use the existing generic bulk path. on is a diagnostic / explicit opt-in, not the recommended default for 4+ clauses.

Verification

cargo test -p lance-index: 749 passed, 1 ignored; doc test passed.
AND / phrase parity and actual dispatch assertions for 2 through 6 clauses under auto, on, and off.
x86_64 / AVX2 parity tests passed on the benchmark VM; scalar tests passed locally on arm64.
cargo clippy --all --tests --benches -- -D warnings passed.
cargo fmt --all and git diff --check passed.

github-actions · 2026-07-04T12:32:39Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

coderabbitai · 2026-07-11T19:05:39Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: cd37e199-b326-4335-bf39-da9d50f44dab

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch yang/lan2-88-fts-simd-merge-kernels

_{Comment @coderabbitai help to get the list of available commands.}

## Feature Linear: [OSS-1344](https://linear.app/lancedb/issue/OSS-1344/make-fts-index-block-size-configurable) ### What is the new feature? FTS inverted index creation now accepts a `block_size` parameter for compressed posting blocks. Supported values are `128` and `256`. ### Why do we need this feature? The posting block size was previously fixed at `128`, which made the block-max granularity impossible to tune for different datasets and query profiles. ### How does it work? - Adds `block_size` to `InvertedIndexParams`, protobuf details, posting-list schema metadata, and cache headers. - Uses `128` as the default for newly created indexes. - Treats older serialized params, schema metadata, and cache entries that omit `block_size` as legacy `128`. - Rejects unsupported values, including `512`, with a clear validation error. - Uses Lance-owned `BitPacker4x` for physical 128-value posting blocks and `BitPacker8x` for physical 256-value posting blocks. - Marks `block_size=256` as experimental in public API docs because it may introduce breaking changes. - Keeps position-stream packing on the legacy 128-value block format. - Keeps downgrade compatibility tests on explicit legacy `block_size=128`, since older wheels cannot read current-created physical 256 FTS posting blocks. - Threads the configured block size through FTS build, read, iterator, WAND, cache, and MemWAL flush paths. - Exposes the parameter in Python and Java FTS index creation APIs, with docs and focused tests. ## Validation - `cargo fmt --all` - `cargo fmt --all --check` - `git diff --check` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo test -p lance-index block_size -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo clippy -p lance-index --tests -- -D warnings` - `uv run make build` from `python/` - `uv run pytest python/tests/test_scalar_index.py::test_create_scalar_index_fts_block_size` from `python/` - `uv run ruff format --check python/tests/test_scalar_index.py python/lance/dataset.py` from `python/` - `uv run ruff check python/tests/test_scalar_index.py python/lance/dataset.py` from `python/` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index block_size -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index test_256_posting_block_uses_single_physical_bitpack_chunk -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-bitpacking` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo clippy -p lance-bitpacking -p lance-index --tests -- -D warnings` - `uv run ruff format --check python/tests/compat/test_scalar_indices.py` from `python/` - `uv run ruff check python/tests/compat/test_scalar_indices.py` from `python/` - `uv run pytest --run-compat -vvv -s python/tests/compat/test_scalar_indices.py::test_FtsIndex_downgrade --durations=30` from `python/` - `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index test_new_training_request_defaults_missing_block_size_to_128` - `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index block_size` - `uv run ruff format --check python/lance/dataset.py` from `python/` - `uv run ruff check python/lance/dataset.py` from `python/` Not run locally: Java focused test / spotless check, because this machine has no Java Runtime installed (`Unable to locate a Java Runtime`). --- ## Update: all V3 breaking changes consolidated here Per review direction, every breaking change for the 256-doc block format now lands in this single PR (the follow-up stack #7602/#7603/#7604/#7624/#7625/#7629 carries none). On top of the configurable block size and PFOR frequency encoding, this PR now also includes: - **Quantized doc-length scoring (Lucene norm semantics), 256-doc blocks only.** BM25 doc lengths are quantized to a SmallFloat-style byte code (4 mantissa bits: 0-7 exact, <= 6.25% relative error, decode = bucket floor). The byte-norm slab bakes lazily per loaded DocSet and quarters the doc-length bytes scoring pulls through the cache (200M docs: 800MB -> 200MB). 128-block indexes keep exact-length scoring bit-for-bit. Measured top-k overlap vs exact scoring on the (score-clustered, synthetic) mmlb corpus: 98.1% mean for phrase, 89.7% for 3-word AND; corpora with more score spread shift less. - **256-doc posting blocks drop the leading block-max-score f32** (~1.5G on a 200M-doc index; 131G -> 130G). Block layout: `[first_doc u32][doc num_bits u8][docs][pfor freqs]`; `posting_block_score_prefix_len(block_size)` keys every reader/writer. The impact skip data from the stacked #7602 supplies a tighter per-block bound; until it lands, 256-block block-max pruning falls back to the (valid, looser) list-level max score. **BREAKING:** 256-doc-block (v3) indexes must be rebuilt; v3 is unreleased so no migration is provided. BM25 scores on v3 differ from exact-length BM25 by the norm quantization, matching Lucene's norm semantics. The format discussion #7606 documents the final layout and scoring semantics. Additional validation for this update: bulk-vs-classic A/B under quantized scoring is score-identical (both paths quantize identically); the full stack's warm benchmarks vs Lucene 10.4 on mmlb-200m: OR k10 0.0249s/318qps and OR k100 0.0467s/170qps (both ahead of Lucene sliced), AND k10 0.0443s (1.29x), AND k100 0.0883s (1.94x). --- ## Standalone results vs main (per-branch-tip wheels) **Legacy (128) read-path parity.** Threading a runtime `block_size` through `PostingIterator` initially replaced the compile-time `BLOCK_SIZE` division (a shift) with real `div` instructions in the `doc()`/`next()` hot loops, measured as +11-14% on 3-word OR against the 200M legacy index (`PostingIterator::next` grew from 16.5% to 23.6% of the profile). Block sizes are validated powers of two, so the iterator now derives block indices with `trailing_zeros` shifts and masks; after that fix the legacy path is at parity with main: 3-word OR k10 0.131s (main 0.132-0.134s), k100 0.255-0.256s (main 0.256-0.257s), single-term 0.025s (main 0.027s) across 3 warm passes on the 200M legacy index, 400G cache. **block_size=256 index size** (5M-doc controlled build, same wheel, `with_position=false`): postings shrink **3.33 GiB → 2.62 GiB (−21%)** from PFOR frequencies + no per-block max-score prefix + half the block headers. Index build 192s → 210s (+10%, PFOR encode cost). Top-10 overlap vs the 128 exact-length scoring on 10 3-word OR queries: 95% mean (5/10 identical sets; the rest differ by 1-2 near-tie docs, from the quantized-norm scoring). **Query wins for 256 land in the stacked PRs.** A 256 index without impact skip data prunes on the (valid, looser) list-level max and is *slower* than 128 — e.g. classic AND k10 0.547s until #7602's impacts restore block-granular bounds (0.115s), and #7603/#7604/#7624/#7625/#7629 take the same index to 0.025s OR k10 / 0.045s AND k10.  ## Summary by CodeRabbit * **New Features** * Added configurable FTS posting `block_size` (128/256) to scalar index creation, including updated examples and APIs. * Enabled FTS format version 3 (`v3`) for `block_size=256`, with quantized doc-length scoring for v3. * **Bug Fixes** * Enforced `block_size`/`format_version` compatibility (invalid combinations now error). * Persisted and restored FTS metadata for format version and posting block size, with legacy indexes defaulting to `128`. * **Documentation** * Updated full-text-search and quickstart guides and parameter docs for `block_size`, defaults, accepted values, and the experimental `256`/`v3` behavior.  --------- Co-authored-by: Yang Cen <yang@lancedb.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

@200M

) Built on the MAXSCORE work merged in #7603; this is the next PR in the Lucene-parity series. This PR now includes the 2/3-clause SIMD and frequency-bound work previously stacked as #7625. ## Performance issue Top-k AND and phrase queries previously leapfrogged doc-at-a-time through boxed `PostingIterator::next` calls. Phrase checks additionally decoded a whole 256-doc position block per candidate and allocated cursor vectors per candidate. ## What changed - Add a bulk conjunction path for compressed top-k AND and phrase queries. It keeps the classic block-max window pruning semantics while intersecting decompressed posting slices in batches. - Specialize the 2- and 3-clause merge kernels. x86_64 uses runtime-dispatched AVX2 catch-up scans, with scalar kernels on other CPUs. - Add a frequency-bucketed score-bound LUT so lead docs that cannot beat the threshold are rejected before follower advances. - Keep a generic merge kernel for 4+ clauses when bulk mode is explicitly forced. - Score candidates in two passes so document-length loads are issued back-to-back. - Decode only the PackedDelta position groups needed by the current phrase candidate instead of the whole position block. - Use an allocation-free exact-phrase check and recycled owned position buffers. - Return contextual errors for malformed packed-position data instead of panicking. ## Runtime selection `LANCE_FTS_BULK_AND` accepts: - `auto` (default): use bulk for 2/3 effective posting clauses and classic for all other widths. - `on` or legacy `1`: force bulk for every eligible compressed AND/phrase query; 4+ clauses use the generic kernel. - `off` or legacy `0`: always use the classic conjunction loop. The clause count is measured after tokenization, deduplication, and expansion. Invalid values warn and fall back to `auto`. The 4/5-clause experiments did not show a consistent specialized-kernel win, and generic bulk regressed against classic in several tiers. Therefore no 4/5-clause specialized kernels are retained and `auto` remains limited to 2/3 clauses. ## Measured performance Per-branch-tip wheels, 1000 queries × 8 concurrent. AND used the warm, fully prewarmed 200M-doc V3/256 index; phrase used the 50M-doc V3/256 positions index. | query | previous stack base | combined PR | |---|---:|---:| | AND 3w k10 @200M | 70 qps | **172 qps (2.46×)** | | AND 3w k100 @200M | 33 qps | **86 qps (2.61×)** | | phrase 3w k10 @50m | 19 qps | **35 qps (1.84×)** | | phrase 2w k10 @50m | 59 qps | **102 qps (1.73×)** | The AWS 4/5-clause follow-up used an `r7i.24xlarge`, the 200M English-only MMLB dataset, a V3/256 index with 338 partitions, k=100, c32, a 600 GB cache, and synchronous full prewarm. ## Compatibility This is a query-time implementation change. It does not change the FTS index format or index version. ## Verification - Bulk/classic/auto parity and dispatch coverage for 2 through 6 clauses, including nonzero phrase slop. - PackedDelta per-doc seek parity across group boundaries and tails. - PackedDelta and VarintDocDelta cursor-recycling coverage. - Malformed packed-position data returns a recoverable search error. - Final WAND suite: 80 passed. - `cargo test -p lance-index`, `cargo clippy --all --tests --benches -- -D warnings`, and `cargo fmt --all -- --check` passed across the reviewed stack.  ## Summary by CodeRabbit * **Performance Improvements** * Added a faster bulk AND/phrase execution path for compressed postings with block-level skipping and slice intersection; availability controlled via `LANCE_FTS_BULK_AND` (auto by default). * Improved phrase verification to decode only required positions per matching document. * Optimized packed-delta position seeking using cached group state and efficient tail handling. * **Bug Fixes** * Ensured bulk AND/phrase results match classic behavior, including filtering and pruning semantics. * Improved rejection of malformed packed-position data. * **Tests** * Added seek accuracy coverage across group boundaries and tail cases, plus malformed-group rejection and cursor independence checks.  --------- Co-authored-by: Yang Cen <yang@lancedb.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

github-actions Bot added A-index Vector index, linalg, tokenizer performance labels Jul 4, 2026

BubbleCal mentioned this pull request Jul 4, 2026

feat(index)!: quantized doc-length scoring and slimmer posting blocks for FTS v3 #7626

Closed

BubbleCal force-pushed the yang/lan2-88-fts-simd-merge-kernels branch from 2f9de34 to 3c60d9a Compare July 4, 2026 18:17

BubbleCal force-pushed the yang/lan2-88-fts-bulk-conjunction branch from fc1af4b to 22e3522 Compare July 4, 2026 18:17

This was referenced Jul 4, 2026

perf(index): norm-addend cache and slim top-k heap for FTS scoring #7629

Open

feat(fts)!: add configurable posting block size #7466

Merged

BubbleCal changed the title ~~perf(index): SIMD merge kernels and frequency-bound pruning for FTS conjunctions~~ perf(index): vectorized merge kernels and frequency-bound pruning for FTS conjunctions Jul 4, 2026

BubbleCal marked this pull request as draft July 5, 2026 17:51

BubbleCal force-pushed the yang/lan2-88-fts-bulk-conjunction branch from 22e3522 to 6122151 Compare July 14, 2026 15:18

BubbleCal force-pushed the yang/lan2-88-fts-simd-merge-kernels branch from 62bd294 to 458a96e Compare July 14, 2026 15:21

BubbleCal force-pushed the yang/lan2-88-fts-bulk-conjunction branch from 6122151 to 6c76a45 Compare July 15, 2026 06:25

BubbleCal force-pushed the yang/lan2-88-fts-simd-merge-kernels branch from 458a96e to 5844cfe Compare July 15, 2026 06:25

Xuanwo mentioned this pull request Jul 15, 2026

perf(index): bulk conjunction path for FTS AND and phrase queries #7624

Merged

BubbleCal force-pushed the yang/lan2-88-fts-bulk-conjunction branch from 6c76a45 to 6311353 Compare July 15, 2026 10:07

BubbleCal force-pushed the yang/lan2-88-fts-simd-merge-kernels branch from 5844cfe to 02a7752 Compare July 15, 2026 10:10

perf(index): specialize SIMD merge kernels for bulk fts conjunctions

110540f

BubbleCal force-pushed the yang/lan2-88-fts-simd-merge-kernels branch from 02a7752 to 110540f Compare July 15, 2026 10:37

BubbleCal merged commit 110540f into yang/lan2-88-fts-bulk-conjunction Jul 15, 2026
5 checks passed

BubbleCal deleted the yang/lan2-88-fts-simd-merge-kernels branch July 15, 2026 11:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(index): vectorized merge kernels and frequency-bound pruning for FTS conjunctions#7625

perf(index): vectorized merge kernels and frequency-bound pruning for FTS conjunctions#7625
BubbleCal merged 1 commit into
yang/lan2-88-fts-bulk-conjunctionfrom
yang/lan2-88-fts-simd-merge-kernels

BubbleCal commented Jul 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 4, 2026

Uh oh!

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

BubbleCal commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance issue

What changed

Benchmark

Existing 2/3-clause benefit vs #7624

4/5-clause follow-up

Verification

Uh oh!

github-actions Bot commented Jul 4, 2026

Uh oh!

coderabbitai Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

BubbleCal commented Jul 4, 2026 •

edited

Loading

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading