[Gabriel39]

Summary

Add opt-in runtime BTree segment selection for ordinary, explicitly fragment-scoped scans. Callers currently choose a physical scalar segment before execution, even when another segment covering the same read domain has a narrower driver predicate.

lance_scanner_set_scalar_index_auto_selection and Scanner::scalar_index_auto_selection compare the current snapshot's BTree page min/max and NULL-count metadata. Estimated work is matching pages times rows per page. A strictly cheaper eligible segment can replace the caller's default; a broad estimate can choose a scoped non-indexed scan. The complete filter, fragment domain, deletion handling, and LIMIT/OFFSET are preserved. No candidate row-ID sets are materialized just to estimate costs.

The option defaults to false. Unknown estimates preserve the selected segment, and Bitmap/LabelList behavior remains unchanged. Candidate count, serialized lookup bytes, decoded lookup buffers, page count, IN values, and estimation work are bounded. Lookup batches share the bounded Session metadata cache. Statistics callbacks report the compared candidates, changed selections, estimated pages/rows, and scan fallbacks.

Decoded lookup buffers have a separate aggregate 8 MiB allowance, checked from fixed-width schema and row count before materialization and charged on both cold loads and cache hits. The current IndexReader API cannot bound decoded variable-width values, so string/binary costs remain unknown: retain such a default or skip such an alternative before opening its lookup for estimation. These indices remain usable through the existing caller-selected path.

The implementation uses existing pinned Lance APIs and does not change dependency revisions or the on-disk format. README documents the API, model limitations, and fallback behavior.

Validation

• cargo test --offline --locked: 501 tests passed, with the three C/C++ compilation tests intentionally ignored by the normal test command.
• Includes all 381 C API tests and 72 unit tests. The compressed-string regression was first observed failing on the original implementation, then passing with the fix. It uses 1,024 BTree pages with repeated 64 KiB strings whose serialized lookup fits the I/O budget but whose decoded extrema would occupy 128 MiB, and verifies retaining the integer driver without caching the string probe.
• Added decoded-budget coverage for cold loads, cumulative cache-hit accounting, wide fixed-width fields, unknown variable-width sizes, and integer overflow.
• Rustfmt and Clippy with --all-targets -- -D warnings.
• Runtime integration tests compare two competing BTree segments and verify the disabled option, broad-filter scan fallback, fixed fragment scope, empty results, and LIMIT/OFFSET.
• On the initial feature commit (b0a79c5), all three C/C++ compilation/execution tests and RUSTDOCFLAGS="-D warnings" cargo doc --offline --locked --no-deps passed. The decoded-memory follow-up changes private Rust logic/tests and README only; upstream CI reruns the full matrix.

[Gabriel39]

Follow-up safety fix in c6cbde1: serialized file size and cache admission do not bound transient decoded lookup memory. Probes now preflight a separate cumulative 8 MiB decoded-buffer allowance before materialization, and cache hits consume the same allowance.

The current IndexReader API cannot provide a safe decoded-size bound for variable-width extrema. Those probes now return unknown before opening the lookup: retain a string/binary default, or skip that alternative. The existing caller-selected index path remains available.

Added a compressed long-string regression (1,024 pages, repeated 64 KiB strings, 128 MiB decoded extrema despite a lookup below the serialized limit), an integer-driver alternative case, cold/cache-hit budget checks, wide fixed-width bounds, and overflow coverage. The original implementation failed the regression; the fix passes all 501 Rust tests, Rustfmt, and Clippy with warnings denied. README and this PR description document the limitation. CI is rerunning.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Verified the decoded-memory follow-up: fixed-width lookup buffers are checked before decoding, and cached lookups consume the same cumulative allowance. String/binary estimates remain unknown; caller-selected execution remains available.

The opt-in selector reduces unnecessary candidate work while preserving fragment scope, the complete filter, deletion handling, and LIMIT/OFFSET. The new memory regressions and focused scalar tests pass.

[yanghua]

+1