[Gabriel39]

What problem does this PR solve?

Related PR: #69332 (lance-c dependency upgrade)

Ordinary SQL against Lance external tables does not use the existing Lance search path for vector Top-N, and score() cannot produce Lance BM25 relevance. Users currently need explicit vector_search() / full_text_search() TVFs.

This PR adds automatic rewrites on branch-4.1, reusing the existing fixed-snapshot search planner, index-segment execution, global candidate merge, and deferred payload reads. It upgrades lance-c to 4fafcd64525b274b113208390bbcca86bbbc093b, as in #69332, and directly uses lance_scanner_set_row_addr_allowlist() introduced by lance-c #96.

SQL examples and behavior changes

The examples assume a Lance table with a three-dimensional Float32 embedding and a supported full-text index on body.

1. Approximate vector Top-N

SELECT doc_id,
       l2_distance_approximate(embedding, [0.1, 0.2, 0.3]) AS distance
FROM lance_catalog.docs.documents
WHERE tenant_id = 7
ORDER BY distance ASC
LIMIT 10 OFFSET 5;

Before: ordinary Lance scanning evaluates the scalar distance and Doris performs Top-N; this SQL does not request native Lance vector search.

After: the optimizer creates a native Lance L2 search with the original scalar predicate applied before candidate selection. Each search split requests 15 candidates with offset 0; Doris merges them and applies the final global OFFSET 5 LIMIT 10. Compatible Lance vector indexes can serve the approximate search through the existing search planner. Candidate selection may therefore be approximate rather than an exhaustive scalar ranking.

The SQL result remains Euclidean distance: Lance's squared L2 result is converted with CAST(sqrt(_distance) AS FLOAT).

Inner-product queries use the corresponding descending order:

SELECT doc_id,
       inner_product_approximate(embedding, [0.1, 0.2, 0.3]) AS similarity
FROM lance_catalog.docs.documents
WHERE tenant_id = 7
ORDER BY similarity DESC
LIMIT 10;

This becomes native Lance dot search. The projection converts Lance dot distance with 1.0f - _distance, so the SQL output remains an inner-product similarity and higher is better.

2. BM25 full-text Top-N

SELECT doc_id, score() AS relevance
FROM lance_catalog.docs.documents
WHERE body MATCH_ANY 'database search'
  AND tenant_id = 7
ORDER BY relevance DESC
LIMIT 10;

Before: score() on this Lance external scan is unsupported.

After: one top-level MATCH predicate becomes a native prepared Lance full-text query, score() reads its generated BM25 score, and Doris merges the highest-scoring candidates. The native prepared query preserves the dataset-wide scoring context across index segments and row-ID filtering.

• MATCH_ANY maps to a match query with OR.
• MATCH_ALL maps to a match query with AND.
• MATCH_PHRASE maps to a phrase query using the stored analyzer and position information.
• A lower score bound such as AND score() >= 0.5 is supported as a residual over the generated score. Upper bounds and other score predicates are rejected because filtering a pruned Top-K with those predicates could lose valid results.

3. Scalar WHERE to native row-ID prefilter

For the scalar predicates in the examples, the execution path is:

Bound Lance snapshot + validated Substrait predicate
    -> row-ID-only native scan (no user-column projection)
    -> native UINT64 _rowid allowlist
    -> vector / prepared full-text search
    -> Doris global Top-N and final projection

The candidate scanner and search scanner use the same pinned dataset snapshot. A query-local cache shares the candidate list across search readers using the dataset URI, version, and filter as its key. An empty allowlist selects zero rows.

The list is capped at 65,536 IDs. Broader predicates discard the entire partial list and retain the complete native Substrait prefilter on the search scanner. This bounds the C++ candidate list and preserves correctness. Profile counters expose prefilter time, candidate count, cache hits, and fallback count.

Scope and compatibility

• Automatic rewriting requires a direct Lance scan, an optional filter, a projection, and a single search ordering key with LIMIT. It does not rewrite across JOIN/EXISTS, add secondary sort keys, or support the internal-table SEARCH() DSL.
• Vector rewriting supports physical Float32 fixed-size-list vectors and constant query arrays. Exact l2_distance(), reversed vector ordering, incompatible vector types, and vector queries with unsupported residual predicates keep scalar execution.
• BM25 requires one supported top-level MATCH on a direct string column, a literal query, the stored analyzer, and ORDER BY score() DESC. Remaining scalar predicates must be fully convertible by the existing Lance predicate converter. Unsupported scored-query shapes fail explicitly rather than silently selecting incorrect candidates.
• The rewrite preserves bound snapshot/version metadata and output expression IDs, retains column pruning, and disables SQL result caching for rewritten searches.
• Existing explicit search TVF SQL-filter behavior is preserved.

Validation

• FE reactor build and the relevant Maven test suite passed; after the final compatibility guard, all 126 focused tests passed, including 12 new optimizer tests.
• FE Checkstyle passed.
• Changed BE production and test translation units passed C++ syntax checks; all affected C++ files passed clang-format 16 checks.
• Dependency download/MD5 and Foyer patch lifecycle tests passed.
• Rebuilt the exact pinned lance-c revision with the Foyer patch; native runtime smoke tests against that library passed for high-bit UINT64 row IDs, vector allowlists, empty allowlists, and prepared FTS allowlists with unchanged BM25 scores.
• Added BE tests for query-local cache reuse, empty masks, and the broad-filter fallback retaining the nearest row. Added an external-table regression suite comparing automatic SQL with explicit TVFs and scalar distance values.
• Full BE unit-test execution and cluster regression execution remain for CI; they were not run locally.

Release note

Support automatic native Lance search for eligible approximate vector Top-N and MATCH/BM25 SQL on Lance external tables.

Check List (For Author)

• Test
  • Regression test added (execution pending CI)
  • Unit Test added (FE passed; BE execution pending CI)
  • Manual native API smoke tests
• Behavior changed:
  • Yes. Eligible approximate vector queries use native search; eligible scored MATCH queries become supported, as shown above.
• Does this need documentation?
  • Yes. The SQL forms and current boundaries are described above; user documentation should accompany the feature release.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review of #69339 at e88159c found four P2 defects: native candidate pruning can suppress a non-movable projection error, NULL vectors lose the scalar distance error, stored NaN vectors can lose rows, and filtered rewrites can fail on older BEs during smooth upgrade. All four are reported inline. No existing inline threads were supplied or independently identified, and no existing P0/P1 comment needs carrying forward.

The intended path was traced from the bound Lance snapshot and SQL TopN through the new rewrite, existing search TVFs and segment planner, BE row-ID prefilter/native search, and Doris projection/global TopN/deferred payload reads. All 14 changed paths were reviewed by the main reviewer and assigned complete reviewers, with separate review of SQL semantics, native identities, and compatibility.

Critical checkpoint conclusions:

Checkpoint	Conclusion
Goal and functional evidence	The automatic approximate-vector and scored-MATCH paths are implemented and reuse existing search execution. The four inline counterexamples show remaining correctness/compatibility defects. Added tests establish intended shapes and nominal differential behavior by inspection; they were not executed in this review.
Scope and clarity	Direct scan/filter/project/single-key TopN guards keep the rewrite focused; existing TVF/planner/reader paths are reused. The dependency pin supplies the new allowlist API.
Concurrency, locks, and lock order	Sibling scanners share a scan-local cache; its shard mutex serializes complete first-fill publication. Native callbacks update existing counters and do not reenter the cache. No new race or reverse lock acquisition was identified. Concurrent first-fill and cancellation still need runtime coverage.
Lifecycle, memory, and initialization	Temporary native scanners and Arrow readers use RAII; native masks copy the IDs. Lists have a bounded 65,536-ID payload and publish only complete results. No new ownership cycle or cross-translation-unit static dependency was found.
Configuration	No new configuration item or dynamic-update contract is introduced.
Compatibility and rolling upgrade	Snapshot version/branch metadata are retained; no persisted-format or public-symbol removal was introduced. M3 identifies missing old-BE gating for SUBSTRAIT search-filter semantics. The exact pinned archive checksum and Foyer patch applicability were checked without building.
Thrift/protobuf definitions	No gensrc/thrift or gensrc/proto definition changed; field-ID/release-branch comparison is inapplicable. Existing request fields now carry SUBSTRAIT, covered by M3.
Parallel and special paths	Explicit TVF SQL filters, ordinary scans, exact/reversed vector ordering, unsupported scalar filters and wider vector types retain their existing paths. BM25 shape/analyzer/upper-bound restrictions fail explicitly. OLAP search rules were checked for interactions.
Conditional checks	Global OFFSET remains on Doris TopN and each split requests limit+offset. Empty masks select no rows; broad masks discard the entire list and retain the full native predicate. R1/R2 concerns about split scope, stable IDs and filter composition were dismissed with source evidence.
Test completeness	FE tests cover metrics, snapshots, filter pushdown, score thresholds, output pruning and fallbacks; BE tests cover empty/filtered masks, cache reuse and broad fallback; regression compares automatic searches with explicit TVFs and scalar L2 values. Missing differential cases include the three semantic findings; no mixed-version test covers M3.
Expected results	No golden-result file changed. New assertions and metric conversions were inspected. Runtime results are unverified under this static-only contract.
Observability	New prefilter time, rows, cache-hit and fallback counters complement existing scan/search profiles and explain output. No per-row timer was added.
Transactions and persistence	No transaction, EditLog, metadata-write/replay or visible-version protocol change occurs. Search and row-ID reads retain the bound positive dataset version; master-failover persistence changes are inapplicable.
Data writes, atomicity, and crashes	This is a read-path change; no new write atomicity or commit path is introduced. Error cleanup is guarded by existing/local ownership. M1/M2 identify errors that are incorrectly suppressed by pruning; M4 identifies NaN-valued rows that are discarded.
FE-to-BE propagation	Vector and FTS factories set the shared request prefilter and optional binary predicate; BE validation/application paths were traced. New wire fields are unnecessary; recipient compatibility is the M3 gap.
Performance	The bounded query-local list is reused across segment readers, wide payload pruning is retained, and partial-list fallback is conservative. Broad filters can incur an extra bounded probe before native fallback; no benchmark was run or performance claim verified.
Other issues and user focus	There was no additional user-provided focus. Complete changed-path and upstream/downstream review produced no further substantiated issue; duplicated specialist observations were merged.

Validation limits: static source and test inspection only; no build, test execution, cluster run, benchmark, production source edit, or submodule initialization. Pinned native sources and tests were inspected, the archive MD5 matched, and the existing Foyer patch passed a read-only applicability check. Runtime concurrency/cancellation, stable-ID compaction and multi-segment integration remain CI validation limits rather than unresolved review candidates.

Review convergence: converged after 3 rounds. In the final round, both complete reviewers and the separate risk specialist returned NO_NEW_VALUABLE_FINDINGS on the unchanged four-comment set; all changed paths, candidates, and risk focuses have explicit dispositions. No unresolved source question remains.

Review converged after 3 round(s).

[github-actions]

[P2] Preserve non-movable projections before pruning candidates

This rule also matches a project containing assert_true, which is explicitly a NoneMovableFunction:

TopN(d ASC, LIMIT 1)
  Project(assert_true(id > 0, 'bad') AS checked,
          l2_distance_approximate(embedding, [0,0]) AS d)
    LanceScan(id, embedding)

Put (id=-1, embedding=[100,100]) and (id=1, embedding=[0,0]) in one fragment without an ANN index. The scalar project raises bad before TopN. The rewrite requests native top_k=1, drops the failing row, and evaluates the retained project only on id=1, so the query succeeds. LogicalProject.pruneOutputs deliberately preserves these expressions, but this guard only checks DISTINCT. Skip the rewrite for non-movable projections, or preserve their evaluation before native pruning, and cover a failing assertion outside the candidate window.

[Gabriel39]

Fixed in #69346: the rewrite now skips projects containing NoneMovableFunction, so native candidate pruning cannot suppress assert_true evaluation on excluded rows. The new optimizer regression failed before the guard and passes after it.

[github-actions]

[P2] Preserve the scalar error for NULL vector inputs

The physical-type guard admits nullable fixed-size-list vectors and nullable elements. For example:

TopN(d ASC, LIMIT 1)
  Project(id, l2_distance_approximate(v, [0,0]) AS d)
    LanceScan(id, v)  // rows: (1,NULL), (2,[1,0])

Both approximate scalar functions call FunctionArrayDistance::_validate_no_nulls, so the original plan throws on row 1. The pinned native flat-KNN path propagates vector validity into distances and filters out NULL distances (vector/flat.rs and io/exec/knn.rs), so the rewritten plan silently returns row 2. NULL elements also lose Doris's explicit validation. Preserve scalar execution unless the required non-null invariants are proven, or validate surviving inputs before native ranking; add outer-NULL and element-NULL differential cases for both functions.

[Gabriel39]

The replacement #69346 explicitly defines eligible _approximate Top-N as provider-native candidate retrieval rather than scalar-equivalent validation. Native NULL vectors are excluded; NULL elements inside vectors are an unsupported input domain and must be absent. Exact functions retain scalar execution. The PR description now states these differences and includes counterexamples; FE/BE tests cover the selected contract. No separate session switch or full-vector validation scan is introduced.

[github-actions]

[P2] Gate Substrait search filters on backend support

During a smooth upgrade, old source BEs remain query/load available and FederationBackendPolicy can assign Lance splits to them. An ordinary query such as SELECT id, l2_distance_approximate(v,[0,0]) AS d FROM catalog.db.items WHERE id > 0 ORDER BY d LIMIT 1 now unconditionally emits this SUBSTRAIT search filter. The pre-change BE accepts only SQL search filters in _validate_external_search_request and returns NotSupported for this request, although it could execute the original scalar scan. The existing LanceScanNode.checkAdditionalTypeBackendCompatibility does not cover ordinary BIGINT/FLOAT/vector schemas, so it does not prevent the failure. Gate the rewrite/request on backend support, retaining scalar execution for vector queries or selecting only capable BEs, and cover the mixed-version path.

[Gabriel39]

Mixed-version backend capability gating is outside the requested scope of the replacement #69346. No rolling-upgrade compatibility guard was added.

[github-actions]

[P2] Preserve rows whose scalar distance is NaN

A supported Lance dataset can contain a non-NULL float32 vector [NaN,0]. With that as its only row and no ANN index, the input plan is:

TopN(d ASC, LIMIT 1)
  Project(id, l2_distance_approximate(v, [0,0]) AS d)
    LanceScan(id, v)

Ordinary Arrow ingestion copies the float values unchanged; the scalar distance returns NaN and Doris TopN retains its only row. The pinned native flat-KNN implementation explicitly removes NaN distances (io/exec/knn.rs:1047-1054), so replacing the scan here returns zero rows. This is a cardinality change even without an approximate index or candidate competition. It also survives the NULL eligibility fix because the vector and its elements are non-NULL. Preserve scalar row eligibility for this case, or restrict the rewrite to a proven compatible input domain, and add a differential regression with stored NaN vectors.

[Gabriel39]

In #69346, NaN-distance exclusion is documented as an intentional part of the provider-native approximate-search contract, including the single-row/all-NaN case. Exact functions remain scalar. Added reader coverage for native NULL-vector/NaN-distance exclusion for both L2 and dot; runtime smoke tests against the pinned library also verified mixed finite/NaN and all-NaN datasets.

[Gabriel39]

Superseded by #69346, targeting branch-4.2. The replacement includes the non-movable-projection guard, additional tests, the Doris/Lance function and metric mapping, explicit native approximate NULL/NaN semantics, and the ACL/EXISTS example with its current unsupported pushdown boundary.