[Gabriel39]

What problem does this PR solve?

Lance profiles do not expose Session cache-miss observation windows, scalar-index task grouping cannot compare runtime costs, and Foyer memory capacity is tied to the data-cache read block size.

This PR targets branch-4.2 and adds:

• LanceSessionIndexCacheMisses and LanceSessionMetadataCacheMisses, with availability and scope information. They report BE-wide Session deltas observed during a reader's dataset-open windows, including opening. Concurrent readers can overlap; these are not query-exclusive counts and must not be summed as an exact query total.
• The startup BE option enable_lance_scalar_index_auto_selection (default true). FE keeps coverage-based task grouping and does not estimate selectivity or add a session variable. In BE, lance-c compares current BTree page min/max and null-count metadata for segments covering the entire task. Estimated work is matching pages times rows per page; a broad estimate can select a scoped full scan. Serialized lookup bytes, decoded buffers and probe work are bounded. Unknown estimates retain the existing segment path. The current native API cannot bound variable-width expansion, so string/binary costs remain unknown: retain such a grouping index or skip such an alternative before opening its lookup. Bitmap/LabelList remain supported on their existing path and are not cost-ranked; vector/FTS semantics are unchanged.
• lance_data_cache_memory_capacity_bytes. Default 0 preserves the previous one-configured-block memory tier, including nondefault block sizes. Explicit capacities configure Foyer independently of block and disk capacities; negative values are rejected before unsigned conversion.

The native selector and decoded-memory fix are now merged upstream in lance-format/lance-c#98. This PR pins upstream main commit a51ab47a9c814a746cb1714c99fad93dd3a188b8, refreshes the source archive checksum, and removes the local runtime-selection patch together with its application/checksum handling. The pre-existing Foyer patch is unchanged and applies with zero fuzz.

Runtime decisions are exposed through Profile counters. Usage, cost-model limits and cache-counter scope are documented in docs/lance-cache-and-index-selection.md.

Release note

Add Lance Session cache-miss diagnostics, bounded BE runtime BTree selection controlled by a BE configuration, and independently configurable Foyer memory capacity.

Validation

• On the newly downloaded upstream archive plus the unchanged Foyer patch, cargo test --offline --locked passed all 515 tests, including 82 unit tests and 385 C API tests. The normal command ignores three C/C++ executable compilation tests. Coverage includes the compressed long-string allocation regression, decoded-budget limits, competing indices, scan fallback, fragment scope, NULL/deletion/stable-ID behavior and LIMIT/OFFSET.
• C and C++ API sources passed syntax-only compilation against the newly patched headers.
• Verified archive MD5, clean zero-fuzz patch application, cached no-op, stale-checksum source replacement, and equality of the resulting tested source. Selector/scanner/segment sources match merged upstream exactly.
• All nine affected C++ files passed the repository formatting runner with clang-format 16.0.6. Shell syntax and staged whitespace checks passed.
• Existing FE validation passed 74 focused tests and FE Checkstyle; this dependency synchronization changes no FE code.
• Updated the array-predicate regression profile checks to account for explicit runtime cost-selected scans while retaining request/search/scan accounting, exact candidate counts and strict LabelList coverage. The actual assertion block passes six valid profile paths and rejects five invalid profiles; the previous assertion reproduces the observed failure. The focused native runtime-selection test and full-suite Groovy 4.0.19 parsing pass. Full external regression execution remains for CI.
• Full Doris BE build and unit-suite validation remain for CI and have not been verified locally.

Check List (For Author)

• Test
  • Unit Test
  • Manual test: native source/patch lifecycle and header compatibility checks above.
  • Regression test
• Behavior changed:
  • Yes. BE runtime BTree selection is enabled by default and can be disabled. Foyer's default memory behavior is preserved.
• Does this need documentation?
  • Yes. Usage and native memory limits are documented.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Reviewed all 18 changed files at db0cbc2, including upstream statistics/predicate translation, split ownership, native cache contracts and downstream reader lifecycle. One new P2 finding: advisory selectivity estimation can abort a valid ordinary Lance query containing a nonfinite IN literal. No existing inline threads or additional user focus points were supplied.

Critical checkpoints:

• Goal and proof: cache diagnostics, selectivity-aware ordinary scalar-index choice and independent Foyer memory capacity are implemented, with the query-availability exception reported inline. Added tests cover the primary behaviors; they were inspected, not executed.
• Scope and clarity: changes are confined to the three described Lance enhancements, supporting configuration, tests and documentation.
• Concurrency: shared initialization remains guarded by call_once; cache snapshots are non-destructive and profile updates use existing synchronized/atomic facilities. No new thread, lock nesting or lock-order change was introduced. Overlapping BE-wide observation windows are explicitly disclosed.
• Lifecycle and static initialization: dataset reuse retains one baseline, dataset close samples once, repeated close is gated by the null handle, and a new open replaces its baseline. The existing function-local session owner remains; no new cycle or cross-translation-unit initializer dependency was found.
• Configuration: the new BE capacity is startup-only and requires restart. Zero retains one configured block, explicit negatives are rejected before conversion, native minimum sizes are enforced, and a disabled data cache ignores Foyer configuration. The FE switch is session-scoped and forwarded.
• Compatibility: existing native APIs are used; storage layouts and serialized scan fields do not change. No new rolling-upgrade format dependency was found.
• Thrift/protobuf definitions: none changed, so field-id/release-branch comparison is not applicable.
• Parallel paths: selection is restricted to ordinary scans; vector/FTS outer-filter semantics, metadata-count handling and uncovered-fragment fallbacks remain intact. Row-id dataset reads share the same cache lifecycle.
• Special conditions: eligibility, expression budgets, exact OR branches, ambiguous-index exclusion and unique fragment ownership remain enforced. Unknown estimates preserve coverage/name tie breaking; correlated predicates and heuristic estimates remain documented cost limitations.
• Test coverage: FE tests inspect estimates, coverage costs, unknown/invalid/zero estimates, residual fields, forwarding, serialization and real column statistics. BE tests inspect defaults, custom capacity, negative rejection, disabled-cache independence, miss deltas and double close. Nonfinite estimator failures are missing and should be covered with the inline fix.
• Test results: no golden result files changed; new assertions were checked against the intended behavior. No build, test or runtime validation was performed in this static-only review.
• Observability: new miss counters include scope and availability strings; shared-window counts are not exact query totals. Existing execution/cache metrics remain separate.
• Transactions and persistence: no transaction state, EditLog path, failover protocol or persisted metadata is modified.
• Data writes and atomicity: no write/commit path is changed; no new crash-time write atomicity issue was identified.
• FE/BE variable propagation: the selector is FE-owned, its session variable uses needForward, and existing selected-segment/filter serialization remains sufficient; no new BE wire variable is required.
• Performance: native snapshots avoid per-cache-entry traversal, selection remains outside per-row execution, and no new hot-loop allocation was found. Runtime speedups and actual cache memory/eviction behavior were not measured.
• Core invariants: snapshot/version, delete, NULL/materialization and fragment-output semantics remain intact in the inspected paths. No new owned BE buffer or cache-handle leak was found. Error handling has the single availability defect reported inline.
• Additional issues and limits: the missed-case sweep found no other verified issue. Concurrent/shared-profile failure injection, reopen/error scenarios and actual Foyer growth/eviction lack new integration coverage. These are validation limits, not unresolved defect candidates.

Review converged in 2 rounds after independent BE, FE and risk-focused rechecks of the complete final finding set. Static inspection does not establish runtime test or benchmark success.

Review converged after 2 round(s).

[github-actions]

[P2] Keep advisory selectivity failures from aborting Lance scans

For an analyzed Lance DOUBLE column d with finite min/max statistics, this valid reduced plan reaches the new estimator with the default switch enabled:

PhysicalFilter(d IN (1.0, 2.0, 3.0, CAST('Infinity' AS DOUBLE)))
  PhysicalFileScan(Lance table)

ExpressionEstimation.visitLiteral gives the Infinity literal equal infinite endpoints, then FilterEstimation.updateInPredicateColumnStatistics checks abs(max - min) < 1e-6; Infinity minus Infinity is NaN, so it throws literal's min/max doesn't equal. Previously StatsCalculator.estimate caught this optional statistics failure and kept the child statistics, and the nonfinite IN stayed a Doris residual. Retrying estimation here without that boundary makes the query fail during translation, even when the table has no scalar index. Leave selectivity unknown when advisory estimation fails and add a regression covering this case.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review of a4e757b9a7dea0b114c27988d87951a83f910dc7: one new P2 finding about decoded memory used by optional BTree lookup probes. No verified P0/P1 finding remains. Review converged after two rounds, including independent normal and risk-focused reviews of all 13 changed files and the exact pinned native dependencies.

Existing inline #4375522650 was independently revalidated: the FE advisory selectivity estimator has been removed from PhysicalPlanTranslator.addConjunctsToPlanNode, so that P2 no longer applies and is not reposted. There were no additional user-provided focus points.

Critical checkpoint	Static conclusion
Goal and behavioral evidence	Session miss windows, BE runtime BTree selection and independent Foyer capacity are implemented; optional probe memory needs the inline fix. Inspected BE/FE and embedded native tests; no test execution is claimed.
Scope and clarity	Changes remain within Lance configuration, reader diagnostics, native selection, tests and documentation; FE production grouping stays coverage-based.
Concurrency and locks	call_once publishes the shared Session and retained initialization Status; snapshots read cumulative counters without resets. No new thread or lock-order dependency. Cache admission does not bound probe loader allocation (P2).
Lifecycle and static initialization	Function-local singleton remains lazy; metrics deregister before Session close, scanners close before datasets, and the dataset guard prevents duplicate close-time deltas. No new static initialization dependency or ownership cycle identified.
Configuration	Both new settings are startup-only and documented. Zero memory preserves one configured block; negative memory is rejected before unsigned conversion when caching is enabled, and native validation enforces memory >= block. Disabled data caching ignores Foyer-only settings.
ABI and storage compatibility	Additive native setter matches the C++ call. Both patch checksums refresh cached sources, and regular thirdparty builds replace header/archive together. No persisted storage or transport format is changed.
Thrift/protobuf definitions	None changed; release-branch field/enum comparison is inapplicable.
Parallel paths	Auto selection is limited to normal explicitly segmented scans. Vector/FTS, uncovered fragments, unsegmented scans, metadata counts, Bitmap/LabelList and unsupported estimates preserve their existing paths.
Conditional safeguards	Alternatives require a top-level keyed field, known coverage of the entire task, compatible scoped expression and a strictly cheaper estimate. Actual index type, exactness and stable-ID/address checks remain downstream; unknown estimates retain the prior path.
Functional and negative tests	Added tests cover config defaults/override/negative values, miss deltas/repeated close, indexed/uncovered domains with the switch off/on, estimator NULL/range/IN bounds, competing indexes, broad fallback and scoped pagination. Missing compressed-string allocation coverage exposes the P2.
Expected results	No golden output changed. Inspected expected rows and 4096-row page estimates agree with the synthetic fixtures.
Observability	Decision counters match native names; Session availability/scope explicitly describe overlapping BE-wide windows. Separate probe allocation/I/O attribution and representative benchmarks remain validation limits.
Transactions and persistence	No commit/publish/journal path changed; fixed snapshot/fragment identity remains the read boundary.
Writes, atomicity and crash cleanup	Production changes are read-only acceleration/diagnostics and cache configuration; existing scanner/dataset cleanup remains in place. No new write-atomicity issue identified.
FE-to-BE variables	No new transmitted fields. BE startup options are wired locally into scanner and Session setup.
Performance and memory	Serialized bytes, page count, candidate count and page/predicate work are capped, but decoded optional lookup memory is not bounded before allocation (P2).
Other core invariants	Traced complete filters, deletion vectors, row-ID domains and LIMIT/OFFSET after mask/refinement within the original explicit fragments. No additional visibility, version, delete or nullable-column defect was verified.

Coverage limits: this review is static-only; no builds, tests or submodule initialization were performed. The new auto-selection fixture does not newly exercise alternatives across deletes, stable IDs, imported/versioned indexes, probe errors or concurrent cache windows, and no external-writer performance matrix/full Doris build was verified. These validation limits are assessed and disclosed; no review candidate remains unresolved.

Review converged after 2 round(s).

[Gabriel39]

The runtime BTree selection feature is now proposed upstream in lance-format/lance-c#98. This PR is draft while that change is reviewed. After the upstream PR merges, I will update the dependency to the latest upstream main commit containing the feature and remove the new local runtime-selection patch and its application/checksum handling. The pre-existing Foyer patch is separate.

The standalone upstream implementation passed 497 Rust tests, all three C/C++ compilation/execution tests, Rustfmt, Clippy, and Rustdoc locally. Upstream CI is running; a full Doris BE build remains subject to Doris CI.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Reviewed pinned head f1d1441169ca046338e4cf8edeca406a493043f3: no new actionable findings after complete static review of all 13 changed files and the pinned native selector. The review converged in two rounds.

Both existing P2 issues no longer apply to this head: comment 4375522650's FE advisory estimator retry has been removed; comment 4375895764's decoded lookup expansion is prevented by the upstream variable-width rejection and pre-materialization decoded budget, including cache-hit charging. Neither is reposted. No existing P0/P1 finding remains to carry forward.

Critical checkpoint conclusions:

1. Goal: Session cache diagnostics, runtime BTree selection and independent Foyer memory capacity are implemented with focused tests inspected.
2. Scope: the BE wiring, native dependency update, documentation and tests are focused; old/new Cargo manifests and lockfiles are byte-identical.
3. Concurrency: call_once publishes the shared Session; reader-local sampling is serial, with atomic/locked Profile updates. No new thread, lock nesting or lock-order issue found.
4. Lifecycle/static initialization: scanner retirement precedes dataset sampling/release; repeated close is guarded; no new ownership cycle or cross-translation-unit initialization dependency found.
5. Configuration: both new options are startup-only as documented. Zero memory capacity preserves one configured read block; enabled negative values fail before unsigned conversion; native validation rejects sub-block memory.
6. Compatibility: pinned headers/library provide the new C API, archive checksum matches, and the unchanged Foyer patch passes a zero-fuzz dry run. Storage/wire formats are unchanged.
7. Thrift/protobuf: no definitions changed; release-field comparison is inapplicable.
8. Parallel paths: uncovered, disabled, debug-grouped, metadata COUNT, vector, FTS and row-ID fetch paths retain their intended behavior; ranking is enabled only for normal segment scans.
9. Conditional guards: explicit whole-task coverage, snapshot, supported-type/schema/format and budget checks preserve conservative fallbacks. The complete filter remains in place after index replacement or scoped full scan.
10. Test coverage: inspected C++/JUnit/native tests cover configuration, diagnostics, enabled/disabled ranking, competing indexes, partial coverage, full scans, decoded budgets and limits. Additional coverage limits are below.
11. Expected results: no result files changed; tests were not executed in this review.
12. Observability: five runtime decision counters, Session miss deltas, availability/scope information and configuration logging match their producers.
13. Persistence/transactions: no EditLog, commit, replay or failover state changed.
14. Data writes/atomicity: no production write path changed; read snapshot, deletion and stable-row-ID semantics remain downstream of selection.
15. FE-BE variables: none added; existing fixed-version task/filter fields are reused, and FE grouping remains coverage-based without selectivity estimation.
16. Performance/memory: serialized lookup bytes, decoded buffers, pages and predicate-probe work are bounded; unknown estimates retain the existing path. Additional metadata sorting was reviewed; no material regression was substantiated.
17. Other issues/user focus: both existing threads were revalidated; no additional user-provided focus, unresolved code candidate or unreviewed changed path remains.

Validation limits: static review only; no builds, tests or performance benchmarks were run. New Doris end-to-end regression coverage is absent. Direct concurrent-reader/reopen/statistics-failure tests and the entire native stable-ID/delete/OR/unsupported-expression matrix with auto-selection enabled are not present; the relevant unchanged downstream guarantees were inspected directly. These are validation limits, not additional verified defects.

Review converged after 2 round(s).

[Gabriel39]

run buildall

[Gabriel39]

/review

[Gabriel39]

Fixed the array-predicate regression assertion in 501aa25.

The fixture contains only 16 rows. Its BTree lookup-page estimate can exceed the scoped scan cost, so the enabled runtime selector deliberately chooses a scan. Result rows were correct; the previous assertion still required one index search and failed with expected 1, actual 0.

The suite now requires requested segments to equal index searches plus explicitly reported automatic full scans. Cost-selected scans must report zero index candidates, and unrelated segment fallbacks still fail. Label-only predicates must continue to search LabelList. Existing result and EXPLAIN checks are preserved.

Validation: reproduced the old assertion failure; the actual updated assertion block passes six valid profile paths and rejects five invalid profiles. The native runtime-selection C API test and full-suite Groovy 4.0.19 parsing pass. All affected PR C++ files pass clang-format 16. Full external regression execution remains for CI; this update does not claim that the pipeline has passed.

[github-actions]

Reviewed all 14 changed files at head 501aa25, plus affected FE grouping, scanner scheduling/profile publication, the exact pinned lance-c implementation and its transitive Lance reader/cache paths. One new P2 finding is attached: dataset-close-only Session statistics can miss the immutable final profile after an early LIMIT. No new P0/P1 finding or applicable existing P0/P1 comment was confirmed.

Review convergence: converged in 3 rounds, including two complete-review scopes and a separate native risk investigation. All candidates and initial risks have evidence-backed dispositions; no unresolved code-review question remains.

Critical checkpoint conclusions:

• Goal and tests: runtime BTree selection and independent Foyer capacity match the intended design; completed-profile diagnostics have the attached publication-timing defect. Direct-reader tests do not cover its scheduler ordering.
• Scope: the changes reuse existing Session, scanner and profile interfaces and stay focused on the three stated features.
• Concurrency: Session initialization remains protected by call_once, shared cache entries retain native ownership, and profile updates use existing synchronized APIs. No new thread or lock-order cycle was found; worker lifetime versus pipeline completion causes the inline issue.
• Lifecycle and static initialization: lazy function-local manager construction and scanner/dataset guards remain intact; metrics hooks are removed before the Session closes. Repeated reader close cannot recount a dataset window.
• Configuration: both options are startup-only as documented. Zero memory capacity preserves one configured block; enabled negative capacity is rejected before unsigned conversion, and native validation enforces at least one block. Disabled Foyer ignores its configuration.
• Compatibility: the immutable archive checksum matches and the existing Foyer patch applies with zero fuzz in a dry-run. Native API signatures and header/archive installation are consistent statically; no persisted format or FE-BE protocol change was introduced.
• Shared definitions: no gensrc Thrift/protobuf definition changed, so release-branch field-ID comparison is inapplicable.
• Parallel paths: normal scans retain explicit snapshot/fragment domains and full filters; alternatives require full task coverage. Unknown costs and unsupported/partial domains preserve the existing path. Bitmap/LabelList, vector/FTS and row-ID safeguards remain separate and intact.
• Conditional/error paths: optional estimates retain conservative fallback; actual scanner configuration/search errors still propagate. Snapshot failures remain marked unavailable rather than changing query rows.
• Test coverage: inspected BE capacity/cache/scalar-segment tests, FE grouping test, array-predicate regression and native selection/budget tests. They cover enabled/disabled selection, uncovered domains, Boolean/NULL/residual behavior, stable IDs/deletes and LIMIT paths; early-stop final-profile publication needs coverage. Broader concurrent/failure-injection and external runtime matrices remain unexecuted.
• Expected results: no result files changed; modified assertions retain exact rows, candidate accounting and strict LabelList expectations while allowing explicit cost-selected scans.
• Observability: native decision metrics are mapped consistently and BE-wide overlapping Session-window semantics are documented. The attached finding addresses final counter publication. INFO logging remains bounded by manager construction.
• Transactions and persistence: no transaction, EditLog, commit visibility or MoW delete-bitmap state is modified; snapshot and native live-row filtering contracts remain preserved.
• Data writes and crash atomicity: no data-write path is added; existing native Session ownership retains cache cleanup responsibilities.
• FE-BE variables: both new settings are BE-local and require no new serialized variable or alternate FE send path.
• Performance and memory: probe inputs, decoded fixed-width buffers and comparison work have explicit per-selection limits; variable-width estimates are declined before lookup opening and cache hits are charged. This is not a whole-query memory cap or a measured performance claim.
• Other concerns: continued across all changed files and special paths after the finding; no additional actionable defect was verified.

Existing review context: 4375522650 is fixed because FE translation no longer retries advisory selectivity estimation. 4375895764 is fixed by the pinned upstream pre-materialization decoded-memory guards and variable-width skip; its compressed-string regression is present. Neither is reposted. No additional user focus was supplied.

Validation limits: static review only. No builds, tests, benchmarks, external regression execution, source edits or submodule initialization were performed. Checksum verification, source inspection and patch dry-run do not establish successful compilation or runtime validation.

Review converged after 3 round(s).

[github-actions]

[P2] Publish Session counters before the final profile is frozen

For a remote Lance scan with concurrent readers and an early LIMIT, one worker can satisfy the limit while another, whose dataset is already open, is still blocked in lance_scanner_next. stop_scanners() requests stopping without joining that worker, which retains its ScannerDelegate; dropping the local state's delegates therefore does not run this dataset-close collection yet. Meanwhile, pipeline task completion calls PipelineFragmentContext::_close_fragment_instance(), which saves an independent TRuntimeProfileTree in QueryContext. When the worker eventually returns and closes its dataset, these counter updates arrive after that final copy and never reach the saved profile. The completed query consequently loses that reader's observation window, leaving the new miss counters unavailable or partial despite successful statistics access. Publish an available snapshot before final profile serialization, with synchronization and delta accounting so later close does not double count, and cover early LIMIT with a second worker held in a native read. The current test explicitly closes the reader before inspecting its profile and misses this path.