Repository navigation
[chore](build) Update lance-c to the latest upstream main revision - #69332
Conversation
### What problem does this PR solve? Related PRs: lance-format/lance-c#97, lance-format/lance-c#96 Update the pinned lance-c revision from `8e9e09ecc795c91ce7f97513db94c2d91eec799d` to `4fafcd64525b274b113208390bbcca86bbbc093b`, the latest upstream `main` commit when this PR was prepared, and update the archive checksum. The new revision includes the upstream fix for cold-session vector prewarm when an index name matches a different schema column. It resolves the indexed field from segment metadata while preserving concurrent segment opening/prewarming. The revision also includes the row-address allowlist/blocklist prefilter APIs from lance-c apache#96. Refresh one C++ test context in the existing Foyer patch to accommodate the test added by apache#96. The patch's added/deleted code lines are unchanged; no local prewarm patch is introduced. ### Release note Fix Lance index prewarm failures when a vector index name matches a different schema column. ### Check List (For Author) - Test - [x] Manual test - Downloaded the exact pinned archive and verified its checksum through the Doris downloader. - `bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src`: passed all scenarios, including fresh extraction, idempotence, cache invalidation, and patch-failure handling. - `cargo metadata --locked --offline --format-version 1` on the patched source: passed dependency resolution, including Foyer. - Compiled the patched upstream C and C++ test sources with `-fsyntax-only`: passed. - `bash -n` on the dependency scripts and `git diff --check`: passed. - Full Doris build and runtime regression tests were not run locally; requested through `run buildall`. - Behavior changed: - [x] Yes. Includes the upstream prewarm fix and new prefilter APIs. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
Replace the local prewarm field-resolution patch with the pinned upstream 4fafcd64525b274b113208390bbcca86bbbc093b revision from apache#69332. Restore the existing Foyer-only downloader and refresh its patch context. Validated the downloader patch/cache harness, locked Cargo dependency resolution, patched C/C++ test source syntax, and shell/diff checks.
There was a problem hiding this comment.
Reviewed the complete two-file diff at eeb54b2 and the exact imported lance-c changes (#96 and #97). No actionable introduced defect was found. There are no existing inline findings to carry forward and no additional user focus points.
Critical checkpoint conclusions:
- Purpose and evidence: the immutable pin includes the cold-session vector index/column-name collision fix and row allow/block APIs. Upstream regression sources cover the collision, shared-session reuse, snapshot behavior and row-mask correctness.
- Scope: the change is focused on the dependency URL/name/source/checksum and Foyer patch context. The overlay's added/deleted code is unchanged.
- Concurrency: the dataset lock is released before I/O; per-segment open and prewarm remain concurrent with fail-fast error propagation/cancellation. No new lock order or Doris thread entry is introduced.
- Lifetimes/static initialization: Arc-owned dataset snapshots, sessions and scanner masks retain their dependencies; no new cross-translation-unit initialization dependency was found.
- Configuration: no configuration item or dynamic-update behavior changes.
- ABI/storage compatibility: the C/C++ APIs are additive, opaque handles preserve the boundary, and the underlying Lance SDK revision and persisted formats are unchanged.
- Thrift/protobuf: no gensrc definitions change.
- Parallel paths: ordinary/colliding index names, vector/scalar/FTS prewarm, stable/nonstable row IDs, indexed/flat search, and synchronous/asynchronous/poll scanning were traced. Match and Phrase plan rewrites both retain the external mask.
- Conditional behavior: invalid inputs and scalar-segment/mask conflicts fail explicitly; same-field, missing-index and scalar/FTS cases preserve SDK handling.
- Test coverage: imported sources cover cold and reopened sessions, multiple segments, deterministic scheduling/cancellation, empty/replaced/combined masks, SQL filters and prepared FTS scoring. Additional pinned-SDK tests cover indexed vector filtering and mask execution; no build or runtime test was executed for this review.
- Expected results: no Doris regression-result files change; inspected assertions check selected rows, scores, errors, cache reuse and cancellation.
- Observability: existing FFI error reporting and underlying SDK diagnostics remain; no new Doris service path requires metrics. The colliding-vector workaround bypasses aggregate SDK prewarm info logs, without a substantiated correctness impact.
- Transactions/persistence: no EditLog, transaction commit/replay or persisted-metadata format changes.
- Writes/crash behavior: the imported changes operate on read/search/prewarm paths; no new data-write atomicity or crash-recovery mechanism is introduced.
- FE/BE variables: no new cross-process fields or propagation paths.
- Performance: concurrent segment I/O is retained; row masks filter before ranking and survive prepared-plan reconstruction. No actionable new performance regression was established.
- Other integration checks: the downloaded archive matches MD5 65e924fb62137c5dd4d58af3fb33923a; all 15 Foyer patch targets pass a zero-fuzz dry run. The unchanged upstream manifests/lockfile, in-memory patched lock graph, downloader invalidation and third-party-only BE linkage were inspected.
Validation limit: this is a static review, including archive verification and patch dry-run only. Doris/upstream builds, link checks, shell test suites and runtime regressions were not run. No unresolved candidate or uncovered static-review scope remains.
Review converged after 1 round(s).
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
… revision #69332 (#69335) ### What problem does this PR solve? Backport #69332 to `branch-4.2` (source commit `eeb54b230ddf7a7a5a06cbaab1bdbd6796d8ff22`). Pin lance-c to `4fafcd64525b274b113208390bbcca86bbbc093b` and update the archive checksum. This includes the upstream fix for cold-session vector prewarm when an index name matches another schema column, plus the row-address allowlist/blocklist prefilter APIs. Refresh the existing Foyer patch's base revision and one C++ test context. Its actual added/deleted code is unchanged. The target branch's only cherry-pick conflict was the previous revision's descriptive comment. ### Validation - Passed `bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src` on this branch: platform definitions, fresh extraction, idempotence, re-extraction, existing markers, stale-cache invalidation, and patch-failure handling. - Verified the pinned archive checksum through the downloader. - Passed shell syntax checks and `git diff --cached --check`. - Confirmed the patch payload is unchanged and the resulting patch matches #69332 exactly. - Full Doris build and runtime regressions were not run locally; requested through `run buildall`. ### Release note Fix Lance index prewarm failures when a vector index name matches a different schema column. ### Check List (For Author) - Test: dependency patch harness and manual checks above. - Behavior changed: Yes, includes the upstream prewarm fix and new prefilter APIs. - Does this need documentation: No.
… revision #69332 (#69336) ### What problem does this PR solve? Backport #69332 to `branch-4.1` (source commit `eeb54b230ddf7a7a5a06cbaab1bdbd6796d8ff22`). Pin lance-c to `4fafcd64525b274b113208390bbcca86bbbc093b` and update the archive checksum. This includes the upstream fix for cold-session vector prewarm when an index name matches another schema column, plus the row-address allowlist/blocklist prefilter APIs. Refresh the existing Foyer patch's base revision and one C++ test context. Its actual added/deleted code is unchanged. The target branch's only cherry-pick conflict was the previous revision's descriptive comment. ### Validation - Passed `bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src` on this branch: platform definitions, fresh extraction, idempotence, re-extraction, existing markers, stale-cache invalidation, and patch-failure handling. - Verified the pinned archive checksum through the downloader. - Passed shell syntax checks and `git diff --cached --check`. - Confirmed the patch payload is unchanged and the resulting patch matches #69332 exactly. - Full Doris build and runtime regressions were not run locally; requested through `run buildall`. ### Release note Fix Lance index prewarm failures when a vector index name matches a different schema column. ### Check List (For Author) - Test: dependency patch harness and manual checks above. - Behavior changed: Yes, includes the upstream prewarm fix and new prefilter APIs. - Does this need documentation: No.
What problem does this PR solve?
Related PRs: lance-format/lance-c#97, lance-format/lance-c#96
Update the pinned lance-c revision from
8e9e09ecc795c91ce7f97513db94c2d91eec799dto4fafcd64525b274b113208390bbcca86bbbc093b, the latest upstreammaincommit when this PR was prepared, and update the archive checksum.The new revision includes the upstream fix for cold-session vector prewarm when an index name matches a different schema column. It resolves the indexed field from segment metadata while preserving concurrent segment opening/prewarming. The revision also includes the row-address allowlist/blocklist prefilter APIs from lance-c #96.
Refresh one C++ test context in the existing Foyer patch to accommodate the test added by #96. The patch's added/deleted code lines are unchanged; no local prewarm patch is introduced.
Release note
Fix Lance index prewarm failures when a vector index name matches a different schema column.
Check List (For Author)
bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src: passed all scenarios, including fresh extraction, idempotence, cache invalidation, and patch-failure handling.cargo metadata --locked --offline --format-version 1on the patched source: passed dependency resolution, including Foyer.-fsyntax-only: passed.bash -non the dependency scripts andgit diff --check: passed.run buildall.Check List (For Reviewer who merge this PR)