Skip to content

[chore](build) Update lance-c to the latest upstream main revision - #69332

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/update-lance-c-main-20261020
Oct 21, 2026
Merged

Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/update-lance-c-main-20261020

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PRs: lance-format/lance-c#97, lance-format/lance-c#96

Update the pinned lance-c revision from 8e9e09ecc795c91ce7f97513db94c2d91eec799d to 4fafcd64525b274b113208390bbcca86bbbc093b, the latest upstream main commit when this PR was prepared, and update the archive checksum.

The new revision includes the upstream fix for cold-session vector prewarm when an index name matches a different schema column. It resolves the indexed field from segment metadata while preserving concurrent segment opening/prewarming. The revision also includes the row-address allowlist/blocklist prefilter APIs from lance-c #96.

Refresh one C++ test context in the existing Foyer patch to accommodate the test added by #96. The patch's added/deleted code lines are unchanged; no local prewarm patch is introduced.

Release note

Fix Lance index prewarm failures when a vector index name matches a different schema column.

Check List (For Author)

  • Test
    • Manual test
      • Downloaded the exact pinned archive and verified its checksum through the Doris downloader.
      • bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src: passed all scenarios, including fresh extraction, idempotence, cache invalidation, and patch-failure handling.
      • cargo metadata --locked --offline --format-version 1 on the patched source: passed dependency resolution, including Foyer.
      • Compiled the patched upstream C and C++ test sources with -fsyntax-only: passed.
      • bash -n on the dependency scripts and git diff --check: passed.
      • Full Doris build and runtime regression tests were not run locally; requested through run buildall.
  • Behavior changed:
    • Yes. Includes the upstream prewarm fix and new prefilter APIs.
  • Does this need documentation?
    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Related PRs: lance-format/lance-c#97, lance-format/lance-c#96

Update the pinned lance-c revision from `8e9e09ecc795c91ce7f97513db94c2d91eec799d` to `4fafcd64525b274b113208390bbcca86bbbc093b`, the latest upstream `main` commit when this PR was prepared, and update the archive checksum.

The new revision includes the upstream fix for cold-session vector prewarm when an index name matches a different schema column. It resolves the indexed field from segment metadata while preserving concurrent segment opening/prewarming. The revision also includes the row-address allowlist/blocklist prefilter APIs from lance-c apache#96.

Refresh one C++ test context in the existing Foyer patch to accommodate the test added by apache#96. The patch's added/deleted code lines are unchanged; no local prewarm patch is introduced.

### Release note

Fix Lance index prewarm failures when a vector index name matches a different schema column.

### Check List (For Author)

- Test
    - [x] Manual test
        - Downloaded the exact pinned archive and verified its checksum through the Doris downloader.
        - `bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src`: passed all scenarios, including fresh extraction, idempotence, cache invalidation, and patch-failure handling.
        - `cargo metadata --locked --offline --format-version 1` on the patched source: passed dependency resolution, including Foyer.
        - Compiled the patched upstream C and C++ test sources with `-fsyntax-only`: passed.
        - `bash -n` on the dependency scripts and `git diff --check`: passed.
        - Full Doris build and runtime regression tests were not run locally; requested through `run buildall`.
- Behavior changed:
    - [x] Yes. Includes the upstream prewarm fix and new prefilter APIs.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 20, 2026
Replace the local prewarm field-resolution patch with the pinned upstream
4fafcd64525b274b113208390bbcca86bbbc093b revision from apache#69332.
Restore the existing Foyer-only downloader and refresh its patch context.

Validated the downloader patch/cache harness, locked Cargo dependency
resolution, patched C/C++ test source syntax, and shell/diff checks.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete two-file diff at eeb54b2 and the exact imported lance-c changes (#96 and #97). No actionable introduced defect was found. There are no existing inline findings to carry forward and no additional user focus points.

Critical checkpoint conclusions:

  • Purpose and evidence: the immutable pin includes the cold-session vector index/column-name collision fix and row allow/block APIs. Upstream regression sources cover the collision, shared-session reuse, snapshot behavior and row-mask correctness.
  • Scope: the change is focused on the dependency URL/name/source/checksum and Foyer patch context. The overlay's added/deleted code is unchanged.
  • Concurrency: the dataset lock is released before I/O; per-segment open and prewarm remain concurrent with fail-fast error propagation/cancellation. No new lock order or Doris thread entry is introduced.
  • Lifetimes/static initialization: Arc-owned dataset snapshots, sessions and scanner masks retain their dependencies; no new cross-translation-unit initialization dependency was found.
  • Configuration: no configuration item or dynamic-update behavior changes.
  • ABI/storage compatibility: the C/C++ APIs are additive, opaque handles preserve the boundary, and the underlying Lance SDK revision and persisted formats are unchanged.
  • Thrift/protobuf: no gensrc definitions change.
  • Parallel paths: ordinary/colliding index names, vector/scalar/FTS prewarm, stable/nonstable row IDs, indexed/flat search, and synchronous/asynchronous/poll scanning were traced. Match and Phrase plan rewrites both retain the external mask.
  • Conditional behavior: invalid inputs and scalar-segment/mask conflicts fail explicitly; same-field, missing-index and scalar/FTS cases preserve SDK handling.
  • Test coverage: imported sources cover cold and reopened sessions, multiple segments, deterministic scheduling/cancellation, empty/replaced/combined masks, SQL filters and prepared FTS scoring. Additional pinned-SDK tests cover indexed vector filtering and mask execution; no build or runtime test was executed for this review.
  • Expected results: no Doris regression-result files change; inspected assertions check selected rows, scores, errors, cache reuse and cancellation.
  • Observability: existing FFI error reporting and underlying SDK diagnostics remain; no new Doris service path requires metrics. The colliding-vector workaround bypasses aggregate SDK prewarm info logs, without a substantiated correctness impact.
  • Transactions/persistence: no EditLog, transaction commit/replay or persisted-metadata format changes.
  • Writes/crash behavior: the imported changes operate on read/search/prewarm paths; no new data-write atomicity or crash-recovery mechanism is introduced.
  • FE/BE variables: no new cross-process fields or propagation paths.
  • Performance: concurrent segment I/O is retained; row masks filter before ranking and survive prepared-plan reconstruction. No actionable new performance regression was established.
  • Other integration checks: the downloaded archive matches MD5 65e924fb62137c5dd4d58af3fb33923a; all 15 Foyer patch targets pass a zero-fuzz dry run. The unchanged upstream manifests/lockfile, in-memory patched lock graph, downloader invalidation and third-party-only BE linkage were inspected.

Validation limit: this is a static review, including archive verification and patch dry-run only. Doris/upstream builds, link checks, shell test suites and runtime regressions were not run. No unresolved candidate or uncovered static-review scope remains.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.20% (35840/46426)
Line Coverage 62.47% (404498/647543)
Region Coverage 58.92% (341502/579564)
Branch Coverage 59.90% (157975/263750)

@Gabriel39
Gabriel39 merged commit bffde19 into apache:master Oct 21, 2026
38 checks passed
yiguolei pushed a commit that referenced this pull request Oct 21, 2026
… revision #69332 (#69335)

### What problem does this PR solve?

Backport #69332 to `branch-4.2` (source commit
`eeb54b230ddf7a7a5a06cbaab1bdbd6796d8ff22`).

Pin lance-c to `4fafcd64525b274b113208390bbcca86bbbc093b` and update the
archive checksum. This includes the upstream fix for cold-session vector
prewarm when an index name matches another schema column, plus the
row-address allowlist/blocklist prefilter APIs.

Refresh the existing Foyer patch's base revision and one C++ test
context. Its actual added/deleted code is unchanged. The target branch's
only cherry-pick conflict was the previous revision's descriptive
comment.

### Validation

- Passed `bash thirdparty/test/lance-prefilter-patch-test.sh
thirdparty/src` on this branch: platform definitions, fresh extraction,
idempotence, re-extraction, existing markers, stale-cache invalidation,
and patch-failure handling.
- Verified the pinned archive checksum through the downloader.
- Passed shell syntax checks and `git diff --cached --check`.
- Confirmed the patch payload is unchanged and the resulting patch
matches #69332 exactly.
- Full Doris build and runtime regressions were not run locally;
requested through `run buildall`.

### Release note

Fix Lance index prewarm failures when a vector index name matches a
different schema column.

### Check List (For Author)

- Test: dependency patch harness and manual checks above.
- Behavior changed: Yes, includes the upstream prewarm fix and new
prefilter APIs.
- Does this need documentation: No.
yiguolei pushed a commit that referenced this pull request Oct 21, 2026
… revision #69332 (#69336)

### What problem does this PR solve?

Backport #69332 to `branch-4.1` (source commit
`eeb54b230ddf7a7a5a06cbaab1bdbd6796d8ff22`).

Pin lance-c to `4fafcd64525b274b113208390bbcca86bbbc093b` and update the
archive checksum. This includes the upstream fix for cold-session vector
prewarm when an index name matches another schema column, plus the
row-address allowlist/blocklist prefilter APIs.

Refresh the existing Foyer patch's base revision and one C++ test
context. Its actual added/deleted code is unchanged. The target branch's
only cherry-pick conflict was the previous revision's descriptive
comment.

### Validation

- Passed `bash thirdparty/test/lance-prefilter-patch-test.sh
thirdparty/src` on this branch: platform definitions, fresh extraction,
idempotence, re-extraction, existing markers, stale-cache invalidation,
and patch-failure handling.
- Verified the pinned archive checksum through the downloader.
- Passed shell syntax checks and `git diff --cached --check`.
- Confirmed the patch payload is unchanged and the resulting patch
matches #69332 exactly.
- Full Doris build and runtime regressions were not run locally;
requested through `run buildall`.

### Release note

Fix Lance index prewarm failures when a vector index name matches a
different schema column.

### Check List (For Author)

- Test: dependency patch harness and manual checks above.
- Behavior changed: Yes, includes the upstream prewarm fix and new
prefilter APIs.
- Does this need documentation: No.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants