Skip to content

[chore](build) Update lance-c to include runtime BTree selection - #69643

Open
Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:dev/update-lance-c-runtime-selection-master
Open

Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:dev/update-lance-c-runtime-selection-master

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: lance-format/lance-c#98

Problem Summary: Update lance-c from 4fafcd64525b274b113208390bbcca86bbbc093b to the merged upstream main commit a51ab47a9c814a746cb1714c99fad93dd3a188b8, and refresh the source archive name and MD5. This brings in the opt-in runtime BTree segment selector and its pre-decode memory guards. Existing master callers retain their selection behavior because the new native option defaults to false. The existing Foyer patch applies without changes and with zero fuzz.

Release note

None. This updates the dependency; it does not enable the new opt-in native feature in Doris.

Validation

  • Verified the downloaded immutable source archive's MD5 and applied the existing Foyer patch with zero fuzz. The same patch is used on master and branch-4.2.
  • cargo test --offline --locked on that patched source: 515 tests passed, including 82 unit tests and 385 C API tests. The normal command ignores three C/C++ executable compilation tests.
  • C and C++ API test sources passed syntax-only compilation against the new patched headers.
  • Exercised the existing source-patching block for first application, a cached no-op, and stale-checksum re-extraction; verified resulting source equality.
  • Shell syntax and staged whitespace checks passed. Full Doris BE build/integration validation remains for CI.

Check List (For Author)

  • Test
    • Unit Test
    • Manual test: source checksum/patch lifecycle and C/C++ header compatibility checks above.
    • Regression test
  • Behavior changed:
    • No. Existing Doris callers do not enable the new native option.
  • Does this need documentation?
    • No. No Doris configuration or user interface changes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Related PR: lance-format/lance-c#98

Problem Summary: Update lance-c from `4fafcd64525b274b113208390bbcca86bbbc093b` to the merged upstream main commit `a51ab47a9c814a746cb1714c99fad93dd3a188b8`, and refresh the source archive name and MD5. This brings in the opt-in runtime BTree segment selector and its pre-decode memory guards. Existing master callers retain their selection behavior because the new native option defaults to false. The existing Foyer patch applies without changes and with zero fuzz.

### Release note

None. This updates the dependency; it does not enable the new opt-in native feature in Doris.

### Validation

- Verified the downloaded immutable source archive's MD5 and applied the existing Foyer patch with zero fuzz. The same patch is used on master and branch-4.2.
- `cargo test --offline --locked` on that patched source: 515 tests passed, including 82 unit tests and 385 C API tests. The normal command ignores three C/C++ executable compilation tests.
- C and C++ API test sources passed syntax-only compilation against the new patched headers.
- Exercised the existing source-patching block for first application, a cached no-op, and stale-checksum re-extraction; verified resulting source equality.
- Shell syntax and staged whitespace checks passed. Full Doris BE build/integration validation remains for CI.

### Check List (For Author)

- Test
    - [x] Unit Test
    - [x] Manual test: source checksum/patch lifecycle and C/C++ header compatibility checks above.
    - [ ] Regression test
- Behavior changed:
    - [x] No. Existing Doris callers do not enable the new native option.
- Does this need documentation?
    - [x] No. No Doris configuration or user interface changes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 485ea23689ac167a6dfe01fe5684d49ef314ffe7. Static review found no verified actionable defect in this dependency update. No new inline findings or existing blocking comments remain. The review converged in one round, including complete review and separate patch/runtime risk reviews.

The URL, archive name, extracted source directory and independently downloaded MD5 (d1282cff3d63acfc0450db1f00d6ab6f) agree. The new pin is the merged upstream runtime-BTree-selection commit. The retained Foyer patch passes a read-only zero-fuzz dry-run across all 15 target files. Old/new upstream Cargo manifests and lockfiles are byte-identical; the retained patch preserves the same dependency graph. The new selector option initializes to false and is only enabled through its new setter. Its cost-selection call is guarded by that option.

Critical checkpoints from the repository code-review skill:

Checkpoint Conclusion
Goal and evidence The immutable dependency pin brings in the opt-in selector and pre-decode guards without enabling it in Doris. Archive identity, upstream source and included test sources support this; runtime behavior was not executed.
Scope and clarity Focused change to the explanatory comment and four matching definitions in thirdparty/vars.sh.
Concurrency and locks No Doris thread or lock changes. Selector probes run sequentially in scan preparation and use the existing session metadata cache; no new static lock-order issue identified.
Lifecycle and static initialization Scanner state starts disabled and rejects setter changes after scan start. Existing opaque-handle/Arc ownership remains; no new cross-TU initialization dependency. Hash-qualified source names isolate old cached source markers.
Configuration No Doris configuration item added. The native per-scanner API is opt-in before scan start.
Compatibility Existing C/C++ signatures are retained, new APIs are additive, and handles remain opaque. Underlying Lance revision, dependency graph and storage format are unchanged. Full link/ABI execution remains unverified.
Thrift and protobuf IDs No definition changes; cross-branch field-ID comparison is inapplicable.
Parallel paths Disabled mode retains the caller-selected segment. Bitmap/LabelList and vector/FTS paths retain their existing behavior; Linux/macOS share the pin and existing build/install wiring.
Conditional checks Existing snapshot, explicit fragment-domain, filter and exact-candidate checks remain. Unknown cost probes preserve the selected path; expensive full-scan fallback retains the original scoped reader.
Test coverage Read upstream selector tests covering disabled/enabled selection, broad-filter fallback, empty results, fragment scope and LIMIT/OFFSET; memory-budget tests cover variable-width rejection, fixed-width/overflow and cache-hit charging. Existing Doris patch-lifecycle harness covers fresh/cached/stale/failure cases. None were run.
Test-result changes No Doris expected-output files changed; no claim that author-reported or inspected tests passed in this review.
Observability Upstream adds selection/candidate/page/row/fallback counters to the existing callback; existing success and fallback reporting remains available. No new Doris logging/metrics requirement identified for the disabled feature.
Transactions and persistence No transaction, EditLog, failover or persistent-format logic changed.
Data writes and atomicity No data-write logic changed. Existing archive publication and selected-source header/library installation paths remain.
FE/BE variables No new transmitted variables or protocol paths.
Performance and memory Cost probes bound candidate probes, serialized bytes, decoded fixed-width buffers, pages, IN values and comparisons; decoded limits are checked before materialization and charged on cache hits. Variable-width estimation is skipped. Metadata enumeration/sorting is behind explicit opt-in. No concrete regression identified for existing callers.
Core invariants and error handling No changed committed-data visibility, version, delete-bitmap or BE allocation path. Existing invalid scanner/snapshot requests propagate errors; advisory estimator failure falls back without removing the full filter or read domain.
Other risks Independently checked retained Foyer/selector interaction, immutable source extraction, patch markers, toolchain and locked build compatibility; no additional substantiated issue.

User focus: none was supplied; the whole changed-file list was reviewed. Existing inline threads: none, confirmed again through the PR API.

Validation limits: this was a static-only review. No builds, tests, source modifications or submodule initialization were performed. This checkout contains link/build wiring but no direct native Lance API consumers, so a full Doris integration run and patched Rust/C/C++ compilation remain CI validation, not established results. There are no unresolved static candidates within the reviewed scope.

Review converged after 1 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants