[pzhdfy]

What problem does this PR solve?

Follow-up to #66227, which introduced the paimon-rust reader. This PR upgrades the Paimon SDK to the community 2.1.0 release and ports the primary-key vector (ANN) search path onto the community stack: ORDER BY l2_distance_approximate|inner_product_approximate(col, vector) LIMIT n is rewritten into a pushdown onto the paimon scan and served by the paimon-rust vector reader on BE.

Problem Summary:

1. Upgrade Paimon to the community 2.1.0

• Port FE off the paimon privilege package removed in 2.1.
• The 2.1 S3FileIO reads S3 credentials only from CatalogContext.options() (verified at bytecode level: S3FileIO#configure -> loadHadoopConfigFromContext never touches the Hadoop Configuration, unlike 1.4; unchanged in the community 2.1.0). Doris keeps a catalog's storage credentials in the Hadoop conf only, so every s3-warehouse paimon catalog initialized the SDK with no credentials and failed with Unable to load credentials. mirrorHadoopS3OptionsToCatalogOptions now mirrors the fs.s3a.* entries onto the catalog options for the FileSystem/HMS/JDBC metastores, keeping user extras (path style, connection limits) flowing as well.
• The community 2.1.0 restores reading manifests whose _FILE field is the legacy nullable-union type (written by paimon versions older than 1.4), so the pre-provisioned paimon1 CI regression baseline and old production tables stay readable; the internal 2.1 BETA build rejected them with expected RECORD but found UNION.
• The aggregate-merge overflow behavior matches 1.4 again in the community release (silent wrap, e.g. a TINYINT sum of 127 and 1 reads back as -128); an intermediate fail-loud behavior only existed in the internal BETA build — the branch history contains the suite adaptation for it (1bac394) followed by its revert (1871ef6), a net zero for reviewers.
• One upstream 2.1.0 behavior change the tests now pin: the persisted timestamp-to-date conversion for pre-epoch values takes the preceding calendar date (floor, matching Arrow) instead of truncating toward zero — a TIMESTAMP(3) of -1ms evolved to DATE reads back as day -1 (UT expectation updated).

2. PK-vector (ANN) search

• New rewrite rule PushDownVectorTopNIntoPaimonScan (registered in Rewriter) matches a vector distascan and pushes the query vector, the indexed vector column, the metric and the limit down to the scan.A pre-flight check refuses incremental scans (t@incr), so an incremental query can never silently resolve to a snapshot read.
• AnnTopNInfo carries the pushed-down info from LogicalFileScan to PhysicalFileScan and onto PaimonScanNode (PhysicalPlanTranslator sets it only inside the PAIMON_EXTERNAL_TABLE branch); the PlanNodes.thrift additions are additive fields.
• BE: paimon_rust_table_reader grows the vector-search path, reading through the rust PK-vector build-op on this path (the rust vector readers do not consume it) — noted in the reader.
• The pushdown reuses the existing enable_paimon_rust_reader and enable_file_scanner_v2 switches; with either off, the query brute-forces through the normal scan.

3. Unify the rust eligibility gates

• PaimonRustEligibility evaluates the 12 table-level/storage/BE checks once and is shared by the batch and vector paths. The vector gate is now computed independently of the batch gate (it previously inherited fallbackRead / scan.mode restrictions from the batch path) and both branches share a common gate chain with a batch-only split-suitability suffix.
• Compaction-merge splits that are raw-convertible skip the merge-reader gates.
• JNI fallback reasons are logged so a silent downgrade to the JNI reader is diagnosable.

Tests

• FE UT: new PushDownVectorTopNIntoPaimonScanTest (positive / incremental-refused / gate-negative coverage), PaimonScanNodeTest gate coverage, planner PaimonPredicateConverter tests, PaimonUtilTest; 485 paimon-related FE UTs pass against the 2.1.0 stack.
• New end-to-end regression test_paimon_pk_vector_ann: the suite builds its own fixture through the Spark ThriftServer — an ivf-rq PK-vector index with nlist = nprobe = 4 on 512 deterministic 16-dim vectors, i.e. exhaustive
  recall, so ANN must equal brute force by construction. It compares against the brute-force answer comsabled, asserts the ANN leg really executed on the paimon-rust vector reader (profile check against asilent brute-force pass), and covers both l2 and inner_product metrics, residual filters and non-index projections.
• Fixed two regression fixtures broken by the 2.1 Spark stack: the creation-time-millis helper was shifted by the session UTC offset (every scan.creation-time-millis leg silently resolved to the latest snapshot; Doris-side resolution verified correct by hand-reproducing the queries), and assertSparkDorisResultEquals now decodes the 0x-hex tokens Doris renders for VARBINARY nested in arrays/maps/structs so identical nested binary values compare equal across engines.
• Adapted two write suites to observable 2.1 changes (oversized-row rejection message gained a qualifier; unset table-property defaults are no longer materialized by SHOW TBLPROPERTIES) — the underlying behaviors are unchanged.
• Swept the locally runnable external_table_p0/paimon regression suites against the 2.1.0 stack: read-path 34/37 and write-path 26/32 green (the rest need a Spark 4.0 ThriftServer for VARIANT DDL or pre-provisioned fixtures that exist only in CI), plus the HDFS-based suites against the original pre-provisioned paimon1 OSS baseline — no Doris-side regressions.

Release note

The Paimon catalog now runs on the community Paimon 2.1.0 release. Queries of the form ORDER BY l2_distance_approximate|inner_product_approximate(<vector col>, <query vector>) LIMIT n over paimon tables with a PK-vector index are
pushed down into the paimon-rust vector reader (requires enable_paimon_rust_reader and `enable_file_sy paimon versions older than 1.4 (legacy union manifests) remain readable.

Check List (For Author)

• Test
  • Unit test
  • Regression test
  • Manual test (add the script if necessary)
  • No need to test
• Behavior changed: yes (the new switch-gated ANN pushdown; paimon-side: pre-epoch timestamp-to-dat
• Does this need documentation?

Check List (For Reviewer who merge this PR)

• Does the PR have a corresponding release note?
• Is the release note filled in correctly?
• Are the test cases added or updated correctly?
• Is the document updated (if necessary)?
• Does this PR need to be picked to other branches? (branch-4.1 only: yes)

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[924060929]

FE-only review of 2dcb9c0. The inline comments cover candidate filtering, vector cast semantics, relation read options, and projection eligibility. These are based on source-path analysis; no build or tests were run for this review.

[Gabriel39]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Review recovery stopped: No final review submission was declared
Workflow run: https://github.com/apache/doris/actions/runs/38353841960

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[morningman]

run buildall

[morningman]

/review

[github-actions]

Static review of PR #68988 at 537b731f92999ca4821ba8645b946e40a23d2888: changes requested. Ten new verified findings cover nested projection correctness, partitioned ANN planning, shared Thrift IDs, BE test compilation, score ordering, NULL query coordinates, BE capability negotiation, first-row filter eligibility, ordinary physical-column preservation, and lossless binary test comparison.

Two existing P1 threads remain applicable and are carried without duplicate inline comments:

• 4254619627 — filtering before bounded TopK: the new whitelist accepts tag IN (1,2,NULL) because every option is a Literal, but the BE converter rejects the entire IN predicate on its NULL literal. With nearest tag=0 and the next tag=2 in one bucket, LIMIT 1 retrieves tag=0 and the Doris residual removes it, returning no row. Typed NULL survives coercion, constant folding, IN simplification, and translation. The original distance-predicate example now declines rewriting, but the same truncation defect remains.
• 4254619638 — resolved file-creation restriction: t@options('scan.creation-time-millis'='<future>') is not excluded by the new raw scan.file-creation-time-millis check. Its resolver falls back to the same pinned internal file-creation bound when no later snapshot exists. Ordinary planning excludes all old files; vector planning ignores that bound and can return their neighbors. The explicit file-creation option is guarded, but this equivalent resolved read state remains affected.

Existing 4254619634 (value-changing vector casts) is addressed by requiring a direct indexed slot. Existing 4254619644 (projected VARIANT / ORC LTZ) has explicit preflight guards; the new ordinary-STRUCT finding is a separate layout failure. No additional user-provided focus was supplied.

Applicable critical checkpoints:

1. Goal and proof: the nominal SDK 2.1 / L2 and inner-product ANN flow is implemented, but the verified failures prevent ordinary-scan parity. Added unit and differential regression tests were inspected; none were executed.
2. Scope and abstraction: reviewed every one of the 53 authoritative changed paths, including dependency/property changes and test utilities. AnnTopNInfo groups transport state and logical/physical copies retain it; the vector branch incorrectly bypasses some compatibility checks.
3. Concurrency and locks: no new Doris threads or lock order introduced. Paimon range construction follows the sequential nonbatch path; a suspected logging-HashSet race was dismissed from actual callers. No heavy new operation was found inside a new lock.
4. Lifecycle and initialization: traced relation/schema binding, pruning, rewrite, translator, processed table, bucket payload, Rust eager execution, Arrow materialization, residual evaluation, and coordinator TopN. Pinned C API setters copy inputs and read/plan ownership allows local RAII destruction. No new cross-translation-unit static initialization dependency or ownership cycle was established.
5. Configuration: existing Rust-reader and FileScannerV2 switches are read per query. No new effective session/BE configuration is introduced; the SessionVariable change is whitespace.
6. Compatibility: compared touched Thrift definitions with live branch-3.1, branch-4.0, branch-4.1, branch-4.2, and master. Two new field IDs collide with master. The generic existing BE capability does not distinguish the new vector protocol. Pinned Rust API availability and Java VECTOR access/type preservation were checked from source.
7. Parallel paths and conditions: compared native, JNI, ordinary Rust, vector, system/delegate, raw-convertible, partition, nested, incremental and read-state paths. Incremental and direct lossy-cast cases decline. Remaining predicate/read-state restrictions are documented above and in the new inlines.
8. Functional tests: new ANN regression uses a small unpartitioned DV-enabled fixture. It does not cover partitioned vector ranges, pruned STRUCT materialization, NULL query coordinates, non-DV first-row filters, mixed-version BEs, or cross-bucket score overflow.
9. Negative/unit coverage: inspected score, nullable-column, payload, rewrite and routing tests. The newly incorrect include prevents the BE test target from compiling. No build or test was attempted under this static-only contract.
10. Expected results: no golden .out files changed. The pre-epoch timestamp-to-date expectation follows the SDK's floor behavior. The new UTF-8 comparator can conceal distinct binary results and needs a lossless comparison.
11. Observability: EXPLAIN exposes vector column, metric, vector and limit; FE logs distinct fallback reasons. Existing reader/open/read/Arrow timers cover the new lifetime; search/refine/materialization costs accumulate during reader opening.
12. Transactions and persistence: production changes are query-side; no new Doris EditLog, commit, replay or master-failover state is introduced. External snapshot/file visibility and deletion-vector constraints were checked; the surviving read-state thread and first-row filter finding identify concrete gaps.
13. Data writes and crashes: no production data-write atomicity change. JNI/native fallback and C error/cleanup paths were inspected statically. This review does not claim fault-injection or crash validation of the upgraded SDK.
14. FE/BE propagation: vector/column/limit/metric travel through logical and physical copies, translator and little-endian FLOAT32 request; bucket payloads travel per split. Partition metadata, nested shape and protocol negotiation have the reported gaps.
15. Performance and memory: projection/filter setup is outside per-row decode. Compacted raw split dispatch and C handle lifetime were inspected. Vector output is bounded by requested TopK, not adaptive scanner batch size; large TopK/wide payload memory, cancellation and full Hive-shade runtime interoperability remain unmeasured.
16. Core invariants and other issues: query correctness/error preservation fail in the identified cases. No new internal rowset-version, MoW delete-bitmap or uncommitted-transaction path is changed. Arrow distance conversion checks concrete nullable columns and restores COW ownership; no additional memory-lifetime defect was established.

Coverage limitation: this is static source review, including pinned external source and live protocol references. No compilation, tests, runtime/external-writer interoperability, or performance measurements were performed. All changed paths have been read; empirical compatibility and memory behavior are not claimed.

Review converged after 3 round(s).

Existing P0/P1 findings confirmed for this head: #68988 (comment), #68988 (comment)

[hello-stephen]

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 25.42% (151/594) 🎉
Increment coverage report
Complete coverage report

[Gabriel39]

/review

[github-actions]

Static review of all 59 authoritative changed paths at 1f40520. Request changes for six new findings: branch index/metric identity, historical scalar value loss, eager materialization memory, generic external column mapping, stored-NULL validation, and nonmovable projection errors.

Existing threads were revalidated and not reposted. Existing P1 4311684734 still applies when an outer field projection leaves an already-pruned bare slot inside the ANN Project: preflight accepts it and the final scan guard throws. Existing P1 4254619638 still applies to its explicitly noted pinned-empty generation bypass: a first commit after binding can be read by vector planning, while ordinary planning returns the fenced empty state. Existing P1 4254619627 still applies with a supported raised expr_children_limit and an IN list exceeding 65535 children: FE accepts it, the Rust predicate converter drops it, and residual filtering occurs after bounded retrieval. The original direct-field, file-time-bound, and ordinary unsupported-filter examples have guards now; these remaining instances preserve the existing blockers. Other earlier P1s (lossy casts, partition metadata, field-ID collisions, missing include and zero-score ordering) are repaired in the inspected head. Known P2 name-collision and type-unaware result-normalization residuals remain covered by their existing threads.

Critical checkpoint conclusions:

1. Goal and proof: the PK-vector ANN/SDK2.1 flow is implemented, but the findings prevent correctness approval. Positive and negative UT/regression sources were inspected; the new boundary cases lack coverage.
2. Scope: planner, request, split, reader and SDK changes follow the feature flow; shared eligibility removes duplicated gates. The generic mapping exception and skipped schema guarantees need correction.
3. Concurrency: query state is per scan; no new lock/order or metadata mutation path found. Heartbeat capability uses existing volatile/replay handling. The empty-snapshot race is identified above.
4. Lifecycle/static initialization: Rust builders/plans clone inputs; local RAII teardown and Arrow release ownership are sound. AnnTopNInfo clones its array. No new cross-TU static dependency or ownership cycle identified; eager buffers have the reported resource issue.
5. Configuration: no new user setting; existing Rust/V2 switches are read during rewrite and scan planning.
6. Compatibility: separate vector capability prevents dispatch to older batch-only BEs. SDK2.1/JNI VECTOR and S3 credential-option changes were traced; historical scalar ANN casts remain unsafe.
7. Shared definitions: touched Thrift definitions were compared with master and branches 4.2/4.1/4.0/3.1. New field IDs 12/21/42/6 are free; no remaining collision identified. New payload is behind an optional envelope. No proto changes.
8. Parallel paths: native/JNI/cpp/batch-Rust routes and raw compaction gates were checked. One-way ANN must preserve their schema, error, identity and memory guarantees.
9. Special conditions: cast, query-NULL, projected VARIANT, creation-time and vector-capability guards are present. Remaining nested/empty/filter edges are recorded against existing threads.
10. Tests: all changed tests/fixtures reviewed statically. Branch metric divergence, stored NULLs, nonmovable assertions, historical scalar ANN evolution and large-K memory require regression coverage.
11. Expected results: pre-epoch DATE floor, unset property and error-message adaptations match the inspected SDK behavior. Existing ResultUtils type-awareness concern remains a duplicate fence; no new golden output file was added.
12. Observability: ANN explain details, Rust lifecycle timers and deduplicated fallback reasons are present. No separate new critical logging defect established.
13. Persistence/failover: no new external data writes or EditLog record type. Heartbeat persistence follows the existing pattern; read snapshot consistency remains blocked as above.
14. Writes/crashes: no new transaction/write atomicity path. The eager memory issue can exhaust a BE during a read.
15. FE-BE flow: vector/column/metric/K and bucket payload survive logical/physical copies and translation; partition values are populated. The branch finding concerns semantic identity despite complete transport.
16. Performance/memory: FFI ownership and ordinary bounded batches were checked; eager all-K retention/copy lacks the ordinary bounds/accounting.
17. Other invariants: NULL and assertion errors must remain failures. No additional deadlock, OLAP delete-bitmap or visible-version modification identified.

No additional user focus was supplied. Validation was static only: no builds, tests, source modifications or submodule initialization. Runtime/load and live mixed-version behavior were not exercised.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #68988 (comment), #68988 (comment), #68988 (comment)

[github-actions]

[P1] Validate the index against the relation's bound table

getPaimonTable(Optional.empty()) fetches the cached main table, while PaimonSource and vector split planning use this relation's retained branch/snapshot. For example:

TopN(dis ASC LIMIT 1)
  Project(id, dis := l2_distance_approximate(vec, [0.25,0,...]))
    Scan(t@branch(b))

Main can have an L2 index while branch b has an independently configured inner-product index. With branch vectors [1,0,...] and [2,0,...] in one bucket, this preflight accepts main's L2 metric, but Rust reads the branch schema and selects the second vector with score 0.5. BE then applies the requested L2 inverse and publishes distance 1; ordinary L2 selects the first vector at 0.75. Full recall still gives this wrong result. Resolve all index/metric/eligibility checks from the scan's bound table before replacing the expression, and cover differing branch index options.

[github-actions]

[P1] Retain historical scalar compatibility checks for vector reads

canRead also protects historical value conversions, which remain relevant when vector search materializes projected columns. On a PK-vector Parquet table, write ts TIMESTAMP(3) = '2300-01-01 00:00:00.123', then widen ts to TIMESTAMP(9) while retaining the old files/index. SELECT ts, l2_distance_approximate(vec,q) AS d FROM t ORDER BY d LIMIT 1 now bypasses that check. Rust's materialization uses DataFileReader -> reconcile_column -> Arrow's safe millisecond-to-nanosecond cast; the year-2300 multiplication overflows and returns NULL. The ordinary path explicitly rejects this evolution in PaimonRustReaderCapabilities and uses JNI, which preserves the timestamp (also documented by testPersistedTimestampEvolutionIntoNanosecondRange). Check current/historical schema compatibility before committing ANN, or provide equivalent materialization. This is a scalar full-projection case, separate from the existing pruned-STRUCT thread.

[github-actions]

[P2] Bound the eager materialization before entering Rust

This call finishes materialization before returning the Arrow reader. In the pinned Rust SDK, materialize_positions retains every projected Top-K batch, then reorder_and_strip_position copies those rows into one output batch. Unlike the ordinary path, this path ignores the scanner's adaptive batch size and has no reservation/accounting bridge for those Rust buffers. A one-bucket query with LIMIT 1000000 and a projected 4-KiB payload can therefore retain roughly 4 GiB plus another 4 GiB reorder copy during prepare_split, even with a tiny probe batch and query memory budget. Raw malloc interposition forwards to jemalloc; it does not perform Doris Allocator tracking. Bound and account for this eager work before the call, or use bounded materialization; splitting the returned batch afterward cannot prevent the allocation spike.

[github-actions]

[P2] Limit this mapping exception to an actual synthetic slot

This method serves every external catalog, so the name alone also excludes a real Hive column. For a Hive text/CSV table (id INT, __paimon_search_score_to_dis STRING), selecting both columns leaves two required file slots but emits only column_idxs=[0]. With FileScannerV2, DelimitedTextReader::_init_reader then rejects the index-count mismatch; Hive positional ORC/Parquet mapping likewise lacks the second ordinal. Preserve the mapping for physical same-named columns and identify the actual Paimon vector-produced slot. This is a separate generic FE/native-scan instance from the already-discussed ordinary Rust _build_read_columns exclusion.

[github-actions]

[P2] Preserve validation of NULL vectors stored in the table

The RHS check above does not cover a live NULL in the indexed column:

TopN(dis ASC LIMIT 1)
  Project(id, dis := l2_distance_approximate(vec, valid_q))
    Scan(rows: vec=NULL; vec=[1,0,...])

Paimon permits nullable VECTOR columns and NULL stored values. The ordinary FunctionArrayDistance::_validate_no_nulls throws First argument ... cannot be null before calculating distances. The Rust vector reader instead skips outer-NULL vectors, so replacing the function makes this query successfully return the finite row, even with full recall. Both supported metrics have this difference. Preserve the stored-vector validation or decline the rewrite when its error contract cannot be guaranteed, and add an enabled/disabled NULL-data regression. This is distinct from the earlier NULL coordinate in the query literal: here the query vector is valid.

[github-actions]

[P2] Keep nonmovable project expressions before truncation

The rewrite accepts a Project containing NoneMovableFunction, such as:

TopN(dis ASC LIMIT 1)
  Project(c := assert_true(id > 0, 'bad id'),
          dis := l2_distance_approximate(vec,q))
    Scan(rows: id=-1/far; id=1/near)

On the ordinary path the Project evaluates before TopN and raises for the far row. withVectorTopN truncates the scan to the near row first, so the assertion never sees id=-1 and the query succeeds. The scalar assertion passes complexColumnsRideAsWholeColumns, and there is no nonmovable-expression guard. Decline this rewrite for project expressions whose evaluation must remain below TopN, and cover the assertion case. This concerns a SELECT expression, separate from the existing WHERE residual-filter thread.