Repository navigation
Conversation
…the FE off the removed privilege package Paimon 2.1 removes the privilege package the Doris FE paimon path depended on; this swaps the dependency to the 2.1-community-release line and ports the FE references to the replacement API surface. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…stack Order-by-distance pushdown over the PK-vector index (l2 / inner_product metrics), executed exclusively by the BE paimon-rust reader: - FE: PushDownVectorTopNIntoPaimonScan rewrite (bounded TopN -> index-only scan with the __paimon_search_score_to_dis synthetic output column) and the AnnTopNInfo / vector payload plumbing down to the split. - Rust eligibility gates: the vector capability bit is negotiated separately from the batch rust bit (a vector payload rides an empty paimon_split, which an older rust-capable BE would reject), the table/storage/backend gates are unified between the batch and vector paths, raw-convertible compacted splits skip the merge-reader gates, and the rewrite pre-flights the gate because a vector split has no JNI fallback. - BE: the paimon-rust vector reader mode (bucket vector search split deserialization, score-to-distance transform, synthetic column fill). - Housekeeping caught by the port: clang-format of the ANN reader files, FileSystemSchemaManager in the rust reader schema tests, the privilege references the 2.1 port missed, and a docs note that read.batch-size is currently a no-op on the PK-vector path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Conflicts: fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java
…s and adapt the test suites Finalization of the 2.1 stack switch: - Pin the Paimon dependency to the released community 2.1.0 (identical content to the snapshot it replaces). - Mirror the s3.* credentials into the catalog options: the 2.1 SDK's S3FileIO reads credentials from CatalogContext.options() where the previous version picked up the endpoint-scoped properties. - Regression suites: a new end-to-end suite for the PK-vector ANN path (rust vs brute-force differential legs), adaptation of the write suites and fixtures to the 2.1 SDK / Spark behaviors, and the before-epoch timestamp-to-date expectation for 2.1.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…review) Addresses the P1/P2 review findings on PushDownVectorTopNIntoPaimonScan: P1 - Keep residual filtering before the bounded vector top-N. The BE paimon-rust predicate converter skips conjuncts it cannot represent (and the synthetic distance column is outside its field registry), and the engine re-applies those residuals on the returned block -- after the top-K rows were already selected. With candidate distances 0, 2 and 3, a `WHERE dis > 1 ORDER BY dis LIMIT 1` fetched only the distance-0 row and returned nothing. The rewrite now fires with a Filter only when every conjunct provably rides into the vector search: a comparison / IN / null-check on a table column whose Doris type is in the converter's supported set (mirrors PaimonRustPredicateConverter._is_supported_slot_type). Anything else declines the rewrite and the query computes the distance in Doris, which is always correct. Verified end to end: the differential leg caught the empty result on the unfixed build and passes on the fixed one. P1 - Do not strip value-changing casts from the indexed vector. A legal cast chain can change vector values (an ARRAY<FLOAT> -> ARRAY<INT> -> ARRAY<FLOAT> round trip loses precision), so searching the original vectors while the query means the casted ones changes the ranking. The rewrite now accepts a direct indexed column slot only. P1 - Preserve the relation's file-creation-time restriction on the vector path. The ordinary path drops manifest entries older than the bound, but the PK-vector search plans its own splits and never consults the bound, so old files' nearest neighbors could leak into the result. A scan carrying scan.file-creation-time-millis now declines the rewrite (snapshot / tag / version selectors are resolved into the table before the vector search and stay eligible). P2 - Check the projection capabilities before committing the rewrite. A PK-vector split has no JNI fallback, so a query whose projected columns would set the projectedVariant bound (nested Variant) or an ORC table with TIMESTAMP_LTZ in its schema -- both scan-node bounds with no vector-path fallback -- would turn a supported query into an error. Both are now pre-flighted and decline instead, keeping the query on an ordinary supported path. Tests: PushDownVectorTopNIntoPaimonScanTest grows from 20 to 27 cases (distance-predicate / unsupported-type / expression filters decline, a simple column filter still rides, file-creation-time declines while a snapshot selector does not, lossy cast declines, Variant projection declines; the two distance-predicate positives flip to the new sound semantics). The ANN regression suite adds the requested differential legs: a distance lower bound, an unconvertible ARRAY<FLOAT> predicate and a file-creation-time bound, each compared against the rust-disabled brute force. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the 3.1.4-SNAPSHOT dependency with its release build. The 4.1.0 artifact is byte-identical to the snapshot it replaces (same class list and file sizes, same shaded paimon 2.1.0 / hive 3.1.3, identical pom dependency declarations), so this only pins the FE build to a stable release version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…NN path Static review of this PR surfaced twelve soundness and compatibility gaps on the PK-vector (ANN) search path; every one was verified against the pinned sources before the fix: - Wire protocol: TPaimonFileDesc.vector_payload moves from field 20 to 21 and TFileScanRangeParams.external_search_request from 41 to 42 -- both IDs are already reserved on master (original_file_path, fluss_properties), and reusing them would make a mixed-version cluster drop the payload instead of decoding it. BE/FE must upgrade together. - Vector capability negotiation: a new heartbeat bit supports_paimon_rust_vector_reader (TBackendInfo field 12, free on all branches) gates the one-way rewrite separately from the batch rust bit, so a rolling upgrade keeps ordinary rust reads working and only declines the vector rewrite against older BEs. - Partitioned vector splits now carry the derived partition values (PaimonSplit.setPaimonPartitionValues), which setPaimonParams demands for any table with partition keys. - The rewrite pre-flight now also declines: a query vector containing a NULL element (getDouble() would silently search a zero coordinate while the un-rewritten function raises); scan.creation-time-millis (resolves into the same internal file-creation bound the vector search never consults); a data predicate on a table without deletion vectors or with merge-on-read (paimon-rust rejects it AFTER the one-way rewrite); a table column named __paimon_search_score_to_dis (collides with the synthetic distance slot); and an expression over a complex column (nested pruning reduces the scan tuple while the vector reader projects only root names -- the struct SerDe then binds children by ordinal). The shape whitelist additionally rejects NullLiteral operands: the BE converter rejects a null literal and with it the whole predicate, leaving it in the post-fetch residual. The scan node keeps the nested-projection bound as a loud backstop for vector splits and logs the vector-capability reason. - BE: _convert_score_to_distance maps zero and out-of-range scores to +inf instead of negative sentinels -- a zero score is reachable from a genuine huge distance (the search-mode=full exact fallback computes squared L2 in f32), and -2 sorted ahead of every finite distance, inverting the coordinator's TopN. _build_read_columns skips the synthetic distance name only in vector mode, so an ordinary scan of a real column with that name keeps reading it. The BE test include path is fixed (the file never compiled). - Regression framework: the 0x-token decode pass only substitutes UTF-8-round-trippable bytes and keeps the lossless hex token otherwise -- replacement-character decoding made 0x80 and 0x81 compare equal. Verified: FE UT 36/36 (rewrite) + 96/0 (scan node) + 22/22 (rust compat) + 4/4 (capability); BE UT 30/30; the full ANN regression suite including the four new differential legs (no-deletion-vector first-row table, partitioned table, NULL query vector, IN with a NULL option). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six new findings, all verified and fixed: - Branch identity: the pre-flight resolved the cached main table while the scan reads the relation's bound branch table, so a branch with a different index metric scored and reordered with the wrong metric. All four pre-flight sites now resolve the bound table (relation snapshot, or the options-applied table for t@options); time-travel relations benefit too. - Historical scalar evolution: the vector path skipped the batch reader's per-file schema gate, so a TIMESTAMP(3)->(9) widening would NULL out-of-range values through Arrow's safe cast. The rewrite now declines any rust-incompatible schema history, and the scan node re-checks per split (loud backstop mirroring the pre-flight exactly). - Generic mapping exception: FileQueryScanNode skipped the synthetic distance name for every external catalog, breaking real same-named Hive columns; the skip now applies only when the schema has no such column. - Nonmovable projection: a NoneMovableFunction (assert_true) in the ANN project must keep evaluating below the TopN; any nonmovable expression outside the order-key alias now declines the rewrite. - Pruned bare slot: a bare complex slot whose type was already reduced by nested pruning now declines (slot type must still equal the original column's), instead of tripping the desc-side guard loudly. - IN beyond the converter cap: the whitelist mirrors the BE converter's 16-bit child count, so a raised expr_children_limit no longer drops the predicate into the post-fetch residual. Carried P1 threads closed at their narrowed edges: the pinned-empty generation is fenced in getVectorSearchSplits exactly like the ordinary path (a first commit after binding is no longer readable by vector planning), and the pruned-bare-slot / IN-cap cases above. The stored-NULL differential is kept as a documented limitation: a nullability-based decline cannot ship until PaimonExternalTable.loadSchema stops hardcoding isAllowNull=true (today every paimon column, even PK columns, maps as nullable, so the gate would disable the rewrite for every table). Follow-ups recorded in the review thread. Tests: rewrite rule 36 -> 42 cases, PaimonScanNode +3 (fence twin, loud per-file backstop, schemaId-matching encode); the ANN e2e suite stays green with a new assert_true error-parity leg. BE untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Paimon 2.1 turned SchemaManager into an interface; the connector test moves to the FileSystemSchemaManager default implementation. - The LogicalFileScan conflict resolution missed a blank line between constructors (checkstyle). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
/review |
Contributor
|
run buildall |
Gabriel39
reviewed
Oct 23, 2026
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static review of #69493 at 8fd3bec. Six verified P2 findings are inline: stored-NULL error suppression, missing TIMESTAMP_NS preflight, reserved materializer names, an overbroad nested-distance exemption, retained ORC/LTZ files, and loss of literal-string identity in the shared regression oracle.
Critical checkpoints:
- Goal and behavior: the SDK upgrade and PK-vector ANN path are connected from Nereids through split serialization to Rust. The six findings prevent full correctness/compatibility; new tests do not cover these inputs.
- Scope and clarity: changes span all 61 authoritative paths; the shared eligibility refactor is relevant. The regression helper affects comparisons beyond ANN and needs the inline correction.
- Concurrency: no new threads or lock order; Backend capability fields are volatile and heartbeat/replay updates were traced. Scan-local and split-local ownership examined; no additional concrete race found.
- Lifecycle/static initialization: C ABI setters copy inputs, filters transfer ownership, returned streams own required state, and RAII handles/Arrow callbacks cover examined error paths. No new cross-translation-unit static dependency identified.
- Configuration: existing Rust-reader and FileScannerV2 session switches gate the rewrite; no new dynamic configuration item.
- Compatibility: Java Paimon 1.4.2 -> 2.1.0 and Hive shade 3.1.3 -> 4.1.0 reviewed, including VECTOR/JNI physical layout, removed privilege wrapper, S3 options and retained schema/file compatibility. Inline findings cover remaining proven input-shape failures.
- Thrift: touched definitions compared with master and branch-4.2/4.1/4.0/3.1. Added IDs 12/21/6/42 avoid collisions, including master reservations 20/41. Required members are inside a new optional, capability-gated payload type; no old-sender required-field failure demonstrated.
- Parallel paths: batch Rust, vector Rust, JNI/native fallback, logical/physical/lazy scan copies, filesystem/HMS/JDBC and relevant REST/DLF paths reviewed. New vector plans have no fallback, which makes the reported preflight gaps observable failures.
- Conditions: predicate whitelists, NULL query elements, IN bounds, incremental/file-time exclusions, branch/snapshot identity and pinned-empty state traced. Canonical field casing is preserved; apparent lowercasing issues were dismissed from actual descriptor code.
- Tests: inspected FE/BE unit and differential regression additions; no builds or tests executed under the static-only contract. Missing cases are stated inline. Profile presence alone is not proof of vector execution.
- Expected results: inspected changed temporal/date, table-property and oversized-row expectations; no additional proven expectation defect. The new shared binary normalization can hide mismatches (inline).
- Observability: ANN explain fields, fallback reason logging and Rust timers are present. No additional critical logging/metrics defect established; large-K behavior was not benchmarked.
- Transactions/persistence: no changed write/commit/EditLog protocol. New heartbeat capability serialization and replay/default-false downgrade behavior checked.
- Data writes/crashes: production changes are read/planning paths; no new table mutation or transaction atomicity change. Reader cleanup/error propagation inspected statically.
- FE/BE state: AnnTopNInfo survives logical/physical copies and translation; query-wide request and per-bucket bytes reach the Rust reader. Old BEs are excluded by independently negotiated vector support.
- Performance/memory: source reads stream into bounded search heaps; selected TopK payloads are eagerly materialized and ignore ordinary batch-size controls. Static inspection does not establish a separate lifetime/tracker defect; runtime memory/cancellation and dependency-linkage validation remain outside this review.
- Other concerns: the whole PR and upstream/downstream special cases were checked beyond the first finding; candidate dispositions and risk coverage are recorded in the shared ledger.
User focus: no additional focus was supplied. Existing thread 4357063534 about reusing the heartbeat flag was reviewed as a duplicate fence and not reposted; no existing P0/P1 blocker was independently confirmed. Coverage limits: static source review only, with no build, test, external-service execution or performance measurements. No repository source changes or submodule initialization were performed.
Review converged after 2 round(s).
Per review: drop the separate supports_paimon_rust_vector_reader field and gate the PK-vector rewrite on supports_paimon_rust_reader alone. The rust reader bit has not shipped in any release yet, and the PK-vector protocol ships in the same release, so no deployed BE can ever report the rust bit without the vector protocol — the separate negotiation (and its mixed-version window) cannot occur in practice. Removes the thrift field, the BE heartbeat report, the FE backend-state plumbing, and the vectorEligible layer; the vector gates now reuse the plain eligible() computation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Per the static review round on 8fd3bec: - Nanosecond pre-flight: a TIMESTAMP_NS column in the CURRENT schema (no evolution history) passed the pre-flight but tripped the scan node's nanosecond-reader gate, which a no-fallback vector split turns into a query error. The pre-flight now mirrors PaimonUtil.containsTimestampNs over the slots the Project and Filter reference. - Reserved materializer names: a user column named __paimon_search_score or _PKEY_VECTOR_POSITION makes the rust table open fail after the one-way rewrite; both names join the synthetic-distance collision check. - Distance exemption narrowed: only the call the rewrite actually replaces (the one equal to the order-key expression) may exempt its slots; a distinct distance call keeps its complex-slot whole-column guard instead of shipping a projection rust cannot materialize. - ORC era detection: the orc-ltz pre-flight checked only the CURRENT file format, so a table written ORC with TIMESTAMP_LTZ and later switched to Parquet passed while setPaimonParams still detected the retained ORC files. Any schema era that declared ORC and carried LTZ columns (its own or the current schema's) now declines. Tests: four new rewrite-rule cases (45 total, all green); the ANN e2e suite stays green against the tighter pre-flight. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ries
The 0x-token substitution ran as a substring replacement on every
normalized string, so a literal-text fragment shaped like a token could
collapse distinct values across engines (Spark "prefixA" vs Doris
"prefix0x41" comparing equal after the decode), and quoted strings and
map keys were equally exposed.
The token pattern now requires a value/element boundary (negative
lookbehind on alphanumerics, quotes and backslash), so only shapes Doris
can only have produced as a binary element ("[0x616C, ", "{0x41: ",
"ROW(0x61") decode; the quantifier also stops matching a bare "0x" with
zero byte pairs. A literal string whose whole value coincides with a
bare token remains the one documented ambiguity — distinguishing it
needs column types the comparison does not have.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
|
/review |
Contributor
|
run buildall |
Contributor
There was a problem hiding this comment.
Reviewed #69493 at 199ad37. Request changes for two new P1 findings (regression-framework syntax and unaccounted eager vector materialization) and one P2 S3 credential-precedence regression.
Critical checkpoint conclusions:
- Goal and proof: the SDK upgrade and PK-vector ANN path are implemented across planner, split transport and BE; focused FE/BE tests and differential regression cases are present, but this static review does not prove runtime success.
- Scope: the production changes form a coherent feature/SDK migration; shared regression normalization expands the affected surface and contains the syntax finding.
- Concurrency and locks: query planning and scanner-local state introduce no new shared lock order; Rust executes through its Tokio runtime. Its unaccounted eager allocations are covered by the memory finding.
- Lifecycle: table reuse keys include schema/options/branch, split cleanup follows ownership order, and the pinned eager API returns an independently owned Arrow stream before local handles are freed. No additional lifetime defect established.
- Configuration: existing Rust-reader and FileScannerV2 session switches gate the feature; no new dynamic server configuration is introduced.
- Compatibility: VECTOR is consistently mapped to Doris ARRAY with the corresponding JNI physical accessor. Unsupported reader shapes mostly decline before the one-way rewrite; existing NULL-vector and retained-ORC concerns remain below.
- Thrift: touched definitions were compared on HEAD, branch-4.2, branch-4.1, branch-4.0, branch-3.1 and master. New parent fields do not reuse IDs; master's reserved fields20/41 are avoided. Required children belong to a wholly new optional payload, so old readers skip that payload.
- Parallel paths: normal batch/raw/JNI paths, Hudi/lazy scan copies and both metrics were checked. The shared filesystem/HMS/JDBC S3-option mirror overwrites explicit provider options, as reported inline; generated Hadoop defaults must preserve explicit catalog choices.
- Conditional checks: filter shapes/types/NULL literals/IN size, snapshot/branch/options/empty generation, nested projection, nanoseconds and reserved names were traced through callers and consumers. Downstream TopN analysis rejects limit overflow; runtime filters do not cross the retained TopN.
- Test coverage: positive and negative planner cases, wire/schema guards, score conversion, partition metadata, filter fallback, assertions and both metrics are represented. No eager-memory admission test covers the new resource defect.
- Changed expected results: timestamp/date, property-default and error-prefix changes were inspected; no golden result files were edited. Framework syntax must be repaired before these suites can run.
- Observability: existing reader open/read/Arrow timers cover execution, and fallback-reason logs explain dispatch without logging credential values. No additional observability blocker established.
- Transactions and persistence: production EditLog, commit and failover logic are unchanged; relation snapshot selection and pinned-empty fencing were traced.
- Writes and atomicity: production write/transaction paths are unchanged; the adjusted write-failure test still checks rows and snapshot state after failure.
- FE-BE variables: AnnTopNInfo survives logical/physical copies and translation; query-wide search parameters and per-split payload agree with BE consumption.
- Performance and memory: table-handle reuse and merge-free compacted batch reads are retained. Eager all-K payload collection/reordering bypasses adaptive batch sizing and tracked admission, as detailed inline.
- Other checks: source casing prevents the suspected ordinary score-name misrouting, and small f32 score/tie differences were not raised as a separate defect under the approximate ANN contract.
Existing review context was revalidated and not reposted. The stored-NULL-vector validation issue in discussion4357228424 still applies. Discussion4357228441 also remains applicable to per-level file formats: a Parquet default plus file.format.per.level='0:orc' can retain ORC/LTZ files that the preflight misses. Its original schema-history trigger is now guarded. Nanosecond projection (4357228428), reserved vector materializer names (4357228431), distinct distance-call exemptions (4357228435), and the redundant heartbeat capability (4357063534) are addressed. Generic string/binary normalization ambiguity remains covered by4357228447; the syntax failure reported here is distinct. No existing P0/P1 inline finding required carrying forward.
User focus: no additional focus points were supplied. All 55 authoritative changed paths and relevant upstream/downstream contracts received static coverage. Builds, tests, live query execution and SDK interoperability matrices were not run, as required by the review contract; no source files or submodules were modified.
Review converged after 3 round(s).
…Groovy The lookbehind's escaped quote terminated the double-quoted string literal mid-pattern, so the shared framework fails to compile and no suite can start. The pattern moves to a slashy string literal, where neither the backslash nor either quote character needs escaping. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
FE UT Coverage ReportIncrement line coverage |
The mirrorHadoopS3OptionsToCatalogOptions loop set every fs.s3a.* entry from the generated Hadoop configuration unconditionally, so for a catalog with static s3 keys plus an explicit assumed-role credentials provider the Hadoop side's SimpleAWSCredentialsProvider replaced the requested provider in the catalog options — the warehouse then re-opened under the base identity and failed with AccessDenied. The mirror now fills only options the catalog has not already set: the user's explicit fs.s3a.* entries keep precedence over the generated defaults. This affects the filesystem, HMS and JDBC paimon catalog variants alike. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Follow-up to #66227, which introduced the paimon-rust reader. This PR upgrades the Paimon SDK to the community 2.1.0 release and ports the primary-key vector (ANN) search path onto the community stack:
ORDER BY l2_distance_approximate|inner_product_approximate(col, vector) LIMIT nis rewritten into a pushdown onto the paimon scan and served by the paimon-rust vector reader on BE.Related PR: #68988 (for branch-4.1 ， current closed)
Problem Summary:
1. Upgrade Paimon to the community 2.1.0
S3FileIOreads S3 credentials only fromCatalogContext.options()(verified at bytecode level:S3FileIO#configure -> loadHadoopConfigFromContextnever touches the HadoopConfiguration, unlike 1.4; unchanged in the community 2.1.0). Doris keeps a catalog's storage credentials in the Hadoop conf only, so every s3-warehouse paimon catalog initialized the SDK with no credentials and failed withUnable to load credentials.mirrorHadoopS3OptionsToCatalogOptionsnow mirrors thefs.s3a.*entries onto the catalog options for the FileSystem/HMS/JDBC metastores, keeping user extras (path style, connection limits) flowing as well._FILEfield is the legacy nullable-union type (written by paimon versions older than 1.4), so the pre-provisioned paimon1 CI regression baseline and old production tables stay readable; the internal 2.1 BETA build rejected them withexpected RECORD but found UNION.1bac394) followed by its revert (1871ef6), a net zero for reviewers.2. PK-vector (ANN) search
PushDownVectorTopNIntoPaimonScan(registered inRewriter) matches a vector distascan and pushes the query vector, the indexed vector column, the metric and the limit down to the scan.A pre-flight check refuses incremental scans (t@incr), so an incremental query can never silently resolve to a snapshot read.AnnTopNInfocarries the pushed-down info fromLogicalFileScantoPhysicalFileScanand ontoPaimonScanNode(PhysicalPlanTranslatorsets it only inside thePAIMON_EXTERNAL_TABLEbranch); thePlanNodes.thriftadditions are additive fields.paimon_rust_table_readergrows the vector-search path, reading through the rust PK-vector build-op on this path (the rust vector readers do not consume it) — noted in the reader.enable_paimon_rust_readerandenable_file_scanner_v2switches; with either off, the query brute-forces through the normal scan.3. Unify the rust eligibility gates
PaimonRustEligibilityevaluates the 12 table-level/storage/BE checks once and is shared by the batch and vector paths. The vector gate is now computed independently of the batch gate (it previously inheritedfallbackRead/scan.moderestrictions from the batch path) and both branches share a common gate chain with a batch-only split-suitability suffix.Tests
PushDownVectorTopNIntoPaimonScanTest(positive / incremental-refused / gate-negative coverage),PaimonScanNodeTestgate coverage, plannerPaimonPredicateConvertertests,PaimonUtilTest; 485 paimon-related FE UTs pass against the 2.1.0 stack.test_paimon_pk_vector_ann: the suite builds its own fixture through the Spark ThriftServer — an ivf-rq PK-vector index withnlist = nprobe = 4on 512 deterministic 16-dim vectors, i.e. exhaustiverecall, so ANN must equal brute force by construction. It compares against the brute-force answer comsabled, asserts the ANN leg really executed on the paimon-rust vector reader (profile check against asilent brute-force pass), and covers both
l2andinner_productmetrics, residual filters and non-index projections.scan.creation-time-millisleg silently resolved to the latest snapshot; Doris-side resolution verified correct by hand-reproducing the queries), andassertSparkDorisResultEqualsnow decodes the0x-hex tokens Doris renders for VARBINARY nested in arrays/maps/structs so identical nested binary values compare equal across engines.SHOW TBLPROPERTIES) — the underlying behaviors are unchanged.external_table_p0/paimonregression suites against the 2.1.0 stack: read-path 34/37 and write-path 26/32 green (the rest need a Spark 4.0 ThriftServer for VARIANT DDL or pre-provisioned fixtures that exist only in CI), plus the HDFS-based suites against the original pre-provisioned paimon1 OSS baseline — no Doris-side regressions.Release note
The Paimon catalog now runs on the community Paimon 2.1.0 release. Queries of the form
ORDER BY l2_distance_approximate|inner_product_approximate(<vector col>, <query vector>) LIMIT nover paimon tables with a PK-vector index arepushed down into the paimon-rust vector reader (requires
enable_paimon_rust_readerand `enable_file_sy paimon versions older than 1.4 (legacy union manifests) remain readable.Check List (For Author)
Check List (For Reviewer who merge this PR)