feat(vortex): add predicate pushdown and optimize selective reads by zhouyc-ali · Pull Request #471 · apache/paimon-cpp

zhouyc-ali · 2026-10-27T10:03:35Z

Purpose

Linked issue: #319.

Push supported predicates, projections and bitmap selections into native Vortex scans while preserving physical file row IDs and Paimon's residual-filter semantics.

Support comparisons, NULL checks, IN and NOT IN, with per-reader membership policies, deduplication and integer-range compression. Fall back when types, timestamp precision or NaN semantics cannot be preserved.
Reduce selective-read overhead by reusing default callback-reader sessions, lazily allocating dictionary filter caches and sharding file-local caches. Add explicit I/O coalescing modes while keeping the object-storage default.
Validate pinned native patch states and extend the benchmark runner to support Vortex. Cover projection/schema alignment, reader reconfiguration and callback lifetimes without changing the existing writer implementation.

Tests

Added and updated tests covering:

Predicate conversion and native scan results for comparisons, NULL checks,
IN and NOT IN, including duplicates, integer boundaries, range compression,
and combined AND/OR predicates.
Safe fallback for unsupported or mismatched types, unbound columns, lossy
timestamps, floating-point semantics, and membership policy limits.
Fixed-seed differential tests check native results before SDK residual filtering.
Reordered, nested, repeated, empty, and missing-column projections, including
NULL filling, schema metadata preservation, and incompatible-type rejection.
Sparse, dense, full, and empty bitmap selections combined with predicates
and projection, preserving physical row IDs across batches. The single-deletion
regression verifies one exclusion index instead of expanding all surviving rows.
Reader reset and invalid reconfiguration, ensuring failed resets preserve the
active scan and returned batches remain valid after reset or reader destruction.
I/O coalescing modes preserving values and row IDs, injected read/write failures,
waiting for in-flight callbacks on abort, and deferred stream closure.
C++ SDK reads with residual filtering and late materialization enabled/disabled,
primary-key updates/deletes with merging before value filtering, and schema
evolution with missing columns and a user column named _row_idx.

Local validation: 89 Vortex tests passed in normal and targeted TSAN runs.
The TSAN run instrumented Vortex C++ code and tests, reusing core/Rust dependencies.
Apache RAT and changed-file lint checks passed.

API and Format

No changes to public C++ API signatures, stored file formats or the table protocol. Add the reader options vortex.read.membership-pushdown and vortex.read.io-coalesce.

The private Vortex FFI is extended; its patched header and native library must be rebuilt together. Readers continue to require SDK residual filtering (EnablePredicateFilter(true)) or equivalent filtering by the caller for exact results.

Documentation

Document the two Vortex reader options, their defaults and the residual-filter requirement in the existing reading guide (docs/source/user_guide/read.rst).

Generative AI tooling

Generated-by: OpenAI Codex 0.163.0

Retain the smaller side of bitmap selections and preserve physical row mapping. Add regression coverage for dense, sparse, and reconfigured selections. Use callback I/O in native-scan test fixtures to avoid the Rust async filesystem path reported by TSAN, without adding suppressions. Add Apache license headers to both Vortex patch files.

lxy-9602 · 2026-10-28T09:04:52Z

+        assert_eq!(source.request_count.load(Ordering::SeqCst), ids.len());
+        Ok(())
+    }
+


Could you please describe the real benefits this change will bring? After all, diffs are hard to maintain.

lxy-9602 · 2026-10-28T09:41:22Z

+++ b/vortex-ffi/src/dtype.rs
+@@ -344,7 +344,9 @@
+ ) -> c_int {
+     try_or(err, 1, || {


[P2] Bound native batches before converting view arrays to Utf8/Binary

Normalizing the target schema here makes execute_arrow materialize ordinary Utf8/Binary arrays for the entire native chunk, before VortexFileBatchReader::NextBatch() slices it to the configured batch_size. Previously, the view-to-standard conversion happened after that slicing.

For dictionary-encoded long strings, this can expand a small encoded chunk into a large contiguous allocation. If a column’s expanded payload exceeds the approximately 2 GiB limit of 32-bit offsets, conversion can fail even though each requested C++ batch would fit. Reducing the C++ batch_size cannot prevent this because conversion has already happened.

Could we split the native output before execute_arrow, or retain view arrays across the FFI and convert after slicing? This would preserve bounded materialization and avoid introducing a larger conversion boundary.

lxy-9602 · 2026-10-28T09:55:48Z

+
+                let data_source = if local {
+                    vx_data_source_new_callback_with_coalesce(
+                        session,


It seems we only support the local and object store modes right now. We have HDFS scenarios that need to support configurable max_gap and max_size, so we can’t just hard-code it based on the mode.

lxy-9602 · 2026-10-28T10:00:25Z

-        : batch_size_(batch_size), pool_(GetDefaultPool()), arrow_pool_(GetArrowPool(pool_)) {}
+    explicit VortexReaderBuilder(
+        int32_t batch_size, const VortexPredicateOptions& predicate_options = {},
+        VortexIoCoalesceMode coalesce_mode = VortexIoCoalesceMode::OBJECT_STORAGE)


Please avoid default parameters in the production code.

lxy-9602 · 2026-10-28T10:05:48Z

+results, retain the full residual predicate: enable ``EnablePredicateFilter(true)``
+with ``TableRead``, or apply equivalent filtering in the caller. ``force`` does not
+bypass this requirement.
+


Please put these parameters into a vortex_defs.h file.

Can you confirm whether this configuration is the default for Java? If it isn’t, please use a more flexible configuration instead of an enum that limits choices to a fixed set of categories.

Also, why is this documentation placed in read.rst? Please check where it should live—it seems intended for external users (the user guide) rather than format-configuration documentation. Please adjust the doc location accordingly.

lxy-9602 · 2026-10-28T13:07:22Z

            return VortexCallbackError("create Vortex scan", error,
                                       input_context_->GetCallbackStatus());
        }
        scan_ = std::move(scan);


Could you please refactor this part of the code? For example, the creation of scan_ could be extracted into a separate function. The current function is a bit too long.

lxy-9602 · 2026-10-28T13:26:04Z

+    selected_row_count_ = selected_row_count;
+    selection_indices_ = std::move(selection_indices);
+    metrics_->SetCounter("vortex.selection-index-count", selection_indices_.size());
    ReleaseStream();


Please define the metric name in defs.h first. Also, could we rename it to vortex.selection-row-count? “index” is a bit ambiguous since it could be interpreted as an index.

lxy-9602 · 2026-10-28T13:41:54Z

-    return std::make_unique<VortexReaderBuilder>(batch_size);
+    VortexPredicateOptions predicate_options;
+    const auto membership_mode = options_.find("vortex.read.membership-pushdown");
+    if (membership_mode != options_.end()) {


Similar changes needed: please adjust the parameter naming and their positions as well.

lxy-9602 · 2026-10-28T13:45:05Z

+        .EnablePrefetch(true)
+        .SetPrefetchBatchCount(2)
+        .SetPrefetchMaxParallelNum(2)
+        .AddOption(Options::READ_BATCH_SIZE, "2");


I’m curious whether vortex supports prefetch. If it doesn’t, why are the EnablePrefetch and related options being configured?

lxy-9602 · 2026-10-28T13:56:54Z

+                // A non-exact reader may prune candidates, but it must retain every matching row.
+                ASSERT_NO_FATAL_FAILURE(
+                    AssertVortexSdkRows(actual, result_type, expected, exact_filter));
+            }


Please tighten your validation. Enabling either late_materializing or exact_filter is sufficient to guarantee correct query results and predicate accuracy.

zjw1111 · 2026-10-29T04:02:02Z

@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one


The "fmm" is from the db name? It's not a public name, maybe need rename it.

lxy-9602 · 2026-10-29T01:10:05Z

+    /// @return An owned expression, or nullptr if no condition can be pushed.
+    static VxExpressionPtr Convert(const std::shared_ptr<Predicate>& predicate,
+                                   const std::shared_ptr<arrow::Schema>& file_schema,
+                                   const VortexPredicateOptions& options = {});


Similar to before: in formal code, please avoid using default parameters as much as possible.

lxy-9602 · 2026-10-29T01:31:11Z

+    if (!timestamp_type.timezone().empty() && timestamp_type.timezone() != "UTC") {
+        return false;
+    }
+    const auto timestamp = literal.GetValue<Timestamp>();


Please carefully review the read/write paths and predicates related to tz, and ensure the behavior is consistent with Java.

lxy-9602 · 2026-10-29T01:36:30Z

+    *result = value + fraction;
+    *result_unit = unit;
+    return true;
+}


This function seems to be primarily handling timestamp precision conversion. Could you please try to use the utility functions from DateTimeUtils instead?

lxy-9602 · 2026-10-29T01:47:56Z

+        case FieldType::FLOAT:
+        case FieldType::DOUBLE:
+            // Native totalOrder distinguishes NaN signs and payloads; Paimon treats all NaNs alike.
+            return scalar;


I checked the Java logic— it looks like they push down floating-point values. For correctness around NaN, could you please open a PR to fix it in Java and keep both sides consistent?

lxy-9602 · 2026-10-29T02:57:29Z

+    if (literal.IsNull() || literal.GetType() != field_type || arrow_type == nullptr) {
+        return false;
+    }
+    switch (field_type) {


Similarly, you can assume the predicate types match the corresponding field types in the file. If there’s a mismatch, just error out—there’s no need for this kind of fallback.

lxy-9602 · 2026-10-29T03:10:44Z

+        nodes += lower == upper ? 1 : 3;
+    }
+    return nodes;
+}


I think these IN-expression optimizations should be handled by the query engine itself. Do we really need to introduce these code changes specifically for this? It seems to add a lot of complexity, and other formats don’t appear to benefit from the same optimization.

Also, pushing down deduplication seems more appropriate to be done in the upstream layer.

lxy-9602 · 2026-10-29T05:13:27Z

+            const auto& decimal_type = checked_cast<const arrow::Decimal128Type&>(*arrow_type);
+            const auto decimal = literal.GetValue<Decimal>();
+            return decimal.Precision() == decimal_type.precision() &&
+                   decimal.Scale() == decimal_type.scale();


If the decimal precisions don’t match, can we do a precision conversion? Or does vortex actually support this kind of pushdown?

lxy-9602 · 2026-10-29T05:37:19Z

+    if (type == Function::Type::AND || type == Function::Type::OR) {
+        const auto compound = std::dynamic_pointer_cast<CompoundPredicate>(predicate);
+        if (compound == nullptr) {
+            return NullExpr();


For compound, besides and/or, are there any other type? It looks like most predicate logic is swallowed and falls back to a “don’t push down” behavior on error. We should distinguish between cases where we silently avoid pushdown because it’s unsupported, versus cases where there’s a real issue that should result in an error.

feat(vortex): add predicate pushdown and optimize selective reads

37dce2a

SteNicholas reviewed Oct 27, 2026

View reviewed changes

Comment thread src/paimon/format/vortex/vortex_file_batch_reader.cpp Resolved

SteNicholas reviewed Oct 27, 2026

View reviewed changes

Comment thread cmake_modules/vortex-fmm-cache-shards.diff Resolved

lxy-9602 reviewed Oct 29, 2026

View reviewed changes

zjw1111 reviewed Oct 29, 2026

View reviewed changes

lxy-9602 reviewed Oct 29, 2026

View reviewed changes

		@@ -0,0 +1,131 @@
		# Licensed to the Apache Software Foundation (ASF) under one

Conversation

zhouyc-ali commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhouyc-ali commented Oct 27, 2026 •

edited

Loading