[jiangxt2]

Description

Forward supported leading structured filters from IngestionRequest.transforms to ray-doris and ray-clickhouse, allowing the database to exclude rows before returning them to Ray. For the 10,000-row validation fixture, id >= 9900 returns 100 rows from the native reader and produces the same final result as Ray-side filtering.

• Validate the complete transform pipeline against source metadata, compile restricted predicates using ClickHouse bound parameters or validated Doris literals, and check the filtered reader's schema.
• Remove exactly pushed predicates from Ray execution, preserve the order of residual transforms, and expose the decisions through existing receipts. Read errors propagate without retrying an unfiltered scan; generated SQL and parameter values stay out of receipts.
• Add an owned single-node Docker Ray database suite, a controlled CI workflow with execution evidence and a stable database-filter-gate check, unit and contract coverage, and user documentation.

Related issues

Fixes #133.

Upstream dependency limitation: ray-project/ray#67230.

Additional information

Validation

• uv run --locked --no-sync python scripts/pr-precheck.py passed, including the complete CI unit suite in isolated dev-only Python 3.12 and 3.13 environments and the documentation static, strict-build, and spelling checks. Six non-blocking inline-import warnings in test helpers were reviewed.
• ./scripts/run_database_filter_it.sh --database clickhouse: 22 tests passed without skips against ClickHouse 26.8.1.2041, covering single, physical-partition, and integer-range reads.
• ./scripts/run_database_filter_it.sh --database doris: 21 tests passed without skips against Doris 4.0.6, covering MySQL and Flight reads.
• Database validation used Linux arm64, Python 3.12, Ray 2.59.0, ray-clickhouse==1.1, and ray-doris==1.3. It verifies successful real-driver SQL execution, bound values, a reduction from 10,000 baseline rows to 100 native output rows, equivalent final rows and schema, and no residual Ray filter for the exact predicate.

Scope and limitations

Public input shapes and dependency versions are unchanged. The scope is bounded table reads through these two Ray bindings. Unsupported predicates and filters after a transform barrier retain ordered Ray execution; arbitrary SQL, other connectors, Daft, and writes are outside this change. Single-node evidence does not establish multi-node behavior, index pruning, or a performance speedup.

Ray 2.59.0 residual filters containing Unicode literals can fail with UnicodeEncodeError in operator metric tags, as tracked above. Exact source execution avoids the corresponding Ray Filter, but residual expressions remain affected. The read guide and support matrix document this dependency limitation; this change does not modify Ray or reorder transforms to avoid it.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were identified.

Summary

Adds source-side filtering for supported leading transforms in the Ray ClickHouse and Doris bindings.

• ClickHouse and Doris apply supported leading filters before matching rows reach Ray.
• A single-node database suite checks filtered reads against ClickHouse and Doris.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Structured read and transforms] --> B[Discover source schema]
  B --> C[Validate complete pipeline]
  C --> D[Compile supported leading filters]
  D --> E{Any source filters?}
  E -->|Yes| F[Create filtered reader]
  F --> G[Check filtered schema]
  E -->|No| H[Keep original reader]
  G --> I[Apply remaining transforms in order]
  H --> I
  I --> J[Return dataset and receipt decisions]

Loading

_{Reviews (1) · Last reviewed commit: "feat(data): push read filters to Doris a..." · Reviewed by Greptile}