Repository navigation

Support datafusion dynamic filters - #10665

Draft

robert3005 wants to merge 21 commits into

developfrom

rk/datafusion-dynamic-exprs

Draft

Support datafusion dynamic filters#10665
robert3005 wants to merge 21 commits into
developfrom
rk/datafusion-dynamic-exprs

Commits on Oct 21, 2026
- Prove comparisons true for every row from statistics
  
  robert3005
  and
  claude
  committed
  58d5cc5
- Push snapshotted dynamic filters into the Vortex DataFusion scan
  
  claude
  authored and
  robert3005
  committed
  c0d3daa
- Skip dynamic filter bounds that file statistics prove always true
  
  claude
  authored and
  robert3005
  committed
  dae7593
- Add dynamic expressions whose predicate can change during a scan
  
  robert3005
  and
  claude
  committed
  ee65614
- Push DataFusion dynamic filters into Vortex scans as dynamic expressions
  
  robert3005
  and
  claude
  committed
  6a4a41b
- Skip optional filter conjuncts that keep most of their rows
  
  robert3005
  and
  claude
  committed
  54f2d34
- Mark DataFusion dynamic filters optional and evaluate implied bounds first
  
  robert3005
  and
  claude
  committed
  bc9720a
- Make the optional filter backoff fit split files and duplicate conjuncts
  
  robert3005
  and
  claude
  committed
  440b8ff
- Convert DataFusion dynamic filters once and push only selective bounds
  
  robert3005
  and
  claude
  committed
  d62e88f
- Skip dynamic conjuncts without a value and filters that keep half the rows
  
  robert3005
  and
  claude
  committed
  57908c1
- Estimate the selectivity of dynamic filters from file statistics
  
  robert3005
  and
  claude
  committed
  61a311d
- Do not prune the zones of optional conjuncts in scans of a few zones
  
  robert3005
  and
  claude
  committed
  d0c95ed
- Build the pushed conjunction of a complete dynamic filter once per file
  
  robert3005
  and
  claude
  committed
  83378ee
- Order filter conjuncts by their cost and selectivity
  
  robert3005
  and
  claude
  committed
  293c4c7
- Convert dynamic filters without blocking concurrent openers
  
  robert3005
  and
  claude
  committed
  000757f
- Evaluate dynamic filter conjuncts Vortex cannot express with DataFusion
  
  robert3005
  and
  claude
  committed
  53eb3f2
- Push DataFusion-evaluated conjuncts only when they save decoding
  
  robert3005
  and
  claude
  committed
  e6a0929
- Measure optional conjuncts on one split and back off from costly ones
  
  robert3005
  and
  claude
  committed
  87feab9
- Pass a demand to scalar functions and short-circuit AND and OR with it
  
  robert3005
  and
  claude
  committed
  d46ce86
- Replace the optional marker with a switch over the predicate
  
  robert3005
  and
  claude
  committed
  dd72d43
- Pass the demand to scalar functions without filtering their inputs
  
  robert3005
  and
  claude
  committed
  42d695e

Commits

Commits on Oct 21, 2026