Repository navigation
Move expression analysis, rewrites and statistics pruning into vortex-expr - #10734
Open
robert3005 wants to merge 2 commits into
Open
robert3005 wants to merge 2 commits into
robert3005 wants to merge 2 commits into
Conversation
…-expr
Expression analysis, the rewrite transforms and statistics pruning are only
used by scan planning (vortex-layout, vortex-file, vortex-spatial and one
`replace` call in vortex-datafusion); nothing in vortex-array's arrays or
execution calls them. They move to a new `vortex-expr` crate:
- `vortex_expr::analysis` (annotations, labeling, field-access analysis,
`referenced_field_paths`), also re-exported at the crate root as before
- `vortex_expr::forms::conjuncts`
- `vortex_expr::transform::{partition_bound, partition_bound_annotations,
BoundPartitionedExpr, replace, find_between}`
- `vortex_expr::stats`: the stat-expression helpers, `StatBinder`, the
falsify/satisfy rewrite rules and `StatsSession`, which turn filter
predicates into pruning predicates for zone maps and file stats
vortex-array keeps the statistics themselves (`StatsSet`, `ArrayStats`, `Stat`,
`Precision`) and the `StatFn` scalar function. `vortex_array::expr::transform`
is removed; `find_between` is no longer part of `BoundExpression::optimize`.
- `BoundExpression::{falsify, satisfy}` were inherent methods; they become the
`BoundExpressionPruneExt` extension trait.
- `LikeVariant`, which only the LIKE pruning rule uses, moves with it.
- `DynamicComparisonExpr` gains `operator()`, `default()` and
`with_operator()`, since the pruning rules can no longer reach its fields.
- `array_session()` no longer adds `StatsSession`; `session.get` creates it
with the built-in rules on first use, and `VortexSessionDefault` still adds
it explicitly.
- The falsification tests for `is_null`, `is_not_null` and `list_contains`
move to `vortex_expr::stats::rewrite`, and the one optimizer test that used
`replace` builds the replaced expression directly.
The umbrella crate's `vortex::expr` module re-exports both `vortex_array::expr`
and `vortex_expr`, and `vortex::expr::stats` re-exports both stats modules, so
existing paths keep working.
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ps4mp7KF5c4p66aoSoXx5j
robert3005
added
changelog/chore
github-actions
Bot
removed
the
action/bench-all
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.015x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.009x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.015x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.890x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (0.852x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.837x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.018x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.910x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.973x ➖, 1↑ 0↓)
|
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.997x ➖, 0↑ 0↓)
vortex / parquet / ns (0.994x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.988x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.995x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.987x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.993x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Merging this PR will regress 8 benchmarks3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Contributor
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.993x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.972x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.995x ➖, 0↑ 0↓)
File Size Changes (200 files changed, -0.0% overall, 90↑ 110↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (0.997x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (0.960x ➖, 0↑ 0↓)
random-access / parquet / ns (0.999x ➖, 1↑ 1↓)
random-access / lance / ns (0.914x ➖, 1↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.978x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (0.949x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.017x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.037x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.984x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.970x ➖, 0↑ 2↓)
|
Bench statics registered every vortex-array session variable whether the bench used it or not, and the spatial benches and three `vortex` benches started from `spatial_session()` or the full `VortexSession::default()`. Each bench's static now registers exactly the session variables its run creates, found by running every bench once in test mode with an empty session and recording each variable `VortexSession::get` created. `main` still forces the static, so no benchmark iteration creates a session variable. The `vortex` benches register the encodings they exercise rather than every default encoding, layout, runtime and plugin. The CUDA benches keep their chains, since they cannot run without a GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ps4mp7KF5c4p66aoSoXx5j Signed-off-by: Claude <noreply@anthropic.com>
Contributor
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.013x ➖, 1↑ 3↓)
random-access / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
random-access / parquet / ns (1.005x ➖, 0↑ 0↓)
random-access / lance / ns (0.998x ➖, 0↑ 0↓)
random-access / litdata / ns (0.995x ➖, 0↑ 0↓)
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
joseph-isaacs
approved these changes
Oct 23, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expression analysis, the rewrite transforms and statistics pruning are only used by scan planning (vortex-layout, vortex-file, vortex-spatial and one
replacecall in vortex-datafusion); nothing in vortex-array's arrays or execution calls them. They move to a newvortex-exprcrate:vortex_expr::analysis(annotations, labeling, field-access analysis,referenced_field_paths), also re-exported at the crate root as beforevortex_expr::forms::conjunctsvortex_expr::transform::{partition_bound, partition_bound_annotations, BoundPartitionedExpr, replace, find_between}vortex_expr::stats: the stat-expression helpers,StatBinder, the falsify/satisfy rewrite rules andStatsSession, which turn filter predicates into pruning predicates for zone maps and file statsvortex-array keeps the statistics themselves (
StatsSet,ArrayStats,Stat,Precision) and theStatFnscalar function.vortex_array::expr::transformis removed;find_betweenis no longer part ofBoundExpression::optimize.BoundExpression::{falsify, satisfy}were inherent methods; they become theBoundExpressionPruneExtextension trait.LikeVariant, which only the LIKE pruning rule uses, moves with it.DynamicComparisonExprgainsoperator(),default()andwith_operator(), since the pruning rules can no longer reach its fields.is_null,is_not_nullandlist_containsmove tovortex_expr::stats::rewrite, and the one optimizer test that usedreplacebuilds the replaced expression directly.