Repository navigation
Separate RowFn batch inspection from execution planning - #10673
connortsui20 wants to merge 1 commit into
Conversation
Merging this PR will degrade performance by 5.32%3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
5541df9 to
abc8689
Compare
80cbea7 to
9663585
Compare
abc8689 to
d3afc24
Compare
Inspect borrowed inputs once for lengths, dtypes, constant classification, and strict validity. Pair collected inputs with a separate execution plan while retaining the existing dispatch and retry paths. Extracted from #9979. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
d3afc24 to
bb199a5
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.021x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.988x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.982x ➖, 2↑ 2↓)
datafusion / parquet / ns (0.844x ➖, 6↑ 0↓)
duckdb / vortex-file-compressed / ns (0.928x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.928x ➖, 1↑ 1↓)
|
|
Closing this performance extraction after the controlled measurements did not establish an independent performance case for the batch-inspection refactor. Its main role in this work was to enable the fused comparison migration, which is now closed in #10676. The smaller internal candidates in #10741 and #10742 target develop independently and do not require the borrowed batch entry point or inspection types. |
Summary
Extracts batch setup from #9979 into a private borrowed-input boundary. Input lengths, dtypes, constants, and strict validity are inspected together before attaching the dispatch-selected execution plan.
Targets
developand can merge independently. #10675 and #10676 use the borrowed batch entry point introduced here.Changes
Keeps constant classification, input ownership, and execution metadata inside the framework. General execution still collects owned inputs and uses the existing dispatch and retry paths. Function IDs are deferred during inspection until an error needs them, with the supporting
CachedIdinlining change kept here.The Vortex test suite was not run locally. Focused native measurements of the combined migration were run separately during the compiler investigation.