Skip to content

Compose and drive a V2 scan in DataFusion - #10708

Open
joseph-isaacs wants to merge 6 commits into
ji/scan-v2-chain-2-plannersfrom
ji/scan-v2-chain-3-drivers
Open

joseph-isaacs wants to merge 6 commits into
ji/scan-v2-chain-2-plannersfrom
ji/scan-v2-chain-3-drivers

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Third of a stack of three that lands the planner-driven V2 scan for DataFusion behind a switch, cut from #10694. Stacked on #10707 (the planner chain), itself on #10706. With this merged, scan_v2 runs TPC-H end to end; the perf and diagnostics commits of #10694 follow as separate PRs on top.

@codspeed

codspeed Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

2 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 23 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2406 untouched benchmarks
⏩ 12 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime bitpack_blocked_compress_avx2 6.7 µs 7.6 µs -11.3%
⚡ Walltime threshold_neon[80] 4 µs 3.5 µs +13.48%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/scan-v2-chain-3-drivers (25458e3) with ji/scan-v2-chain-2-planners (26c4265)2

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on ji/scan-v2-chain-2-planners (c441bc4) during the generation of this report, so 872b4d1 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-2-planners branch from d091576 to a6f15f1 Compare October 23, 2026 13:08
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-3-drivers branch from da8a7c8 to a3ef12f Compare October 23, 2026 13:08
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-2-planners branch from a6f15f1 to 1c7e1c8 Compare October 23, 2026 13:49
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-3-drivers branch from a3ef12f to 10ac271 Compare October 23, 2026 13:49
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-2-planners branch from 1c7e1c8 to 26c4265 Compare October 23, 2026 13:56
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-3-drivers branch from 10ac271 to f23da3a Compare October 23, 2026 13:56
joseph-isaacs and others added 6 commits October 23, 2026 15:07
A planner says what it needs, compute or reads. Where either runs is an
engine's decision, so this crate stops driving: `ScanBuilder::prepare`
lowers and plans a scan without IO into a `PreparedScan`, which exposes
its rows, selection, filter plans and IO service, and `PrunePlanner` and
`ChunkPlanner` are public for an engine to compose, the prune planner
continuing into whatever it is built with and passing `None` when no
proof ran. The builder's ordering, concurrency, mapping and streams go
with the driver, and the layout's tests drive their scans on a run of
their own.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener composes a scan's planners, a prune planner into a chunk
planner when the scan has a filter and the chunk planner alone when it
does not, and a driver of this crate runs them: the root run stepped on
`spawn_cpu` tasks as DataFusion polls the stream, each spawned chunk on
a task of its own up to the concurrency, or on the root run when the
engine's partitions already fill the cores, with the concurrency
derived from cores per partition unless configured. Batches are mapped
to Arrow on the task that produced them. Every spawning decision of a
V2 scan under DataFusion is in this crate.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`vortex_file::v2::ScanDriver` composes a scan's planners and runs them
on the session's runtime, the way an engine's driver does, for the file
API's callers: the compress and random-access benchmarks move to it.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener prepares the scan where compute runs, composes its planners
itself, a prune planner continuing into a chunk planner when there is a
filter and the chunk planner alone when there is not, and hands the
driver the first planner. The driver only drives.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The prune planner skipped prefetching the zone maps of proofs an earlier
scan of the file had proved. The segment list now comes from compiling
the proof, and a proved proof compiles to its kept result without a
segment scan, so the check repeated what compile already knew.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The scan code named the filter's zone-map plans proofs. They are the
filter's pruning expressions evaluated over the file's zone map, so the
layout, file and DataFusion code now says pruning, matching
prune_zones, PrunedZones and the pruning field they already used.
Behaviour is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-v2-chain-3-drivers branch from f23da3a to 25458e3 Compare October 23, 2026 14:07
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action/bench-sql Run SQL benchmarks without Vortex Compact on this PR changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant