Repository navigation
Compose and drive a V2 scan in DataFusion - #10708
joseph-isaacs wants to merge 6 commits into
Conversation
Merging this PR will regress 1 benchmark2 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
d091576 to
a6f15f1
Compare
da8a7c8 to
a3ef12f
Compare
a6f15f1 to
1c7e1c8
Compare
a3ef12f to
10ac271
Compare
1c7e1c8 to
26c4265
Compare
10ac271 to
f23da3a
Compare
A planner says what it needs, compute or reads. Where either runs is an engine's decision, so this crate stops driving: `ScanBuilder::prepare` lowers and plans a scan without IO into a `PreparedScan`, which exposes its rows, selection, filter plans and IO service, and `PrunePlanner` and `ChunkPlanner` are public for an engine to compose, the prune planner continuing into whatever it is built with and passing `None` when no proof ran. The builder's ordering, concurrency, mapping and streams go with the driver, and the layout's tests drive their scans on a run of their own. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener composes a scan's planners, a prune planner into a chunk planner when the scan has a filter and the chunk planner alone when it does not, and a driver of this crate runs them: the root run stepped on `spawn_cpu` tasks as DataFusion polls the stream, each spawned chunk on a task of its own up to the concurrency, or on the root run when the engine's partitions already fill the cores, with the concurrency derived from cores per partition unless configured. Batches are mapped to Arrow on the task that produced them. Every spawning decision of a V2 scan under DataFusion is in this crate. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`vortex_file::v2::ScanDriver` composes a scan's planners and runs them on the session's runtime, the way an engine's driver does, for the file API's callers: the compress and random-access benchmarks move to it. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The opener prepares the scan where compute runs, composes its planners itself, a prune planner continuing into a chunk planner when there is a filter and the chunk planner alone when there is not, and hands the driver the first planner. The driver only drives. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The prune planner skipped prefetching the zone maps of proofs an earlier scan of the file had proved. The segment list now comes from compiling the proof, and a proved proof compiles to its kept result without a segment scan, so the check repeated what compile already knew. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The scan code named the filter's zone-map plans proofs. They are the filter's pruning expressions evaluated over the file's zone map, so the layout, file and DataFusion code now says pruning, matching prune_zones, PrunedZones and the pruning field they already used. Behaviour is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
f23da3a to
25458e3
Compare
Third of a stack of three that lands the planner-driven V2 scan for DataFusion behind a switch, cut from #10694. Stacked on #10707 (the planner chain), itself on #10706. With this merged,
scan_v2runs TPC-H end to end; the perf and diagnostics commits of #10694 follow as separate PRs on top.