Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #2652 and #2657.
Rationale for this change
The benchmarking page shows one Ballista column from
c76278cd(2026-10-07). The pipelined execution mode in #2657 changes TPC-H SF1000 times a lot, so the page should show it next to the default blocking mode.What changes are included in this PR?
benchmarking/index.md: the single Ballista column becomes Ballista blocking and Ballista pipelined, both from one image built atcb49f30f(feat: Add experimental pipelined execution mode (all tasks placed up front, streaming exchange) #2657, on top ofmain98d724b3), with DataFusion55.2.0. The version row is updated to match, and a short paragraph says what the two modes are. The Spark, Comet and Trino columns are unchanged.benchmarking/ballista.md: a section listing the settings that differ for pipelined mode (push-staged scheduler and executors,ballista.execution.mode = pipelined), and the commands to reproduce it.Totals, in seconds (mean of 2 iterations, cold first iteration included):
Pipelined is faster on every query, from 7% (q1) to 80% (q14). The blocking column replaces the old 502.43 s from
c76278cd, since the page asks for the version, config and results tables to change together.Why this is a draft
cb49f30fis on the feat: Add experimental pipelined execution mode (all tasks placed up front, streaming exchange) #2657 branch, not onmain, and the pin should be amaincommit. When feat: Add experimental pipelined execution mode (all tasks placed up front, streaming exchange) #2657 (and feat: add pipelined exchange operators #2670 under it) merge, I'll rerun both modes at the merge commit and replace the numbers and the pin before this leaves draft.mainuntil then, and the commands I added for it won't work.How are these changes tested?
Documentation only. The numbers come from the runs in this comment: same image, cluster (32 executors, 8 vCPU and 64 GiB each,
gp3work-dir) and runner, one suite run per mode. The 22 queries return the same row counts in both modes (q11 returns 0 rows and q13 returns 30 in both). I haven't run a Markdown formatter on the tables.Are there any user-facing changes?
No.