Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datafusion/core/tests/physical_optimizer/join_selection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -472,7 +472,7 @@ async fn test_join_with_swap_semi() {
/// `PartitionMode::Partitioned` is deliberately not a case here: a null-aware
/// `LeftAnti` cannot be constructed in that mode at all, so there is no input
/// plan to optimize. See
/// `HashJoinExec::null_aware_mode`/`NullAwareMode::try_new`.
/// `HashJoinExec::null_aware_mode`/`NotIn::try_new`.
#[rstest]
#[case(PartitionMode::CollectLeft)]
#[case(PartitionMode::Auto)]
Expand Down
2 changes: 1 addition & 1 deletion datafusion/physical-optimizer/src/join_selection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ fn statistical_join_selection_subrule(
&& should_swap_join_order(left, right, context)?
{
// A null-aware join is never in `Partitioned` mode
// (`NullAwareMode::try_new` rejects it), so the swap
// (`NotIn::try_new` rejects it), so the swap
// keeps the mode this arm matched on.
hash_join
.swap_inputs(PartitionMode::Partitioned)
Expand Down
3 changes: 3 additions & 0 deletions datafusion/physical-plan/src/joins/array_map.rs
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,9 @@ impl ArrayMap {
self.data.capacity() * size_of::<u32>() + self.next.capacity() * size_of::<u32>()
}

// Kept out of line: inlined into `HashJoinStream::process_probe_batch`,
// this loop measured ~4-6% slower on high-fanout probes.
#[inline(never)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The effects measured for this pin and for the #[inline(always)] on process_probe_batch (4–9%) aren't far above the 2.5% that symbols alone moved Q24, and they were measured on one aarch64 machine. Do they hold on x86_64? If not clearly, I'd lean toward leaving both out of this refactor or moving them to a separate perf PR. They encode one LLVM version's inlining decisions, and the percentages in the comments will go stale as the surrounding code changes. This one is also on a pub method, so it applies to every caller, not just the hash join probe.

pub fn get_matched_indices_with_limit_offset(
&self,
prob_side_keys: &[ArrayRef],
Expand Down
421 changes: 113 additions & 308 deletions datafusion/physical-plan/src/joins/hash_join/exec.rs

Large diffs are not rendered by default.

6 changes: 2 additions & 4 deletions datafusion/physical-plan/src/joins/hash_join/exec/prepared.rs
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,10 @@ impl PreparedHashJoinBuild {
pub(super) fn probe_data(&self, probe_threads: usize) -> JoinLeftData {
JoinLeftData {
build: Arc::clone(&self.build),
null_aware_scope_map: None,
null_value_build_rows: None,
// Prepared builds exclude null-aware joins.
not_in: None,
visited_indices_bitmap: Mutex::new(BooleanBufferBuilder::new(0)),
null_indices_bitmap: Mutex::new(BooleanBufferBuilder::new(0)),
probe_completion: ProbeCompletion::new(probe_threads),
build_side_has_null: false,
}
}

Expand Down
1 change: 1 addition & 0 deletions datafusion/physical-plan/src/joins/hash_join/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ pub use partitioned_hash_eval::{HashExpr, SeededRandomState};

mod exec;
mod inlist_builder;
mod not_in;
mod partitioned_hash_eval;
mod probe_completion;
mod shared_bounds;
Expand Down
Loading
Loading