Skip to content

perf: Optimize RowSelection::and_then for dense masks - #10968

Open
haohuaijin wants to merge 11 commits into
apache:mainfrom
haohuaijin:perf/dense-mask-and-then
Open

perf: Optimize RowSelection::and_then for dense masks#10968
haohuaijin wants to merge 11 commits into
apache:mainfrom
haohuaijin:perf/dense-mask-and-then

Conversation

@haohuaijin

@haohuaijin haohuaijin commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The mask-backed RowSelection::and_then implementation has high per-selected-row overhead for large, dense masks: it walks the outer set bits one at a time and appends each surviving row individually. Expanding the inner mask one 64-bit word at a time is substantially faster for these inputs.

What changes are included in this PR?

  • Add a word-at-a-time dense mask expansion path. Each output word deposits the next popcount bits of the inner mask into the set positions of the outer word (a software pdep), iterating over whichever side of the word has fewer bits.
  • Keep the existing set-index path for short or sparse masks.
  • Select the dense path for masks with at least 8,192 rows, at least ~75% outer selectivity, and at least ~5% inner selectivity.
  • Add unit tests for the new path.

The dense path costs a fixed amount per output word regardless of how many inner rows survive. The set-index path only walks the outer set bits up to the last inner survivor, so it stays faster when the outer mask is sparse, or when only a few inner rows survive and they are clustered near the start of the selection. The thresholds keep the set-index path for those inputs.

The benchmark suite is submitted separately in #10969.

Are these changes tested?

Yes. Tests cover bitmap offsets, threshold boundaries, and randomized 64-bit deposit inputs across the full mask density range, including every mask with at most one set or unset bit.

cargo test -p parquet --lib arrow::arrow_reader::selection::algebra::tests

test result: ok. 23 passed; 0 failed

Using the benchmark suite in #10969, all 38 cases that select the dense path improved, with mean execution time reduced by 41.7% to 97.6% on an Apple M4 with Rust 1.97.1. The remaining 26 cases take the unchanged set-index path and show no measurable difference.

Distribution Rows Outer Inner main PR Change
Pseudo-random 8,192 75% 5% 3,691 ns 1,456 ns -60.7%
Clustered 8,192 75% 5% 882 ns 513 ns -41.7%
Pseudo-random 8,192 99% 99% 18,255 ns 553 ns -96.9%
Clustered 65,536 99% 99% 142,840 ns 3,389 ns -97.6%

Are there any user-facing changes?

No. This is an internal performance optimization with no API or behavior changes.

@haohuaijin haohuaijin changed the title Optimize RowSelection::and_then for dense masks perf: Optimize RowSelection::and_then for dense masks Sep 3, 2026
@haohuaijin haohuaijin changed the title perf: Optimize RowSelection::and_then for dense masks perf: Optimize RowSelection::and_then for dense masks Sep 3, 2026
@haohuaijin
haohuaijin marked this pull request as draft September 3, 2026 05:38
@haohuaijin
haohuaijin marked this pull request as ready for review September 3, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize RowSelection::and_then for dense masks

1 participant