perf: Optimize RowSelection::and_then for dense masks - #10968
Open
haohuaijin wants to merge 11 commits into
Open
perf: Optimize RowSelection::and_then for dense masks#10968haohuaijin wants to merge 11 commits into
RowSelection::and_then for dense masks#10968haohuaijin wants to merge 11 commits into
Conversation
RowSelection::and_then for dense masks
haohuaijin
marked this pull request as draft
September 3, 2026 05:38
haohuaijin
marked this pull request as ready for review
September 3, 2026 11:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
RowSelection::and_thenfor dense masks #10967.Rationale for this change
The mask-backed
RowSelection::and_thenimplementation has high per-selected-row overhead for large, dense masks: it walks the outer set bits one at a time and appends each surviving row individually. Expanding the inner mask one 64-bit word at a time is substantially faster for these inputs.What changes are included in this PR?
popcountbits of the inner mask into the set positions of the outer word (a softwarepdep), iterating over whichever side of the word has fewer bits.The dense path costs a fixed amount per output word regardless of how many inner rows survive. The set-index path only walks the outer set bits up to the last inner survivor, so it stays faster when the outer mask is sparse, or when only a few inner rows survive and they are clustered near the start of the selection. The thresholds keep the set-index path for those inputs.
The benchmark suite is submitted separately in #10969.
Are these changes tested?
Yes. Tests cover bitmap offsets, threshold boundaries, and randomized 64-bit deposit inputs across the full mask density range, including every mask with at most one set or unset bit.
Using the benchmark suite in #10969, all 38 cases that select the dense path improved, with mean execution time reduced by 41.7% to 97.6% on an Apple M4 with Rust 1.97.1. The remaining 26 cases take the unchanged set-index path and show no measurable difference.
mainAre there any user-facing changes?
No. This is an internal performance optimization with no API or behavior changes.