[rok]

Rationale for this change

apache#43761 (apacheGH-43187) added is_in predicate simplification against row group statistics by filtering the value set and rebinding the call for every guarantee. That made large value sets prohibitively slow (apache#46777), so apache#46859 disabled the simplification above 50 values. Since then is_in predicates with hundreds or millions of values cannot prune any row groups, even though that is exactly the case where pruning pays off most (see G-Research/gr-oss#1440 and the equivalent Polars change pola-rs/polars#29923).

The closed PR apache#43256 already explored a sorted value set with binary search; this PR picks that up with the constraints raised in its review: no flag on SetLookupOptions, no mutation of the options, the sorted copy is kept in the kernel state and written at most once, and the result is exact rather than hull based.

What changes are included in this PR?

Sorted value set in the is_in kernel state (scalar_set_lookup_internal.h, scalar_set_lookup.cc)

• is_in and index_in get separate init functions. For is_in with more than kIsInSimplificationMaxValueSet (50) values of a searchable type (integers, floats, temporal types, decimals, binary-like types), the init keeps a copy of the value set sorted in the order of the comparison kernels, with nulls removed.
• An already sorted value set (e.g. a range of ids) is reused as is after a single O(N) check. An unsorted one is sorted lazily and thread-safely on the first simplification, so eager kernel calls never pay for sorting.
• Primitive value sets are sorted by copying the values and calling std::sort, which is about 14x faster than sort_indices + take for 10M shuffled int64 values. Binary and decimal sets use the generic path.
• Value sets containing NaN are not searchable, since statistics cannot bound NaN.
• SetLookupStateBase moves to a header shared between the kernel (libarrow_compute) and expression simplification (libarrow); everything in it is header-only.

Simplification against all inequalities at once (expression.cc)

• SimplifyWithGuarantee collects every inequality conjunct of the guarantee and applies all of them to each large is_in call in one pass, narrowing an index range into the sorted value set with lower/upper bound searches. If the range is empty the call simplifies to false, subject to the same null matching rules as the small set path; otherwise it is left unchanged, so nothing is rebound.
• Applying both bounds of a row group together is what allows pruning a row group that lies inside the hull of the value set but holds none of its values. A per-bound (hull) check cannot do that.
• Bounds are cast to the value set type when they differ; a bound that cannot be safely cast is ignored, which only loses precision.
• Value sets up to 50 values keep using the existing filter-and-rebind path unchanged.

Are these changes tested?

• Expression.SimplifyIsInLargeValueSet covers hull pruning, gap pruning, redundant bounds, guarantees on other fields, casts in both directions, unsorted and duplicated sets, strings (sorted, unsorted, with nulls), floats with and without NaN, dictionaries (not searchable), the full null matching matrix including a guarantee that is nullable on one bound only, equality-or-null guarantees, and composition with other predicates.
• TestParquetFileFormatScan.PredicatePushdownRowGroupFragmentsLargeIsIn prunes 30 of 80 row groups that lie inside the hull of a 60 value set but hold none of its values, with sorted, shuffled, duplicated and int32 value sets, and combined with another predicate.
• Local sweep of the arrow-compute-*, arrow-acero-* and arrow-dataset-* test suites at the base commit and at each commit of this PR: all 43 suites pass at each of the three points (base e7b0e50dfa, commit 1, commit 2).

End-to-end scan sweep. Real Parquet scans through arrow::dataset (ScannerBuilder::Filter + ToTable, threads on, min of 3 runs, files in page cache, Apple Silicon, release build) at the base commit (main), commit 1 (sorted value set + interval search) and commit 2 (+ std::sort fast path). The big file follows apache#46777: 47 row groups of 1Mi rows with a sorted int64 id column (1..49.3M) and a float64 column. The small file follows gr-oss#1440: 4096 ids x 8 rows in 4096-row row groups. Value set layouts: sorted contiguous ids 1..N; the same ids shuffled; two clusters of N/2 ids at the start and at the end of the data, whose hull spans every row group so only an exact check can prune anything; and N ids spread evenly over the data, where every row group holds a match and nothing can be pruned (the no-benefit case, which shows the overhead).

49.3M rows, 47 row groups of 1Mi rows (apache#46777 layout)

Value set	N	rows returned	scan main	scan commit 1	scan commit 2	speedup (commit 2 vs main)
no filter		49,283,072	84 ms	102 ms	94 ms	0.9x
sorted	100	100	104 ms	3.7 ms	3.7 ms	28.1x
shuffled	100	100	101 ms	3.6 ms	3.9 ms	26.0x
clusters	100	100	111 ms	5.7 ms	6.6 ms	16.9x
spread	100	100	103 ms	119 ms	117 ms	0.9x
sorted	1,000	1,000	116 ms	4.3 ms	4.1 ms	28.4x
shuffled	1,000	1,000	117 ms	4.0 ms	4.1 ms	28.5x
clusters	1,000	1,000	117 ms	6.9 ms	7.4 ms	15.9x
spread	1,000	1,000	119 ms	138 ms	132 ms	0.9x
sorted	10,000	10,000	104 ms	3.7 ms	3.7 ms	28.1x
shuffled	10,000	10,000	117 ms	4.1 ms	3.9 ms	30.1x
clusters	10,000	10,000	106 ms	6.0 ms	5.8 ms	18.3x
spread	10,000	10,000	109 ms	117 ms	114 ms	1.0x
sorted	100,000	100,000	126 ms	6.3 ms	6.4 ms	19.7x
shuffled	100,000	100,000	132 ms	12 ms	7.2 ms	18.4x
clusters	100,000	100,000	130 ms	8.4 ms	7.7 ms	16.9x
spread	100,000	100,000	138 ms	147 ms	141 ms	1.0x
sorted	1,000,000	1,000,000	295 ms	26 ms	26 ms	11.4x
shuffled	1,000,000	1,000,000	307 ms	112 ms	42 ms	7.3x
clusters	1,000,000	1,000,000	306 ms	33 ms	34 ms	8.9x
spread	1,000,000	1,000,000	344 ms	347 ms	354 ms	1.0x
sorted	10,000,000	10,000,000	454 ms	267 ms	256 ms	1.8x
shuffled	10,000,000	10,000,000	474 ms	3.01 s	461 ms	1.0x
clusters	10,000,000	10,000,000	513 ms	277 ms	284 ms	1.8x
spread	10,000,000	10,000,000	474 ms	452 ms	431 ms	1.1x

32K rows, 8 row groups of 4096 rows (gr-oss#1440 layout)

Value set	N	rows returned	scan main	scan commit 1	scan commit 2	speedup (commit 2 vs main)
no filter		32,768	0.3 ms	0.3 ms	0.3 ms	1.0x
sorted	64	512	0.3 ms	0.1 ms	0.1 ms	3.0x
sorted	256	2,048	0.3 ms	0.1 ms	0.1 ms	3.0x
sorted	1,024	8,192	0.3 ms	0.2 ms	0.2 ms	1.5x

Reading: up to 50 values nothing changes. Above that, main reads every row group and the scan costs the full file regardless of N, while this PR reads only the row groups that can hold a match: 16-30x faster for sets up to 100K ids, 7-11x at 1M, 1.8x at 10M where 10 of 47 row groups genuinely contain matches and the hash table build dominates. The spread layout and the unfiltered scan vary by the same few milliseconds between runs, i.e. the pruning attempt costs nothing measurable when it cannot prune. The one place commit 1 alone was a regression, a 10M shuffled set (3.0 s vs 0.47 s, from the generic sort), is what commit 2 fixes (0.46 s).

Simplification-only microbenchmark (bind + SimplifyWithGuarantee over 47 row group guarantees, no I/O)

Sorted contiguous ids 1..N

N	main simplify	commit 1 simplify	commit 2 simplify	main pruned	commit 1 pruned	commit 2 pruned	bind main	bind commit 2
10	141 µs	148 µs	146 µs	46 / 47	46 / 47	46 / 47	1.0 µs	2.0 µs
50	131 µs	138 µs	138 µs	46 / 47	46 / 47	46 / 47	1.0 µs	2.0 µs
100	20 µs	27 µs	26 µs	0 / 47	46 / 47	46 / 47	2.0 µs	2.0 µs
1,000	19 µs	28 µs	27 µs	0 / 47	46 / 47	46 / 47	7.0 µs	7.0 µs
10,000	18 µs	32 µs	30 µs	0 / 47	46 / 47	46 / 47	43 µs	46 µs
100,000	19 µs	31 µs	30 µs	0 / 47	46 / 47	46 / 47	449 µs	481 µs
1,000,000	18 µs	31 µs	38 µs	0 / 47	46 / 47	46 / 47	18 ms	21 ms
10,000,000	20 µs	53 µs	54 µs	0 / 47	37 / 47	37 / 47	195 ms	193 ms

The same ids, shuffled

N	main simplify	commit 1 simplify	commit 2 simplify	main pruned	commit 1 pruned	commit 2 pruned	bind main	bind commit 2
10	130 µs	146 µs	137 µs	46 / 47	46 / 47	46 / 47	1.0 µs	1.0 µs
50	131 µs	137 µs	138 µs	46 / 47	46 / 47	46 / 47	1.0 µs	1.0 µs
100	20 µs	31 µs	27 µs	0 / 47	46 / 47	46 / 47	2.0 µs	2.0 µs
1,000	19 µs	58 µs	34 µs	0 / 47	46 / 47	46 / 47	7.0 µs	7.0 µs
10,000	19 µs	419 µs	128 µs	0 / 47	46 / 47	46 / 47	42 µs	43 µs
100,000	18 µs	5.5 ms	1.6 ms	0 / 47	46 / 47	46 / 47	431 µs	428 µs
1,000,000	17 µs	80 ms	16 ms	0 / 47	46 / 47	46 / 47	17 ms	19 ms
10,000,000	20 µs	2.42 s	180 ms	0 / 47	37 / 47	37 / 47	215 ms	218 ms

Two clusters: N/2 ids at the start and N/2 at the end of the data (hull spans every row group)

N	main simplify	commit 1 simplify	commit 2 simplify	main pruned	commit 1 pruned	commit 2 pruned	bind main	bind commit 2
10	286 µs	311 µs	295 µs	45 / 47	45 / 47	45 / 47	1.0 µs	1.0 µs
50	295 µs	299 µs	304 µs	45 / 47	45 / 47	45 / 47	1.0 µs	1.0 µs
100	19 µs	27 µs	26 µs	0 / 47	45 / 47	45 / 47	1.0 µs	2.0 µs
1,000	20 µs	29 µs	27 µs	0 / 47	45 / 47	45 / 47	6.0 µs	7.0 µs
10,000	19 µs	30 µs	29 µs	0 / 47	45 / 47	45 / 47	42 µs	46 µs
100,000	19 µs	27 µs	28 µs	0 / 47	45 / 47	45 / 47	644 µs	639 µs
1,000,000	17 µs	36 µs	34 µs	0 / 47	45 / 47	45 / 47	19 ms	20 ms
10,000,000	20 µs	57 µs	52 µs	0 / 47	37 / 47	37 / 47	208 ms	213 ms

Are there any user-facing changes?

No API changes. is_in filters with more than 50 values now prune Parquet row groups again, exactly.

Follow-ups not included here: a benchmark for SimplifyWithGuarantee with is_in, and distributing or(and(ge, le), is_null) row group guarantees so that nullable columns get inequality simplification at all.

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview