[mhk197]

Summary

Adds rebuild_with_referenced_valid_elements to ListArray and ListViewArray: the same lists over only the elements valid lists reference, with nothing before the first list, after the last, between lists or under a null list. A function that can fail on an element must not see the others; list_transform (#10503) is the first user.

Changes

• ListArray: slices the elements when only the ends go. When null lists hold elements, it copies the runs of valid lists between them (a filter with the ranges cached on the mask, so it copies range by range) and shifts the offsets in one pass. An array with nothing to drop comes back as it is.
• ListViewArray: a view not marked zero-copy to a list gets the existing MakeExact rebuild, which already skips null lists. A zero-copy view keeps its layout and only loses its ends, unless a null list holds elements; then it goes through the ListArray method.
• Both walk the runs of valid lists that the validity mask already caches.

Benchmark

list_transform with a checked x + 1 over 100k lists (~800k i32 elements, ~1/8 of the lists null and holding elements), through #10503, median µs, local scratch divan benchmark (runs vary by about ±5%):

Dropping the null lists' elements	µs
List view rebuild, before	1,399
ListArray::rebuild_with_referenced_valid_elements	339
DataFusion 55.2 array_transform, same lists	260

Testing

rstest cases for both methods: nothing to drop, leftovers at both ends, null lists holding elements at the start, middle and end, touching ranges, every list null, and for views a gap and lists out of order. Each checks the resulting elements, offsets and sizes, and that the lists read the same as the input.

Ran cargo +nightly-2026-09-10 fmt, cargo clippy -p vortex-array --all-targets --all-features -- -D warnings, cargo nextest run -p vortex-array, cargo test --doc -p vortex-array and the CI docs build.

Follow-up

• A zero-copy list view whose null lists hold elements converts to a list and back; copying its runs directly would save two passes over the offsets.

[codspeed]

Merging this PR will degrade performance by 1%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 2 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 21 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 4 improved benchmarks
❌ 8 regressed benchmarks
✅ 2396 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	test_scan_filter_to_arrow_array[int_float]	3.5 ms	4.1 ms	-13.82%
❌	CPU Simulation	filter_powerlaw_by_mostly_false[250000]	138.7 µs	158.5 µs	-12.48%
❌	CPU Simulation	test_repeated_scan_filter[int_float]	3.4 ms	3.9 ms	-11.22%
❌	CPU Simulation	filter_random_by_mostly_false[250000]	163.2 µs	183.4 µs	-11.03%
❌	CPU Simulation	test_repeated_scan_filter[int]	2.4 ms	2.7 ms	-10.39%
❌	CPU Simulation	test_scan_filter_to_arrow_array[int]	2.5 ms	2.8 ms	-10.34%
❌	CPU Simulation	filter_powerlaw_by_power_law[250000]	166.2 µs	185.2 µs	-10.27%
❌	CPU Simulation	filter_random_by_power_law[250000]	166.1 µs	185 µs	-10.21%
⚡	CPU Simulation	chunked_dict_primitive_canonical_into[f64, (1000, 1000, 10)]	205.2 µs	152.3 µs	+34.75%
⚡	CPU Simulation	bench_compare_primitive[(10000, 32)]	97 µs	72.7 µs	+33.55%
⚡	Walltime	starts_with_avx2[64, VarBin]	60.5 µs	53.4 µs	+13.18%
⚡	CPU Simulation	or_true_constant	21 µs	18.6 µs	+12.81%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/list-rebuild-referenced-valid-elements (a0ac766) with develop (c7a6cc9)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩