[thorfour]

Summary

• Tracking Issue: Support file stats on nested fields #6389

Being able to perform file level pruning on nested columns speeds up queries.

Changes

Adds post order stats to the file stats footer for nested columns. It adds a new is_nested: bool = false to the stats set flatbuffer for backward compatibility.

Additionally it modifies the way the stats footer is accumulated to use the same AggRef functions that the zone maps use as well.

Only goofy thing it it also has to track truncatable stats separately because the aggregate partials don't have a way to express exactness.

[codspeed]

Merging this PR will degrade performance by 13.66%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 22 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 2 improved benchmarks
❌ 11 regressed benchmarks
✅ 2395 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Walltime	filter_neon[8]	911 ns	1,686 ns	-45.97%
❌	Walltime	threshold_neon[8]	1.3 µs	2.1 µs	-37.89%
❌	Walltime	threshold_neon[64]	2.6 µs	3.5 µs	-25.99%
❌	CPU Simulation	test_scan_filter[int]	2.8 ms	3.4 ms	-19.53%
❌	CPU Simulation	test_scan_filter_to_arrow_array[int_float]	3.5 ms	4.2 ms	-16.36%
❌	Walltime	filter_neon[64]	1.7 µs	2 µs	-13.74%
❌	CPU Simulation	test_repeated_scan_filter[int_float]	3.4 ms	3.9 ms	-10.9%
❌	Walltime	threshold_neon[80]	4.1 µs	4.6 µs	-10.49%
❌	CPU Simulation	test_repeated_scan_filter[int]	2.4 ms	2.7 ms	-10.26%
❌	Walltime	take_u16_nullable_from_65536_avx512[65536]	24 µs	26.7 µs	-10.16%
❌	CPU Simulation	test_scan_filter_to_arrow_array[int]	2.5 ms	2.8 ms	-10.03%
⚡	CPU Simulation	decode_primitives[i64, (1000, 8)]	37.2 µs	23.8 µs	+56.68%
⚡	Walltime	bitpack_blocked_compress_avx512	5.6 µs	4.9 µs	+13.44%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing polarsignals:statistices-nested-columns (1e91e81) with develop (b702068)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

I think for maximum compatibility this has to be a new top level field in https://github.com/vortex-data/vortex/pull/9775/changes#diff-9c5a36ce8f55480706984e8996f192cd3abbdd1b97afa6f702f646b3ce42e039L25. The problem you run into is that old readers already shipped code that make assumption about existing footer statistics. One issue is that they validate field count and if the count doesn't add up they will assume it's a corrupted file. The writer could optionally populate both of the segments and the new version will only ever read the new field.

If you go that route you will see all the places that make assumptions about field stats that need fixing, i.e. datafusion, duckdb and jni

[joseph-isaacs]

We also want to use aggregate partials here?

[thorfour]

@joseph-isaacs I've modified the summary stats to use aggregate partials.
@robert3005 I've moved the stats to a new top-level field, and left the legacy stats behavior. The writer can opt out of writing the legacy stats.