You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bench: add the shuffle benchmark cases that the #5905 review found missing #7080
The shuffle benches in native/shuffle/benches do not cover several shapes that the performance review in #5905 flagged, so changes to those paths cannot show a win or catch a regression. What exists today: shuffle_writer.rs (codecs None/LZ4/Snappy/Zstd, a high-partition group, partitioning placement, schema encoding, IPC context), shuffle_reader.rs (codecs none and lz4, dictionary blocks, schema parse) and row_columnar.rs (primitive, struct, list and map conversion).
Describe the potential solution
Add the missing cases, each as its own small PR:
Writer: end-to-end wide schema (100+ columns); List, Struct and Map through interleave; a spill run under a memory limit; a compressed high-partition case (the existing high-partition group uses codec None).
Reader: zstd and snappy blocks; List and Map blocks; a cached-schema variant against decode_block_uncached for nested schemas.
Activity
HarshilMital commentedon Oct 18, 2026
take
bench: add nested-schema reader cases for the schema cache (apache#7080)
bench: add a wide-schema end-to-end writer case (apache#7080)
bench: add nested-column gather cases (apache#7080)
bench: add zstd and snappy reader cases (apache#7080)
bench: add list and map reader cases (apache#7080)
bench: add a compressed high-partition writer case (apache#7080)
bench: add a spilling end-to-end writer case (apache#7080)
bench: add nested-schema reader cases for the schema cache (apache#7080)
bench: add zstd and snappy reader cases (apache#7080)
bench: add list and map reader cases (apache#7080)
bench: add zstd and snappy reader cases (apache#7080)
bench: add a compressed high-partition writer case (apache#7080)
bench: add a wide-schema end-to-end writer case (apache#7080)
bench: add a spilling end-to-end writer case (apache#7080)
bench: add nested-column gather cases (apache#7080)
bench: add list and map reader cases (apache#7080)
bench: add nested-schema reader cases for the schema cache (apache#7080)
bench: add string, Map<Utf8, Int64> and nullable row-to-columnar cases (