You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reduce per-batch allocation in broadcast serialization #7309
Broadcast serialization allocates a 1 MiB heap chunk and constructs a compression codec for every input batch. When the input contains many tiny Arrow batches, this repeats allocation and codec setup thousands of times even though each serialized stream is small.
Size output chunks from the Arrow field and dictionary buffers, bounded between 4 KiB and 1 MiB, and reuse a compression codec within one serializer iterator while creating an independent output stream for each batch. Keep output framing and decoding unchanged.
This is the shared serializer improvement discussed in #7258. It should land independently of the experimental native producer-batching setting because it applies to every broadcast input.
Activity