Skip to content

Reduce per-batch allocation in broadcast serialization #7309

Open

Description

@sunchao

Broadcast serialization allocates a 1 MiB heap chunk and constructs a compression codec for every input batch. When the input contains many tiny Arrow batches, this repeats allocation and codec setup thousands of times even though each serialized stream is small.

Size output chunks from the Arrow field and dictionary buffers, bounded between 4 KiB and 1 MiB, and reuse a compression codec within one serializer iterator while creating an independent output stream for each batch. Keep output framing and decoding unchanged.

This is the shared serializer improvement discussed in #7258. It should land independently of the experimental native producer-batching setting because it applies to every broadcast input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions