Skip to content

[C++][Python][ORC] Support batch iteration over selected stripes #54921

Open

Description

@jiangxt2

Describe the enhancement requested

Add a public API to iterate over RecordBatch objects from selected stripes of an ORC file, with a configurable maximum number of rows per batch and column projection.

ORCFile currently exposes whole-file and whole-stripe reads. When an application already knows which stripes it needs, it can call read_stripe() for each one, but each selected stripe is materialized as a single Arrow batch before processing can start. Slicing that result into smaller batches does not avoid the initial allocation. For large stripes, this creates memory pressure and unnecessary conversion when the consumer stops early.

The Dataset API already supports streaming full-file ORC scans. The missing capability is combining explicit stripe selection with batch iteration through the public Python API.

A possible interface, subject to API review, is ORCFile.iter_batches(batch_size=65536, stripes=None, columns=None), analogous to ParquetFile.iter_batches(row_groups=...). It should yield projected batches of at most batch_size rows without first materializing an entire selected stripe, and allow consumption to stop before the remaining rows are converted. Existing read() and read_stripe() behavior would remain unchanged.

The C++ ORC adapter already provides Seek(), NextStripeReader(batch_size), and a full-file GetRecordBatchReader(), but these batch-reading entry points are not exposed by the ORC Python bindings.

This request covers reading caller-supplied stripe IDs, independently of automatic statistics-based pruning or distributed task splitting. The batch size would limit output row counts; ORC dictionaries and decoder buffers could still require additional memory.

Related work: #49288 covers Dataset stripe subsetting, and #49360 covers statistics and selective reads. This request focuses on Python batch iteration over selected stripes.

Drafted with assistance from Codex.

Component(s)

Python, C++

Activity

  1. changed the title [-][Python][ORC] Support batch iteration over selected stripes[/-] [+][C++][Python][ORC] Support batch iteration over selected stripes[/+] on Oct 22, 2026
  2. added a commit that references this issue on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions