Skip to content

"end to end" test for reading parquet using custom PageIndexProvider #11029

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

#10842 introduces the PageIndexProvider trait so that applications can plug in custom page index implementations (for example, sparse or lazily-loaded indexes) into ParquetMetaData.

However, no test currently drives reading with a custom PageIndexProvider through the actual read path.

Several places in the code use as_any().downcast_ref::<PageIndex>() and take a different path for custom providers (for example ParquetMetaData::memory_size, PartialEq for ParquetMetaData, and ParquetMetaDataWriter), so adding specific reader coverage would be good.

  • The custom_page_index.rs example implements OnDemandPageIndexProvider, but it only prints index availability, and examples are compiled — not executed — in CI.
  • The existing reader tests exercise the built-in PageIndex implementation only.

Describe the solution you'd like

Add an integration test that:

  1. Writes a parquet file with page indexes (multiple row groups, multiple columns, multiple pages per column).
  2. Loads the metadata with PageIndexPolicy::Skip, then installs a custom PageIndexProvider implementation (similar to the example's OnDemandPageIndexProvider) via ParquetMetaDataBuilder::set_page_index, populated for only a subset of row groups / columns.
  3. Reads data through ParquetRecordBatchReader (and/or the async reader / push decoder) with a RowSelection that triggers page skipping, and verifies:
    • correct results are returned,
    • the offset-index-driven fetch path is used for columns where the provider returns Some,
    • columns where the provider returns None fall back to whole-column-chunk fetching.

Describe alternatives you've considered

Converting the custom_page_index.rs example into an executed test, or adding assertions to it and running it in CI.

Additional context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

parquetChanges to the parquet crate

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions