Is your feature request related to a problem or challenge? Please describe what you are trying to do.
#10842 introduces the PageIndexProvider trait so that applications can plug in custom page index implementations (for example, sparse or lazily-loaded indexes) into ParquetMetaData.
However, no test currently drives reading with a custom PageIndexProvider through the actual read path.
Several places in the code use as_any().downcast_ref::<PageIndex>() and take a different path for custom providers (for example ParquetMetaData::memory_size, PartialEq for ParquetMetaData, and ParquetMetaDataWriter), so adding specific reader coverage would be good.
- The
custom_page_index.rs example implements OnDemandPageIndexProvider, but it only prints index availability, and examples are compiled — not executed — in CI.
- The existing reader tests exercise the built-in
PageIndex implementation only.
Describe the solution you'd like
Add an integration test that:
- Writes a parquet file with page indexes (multiple row groups, multiple columns, multiple pages per column).
- Loads the metadata with
PageIndexPolicy::Skip, then installs a custom PageIndexProvider implementation (similar to the example's OnDemandPageIndexProvider) via ParquetMetaDataBuilder::set_page_index, populated for only a subset of row groups / columns.
- Reads data through
ParquetRecordBatchReader (and/or the async reader / push decoder) with a RowSelection that triggers page skipping, and verifies:
- correct results are returned,
- the offset-index-driven fetch path is used for columns where the provider returns
Some,
- columns where the provider returns
None fall back to whole-column-chunk fetching.
Describe alternatives you've considered
Converting the custom_page_index.rs example into an executed test, or adding assertions to it and running it in CI.
Additional context
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
#10842 introduces the
PageIndexProvidertrait so that applications can plug in custom page index implementations (for example, sparse or lazily-loaded indexes) intoParquetMetaData.However, no test currently drives reading with a custom
PageIndexProviderthrough the actual read path.Several places in the code use
as_any().downcast_ref::<PageIndex>()and take a different path for custom providers (for exampleParquetMetaData::memory_size,PartialEq for ParquetMetaData, andParquetMetaDataWriter), so adding specific reader coverage would be good.custom_page_index.rsexample implementsOnDemandPageIndexProvider, but it only prints index availability, and examples are compiled — not executed — in CI.PageIndeximplementation only.Describe the solution you'd like
Add an integration test that:
PageIndexPolicy::Skip, then installs a customPageIndexProviderimplementation (similar to the example'sOnDemandPageIndexProvider) viaParquetMetaDataBuilder::set_page_index, populated for only a subset of row groups / columns.ParquetRecordBatchReader(and/or the async reader / push decoder) with aRowSelectionthat triggers page skipping, and verifies:Some,Nonefall back to whole-column-chunk fetching.Describe alternatives you've considered
Converting the
custom_page_index.rsexample into an executed test, or adding assertions to it and running it in CI.Additional context
PageIndexBuilderandPageIndexProviderfor Parquet page indexes #10842