Skip to content

fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary - #26668

Open
ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:fix-nested-pruning-wrapper-leaf-subset
Open

ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:fix-nested-pruning-wrapper-leaf-subset

Conversation

@ranflarion

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A table that declares a narrower element struct under a FixedSizeList, ListView, LargeListView or Dictionary column still reads every Parquet leaf. The read plan derives the emitted type with type_for_leaf_subset, which rejects partial selections under those wrappers, so the clip #24315 computes for them always falls back to a full read.

What changes are included in this PR?

project_type_for_leaves descends through the same wrappers as clip_type, rebuilding each around its projected child with the physical outer parameters (FixedSizeList keeps its size, Dictionary its key type). That is the type #24315's reader tests pin for those wrappers.

What is the testing strategy for this PR?

type_for_leaf_subset_matches_clip_through_wrappers checks that type_for_leaf_subset reproduces clip_for_cast's type for every wrapper the clip descends through; on main it fails at ListView. type_for_leaf_subset_rejects_unsupported_partial_wrapper now uses RunEndEncoded, which stays opaque. A FixedSizeList section in parquet_nested_schema_pruning.slt pins bytes_scanned for the narrow table at 83 B against 174 B for the full schema; on main both read 174 B. ListView and Dictionary columns can't be declared in SQL, so the unit test is what covers them.

Are there any user-facing changes?

No API changes. Scans of such columns read fewer bytes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Oct 23, 2026

@rgbuilds rgbuilds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b474db1a. The wrapper reconstruction matches the existing clip_for_cast behavior, preserves the relevant outer type and field properties, and retains conservative fallback for unsupported partial wrappers.

I also verified the focused nested-pruning and projection-read-plan tests and the Parquet SLT locally. Restoring the previous behavior makes the wrapper regression fail and causes the narrow FixedSizeList scan to read the full 174 B instead of 83 B. The change looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested Parquet pruning reads every leaf through FixedSizeList, ListView and Dictionary wrappers

2 participants