Repository navigation
Blog / paper post about parquet push decoder #8035
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
-
Part of [Epic] Parquet Reader Improvement Plan / Proposal - July 2025 #8000
-
Related to Decouple IO and CPU operations in the Parquet Reader (push decoder) #7983
Describe the solution you'd like
I would like to explain the benefits of a push decoder in a more digestable format than a PR (even if the code is well documented 
)
I think this will also help reviewers
Describe alternatives you've considered
Idea is to write a blog post about the push parquet decoder, and how it can be used to offer more fine grained control over IO and CPU work in the parquet reader.
The blog post would cover:
- Motivation: why do we need a push decoder?
- Design: how does the push decoder work?
- Examples: how to use the push decoder in practice
- Performance: how does the push decoder perform compared to the existing parquet reader?
- Future work: what are the next steps for the push decoder?
Additional context
Activity
[-]Blog post about parquet pushdown ecoder[/-][+]Blog post about parquet push encoder[/+]yshcz commentedon Aug 3, 2025
Should the title be push decoder, not encoder?
alamb commentedon Aug 4, 2025
🤦 yes thank you @yshcz -- fixed
[-]Blog post about parquet push encoder[/-][+]Blog post about parquet push decoder[/+]Rewrite `ParquetRecordBatchStream` in terms of the PushDecoder (#8159)
alamb commentedon Feb 27, 2026
Another thing to highlight would be an example of how to read a parquet file in smaller chunks than row groups (e.g. if you wanted to read it 100K rows at a time) -- as discussed here: #6946 (comment)
[-]Blog post about parquet push decoder[/-][+]Blog / paper post about parquet push decoder[/+]alamb commentedon Jun 8, 2026
Update here is that @xudong963 @HippoBaro @neilconway and @Dandandan are working on a paper to submit to CIDR for this work. When complete we'll post it here, and perhaps cross post a blog out if it
alamb commentedon Aug 12, 2026
Draft post here from @xudong963