Skip to content

Deprecate .withProjection from ParquetIO #20770

Closed

Description

@damccorm

There are multiple issues wrong with the API of withProjection:

  1. The current API requires an extra encoderSchema that is not needed when projecting data in Parquet. The simplest way to get this with the Parquet API is by passing the projectionSchema like this:

{color:#000000}AvroReadSupport{color}.setAvroReadSchema({color:#871094}conf{color}, {color:#871094}projectionSchema{color});
olor:#000000}AvroReadSupport{color}.setRequestedProjection({color:#871094}conf{color}, {color:#871094}projectionSchema{color});
We can offer an alternative method withProjection(Configuration conf, List<String\> fields) so users don't have to build their own projection Schema, but historically we have let users to rely on the upstream connector API. If we follow this we can better document in ParquetIO how to project fields by relying in the Parquet APIs and avoid maintaining this extra code in the Beam side.

 

Imported from Jira BEAM-11908. Original Jira may contain additional context.
Reported by: iemejia.

Activity

  1. DevPJ9 commented on Apr 4, 2023

    @DevPJ9
    Contributor

    Hello @damccorm, I would like to contribute.

  2. nitinware commented on Sep 7, 2026

    @nitinware
    Contributor

    .take-issue

  3. added a commit that references this issue on Oct 5, 2026
    976f052
  4. added this to the 2.78.0 Release milestone on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions