Repository navigation
[Feature Request]: Add a Variant type to Beam Schemas #38251
Description
What would you like to happen?
PCollections of Beam Rows are required to have a fixed schema, making it hard to read or write records with varying logical schemas in the same PCollection.
We need a semi-structured type that can contain a nested level of variable columns. Transforms can choose to unwrap this type to reconstruct the variable columns accordingly.
The Variant type is becoming a good standard across different projects (see Parquet, Spark, Flink). It could also be a good candidate for Beam.
Bonus if such a type is made portable so that other SDKs can use it too
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
chamikaramj commentedon May 11, 2026
Noting that lakehouse solutions Delta Lake and Iceberg recently introduced support for the Variant type.
https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark
https://www.snowflake.com/en/engineering-blog/apache-iceberg-v3-variant-type/
So probably a good idea to add this support natively to Beam as well.
github-actions commentedon Oct 9, 2026
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
CH-Abhinav commentedon Oct 13, 2026
Commenting to keep this issue active.
With discussions like PR #39236 highlighting this again, having a
Varianttype in Beam Schemas seems like the core foundational feature needed to properly unblock dynamic schemas across connectors like Iceberg and BigQuery.I'd really like to explore and give a shot at this. I've started looking into the open Parquet/Iceberg Variant specifications to see how they could map to Beam.
@ahmedabu98 — what do you think would be the best starting point or path forward for this?