Variant Data Type Support

Proposed Change

We would like to propose to add Variant type to Iceberg data types.

Variant data types allow for the efficient binary encoding of dynamic semi-structured data such as JSON, Avro,Parquet, etc. By encoding semi-structured data as a variant column, we retain the flexibility of the source data, while allowing query engines to more efficiently operate on the data.

With the support of Variant type, such data can be encoded in an efficient binary representation internally for better performance. Without that, we need to parse the data in its format inefficiently.

This will allow the following use cases:

Create an Iceberg table with a Variant column
CREATE OR REPLACE TABLE car_sales(record Variant);
Insert semi-structured data into the Variant column
INSERT INTO car_sales SELECT PARSE_JSON(<json_string>)
Query against the semi-structured data
SELECT VARIANT_GET(record, '$.dealer.ship', 'string') FROM car_sales

Proposal document

https://docs.google.com/document/d/1sq70XDiWJ2DemWyA5dVB80gKzwi0CWoM0LOWM7VJVd8/edit?tab=t.0

Specifications

+1

Any ideas on release dates ?

+1

Any updates on release date for Varient Feature ?

Hi folks. We are actively working on the Variant support. I have made a POC with Spark implementation and currently we are working on individual PRs to move forward. The full Variant support includes part1 - basic Variant type encoding/decoding (without shredding) and part2 - Variant with shredding. We will get part1 to work soon. The spec for Part2 is getting finalized and we will work on the design.

@aihuaxu

@aihuaxu Will this also work for protobuf encoded columns? I have a dataset with
event_bytes | event_name

101010100 | e1
101010100 | e2

@aihuaxu

@aihuaxu Will this also work for protobuf encoded columns? I have a dataset with event_bytes | event_name

101010100 | e1 101010100 | e2

That shouldn't work. Variant has its own specific encoding. We need to store the value in a Variant column which will convert and store in its specific binary format. Protobuf encoding is stored differently.

@RussellSpitzer

@RussellSpitzer Can you help reopen this? It's automatically closed when one task #10831 (#10831) mentioned "fixed this issue" . Thanks.

+1

@aihuaxu

@aihuaxu Hi, does Spark support pushing the variant_get function down to Iceberg, so that it avoids reading the entire variant column and instead only reads its subcolumn (in shredding mode)? Or will this be implemented later?

@eldenmoon

@aihuaxu Hi, does Spark support pushing the variant_get function down to Iceberg, so that it avoids reading the entire variant column and instead only reads its subcolumn (in shredding mode)? Or will this be implemented later?

@eldenmoon Sorry for the late reply. We need some work to support such pushdown to Iceberg. That will be implemented later.

@aihuaxu

@aihuaxu Hi, does Spark support pushing the variant_get function down to Iceberg, so that it avoids reading the entire variant column and instead only reads its subcolumn (in shredding mode)? Or will this be implemented later?

@eldenmoon Sorry for the late reply. We need some work to support such pushdown to Iceberg. That will be implemented later.

Any updates on release date or time line @aihuaxu ?

Hi there Varient feature has been released please check out example here
Learn how to use Variant Feature in Iceberg 1.10.0 V3 with PySpark Code Example

https://medium.com/@shahsoumil519/learn-how-to-use-variant-feature-in-iceberg-1-10-0-v3-with-pyspark-code-example-2c2ae84829cf

@aihuaxu Hi, does Spark support pushing the variant_get function down to Iceberg, so that it avoids reading the entire variant column and instead only reads its subcolumn (in shredding mode)? Or will this be implemented later?

@eldenmoon Sorry for the late reply. We need some work to support such pushdown to Iceberg. That will be implemented later.

Any updates on release date or time line @aihuaxu ?

To have the pushdown to Iceberg from Spark, it needs apache/spark#52578 which will be in Spark 4.1.0 release. We are waiting for the release.

Added PRs to support file level skipping based on variant filter

The feature is important for my product. Any milestones to release?

in #15628/#15629 I'm adding benchmarks for variant read (plus something similar in parquet).
The current numbers (see the issue) show a major performance penalty doing simple queries over shedded variants -things get about twice as slow as unshedded. That hints to me that there's reconstruction of the variant struct going on before query execution

Hi everyone, I've seen the requests on this issue and others to help track the Variant progress better. To that end, I've proposed to set up some time on a monthly cadence to sync on all things Variant.
To keep things tracked, I've created a Community document that covers the current status, tracking of issues and PR, and the benchmark results from the community. Please take a look here - Variant Community Document. If you are interested in Variant, please chime in on the mailing list thread: https://lists.apache.org/thread/f8l27lr55nlrdg4qwno29r796d4v2p8l. Thanks.

I"ve put up an issue to track what's needed for shredded variants in iceberg, spark and parquet #16172

What about Kafka Connect auto-create / schema inference: Debezium io.debezium.data.Json (PG json & jsonb) -> Iceberg VariantType ?

Variant update ahead of the 1.12 release, with the PRs collected here so this issue is easy to follow.

You can define a VARIANT column, write it with Spark, Flink, and Kafka Connect, and read it back either row-by-row or with the Spark vectorized reader. Parquet, Avro, and ORC are supported, along with metrics-based row-group filtering. Shredded writes work in Spark and Flink.

Current work: type uniformity for shredding, shredded-read extraction and predicate pushdown, and wider engine and Record/Data coverage.

Merged

Spec - #10831 variant type · #12658 lower/upper bounds · #16836 type classification & primitive scoping

Type + binary encoding (API / Core)

Type & API move: #11324, #11831, #12105, #12167, #12304, #12374
Values, toString, extra types: #11415, #12138, #12531, #12317, #12682 (timestamp_ns/time/UUID), #13151, #13284 (short string), #12835 (JSON path)
Bounds, sanitizing, hardening, serialization: #14362, #13137, #16568, #17260, #13629 (tests)

Parquet

Readers / writers / metrics: #12139, #12323, #12496, #12512 + #12847/#12933 (arrays), #12559, #14306
Row-group filter / metrics handling: #14081, #14279, #14588, #17377
Shredding + fixes: #15087 (partial-shred data loss), #16818, #17002 (large decimals), #16852 (metadata cache), #16880 + #17342 + #17397 (bounds), #15934 + #16585 (crashes), #17469 (encrypted round-trip)
Backports: #14467, #14624, #17352, #17354, #17588, #17589

Avro / ORC - #12238, #12457 (Avro) · #12567 (ORC generics)

Engines

Spark: #13219 (read), #14297 + #16241 (shredded write), #16292 (vectorized read for unshredded), #16087, #15964, #14276 (test)
Flink: #15265 (support), #15596 (shredded write), #16450
Kafka Connect: #15283

REST / OpenAPI - #17256 (VariantType in spec) · #17500 (RCK coverage)

Docs - #13828

Shredding writes - #17424 (type uniformity) · #17520 (Data + Kafka Connect)

Active / open

Shredded reads / pushdown - #16714 + #16715 (Spark extraction) · #15385 (variant_get file skipping) · #15384 (extract + manifest bounds) · #17431 (ordered predicates)

Engines / tooling - #16735 (Hive nested VARIANT) · #17617 (file-format TCK) · #17553 (partial/multi-column tests) · #15629 (JMH benchmarks) · #17537 (blog post)

@nssalian

hi @nssalian , super excited to see the progress with Variant type support esp with Kafka Connect!

do you know if there's a rough timeline for 1.12 release?

@trevorpburke

Thanks @trevorpburke. I'm hoping soon once some of the correctness fixes are in. I keep the dev list thread of the release up to date if you need to look at what is pending.

Navigation Menu

Repository navigation

Variant Data Type Support #10392

Proposed Change

Proposal document

Specifications

11 remaining items

Merged

Active / open

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Variant Data Type Support #10392

Description

Proposed Change

Proposal document

Specifications

Activity

soumilshah1995 commented on Oct 1, 2024

aihazm commented on Oct 25, 2024

soumilshah1995 commented on Oct 29, 2024

NurFaizin commented on Nov 4, 2024

tmnd1991 commented on Nov 22, 2024

soumilshah1995 commented on Nov 29, 2024

aihuaxu commented on Dec 2, 2024

stym06 commented on Jan 6, 2025

aihuaxu commented on Jan 6, 2025

aihuaxu commented on Jan 30, 2025

desairavi1101 commented on Feb 2, 2025

11 remaining items

eldenmoon commented on Sep 5, 2025

aihuaxu commented on Sep 12, 2025

soumilshah1995 commented on Sep 12, 2025

soumilshah1995 commented on Sep 20, 2025

aihuaxu commented on Nov 2, 2025

qlong commented on Feb 28, 2026

ndcuong869 commented on Mar 1, 2026

steveloughran commented on Mar 25, 2026

nssalian commented on Apr 19, 2026

steveloughran commented on Apr 30, 2026

dantrofm commented on Jul 15, 2026

nssalian commented on Aug 12, 2026

Merged

Active / open

trevorpburke commented on Sep 3, 2026

nssalian commented on Sep 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions