EPIC: v3 Support Tracking

This tracks the progress on adding full v3 support to iceberg-rust.
Based on the pyiceberg tracking (apache/iceberg-python#1818)
and iceberg-go tracking (apache/iceberg-go#589).

New data types

Variant

Variant type in the schema: #2188
Recognize arrow.parquet.variant on Arrow schema import: #2840
Unshredded read: #2880 (reference implementation in iceberg-go: iceberg-go#929, iceberg-go#932)
Logical type annotation on write: #2546
Non-shredded write: no issue yet
Shredded read: no issue yet (reference implementation in iceberg-go: iceberg-go#986)
Shredded write: no issue yet (reference implementation in iceberg-go: iceberg-go#987)
Statistics: no min/max on the variant column itself, allow them on shredded children: no issue yet (reference implementation in iceberg-go: iceberg-go#988)
Predicate evaluation and partition-transform behavior on variant fields: no issue yet (reference implementation in iceberg-go: iceberg-go#1016)

Geometry / Geography (umbrella: #1884)

Recognize geometry and geography types [1/3]: #2933
Geospatial read support [2/3]: #3341
Geospatial Parquet write support [3/3]: #3342 (reference implementation in iceberg-go: iceberg-go#992, iceberg-go#995)
Manifest-level geo bounds: no issue yet (reference implementation in iceberg-go: iceberg-go#993)
Bounding-box predicate evaluation and pruning: no issue yet (reference implementation in iceberg-go: iceberg-go#994)

Unknown (umbrella: #3373)

unknown primitive type: #2773
Omit unknown fields from Parquet writes: #3072
Reconstruct unknown fields on read: #3073 (reference implementation in iceberg-go: iceberg-go#1003)

Nanosecond timestamps

timestamp_ns / timestamptz_ns types
Reject nanosecond types on v1/v2 tables for non-current schemas: #2418

Type promotion (date to timestamp / timestamp_ns, v3 only)

Manifest read fails on 4-byte bounds of promoted columns: #3387
Promoted column reads as NULL when another column is projected: #3404 (PR #3485)
Promote partition values and column bounds in the reader: #3503

Default values

initial-default on read for top-level fields: #737
Read timestamp_ns / timestamptz_ns default values in the JSON codec: #2832
initial-default for nested fields: #3261
Filters on columns missing from a data file must use the identity partition value or initial-default: #3298, #3301 (PR #3302)
JSON literal fixes: #3331, #3332 open (#3334 merged)
Apply write-default on write: #3250
Default values in UpdateSchema: #1357
DataFusion INSERT fills omitted columns with NULL instead of write-default: #2803 (the DataFusion integration moved to apache/datafusion-iceberg, #3029)

Row lineage

Assign first-row-id when writing the manifest list, advance next-row-id, and reject snapshots whose first-row-id is behind next-row-id: #1682 (reference implementation in iceberg-go: iceberg-go#741, iceberg-go#1004). #1652 can be closed.
Inherit data-file first_row_id on manifest read: #2922
Carry first_row_id and data sequence number on FileScanTask: #2952
_row_id and _last_updated_sequence_number metadata columns on read: #2959, #2966, #2985, #3058 (reference implementation in iceberg-go: iceberg-go#1010). #2879 looks complete and can likely be closed.
_last_updated_sequence_number physical values discarded when first_row_id is null: #2991
Carry row lineage into replacement files on copy-on-write rewrite: #3401 (reference implementation in iceberg-go: iceberg-go#999)

Deletion vectors

Read (epic: #2792, closed)

Decode deletion-vector-v1 Puffin blobs: #2866
Carry DV coordinates on FileScanTaskDeleteFile: #2868
Read and apply V3 deletion vectors: #3035
Validate deletion vector delete tasks: #3145
Make the delete_vector module public: #3461
DeleteVectorIterator::advance_to mishandles positions at the 2^32 boundary: #3400 (PR #3454)

Write

Serialize DVs to the deletion-vector-v1 blob layout: #3232
Deletion-vector writer and RowDelta action: #2580, #2203
Delete files in the snapshot producer: #2367
Commit-time rules from the spec: at most one DV per data file, merge existing position deletes into the new DV, and no new position delete files on v3 tables (reference implementation in iceberg-go: iceberg-go#1135, iceberg-go#1007)

Interop

Golden tests against Iceberg-Java-written DV blobs: no issue yet (reference implementation in iceberg-go: iceberg-go#1008, iceberg-go#1131)

Multi-arg transforms

source-ids on UnboundPartitionField: #3267
source-ids on bound PartitionField: #2801 (PR #2802)
source-ids on SortField: no issue yet
Parse unrecognized transform names as Transform::Unknown (the spec requires readers to accept unknown partition transforms): #2789 (PR #2790)
Eval support: parked until the spec or Java adds a concrete multi-arg transform (iceberg-go parks this for the same reason)

Metadata and format-version gating

V3 table metadata: #1682
V3 manifest writes the content field: #1979 (fixes #1973)
v2 to v3 upgrade with existing snapshots: #2010 (fixes #2009)
No -1 for an empty current-snapshot-id: the v2/v3 serializer omits the field when there is no current snapshot and reads -1 as none (reference implementation in iceberg-go: iceberg-go#1002)
v2 manifest Avro schema includes v3-only data file fields 142, 144 and 145: #3399 (PR #3452)
Reject field ids in the reserved metadata-column range: #3403 (PR #3451) (reference implementation in iceberg-go: iceberg-go#821)
Gate each transaction action by supported format version: #3041
Central per-type capability gating: #3258
REST create_table ignores the requested format version: #2923 (PR #3055)

Encryption

encryption-keys in table metadata and snapshot key-id: #1682
Client-side table encryption (umbrella): #2034. Merged so far: encrypted reads of manifests and Parquet data files, encrypted writes of manifests, manifest lists and Parquet data files, and encrypted Puffin read/write (#2584, #2586, #2677, #2701, #2821, #2822, #3018, #3236). Open: AWS KMS client #3173, resolving KMS client factory #3251, StaticTable #2915
Commits to v1/v2 tables with encryption.key-id write plaintext files: #3402

Unfiled leads

From the audit comment on this issue (2026-10-11), reproduced but not filed yet:

Manifest read fails on a variant lower_bounds / upper_bounds entry (spec/manifest/_serde.rs)
timestamptz / timestamptz_ns Avro schemas omit adjust-to-utc: true (limited by apache-avro, avro-rs#368)

Related, not v3 spec items

zstd metadata compression: #2894. The spec only mentions GZIP for metadata JSON ("A metadata JSON file may be compressed with GZIP").
Copy-on-write and merge-on-read: #2186
v3 spec types umbrella: #2581
Equality-delete column projection: #3369
Delete I/O metrics and unknown delete-file sizes: #3283, #3344 (PR #3433)
File-scoped position deletes: #3287 (PRs #3291, #3329)
v1 manifest entry with null snapshot_id: #3371

References

Iceberg spec v3 changes: https://iceberg.apache.org/spec/#version-3
iceberg-go v3 tracking: iceberg-go#589
pyiceberg v3 tracking: iceberg-python#1818
Java variant shredding: Spark: Support writing shredded variant in Iceberg-Spark iceberg#14297

Describe the solution you'd like

These would be grouped and owned by one or more contributors.

Joined the Iceberg - Variant Community Sync call and happy to take Deletion Vectors → read support . Already following it to have in datafusion-comet apache/datafusion-comet#3376

@nssalian

There's these issues we can track under this epic:

@nssalian can we add it to the main issue body?

Geometry/Geography support is now in progress in draft PR #2933, replacing #2653 after it was closed automatically by the stale bot.

feat(spec,arrow): variant Arrow value support (fills the PR 2188 stub)

#2792 deletion vector read epic.

Opened #2894 to implement the Support zstd metadata compression codec tracking item. The PR adds case-insensitive zstd configuration, canonical and alternate metadata suffix handling, content-based zstd/gzip detection, zstd writes, and interoperability/error-path coverage. This addresses that item without closing the broader v3 epic.

@nssalian

Just because I saw it, the Misc item Stop writing distinct_counts on v3 manifests, is not relevant.

There is no such writing in the rust implementation.
see: 6781ba9 that removed it on Dec 13, 2023 on a in-progress branch.

I think it is just here from copying the tasks from go ? @nssalian

feat: support zstd metadata compression

Hi, would you accept agent-written contributions to move iceberg v3 support forward? I'm working on a downstream project that uses iceberg-rust internally and would really benefit from having full v3 support.

I'm not a Rust expert myself but we have a large and complex data intelligence product with a data lakehouse and many use cases to verify against. I'd be testing and verifying any changes and contributions in a production environment with real data.

Perhaps by referencing the official spec and implementations in other languages (Go seems complete?) agents might be good enough to produce a suitable outcome?

@mfellner

Hi, would you accept agent-written contributions to move iceberg v3 support forward? I'm working on a downstream project that uses iceberg-rust internally and would really benefit from having full v3 support.

I'm not a Rust expert myself but we have a large and complex data intelligence product with a data lakehouse and many use cases to verify against. I'd be testing and verifying any changes and contributions in a production environment with real data.

Perhaps by referencing the official spec and implementations in other languages (Go seems complete?) agents might be good enough to produce a suitable outcome?

Hey @mfellner, thank you for offering. I think we broadly have implementations open for all of V3 support. The bottleneck is on reviewer capacity at the moment. Additional review and scrutiny is welcome!

I think there may be the odd bit detail that's not quite right at the moment. For example, I think I may have come across V3 fields being written into V1+V2 manifests. Identifying any gaps like that would be great to address.

Manu has been working on V3 Unknown, I have opened a tracking issue: #3373

@dannycjones

Thanks again for the pointers @dannycjones. I ran an agent-assisted audit of the v3 spec (1.12.0) against main and v0.11.0-rc.2. Findings I could reproduce are now separate issues, so each can be triaged on its own. Everything else below is from reading code and issue state on 2026-10-11, so please treat it as a lead list and correct me where I am wrong.

New issues (each has a test that fails on current main)

v2 manifest Avro schema includes v3-only data file fields 142, 144 and 145 #3399 v2 manifest Avro schema includes v3-only data file fields 142, 144 and 145 (the "V3 fields in V1+V2 manifests" case; schema-level only, v1 and manifest lists are clean)
Date-to-timestamp promoted column reads as NULL when another column is projected #3404 Date-to-timestamp promoted column reads as NULL when another column is projected
COW rewrite does not carry row lineage into replacement files on v3 tables #3401 COW rewrite does not carry row lineage into replacement files on v3 tables
Commits to v1/v2 tables with encryption.key-id write plaintext files #3402 Commits to v1/v2 tables with encryption.key-id write plaintext files
Table metadata accepts field ids above 2147483447, which the spec reserves for metadata columns #3403 Table metadata accepts field ids above 2147483447
DeleteVectorIterator::advance_to mishandles positions at the 2^32 boundary #3400 DeleteVectorIterator::advance_to mishandles positions at the 2^32 boundary (only reachable on data files with at least 2^32 rows)

I also left comments on #3373 (a required unknown field is accepted), #2418 (the current-schema case looks fixed on main; non-current schemas are the remaining gap) and #3041 (concrete cases for the version gate).

I have two more reproduced findings that I did not file, to avoid adding noise: manifest read fails on a variant lower_bounds/upper_bounds entry (spec/manifest/_serde.rs), and timestamptz/timestamptz_ns Avro schemas omit adjust-to-utc: true (limited by apache-avro, avro-rs#368). Happy to file either if you want them tracked.

Epic body items with no child issue or PR today. These are big enough that I did not want to open issues for work other people may already be doing; I can open child issues if the owners prefer.

Variant: non-shredded write, shredded read, shredded write, statistics (feat(arrow): read unshredded variant columns #2880 is the unshredded read; Varaint Type: Add logical type annotation for writer #2546 is the writer annotation)
Geometry/Geography: manifest bounds and bbox pruning (feat: recognize geometry and geography types [1/3] #2933 and feat: add geospatial parquet write support [3/3] #3342 defer them)
Multi-arg transforms: SortField source-ids (refactor(spec): make PartitionField fields private and carry source-ids #2802 mentions it as a follow-up) and eval
write-default: applying it on write (Support for write default and initial default #3250 covers validation; DataFusion INSERT fills omitted columns with NULL instead of the column's write-default #2803 refers to the DataFusion integration that has moved to apache/datafusion-iceberg)

Possible updates to the epic body, if useful: #2188 is merged; #1652 and #2879 look implemented by merged work (#1682, #3058); "Gate v3-only metadata fields when reading v1/v2" could point at #3399, #2418 and #3041; zstd (#2894) is not a v3 spec item; the "Stop writing distinct_counts" bullet is gone. Below is a checklist that links every v3 item I found, as of 2026-10-11. It is meant to be copied into the body if it helps.

Proposed checklist

Types

Variant type and Arrow recognition: feat: Variant Support #2188, feat(arrow): recognize arrow.parquet.variant on schema import #2840
Variant unshredded read: feat(arrow): read unshredded variant columns #2880
Variant writer annotation: Varaint Type: Add logical type annotation for writer #2546
Variant non-shredded write, shredded read/write, statistics: no issue yet
Unknown type: feat(spec): add unknown primitive type support #2773. [ ] Parquet omit/reconstruct: feat(parquet): omit unknown fields from writes #3072, feat(parquet): reconstruct unknown fields on reads #3073. [ ] Umbrella: Support for V3 Unknown data type #3373
Geometry/Geography: Support geo types #1884, feat: recognize geometry and geography types [1/3] #2933, feat: add geospatial read support [2/3] #3341, feat: add geospatial parquet write support [3/3] #3342. Bounds and pruning: no issue yet
Promoted date to timestamp(_ns): manifest bounds Manifest read fails on 4-byte bounds of timestamp and timestamp_ns columns promoted from date #3387; data read Date-to-timestamp promoted column reads as NULL when another column is projected #3404

Defaults

initial-default on read for top-level fields (Implement initial-default #737, feat(spec): read timestamp_ns/timestamptz_ns default values in the Literal JSON codec #2832)
Nested initial-default: RecordBatchTransformer does not apply initial-default for nested fields #3261
Filters on columns missing from the file: Rows are dropped when a filter references a column that is missing from the data file but has an identity partition value or initial-default #3298, A null identity partition value is replaced by initial-default when the partition column is missing from the data file #3301, fix(arrow): evaluate filters on columns missing from a data file against their identity partition value or initial-default #3302
JSON literal fixes: fix(spec): reject overflowing JSON integer literals #3330, fix(spec): reject non-finite decimal values #3331, fix(spec): accept null nested JSON literals #3332, fix(spec): reject float overflow in raw literals #3333 (fix(spec): reject boolean literals for non-boolean types #3334 merged)
write-default application and validation: Support for write default and initial default #3250, DataFusion INSERT fills omitted columns with NULL instead of the column's write-default #2803, feat: Support default values in UpdateSchema #1357

Row lineage

first_row_id inheritance and metadata columns: feat: inherit data-file first_row_id on manifest read #2922, feat: carry first_row_id and data sequence number on FileScanTask #2952, feat: synthesize _row_id metadata column on read #3058
Open housekeeping: iceberg v3 has to set first-row-id #1652, Support reading _row_id and _last_updated_sequence_number metadata columns #2879, _last_updated_sequence_number: physical per-row values discarded when first_row_id is null #2991
Lineage on COW rewrite: COW rewrite does not carry row lineage into replacement files on v3 tables #3401

Deletion vectors and deletes

Transforms

Unbound source-ids: refactor(spec): make UnboundPartitionField fields private and carry source-ids #3267. [ ] Bound PartitionField: refactor(spec): make PartitionField fields private and carry source-ids #2802, Reading v3 metadata with multi-argument transform fields (source-ids) fails #2801. [ ] SortField: no issue yet
Unknown transform names: Unrecognized transform names fail parsing instead of becoming Transform::Unknown #2789, Parse unrecognized transform names as Transform::Unknown #2790

Metadata and version gating

v2 manifest schema carries v3 fields: v2 manifest Avro schema includes v3-only data file fields 142, 144 and 145 #3399
Non-current schemas not checked per format version: No validation that prevents creating a V2 table with timestamp_ns/timestamptz_ns types #2418
Per-action version gating: Gate each transaction action by supported table format versions #3041
REST create-table format version: TableCreation::format_version is ignored when creating a table using the Iceberg REST catalog interface #2923, fix(rest): send the requested format version when creating a table #3055
Field ids above 2147483447: Table metadata accepts field ids above 2147483447, which the spec reserves for metadata columns #3403
zstd metadata (not in the spec): feat: support zstd metadata compression #2894
v1 manifest entry with null snapshot_id: Reading a V1 manifest entry with a null snapshot_id fails #3371

Encryption

Umbrella: Support client-side encryption #2034. [ ] KMS: feat(encryption/kms): Add AWS KMS client #3173. [ ] StaticTable: StaticTable does not support encryption today #2915. [ ] v1/v2 with encryption.key-id: Commits to v1/v2 tables with encryption.key-id write plaintext files #3402

Found during an agent-assisted audit of v3 support. I am not a Rust expert, so I would welcome review of the issues I filed. Happy to help where it is useful.

Updated the issue to capture the latest. Referenced the equivalent work in iceberg-go.

Navigation Menu

Repository navigation

EPIC: v3 Support Tracking #2411

New data types

Default values

Row lineage

Deletion vectors

Multi-arg transforms

Metadata and format-version gating

Encryption

Unfiled leads

Related, not v3 spec items

References

Describe the solution you'd like

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

EPIC: v3 Support Tracking #2411

Description

New data types

Default values

Row lineage

Deletion vectors

Multi-arg transforms

Metadata and format-version gating

Encryption

Unfiled leads

Related, not v3 spec items

References

Describe the solution you'd like

Activity

Shekharrajak commented on May 7, 2026

dannycjones commented on Jun 3, 2026

wirybeaver commented on Jun 16, 2026

mbutrovich commented on Jul 9, 2026

wirybeaver commented on Jul 25, 2026

Baunsgaard commented on Jul 27, 2026

mfellner commented on Oct 6, 2026

dannycjones commented on Oct 8, 2026

dannycjones commented on Oct 8, 2026

mfellner commented on Oct 11, 2026

nssalian commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions