Repository navigation
Reject v3-only features on pre-v3 tables #4156
Description
Feature Request / Improvement
Several features are only valid in format v3 (spec, Appendix E), e.g. the new types (variant, geometry, geography, unknown, timestamp_ns, timestamptz_ns), non-null initial-default, encryption (encryption.key-id, encryption-keys, snapshot key-id), deletion vectors, and row lineage.
PyIceberg handles these inconsistently on pre-v3 tables today. Some checks only run at create_table, some features are silently dropped, and others are accepted. We should follow Java here, but Java doesn't cover every case either, so we need to go further.
Rules
- Reject v3 features on pre-v3 tables. Creating a pre-v3 table, or changing its metadata (schema, properties, partition specs, sort orders), must raise if the change introduces a v3-only feature. Never silently accept, drop, or ignore it.
- Allow reads and repairs. Loading and scanning a pre-v3 table that already contains v3-only features must work. Metadata changes that only remove v3-only features (e.g.
remove_properties("encryption.key-id"), dropping a column) must succeed, so the table can be repaired from invalid to valid. - Check the version on the write side. Writers must check the format version before writing v3-only structures (e.g. deletion vectors, encrypted manifest lists, row lineage fields), and raise instead of falling back to a pre-v3 equivalent.
- Reject everything else on an invalid table. If a pre-v3 table contains v3-only features, reject all other writes until it is repaired: data writes (append, overwrite, delete), metadata updates other than repairs, and format version upgrades. Upgrading must not turn an invalid table into a valid one.
Java coverage
| Rule | Java |
|---|---|
| 1. Reject v3 features | Types and initial-default are checked on every schema change (Schema.checkCompatibility). Encryption properties are checked only at table creation (EncryptionUtil.checkCompatibility), not on property updates |
| 2. Reads and repairs | Allowed, since metadata isn't validated on load |
| 3. Write-side checks | Deletion vectors are checked on commit (validateDeleteFileForVersion). Manifest list encryption is not gated by version |
| 4. Reject on invalid tables | Not covered: upgrades and other writes are not blocked |
Note: Java doesn't fully handle this yet either. Where PyIceberg goes further than Java (rules 1, 3 and 4), we should raise the gaps on the dev list so implementations stay consistent.
Activity
Ankitraj-builds commentedon Oct 26, 2026
I opened #4160 for the first piece of rule 1: schema changes on v1/v2 tables now raise for v3-only types (timestamp_ns, unknown, etc.) and for a non-null initial-default. Before, the format version check only ran in create_table, so update_schema().add_column(...) accepted these.
Dropping a v3-only column still works, so a table can be repaired. Happy to take the next pieces (properties like encryption.key-id, then the write-side checks) as separate PRs if that works for you.
karmendra8386 commentedon Nov 2, 2026
Hi @kevinjqliu, I'd like to pick up the property-updates piece of rule 1 if it's still free — reject setting encryption.key-id / encryption-keys on pre-v3 tables in update_properties, while still allowing their removal so the repair path in rule 2 keeps working. I'd follow the pattern @Ankitraj-builds set in #4160 for the schema side, with tests alongside theirs.
Since this touches behavior beyond what Java checks today, I'm also happy to flag it on the dev list if you'd like that first. Does this slice sound right before I start?