Manifest Avro schema repeats the decimal type definition when partition fields share a decimal type #3466

TL;DR: when two or more partition fields have the same decimal(P,S) type, iceberg-rust ≤ 0.11 writes the full Avro definition of that type once per field in the manifest schema. The Avro spec only allows one definition per type, so strict Avro readers reject these manifests. Older iceberg-rust also can't read the manifests Java writes in this case.

#3354 fixes this on main (it isn't in v0.11.0-rc.2, so it isn't released yet):

• write (strict): follows the Avro spec. Define the type once, and refer to it by name for the other fields.
• read (lenient): also reads manifests that repeat the definition, like the ones older iceberg-rust and PyIceberg wrote.

What's the issue?

Iceberg stores decimal(P,S) as an Avro fixed type, which needs a name, like decimal_10_2. The Avro spec says a name can only be defined once per schema.

This matters for manifests. Each manifest's Avro schema includes the table's partition spec as a record, with one field per partition field. If two partition fields have the same decimal type, for example identity(a), identity(b) where a and b are both decimal(10,2), then decimal_10_2 shows up twice in the schema.

There are two ways to write the second use:

Reference form ✅ follows the spec. Define it once, then refer to it by name:

{"name": "a", "type": ["null", {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2}]},
{"name": "b", "type": ["null", "decimal_10_2"]}

Repeated form ❌ doesn't follow the spec. Write out the full definition every time:

{"name": "a", "type": ["null", {"type": "fixed", "name": "decimal_10_2", ...}]},
{"name": "b", "type": ["null", {"type": "fixed", "name": "decimal_10_2", ...}]}

The data is the same in both; only the schema in the file header is different. A strict reader follows the Avro spec and only accepts the reference form; a lenient reader accepts both.

What happened in iceberg-rust, and how #3354 fixed it

Before #3354, schema_to_avro_schema built a new decimal definition for every field, so iceberg-rust wrote the repeated form. This affects all releases through v0.10.1 and v0.11.0-rc.2.

apache-avro 0.21 doesn't keep track of decimal names. So it never noticed the repeat, but it also can't resolve a reference to a decimal, which means older iceberg-rust can't read Java's manifests in this case.

#3354 upgraded to apache-avro 0.22, which is strict: it reads the reference form, and Reader::new rejects the repeated form (AmbiguousSchemaDefinition). On its own, the upgrade would have broken reading manifests that already exist in the repeated form, so #3354 also changed both sides:

• Write: define_once in avro/schema.rs keeps track of the names it has already defined, and writes a reference for every later use.
• Read: apache-avro has no lenient option, so we added custom code. avro/named_types.rs catches the error, rewrites identical repeated definitions in the file header into references, and reads the file again. Repeats that differ still fail.

How does this line up with Java and PyIceberg?

The goal is Postel's law: be lenient in what we read, and strict in what we write.

Java works the same as iceberg-rust main:

• ✅ Reads the reference form. Avro's parser resolves references as it goes.
• ✅ Reads the repeated form. Avro's parser accepts a repeat if it's identical to the first definition.
• ✅ Writes the reference form. TypeToSchema names the type decimal_P_S, and Avro's writeNameRef writes every later use as a reference.

PyIceberg is the odd one out:

• ✅ Reads the repeated form. AvroSchemaConversion doesn't track names, so a repeated definition is just parsed again.
• ❌ Can't read the reference form. AvroSchemaConversion treats "decimal_10_2" as an unknown primitive type, so it fails on manifests from Java and iceberg-rust main. That includes iceberg-rust's manifests since perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value #3354, which PyIceberg could read before.
• ❌ Writes the repeated form. iceberg_to_avro writes the full definition every time, so strict readers like apache-avro 0.22 reject PyIceberg's manifests.

Repro for the read failure:

from pyiceberg.utils.schema_conversion import AvroSchemaConversion

dec = {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2}
AvroSchemaConversion().avro_to_iceberg({"type": "record", "name": "r102", "fields": [
    {"name": "a", "type": ["null", dec], "field-id": 1000, "default": None},
    {"name": "b", "type": ["null", "decimal_10_2"], "field-id": 1001, "default": None}]})
# TypeError: Type not recognized: decimal_10_2

Anything else?

A few follow-ups:

• avro-rs: move the custom Avro code from perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value #3354 into avro-rs. Two avro-rs issues get us there:
  • Accept identical repeated named type definitions when reading (like Java Avro 1.12) avro-rs#708: accept identical repeated definitions on read, like Java Avro 1.12. Lets us drop avro/named_types.rs.
  • Feature request: SchemaAwareResolvingDeserializer — direct-to-type schema evolution without the intermediate Value avro-rs#575: deserialize directly into a type while resolving against a reader schema. Lets us drop Resolved<T> in avro/deserializer.rs and RawLiteral::project_by_name.
• PyIceberg: read the reference form and write it too, so PyIceberg can read manifests from Java and iceberg-rust, and strict readers can read PyIceberg's. Tracked in Avro: can't read named type references, and writes repeated definitions (e.g. multiple decimal partition fields) iceberg-python#4116.