You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Accept identical repeated named type definitions when reading (like Java Avro 1.12) #708
Some Avro writers repeat the full definition of a named type instead of referring to it by name after the first use. For example, PyIceberg and iceberg-rust 0.21 do this for two Iceberg partition fields that are both decimal(10,2):
The spec doesn't allow this, but these files already exist and won't go away until someone rewrites them. In 0.22:
let schema = Schema::parse_str(REPEATED)?;// OkResolvedSchema::try_from(&schema)?;// Err: Two named schema defined for same fullname: decimal_10_2.
Reader::new on a file with this schema in its header fails with the same error, unless a reader schema is passed.
What Java does
Java Avro 1.12 accepts a repeat if it's identical to the first definition, and only rejects one that differs (ParseContext.put):
if (alreadyKnownSchema != null) {
if (!schema.equals(alreadyKnownSchema)) thrownewSchemaParseException("Can't redefine: " + fullName);
}
Proposal
During name resolution, treat a repeated definition that's equal to the first one as a reference, and keep the error when it differs. That's lenient on read, like Java, and a repeat that differs is still caught as genuinely ambiguous.
Why
iceberg-rust upgraded to 0.22 in apache/iceberg-rust#3354. To keep reading existing manifests, we had to add a workaround that rewrites the schema JSON in the file header (repeats become references) and reads the file again (named_types.rs). With this change, plus #575 for the other custom code, we could move all of it into avro-rs. More background in apache/iceberg-rust#3466.
if a schema uses the same named type twice, it'd be great if avro-rs wrote later uses as a reference, like Java's writeNameRef. canonical_form() already does this, but Serialize doesn't, and Writer::new rejects the schema:
schema.canonical_form();// ✅ {"name":"b","type":"decimal_10_2"}
serde_json::to_string(&schema)?;// ❌ repeats the full definitionWriter::new(&schema,Vec::new());// ❌ Two named schema defined for same fullname
with both sides, iceberg-rust can drop its custom code for this too (define_once).
Activity
kevinjqliu commentedon Oct 15, 2026
same issue on the write side, adding it here 😄
if a schema uses the same named type twice, it'd be great if avro-rs wrote later uses as a reference, like Java's
writeNameRef.canonical_form()already does this, butSerializedoesn't, andWriter::newrejects the schema:with both sides, iceberg-rust can drop its custom code for this too (
define_once).