Skip to content

Accept identical repeated named type definitions when reading (like Java Avro 1.12) #708

Open

Description

@kevinjqliu

Problem

Some Avro writers repeat the full definition of a named type instead of referring to it by name after the first use. For example, PyIceberg and iceberg-rust 0.21 do this for two Iceberg partition fields that are both decimal(10,2):

{"type": "record", "name": "r", "fields": [
  {"name": "a", "type": {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2}},
  {"name": "b", "type": {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2}}
]}

The spec doesn't allow this, but these files already exist and won't go away until someone rewrites them. In 0.22:

let schema = Schema::parse_str(REPEATED)?;  // Ok
ResolvedSchema::try_from(&schema)?;         // Err: Two named schema defined for same fullname: decimal_10_2.

Reader::new on a file with this schema in its header fails with the same error, unless a reader schema is passed.

What Java does

Java Avro 1.12 accepts a repeat if it's identical to the first definition, and only rejects one that differs (ParseContext.put):

if (alreadyKnownSchema != null) {
  if (!schema.equals(alreadyKnownSchema)) throw new SchemaParseException("Can't redefine: " + fullName);
}

Proposal

During name resolution, treat a repeated definition that's equal to the first one as a reference, and keep the error when it differs. That's lenient on read, like Java, and a repeat that differs is still caught as genuinely ambiguous.

Why

iceberg-rust upgraded to 0.22 in apache/iceberg-rust#3354. To keep reading existing manifests, we had to add a workaround that rewrites the schema JSON in the file header (repeats become references) and reads the file again (named_types.rs). With this change, plus #575 for the other custom code, we could move all of it into avro-rs. More background in apache/iceberg-rust#3466.

Activity

  1. kevinjqliu commented on Oct 15, 2026

    @kevinjqliu
    Author

    same issue on the write side, adding it here 😄

    if a schema uses the same named type twice, it'd be great if avro-rs wrote later uses as a reference, like Java's writeNameRef. canonical_form() already does this, but Serialize doesn't, and Writer::new rejects the schema:

    schema.canonical_form();          // ✅ {"name":"b","type":"decimal_10_2"}
    serde_json::to_string(&schema)?;  // ❌ repeats the full definition
    Writer::new(&schema, Vec::new()); // ❌ Two named schema defined for same fullname

    with both sides, iceberg-rust can drop its custom code for this too (define_once).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions