You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ParquetWriter: opaque error when record batch schema is missing field ids #2654
When matching an Arrow record batch to an Iceberg schema by field id, ParquetWriter (via ArrowArrayAccessor::field_partner in crates/iceberg/src/arrow/value.rs) reads the PARQUET:field_id (PARQUET_FIELD_ID_META_KEY) metadata off each Arrow field. When a caller hands it a record batch whose Arrow schema was built by hand--without that metadata--the match fails with:
DataInvalid => Field id N not found in struct array
The message points at the symptom, not the cause: the Arrow field carries no field id at all. The right way to build the batch schema is to derive it from the table schema via table.metadata().current_schema().as_ref().try_into(), which stamps the field-id metadata--but nothing surfaces that, so it's an easy trap for downstream consumers.
Reported by @malon64 while testing #2185 from a downstream Rust ingestion tool.
Proposed fix
Enrich the error at its source in ArrowArrayAccessor::field_partner. In FieldMatchMode::Id, name the field and note the likely cause (a missing PARQUET:field_id), pointing at current_schema().as_ref().try_into(); FieldMatchMode::Name gets a matching by-name message.
Message-only change--the error fires in exactly the same cases as before, so there is no behavior change. (An earlier attempt validated the batch schema up front in the writer, but that wrongly rejected valid batches carrying extra non-Iceberg columns, e.g. the datafusion _partition column, so the source-error approach is the correct one.)
Willingness to contribute
I can contribute this fix independently--PR incoming.
Activity
fix(writer): fail fast when record batch schema is missing field ids
fix(writer): validate field ids in record batch schema
fix(writer): validate field ids in record batch schema
fix(arrow): clarify error when an Arrow field is missing its field id
glitchy commentedon Jun 16, 2026
original PR was too strict, breaking downstream consumers. closing.