Spark: vectorized ORC reader ignores row lineage for _row_id and _last_updated_sequence_number #18543

Apache Iceberg version

main (development)

Query engine

Spark

Please describe the bug 🐞

When reading an ORC file with the vectorized Spark reader, the _row_id metadata column is wrong. Every row gets the file's first_row_id instead of its own value.

This happens because VectorizedSparkOrcReaders.StructConverter checks idToConstant before it checks for metadata columns, so _row_id is treated as a plain constant. Values stored in the file are ignored, and for rows with no stored value the row position is not added. _last_updated_sequence_number has the same problem.

Example: a file with first_row_id = 100, where row 0 stores _row_id 555 and row 1 stores null. The vectorized reader returns 100 for both. It should return 555 and 101.

The row-based ORC reader does this correctly (OrcValueReaders, added in #15776 and #16534). The Parquet vectorized reader is also correct.

Found while adding vectorized coverage to the format model TCK in #17610. #17635 has an earlier attempt at a fix.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time