[Python][C++][Parquet] A damaged footer makes RowGroupMetaData.column() / ColumnChunkMetaData.statistics abort the interpreter (uncaught ParquetException -> std::terminate) #54957

Describe the bug, including details regarding any error messages, version, and platform.

When a Parquet footer is damaged, pq.ParquetFile(path).metadata.row_group(i).column(j) (and, for a different kind of damage, .statistics on the returned ColumnChunkMetaData) can kill the whole Python process with std::terminate. A C++ parquet::ParquetException escapes through the Cython wrapper, so Python never gets a chance to raise or catch anything: the process dies with SIGABRT (exit status 134 from a shell, -6 from subprocess), and try/except around the call is useless.

I found this by flipping or zeroing a few bytes inside the Thrift FileMetaData of ordinary files written by pyarrow.parquet.write_table: 178 of 45,000 mutants (0.4%) aborted the interpreter, and every one of them changed at most three bytes of the footer. The footer is not checksummed, so a torn write or a flipped bit on disk produces exactly this input, and a reader that only wants the footer statistics of many files (and expects OSError/ArrowInvalid for a bad one) cannot survive it.

Minimal reproduction (14 lines; one byte of the footer is changed):

import pyarrow as pa
import pyarrow.parquet as pq

pq.write_table(pa.table({"x": pa.array([1, 2, 3], pa.int64())}), "repro_clean.parquet")
data = bytearray(open("repro_clean.parquet", "rb").read())
# Footer, SchemaElement for "x": repetition_type (Thrift zigzag 2 = OPTIONAL), name "x".
i = data.index(b"\x25\x02\x18\x01x")
data[i + 1] = 0  # one byte: repetition_type OPTIONAL -> REQUIRED, now at odds with the column's size_statistics
open("repro_crash.parquet", "wb").write(data)

md = pq.ParquetFile("repro_crash.parquet").metadata  # fine: the footer parses
print("footer opened, calling row_group(0).column(0)", flush=True)
md.row_group(0).column(0)  # aborts the process: std::terminate, not a Python exception
print("not reached")

Output (macOS arm64, pyarrow 25.0.1; identical on 26.0.0):

footer opened, calling row_group(0).column(0)
libc++abi: terminating due to uncaught exception of type parquet::ParquetException: Definition level histogram size mismatch, size: 2, expected: 1

The shell reports exit status 134 (SIGABRT). I ran it three times and got the same result each time.

A second trigger with the same effect, on .statistics rather than .column(): change the physical type in the schema so that it disagrees with the ColumnMetaData of the chunk.

import pyarrow as pa
import pyarrow.parquet as pq

pq.write_table(pa.table({"x": pa.array([1.0, 2.0, 3.0], pa.float64())}), "repro_stats_clean.parquet")
data = bytearray(open("repro_stats_clean.parquet", "rb").read())
# Footer, SchemaElement for "x": type (Thrift zigzag 10 = DOUBLE), repetition_type, name "x".
i = data.index(b"\x15\x0a\x25\x02\x18\x01x")
data[i + 1] = 0  # one byte: physical type DOUBLE -> BOOLEAN, but the ColumnMetaData still says DOUBLE
open("repro_stats_crash.parquet", "wb").write(data)

col = pq.ParquetFile("repro_stats_crash.parquet").metadata.row_group(0).column(0)  # fine
print("column(0) ok, reading .statistics", flush=True)
col.statistics  # aborts the process

column(0) ok, reading .statistics
libc++abi: terminating due to uncaught exception of type parquet::ParquetException: ColumnMetaData type does not match ColumnDescriptor physical type: DOUBLE vs. BOOLEAN

In my fuzzing run, 65 of the 178 aborts were in .column() (definition/repetition level histogram size mismatches) and 113 were in .statistics (column type mismatches). Walking num_row_groups, row_group(i), num_columns, the file-level num_rows, and path_in_schema / has_min_max / min / max of a chunk that did construct never aborted. The fuzz harness is deterministic and I can attach it if useful.

Expected behaviour. A Python exception, not a process abort. The same validation already surfaces as a regular exception on the read path: for the file produced by the first snippet, pq.ParquetFile("repro_crash.parquet").read_row_group(0) (and .read()) raises

OSError: Definition level histogram size mismatch, size: 2, expected: 1

so OSError (or ArrowInvalid) from column() / .statistics would be consistent with the rest of the Python API.

What I think is going on (from the pyarrow wheel; I have not built from source). pyarrow/includes/libparquet.pxd declares CRowGroupMetaData.ColumnChunk(int i) and CColumnChunkMetaData.statistics() inside cdef extern from "parquet/api/reader.h" ... nogil: without except +. RowGroupMetaData.column() reaches parent.metadata.ColumnChunk(i) through ColumnChunkMetaData.init (_parquet.pxd), and ColumnChunkMetaData.statistics calls self.metadata.statistics(). Both appear to run C++ that validates the chunk metadata against the schema descriptor and throws ParquetException when they disagree, and with no translation at the Cython boundary the exception unwinds into the interpreter and ends in std::terminate. RowGroupInfo.statistics in _dataset_parquet.pyx calls self.metadata.column(i) and col.statistics and aborts the same way, so ds.dataset(path, format="parquet").get_fragments() -> fragment.row_groups[i].statistics is another way to reach it. The other metadata accessors I tried do not abort, but I have not audited the rest of the file's extern declarations for the same missing except +; other accessors that build per-column metadata may be affected as well.

Versions.

• The .column() abort (first snippet): aborts on pyarrow 20.0.0, 21.0.0, 22.0.0, 23.0.1, 24.0.0, 25.0.1 and 26.0.0 (26.0.0 is the latest on PyPI today). 15.0.2, 17.0.0, 18.1.0 and 19.0.1 do not abort (on 19.0.1 column() returns normally for this file; read_row_group(0) there fails with OSError: Unexpected end of stream).
• The .statistics abort (second snippet): aborts on 21.0.0, 22.0.0, 23.0.1, 24.0.0, 25.0.1 and 26.0.0. 15.0.2, 17.0.0, 18.1.0, 19.0.1 and 20.0.0 do not abort.
• Platform: macOS 27.0.1 (Darwin 27.0.0), Apple M4 (arm64), Python 3.14.8 for 25.0.1 and 26.0.0 and Python 3.12.13 for the older versions; wheels from PyPI. I have not tried Linux. I would expect the message there to read terminate called after throwing an instance of 'parquet::ParquetException'.

I searched the tracker for this and found nothing that matches. The closest is GH-45283 ("OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0), which is the same validation reported as a regular OSError on the read path.

Component(s)

Parquet, Python