[tobias-fire]

Background

A column chunk whose dictionary grows past dictionary_pagesize_limit continues in PLAIN, hard-coded (FallbackToPlainEncoding), and a column's configured encoding only applies with dictionary encoding disabled. parquet-rs instead continues in the column's encoding, so a writer gets dictionary pages where a dictionary pays off and a better encoding than PLAIN where it does not, without knowing a column's cardinality up front. PackDB's native Parquet reader decodes DELTA_LENGTH_BYTE_ARRAY 1–6× faster than PLAIN (firebolt-analytics/packdb#27086), so that is what its tablet writer wants to fall back to.

Summary

WriterProperties::Builder::enable_dictionary_fallback_encoding(): with it, the fallback is the column's encoding. Only PLAIN and DELTA_LENGTH_BYTE_ARRAY are accepted there, the latter for BYTE_ARRAY columns; a column writer for any other combination throws when it opens rather than mid-write. The delta encodings are format 2.0 encodings, so they are written to 2.x files with V2 data pages only, as RLE for booleans is (ColumnWriter::Make); a 1.0 file or V1 pages keep PLAIN. Off by default: nothing changes without the option. UNKNOWN, the column default that ColumnWriter::Make resolves to PLAIN, resolves to PLAIN here too.

Test Plan

• column_writer_test.cc: DictionaryFallbackToDeltaLengthByteArray writes a BYTE_ARRAY column that outgrows its dictionary with the option on and reads every row back; the chunk's data pages are RLE_DICTIONARY and then DELTA_LENGTH_BYTE_ARRAY, with PLAIN only on the dictionary page. DictionaryFallbackEncodingNeedsVersion2AndV2Pages keeps PLAIN for 1.0 files and for V1 pages. DictionaryFallbackEncodingIsValidated and DeltaLengthByteArrayIsNoDictionaryFallbackForInt32 reject DELTA_BYTE_ARRAY and a DELTA_LENGTH_BYTE_ARRAY fallback on an INT32 column at writer creation. The existing DictionaryFallbackVersion* tests cover the default (option off).
• These tests were written against the fixture but not run here: PackDB builds libparquet from this tree without Arrow's test targets. The change itself is exercised end to end by PackDB's ParquetFormatTest.AColumnThatOutgrowsItsDictionaryContinuesInDeltaLengthByteArray (firebolt-analytics/packdb PR to follow), which writes a tablet through this code and checks the chunk's page encodings and the read-back; the 682 Parquet and ManagedV2 tests pass with this tree.

🤖 Generated with Claude Code

https://claude.ai/code/session_011hhJuKWxLQsA1Mgyh4Ni9d

---

Note

Medium Risk
Opt-in writer behavior change affects on-disk page encodings when dictionaries overflow; misconfiguration is caught at writer open, but enabled paths must be readable by consumers expecting PLAIN fallback.

Overview
Adds an opt-in WriterProperties::Builder::enable_dictionary_fallback_encoding() so that when a column’s dictionary hits dictionary_pagesize_limit, later data pages use the column’s configured encoding (parquet-rs-style) instead of always PLAIN. Default behavior is unchanged.

DictionaryFallbackEncoding picks the fallback at writer construction: PLAIN unless the option is on, dictionary encoding is used, the file is 2.x with V2 data pages, and the column encoding is PLAIN or DELTA_LENGTH_BYTE_ARRAY (BYTE_ARRAY only). Other encodings or type mismatches throw ParquetException at open time. FallbackToNonDictionaryEncoding (renamed from plain-only fallback) switches encoders to that stored fallback_encoding_ when the dictionary limit is hit or dictionary writes must stop.

Tests cover DELTA_LENGTH_BYTE_ARRAY fallback after dictionary overflow, version/page gating, and validation errors.

^{Reviewed by Cursor Bugbot for commit a71dc73. Bugbot is set up for automated code reviews on this repo. Configure here.}