fix(parquet): re-encode buffered values on dictionary fallback when the dictionary would be unreferenced

[adriangb]

Closes #9739.

When the dictionary overflows its size limit, the writer currently flushes the buffered dictionary ids as one more dictionary-encoded page and writes the dictionary page. If no data page has been sealed yet, that dictionary page exists solely to serve the values still buffered, and the chunk pays for those values twice: once in the dictionary, once in the ids that index it.

This changes the fallback to re-encode the buffered ids through the fallback encoder and drop the dictionary, but only in that case. Once a dictionary-encoded data page has been sealed, the dictionary page has to be written whatever happens next and every value it covers is already paid for, so re-encoding would pay for them a second time. Measurements agree: re-encoding unconditionally makes common files whose dictionary overflows after several sealed pages 3 to 15 percent larger, while the narrow case is where the saving is, with a TPC-H orders column dropping a roughly 1 MB dictionary whose only reference was the re-encoded buffer, for a 0.8 percent smaller file.

While dictionary encoding is active a data page holds small RLE indices, so it can buffer far more bytes worth of values than the data page size limit before the limit is consulted, and re-encoding that buffer in one go could produce a single page many times the limit. The re-encode drains the buffer in increments sized from the bytes the previous increment produced, sealing a page whenever the limit is reached. Sealing a page part-way through means splitting the levels already buffered for it, which the streaming level encoders cannot do, so this applies to columns with no repetition or definition levels; other columns re-encode into a single page as before.

The dual Option<DictEncoder> bookkeeping is gone. A single DictionaryState enum holds the three states a chunk can be in — encoding, draining the ids buffered at the fallback point, or without a dictionary — so the encoder cannot be consulted as an encoder once it is only a source of values to re-encode.

[etseidl]

I need some time to mull this over. One issue as I see it is that the fallback here always re-encodes the buffered values. But I see many instances where the dictionary is effective, but too many keys accrete within a large column chunk, so fallback occurs anyway. The current page would still benefit from the dictionary, but with this change that partial page will be re-encoded with PLAIN. Perhaps this re-encode should only occur if it's the first page in the chunk (i.e. retain_dictionary is false).

I'll also admit I find the presence of two Option<DictEncoder>s a bit confusing. Maybe a boolean to indicate if fallback has occurred would be better?

[cakeni]

Could this let fallback bypass the data-page size limit? fall_back_from_dictionary re-encodes the entire buffered dictionary-id buffer before should_add_data_page() runs. Since add_data_page() then flushes the buffer as a whole, it can’t split it back into page-sized chunks.
With 1 KiB dictionary/data page limits and write_batch_size = 1, I get a 41,612-byte PLAIN page with this PR, while the parent keeps pages under 4 KiB. Should we re-encode the buffered ids incrementally during fallback so the page limit is still enforced?

[adriangb]

I'm going to close this as it's not a clear win and I don't want to consume more reviewer time on it without clear evidence of a global improvement in behavior.

[etseidl]

Thanks @adriangb