feat(parquet): create a subset of column writers with per-row-group properties - #10917
Open
adriangb wants to merge 2 commits into
Open
feat(parquet): create a subset of column writers with per-row-group properties#10917adriangb wants to merge 2 commits into
adriangb wants to merge 2 commits into
Conversation
…roperties `ArrowRowGroupWriterFactory` can only build column writers for every leaf column of a row group, at the properties the file writer was created with. This adds two accessors: * `create_column_writer(row_group_index, column_index, props)` builds the writer for one leaf column with the given `WriterProperties`. `column_index` indexes the parquet leaf columns in schema order, the order of `SchemaDescriptor::columns()` and of the `ArrowLeafColumn`s that `compute_leaves` yields, not the Arrow schema's top level fields; an index past the last leaf column is an error naming the valid range. The writer shares the factory's page store factory and, with the `encryption` feature, its file encryptor, so the chunk it produces is appendable to the same file writer. * `page_store_factory()` returns the configured `PageStoreFactory`, so a caller that builds some of a row group's column chunks by another route can buffer them through the same page store. Both this accessor and `create_column_writers` go through one private construction path, of which they are the two degenerate cases: one column, or every column at the file writer's own properties. The default write path is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MwnUAoPDMQaYaQPVP2iUcz
adriangb
force-pushed
the
chunk-probe-writers
branch
from
August 30, 2026 17:07
554c930 to
8554400
Compare
adriangb
commented
Aug 30, 2026
Comment on lines
+35
to
+41
| //! The cost model is deliberately simple. A column that is still deciding | ||
| //! encodes its probe prefix K + 1 times: K throwaway passes plus the real one. | ||
| //! The prefix is one data page worth of rows rather than a whole row group, so | ||
| //! the extra work is bounded by the page size and not by the data. A column | ||
| //! that has settled costs nothing extra at all: no probe writer is built for | ||
| //! it, so no page store is allocated for it either. That is what addressing one | ||
| //! column at a time buys over building every column writer at once. |
Contributor
Author
There was a problem hiding this comment.
Can we clarify how this interacts w/ dictionary pages, and in general how this adaptive writer handles "is the dictionary encoding earning it's keep"? How does it measure "is this dictionary worth it"? Does it fall back to avoid dictionary pages being larger than memory (there can only be 1)?
adriangb
force-pushed
the
chunk-probe-writers
branch
2 times, most recently
from
August 30, 2026 17:47
af4dcc2 to
c5c3e72
Compare
adriangb
commented
Aug 31, 2026
adriangb
left a comment
adriangb
left a comment
Contributor
Author
There was a problem hiding this comment.
some suggestions for the example
adriangb
force-pushed
the
chunk-probe-writers
branch
2 times, most recently
from
August 31, 2026 03:33
b060d08 to
0a5ba75
Compare
An adaptive writer built on `create_column_writer`: for each row group, each column that is due to race encodes a probe of its leading rows through several throwaway single-column writers, one per candidate property set (dictionary, delta where the physical type has one, and plain). The compressed sizes on the returned `ColumnCloseResult`s pick a winner, the probe chunks are dropped, and the row group is then written through one ordinary column writer per column at its current choice. Two small mechanisms make the measurement pay for itself only where it is worth it. `DECISIVE_MARGIN` is the fraction by which the leading chunk must beat the runner-up for the race to count as decided. Closer than `NEAR_TIE_BAND`, the sizes are not telling the candidates apart at all, and the tie is broken toward the page a reader decodes most cheaply, dictionary and plain ahead of the delta encodings, rather than toward a nominally smaller chunk. The band is a separate and much narrower constant than `DECISIVE_MARGIN` for a measured reason: at 10 percent it gives away real wins, since delta beats plain or dictionary by 5 to 10 percent on many TPC-H columns, and at 2 percent the tie-break is free on that data. The tie-break is deterministic and not random, because the example promises that the same input produces the same file. The interval before a column races again scales with the margin it won by, doubling after each decisive race up to `MAX_RERACE_INTERVAL` row groups and dropping back to one row group after a race the leader did not win decisively. A settled column then costs nothing at all: no probe writer is built for it, so no page store is allocated for it either. The probe itself stays one data page long. Growing it a page at a time until the leader is decisive was tried and measured: over TPC-H and ClickBench it bought under a point of compression for 10 to 30 percent more write CPU, since a chunk's compressed size can only be read once the chunk is closed, so each extra page re-encodes the ones before it. The module documentation records that result and leaves adaptive probe length to callers whose data rewards it. The example generates a deterministic dataset whose best encoding differs per column, writes it both with the probe writer and with a stock `ArrowWriter` at identical properties and row group boundaries, verifies that both read back exactly, and prints the total bytes, elapsed time and peak allocation of each arm alongside the encodings each column ended on. The two measured numbers are there because the technique trades work for size, so the report should show both sides of that trade. Neither needs a dependency: elapsed time is `std::time::Instant`, standing in for CPU time, which the standard library cannot read, and peak allocation comes from a short counting `#[global_allocator]` in the example itself, sampled as each arm's increment over the bytes live when it started so that the input batches do not drown the signal. Both arms are measured after the input is materialized and each runs once, so the byte totals are deterministic while the time and memory figures are indicative. The module documentation also spells out how a dictionary candidate is handled, since that is the case where a measured comparison differs most from a guess: a probe's compressed size covers the dictionary page as well as the data pages indexing into it, so a dictionary is charged its full cost and wins only by being smallest; and a dictionary that a short probe flattered is corrected by the writer's own single-dictionary-page fallback and by the next re-race. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MwnUAoPDMQaYaQPVP2iUcz
adriangb
force-pushed
the
chunk-probe-writers
branch
from
August 31, 2026 04:08
0a5ba75 to
b99c55f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured results
Externally measured with
/usr/bin/time -v, release build, ZSTD, five interleaved runs per arm per file, reported net of a read-only baseline arm that stops after loading the input. CPU is user+sys as median ± half the observed min-max range (five runs, not a standard deviation); peak memory is maximum resident set size; MB is 10^6 bytes.The probe writer costs less CPU than the stock writer on every file, because racing a prefix of rows per candidate is cheaper than the dictionary builds the stock writer starts and then abandons. It costs more peak memory on the wider schemas, where it holds a throwaway writer per candidate alongside the committed ones while a column is being probed; a caller that probes columns sequentially rather than all at once bounds that to one column's candidates. Every output file was read back and compared row for row against the input, and the synthetic timestamp and f64 results from the included example (-84.8% and -20.5%) reproduce deterministically.