[Parquet] Populate bloom filters from the dictionary while a column is dictionary encoded - #10966
Open
ranflarion wants to merge 1 commit into
Open
[Parquet] Populate bloom filters from the dictionary while a column is dictionary encoded#10966ranflarion wants to merge 1 commit into
ranflarion wants to merge 1 commit into
Conversation
Contributor
|
run benchmark arrow_writer env:
BENCH_FILTER: bloom |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing bloom-filter-defer-dictionary-values (345ae93) to c134baf (merge-base) diff Run configurationrun benchmark arrow_writer
env:
BENCH_FILTER: "bloom"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing bloom-filter-defer-dictionary-values (345ae93) to c134baf (merge-base) diff Run configurationrun benchmark arrow_writer
env:
BENCH_FILTER: "bloom"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
|
run benchmark arrow_writer env:
BENCH_FILTER: bloom |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing bloom-filter-defer-dictionary-values (345ae93) to c134baf (merge-base) diff Run configurationrun benchmark arrow_writer
env:
BENCH_FILTER: "bloom"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing bloom-filter-defer-dictionary-values (345ae93) to c134baf (merge-base) diff Run configurationrun benchmark arrow_writer
env:
BENCH_FILTER: "bloom"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
|
run benchmark arrow_writer env:
BENCH_FILTER: bloom |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing bloom-filter-defer-dictionary-values (345ae93) to c134baf (merge-base) diff Run configurationrun benchmark arrow_writer
env:
BENCH_FILTER: "bloom"BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing bloom-filter-defer-dictionary-values (345ae93) to c134baf (merge-base) diff Run configurationrun benchmark arrow_writer
env:
BENCH_FILTER: "bloom"CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
|
I've run the benches on my workstation, and the low cardinality dict tests have been consistently better. All others are mixed from run to run, but no consistent regressions. |
etseidl
approved these changes
Sep 4, 2026
etseidl
left a comment
etseidl
left a comment
Contributor
There was a problem hiding this comment.
The changes look good to me and seem well tested. Thanks @ranflarion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
While a column is dictionary encoded every row was still hashed into the bloom filter, although the interner already holds the distinct values. For a low-cardinality column that is N inserts where D carry the same information, and those are exactly the columns that stay dictionary encoded. Suggested by @etseidl in #10963.
What changes are included in this PR?
ColumnValueEncoderImpl::write_sliceand the byte arrayencodeinsert into the filter only when no dictionary encoder is active.flush_dict_pageon both encoders inserts every interned value before handing the dictionary page over. It runs on fallback and at chunk close, so after a fallback the filter holds the dictionary's values plus every value written plain afterwards.DictEncoder::uniquesexposes the interned values for the primitive path.The set of values inserted is unchanged and folding decides from the final fill rate, so the serialized filter is byte-identical; only the write-side cost changes.
Benchmark (
cargo bench -p parquet --bench arrow_writer -- '<batch>/bloom_filter', Apple M-series, criterion, main vs this branch):Are these changes tested?
Yes. New round-trip tests for
StringArrayandInt64Arraycover a chunk that stays dictionary encoded (asserted through the page encoding mask) and a chunk that falls back to plain after a small dictionary page limit, checking the filter for every written value and for absent ones. The existing bloom filter round-trip tests already sweep dictionary disabled, immediate fallback and dictionary enabled and pass unchanged.Are there any user-facing changes?
No API or output change.
DictEncoder::uniquesis new but the type is not exported from the crate.