Repository navigation
perf: preserve dictionary encoding through regexp_replace - #26186
Draft
Rich-T-kid wants to merge 2 commits into
Draft
Rich-T-kid wants to merge 2 commits into
Rich-T-kid wants to merge 2 commits into
Conversation
github-actions
Bot
added
sqllogictest
Rich-T-kid
force-pushed
the
rich-T-kid/regexp-replace-dict-encoding
branch
from
October 10, 2026 17:41
70cbf41 to
62e3ba4
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26186 +/- ##
==========================================
+ Coverage 82.79% 82.89% +0.10%
==========================================
Files 1149 1154 +5
Lines 451645 458074 +6429
Branches 451645 458074 +6429
==========================================
+ Hits 373940 379731 +5791
- Misses 54929 55298 +369
- Partials 22776 23045 +269 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Keep Dictionary(_, Utf8/LargeUtf8/Utf8View) through regexp_replace instead of casting it to the inner string type. When the pattern, replacement, and flags args are all scalar the function runs on the dict values and re-attaches the keys; otherwise it falls back to a flat value-type output.
Rich-T-kid
force-pushed
the
rich-T-kid/regexp-replace-dict-encoding
branch
from
October 12, 2026 03:33
62e3ba4 to
7a2ee99
Compare
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned wide_schema |
Contributor
Author
|
run benchmark clickbench_partitioned |
Rich-T-kid
marked this pull request as ready for review
October 12, 2026 03:52
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark wide_schema
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
forgot to flip the flag , re-running the last one |
Contributor
Author
|
run benchmark clickbench_partitioned |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
TODO fix the benchmark run. the baseline needs to be latest main with the flag enabled and the changed needs to be this branch with the flag enabled |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark wide_schema
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagewide_schema — base (merge-base)
wide_schema — branch
File an issue against this benchmark runner |
Contributor
Author
|
actually no thats exactly what this does #26186 (comment), this is the exact query this PR targets https://github.com/apache/datafusion/blob/main/benchmarks/queries/clickbench/queries/q28.sql |
Contributor
Author
|
@kumarUjjawal would you mind taking a look at this as its related to #24111 this comment also provides some context #26186 (comment) |
Contributor
Thank you @Rich-T-kid for working on this. I'm currently on vacation. I will get to it when I return. |
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
jayzhan211
reviewed
Oct 24, 2026
| && let DataType::Dictionary(_, value_type) = array.data_type().clone() | ||
| { | ||
| if preserve_dictionary { | ||
| let dict = array.as_any_dictionary(); |
Contributor
There was a problem hiding this comment.
This runs the regex over every value in the dictionary, including values no row references. After a filter, or with a Parquet dictionary shared across batches, that can be much larger than the batch. In a same-build microbench (URL-shaped values, the Q28 pattern) I measured 5.3x slower than the old cast path at 50k values / 8192 rows, and 34x slower at 50k values / 1000 rows.
Compacting the dictionary first fixes it. With this change every case I tried was ≤1.01x the old path, and the small-dictionary win stays:
if preserve_dictionary {
- let dict = array.as_any_dictionary();
+ // A filtered batch or a shared Parquet dictionary can hold far more
+ // values than rows; drop unreferenced values so the regex runs at
+ // most once per row.
+ let compacted;
+ let mut dict = array.as_any_dictionary();
+ if dict.values().len() > dict.keys().len() {
+ compacted = garbage_collect_any_dictionary(dict)?;
+ dict = compacted.as_any_dictionary();
+ }with use arrow_select::dictionary::garbage_collect_any_dictionary; and arrow-select = { workspace = true } added to datafusion/functions/Cargo.toml.
This is the same cost issue as the substr fast path in #25874, and #26696 discusses handling it once for all functions that map over dictionary values, so this guard may later move into a shared helper.
| Utf8bd | ||
|
|
||
| statement ok | ||
| DROP TABLE regexp_replace_dict_runtime_scalars; |
Contributor
There was a problem hiding this comment.
Could you add a few cases for the other dictionary value types and the non-literal fallback? Dictionary(Int32, Utf8View) used to be a plan error and now works, and Binary values now come back as Dictionary(Int32, Utf8). I checked these locally and they all return the right values. Fine to do in a follow-up if you prefer.
CREATE TABLE t AS SELECT
arrow_cast(column1, 'Dictionary(Int32, LargeUtf8)') AS dl,
arrow_cast(column1, 'Dictionary(Int32, Utf8View)') AS dv,
arrow_cast(column1, 'Dictionary(Int32, Binary)') AS db,
arrow_cast(column1, 'Dictionary(Int32, Utf8)') AS d8,
column1 AS s
FROM (VALUES ('abc'), ('abd'), (NULL)) AS v;
-- expect Dictionary(Int32, LargeUtf8) / Dictionary(Int32, Utf8View) / Dictionary(Int32, Utf8)
SELECT arrow_typeof(regexp_replace(dl, 'a', 'x')), regexp_replace(dl, 'a', 'x') FROM t;
SELECT arrow_typeof(regexp_replace(dv, 'a', 'x')), regexp_replace(dv, 'a', 'x') FROM t;
SELECT arrow_typeof(regexp_replace(db, 'a', 'x')), regexp_replace(db, 'a', 'x') FROM t;
-- non-literal pattern: expect Utf8
SELECT arrow_typeof(regexp_replace(d8, s, 'x')), regexp_replace(d8, s, 'x') FROM t;
Contributor
|
This PR appears to be waiting on the author to respond to reviewer feedback so marking as draft. Please mark ready to review when it is ready for another look |
alamb
marked this pull request as draft
October 24, 2026 10:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Preserves Dictionary encoding on the input column by running regexp_replace over the dict's unique values and reusing the keys, avoiding the full materialization that previously happened when the column was dict-encoded.
What changes are included in this PR?
user_definedwithcoerce_typesthat keepsDictionaryon arg[0].return_type/return_field_from_argscarry the dict wrapper through, dropping it only when the other args are arrays (keys can't round-trip).invoke_with_argsruns the regex ondict.values()and re-attaches the keys;What is the testing strategy for this PR?
added one sqllogictest in
datafusion/sqllogictest/test_files/regexp/regexp_replace.sltthatasserts
arrow_typeof(regexp_replace(dict_col, pattern, replacement))returns
Dictionary(Int32, Utf8). Existing regex / string sqllogictestsand the
regexp_replaceunit tests still pass unchanged.Are there any user-facing changes?
no