Summary
Group-by aggregation over Dictionary(Int32, Utf8View) group keys is measurably slower than the identical query over plain Utf8View columns, on both DataFusion 54.0.1 and 55.0.0 (rev-checked). On TPC-H SF=10 in-memory:
- Q2 (p_type/p_size keys, 2 passes): dictionary -27% vs Utf8View
- Q19 (ps_shipmode key): dictionary -37%
- Q20 (s_name key): dictionary -33%
Reproduced with a self-contained bench: same RecordBatches, only the group-key column encoding differs (via Arrow cast before planning). No disk, no confounding.
Repro sketch
// two sessions over the same batches; key column cast:
// a) Utf8View (baseline)
// b) Dictionary(Int32, Utf8View) (taxed)
let agg = ctx.sql("select l_returnflag, count(*) from t group by l_returnflag").await;Timings (5-round median, single machine, Win11, DF 55.0.0):
| query |
Utf8View |
Dictionary |
delta |
| Q2-like |
21.1ms |
28.9ms |
-27% |
| Q19-like |
8.6ms |
13.6ms |
-37% |
| Q20-like |
12.4ms |
18.5ms |
-33% |
Why this matters
Dictionary encoding is the natural choice for low-cardinality string columns
(exact-dictionary scans, smaller memory). But as soon as such a column is a
group key, users pay a ~1.3-1.6x aggregation penalty — the opposite of what
encoding choice should do to query time. In our columnar engine (custom
TableProvider over PAX storage) this forces a lose-lose choice between scan
speed and aggregation speed on every low-cardinality string column.
Suspect area
Group-aggregation kernels appear to upcast/normalize dictionary keys per
batch rather than grouping on the codes directly (for single-batch
aggregations the dictionary is already a perfect group id in the common
no-duplicate case). Happy to share the standalone bench if useful.
DF 54.0.1 and 55.0.0 both affected; can test main on request.
Environment
DataFusion 55.0.0 / 54.0.1, Arrow 57/56, Windows 11, rustc 1.93-nightly,
in-memory RecordBatches (8192-row batches), single-threaded execution to
isolate kernel cost.
Summary
Group-by aggregation over
Dictionary(Int32, Utf8View)group keys is measurably slower than the identical query over plainUtf8Viewcolumns, on both DataFusion 54.0.1 and 55.0.0 (rev-checked). On TPC-H SF=10 in-memory:Reproduced with a self-contained bench: same RecordBatches, only the group-key column encoding differs (via Arrow cast before planning). No disk, no confounding.
Repro sketch
Timings (5-round median, single machine, Win11, DF 55.0.0):
Why this matters
Dictionary encoding is the natural choice for low-cardinality string columns
(exact-dictionary scans, smaller memory). But as soon as such a column is a
group key, users pay a ~1.3-1.6x aggregation penalty — the opposite of what
encoding choice should do to query time. In our columnar engine (custom
TableProvider over PAX storage) this forces a lose-lose choice between scan
speed and aggregation speed on every low-cardinality string column.
Suspect area
Group-aggregation kernels appear to upcast/normalize dictionary keys per
batch rather than grouping on the codes directly (for single-batch
aggregations the dictionary is already a perfect group id in the common
no-duplicate case). Happy to share the standalone bench if useful.
DF 54.0.1 and 55.0.0 both affected; can test main on request.
Environment
DataFusion 55.0.0 / 54.0.1, Arrow 57/56, Windows 11, rustc 1.93-nightly,
in-memory RecordBatches (8192-row batches), single-threaded execution to
isolate kernel cost.