Repository navigation
GROUPS frames and bounded rank split peers with logically equal nested dictionary nulls #26689
Description
Describe the bug
Two one-element Arrow lists can both represent [NULL] when one contains a null dictionary key and the other contains a key pointing to a null dictionary value. GROUPS BETWEEN CURRENT ROW AND CURRENT ROW counts one row in each group. rank() and dense_rank() also split the peers when the planner chooses BoundedWindowAggExec.
To reproduce
Register this two-row batch in a Rust SessionContext.
use std::sync::Arc;
use datafusion::arrow::{
array::{ArrayRef, DictionaryArray, Int8Array, Int32Array, ListArray, RecordBatch},
buffer::OffsetBuffer,
datatypes::{Field, Int8Type},
};
use datafusion::prelude::{SessionConfig, SessionContext};
let children: ArrayRef = Arc::new(DictionaryArray::<Int8Type>::try_new(
Int8Array::from(vec![None, Some(0)]),
Arc::new(Int32Array::from(vec![None])),
)?);
let key: ArrayRef = Arc::new(ListArray::new(
Arc::new(Field::new("item", children.data_type().clone(), true)),
OffsetBuffer::from_lengths([1, 1]), children, None,
));
let ctx = SessionContext::new_with_config(SessionConfig::new().with_target_partitions(1));
ctx.register_batch("t", RecordBatch::try_from_iter([("k", key)])?)?;Run these queries in that context.
SELECT count(*) OVER (ORDER BY k GROUPS BETWEEN CURRENT ROW AND CURRENT ROW) FROM t;
SELECT rank() OVER w, dense_rank() OVER w FROM t WINDOW w AS (ORDER BY k);
SELECT rank() OVER w, dense_rank() OVER w, percent_rank() OVER w
FROM t WINDOW w AS (ORDER BY k);| Query | Physical operator observed | Output for the two rows | Expected |
|---|---|---|---|
| GROUPS count | BoundedWindowAggExec (Sorted) |
1, 1 |
2, 2 |
| Rank and dense rank | BoundedWindowAggExec (Sorted) |
1, 2 for each function |
1, 1 for each function |
| Rank and dense rank alongside percent rank | WindowAggExec |
1, 1 for each rank function and 0.0, 0.0 for percent rank |
Same |
Reproduced at 99100274dd during review of #26067. GROUPS frame boundaries and bounded rank compare extracted ScalarValues. Batch rank uses Arrow partition boundaries, which group the two rows together. GROUPS boundaries · bounded rank · batch rank
Activity
fix: handle nested RANGE CURRENT ROW peers (apache#26067)