[sunchao]

Which issue does this PR close?

Related to #24937. This PR fixes RANGE-window comparisons. #26689 tracks the related GROUPS, rank(), and dense_rank() results for logically equal dictionary nulls.

Rationale for this change

RANGE windows can return incorrect counts for ordinary lists containing nulls.

SELECT count(*) OVER (ORDER BY l NULLS FIRST)
FROM (VALUES ([NULL::int]), ([1]), ([NULL::int])) t(l);

The result should be 2, 2, 3, but the old peer search returns 3, 3, 3. Lists containing lists or structs can also fail with Uncomparable values, including during streaming execution in Linear mode.

What changes are included in this PR?

Use Arrow's logical comparison to find nested CURRENT ROW peers. A null dictionary key and a key pointing to a null dictionary value can encode the same nested value. The window frame state owns its comparator and reuses it while the ordering arrays are identical. Pruning releases the old arrays. Cloned states construct a comparator on demand. The public WindowFrameStateRange keeps its unwind-safety traits.

Arrow's partition also recognizes the dictionary-null peers. It checks adjacent rows for every ordering column and materializes the group ranges when ranges() is called. A bounded window can resume partway through a partition after appending a batch. Keeping the boundary scan lets it resume from the previous frame without enumerating every group in the replacement buffer. Nested-type dispatch runs once per range calculation when either bound is CURRENT ROW.

Linear-mode completion uses Arrow comparison and computes its Boolean result lazily once per partition evaluation. The SQL analyzer governs type admission, and finite RANGE offsets use the scalar arithmetic path.

What is the testing strategy for this PR?

window.slt covers the ordinary-list wrong result and tied List<List> and List<Struct> keys with a secondary sort key and one-row batches. The bounded and unbounded-following queries exercise BoundedWindowAggExec (Sorted) and WindowAggExec, respectively. The physical integration tests cover struct keys, Linear execution, and emission before EOF. In the descending streaming case, a row in another partition completes the waiting peers under DESC NULLS LAST, both when its key is smaller and when it is NULL. State tests cover nested dictionary nulls, cloning, pruning, and unwind safety.

Validation at 39358f523 on upstream base 364613296 included the extended workspace suite (12,745 Rust tests passed, eight ignored, 535 SQL-logic files completed), formatting, all-target and all-feature Clippy with warnings denied, and the CI Clippy configuration. All ten bounded-window benchmark cases passed in smoke mode on both the base and the patch.

License, spelling, TOML formatting, local Markdown links, and unused dependency checks passed. Earlier validation at 99100274 also covered generated docs and the Rust and website documentation builds.

cargo audit reports RUSTSEC-2026-0382, RUSTSEC-2026-0383, and RUSTSEC-2026-0385 on both the PR and its unmodified upstream base, which share a lockfile. Current main contains the fixed versions of all three crates. Certificate dependency updates · ctutils update

Are there any user-facing changes?

Already-admitted SQL RANGE windows over nested keys return correct peer counts. Physical window execution handles these peer boundaries in both Sorted and Linear modes.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 87.25490% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.92%. Comparing base (63af08d) to head (3ee0478).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/expr/src/window_state.rs	87.25%	6 Missing and 7 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #26067    +/-   ##
========================================
  Coverage   82.91%   82.92%            
========================================
  Files        1155     1155            
  Lines      458058   458304   +246     
  Branches   458058   458304   +246     
========================================
+ Hits       379813   380038   +225     
- Misses      55202    55216    +14     
- Partials    23043    23050     +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sunchao]

cc @viirya @alamb @comphead for review

[viirya]

Thanks for this. Using Arrow's comparator to find CURRENT ROW peers by equality is more robust than the ordering-based scan, since it no longer relies on ScalarValue ordering matching the sort. Two larger points, plus details inline:

Linear / PartiallySorted mode still fails for nested keys. is_row_ahead in window_expr.rs (outside this diff) still calls compare_rows on order_bys[0]. BoundedWindowAggExec reaches it for RANGE ... CURRENT ROW frames whenever it is in linear search mode. I took nested_batches() from this PR, added a constant p column, and ran PARTITION BY p with InputOrderMode::Linear. All three key types fail with Internal("Uncomparable values: List([[]]), List([[]])"). So "the same boundary fix applies to windows constructed through the physical API" currently holds only for Sorted. is_row_ahead needs an ordering rather than equality, so make_comparator(old_col, current_col, opts) over the two arrays looks like a natural fit. Could this be covered here?

This also fixes a wrong result for plain lists, which seems worth claiming and testing. On main,

select l, count(*) over (order by l nulls first)
from (values ([NULL::int]), ([1]), ([NULL::int])) t(l)

returns 3, 3, 3. With this PR it returns the correct 2, 2, 3, because partial_cmp_list orders a NULL element the opposite way from the sort. List<Int> keys are much more common than lists of lists, so this is probably the most practical benefit of the change.

[comphead]

Thanks for the update. The Arrow comparator is a good fit for peer detection. Points not raised earlier:

• is_row_ahead is shared with GROUPS frames, but GROUPS group boundaries still come from ScalarValue equality (check_equality in WindowFrameStateGroups). The nested dictionary null case described here would still split into separate groups there, unlike RANGE peers. A follow-up issue or a sentence in the description would make the scope clear. I did not run this case.
• Design question: peer groups are contiguous equal runs, which evaluate_partition_ranges (arrow partition) already computes in one vectorized pass for PARTITION BY and rank. Caching those ranges per array instead of a comparator would turn the scan into a lookup and might remove the custom Debug and the unwind-safety impls. I have not checked that partition handles the dictionary-inside-list null case, so the existing regression would need to confirm it.
• Nit: peer_comparator ends with .as_ref().unwrap() right after assigning the value. Option::insert on the rebuild branch avoids an unwrap in library code.
• Nit: column.data_type().is_nested() runs for every row on both the start and end search, and it recurses through dictionary and run-end wrappers. Deciding once per calculate_range, or when the cache is built, avoids the repeated work.

[viirya]

Thanks for the thorough update. All of my earlier points are addressed, and the Linear-mode fix through make_comparator with the per-evaluation row_ahead reuse looks good to me.

A few remaining notes, none blocking:

• The dictionary-null argument for not using ScalarValue equality is convincing, but it also applies to other peer checks that still use it. One is GROUPS boundaries, as @comphead noted. The other is the bounded rank evaluator: RankEvaluator::evaluate compares get_row_at_idx values with !=. The non-bounded path uses evaluate_partition_ranges, i.e. Arrow partition, which falls back to make_comparator for nested types. So the same query could group rank peers differently under BoundedWindowAggExec and WindowAggExec. Could you open a follow-up issue covering both?
• +1 to @comphead's point that the nested slt queries need a tied row to actually exercise peers.
• On the Arc around the comparator: it is also what keeps #[derive(Clone)] working, since LexicographicalComparator isn't Clone. A manual Clone that drops the cache would work too, because the only production clone (standard.rs) clones a fresh state. Either is fine with me.