Skip to content

perf: memoize LAST_VALUE and NTH_VALUE with IGNORE NULLS - #26664

Open
comphead wants to merge 1 commit into
apache:mainfrom
comphead:nth-value-ignore-nulls
Open

comphead wants to merge 1 commit into
apache:mainfrom
comphead:nth-value-ignore-nulls

Conversation

@comphead

@comphead comphead commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

With IGNORE NULLS, LAST_VALUE and NTH_VALUE over a frame that starts at the partition's first row kept the whole partition buffered and rescanned long NULL runs for every row. #26533 already fixed the common case of #26538. FIRST_VALUE, LAST_VALUE and NTH_VALUE also built a ScalarValue per row, which is slow for nested types.

What changes are included in this PR?

  • memoize moves the frame start under IGNORE NULLS too. LAST_VALUE keeps its last non-NULL value and drops the whole frame.
  • A new PartitionEvaluator::evaluate_ranges evaluates all frames of a batch at once, and NthValueEvaluator implements it with one take. Unlike the row index that FIRST_VALUE, LAST_VALUE and NTH_VALUE build a ScalarValue for every row #26540 suggests, this also covers a final FIRST_VALUE result whose row is already pruned.

NTH_VALUE(expr, ±n) with n > 1 still keeps the rows from the first valid row it needs.

What is the testing strategy for this PR?

New window.slt cases and a memoize unit test. In datafusion-cli with the release-nonlto profile:

Query main This PR
last_value(v) IGNORE NULLS, one 10M-row partition 5.64 s, 299 MiB 0.67 s, 43 MiB
last_value(v) IGNORE NULLS, 1M rows with one non-NULL value 3.79 s 0.13 s
last_value(st) over a struct, 64-row partitions 0.25 s 0.10 s

The built-in function cases of window_query_sql are 9% to 73% faster, except one-row partitions (u64_wide), which are unchanged.

Are there any user-facing changes?

PartitionEvaluator gains evaluate_ranges, with a default implementation.

@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Oct 23, 2026
@comphead
comphead force-pushed the nth-value-ignore-nulls branch from 9384811 to 40c848f Compare October 23, 2026 16:00
@comphead
comphead marked this pull request as ready for review October 23, 2026 16:07
@comphead comphead added the performance Make DataFusion faster label Oct 23, 2026
@comphead comphead changed the title perf: memoize LAST_VALUE and NTH_VALUE with IGNORE NULLS, gather results with take perf: memoize LAST_VALUE and NTH_VALUE with IGNORE NULLS Oct 23, 2026
@comphead
comphead requested review from andygrove and sunchao October 23, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation logical-expr Logical plan and expressions performance Make DataFusion faster physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

1 participant