Skip to content

[python] Support window joins in temporal alignment - #9760

Open
XiaoHongbo-Hope wants to merge 12 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-window-aggregate
Open

[python] Support window joins in temporal alignment#9760
XiaoHongbo-Hope wants to merge 12 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-window-aggregate

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

Join higher-frequency samples, such as IMU, to temporal anchor rows and aggregate each matched window.

Changes

  • add top-level and chainable join_window
  • support group-local preceding / following ranges with configurable endpoint closure
  • support named outputs and multiple aggregations per source column
  • support mean, min, max, first, last, and count; the latter three also accept non-numeric values
  • prune right payload reads to aggregation columns
  • reuse snapshot pinning, authorization masking, batched row-ID fetches, and the bounded row-group cache

API choice

rolling().agg() aggregates windows within one input. This operation uses each left timestamp to select rows from a right input, then aggregates the matches, so it follows established window-join terminology. The verb-first join_window also matches the public join_asof API.

Named (source, operation) pairs allow one source column to produce multiple outputs; same-name operation strings remain shorthand.

Tests

147 temporal and multimodal table tests passed on Python 3.11 / PyArrow 19, including regressions for preserving masked field types after pruning colliding projection aliases. The regressions cover mean after STRING-to-DOUBLE masking, first/last after INT-to-STRING masking, unmatched windows, and rejection of mean/min/max on masked strings. Both integer-boundary regression tests also passed on Python 3.6 / NumPy 1.19.5 / PyArrow 6.0.1. An additional 2,400 boundary cases passed on each of NumPy 1.19.5 and 2.2.6. Python 3.6 syntax and flake8 checks passed.

The masked-alias regressions cannot complete on PyArrow 6 because the existing masking reader requires RecordBatch.set_column, which is unavailable in that version.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add window aggregation to temporal alignment [python] Add temporal window join Sep 12, 2026
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add temporal window join [python] Support window joins in temporal alignment Sep 12, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 13, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant