Skip to content

[Task]: Spark 4 streaming sibling scans of one unbounded source can disagree on a deadline cut batch #40786

Open

Description

@tkaymak

Part of #36841. When one unbounded source PCollection feeds two branches of a single streaming query, for example Flatten of a PCollection with itself, Spark plans one MicroBatchScanExec per branch and does not deduplicate streaming relations, so each micro batch opens two readers for the same split. The sibling scan fix that follows #40766 gives the second scan a detached reader restored from the start mark, never cached and writing no marks. The two scans still agree only when the source yields the same records up to the same cut. With the default configuration (maxBatchDurationMillis 500 ms, no record quota) a batch cut by the deadline can end at different records per sibling, so the detached branch can lose or duplicate the tail of that batch while the next batch continues from the cached reader's mark. Bounded inputs via Create that drain within the deadline are exact. Possible fixes are recording the cached scan's cut (record count or end mark) so siblings replay to it, or a per batch buffer keyed by split and epoch shared by all scans of the key.

Activity

  1. tkaymak commented on Oct 25, 2026

    @tkaymak
    ContributorAuthor

    .take-issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions