Skip to content

Fix FlinkOrderedListState: preserve elements with same timestamp - #39795

Open
waterWang wants to merge 1 commit into
apache:masterfrom
waterWang:fix/flink-orderedlist-same-timestamp-overwrite
Open

waterWang wants to merge 1 commit into
apache:masterfrom
waterWang:fix/flink-orderedlist-same-timestamp-overwrite

Conversation

@waterWang

Copy link
Copy Markdown

Fix FlinkOrderedListState data loss when multiple elements with the same timestamp are added to an OrderedListState.

Root cause: The readAsMap() method builds a TreeMap<Instant, TimestampedValue<T>> keyed by the element's timestamp. When a timestamp is shared by multiple elements, Map.put() silently overwrites the previous entry, destroying data.

Fix: Change the internal structure to SortedMap<Instant, List<TimestampedValue<T>>>, using computeIfAbsent to append same-timestamp elements to a list instead of overwriting. All callers (read(), readRange(), clearRange()) are updated to flatten the list via Iterables.concat().

Fixes #39782

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@je-ik
je-ik self-requested a review September 2, 2026 08:53

@je-ik je-ik left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

LGTM in general, but it would be good to cover this change in flink-specific tests. Even better would be if we could cover this in ParDoTest suite (next to testOrderedListStateUnbounded). This might reveal multiple runners where similar fix should be applied (in order not to block this PR, I'd suggest doing both - add flink-specific test and a different PR that will change the ParDoTest that is applied to multiple runners).

Also the PR fails spotless check - it can be fixed by ./gradlew spotlessApply

@shunping

shunping commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

@waterWang: any update regarding the last review?

@shunping

shunping commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

waiting on author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Java FlinkRunner OrderedListState silently overwrites duplicate elements sharing same timestamps

3 participants