Repository navigation
Conversation
Explain in the Trigger proto comments that when a window's root trigger finishes, the window closes: that firing is the final pane, no further panes are emitted, and later elements are dropped. Note that such triggers are "unsafe" because they can lose data, and document when each trigger type finishes.
There was a problem hiding this comment.
📋 Review Summary
This PR significantly improves the documentation of trigger messages in beam_runner_api.proto by explicitly specifying the finishing semantics and final pane behaviors for each trigger type, as well as the top-level window closing and garbage collection rules. The added comments are clear, thorough, and well-aligned with the Beam model and runner implementations.
🔍 General Feedback
- The distinction between trigger finishing (releasing buffered data and dropping subsequent elements) and window expiration (
end of window + allowed lateness) is clearly articulated in the top-levelTriggermessage. - A minor correction is suggested on
Neverto avoid conflating trigger-level firing with runner-level final pane emission at window garbage collection.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
💬 Inline Findings
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto(line 1244): 🟢 Low: TheNevertrigger itself never fires and never finishes (shouldFireis alwaysfalse). When used as a root trigger, the output atend of window + allowed latenessis produced by the runner's window garbage collection mechanism (as described in the top-levelTriggercomment) rather thanNeveritself firing or finishing. IfNeveris used as a subtrigger (e.g., insideAfterAllorAfterEach), it will never fire even at window GC.
Clarifying that Never itself never fires or finishes avoids ambiguity when it is composed inside other triggers.
// Never fires or finishes. When used as the root trigger, a final pane is still emitted at end of window + allowed lateness based on ClosingBehavior.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40879 +/- ##
===========================================
+ Coverage 0 79.82% +79.82%
===========================================
Files 0 494 +494
Lines 0 83573 +83573
===========================================
+ Hits 0 66715 +66715
- Misses 0 16858 +16858
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
|
||
| // Ready when all subtriggers are ready. | ||
| // | ||
| // Finishes after it fires once. |
There was a problem hiding this comment.
I would assume this means all the sub triggers have fired and finished, since they are all ready?
Finishes after firing all subtriggers once?
|
The root trigger should never finish. That's sort of the point of the doc linked. |
kennknowles
left a comment
There was a problem hiding this comment.
The comments in the proto seems useful. However, we should not revive the idea of triggers "finishing", and we should remove it from any SDK it has snuck into. It is not very useful for saying "good enough, throw away the rest of the data" because triggers are not expressive enough for that - instead, use state/timers or some other filter. It is very easy to accidentally make a trigger that drops data that you don't realize. This happened a lot in the early days, hence I made the breaking change to remove it.
|
The usefulness of triggers finishing is moving from one trigger to another in a Sequence trigger. Which, TBH, is also not very useful. |
|
Assigning reviewers: R: @chamikaramj added as fallback since no labels match configuration This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Summary
The
Triggermessage inbeam_runner_api.protoonly says when each trigger is ready to fire. It doesn't say what happens when a trigger finishes, meaning it will never fire again. Runners depend on that: when a window's root trigger finishes, the window is closed, no more panes are emitted, and later elements are dropped.Java and Python already behave this way, but the rules live only in SDK and runner code. Prism behaved differently until #40878. This PR adds the rules to the model as comments only; there are no schema changes.
A similar topic has been covered in Guard against "Trigger Finishing", a data loss risk by @kennknowles.
Changes
Triggerheader commentPaneInfo.is_last);ClosingBehavior/OnTimeBehavior;Repeatmakes a trigger never finish. Runners must still implement the closed-window rules for pipelines that use finishing triggers.When each trigger type finishes
AfterAllAfterAnyAfterEachAfterEndOfWindowlate_firings, on its firing after end of window; withlate_firings, never before expiry (late trigger repeats implicitly)AfterProcessingTime,AfterSynchronizedProcessingTimeElementCountOrFinallyfinallyfires, ormainfinishes, whichever comes firstRepeatDefault,AlwaysNeverClosingBehavior.EMIT_ALWAYS: notes that it doesn't apply if the window was already closed by a finished root trigger. In that case no pane is emitted at expiry.References
What "finishing" a root trigger means
Triggerjavadoc, Trigger.java#L41-L45:droppedDueToClosedWindow.windowIsActiveAndOpen(), i.e. the window is active and its trigger hasn't finished. So a closed window gets no extra pane, whateverClosingBehaviorsays.GeneralTriggerDriverin trigger.py marks it with aTOMBSTONEand skips later elements for it.orFinallymakes a trigger "fire one final time and never fire again"."Unsafe" triggers
Trigger.mayFinish(), Trigger.java#L143-L147: "A top level trigger that finishes can cause data loss, so is rejected by GroupByKey validation."GroupByKey, GroupByKey.java#L185-L245:triggerIsSaferejects root triggers that may finish, exceptNever, and end-of-window triggers with zero allowed lateness or with late firings. The error reads "Unsafe trigger '…' may lose data, did you mean to wrap it inRepeatedly.forever(...)?".DataLossReason.MAY_FINISH, "data that comes in after the trigger may be lost. Example: AfterCount(1) will stop firing after the first element."may_lose_data().GroupByKeyraises "Unsafe trigger … may lose data" unless--allow_unsafe_triggersis set.