Repository navigation
[Feature Request]: Wait.on for Go SDK #39909
Description
What would you like to happen?
I'd like to port the Wait.On functionality of Java to the Go SDK, so that runtime ordering constraints can be added between transforms without requiring modification of the transforms.
I often have operations which rely on side-effects (eg database writes having completed, GCS storage, BigTable updates, etc). In order to sequence these operation in Go, I end up wiring in completion signals manually into producers and then side-inputs into consumers. This is effectively a manual implementation of wait.On which reduces composability of the operations since it means the shape of the operation has to be modified for the particular situation.
The goal would be to provide a transforms/wait package containing an On method:
// To finish writing each window to one database before writing it
// to another:
firstWriteResults := beam.ParDo(s, writeToFirstDB, data)
delayed := wait.On(s, data, firstWriteResults)
beam.ParDo0(s, writeToSecondDB, delayed)Where On would look like this:
// On returns col unchanged, but delays each main window until its mapped window
// has closed in every signal.
//
// With no signals, On validates s and col and returns col directly. Otherwise,
// it panics at pipeline construction time for an invalid scope or PCollection,
// CoGBK inputs, or session-windowed signals.
func On(s beam.Scope, col beam.PCollection, signals ...beam.PCollection) beam.PCollection {Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
Gaurav598 commentedon Aug 30, 2026
Hi @vgough, I’ve investigated this issue in detail, including the Java
Wait.onimplementation, its windowing/side-input semantics, the relevant Go SDK architecture, and the available Go primitives and testing infrastructure.I’d like to work on implementing
Wait.Onfor the Go SDK..take-issue
vgough commentedon Aug 31, 2026
Hi. I actually ported the Java implementation to Go already. I haven't posted it because I wanted to do more testing. I can create a pull request though while I continue testing, in case you want to give feedback?
Gaurav598 commentedon Aug 31, 2026
Thanks for the update, @vgough
Since you’ve already ported the Java implementation to Go, please go ahead and open the PR while you continue testing.
I’d be happy to review the implementation and test it against the Go SDK semantics, especially the windowing, side-input, and sequencing behavior, and provide feedback if I find anything.
Gaurav598 commentedon Aug 31, 2026
@vgough Since you've already implemented the Go port and are preparing the PR, I’m happy to step back from the issue so there’s no overlap.
I don’t have permission to remove myself as the assignee. Could you or a maintainer please unassign me when convenient?
I’ll still be happy to review/test the PR once it’s opened.
vgough commentedon Aug 31, 2026
@Gaurav598 I've pushed a PR request. I primarily use Beam for batch processing, so much of the streaming support is based on theory rather than practice. There are some differences over the Java support, like the way that side-inputs are bound in Go function signatures and window emission options. Thanks for offering to help.