Repository navigation
[New Feature] IBM MQ IO #20990
Description
We want to create a data pipeline that will read message from IBM MQ. And process the messages in the order it receive the message from IBM MQ. IBM MQ will deliver the message in order. The message does not have any intelligence (for example a timestamp or sequence number)..Would appreciate if it possible to achieve it without using single threaded model.
We want to scale it we would like to have ordering & parallelism, not sure whether its doable or not. Request your help. The runner is google dataflow.
Imported from Jira BEAM-12499. Original Jira may contain additional context.
Reported by: ashwinidas.
Activity
kennknowles commentedon Dec 2, 2022
Order and parallelism are logically opposite. By definition things that are in parallel are not ordered, and vice versa. You can achieve parallelism with per-shard order, which is what people tend to do.
kennknowles commentedon Dec 2, 2022
I don't know anything about IBM MQ but it sounds like this is a feature request for an IO
[-]Processing message in sequence in a pipeline from IBM MQ (deliver message in order)[/-][+]IBM MQ IO[/+][-]IBM MQ IO[/-][+][New Feature] IBM MQ IO[/+]Abacn commentedon Jul 23, 2026
Currently one can use Beam's JmsIO with IBM MQ client as a solution. But we should add integration test for that.