Repository navigation
[GSoC 2026] Kafka Streams Runner — translation framework + Impulse translator #38616
Description
Tracking issue: #18479
Depends on: #38465 (skeleton, merged via #38534)
Summary
Second sub-issue under the Kafka Streams Runner GSoC 2026 project. The
skeleton from #38465 currently throws UnsupportedOperationException
for every transform URN. This issue adds:
- A URN-dispatch framework in
KafkaStreamsPipelineTranslator(a
Map<String, TransformTranslator>populated at construction time,
walked viaQueryablePipelinein topological order — same shape as
FlinkStreamingPortablePipelineTranslator). - The first concrete translator: Impulse
(beam:transform:impulse:v1). Per the design doc §4.1, this uses a
dedicated bootstrap topic (e.g.__beam_impulse) so Kafka Streams
has a real source to consume from, emits one emptybyte[]element,
records in a state store that it has already fired, and advances
the watermark toBoundedWindow.TIMESTAMP_MAX_VALUE. - The translation context starts holding the Kafka Streams
Topology
being built and theMap<String, String>from PCollection ID to
processor node name.
After this issue, an Impulse-only pipeline translates and starts a
Kafka Streams topology. Pipelines containing any other URN still fail
fast with No translator registered for URN ... — the message format
unchanged from #38465.
Design doc reference
Portable Kafka Streams Runner for Apache Beam — design doc §4.1, §11.5.
Scope
-
KafkaStreamsTranslationContext: holdTopology, plus
Map<String, String> pcollectionIdToProcessorNameand accessors. -
KafkaStreamsPipelineTranslator:Map<String, TransformTranslator>
populated at construction; walk viaQueryablePipeline/
topological order. -
TransformTranslatorinterface (single method
translate(PTransformNode, RunnerApi.Pipeline, KafkaStreamsTranslationContext)). -
translation/ImpulseTranslatorimplementing the dedicated-topic
pattern from design doc §4.1. - Bootstrap-topic creation hook (auto-create via
AdminClientor
require pre-created — open question 12.1; pick one with note). - Wire
KafkaStreamsPipelineRunner.runto actually call
KafkaStreams.start()on the built topology (instead of
throwing). Returns aPortablePipelineResultthat tracks the
KafkaStreamsinstance state.
Acceptance criteria
-
./gradlew :runners:kafka-streams:checkgreen. - Unit test using
TopologyTestDriver: build an Impulse-only
pipeline proto, translate it, run the resulting topology, assert
exactly one emptybyte[]element appears at the output node
and is not re-emitted on restart. - Unit test asserting any non-Impulse URN still throws the same
"No translator registered for URN ..." message from [GSoC 2026] Kafka Streams Runner — skeleton Gradle module + pipeline entry points #38465. - PreCommit workflow added in [GSoC 2026] Kafka Streams runner skeleton module + portable entry points #38534 still triggers and passes.
Out of scope (deferred to follow-up sub-issues)
- ExecutableStage / stateless ParDo (next sub-issue).
- GroupByKey, Combine, Window assignment, Flatten.
- Watermark manager (per Jan: "watermark manager comes last when GBK
forces it"). - Splittable DoFn.
Reference implementation
runners/flink/2.0/src/main/java/.../FlinkStreamingPortablePipelineTranslator.java
— URN dispatch map pattern.runners/flink/2.0/src/main/java/.../translation/.../ImpulseSourceFunction.java
(and surrounding wiring) for the Flink-side analog. Kafka Streams
needs a different approach because KS requires a real input topic.
cc @je-ik
Activity
durgaprasadml commentedon May 30, 2026
Hi @junaiddshaukat, I’m interested in working on this issue and have started studying the existing Kafka Streams runner skeleton merged in #38534 along with the follow-up design discussion.
I’ll first sync with the latest integration branch and review the Flink portable runner translation flow (FlinkStreamingPortablePipelineTranslator) to align the implementation patterns before starting work on the translation framework and Impulse translator.
My plan is to focus on:
I’ll keep the implementation incremental and Beam-style compliant (nullness-clean, tests included, no unnecessary suppressions).
junaiddshaukat commentedon May 30, 2026
Thanks for your interest, but this is a reserved for Google summer of code
durgaprasadml commentedon May 30, 2026
Understood, thanks for the clarification @junaiddshaukat.
junaiddshaukat commentedon May 30, 2026
@durgaprasadml thanks for the interest here, and sorry about the earlier wording — it came out more restrictive than intended.
This issue is part of the current Kafka Streams runner translation work, which is aligned with the recently merged skeleton + the URN-based dispatch and Impulse translator design from the doc. Because of that, I wanted to avoid anyone starting in a direction that might conflict with the active implementation path.
That said, there’s a lot of related work in the runner that is not blocked by this issue and would be great for collaboration. I’ll sync with my mentor and put together a clearer breakdown under the tracking issue (#18479) showing:
what is currently in active GSoC execution path
what is open for independent contribution
what can be picked up in parallel once dependencies land
Happy to help guide you to a good entry point once that list is up.
durgaprasadml commentedon Jun 1, 2026
Thanks for the clarification @junaiddshaukat — that makes sense.
I appreciate the detailed context around the active implementation path and dependency flow. I’ll keep following the tracking issue and would definitely be interested in contributing to any parallel or independent runner tasks once they’re identified.
je-ik commentedon Jun 15, 2026
Closed via #38689.