Skip to content

KafkaUnboundedReader doesn't support updating pipeline with re-partition case. #20836

New issue
New issue
Closed as not planned
Closed as not planned
KafkaUnboundedReader doesn't support updating pipeline with re-partition case.#20836
Labels
P3improvementiojavakafka

Description

@damccorm
damccorm
opened on Jun 4, 2022

When updating pipelines with re-partition topic, KafkaUnboundedReader will throw exceptions from https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L470-L474

Imported from Jira BEAM-11947. Original Jira may contain additional context.
Reported by: boyuanz.

Activity

  1. damccorm
    added
    improvement
    io-java-kafka
    P3
    on Jun 4, 2022
  2. damccorm
    added
    io
    java
    kafka
    and removed
    io-java-kafka
    on Jun 16, 2022

  3. johnjcasey commented on Sep 8, 2022

    @johnjcasey
    johnjcasey
    on Sep 8, 2022
    Contributor

    Closing as "will not do". This is supported using the SDF reader implementation of KafkaIO

  4. johnjcasey
    closed this as completedon Sep 8, 2022
  5. Abacn
    closed this as not plannedon Sep 17, 2026

  6. Abacn commented on Sep 17, 2026

    @Abacn
    Abacn
    on Sep 17, 2026
    Contributor

    For future reference, the SDF implementation support this via "withDynamicRead(Duration)" transform configuration (done in #21752)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3improvementiojavakafka

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions