[Bug]: job launch problem 2.75.0 #39353

Activity

1. aprochko

   added

   bug

   awaiting triage

   on Jul 16, 2026
2. github-actions

   added

   java

   P2

   on Jul 16, 2026
3. shunping commented on Jul 21, 2026

   

   shunping

   on Jul 21, 2026

   Collaborator

   More actions

   Could you provide more information of what your pipeline looks like?

   What is the source and sink? What kind of steps/transforms you use in the middle?
4. shunping commented on Jul 21, 2026

   

   shunping

   on Jul 21, 2026

   Collaborator

   More actions

   cc'ed @Abacn
5. Abacn commented on Jul 21, 2026

   

   Abacn

   on Jul 21, 2026

   Contributor

   More actions

   also which Beam version were you upgrade from? If it's Dataflow runner please open a customer support ticket so engineers can inspect the job
6. aprochko commented on Jul 22, 2026

   

   aprochko

   on Jul 22, 2026

   Author

   More actions

   Hi @shunping @Abacn

   thank you for the response

   1. From 2.74.0
   2. Typical pipeline

   KafkaIO -> some aggregation functions with state/timers and BigTable/Bigquery clients (not IO-ed) -> Flatten Pcollections -> Aggregation with state/timers -> BigTableIO + io.requestresponse

   I will create support ticket
7. aprochko commented on Sep 14, 2026

   

   aprochko

   on Sep 14, 2026

   Author

   More actions

   Follow-up: traced this to a Beam POM change between 2.74.0 → 2.75.0

   We dug into this a bit more on our side. Our pipeline: KafkaIO → aggregation functions with state/timers, calling BigTable/BigQuery clients directly (not via IO transforms) → Flatten PCollections → aggregation with state/timers → BigTableIO + io.requestresponse.

   What changed between 2.74.0 and 2.75.0

   Comparing beam-sdks-java-io-google-cloud-platform POMs on Maven Central:

   In 2.74.0, the google-cloud-bigtable dependency is explicitly pinned:

   <dependency>
     <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-bigtable</artifactId>
     <version>2.73.1</version>
     ...

   In 2.75.0, the same dependency has no explicit version anymore. It now resolves transitively through:

   libraries-bom:26.83.0 → google-cloud-bom:0.264.0 → gapic-libraries-bom:1.86.2 → google-cloud-bigtable-bom:2.78.1
   

   (confirmed directly in gapic-libraries-bom-1.86.2.pom). For Beam 2.76.0, the same chain resolves to bigtable 2.80.0.

   Why this matters

   Starting with google-cloud-bigtable 2.76.0, DirectPath is enabled by default (bypasses the Cloud Front End, direct gRPC), and 2.77.0 introduced a new session-based ("VRpc") client stack. The versions Beam now pulls in transitively (2.78.1 for 2.75.0, 2.80.0 for 2.76.0) both sit inside this new stack, and the client's own changelog shows a sequence of concurrency fixes across 2.78.0–2.81.0 (session pool leaks, deadlocks, and in 2.81.0 specifically "Bound SessionPoolImpl lock to prevent pod-wide wedge") — which lines up closely with the "job gets stuck, 100% CPU, no progress" symptom.

   Short stack trace we saw (appeared once, after ~2 hours of the job running):

   com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL:
     Abrupt GOAWAY closed sent stream. HTTP/2 error code: INTERNAL_ERROR,
     debug data: Cancelling all calls
       at com.google.cloud.bigtable.data.v2.stub.BigtableChannelPool...
   

   Is there anything that can realistically be improved on Beam's side here — re-pinning explicitly, or at least surfacing this as a known-risky transitive upgrade?

   For context on our side, we use both BigtableIO (for writes) and our own custom repository classes (for reads). We initially had the read-side client created lazily and never explicitly closed, and suspected that pattern as a contributing factor, so we changed it to create the BigtableDataClient explicitly in each DoFn's @Setup and close it in @Teardown. That didn't resolve it — we're still seeing the same freeze on 2.83.0 with the client properly scoped and closed, which points more toward the client/DirectPath negotiation itself rather than our lifecycle management around it.

   We also run in a fairly locked-down private network (restrictive firewall rules), and suspect DirectPath's connection negotiation may not be degrading cleanly in that environment — which could explain why we see a silent freeze instead of a clean connection error. We're not sure how much of this is on us vs. the client/Beam interaction, but I guess it's important to share it there :)