The PostCommit Java ValidatesRunner Dataflow Streaming Engine job is flaky

The PostCommit Java ValidatesRunner Dataflow Streaming Engine is failing over 50% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=4&var-Workflow=PostCommit%20Java%20ValidatesRunner%20Dataflow%20Streaming%20Engine

Stabilized

Reopening since the workflow is still flaky

@damccorm

@damccorm , @Abacn The workflow is failing on different tests like PAssertTest, WindowingTest and always with the same error:

java.lang.RuntimeException: Startup of the worker pool in us-central1 failed to bring up any of the desired 1 workers. This is likely a quota issue or a Compute Engine stockout. The service will retry. For troubleshooting steps, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-pool-failure for help troubleshooting. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: Instance 'windowingtest0testwindowp-09111349-676d-harness-q6c7' creation failed: The zone 'projects/apache-beam-testing/zones/us-central1-c' does not have enough resources available to fulfill the request.  'NULL:0/NULL:0/NULL:0 (state:STOCKOUT, sub-state:STOCKOUT, resource type:compute)'.
	at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:149)
	at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:101)
	at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:56)

Stabilized

Reopening since the workflow is still flaky

I checked the recent failures and I found out it’s not a real bug as different runs fail on different tests like ParDo timers one day, combine/flatten another and even the same SHA can pass then fail

and what’s actually happening is the Dataflow jobs get stuck in RUNNING and hit the 15 min test timeout (or sometimes gcp just can’t spin up workers in us central1 because of stockouts)

maybe we can run this job at a quieter time and monitor it?

@tomstepp

The timeline is designed such that different variants spans throughout the day

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml?query=event%3Aschedule

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.yml?query=event%3Aschedule

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.yml?query=event%3Aschedule

maybe we can swap the time between beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming (takes 5h) and beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine (takes 8.5h). Not sure why one variant is more flaky than the others, probably due to the busy time hour, or others

cc: @tomstepp in case there is a real issue.

There is an ongoing effort to improve Validates runner test, including group different validate runner test into a single Dataflow pipeline, which will mitigate capacity issues.

Stabilized

Reopening since the workflow is still flaky

Navigation Menu

Repository navigation

The PostCommit Java ValidatesRunner Dataflow Streaming Engine job is flaky #39835

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

The PostCommit Java ValidatesRunner Dataflow Streaming Engine job is flaky #39835

Description

Activity

Amar3tto commented on Aug 24, 2026

github-actions commented on Sep 11, 2026

aIbrahiim commented on Sep 13, 2026

Amar3tto commented on Sep 15, 2026

github-actions commented on Sep 28, 2026

aIbrahiim commented on Sep 29, 2026

Abacn commented on Sep 29, 2026

Amar3tto commented on Oct 8, 2026

github-actions commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions