Repository navigation
The Inference Python Benchmarks Dataflow job is flaky #38782
Description
The Inference Python Benchmarks Dataflow is failing over 50% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=9&var-Workflow=Inference%20Python%20Benchmarks%20Dataflow
Activity
github-actions commentedon Jun 13, 2026
Reopening since the workflow is still flaky
shunping commentedon Jun 30, 2026
I think the root cause of the
resolution too deepfailure is due to dependency conflict introduced by #37639.Particularly, the following dep is added to
setup.py:beam/sdks/python/setup.py
Line 535 in 961ed7c
While tensorflow-transform-1.4(https://github.com/tensorflow/transform/blob/r1.14.0/setup.py#L48), tensorflow-metadata-1.14(https://github.com/tensorflow/metadata/blob/r1.14.0/setup.py#L130), tfx-bsl-1.14(https://github.com/tensorflow/tfx-bsl/blob/r1.14.0/setup.py#L174) together has protobuf requirement as
protobuf>=3.20.3,<4.21,google-cloud-dataflow-client 0.13.0depends onprotobuf<8.0.0 and >=4.25.8. This leads to an unsatisfied requirement for the latest new Beam version (2.75+).As a result, pip has difficulty to resolve the dependencies (causing
resolution too deeperror); uv is able to resolve it, but it has to downgrade Beam version to 2.74 to get rid of the newly introducedgoogle-cloud-dataflow-clientrequirement.shunping commentedon Jun 30, 2026
It is most likely that the
resolution too deepfailure from #38780 has the same root cause of this. I see there is a PR to fix that, but maybe we can simplify that to only upgrading tensorflow-transform, tensorflow-metadata and tfx_bsl version like #39163? @aIbrahiimI am experimenting this idea at #39164.
aIbrahiim commentedon Jun 30, 2026
Agree it’s the same root cause as old TFT stack conflicting with apache beam gcp and #39163’s minimal bump of tensorflow-transform / tensorflow-metadata / tfx-bsl to 1.21.x is a cleaner fix than full pinning + constraints if CI confirms it resolves ResolutionTooDeep and I also hit a second failure after driver install (worker staging with pyarrow 10 / TF 2.16 wheels), which 1.21 may address via newer pyarrow wheels
aIbrahiim commentedon Jun 30, 2026
I looked at last run https://github.com/apache/beam/actions/runs/28416310321/job/84199856147 and the Generate Vocab job is failing with error is that the GCS path for MLTransform artifacts is always the same and attributes.json is already there from a previous run also the job uses gs://temp-storage-for-perf-tests/loadtests/mltransform_vocab_artifacts every time because it don’t pass --artifact_location. MLTransform refuses to overwrite that folder so we fix it like One Hot Encoding by adding a unique path per run like --artifact_location=gs://temp-storage-for-perf-tests/mltransform/vocab_artifacts_${NOW_UTC} in the workflow step @shunping
shunping commentedon Jun 30, 2026
Thanks for checking @aIbrahiim. Let me add that and rerun the test.
github-actions commentedon Jul 22, 2026
Reopening since the workflow is still flaky
damccorm commentedon Aug 6, 2026
We should fix this, but it is not a release blocker, it is likely a test issue