[stankiewicz]

Fixes #39667

Description

This PR updates dataflow_cloudml_benchmark_tests_py_versions in sdks/python/test-suites/gradle.properties from 3.10 to 3.11.

Background & Root Cause

The Dataflow TensorFlow Transform benchmark suite (:sdks:python:test-suites:dataflow:tftTests) experienced a worker startup and execution regression (sdk boot latency ballooned from ~1.5 minutes to ~6 minutes).

Investigation of worker startup logs revealed a pip dependency resolution conflict triggered by cloudml/requirements.txt:

1. Mutually Exclusive Protobuf Range on Python 3.10:
   • tensorflow-metadata==1.21.0 (required by tfx_bsl and tensorflow-transform) contains a Python <3.11-specific constraint:
     protobuf<=6.32;python_version<"3.11".
   • Beam master recently bumped google-cloud-bigtable to >=2.42.0, which strictly requires protobuf>=6.33.5.
   • On Python 3.10, these ranges do not overlap.
2. Pip Backtracking & On-Worker Compilation:
   • Pip rejects the pre-installed Beam master build and resolves the conflict by temporarily downloading apache-beam==2.75.0 and downgrading pyarrow to 23.0.1.
   • Right after, Gradle's staged --extra_package (apache-beam.tar.gz) forces a reinstall of master code, which compiles all Cython/C++ extensions from source directly on the worker VM. This alone consumes over 4 minutes of 100% CPU on each worker before the harness can even start.

Solution

In tensorflow-metadata==1.21.0's setup.py, the protobuf constraint for Python 3.11+ has no upper bound:

'protobuf>=4.25.2;python_version>="3.11"'

[stankiewicz]

R: @tvalentyn

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[tvalentyn]

thanks!