Skip to content

Performance Regression or Improvement: test_cloudml_benchmark_criteo_10GB-runtime_sec:runtime_sec  #39667

Closed
#40333
Closed
Performance Regression or Improvement: test_cloudml_benchmark_criteo_10GB-runtime_sec:runtime_sec #39667
#40333
Labels
perf-alertAutomatically filed performance-related alerts.
Milestone
2.78.0 Release

Description

@github-actions
github-actions
bot
opened on Aug 7, 2026

Performance change found in the
test: test_cloudml_benchmark_criteo_10GB-runtime_sec for the metric: runtime_sec.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

Test description: TFT Criteo test on 10 GB data.
Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#LL104C7-L104C41


timestamp: Wed Aug  5 23:21:10 2026, metric_value: 2800.29
timestamp: Tue Aug  4 23:22:17 2026, metric_value: 2808.33
timestamp: Mon Aug  3 23:37:30 2026, metric_value: 2774.47
timestamp: Sun Aug  2 23:07:44 2026, metric_value: 2729.92
timestamp: Sat Aug  1 23:09:53 2026, metric_value: 2732.18
timestamp: Fri Jul 31 23:09:55 2026, metric_value: 2486.15
timestamp: Thu Jul 30 23:08:38 2026, metric_value: 2467.28
timestamp: Wed Jul 29 22:53:13 2026, metric_value: 2283.78
timestamp: Tue Jul 28 23:26:00 2026, metric_value: 2327.85
timestamp: Mon Jul 27 23:07:24 2026, metric_value: 2777.71
timestamp: Sun Jul 26 23:03:46 2026, metric_value: 2858.59 <---- Anomaly
timestamp: Sat Jul 25 22:54:07 2026, metric_value: 2398.88
timestamp: Fri Jul 24 23:00:38 2026, metric_value: 2379.31
timestamp: Thu Jul 23 23:14:49 2026, metric_value: 2403.32
timestamp: Wed Jul 22 22:56:31 2026, metric_value: 2223.12
timestamp: Tue Jul 21 22:56:42 2026, metric_value: 2191.53
timestamp: Mon Jul 20 22:55:29 2026, metric_value: 2081.13
timestamp: Sun Jul 19 22:49:13 2026, metric_value: 2286.14
timestamp: Sat Jul 18 22:48:21 2026, metric_value: 2334.37
timestamp: Fri Jul 17 22:44:25 2026, metric_value: 2140.38
timestamp: Thu Jul 16 22:48:39 2026, metric_value: 2199.59

Activity

  1. github-actions
    added
    awaiting triage
    perf-alertAutomatically filed performance-related alerts.
    on Aug 7, 2026

  2. github-actions commented on Aug 7, 2026

    @github-actions
    github-actionsBot
    on Aug 7, 2026 – with GitHub Actions
    ContributorAuthor

    Performance change found in the
    test: test_cloudml_benchmark_criteo_10GB-runtime_sec for the metric: runtime_sec.

    For more information on how to triage the alerts, please look at
    Triage performance alert issues section of the README.

    Test description: TFT Criteo test on 10 GB data.
    Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#LL104C7-L104C41

    
timestamp: Fri Aug  7 01:14:45 2026, metric_value: 2999.65
timestamp: Wed Aug  5 23:21:10 2026, metric_value: 2800.29
timestamp: Tue Aug  4 23:22:17 2026, metric_value: 2808.33
timestamp: Mon Aug  3 23:37:30 2026, metric_value: 2774.47
timestamp: Sun Aug  2 23:07:44 2026, metric_value: 2729.92
timestamp: Sat Aug  1 23:09:53 2026, metric_value: 2732.18 <---- Anomaly
timestamp: Fri Jul 31 23:09:55 2026, metric_value: 2486.15
timestamp: Thu Jul 30 23:08:38 2026, metric_value: 2467.28
timestamp: Wed Jul 29 22:53:13 2026, metric_value: 2283.78
timestamp: Tue Jul 28 23:26:00 2026, metric_value: 2327.85
timestamp: Mon Jul 27 23:07:24 2026, metric_value: 2777.71
timestamp: Sun Jul 26 23:03:46 2026, metric_value: 2858.59
timestamp: Sat Jul 25 22:54:07 2026, metric_value: 2398.88
timestamp: Fri Jul 24 23:00:38 2026, metric_value: 2379.31
timestamp: Thu Jul 23 23:14:49 2026, metric_value: 2403.32
timestamp: Wed Jul 22 22:56:31 2026, metric_value: 2223.12

  3. Amar3tto commented on Sep 1, 2026

    @Amar3tto
    Amar3tto
    on Sep 1, 2026
    Collaborator
    Image Seems like it has stabilized. I’ll keep monitoring it.

  4. aIbrahiim commented on Sep 14, 2026

    @aIbrahiim
    aIbrahiim
    on Sep 14, 2026
    Contributor

    I investigated CloudML Dataflow runs for this test and checked Grafana:

    Grafana shows that it was mostly around 35 to 42 min in July to August. From early September it jumps to around 50 min and stays there around 25 to 30% slower and The no_shuffle series moves the same way at the same time

    Deps didn’t move (Beam 2.77.0.dev0, TFT 1.21, TF 2.21) but on 2 September the container went from beam-master-20260817 to …0831 (#39958)

    as @Amar3tto earlier comment that it stabilized was before this September step. After the container update it looks like a real lasting regression again Image

  5. tvalentyn commented on Sep 22, 2026

    @tvalentyn
    tvalentyn
    on Sep 22, 2026
    Contributor

    https://metrics.beam.apache.org/d/C9nj1oxVk/tft-criteo-benchmarks?orgId=1&from=now-60d&to=now

  6. tvalentyn commented on Sep 22, 2026

    @tvalentyn
    tvalentyn
    on Sep 22, 2026
    Contributor

    Let's try to rootcause this.

  7. tvalentyn
    modified the milestones: 2.78.0 Release, 2.77.0 Release on Sep 22, 2026

  8. tvalentyn commented on Sep 22, 2026

    @tvalentyn
    tvalentyn
    on Sep 22, 2026
    Contributor

    cc: @ahmedabu98

  9. tvalentyn commented on Sep 22, 2026

    @tvalentyn
    tvalentyn
    on Sep 22, 2026
    Contributor

    Next steps: try to revert locally: https://github.com/apache/beam/pull/39958/changes , and run the test 5 times, and see if performance improves.

    THen, analyze dependecy diff between two container images.

  10. tvalentyn commented on Sep 22, 2026

    @tvalentyn
    tvalentyn
    on Sep 22, 2026
    Contributor

    looked closer, looks like the regression originally triggered a month earlier (around 08/01)

    Next steps: try to revert locally: #39958 (changes) , and run the test 5 times, and see if performance improves.

    THen, analyze dependecy diff between two container images.

  11. tvalentyn commented on Sep 22, 2026

    @tvalentyn
    tvalentyn
    on Sep 22, 2026
    Contributor

    cc: @stankiewicz

  12. kennknowles commented on Sep 28, 2026

    @kennknowles
    kennknowles
    on Sep 28, 2026
    Member

    Any progress on this one?

  13. stankiewicz commented on Sep 28, 2026

    @stankiewicz
    stankiewicz
    on Sep 28, 2026
    Contributor

    run 3-4 tests fro each beam-master version

    AVERAGE of duration test      
    version long short-one very-long
    beam-master-20260817 51.33333333 18  
    beam-master-20260831 60.5 21.25 67

  14. stankiewicz commented on Sep 28, 2026

    @stankiewicz
    stankiewicz
    on Sep 28, 2026
    Contributor

    I'm on some path to finding it, there is dependency conflict between tft reqs and what is on container after bigtable bump

  15. stankiewicz commented on Sep 28, 2026

    @stankiewicz
    stankiewicz
    on Sep 28, 2026
    Contributor

    With help from AI:
    The performance regression is caused by a pip dependency conflict during Dataflow worker startup. Because the pipeline option specifies --requirements_file=apache_beam/testing/benchmarks/cloudml/requirements.txt, every Dataflow worker runs pip install -r requirements.txt at boot time. On 20260831, pip discovers an irreconcilable conflict between TensorFlow Transform's dependency tree and Apache Beam's new Google Cloud Bigtable requirement, causing pip to uninstall apache-beam 2.77.0.dev0 and downgrade it to apache-beam==2.75.0

    tensorflow-metadata 1.21.0 Caps Protobuf - tfx_bsl==1.21.0 and tensorflow-transform==1.21.0 both depend on tensorflow-metadata==1.21.0. Its distribution metadata specifies: protobuf<=6.32 and >=4.21.6; python_version < "3.11"

    Beam Bumped google-cloud-bigtable in Master: In google-cloud-bigtable==2.42.0 protobuf<8.0.0, >=6.33.5

    When pip installs requirements.txt on the worker:

    1. tensorflow-metadata 1.21.0 is explicitly pinned in requirements.txt, so pip cannot change it (protobuf <= 6.32 is mandatory).

    2. The pre-installed apache-beam 2.77.0.dev0 requires bigtable >= 2.42.0, which demands protobuf >= 6.33.5.

    3. Because [6.0.0, 6.32] and [6.33.5, 8.0.0) do not overlap, pip concludes that the pre-installed apache-beam 2.77.0.dev0 is incompatible.

    4. tfx_bsl 1.21.0 only specifies apache-beam[gcp]<3, >=2.53. Therefore, pip is allowed to replace Apache Beam with any older release from PyPI matching <3, >=2.53.

    5. Pip checks PyPI:
      5.1 . 2.76.0: Also requires bigtable >= 2.42.0 (rejected).
      5.2 . 2.75.0: Allowed bigtable >= 2.19.0 (selected, because it allows bigtable==2.39.0 which supports protobuf <= 6.32).

  16. stankiewicz commented on Sep 28, 2026

    @stankiewicz
    stankiewicz
    on Sep 28, 2026
    Contributor

    will test container:

    FROM us-central1-artifactregistry.gcr.io/google.com/dataflow-containers/worker/v1beta3/beam_python3.10_sdk:beam-master-20260831
# Install TensorFlow, TF-Keras, TF-Serving-API, and dill
RUN pip install --no-cache-dir \
    tensorflow \
    dill==0.4.1 \
    tensorflow-serving-api \
    tf-keras
# Install TFT stack with --no-deps to preserve the pre-installed Apache Beam SDK
RUN pip install --no-cache-dir --no-deps \
    tfx_bsl==1.21.0 \
    tensorflow-transform==1.21.0 \
    tensorflow-metadata==1.21.0

  17. tvalentyn commented on Sep 28, 2026

    @tvalentyn
    tvalentyn
    on Sep 28, 2026
    Contributor

    thanks, in case of the dependency resolution problem it shouldn't be a release blocker.

  18. stankiewicz commented on Sep 29, 2026

    @stankiewicz
    stankiewicz
    on Sep 29, 2026
    Contributor

    The separate container didn't help because --requirements_file was still passed, forcing the worker to run pip install -r requirements.txt and temporarily pull apache-beam 2.75.0. While Gradle's injected --extra_package did successfully uninstall 2.75.0 and restore master/head build (2.78.0.dev0), it did so by compiling Beam’s source tarball (.tar.gz) directly on the worker VM—costing over 4 minutes of CPU compilation. Furthermore, the temporary 2.75.0 resolution left behind downgraded companion dependencies in the final environment, notably pyarrow 23.0.1 (down from 25.0.1).

    Impact: Worker startup time degraded from ~1.5 minutes to nearly 6 minutes (3.4x slower) due to on-worker compilation, while TFT still executed against an older, downgraded PyArrow runtime.

  19. stankiewicz commented on Sep 29, 2026

    @stankiewicz
    stankiewicz
    on Sep 29, 2026
    Contributor

    Why the Regression Happens on 3.10 vs. 3.11

    On Python 3.10, tensorflow-metadata enforces an environment-specific ceiling: protobuf<=6.32;python_version<"3.11". Because Beam master (20260831) bumped Bigtable to >=2.42.0 (which requires protobuf>=6.33.5), these mutually exclusive ranges force pip into an aggressive backtracking loop on every worker node. Even on standard default containers without any custom image, pip resolves the conflict by downgrading Beam to 2.75.0 and pyarrow to 23.0.1, followed by compiling Beam master from source on the worker VM. On Python 3.11, this regression vanishes completely because tensorflow-metadata drops the upper bound (protobuf>=4.25.2;python_version>="3.11"), allowing the container’s pre-installed protobuf 6.33.6 to satisfy both libraries and resolve dependencies cleanly in seconds without touching Beam or PyArrow.

    How 3.10 Eventually Still Works on Head/Master (Just Slower)

    The pipeline on Python 3.10 does not actually fail and still runs on master/head code. This is because Gradle's tftTests task automatically stages local Beam checkout as an --extra_package (apache-beam.tar.gz). After pip finishes downgrading to 2.75.0 during the requirements phase, the worker boot daemon immediately runs pip install --force-reinstall --no-deps apache-beam.tar.gz. This successfully replaces 2.75.0 with local master build (2.78.0.dev0). However, it runs much slower because:

    1. Every worker node spends over 4 minutes of 100% CPU compiling Beam’s Cython/C++ extensions from source during boot.
    2. The transient 2.75.0 resolution permanently leaves behind older companion packages in the final environment, notably pyarrow 23.0.1 instead of 25.0.1.

    Proposed Solution: Bump gradle.properties to 3.11

    We should update sdks/python/test-suites/gradle.properties:

    -# TFX_BSL is not yet supported on Python 3.10.
-dataflow_cloudml_benchmark_tests_py_versions=3.10
+# Run cloudml TFT benchmark tests on Python 3.11 to avoid Python 3.10 protobuf<=6.32 cap
+dataflow_cloudml_benchmark_tests_py_versions=3.11

    How Gradle Passes It to the Test:

    1. Task Delegation: When you run ./gradlew :sdks:python:test-suites:dataflow:tftTests, dataflow/build.gradle iterates over dataflow_cloudml_benchmark_tests_py_versions and adds :sdks:python:test-suites:dataflow:py311:tftTests to its dependsOn list.
    2. Environment Configuration: The py311 subproject sets pythonVersion = '3.11', which configures Gradle to:
      • Build the manylinux Python 3.11 Beam wheel (bdistPy311linux).
      • Provision a local Python 3.11 virtual environment for the test runner.
      • Automatically tell DataflowRunner to use the official Python 3.11 worker harness image (beam_python3.11_sdk:beam-master-20260831), bypassing the protobuf clash and avoiding the 4-minute on-worker compilation entirely.

  20. damccorm commented on Sep 29, 2026

    @damccorm
    damccorm
    on Sep 29, 2026
    Contributor

    Removing the release blocker, thanks for the investigation!

  21. damccorm
    removed this from the 2.77.0 Release milestone on Sep 29, 2026
  22. stankiewicz
    removed
    awaiting triage
    on Sep 29, 2026
  23. tvalentyn
    closed this as completedin #40333on Sep 29, 2026
  24. github-actions
    added this to the 2.78.0 Release milestone on Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    perf-alertAutomatically filed performance-related alerts.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions