Repository navigation
Performance Regression or Improvement: test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec:runtime_sec #40067
Description
Performance change found in the
test: test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec for the metric: runtime_sec.
For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.
Test description: TFT Criteo test on 10 GB data with no Reshuffle.
Test link - Test link
timestamp: Mon Sep 7 23:19:16 2026, metric_value: 2998.61
timestamp: Sun Sep 6 23:09:38 2026, metric_value: 3075.77
timestamp: Sat Sep 5 23:39:23 2026, metric_value: 3069.05
timestamp: Fri Sep 4 23:08:06 2026, metric_value: 2910.89
timestamp: Thu Sep 3 23:05:58 2026, metric_value: 3123.24
timestamp: Wed Sep 2 23:00:42 2026, metric_value: 3080.73 <---- Anomaly
timestamp: Tue Sep 1 23:23:12 2026, metric_value: 2366.35
timestamp: Mon Aug 31 22:52:36 2026, metric_value: 2537.10
timestamp: Mon Aug 31 20:43:04 2026, metric_value: 2407.92
timestamp: Sun Aug 30 22:39:01 2026, metric_value: 2299.40
timestamp: Sat Aug 29 23:00:03 2026, metric_value: 2421.35
timestamp: Fri Aug 28 23:58:09 2026, metric_value: 2360.65
timestamp: Fri Aug 28 01:54:40 2026, metric_value: 2298.42
timestamp: Wed Aug 26 23:28:17 2026, metric_value: 2362.46
timestamp: Wed Aug 26 12:11:02 2026, metric_value: 2356.62
timestamp: Wed Aug 19 22:04:50 2026, metric_value: 2308.39
Activity
aIbrahiim commentedon Sep 10, 2026
I investigated CloudML Dataflow runs from 30 August to 9 September and here's my findings:
Deps didn’t move (Beam 2.77.0.dev0, TFT 1.21, TF 2.21) and on 2 September the container went from beam-master-20260817 to …0831 (#39958), and both no_shuffle and criteo_10GB stayed around 25 to 30% slower after that and it feels like a real regression tied to that container update