Skip to content

ci: Replace per-run flaky issues with a weekly report - #24707

Merged
nicohrubec merged 11 commits into
developfrom
feat/weekly-flaky-test-report
Sep 25, 2026
Merged

nicohrubec merged 11 commits into
developfrom
feat/weekly-flaky-test-report

Conversation

@nicohrubec

@nicohrubec nicohrubec commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Replace the flaky test issue workflow with a weekly report so that we can focus on the most severe flakes instead of being flooded with issues from single run failures. This is fully vibed but works as follows:

  • Report all failing tests from the past seven days on develop, sorted by affected runs, with every run linked.
  • Deduplicate matrix variants and reruns.
  • Link each test to a prefilled flaky test issue template.
  • Run Mondays at 07:00 Europe/Vienna and support manual runs.
  • Move normalization helpers into the weekly reporter.
  • Remove per-run issue creation, its script and template, and its token permissions.

Example run: https://github.com/getsentry/sentry-javascript/actions/runs/36121753520

I'll also set up a weekly reminder for the team channel, maybe we can then look at this as part of triage.

Closes #24691

nicohrubec and others added 2 commits September 24, 2026 18:39
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 24c6931. Configure here.

Comment thread scripts/weekly-flaky-tests.mjs
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size % Change Change
@sentry/browser 29.24 kB - -
@sentry/browser - with treeshaking flags 27.5 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 27.4 kB - -
@sentry/browser (incl. Tracing) 51.15 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 51.17 kB - -
@sentry/browser (incl. Tracing, Profiling) 54.18 kB - -
@sentry/browser (incl. Tracing, Replay) 90.76 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 79.86 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 95.46 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 108.41 kB - -
@sentry/browser (incl. Feedback) 46.76 kB - -
@sentry/browser (incl. sendFeedback) 34.3 kB - -
@sentry/browser (incl. FeedbackAsync) 39.41 kB - -
@sentry/browser (incl. Metrics) 30.25 kB - -
@sentry/browser (incl. Logs) 30.51 kB - -
@sentry/browser (incl. Metrics & Logs) 31.18 kB - -
@sentry/react 31 kB - -
@sentry/react (incl. Tracing) 53.45 kB - -
@sentry/vue 36.74 kB - -
@sentry/vue (incl. Tracing) 53.7 kB - -
@sentry/svelte 29.26 kB - -
CDN Bundle 30.93 kB - -
CDN Bundle (incl. Tracing) 51.69 kB - -
CDN Bundle (incl. Logs, Metrics) 33.2 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 53.66 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 73.92 kB - -
CDN Bundle (incl. Tracing, Replay) 89.28 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 91.25 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 95.45 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 97.42 kB - -
CDN Bundle - uncompressed 91.4 kB - -
CDN Bundle (incl. Tracing) - uncompressed 153.77 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 97.97 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 159.73 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 227.54 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 273.5 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 279.44 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 287.2 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 293.13 kB - -
@sentry/nextjs (client) 55.77 kB - -
@sentry/sveltekit (client) 51.59 kB - -
@sentry/core/server 39.95 kB - -
@sentry/core/browser 13.63 kB - -
@sentry/node 133.9 kB +0.01% +6 B 🔺
@sentry/node/import (ESM hook with diagnostics-channel injection) 82.43 kB - -
@sentry/node - without tracing 90.46 kB +0.02% +14 B 🔺
@sentry/node - without channel injection 112.42 kB +0.01% +11 B 🔺
@sentry/aws-serverless 98.72 kB +0.01% +4 B 🔺
@sentry/cloudflare (withSentry) - minified 206.49 kB - -
@sentry/cloudflare (withSentry) 513.73 kB - -

View base workflow run

nicohrubec and others added 3 commits September 24, 2026 19:11
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>
@nicohrubec nicohrubec changed the title chore(ci): Preview weekly flaky test report chore(ci): Replace per-run flaky issues with a weekly report Sep 25, 2026
nicohrubec and others added 3 commits September 25, 2026 11:02
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>
@nicohrubec
nicohrubec marked this pull request as ready for review September 25, 2026 09:35
nicohrubec and others added 3 commits September 25, 2026 11:50
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>
Co-Authored-By: GPT-6 <codex@openai.com>

@JPeer264 JPeer264 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is such an amazing improvement <3
Hope this will work out better

@nicohrubec nicohrubec changed the title chore(ci): Replace per-run flaky issues with a weekly report ci: Replace per-run flaky issues with a weekly report Sep 25, 2026
@nicohrubec
nicohrubec merged commit b7b7e04 into develop Sep 25, 2026
637 of 641 checks passed
@nicohrubec
nicohrubec deleted the feat/weekly-flaky-test-report branch September 25, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rework flaky test detector

3 participants