Skip to content

[YAML] Clean up BigQuery datasets when fixtures fail - #40863

Draft
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/cleanup-yaml-bigquery-datasets
Draft

bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/cleanup-yaml-bigquery-datasets

Conversation

@bvolpato

Copy link
Copy Markdown
Contributor

YAML integration fixtures use ExitStack. If a later fixture fails during setup, the exception reaches the BigQuery fixture at yield and skips its dataset deletion. In this run, datasets were created before Bigtable setup failed, and their cleanup was never reached.

Move dataset cleanup into finally after creation, covering both optional table setup and errors from the test or another fixture. The fixture now calls deletion as the stack unwinds, using the existing request to delete the dataset and its contents.

Focused regression cases failed against the original helper because deletion was never called, then passed with the fix. They use the real Beam and BigQuery API modules with mocked cloud clients. Formatting and lint checks passed, including a check that missing optional dependencies skip the tests. No live cloud resources were used.

@github-actions github-actions Bot added python yaml size/m M: 50-249 lines changed labels Oct 28, 2026
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.11111% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.08%. Comparing base (0f1ac43) to head (f0f3cea).
⚠️ Report is 16 commits behind head on master.

Files with missing lines Patch % Lines
sdks/python/apache_beam/yaml/integration_tests.py 61.11% 7 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #40863       +/-   ##
===========================================
+ Coverage        0   80.08%   +80.08%     
===========================================
  Files           0      494      +494     
  Lines           0    83574    +83574     
===========================================
+ Hits            0    66927    +66927     
- Misses          0    16647    +16647     
Flag Coverage Δ
python 80.08% <61.11%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bvolpato
bvolpato force-pushed the bvolpato/cleanup-yaml-bigquery-datasets branch from 6ada839 to f0f3cea Compare October 28, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python size/m M: 50-249 lines changed yaml

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant