Skip to content

[Python] Keep BigQuery load jobs distinct for table aliases - #40001

Merged
tvalentyn merged 4 commits into
apache:masterfrom
bvolpato:bvolpato/python-bigquery-destination-job-ids
Oct 22, 2026
Merged

tvalentyn merged 4 commits into
apache:masterfrom
bvolpato:bvolpato/python-bigquery-destination-job-ids

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Python BigQuery FILE_LOADS can silently omit files when dynamic destinations use different spellings of the same table, such as project:dataset.table and dataset.table. File grouping keeps those strings separate, but load-job IDs hash the resolved table and collide. BigQuery then treats a later submission as an existing job.

This change uses the destination identity from file grouping to generate load-job IDs, so equivalent destination spellings do not collide. Retries remain deterministic. Callback arguments and copy write-disposition handling are preserved. The tox change applies Beam's existing GRPC_ENABLE_FORK_SUPPORT=0 workaround before Python imports gRPC.

The FILE_LOADS unit suite passed with fake jobs and stubbed credential discovery. No live BigQuery service was used. Run the Python cloud unit suite with:

./gradlew :sdks:python:test-suites:tox:py312:testPy312Cloud

Job IDs change for noncanonical destination spellings. Replaying unfinished work across an in-place SDK update can resubmit completed work and duplicate rows. Canonical project:dataset.table destinations keep their existing IDs, and retries within either version remain stable.


  • If this contribution is large, please file an Apache Individual Contributor License Agreement.
  • Replace these instructions with a description of your change and why you have made it.
  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.

See CONTRIBUTING.md for contribution guidelines.

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

I checked the macOS Python 3.14 failure. JavaJarServerTest.test_classpath_jar fails because the Java subprocess exits with SIGABRT; that test and its implementation are unchanged by this PR. The FILE_LOADS tests are skipped in that CI lane because it has no GCP dependencies. All 38 FILE_LOADS unit tests pass locally with those dependencies installed.

Could someone with rerun access retry the macOS job? My fork account cannot rerun upstream Actions. Thanks!

@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.75%. Comparing base (76f6e6d) to head (48a3f65).
⚠️ Report is 216 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40001       +/-   ##
=============================================
+ Coverage     56.16%   79.75%   +23.59%     
=============================================
  Files          1124      487      -637     
  Lines        178336    82280    -96056     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65626    -34544     
+ Misses        75645    16654    -58991     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 79.75% <ø> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bvolpato
bvolpato marked this pull request as ready for review October 18, 2026 01:13

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves a bug in Python's BigQuery FILE_LOADS sink where dynamic destinations returning different string representations of the same table (e.g., project:dataset.table vs. dataset.table) grouped files separately by their raw destination string, but hashed the normalized TableReference when generating load-job IDs. By hashing bigquery_tools.get_hashable_destination(destination) in TriggerLoadJobs, load-job IDs now match the file-grouping identity and avoid silent data omission caused by 409 job-ID collisions.

🔍 General Feedback

  • The fix in bigquery_file_loads.py is minimal, targeted, and preserves existing job IDs for canonical project:dataset.table destinations while maintaining deterministic retry behavior.
  • Unit test coverage in bigquery_file_loads_test.py thoroughly verifies load job distinctness, retry idempotency, callback arguments, and downstream TriggerCopyJobs write-disposition behavior (WRITE_TRUNCATE vs WRITE_APPEND).

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @tvalentyn for label python.
Backup reviewer: shunping (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@tvalentyn tvalentyn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Bruno!

@tvalentyn
tvalentyn merged commit 9f5f83e into apache:master Oct 22, 2026
91 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants