Repository navigation
Conversation
|
I checked the macOS Python 3.14 failure. Could someone with rerun access retry the macOS job? My fork account cannot rerun upstream Actions. Thanks! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40001 +/- ##
=============================================
+ Coverage 56.16% 79.75% +23.59%
=============================================
Files 1124 487 -637
Lines 178336 82280 -96056
Branches 1489 0 -1489
=============================================
- Hits 100170 65626 -34544
+ Misses 75645 16654 -58991
+ Partials 2521 0 -2521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
📋 Review Summary
This PR resolves a bug in Python's BigQuery FILE_LOADS sink where dynamic destinations returning different string representations of the same table (e.g., project:dataset.table vs. dataset.table) grouped files separately by their raw destination string, but hashed the normalized TableReference when generating load-job IDs. By hashing bigquery_tools.get_hashable_destination(destination) in TriggerLoadJobs, load-job IDs now match the file-grouping identity and avoid silent data omission caused by 409 job-ID collisions.
🔍 General Feedback
- The fix in
bigquery_file_loads.pyis minimal, targeted, and preserves existing job IDs for canonicalproject:dataset.tabledestinations while maintaining deterministic retry behavior. - Unit test coverage in
bigquery_file_loads_test.pythoroughly verifies load job distinctness, retry idempotency, callback arguments, and downstreamTriggerCopyJobswrite-disposition behavior (WRITE_TRUNCATEvsWRITE_APPEND).
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Python BigQuery
FILE_LOADScan silently omit files when dynamic destinations use different spellings of the same table, such asproject:dataset.tableanddataset.table. File grouping keeps those strings separate, but load-job IDs hash the resolved table and collide. BigQuery then treats a later submission as an existing job.This change uses the destination identity from file grouping to generate load-job IDs, so equivalent destination spellings do not collide. Retries remain deterministic. Callback arguments and copy write-disposition handling are preserved. The tox change applies Beam's existing
GRPC_ENABLE_FORK_SUPPORT=0workaround before Python imports gRPC.The FILE_LOADS unit suite passed with fake jobs and stubbed credential discovery. No live BigQuery service was used. Run the Python cloud unit suite with:
Job IDs change for noncanonical destination spellings. Replaying unfinished work across an in-place SDK update can resubmit completed work and duplicate rows. Canonical
project:dataset.tabledestinations keep their existing IDs, and retries within either version remain stable.addresses #123orfixes #456).CHANGES.mdwith noteworthy changes.See CONTRIBUTING.md for contribution guidelines.