[mxtymoshyk]

Fixes #27069.

Adds integer-range partitioning to BigQueryIO.Write in the Java SDK. Python already supports it through additional_bq_parameters.

Usage

BigQueryIO.writeTableRows()
    .to("project:dataset.table")
    .withSchema(schema)
    .withRangePartitioning(
        new RangePartitioning()
            .setField("customer_id")
            .setRange(new RangePartitioning.Range().setStart(0L).setEnd(1000L).setInterval(10L)));

With to(SerializableFunction) or to(DynamicDestinations), set the RangePartitioning on each returned TableDestination and call withRangePartitioning() with no arguments. A custom DynamicDestinations must also return TableDestinationCoderV4 from getDestinationCoder().

Update compatibility: drain and restart required

Turning on range partitioning switches the destination coder to a new TableDestinationCoderV4. V4 writes the V3 encoding followed by the range partitioning, and it can't read state written by V2 or V3. A running pipeline can't be updated in place to turn this on. Drain it and start a new one.

Pipelines that don't call withRangePartitioning keep TableDestinationCoderV2 or V3, so their encoding is unchanged and in-place updates keep working. V4 is registered in the Dataflow runner's DefaultCoderCloudObjectTranslatorRegistrar.

Changes

• TableDestination carries an optional range partitioning as a JSON string, the same way it stores time partitioning and clustering.
• Table creation (CreateTableHelpers) and load jobs (WriteTables, UpdateSchemaDestination) pass range partitioning to BigQuery. This covers FILE_LOADS, STREAMING_INSERTS and the Storage Write API.
• FILE_LOADS temp tables copy the range partitioning of an existing destination table, so copy jobs into it succeed. This is the Java counterpart of Preserve partitioning on temp FILE_LOADS tables #38833.
• PrepareWrite fails with an error if a destination has range partitioning but the destination coder would drop it. Without the check, the table would be created without partitioning and no error.
• Setting both withTimePartitioning and withRangePartitioning fails at construction, since a BigQuery table can have only one.
• BigQueryIOTranslation adds json_range_partitioning to the transform upgrade schema, read only when upgrading from 2.78.0 or later. If this misses the 2.78.0 cut, I'll move that gate and the CHANGES.md entry to 2.79.0.

Tests

• Unit: TableDestinationCoderV4Test, plus new cases in BigQueryIOWriteTest and BigQueryIOTranslationTest.
• Integration: BigQueryRangePartitioningIT writes to real BigQuery: a single table, a table function with clustering, the Storage Write API, and temp tables into an existing range-partitioned table. It runs in beam_PreCommit_Java_GCP_IO_Direct_BigQuery. I haven't run it against my own project, so that job is its first real run.

[github-actions]

📋 Review Summary

This PR adds support for BigQuery integer range partitioning (RangePartitioning) across BigQueryIO.Write, TableDestination, DynamicDestinations, and the underlying write transforms (Batch Loads, Streaming Inserts, and Storage Write API). It also introduces TableDestinationCoderV4 while carefully preserving update compatibility for existing streaming pipelines via TableDestinationCoderV3 when range partitioning is not enabled.

🔍 General Feedback

• The backward compatibility design around TableDestinationCoderV3 vs TableDestinationCoderV4 and fail-fast validation in PrepareWrite is well thought out.
• In TableDestination, equals() and hashCode() omit jsonClustering, which causes destination collisions when TableDestination is used as a map/grouping key and breaks CoderProperties equality checks when clustering is populated.

---

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

💬 Inline Findings

• sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java (line 274): 🟠 High: equals (and hashCode on line 280) omits this.jsonClustering. Because TableDestination is used as a Map key in WriteBundlesToFiles and WritePartition and as a GroupByKey key across BigQuery write transforms, two TableDestination instances with the same tableSpec but different jsonClustering configurations will be treated as equal. Additionally, TableDestinationCoderV3 and TableDestinationCoderV4 encode jsonClustering, so omitting it from equals/hashCode violates Beam's Coder equality contract.

Please include Objects.equals(this.jsonClustering, other.jsonClustering) in equals and jsonClustering in Objects.hash(...).

        && Objects.equals(this.jsonClustering, other.jsonClustering) && Objects.equals(this.jsonRangePartitioning, other.jsonRangePartitioning);

• sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java (line 280): 🟠 High: Include jsonClustering in hashCode() to match equals() and the fields serialized by TableDestinationCoderV3 / TableDestinationCoderV4.

    return Objects.hash(tableSpec, tableDescription, jsonTimePartitioning, jsonClustering, jsonRangePartitioning);

[mxtymoshyk]

On the automated review's TableDestination.equals/hashCode finding:

BEAM-5191 (2018) added clustering without adding it to equals/hashCode, so this predates the PR. This PR adds range partitioning to both, next to time partitioning.

The TableDestination coders don't override consistentWithEquals(), so GroupByKey compares destinations by their encoded bytes, which include clustering. The in-memory maps in WriteBundlesToFiles and WritePartition do use equals, but they can only merge destinations for the same table, and a BigQuery table has one clustering spec.

I'd rather not change grouping for existing clustering users in this PR. If reviewers want it changed, I'll open a follow-up issue.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[mxtymoshyk]

assign set of reviewers

The one failing check, beam_PreCommit_Java_GCP_IO_Direct_BigQuery, failed in StorageApiDataTriggeredSchemaUpdateIT, which this PR doesn't touch. BigQuery returned a 503 backendError ("Retrying may solve the problem") while the test was polling for query results. The new BigQueryRangePartitioningIT passed all 4 cases in that same run. Could a committer re-run the failed job?

[github-actions]

Assigning reviewers:

R: @Abacn for label java.
Backup reviewer: kennknowles (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (a8d6afd) to head (2121346).
⚠️ Report is 74 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #40790       +/-   ##
===========================================
- Coverage   80.04%        0   -80.05%     
===========================================
  Files         490        0      -490     
  Lines       82916        0    -82916     
===========================================
- Hits        66373        0    -66373     
+ Misses      16543        0    -16543     

Flag	Coverage Δ
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mxtymoshyk]

I opened #40793 to fix the cause of the StorageApiDataTriggeredSchemaUpdateIT failure above. BigqueryClient.queryUnflattened didn't retry the 503 (#40792).

[mxtymoshyk]

/gemini review