Skip to content

[Java] Honor non-default universe domains in GcsUtilV2 - #40796

Open
shunping wants to merge 1 commit into
apache:masterfrom
shunping:tpc-8
Open

shunping wants to merge 1 commit into
apache:masterfrom
shunping:tpc-8

Conversation

@shunping

@shunping shunping commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #40569

Problem

Since GcsUtil V2 (java-storage) became the default (#40713), Java GCS ignores GOOGLE_CLOUD_UNIVERSE_DOMAIN. Unlike the Google API client used by GcsUtil V1 (and by the Dataflow client, #40672), java-storage only takes the universe domain from StorageOptions#setUniverseDomain. Without --gcsEndpoint, every request goes to storage.googleapis.com, even with credentials for another universe.
On Trusted Partner Cloud this fails right at pipeline construction:

Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://...
...
401 Unauthorized
GET https://storage.googleapis.com/storage/v1/b/<bucket>?fields=name&projection=full

Fix

When --gcsEndpoint is not set, GcsUtilV2 resolves the universe domain and sets it on StorageOptions:

  1. GOOGLE_CLOUD_UNIVERSE_DOMAIN, if set; otherwise
  2. the credentials' universe domain.
    If neither is set, or the value is googleapis.com, nothing changes. An explicit --gcsEndpoint still takes precedence. This mirrors the Python fix ([Python] Fix GCS client creation in non-default universes #40674).

java-storage ignores GOOGLE_CLOUD_UNIVERSE_DOMAIN and the credentials'
universe domain, so GcsUtilV2 always targeted storage.googleapis.com.
When no gcsEndpoint is set, resolve the universe domain (env var, then
credentials) and set it on StorageOptions. Add unit tests.
@shunping
shunping marked this pull request as ready for review October 26, 2026 01:59

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves an issue in GcsUtilV2 where non-default universe domains (such as Trusted Partner Cloud) were ignored when --gcsEndpoint was not explicitly configured. It properly resolves the universe domain from GOOGLE_CLOUD_UNIVERSE_DOMAIN or the configured Credentials and applies it to StorageOptions, while preserving --gcsEndpoint precedence and default googleapis.com behavior.

🔍 General Feedback

  • The resolution precedence (--gcsEndpoint > GOOGLE_CLOUD_UNIVERSE_DOMAIN > Credentials#getUniverseDomain() > default) is clean and consistent with the Google API client library behavior and the Python SDK implementation (#40674).
  • Unit tests in GcsUtilV2Test thoroughly cover environment variable precedence, credential fallback, default universe handling, error handling on getUniverseDomain(), and --gcsEndpoint override.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @damccorm for label java.
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (a8d6afd) to head (357619d).
⚠️ Report is 75 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #40796       +/-   ##
===========================================
- Coverage   80.04%        0   -80.05%     
===========================================
  Files         490        0      -490     
  Lines       82916        0    -82916     
===========================================
- Hits        66373        0    -66373     
+ Misses      16543        0    -16543     
Flag Coverage Δ
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant