[Python] Don't override the Dataflow endpoint by default by shunping · Pull Request #40568 · apache/beam

shunping · 2026-10-14T03:28:13Z

Problem

GoogleCloudOptions.dataflow_endpoint defaulted to https://dataflow.googleapis.com, and DataflowApplicationClient always passed it to the Dataflow GAPIC clients as ClientOptions(api_endpoint=...). In the GCP client libraries, an explicit api_endpoint takes precedence over GOOGLE_CLOUD_UNIVERSE_DOMAIN, so the default Google universe (GDU) URL was always used. This breaks job submission in non-GDU universes such as TPC.

Changes

--dataflow_endpoint defaults to '' (same as the Go SDK). When unset, no api_endpoint override is passed, so the client library resolves the endpoint (honoring GOOGLE_CLOUD_UNIVERSE_DOMAIN, otherwise dataflow.googleapis.com). An explicit value still overrides as before.
Validator: dataflow_endpoint is now optional; the URL is only validated when provided.
Remove legacy taskrunner_settings from the worker pool. They only applied to the legacy (Runner v1) worker, which Python no longer supports, and hard-coded the GDU URL as base_url. The Go and Java SDKs never set them, and the Dataflow service doesn't echo them back.

Compatibility

Code that reads options.view_as(GoogleCloudOptions).dataflow_endpoint and expects the GDU URL will now get '' when the flag isn't set.

Part of #40569

codecov · 2026-10-14T05:11:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.90%. Comparing base (76f6e6d) to head (5fbce4b).
⚠️ Report is 178 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #40568       +/-   ##
=============================================
+ Coverage     56.16%   79.90%   +23.73%     
=============================================
  Files          1124      488      -636     
  Lines        178336    82382    -95954     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65827    -34343     
+ Misses        75645    16555    -59090     
+ Partials       2521        0     -2521

Flag	Coverage Δ
go	`?`
java	`?`
python	`79.90% <100.00%> (+0.34%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-10-14T13:36:41Z

Assigning reviewers:

R: @tvalentyn for label python.
Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

jrmccluskey

LGTM, I would suggest cloning the PR internally and running TAP against it just as a sanity check before merging if you haven't already

tvalentyn · 2026-10-14T20:58:19Z

          ' at dataflow service endpoint %s',
          e.code,
-          self.google_cloud_options.dataflow_endpoint)
+          self.google_cloud_options.dataflow_endpoint or


can we print the actual attempted endpoint here? is it resolved by now?

Sure, that's a good idea. Changed in the latest code.

tvalentyn · 2026-10-14T20:59:26Z

-            'The URL for the Dataflow API. If not set, the default public URL '
-            'will be used.'))
+            'The URL for the Dataflow API. If not set, the endpoint is '
+            'resolved by the Google Cloud client library, which honors '


the Google Cloud client library

For my education:

Where in the control flow this resolution happens?

Which library is responsible for this resolution?

The control flow happens in Dataflow Client Library. Particularly, it is when we initialize JobsV1Beta3Client, MessagesV1Beta3Client and MetricsV1Beta3Client:

For example, you can find the logic in __init__ of JobsV1Beta3Client below.
https://github.com/googleapis/google-cloud-python/blob/8e640e06da613caa23bb4cc193e839e7ddbb67c7/packages/google-cloud-dataflow-client/google/cloud/dataflow_v1beta3/services/jobs_v1_beta3/client.py#L410

tvalentyn · 2026-10-14T21:01:14Z

-    if dataflow_endpoint is None:
-      return False
-    else:
+    # An unset endpoint means the default Dataflow service endpoint (resolved


nit: we don't have to repeat these comment, seems like a distraction here.

I reworded it slightly to make it less distracting, but I'd prefer to keep it. I think having the context right next to the logic makes it much easier for future readers to follow.

tvalentyn · 2026-10-14T21:02:08Z

-      errors.extend(
-          self._validate_error(self.ERR_MISSING_OPTION, dataflow_endpoint))
-    else:
+    # dataflow_endpoint is optional. When unset, the client library resolves


shunping · 2026-10-14T23:07:54Z

LGTM, I would suggest cloning the PR internally and running TAP against it just as a sanity check before merging if you haven't already

Just verified that TAP is green with the change in this PR.

The default --dataflow_endpoint (GDU URL) was always passed as an api_endpoint override, ignoring GOOGLE_CLOUD_UNIVERSE_DOMAIN (e.g. TPC). Default it to '' so the client library resolves the endpoint, and drop the legacy TaskRunner worker settings that hard-coded the GDU URL.

tvalentyn · 2026-10-15T01:47:10Z

  `jakarta.jms.*`. The module now requires Java 17 at runtime ([#38729](https://github.com/apache/beam/issues/38729)).
 * (Python) `Regex.find_all` with `group=Regex.ALL` now returns every group, as `(match, group1, group2, ...)`, instead of only group 1, and with `outputEmpty=False` it drops a match only when the whole match is empty, matching the `group=0` behavior ([#40399](https://github.com/apache/beam/pull/40399)).
 * (Python) `Regex.all_matches` now returns every group of the regex, instead of failing with a `TypeError` when the regex has no groups and dropping nested or unmatched trailing groups ([#40562](https://github.com/apache/beam/issues/40562)).
+* (Python) `--dataflow_endpoint` now defaults to `''` instead of `https://dataflow.googleapis.com`. When unset, the Google Cloud client library resolves the Dataflow endpoint and honors `GOOGLE_CLOUD_UNIVERSE_DOMAIN`, so Dataflow jobs can be submitted in non-default universes. Code that reads `GoogleCloudOptions.dataflow_endpoint` should handle an empty value ([#40569](https://github.com/apache/beam/issues/40569)).


Suggested change

* (Python) `--dataflow_endpoint` now defaults to `''` instead of `https://dataflow.googleapis.com`. When unset, the Google Cloud client library resolves the Dataflow endpoint and honors `GOOGLE_CLOUD_UNIVERSE_DOMAIN`, so Dataflow jobs can be submitted in non-default universes. Code that reads `GoogleCloudOptions.dataflow_endpoint` should handle an empty value ([#40569](https://github.com/apache/beam/issues/40569)).

* (Python) `--dataflow_endpoint` now defaults to `''` instead of `https://dataflow.googleapis.com`. When unset, the Google Cloud Dataflow client library resolves the Dataflow endpoint and honors `GOOGLE_CLOUD_UNIVERSE_DOMAIN`, so Dataflow jobs can be submitted in non-default universes. Code that reads `GoogleCloudOptions.dataflow_endpoint` should handle an empty value ([#40569](https://github.com/apache/beam/issues/40569)).

I think Google Cloud client library is the official term of that.

https://docs.cloud.google.com/apis/docs/cloud-client-libraries

tvalentyn · 2026-10-15T01:50:30Z

-            'The URL for the Dataflow API. If not set, the default public URL '
-            'will be used.'))
+            'The URL for the Dataflow API. If not set, the endpoint is '
+            'resolved by the Google Cloud client library, which honors '


Suggested change

'resolved by the Google Cloud client library, which honors '

'resolved by the Python Client for Dataflow, which honors '

seems to be the official name from https://docs.cloud.google.com/python/docs/reference/dataflow/latest

See #40568 (comment)

github-actions Bot added python runners labels Oct 14, 2026

shunping marked this pull request as ready for review October 14, 2026 13:35

github-actions Bot requested a review from tvalentyn October 14, 2026 13:36

github-actions Bot added the Next Action: Reviewers label Oct 14, 2026

shunping requested a review from jrmccluskey October 14, 2026 13:58

jrmccluskey approved these changes Oct 14, 2026

View reviewed changes

shunping force-pushed the tpc-1 branch from a560e49 to c9ebd71 Compare October 14, 2026 14:50

tvalentyn reviewed Oct 14, 2026

View reviewed changes

shunping added 5 commits October 14, 2026 21:29

Reformat

8e42f4b

Edited CHANGES.md

53a18ab

Print out the actual endpoint where the job request is sent to.

b2715db

Reword comments

5fbce4b

shunping force-pushed the tpc-1 branch from befa4f1 to 5fbce4b Compare October 15, 2026 01:30

tvalentyn reviewed Oct 15, 2026

View reviewed changes

shunping merged commit 1a017df into apache:master Oct 15, 2026
87 checks passed

shunping deleted the tpc-1 branch October 15, 2026 13:05

	'resolved by the Google Cloud client library, which honors '
	'resolved by the Python Client for Dataflow, which honors '

Conversation

shunping commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Compatibility

Uh oh!

codecov Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

jrmccluskey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping commented Oct 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shunping commented Oct 14, 2026 •

edited

Loading

codecov Bot commented Oct 14, 2026 •

edited

Loading