[Bug]:KafkaIO OAUTHBEARER auth against GCP Managed Service for Apache Kafka broken starting 2.69.0 (works on 2.68.0)

What happened?

KafkaIO OAUTHBEARER auth against GCP Managed Service for Apache Kafka broken starting 2.69.0 (works on 2.68.0)

Using Python ReadFromKafka with sasl.mechanism=OAUTHBEARER and
sasl.login.callback.handler.class=com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler
to connect to GCP Managed Service for Apache Kafka from a Dataflow streaming job
(DataflowRunner, --experiments=use_runner_v2), the Kafka consumer fails 100% of
connection attempts with:

org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
  at org.apache.beam.sdk.io.kafka.KafkaIO$Read$GenerateKafkaSourceDescriptor...

with worker logs showing, for every connection attempt:

[Consumer clientId=..., groupId=...] Connection to node -1 (<broker>:9092) terminated
during authentication. This may happen due to any of the following reasons:
(1) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic),
(2) Transient network issue.

This is not a network/firewall issue — same VPC/subnetwork/service-account/broker
worked correctly on Beam 2.68.0 and earlier.

Bisection result

Tested directly against a live GCP Managed Kafka cluster (Dataflow, DataflowRunner,
identical pipeline code, identical JDK build (17.0.18+8-Debian-1deb12u1, controlled via
a pinned custom SDK container image so the JDK is not a variable), identical
kafka-clients (3.9.0) and identical GcpLoginCallbackHandler.class bytecode
(SHA256-verified) across every version tested:

Beam version	Result
2.67.0	✅ Works
2.68.0	✅ Works (confirmed live data flowing)
2.69.0	❌ Fails — 100% auth termination
2.71.0	❌ Fails
2.75.0	❌ Fails

The break happens exactly between the 2.68.0 → 2.69.0 release.

Dependency diff, 2.68.0 → 2.69.0

Diffing every META-INF/maven/*/pom.properties entry bundled in
beam-sdks-java-io-expansion-service-{2.68.0,2.69.0}.jar, after excluding
unrelated file-format IO connectors (ORC/Parquet/JTS, bundled in the same
shadowJar but irrelevant to Kafka), the changes touching the HTTP/auth path are:

org.apache.httpcomponents.client5:httpclient5   5.4.3 -> 5.5
io.confluent:kafka-schema-registry-client       7.6.1 -> 7.5.5   (downgrade)
io.confluent:common-utils                       7.6.1 -> 7.5.5   (downgrade)
net.bytebuddy:byte-buddy                        1.14.12 -> 1.17.7

google-auth-library-oauth2-http and google-auth-library-credentials are
unchanged across this specific boundary (both 1.37.1 / 1.30.0), so this
looks like a transport-layer (httpclient5) or Confluent schema-registry-client
regression rather than a google-auth-library version bump, despite
google-auth-library-oauth2-http moving significantly (1.36.0 → 1.47.0) across
the wider 2.67.0 → 2.75.0 range.

What's been ruled out

kafka-clients version: identical (3.9.0) at every version tested
GcpLoginCallbackHandler.class: byte-identical (SHA256-verified) at every version tested
managed-kafka-auth-login-handler (Google's own artifact): identical, 1.0.5
JDK patch version: identical (17.0.18+8-Debian-1deb12u1), controlled via a pinned
custom SDK container image built via multi-stage COPY from a known-good image,
ruling out JDK/TLS-default drift as a factor
Network/firewall/subnetwork/service account/IAM: identical config across all tests;
ruled out because the same infra with only the Beam SDK version changed reproduces
the failure deterministically

Environment

Runner: DataflowRunner, --experiments=use_runner_v2,
--dataflow_service_options=streaming_mode_at_least_once
SDK: Python 3.12, apache-beam[gcp]
Kafka: GCP Managed Service for Apache Kafka, SASL_SSL + OAUTHBEARER
Auth: com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler (Application
Default Credentials via the pipeline's attached service account)

Reproduction

Minimal consumer_config used:

{
    "bootstrap.servers": "<managed-kafka-bootstrap>:9092",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "OAUTHBEARER",
    "sasl.login.callback.handler.class": "com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler",
    "sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;",
}

Passed to ReadFromKafka(consumer_config=consumer_config, topics=[...]) in a
streaming pipeline submitted with --sdk_container_image pinned to a custom image
per Beam version tested (JDK controlled/identical across all).

Happy to share the full bisection harness (build + throwaway-consumer-group test
script) if useful for reproducing this quickly across future release candidates.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

I'll investigate this regression. The bisection is very helpful. Looking at the dependency diff between 2.68.0 and 2.69.0, the httpclient5 upgrade (5.4.3→5.5) is the most likely culprit — it's a transitive dependency that affects the Google auth library's HTTP transport layer.

@johnjcasey

@johnjcasey does this ring any bells for you?

@c-broggy

@c-broggy can you run failing pipeline with addiitional parameter : --sdk_harness_log_level_overrides='{"org.apache.kafka.common.network":"DEBUG","org.apache.kafka.common.security":"DEBUG"}'

Kafka's selector has debugs on disconnect that would help troubleshoot that.

I'm not sure if were testing with correct way of pinning, give it a try with
--sdk_harness_container_image_overrides=".*java.*,gcr.io/cloud-dataflow/v1beta3/beam_java17_sdk:2025-10-27 jdk-17.0.16 ; 2026-08-31 is jdk-17.0.20

I've tested freshly created pipeline on beam 2.75, runner v2, python 3.10 and 3.12 with managed kafka and gcr.io/cloud-dataflow/v1beta3/beam_java21_sdk:2.75.0

if you have minimal pipeline - pipeline + reqs + startup script if you have some customization, this would help reproduce that.

@stankiewicz

@stankiewicz this is what I have found :
Enabled -- sdk_harness_log_level_overrides='{"org.apache.kafka.common.network":"DEBUG","org.apache.kafka.common.security":"DEBUG"}' on the default (unpinned) beam_java25_sdk harness, and the debug logging surfaces the actual cause behind the TimeoutException: Timeout expired while fetching topic metadata:

java.lang.UnsupportedOperationException: getSubject is not supported
at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:100)
at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:93)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:536)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:335)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:276)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:563)
...
at org.apache.kafka.clients.consumer.internals.TopicMetadataFetcher.getTopicMetadata(TopicMetadataFetcher.java:102)
at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1434)
at org.apache.beam.sdk.io.kafka.KafkaIO$Read$GenerateKafkaSourceDescriptor.processElement(KafkaIO.java:2169)
The OAUTHBEARER token fetch itself succeeds ("Successfully logged in."), but the SASL handshake then fails immediately with the exception above, the connection is torn down, and it retries in a loop — which eventually surfaces as the metadata-fetch timeout everyone's seeing.

This looks like a JDK-version compatibility issue, not anything cluster- or config-specific. JDK 24+ finalized JEP 486 (permanently disabling the Security Manager), which turns Subject.getSubject(AccessControlContext) into a hard UnsupportedOperationException. Kafka's OAuthBearerSaslClientCallbackHandler still calls that legacy API to retrieve the current Subject during the SASL handshake — so any environment where the SDK harness runs on JDK 24+ (e.g. beam_java25_sdk) will hit this on every OAUTHBEARER auth attempt against Kafka.

Consistent with that: pinning sdk_harness_container_image_overrides to a harness on JDK 21 or earlier (predates JEP 486) avoids the issue entirely — clean connect and auth, no disconnects, no timeouts. So the fix/workaround is specifically about the JDK version the harness runs on, not the Beam version or SDK harness image name per se — any harness bundling JDK 24+ should reproduce this with SASL/OAUTHBEARER Kafka auth, and any harness on JDK ≤21 should not.

Root Cause

In JDK 24+, JEP 486 permanently disabled the Security Manager, causing calls to Subject.getSubject(AccessControlContext) to throw an unconditional UnsupportedOperationException. Beam's Java expansion service forcibly pins org.apache.kafka:kafka-clients:3.9.0, which still invokes this legacy API inside OAuthBearerSaslClientCallbackHandler during the SASL handshake. When cross-language pipelines execute on an unpinned or JDK 24+ harness (such as beam_java25_sdk), this exception causes repeated authentication failures that surface to the user as a metadata fetch TimeoutException.

Solution

Bump the pinned dependency in sdks/java/io/expansion-service/build.gradle from org.apache.kafka:kafka-clients:3.9.0 to 3.9.1 (or 3.9.2). Kafka 3.9.1 backported KAFKA-17078, which adds the SecurityManagerCompatibility reflective shim to safely fall back to modern Subject.current() on Java 18+ runtimes. Because 3.9.1 is a backward-compatible patch release, it eliminates the crash on newer JDKs without introducing breaking API or protocol changes.

Workaround (Until Solution is Deployed)

Until the expansion service patch is deployed, pipelines should pin the Java SDK worker harness to a JDK 21 (LTS) image where Subject.getSubject is still supported. This can be configured by supplying --sdk_harness_container_image_overrides=".*java.*,gcr.io/cloud-dataflow/v1beta3/beam_java21_sdk:<beam-version>" when launching the pipeline. On JDK 21, the SASL callback handler executes without error and establishes the connection to the Kafka cluster normally.

Alternative Considered but Rejected

Upgrading kafka-clients directly to 4.0.0+ was considered because it natively supports modern JDKs and includes the compatibility shim. However, Kafka 4.0 is a major version release that introduces breaking changes, including the complete removal of ZooKeeper and changes to client configurations and protocol defaults. Upgrading to the 3.9.1 patch release delivers the necessary fix with zero disruption to existing Beam KafkaIO consumers.

Navigation Menu

Repository navigation

[Bug]:KafkaIO OAUTHBEARER auth against GCP Managed Service for Apache Kafka broken starting 2.69.0 (works on 2.68.0) #39789

What happened?

Bisection result

Dependency diff, 2.68.0 → 2.69.0

What's been ruled out

Environment

Reproduction

Issue Priority

Issue Components

Root Cause

Solution

Workaround (Until Solution is Deployed)

Alternative Considered but Rejected

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]:KafkaIO OAUTHBEARER auth against GCP Managed Service for Apache Kafka broken starting 2.69.0 (works on 2.68.0) #39789

Description

What happened?

Bisection result

Dependency diff, 2.68.0 → 2.69.0

What's been ruled out

Environment

Reproduction

Issue Priority

Issue Components

Activity

waterWang commented on Aug 17, 2026

kennknowles commented on Sep 1, 2026

stankiewicz commented on Sep 25, 2026

stankiewicz commented on Sep 25, 2026

stankiewicz commented on Sep 25, 2026

c-broggy commented on Sep 28, 2026

stankiewicz commented on Sep 28, 2026

Root Cause

Solution

Workaround (Until Solution is Deployed)

Alternative Considered but Rejected

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions