Repository navigation
Conversation
Kafka supervisors and Kafka lookups could not authenticate to an IAM-enabled Amazon MSK cluster, because neither extension shipped the aws-msk-iam-auth plugin that registers the AWS_MSK_IAM SASL mechanism. Add aws-msk-iam-auth 2.3.9 to both extensions at runtime scope and exclude all of its transitives, so only the plugin jar ships. The AWS SDK v2 it needs is already in lib/ through druid-server and druid-aws-common, and extensions resolve it parent-first, so bundling another copy would put two SDK versions in one JVM. The plugin cannot live in core itself: its callback handlers implement Kafka client interfaces, and kafka-clients is bundled per extension. No code change is needed in either extension. Both pass consumer properties through unfiltered and set the thread context classloader to the extension classloader around consumer construction, which is where Kafka loads the login module and callback handler. The docs list the IAM actions a consumer needs and recommend awsAddDefaultProviders="false" with assume-role, so that a failed assumption does not fall back to the ambient identity. The tests build a consumer through each extension's own factory with the documented properties. Login happens in the constructor, so no broker is needed, and a missing plugin or SDK artifact fails the test.
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
The IAM mechanism and callback configuration are wired through both consumer factories, but the wildcard exclusions in both extension POMs drop the AWS SSO provider modules used by aws-msk-iam-auth 2.3.9. That leaves SSO-backed profiles in the advertised default credential chain unable to resolve credentials at runtime; the inline finding identifies the packaging fix.
Reviewed 8 of 8 changed files, including both dependency declarations, both smoke tests, both documentation updates, license metadata, and root dependency management. Static review also covered the surrounding consumer construction and extension classloader paths plus the upstream 2.3.9 provider/dependency behavior.
Narrow validation: git diff --check e855cd9c3316710361c7f2cf0b1b8331141c20ad ebf673d4c13d903eaccbe4995a2308733b7ddc32 passed. No broad builds or test suites were run, per the requested static-review scope.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
| <scope>runtime</scope> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> |
There was a problem hiding this comment.
[P2] Default-provider SSO profiles fail at runtime
Finding: Both extension POMs exclude every transitive of aws-msk-iam-auth 2.3.9, but this release's ProfileCredentialsProvider loads software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory for profiles using SSO fields and declares the sso and ssooidc modules explicitly. druid-aws-common in this tree supplies the shared auth, regions, and STS modules but not those SSO modules, so a user relying on the default AWS provider chain with an SSO-backed profile cannot resolve credentials and the Kafka consumer fails before authenticating.
Suggestion: Add compatible AWS SDK sso and ssooidc runtime modules to the shared classpath, or retain these plugin transitives in both extension packaging paths, and cover an SSO-backed profile in a smoke test.
There was a problem hiding this comment.
Thanks, this is deliberate. The gap is not specific to this PR. Druid's own chain (AWSCredentialsUtils.defaultAWSCredentialsProviderChain, used by S3 and Kinesis) fails on the same sso-session profile with the same "the 'sso' service module must be on the class path" error, because lib/ lacks these modules.
I had Claude look up some specifics around how AWS SSO works:
SDK SSO profiles (
sso_session,sso_start_url) use a token from a user's IAM Identity Center sign-in, andssooidconly refreshes that user token. Workloads use other sources: web identity federation (OIDC token via STSAssumeRoleWithWebIdentity, as with IRSA),credential_process(for example IAM Roles Anywhere), instance profiles, pod identity and task roles. I verified that web identity andcredential_processresolve with the shippedlib/. No issue has asked for SSO support, so I left these modules out.Keeping the plugin's transitives in the Kafka extensions would not fix it.
ProfileCredentialsUtilsinlib/loadsSsoProfileCredentialsProviderFactoryby name from the thread context classloader or its own. Credentials resolve during the SASL handshake on the polling thread, outside the context-classloader switch around consumer construction, so an extension copy is never found. In a reproduction with an extension classloader over the shippedlib/, it worked only when the context classloader was the extension's.
That said, users who need SSO can add sso, ssooidc and aws-json-protocol to lib/ in their own Druid install. sso needs aws-json-protocol; without it, resolution fails with NoClassDefFoundError. I verified this layout for both the plugin and Druid's chain. We could always add them to druid-aws-common if there is demand (in a separate PR).
There was a problem hiding this comment.
Follow-up assessment
Thanks for the clarification. I rechecked the current head across all 8 changed files, including both extension POMs and Druid's existing AWSCredentialsUtils.defaultAWSCredentialsProviderChain. The missing SSO modules are already absent from the shared druid-aws-common classpath, and this PR does not change that path, so the SSO limitation is not a PR-caused regression. I am withdrawing the P2 finding; no further action is needed on this thread.
Reviewed 8 of 8 changed files; this follow-up has 0 remaining findings. Static git diff --check e855cd9c3316710361c7f2cf0b1b8331141c20ad ebf673d4c13d903eaccbe4995a2308733b7ddc32 passed; no tests or builds were run.
…-iam-auth # Conflicts: # pom.xml
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
⚪ Review incomplete
Review deferred for apache/druid#20471: the current head feature/kafka-msk-iam-auth (f245cf20b5c650d8741ecce8418942eda1bcf356) conflicts with the target branch master. Please resolve the conflicts with master and push the resolved head so the code can be reviewed later.
No files were reviewed because the conflicting head was not prepared.
This is an automated review by Codex GPT-5.6-Luna(max)
…-iam-auth # Conflicts: # pom.xml
2 participants
Description
Kafka supervisors and Kafka lookups cannot authenticate to an Amazon MSK cluster that uses IAM access control, because neither extension ships the aws-msk-iam-auth plugin that provides the
AWS_MSK_IAMSASL mechanism. This PR bundles the plugin in both extensions, so IAM authentication works with configuration alone.This PR replaces #17355 from @zachjsh, which took the same approach (but Druid move to AWS SDK v2 since) but updates the plugin to 2.3.9 and adds tests, docs and licence entries. Imply already runs the plugin internally, and the jar is 60 KB.
Release note
Kafka supervisors and Kafka lookups can now connect to Amazon MSK clusters that use IAM access control. The
druid-kafka-indexing-serviceanddruid-kafka-extraction-namespaceextensions bundle the Amazon MSK IAM authentication library, so setsasl.mechanismtoAWS_MSK_IAMand the related consumer properties, without installing extra jars. Druid picks up credentials from the default AWS provider chain and can assume a role.Closes #17355.
This PR has: