[aws] Enable Identity Federation on all agentless policy templates - #20302
[aws] Enable Identity Federation on all agentless policy templates#20302seanrathier wants to merge 3 commits into
Conversation
Remove the hide_in_var_group_options credential_type gates from all 23 agentless inputs and point iac_template_url at the new federated-identity-aws CloudFormation template (cloudbeat#7422), which unconditionally grants the read-only permissions required by every agentless-enabled AWS integration. Users deploy one stack and paste the Role ARN + External ID back into Kibana regardless of which integrations they select. Bumps the package to 7.1.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
89d00a4 to
cc6f9da
Compare
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
andrewkroh
added
Integration:aws
Remove hand-rolled SigV4 transforms (X-Amz-Date header, Authorization header, pagination re-signing) from inspector, securityhub_findings, securityhub_findings_full_posture, and securityhub_insights httpjson streams. Add auth.aws block (matching the guardduty pattern from elastic#19828) so the agent handles AWS auth natively and use_cloud_connectors works. Revert identity federation enablement on aws.config: the CEL program implements SigV4 inline and explicitly rejects non-key credentials; cloud connectors cannot inject temporary creds as template variables, so the option would silently fail at runtime. Restore hide_in_var_group_options until the CEL program is rewritten. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🚀 Benchmarks reportTo see the full report comment with |
andrewkroh
added
the
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
@vera-review-bot review The earlier review attempt on this PR failed with a gateway authentication error during an LLM credential rotation on our side — nothing to do with the changes here. The new credential is in place, so re-triggering the initial review. Comment posted with 🤖 Cursor/Claude Opus 5 under my supervision. |
|
@vera-review-bot review |
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
cc @seanrathier |
|
@vera-review-bot review |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| hide_in_deployment_modes: [default] | ||
| provider: aws | ||
| iac_template_url: https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-cloud-connectors-guardduty-9.4.0.yml¶m_ElasticResourceId=RESOURCE_ID | ||
| iac_template_url: https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-federated-identity-aws-9.4.0.yml¶m_ElasticResourceId=RESOURCE_ID |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/aws/manifest.yml:133
Identity Federation is now selectable for the aws/metrics, aws-cloudwatch and aws-s3 inputs, but none of those stream templates emit use_cloud_connectors (and the metrics/cloudwatch ones drop external_id), so the option renders an unusable config; keep hide_in_var_group_options on those inputs until their templates support it.
Details
This PR removes hide_in_var_group_options: credential_type: [identity_federation] from ~20 policy templates, but only the four httpjson data streams (inspector, securityhub_findings, securityhub_findings_full_posture, securityhub_insights) gained an auth.aws block. use_cloud_connectors exists in exactly five stream templates in the package (the four above plus guardduty/httpjson) — grep confirms no aws/metrics, aws-cloudwatch or aws-s3 template references it.
Concretely, with Identity Federation selected today:
data_stream/ec2_metrics/agent/stream/stream.yml.hbs(and every other aws/metrics template) renders onlyrole_arn:—external_idanduse_cloud_connectorsare never written.data_stream/ec2_logs/agent/stream/aws-cloudwatch.yml.hbslikewise renders onlyrole_arn:.data_stream/guardduty/agent/stream/aws-s3.yml.hbsrendersrole_arnandexternal_idbut nouse_cloud_connectors, yet its input lost the hide at manifest.yml:1060.
So the agentless UI now offers Identity Federation for CloudWatch/EBS/EC2/ECS/S3/RDS/ELB/Lambda/SNS/SQS/Transit Gateway/Billing/AWS Health and the GuardDuty S3 input, and the resulting policy contains a bare role_arn with no cloud-connector signal and no external ID. That is the exact situation the 7.0.0 changelog entry describes as unsupported. The AWS Config template (cel input) correctly keeps its hide, which shows the mechanism is still needed for inputs that lack support.
Recommendation:
Restore the exclusion on every input whose stream template does not emit use_cloud_connectors, and only drop it once the template is migrated. For example, in the guardduty policy template:
inputs:
- type: httpjson
title: Collect Amazon GuardDuty logs via API
description: Collecting Amazon GuardDuty logs via API.
- type: aws-s3
title: Collect Amazon GuardDuty logs via AWS S3 or SQS
description: Collecting Amazon GuardDuty logs via AWS S3 or SQS input.
hide_in_var_group_options:
credential_type: [identity_federation]Alternatively, if these inputs are meant to be covered in this PR, add the cloud-connector plumbing to their templates, e.g. in aws-cloudwatch.yml.hbs / stream.yml.hbs:
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if external_id}}
external_id: {{external_id}}
{{/if}}
{{#if supports_identity_federation}}
use_cloud_connectors: {{supports_identity_federation}}
{{/if}}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| # newer versions go on top | ||
| - version: "7.1.0" | ||
| changes: | ||
| - description: Enable the Identity Federation setup access option on all agentless-enabled policy templates. The CloudFormation quick-create link now deploys the `federated-identity-aws` template, which grants the read-only permissions required by every agentless AWS integration. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/aws/changelog.yml:4
The 7.1.0 changelog only mentions the federation toggle and the CloudFormation link, but this PR also switches four Security Hub/Inspector httpjson streams from manual SigV4 signing to auth.aws; add a changelog entry for that migration.
Details
The diff rewrites authentication for inspector, securityhub_findings, securityhub_findings_full_posture and securityhub_insights: the inline header.X-Amz-Date / header.Authorization HMAC transforms and the pagination re-signing steps are deleted and replaced by an auth.aws block. This is a user-visible behaviour change for existing non-federated users too — credentials now flow through the input's AWS credential chain, so variables that the hand-rolled signer ignored (session_token, shared_credential_file, credential_profile_name, assume_role.duration, assume_role.expiry_window) start taking effect.
The equivalent GuardDuty migration in 7.0.0 got its own changelog entry (Switch the GuardDuty httpjson stream from manual SigV4 HMAC signing to the native auth.aws block.), so the omission here is inconsistent with the precedent set one release earlier.
Recommendation:
Add a second entry under 7.1.0:
- version: "7.1.0"
changes:
- description: Enable the Identity Federation setup access option on all agentless-enabled policy templates. The CloudFormation quick-create link now deploys the `federated-identity-aws` template, which grants the read-only permissions required by every agentless AWS integration.
type: enhancement
link: https://github.com/elastic/integrations/pull/20302
- description: Switch the Amazon Inspector and AWS Security Hub CSPM httpjson streams from manual SigV4 HMAC signing to the native `auth.aws` block.
type: enhancement
link: https://github.com/elastic/integrations/pull/20302🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| processors: | ||
| {{processors}} | ||
| {{/if}} | ||
| auth.aws: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs:74
The four streams migrated to auth.aws ship no policy tests, unlike guardduty which has both a cloud-connector and a legacy-credentials .expected fixture; add equivalent policy tests so the rendered credential block is covered.
Details
data_stream/guardduty/_dev/test/policy/ contains test-httpjson-agentless-cloud-connector.{yml,expected} and test-httpjson-legacy-credentials.{yml,expected}, which assert exactly how the auth.aws block renders for the federated and access-key paths. The four data streams converted in this PR have no _dev/test/policy/ directory at all (the only policy tests in the package are guardduty's), so nothing verifies that use_cloud_connectors, role_arn and external_id are emitted correctly for these streams — and buildkite/integrations passing here does not exercise that rendering.
This matters most for supports_identity_federation: it is a package-level var with show_user: false and no default, so a rendering regression in these blocks would surface only at runtime as an authentication failure.
Recommendation:
Mirror the guardduty fixtures for each migrated stream, e.g. packages/aws/data_stream/inspector/_dev/test/policy/test-httpjson-agentless-cloud-connector.yml:
vars:
role_arn: arn:aws:iam::123456789012:role/ElasticInspectorReadOnly
external_id: abcd1234-ef56-7890-ab12-cd34ef567890
supports_identity_federation: true
data_stream:
vars:
aws_region: us-east-1
interval: 24h
initial_interval: 2160h
lag_time: 5m
http_client_timeout: 30sThen generate the .expected file with elastic-package test policy --generate and confirm it contains:
- auth.aws:
external_id: ${SECRET_0}
role_arn: arn:aws:iam::123456789012:role/ElasticInspectorReadOnly
use_cloud_connectors: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits d65362c — 1 high, 2 medium
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
4 participants
Proposed commit message
Remove the
hide_in_var_group_options: credential_type: [identity_federation]gates from all 23 agentless inputs and pointiac_template_urlat the newfederated-identity-awsCloudFormation template (elastic/cloudbeat#7422), which unconditionally grants the read-only permissions required by every agentless-enabled AWS integration. Users deploy one stack and paste the Role ARN + External ID back into Kibana regardless of which integrations they select. Bumps the package to 7.1.0.Background
Part of https://github.com/elastic/ingest-dev/issues/8887.
When var_groups landed in 7.0.0 (#19828), the Identity Federation setup access option was hidden on every agentless input except GuardDuty's API input, because the published CloudFormation template (
cloud-connectors-guardduty-9.4.0.yml) only granted GuardDuty permissions.elastic/cloudbeat#7422 adds
federated-identity-aws.yml: a single static template whose read-only role covers all agentless-enabled AWS policy templates (transport, metrics, security findings, and per-service inventory permissions). With that template as the quick-create target, the gates are no longer needed — Identity Federation becomes selectable on awshealth, billing, cloudwatch, config, dynamodb, ebs, ec2, ecs, elb, lambda, rds, s3, sns, sqs, transitgateway, securityhub, inspector, and guardduty.No Kibana changes are required: the existing var_group
iac_template_url→ quick-create URL flow handles the new template as-is.Depends on elastic/cloudbeat#7422 merging and the template being published to
s3://elastic-cspm-cft/before this ships.Checklist
changelog.ymlfile.🤖 Generated with Claude Code