Skip to content

fix(eks): apply Karpenter node class updates in place - #817

Merged
Nikolaichukr merged 1 commit into
mainfrom
fix/karpenter-nodeclass-in-place
Nov 2, 2026
Merged

Nikolaichukr merged 1 commit into
mainfrom
fix/karpenter-nodeclass-in-place

Conversation

@Nikolaichukr

@Nikolaichukr Nikolaichukr commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

An AMI update can stall an EKS apply because the provisioner deletes the Karpenter EC2NodeClass before applying its replacement.

The current sequence clears the class's finalizers, deletes it, and recreates it. Karpenter can restore its termination finalizer between the first two steps. When existing NodeClaims still reference the class, deletion remains pending and the replacement manifest is never applied.

Update the class in place with kubectl apply. Mutable changes such as the AMI selector no longer require deleting the class or interfering with Karpenter's finalizers. Karpenter can reconcile the updated configuration through its normal node lifecycle.

The provisioner also:

  • Limits each Kubernetes API request to 60 seconds.
  • Uses a quoted heredoc so manifest values reach kubectl unchanged.
  • Runs Bash with strict error handling so kubectl failures fail the apply.

Regression tests execute the provisioner with a fake kubectl and verify that it issues a single apply operation, preserves literal manifest values, and propagates command failures. All 18 focused IaC tests, the OpenTofu formatting check, and the diff checks passed.

@Nikolaichukr
Nikolaichukr requested a review from a team as a code owner November 2, 2026 22:13
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

OPA/conftest tenant-isolation gate

Result: passed

Gate Result
conftest verify --policy policy/opa passed

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Offline IaC contract tests

Result: passed

Suite Passed Failed Errors Skipped Total Duration Result
Offline IaC contracts 131 0 0 0 131 5.47s passed

Coverage: 95.27% (1509/1584 lines)

Lowest-covered Lambda files

File Coverage Missed lines

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Mutation test critical Lambda boundaries

Result: passed

Suite Passed Failed Errors Skipped Total Duration Result
Mutation tests 17 0 0 0 17 1.60s passed

Coverage: 39.67% (1215/3063 lines)

Lowest-covered Lambda files

File Coverage Missed lines

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Automation gate tests

Result: passed

Suite Passed Failed Errors Skipped Total Duration Result
Automation gate tests 51 0 0 0 51 27.90s passed

Coverage: 88.30% (906/1026 lines)

Lowest-covered Lambda files

File Coverage Missed lines

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Lambda unit tests

Result: passed

Suite Passed Failed Errors Skipped Total Duration Result
Lambda unit tests 255 0 0 0 255 52.53s passed

Coverage: 92.76% (4641/5003 lines)

Lowest-covered Lambda files

File Coverage Missed lines
modules/integrations/splunk_aws_billing/lambda/handler_per_resource_process.py 73.13% 18
modules/platform/forge_runners/forge_trust_validator/lambda/trust_common.py 77.57% 48
modules/integrations/splunk_stuck_workflow_job_dispatcher/lambda/worker.py 78.83% 58
modules/integrations/splunk_aws_billing/lambda/handler_per_service.py 79.41% 14
modules/platform/forge_runners/forge_trust_validator/lambda/trust_validator.py 79.80% 20
modules/platform/forge_runners/forge_trust_validator/lambda/trust_preparer.py 80.00% 10
modules/integrations/github_webhook_relay_destination_receivers/webex_webhook_relay/lambda/handler.py 80.00% 24
modules/integrations/splunk_cloud_data_manager/log_group_reconciler/lambda/log_group_reconciler.py 82.52% 18

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

OpenTofu module tests (latest-stable)

Result: passed

OpenTofu version: 1.13.2

Metric Count
Passed modules 72
Failed modules 0
Total modules 72
Module Result
modules/helpers/ami_policy passed
modules/helpers/ami_sharing passed
modules/helpers/aws_config_recording passed
modules/helpers/cloud_custodian passed
modules/helpers/cloud_formation passed
modules/helpers/dedicated_mac_hosts passed
modules/helpers/ecr passed
modules/helpers/forge_subscription passed
modules/helpers/microvm passed
modules/helpers/opt_in_regions passed
modules/helpers/service_linked_roles passed
modules/helpers/storage passed
modules/infra/eks passed
modules/integrations/github_webhook_relay_destination passed
modules/integrations/github_webhook_relay_destination_receivers passed
modules/integrations/github_webhook_relay_destination_receivers/webex_webhook_relay passed
modules/integrations/splunk_aws_billing passed
modules/integrations/splunk_cloud_conf_shared passed
modules/integrations/splunk_cloud_data_manager passed
modules/integrations/splunk_cloud_data_manager/data_input passed
modules/integrations/splunk_cloud_data_manager/log_group_reconciler passed
modules/integrations/splunk_cloud_data_manager/sec_meta_ec2_tags passed
modules/integrations/splunk_cloud_data_manager_common passed
modules/integrations/splunk_dependency_monitor passed
modules/integrations/splunk_o11y_aws_integration passed
modules/integrations/splunk_o11y_aws_integration_common passed
modules/integrations/splunk_o11y_conf_shared passed
modules/integrations/splunk_o11y_conf_shared/dashboards/arc_runner_operations passed
modules/integrations/splunk_o11y_conf_shared/dashboards/aws_regional_health passed
modules/integrations/splunk_o11y_conf_shared/dashboards/aws_service_limits passed
modules/integrations/splunk_o11y_conf_shared/dashboards/billing passed
modules/integrations/splunk_o11y_conf_shared/dashboards/dependency_probes passed
modules/integrations/splunk_o11y_conf_shared/dashboards/dynamodb passed
modules/integrations/splunk_o11y_conf_shared/dashboards/ebs passed
modules/integrations/splunk_o11y_conf_shared/dashboards/forge_impact passed
modules/integrations/splunk_o11y_conf_shared/dashboards/k8s_control_plane passed
modules/integrations/splunk_o11y_conf_shared/dashboards/lambda passed
modules/integrations/splunk_o11y_conf_shared/dashboards/lambda_control_plane passed
modules/integrations/splunk_o11y_conf_shared/dashboards/metric_ingest passed
modules/integrations/splunk_o11y_conf_shared/dashboards/opencost passed
modules/integrations/splunk_o11y_conf_shared/dashboards/runner_ec2 passed
modules/integrations/splunk_o11y_conf_shared/dashboards/runner_k8s passed
modules/integrations/splunk_o11y_conf_shared/dashboards/runner_usage passed
modules/integrations/splunk_o11y_conf_shared/dashboards/s3 passed
modules/integrations/splunk_o11y_conf_shared/dashboards/s3_control_plane passed
modules/integrations/splunk_o11y_conf_shared/dashboards/sqs passed
modules/integrations/splunk_o11y_conf_shared/dashboards/sqs_control_plane passed
modules/integrations/splunk_o11y_conf_shared/detectors/aws_regional_health passed
modules/integrations/splunk_o11y_conf_shared/detectors/dependency_probes passed
modules/integrations/splunk_o11y_conf_shared/detectors/ec2_runner_health passed
modules/integrations/splunk_o11y_conf_shared/detectors/k8s passed
modules/integrations/splunk_opencost_eks passed
modules/integrations/splunk_otel_eks passed
modules/integrations/splunk_secrets passed
modules/integrations/splunk_stuck_workflow_job_dispatcher passed
modules/integrations/teleport passed
modules/integrations/teleport/tenant passed
modules/platform/arc passed
modules/platform/arc/scale_set passed
modules/platform/arc/scale_set_controller passed
modules/platform/arc_deployment passed
modules/platform/ec2_deployment passed
modules/platform/ec2_deployment/ec2_update_runner_ssm_ami passed
modules/platform/ec2_deployment/ec2_update_runner_tags passed
modules/platform/forge_runners passed
modules/platform/forge_runners/forge_trust_validator passed
modules/platform/forge_runners/github_actions_job_logs passed
modules/platform/forge_runners/github_app_runner_group passed
modules/platform/forge_runners/github_global_lock passed
modules/platform/forge_runners/github_webhook_relay passed
modules/platform/forge_runners/github_webhook_relay/source passed
modules/platform/forge_runners/redrive_deadletter passed

@Nikolaichukr
Nikolaichukr force-pushed the fix/karpenter-nodeclass-in-place branch from 0289351 to a7c2d58 Compare November 2, 2026 22:24
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

OpenTofu module tests (minimum-supported)

Result: passed

OpenTofu version: 1.11.0

Metric Count
Passed modules 72
Failed modules 0
Total modules 72
Module Result
modules/helpers/ami_policy passed
modules/helpers/ami_sharing passed
modules/helpers/aws_config_recording passed
modules/helpers/cloud_custodian passed
modules/helpers/cloud_formation passed
modules/helpers/dedicated_mac_hosts passed
modules/helpers/ecr passed
modules/helpers/forge_subscription passed
modules/helpers/microvm passed
modules/helpers/opt_in_regions passed
modules/helpers/service_linked_roles passed
modules/helpers/storage passed
modules/infra/eks passed
modules/integrations/github_webhook_relay_destination passed
modules/integrations/github_webhook_relay_destination_receivers passed
modules/integrations/github_webhook_relay_destination_receivers/webex_webhook_relay passed
modules/integrations/splunk_aws_billing passed
modules/integrations/splunk_cloud_conf_shared passed
modules/integrations/splunk_cloud_data_manager passed
modules/integrations/splunk_cloud_data_manager/data_input passed
modules/integrations/splunk_cloud_data_manager/log_group_reconciler passed
modules/integrations/splunk_cloud_data_manager/sec_meta_ec2_tags passed
modules/integrations/splunk_cloud_data_manager_common passed
modules/integrations/splunk_dependency_monitor passed
modules/integrations/splunk_o11y_aws_integration passed
modules/integrations/splunk_o11y_aws_integration_common passed
modules/integrations/splunk_o11y_conf_shared passed
modules/integrations/splunk_o11y_conf_shared/dashboards/arc_runner_operations passed
modules/integrations/splunk_o11y_conf_shared/dashboards/aws_regional_health passed
modules/integrations/splunk_o11y_conf_shared/dashboards/aws_service_limits passed
modules/integrations/splunk_o11y_conf_shared/dashboards/billing passed
modules/integrations/splunk_o11y_conf_shared/dashboards/dependency_probes passed
modules/integrations/splunk_o11y_conf_shared/dashboards/dynamodb passed
modules/integrations/splunk_o11y_conf_shared/dashboards/ebs passed
modules/integrations/splunk_o11y_conf_shared/dashboards/forge_impact passed
modules/integrations/splunk_o11y_conf_shared/dashboards/k8s_control_plane passed
modules/integrations/splunk_o11y_conf_shared/dashboards/lambda passed
modules/integrations/splunk_o11y_conf_shared/dashboards/lambda_control_plane passed
modules/integrations/splunk_o11y_conf_shared/dashboards/metric_ingest passed
modules/integrations/splunk_o11y_conf_shared/dashboards/opencost passed
modules/integrations/splunk_o11y_conf_shared/dashboards/runner_ec2 passed
modules/integrations/splunk_o11y_conf_shared/dashboards/runner_k8s passed
modules/integrations/splunk_o11y_conf_shared/dashboards/runner_usage passed
modules/integrations/splunk_o11y_conf_shared/dashboards/s3 passed
modules/integrations/splunk_o11y_conf_shared/dashboards/s3_control_plane passed
modules/integrations/splunk_o11y_conf_shared/dashboards/sqs passed
modules/integrations/splunk_o11y_conf_shared/dashboards/sqs_control_plane passed
modules/integrations/splunk_o11y_conf_shared/detectors/aws_regional_health passed
modules/integrations/splunk_o11y_conf_shared/detectors/dependency_probes passed
modules/integrations/splunk_o11y_conf_shared/detectors/ec2_runner_health passed
modules/integrations/splunk_o11y_conf_shared/detectors/k8s passed
modules/integrations/splunk_opencost_eks passed
modules/integrations/splunk_otel_eks passed
modules/integrations/splunk_secrets passed
modules/integrations/splunk_stuck_workflow_job_dispatcher passed
modules/integrations/teleport passed
modules/integrations/teleport/tenant passed
modules/platform/arc passed
modules/platform/arc/scale_set passed
modules/platform/arc/scale_set_controller passed
modules/platform/arc_deployment passed
modules/platform/ec2_deployment passed
modules/platform/ec2_deployment/ec2_update_runner_ssm_ami passed
modules/platform/ec2_deployment/ec2_update_runner_tags passed
modules/platform/forge_runners passed
modules/platform/forge_runners/forge_trust_validator passed
modules/platform/forge_runners/github_actions_job_logs passed
modules/platform/forge_runners/github_app_runner_group passed
modules/platform/forge_runners/github_global_lock passed
modules/platform/forge_runners/github_webhook_relay passed
modules/platform/forge_runners/github_webhook_relay/source passed
modules/platform/forge_runners/redrive_deadletter passed

@rajeshpodipati rajeshpodipati left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Nikolaichukr
Nikolaichukr merged commit e8369b7 into main Nov 2, 2026
15 of 16 checks passed
@Nikolaichukr
Nikolaichukr deleted the fix/karpenter-nodeclass-in-place branch November 2, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants