Skip to content

[infra] make the general node groups min and max sizes code-owned, leaving only desired size out of band - #773

Merged
the-gigi-pplx merged 1 commit into
mainfrom
gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership
Oct 21, 2026
Merged

the-gigi-pplx merged 1 commit into
mainfrom
gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership

Conversation

@the-gigi-pplx

@the-gigi-pplx the-gigi-pplx commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Makes the general node groups' min_size and max_size code-owned again, so Terraform ignores only their desired_size. Until now the modules ignored all three after creation. That made a tfvars change to a general pool's bounds a silent no-op, and the ceiling cluster-autoscaler can't exceed could only be moved by an out-of-band EKS call. Each general group's recorded floor now applies only while it serves; a parked group's minimum is zero, so an operator can still scale it to zero through EKS without the next apply raising it again.

What changed

  • eks-cluster/node_groups.tf: aws_eks_node_group.managed ignores only scaling_config[0].desired_size. The comment above it explains the split, and the constraint that comes with it: the provider sends min, desired and max in one UpdateNodegroupConfig call with the desired size from the plan, and EKS requires min <= desired <= max. So raising min above a group's live desired size, or lowering max below it, fails until the live size moves first. Applying a saved plan on a pool cluster-autoscaler has resized since the plan sends the old desired size back, so re-plan right before a min/max apply on a cluster running the autoscaler. An out-of-band min/max change through the EKS API is now drift that every later apply plans to revert.
  • space/locals.tf: the general pool's min_size is general_min_size while general_serving_node_group = "general", and zero otherwise.
  • space/node_groups.tf:
    • general-2 ignores only desired_size. Its min_size is recorded_active_size while it serves, and zero otherwise.
    • The per-AZ groups get the same split. Each one's min_size is its zone's min_size while general_serving_node_group = "general-per-az", and zero otherwise. The per-AZ migration steps say that the minimums move with the serving switch, and that a per-AZ group scaled below its floor has to be grown through EKS before that apply.
  • Two plan-time preconditions, on eks-cluster's managed_node_groups output and on space's general_serving_node_group output, refuse a plan whose new bounds don't contain a group's live desired size. Desired is ignored, so the plan carries the size EKS reported at refresh, and EKS would otherwise refuse the update partway through the apply. Raising cred_authority_node_count on an existing pool, or switching general_serving_node_group to a group parked at zero, now fails the plan with a message saying to move the desired size through EKS first.
  • Descriptions and comments that restated the old contract now describe the new one: general_min_size, general_blue_green_node_group, general_per_az_node_groups, cluster_autoscaler_enabled, node_pools in eks-cluster, the controllers module's cluster_autoscaler input and its discovery comment, the serving-floor precondition, and the general_serving_min_size output.

Rollout

A cluster's first plan after this change applies any configured bounds that differ from the live ones, which an out-of-band EKS resize may have left. A parked general or general-2 plans its minimum down to zero, which leaves its desired size as the plan read it. If a group's live desired size falls outside its configured bounds, the plan fails until the size is moved through EKS or the bounds are recorded in tfvars. This covers every EKS-managed-AMI node group, not only the general ones: cred-authority, and builders where it runs without a custom AMI. Compare each such group's describe-nodegroup scaling config with its tfvars before the first apply. Any bound change sends the desired size the plan read, so on a cluster running cluster-autoscaler a saved plan applied after the autoscaler has resized the pool resizes it back. Setting the new min and max on the live group through EKS before planning avoids that, since the plan then has no bound change to send. Otherwise, plan immediately before applying.

Dependencies

None.

Test plan

  • terraform fmt -check -recursive infra/space/aws/eks/modules
  • terraform validate on eks-cluster, space, controllers and reference/cluster
  • tflint on eks-cluster, space, controllers and reference/cluster
  • A local terraform_data scratch config with ignore_changes on desired confirmed that the plan carries the prior desired size, and that the bounds precondition refuses a min raised above it

@the-gigi-pplx
the-gigi-pplx force-pushed the gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership branch 6 times, most recently from 2c733d8 to 294925e Compare October 21, 2026 09:52
@the-gigi-pplx
the-gigi-pplx force-pushed the gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership branch from 294925e to 05c99a7 Compare October 21, 2026 09:59
@the-gigi-pplx
the-gigi-pplx marked this pull request as ready for review October 21, 2026 10:10

@nickdujay-pplx nickdujay-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 05c99a7.

  • The ownership split is correct: only desired_size is ignored on managed, general_blue_green, and general_per_az. Custom-AMI pools (workers) already sit in aws_eks_node_group.custom and are unaffected. The rollout-mng and cluster-controller UpdateNodegroupConfig paths are scoped to worker nodegroup ARNs, so no in-repo automation writes general/cred-authority/builders bounds out of band that would now show up as drift.
  • Parked pools get min_size = 0 and the serving pool gets its recorded floor. The serving-floor precondition still holds by construction, and the single-pool general_min_size = 0 gate is kept.
  • The bounds preconditions check the planned desired size, which is the refreshed live size because of ignore_changes. They turn EKS's mid-apply min <= desired <= max rejection into a plan-time failure. New per-AZ groups come up with desired = zone min ≤ max, so they pass.

Before merge or first apply (not a code blocker): as the Rollout section says, diff each existing EKS-AMI group's live describe-nodegroup scaling config against tfvars on every cluster, and plan right before applying on clusters running cluster-autoscaler.

@the-gigi-pplx
the-gigi-pplx added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit 51434d1 Oct 21, 2026
26 checks passed
@the-gigi-pplx
the-gigi-pplx deleted the gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership branch October 21, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants