Repository navigation
[infra] make the general node groups min and max sizes code-owned, leaving only desired size out of band - #773
Merged
the-gigi-pplx merged 1 commit intoOct 21, 2026
Conversation
the-gigi-pplx
force-pushed
the
gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership
branch
6 times, most recently
from
October 21, 2026 09:52
2c733d8 to
294925e
Compare
…aving only desired size out of band
the-gigi-pplx
force-pushed
the
gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership
branch
from
October 21, 2026 09:59
294925e to
05c99a7
Compare
the-gigi-pplx
marked this pull request as ready for review
October 21, 2026 10:10
nickdujay-pplx
approved these changes
Oct 21, 2026
nickdujay-pplx
left a comment
Collaborator
There was a problem hiding this comment.
Approving at 05c99a7.
- The ownership split is correct: only
desired_sizeis ignored onmanaged,general_blue_green, andgeneral_per_az. Custom-AMI pools (workers) already sit inaws_eks_node_group.customand are unaffected. The rollout-mng and cluster-controllerUpdateNodegroupConfigpaths are scoped to worker nodegroup ARNs, so no in-repo automation writes general/cred-authority/builders bounds out of band that would now show up as drift. - Parked pools get
min_size = 0and the serving pool gets its recorded floor. The serving-floor precondition still holds by construction, and the single-poolgeneral_min_size = 0gate is kept. - The bounds preconditions check the planned desired size, which is the refreshed live size because of
ignore_changes. They turn EKS's mid-applymin <= desired <= maxrejection into a plan-time failure. New per-AZ groups come up with desired = zone min ≤ max, so they pass.
Before merge or first apply (not a code blocker): as the Rollout section says, diff each existing EKS-AMI group's live describe-nodegroup scaling config against tfvars on every cluster, and plan right before applying on clusters running cluster-autoscaler.
the-gigi-pplx
deleted the
gigi/pla-6800-port-cluster-autoscaler-node-group-bound-ownership
branch
October 21, 2026 11:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the general node groups'
min_sizeandmax_sizecode-owned again, so Terraform ignores only theirdesired_size. Until now the modules ignored all three after creation. That made a tfvars change to a general pool's bounds a silent no-op, and the ceiling cluster-autoscaler can't exceed could only be moved by an out-of-band EKS call. Each general group's recorded floor now applies only while it serves; a parked group's minimum is zero, so an operator can still scale it to zero through EKS without the next apply raising it again.What changed
eks-cluster/node_groups.tf:aws_eks_node_group.managedignores onlyscaling_config[0].desired_size. The comment above it explains the split, and the constraint that comes with it: the provider sends min, desired and max in oneUpdateNodegroupConfigcall with the desired size from the plan, and EKS requiresmin <= desired <= max. So raising min above a group's live desired size, or lowering max below it, fails until the live size moves first. Applying a saved plan on a pool cluster-autoscaler has resized since the plan sends the old desired size back, so re-plan right before a min/max apply on a cluster running the autoscaler. An out-of-band min/max change through the EKS API is now drift that every later apply plans to revert.space/locals.tf: thegeneralpool'smin_sizeisgeneral_min_sizewhilegeneral_serving_node_group = "general", and zero otherwise.space/node_groups.tf:general-2ignores onlydesired_size. Itsmin_sizeisrecorded_active_sizewhile it serves, and zero otherwise.min_sizeis its zone'smin_sizewhilegeneral_serving_node_group = "general-per-az", and zero otherwise. The per-AZ migration steps say that the minimums move with the serving switch, and that a per-AZ group scaled below its floor has to be grown through EKS before that apply.eks-cluster'smanaged_node_groupsoutput and onspace'sgeneral_serving_node_groupoutput, refuse a plan whose new bounds don't contain a group's live desired size. Desired is ignored, so the plan carries the size EKS reported at refresh, and EKS would otherwise refuse the update partway through the apply. Raisingcred_authority_node_counton an existing pool, or switchinggeneral_serving_node_groupto a group parked at zero, now fails the plan with a message saying to move the desired size through EKS first.general_min_size,general_blue_green_node_group,general_per_az_node_groups,cluster_autoscaler_enabled,node_poolsineks-cluster, the controllers module'scluster_autoscalerinput and its discovery comment, the serving-floor precondition, and thegeneral_serving_min_sizeoutput.Rollout
A cluster's first plan after this change applies any configured bounds that differ from the live ones, which an out-of-band EKS resize may have left. A parked
generalorgeneral-2plans its minimum down to zero, which leaves its desired size as the plan read it. If a group's live desired size falls outside its configured bounds, the plan fails until the size is moved through EKS or the bounds are recorded in tfvars. This covers every EKS-managed-AMI node group, not only the general ones:cred-authority, andbuilderswhere it runs without a custom AMI. Compare each such group'sdescribe-nodegroupscaling config with its tfvars before the first apply. Any bound change sends the desired size the plan read, so on a cluster running cluster-autoscaler a saved plan applied after the autoscaler has resized the pool resizes it back. Setting the new min and max on the live group through EKS before planning avoids that, since the plan then has no bound change to send. Otherwise, plan immediately before applying.Dependencies
None.
Test plan
terraform fmt -check -recursive infra/space/aws/eks/modulesterraform validateoneks-cluster,space,controllersandreference/clustertflintoneks-cluster,space,controllersandreference/clusterterraform_datascratch config withignore_changeson desired confirmed that the plan carries the prior desired size, and that the bounds precondition refuses a min raised above it