[the-gigi-pplx]

Summary

Prepares the EKS modules for cluster-autoscaler on the general pool. Three changes, all off by default except the budgets:

• A blue/green cluster can retire the original general managed node group once general-2 serves, keeping the launch template and node role general-2 runs on.
• A cluster-autoscaler can scale the serving general node group and nothing else. It reads that group's bounds live from its ASG, so Terraform keeps leaving every node group's sizes alone after creation.
• The kube-system controllers this stack installs on general get PodDisruptionBudgets, so the autoscaler can't evict them freely once scale-down is turned on.

What changed

• eks-cluster
  • node_pools gains launch_template_and_role_only (default false). When set, the pool keeps its launch template, node IAM role and policy attachments and gets no node group, so a node group declared outside the module can run on them.
  • New input external_node_group_hosts_addons lets such an outside group count as capacity for the deployment-backed addons check.
  • New output managed_node_groups exposes each EKS-managed-AMI group's ARN, ASG names and scaling config, without depending on custom-AMI pools.
• controllers/cluster_autoscaler.tf (new)
  • An IRSA role, and a policy whose write actions are scoped to the one serving ASG.
  • An aws_autoscaling_group_tag that marks that ASG with a per-cluster key, space.pplx.net/cluster-autoscaler/<cluster>.
  • A cluster-autoscaler Helm release, pinned per Kubernetes minor, that discovers only ASGs with that key.
  • It doesn't use the stock k8s.io/cluster-autoscaler/* pair: EKS stamps that pair on every managed ASG, and the worker pools carry Firecracker sandboxes the autoscaler can't drain safely.
  • Because the group is discovered, its min and max are the ASG's own. A resize through the EKS API moves the autoscaler's bounds, and no apply sends a desired size back to the group.
  • Memory is set per cluster: request equals limit, at least 256 MiB, with GOMEMLIMIT at 90%.
  • Two optional knobs, each passed to the chart only when set:
    • max_graceful_termination_sec caps an evicted pod's grace period during a scale-down drain; whole seconds from 1 to 86400.
    • new_pod_scale_up_delay makes cluster-autoscaler ignore Pending pods younger than the delay when it decides on a scale-up; written like 90s or 2m.
    • Left unset, the upstream defaults (600s and 0s) stay in force, and the release's Helm values are byte-identical to leaving the knobs out.
  • Like the other controller releases, it waits for the general pool.
• PodDisruptionBudgets for the general-pool controllers: controllers/disruption_budgets.tf (new), controllers/main.tf, and the new chart charts/pod-disruption-budget.
  • Every budget is maxUnavailable: 1. None is minAvailable: 1 over one replica, which would allow no disruption and pin the node from scale-down.
  • Multi-replica controllers take the budget their own chart renders. That covers the ALB controller (two replicas) and the external-secrets webhook, which goes to two replicas with preferred hostname anti-affinity. The webhook's validating webhooks fail closed, so with one replica an eviction would reject every ExternalSecret and SecretStore write.
  • Single-replica controllers get their budget from charts/pod-disruption-budget, which sets unhealthyPodEvictionPolicy: AlwaysAllow. Their own charts render none. These are external-dns, the external-secrets controller and cert-controller, and cluster-autoscaler.
  • Under the default policy, the eviction API refuses an unready pod when only one is desired healthy, so a chart budget on a crashlooping singleton would block every drain of its node, including a node-group roll.
  • Each in-repo budget selects its Deployment's pods and references the owning release, so the release's upgrade removes any budget its chart rendered before.
  • The chart refuses an empty selector and any maxUnavailable but a positive whole number or a percentage up to 100%, and its content checksum goes into the release values so a template edit plans.
  • The ALB controller's chart budget sets unhealthyPodEvictionPolicy: AlwaysAllow. The external-secrets webhook's chart can't render that field, so its budget still refuses evictions while both replicas are unready.
  • The external-secrets webhook and cert-controller are now pinned to general; before, only the controller was.
  • The new required controllers input disruption_budget_chart_path names that chart; the space module passes <charts_dir>/pod-disruption-budget.
  • controllers now declares its provider version constraints (aws >= 6.40.0, helm >= 2.17.0).
• space
  • New inputs general_node_group_retired, cluster_autoscaler_enabled, cluster_autoscaler_memory_mib, cluster_autoscaler_scale_down_enabled, cluster_autoscaler_max_graceful_termination_sec and cluster_autoscaler_new_pod_scale_up_delay; the last two default to null.
  • Changing general_serving_node_group moves the discovery tag, so the autoscaler retargets in the same apply.
  • Retirement is refused at plan time until general is parked: its ASG wants no instances and none are left. The ASG is found through the general node group itself. This also stops a cutover and a retirement from landing in one apply.
  • Retirement is also refused on a cluster that doesn't exist yet. A new cluster comes up with general, because its deployment-backed addons wait only on node groups inside module.eks.
  • The general pools are read through one map keyed by pool name: nodes_ready, the autoscaler wiring, the serving-floor precondition and general_serving_min_size all still resolve after general is retired.
  • The ASG IAM ARN format now lives in one local.

The module interface is otherwise unchanged: the ci_* names, moved_ci_access.tf and additional_external_secrets all stay as they are.

Effect on existing clusters

The node groups and the autoscaler are untouched unless a cluster sets the new inputs. The budgets apply everywhere:

• the external-secrets webhook scales to two replicas and rolls onto general, and the cert-controller rolls onto general;
• in-repo budget releases install for the external-secrets controller and cert-controller, and for external-dns where it runs;
• the ALB controller only gains a budget.

Dependencies

• None.

Test plan

• terraform fmt -check -recursive infra/space/aws/eks/modules
• terraform init -backend=false && terraform validate on controllers, eks-cluster, space and eks-cluster/examples/minimal
• tflint on the three modules: no findings.
• A disposable mock-provider terraform test of controllers, kept out of the tree per the repository's test policy for infrastructure-only changes: 11 runs passed.
  • Autoscaler (8 runs):
    • disabled installs nothing;
    • discovery uses only the per-cluster tag, with no --nodes bounds;
    • the tag sits on the serving ASG only, not propagated;
    • an unpinned minor fails the plan;
    • the memory validations, and GOMEMLIMIT following memory_mib;
    • a null scale_down_enabled renders false.
  • Disruption budgets (3 runs):
    • each budget installs only with its workload;
    • each selects the pods its upstream chart's Deployment renders, in kube-system, pinned to the chart versions those renders came from;
    • the single-replica controllers get no chart-rendered budget;
    • no budget sets minAvailable or allows zero unavailable;
    • the webhook runs at least two replicas under its chart budget.
  • Breaking a selector, putting a chart budget back on a singleton, or dropping the webhook replica count each fails the run that covers it.
• A disposable mock-provider terraform test of space plans eight scenarios:
  • general retired with the autoscaler on;
  • retirement refused while general's ASG has a desired capacity of 2, refused at 0 while an instance remains, and allowed once none do;
  • retirement refused on a cluster that doesn't exist yet;
  • the autoscaler enabled without memory, refused;
  • general serving without retirement;
  • a single-pool cluster.
• The cluster-autoscaler Helm values, captured from a disposable mock-provider apply of controllers:
  • With both new knobs unset, they're byte-identical (913 bytes) to the values before the knobs existed.
  • With max_graceful_termination_sec = 120 and new_pod_scale_up_delay = "90s", chart 9.59.0 renders exactly two more args: --max-graceful-termination-sec=120 and --new-pod-scale-up-delay=90s.
  • The validations refuse 0, 1.5, 86401, "1h" and "90", and accept 86400 and "2m".
• helm template of the pinned upstream charts with these values:
  • cluster-autoscaler 9.59.0 renders only --node-group-auto-discovery=asg:tag=space.pplx.net/cluster-autoscaler/<cluster>.
  • The external-secrets webhook runs two replicas, and all three external-secrets components sit on general.
  • aws-load-balancer-controller 1.10.0 renders its budget over two replicas, with unhealthyPodEvictionPolicy: AlwaysAllow.
  • external-secrets 0.16.1 renders only the webhook's budget, and cluster-autoscaler 9.59.0 renders none.
  • The external-dns 1.20.0 Deployment selector equals the budget's.
• helm lint on charts/pod-disruption-budget. The chart refuses an empty matchLabels and any maxUnavailable that isn't a positive whole number or a percentage from 1% to 100% written without a leading zero (0, 00, 010, +1, 0%, 101%, 0.5 among them), and accepts 1, 10, 1%, 25% and 100%.

[nikhilgupta-pplx]

Looks good. Retirement preserves the launch template and role used by general-2, and the guard checks both desired capacity and remaining instances. Autoscaler discovery and write permissions stay scoped to the serving general group, without taking Terraform ownership of its sizing.

Rollout notes, not blockers: the PDBs and the external-secrets webhook/cert-controller move onto general apply even with the autoscaler off, so watch scheduling headroom and webhook readiness on the first apply. Before enabling scale-down, check root-owned kube-system workloads for budgets and drain tolerance.