You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
GPU pools can't scale to zero on clouds using vanilla cluster-autoscaler #345
#306 fixed GPU pools cold-starting from zero on GKE. That fix works because GKE's autoscaler has custom support for DRA: it recognises the cloud.google.com/gke-nvidia-gpu-dra-driver=true node label and models the ResourceSlices a not-yet-existing node would publish, so it can tell a scaled-to-zero pool would satisfy a GPU ResourceClaim. Nebius's autoscaler appears to have similar custom support.
The upstream Kubernetes cluster-autoscaler doesn't have this in general. Scale-from-zero relies on each cloud provider's TemplateNodeInfo() synthesising the node a pool would create, and only the cluster-api provider models DRA ResourceSlices there (via the capacity.cluster-autoscaler.kubernetes.io/dra-driver annotation added in kubernetes/autoscaler#7804). The AWS provider passes nil slices, so an EKS GPU pool at zero presents no DRA capacity to the autoscaler and never wakes up. Scaling 1→N works because the autoscaler templates off a real node's live slices; 0→1 doesn't. This is tracked upstream, broadly, in kubernetes/autoscaler#7799.
We can't sidestep this on EKS with Karpenter or EKS Auto Mode, because neither supports DRA yet, which is why we run the vanilla cluster-autoscaler. So any Modelplane cluster on a provider without custom autoscaler DRA support — EKS today, and any self-managed cluster running the stock cluster-autoscaler — can't scale GPU pools to zero.
The immediate user-facing problem: the InferenceCluster API accepts minNodeCount: 0 on GPU pools regardless of provider, but on these providers a pool set to zero silently stays pending with cannot allocate all claims instead of cold-starting. Nothing tells the user it won't work.
How could Modelplane help solve your problem?
Mostly this issue is to track the limitation.
In the meantime, we could stop the API silently accepting a configuration that won't work, e.g. reject or warn on minNodeCount: 0 for GPU pools on providers we know don't support it, and document per-provider support.
CoreWeave's CKS has the same limitation. Its managed cluster-autoscaler took an idle GPU pool to zero, and a pending pod with a GPU ResourceClaim then got pod didn't trigger scale-up: 1 cannot allocate all claims on every pass. The pool had nodes before it scaled down, so the autoscaler had already cached a node's resources. It still can't model DRA devices for a pool at zero. #655 documents it on CoreWeaveCluster's minNodeCount, as AKS does.
@negz I'd like to do the docs half of this. EKSCluster's minNodeCount doesn't carry the warning AKS and CoreWeave now have, InferenceCluster's minNodeCount has no description, and both the EKSCluster reference example and the Llama 3.1 8B EKS recipe show GPU and CPU pools scaling to zero. The PR would:
add the warning to the minNodeCount field on both EKSCluster and InferenceCluster
keep the pools in those two EKS examples at one node
It wouldn't reject or warn on minNodeCount: 0 at runtime, since which providers that should cover isn't settled here. The PR would say "Towards #345" and leave this issue open to track the limitation.
I've worked out the plan and started on the code, so if this is assigned to me I can raise the PR. Thanks!
Activity
negz commentedon Nov 6, 2026
CoreWeave's CKS has the same limitation. Its managed cluster-autoscaler took an idle GPU pool to zero, and a pending pod with a GPU
ResourceClaimthen gotpod didn't trigger scale-up: 1 cannot allocate all claimson every pass. The pool had nodes before it scaled down, so the autoscaler had already cached a node's resources. It still can't model DRA devices for a pool at zero. #655 documents it onCoreWeaveCluster'sminNodeCount, as AKS does.Document that EKS GPU and CPU pools can't scale up from zero
chethanuk commentedon Nov 13, 2026
@negz I'd like to do the docs half of this. EKSCluster's
minNodeCountdoesn't carry the warning AKS and CoreWeave now have, InferenceCluster'sminNodeCounthas no description, and both the EKSCluster reference example and the Llama 3.1 8B EKS recipe show GPU and CPU pools scaling to zero. The PR would:minNodeCountfield on both EKSCluster and InferenceClusterIt wouldn't reject or warn on
minNodeCount: 0at runtime, since which providers that should cover isn't settled here. The PR would say "Towards #345" and leave this issue open to track the limitation.I've worked out the plan and started on the code, so if this is assigned to me I can raise the PR. Thanks!
Document that EKS GPU and CPU pools can't scale up from zero