Skip to content

GPU pools can't scale to zero on clouds using vanilla cluster-autoscaler #345

Open

Description

@negz

What problem are you facing?

#306 fixed GPU pools cold-starting from zero on GKE. That fix works because GKE's autoscaler has custom support for DRA: it recognises the cloud.google.com/gke-nvidia-gpu-dra-driver=true node label and models the ResourceSlices a not-yet-existing node would publish, so it can tell a scaled-to-zero pool would satisfy a GPU ResourceClaim. Nebius's autoscaler appears to have similar custom support.

The upstream Kubernetes cluster-autoscaler doesn't have this in general. Scale-from-zero relies on each cloud provider's TemplateNodeInfo() synthesising the node a pool would create, and only the cluster-api provider models DRA ResourceSlices there (via the capacity.cluster-autoscaler.kubernetes.io/dra-driver annotation added in kubernetes/autoscaler#7804). The AWS provider passes nil slices, so an EKS GPU pool at zero presents no DRA capacity to the autoscaler and never wakes up. Scaling 1→N works because the autoscaler templates off a real node's live slices; 0→1 doesn't. This is tracked upstream, broadly, in kubernetes/autoscaler#7799.

We can't sidestep this on EKS with Karpenter or EKS Auto Mode, because neither supports DRA yet, which is why we run the vanilla cluster-autoscaler. So any Modelplane cluster on a provider without custom autoscaler DRA support — EKS today, and any self-managed cluster running the stock cluster-autoscaler — can't scale GPU pools to zero.

The immediate user-facing problem: the InferenceCluster API accepts minNodeCount: 0 on GPU pools regardless of provider, but on these providers a pool set to zero silently stays pending with cannot allocate all claims instead of cold-starting. Nothing tells the user it won't work.

How could Modelplane help solve your problem?

Mostly this issue is to track the limitation.

In the meantime, we could stop the API silently accepting a configuration that won't work, e.g. reject or warn on minNodeCount: 0 for GPU pools on providers we know don't support it, and document per-provider support.

Activity

  1. negz commented on Nov 6, 2026

    @negz
    CollaboratorAuthor

    CoreWeave's CKS has the same limitation. Its managed cluster-autoscaler took an idle GPU pool to zero, and a pending pod with a GPU ResourceClaim then got pod didn't trigger scale-up: 1 cannot allocate all claims on every pass. The pool had nodes before it scaled down, so the autoscaler had already cached a node's resources. It still can't model DRA devices for a pool at zero. #655 documents it on CoreWeaveCluster's minNodeCount, as AKS does.

  2. chethanuk commented on Nov 13, 2026

    @chethanuk

    @negz I'd like to do the docs half of this. EKSCluster's minNodeCount doesn't carry the warning AKS and CoreWeave now have, InferenceCluster's minNodeCount has no description, and both the EKSCluster reference example and the Llama 3.1 8B EKS recipe show GPU and CPU pools scaling to zero. The PR would:

    • add the warning to the minNodeCount field on both EKSCluster and InferenceCluster
    • keep the pools in those two EKS examples at one node

    It wouldn't reject or warn on minNodeCount: 0 at runtime, since which providers that should cover isn't settled here. The PR would say "Towards #345" and leave this issue open to track the limitation.

    I've worked out the plan and started on the code, so if this is assigned to me I can raise the PR. Thanks!

  3. added
    g/clustersEnable Platform Engineers to operate inference clusters on any infrastructure as hardware evolves
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestg/clustersEnable Platform Engineers to operate inference clusters on any infrastructure as hardware evolves

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions