Skip to content

Document that EKS GPU and CPU pools can't scale up from zero - #673

Open
chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/345-gpu-scale-to-zero-docs
Open

chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/345-gpu-scale-to-zero-docs

Conversation

@chethanuk

@chethanuk chethanuk commented Nov 13, 2026 •

Copy link
Copy Markdown

Description of your changes

Towards #345.

EKS GPU and CPU pools can't scale up from zero nodes. The upstream cluster-autoscaler's AWS provider doesn't model the DRA ResourceSlices a new node would publish, so a pod whose claim binds through DRA waits forever on an empty group. The AKS and CoreWeave XRDs already say so on minNodeCount. EKSCluster didn't, InferenceCluster's minNodeCount had no description, and both the EKSCluster reference example and the Llama 3.1 8B EKS recipe showed a GPU pool scaling to zero.

This adds the warning to EKSCluster and InferenceCluster minNodeCount and keeps the EKS examples at one node. It doesn't reject or warn on minNodeCount: 0 at runtime, since the issue hasn't settled which providers that should cover.

I regenerated the models and schema lock with the Crossplane CLI build that nix/deps.nix pins (same hash), since nix isn't available here.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes. Ran the docs manifest validator and the compose-inference-cluster and compose-eks-cluster tests; nix isn't available here.
  • Added or updated tests covering any composition function changes. No composition function changes.
  • Signed off every commit with git commit -s.

@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-673.vercel.app (ready once the site's Content workflow finishes)

The upstream cluster-autoscaler's AWS provider doesn't model the DRA
ResourceSlices a node would publish, so an EKS node group at zero never
wakes for a pod whose GPU or CPU claim binds through DRA. The AKS and
CoreWeave XRDs already warn about the same limit on minNodeCount;
EKSCluster didn't, and InferenceCluster's minNodeCount had no
description at all. The EKSCluster reference example and the Llama 3.1
8B EKS recipe went further and showed a GPU pool that scales to zero.

This adds the warning to EKSCluster and InferenceCluster minNodeCount,
regenerates the models and schema lock, and keeps the EKS examples at
one node. InferenceCluster lists AKS for GPU pools only, matching
AKSCluster. It doesn't reject minNodeCount: 0, since the issue hasn't
settled which providers that should cover.

Towards modelplaneai#345.

Signed-off-by: ChethanUK <chethanuk@outlook.com>
@chethanuk
chethanuk force-pushed the oss/345-gpu-scale-to-zero-docs branch from c4a5f79 to 17992ad Compare November 17, 2026 04:41
@chethanuk
chethanuk marked this pull request as ready for review November 17, 2026 04:58
@haarchri

Copy link
Copy Markdown
Contributor

i wonder If we should try to switch to EKS AutoMode with Karpenter DRA: https://aws.amazon.com/de/about-aws/whats-new/2026/11/amazon-eks-auto-mode-karpenter-dra/

@negz

negz commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs another look at the EKS defaults before it's ready to merge. EKSCluster still defaults minNodeCount to 0, so an InferenceCluster pool that leaves the field out ends up in the state the new warning describes. Separately, the InferenceCluster description raises a question about providers: which clouds it lists, and whether it should list them at all.

Reviewed by modelplane-review 22e194c. Reply, or react 👍 or 👎, on each comment to help tune it.

description: >-
Minimum number of nodes for autoscaling. Set to 1 or
higher for groups that must always be available.
Minimum number of nodes for autoscaling. Set it to 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change the default here rather than only warn about it? minNodeCount still defaults to 0 and maxNodeCount to 8, and the InferenceCluster function passes pool.minNodeCount straight through (compose-inference-cluster/function/fn.py:1464) to scalingConfig.minSize. So a pool with only nodeCount: 1 gets a 0 to 8 node group, and once it idles to zero it can't come back.

Could the default follow nodeCount, as CoreWeave's does? The PR leaves runtime rejection until #345 settles, but a default isn't a rejection.

minNodeCount:
type: integer
description: >-
Minimum node count for autoscaling. Keep GPU pools on

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are AKS CPU pools safe at zero? It looks like they bind through DRA just as EKS ones do: compose-aks-cluster labels them for dra-driver-cpu (fn.py:106-111), and engine pods claim through the dra.cpu DeviceClass. The AKS function's reason for the GPU limit is that no live node publishes ResourceSlices, which would hold for CPU slices too. GKE only models slices for nodes carrying the gke-nvidia-gpu-dra-driver label, so GKE CPU pools may be in the same position. If so, could the CPU clause list them too?

type: integer
description: >-
Minimum node count for autoscaling. Keep GPU pools on
EKS, AKS and CoreWeave, and CPU pools on EKS and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the provider-neutral API name clouds here? This list repeats facts the EKS, AKS and CoreWeave XRDs already carry, and goes stale whenever a provider's autoscaler gains DRA support. What if this said that some providers' autoscalers can't scale a DRA pool up from zero, and pointed to each provider's minNodeCount? Users would then look one level down to learn whether the limit hits their cloud. If the list stays, the question above about AKS and GKE CPU pools applies.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants