Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions apis/eksclusters/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,12 @@ spec:
type: integer
default: 0
description: >-
Minimum number of nodes for autoscaling. Set to 1 or
higher for groups that must always be available.
Minimum number of nodes for autoscaling. Set it to 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change the default here rather than only warn about it? minNodeCount still defaults to 0 and maxNodeCount to 8, and the InferenceCluster function passes pool.minNodeCount straight through (compose-inference-cluster/function/fn.py:1464) to scalingConfig.minSize. So a pool with only nodeCount: 1 gets a 0 to 8 node group, and once it idles to zero it can't come back.

Could the default follow nodeCount, as CoreWeave's does? The PR leaves runtime rejection until #345 settles, but a default isn't a rejection.

or higher for groups that must always be available,
and for GPU and CPU groups: the upstream cluster
autoscaler's AWS provider can't scale a group whose
devices bind through DRA up from zero nodes, so a pod
scheduled to an empty group waits forever.
minimum: 0
maximum: 1000
maxNodeCount:
Expand Down
7 changes: 7 additions & 0 deletions apis/inferenceclusters/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,13 @@ spec:
minimum: 0
minNodeCount:
type: integer
description: >-
Minimum node count for autoscaling. Keep GPU pools on

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are AKS CPU pools safe at zero? It looks like they bind through DRA just as EKS ones do: compose-aks-cluster labels them for dra-driver-cpu (fn.py:106-111), and engine pods claim through the dra.cpu DeviceClass. The AKS function's reason for the GPU limit is that no live node publishes ResourceSlices, which would hold for CPU slices too. GKE only models slices for nodes carrying the gke-nvidia-gpu-dra-driver label, so GKE CPU pools may be in the same position. If so, could the CPU clause list them too?

EKS, AKS and CoreWeave, and CPU pools on EKS and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the provider-neutral API name clouds here? This list repeats facts the EKS, AKS and CoreWeave XRDs already carry, and goes stale whenever a provider's autoscaler gains DRA support. What if this said that some providers' autoscalers can't scale a DRA pool up from zero, and pointed to each provider's minNodeCount? Users would then look one level down to learn whether the limit hits their cloud. If the list stays, the question above about AKS and GKE CPU pools applies.

CoreWeave, at 1 or higher: their cluster autoscalers
can't scale a pool whose devices bind through DRA up
from zero nodes, so a pod scheduled to an empty pool
waits forever.
minimum: 0
maxNodeCount:
type: integer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,5 @@ spec:
nodeCount: 1
zones:
- us-west-2a
minNodeCount: 0
minNodeCount: 1
maxNodeCount: 4
5 changes: 4 additions & 1 deletion docs/manifests/reference/eksclusters.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ spec:
role: GPU
instanceType: g6.12xlarge
diskSizeGb: 200
nodeCount: 0
# GPU groups need at least one node: the cluster autoscaler cannot
# scale a group whose GPUs bind through DRA up from zero.
nodeCount: 1
minNodeCount: 1
maxNodeCount: 8
zones: [us-west-2a, us-west-2b]
gpu:
Expand Down
2 changes: 1 addition & 1 deletion schemas/.lock.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"packages": {
"fs://apis": "7765fe0a737410e9a3f3504455b088e8d2c9ba91fb33046a56acb5441f1ce741",
"fs://apis": "44bfccfab3ee8fdaccac6139473f1f96bb3f1a7d21fc3709b962bb6f3599b962",
"git://https://github.com/crossplane/crossplane/cluster/crds": "14c53aac914d393b5bcc00962fbc90d4be5d35bc",
"git://https://github.com/modelplaneai/cluster-agent/cluster/charts/cluster-agent/files/crds": "a833d3f6f7d9a6ac6083e7453375b58b65ee741a",
"xpkg://xpkg.upbound.io/upbound/provider-aws-ec2:v2.9.4": "sha256:7a0febdfb8da2d59a9fdcdbc2d476811f00566fcb941880313fc2034b0ca6177",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,9 @@ class NodePool(BaseModel):
Maximum node count for autoscaling. Omit for fixed-size pools.
"""
minNodeCount: conint(ge=0) | None = None
"""
Minimum node count for autoscaling. Keep GPU pools on EKS, AKS and CoreWeave, and CPU pools on EKS and CoreWeave, at 1 or higher: their cluster autoscalers can't scale a pool whose devices bind through DRA up from zero nodes, so a pod scheduled to an empty pool waits forever.
"""
name: constr(pattern=r'^[a-z0-9]([-a-z0-9]*[a-z0-9])?$', max_length=40)
"""
Name of the pool. Its nodes carry it as the value of the modelplane.ai/pool label, and the cluster agent observes the pool through an object of this name.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ class NodePool(BaseModel):
"""
minNodeCount: conint(ge=0, le=1000) | None = 0
"""
Minimum number of nodes for autoscaling. Set to 1 or higher for groups that must always be available.
Minimum number of nodes for autoscaling. Set it to 1 or higher for groups that must always be available, and for GPU and CPU groups: the upstream cluster autoscaler's AWS provider can't scale a group whose devices bind through DRA up from zero nodes, so a pod scheduled to an empty group waits forever.
"""
name: constr(min_length=1, max_length=40)
"""
Expand Down
Loading